Archive of FSD files?

[scarybeasts]

Wow, that Boulderdash format is weird. I just looked to see if there were any way it could fit, and even reducing gaps 1 and 3 to zero, it still needs 3146 bytes. So is data slightly more squeezed together (written at less than 300rpm?) or is the sector actually incomplete?

My interpretation of track 0 is that it's likely a normal 10x 256 byte track with the following differences:
- The last sector (ID: 9) is declared in the sector header as 512 bytes, but is valid for a 256 byte read.
- There are some copy protection bytes hidden after the end of the sector, i.e. in GAP4.

When creating an FSD, there's no easy way to know how many copy protection bytes will be "overread" after the end of a sector, so the FSD format errs on the side of caution and captures them all.

Once we have a better idea of the floppy controllers' GAP2 constraints, I can re-juggle how I reconstruct FSDs to make this work.


Cheers
Chris

[scarybeasts]

Can you tell how large the inter-sector gaps are on that?

Nice visualization tool. Perhaps cheating for me to look in my code but for sector packing problems:

Code: Select all

        gap1_ff_count = 4;
        gap2_ff_count = 3;
        gap3_ff_count = 3;

Obviously, this won't do. Rich and I chatted and suspect GAP2 is the most likely to be sensitive to messing around like this.

Any chance you can see if any of the attached Arcadians HFEs load on your Gotek + 1770? It's just a conversion of the STH Arcadians.ssd to HFE but with a variety of values for the number of 0xFF's in GAP2: 9, 7, 5, 3. Standard is 11. We think 3 will fail, assuming that's the condition making the Boulderdash HFE fail. But it'll be interesting to see if any of 5, 7, or 9 are acceptable to the 1770.


Cheers
Chris

[danielj]

9 was the only one that worked

d.

[scarybeasts]

9 was the only one that worked

Ah thanks mate, you're a superstar. So we just uncovered a 1770 tolerance that probably isn't documented anywhere else.
The closest I found is in the IPF decoder source code referenced here: https://forum.kryoflux.com/viewtopic.php?f=2&t=265

It contains an allegedly accurate 1770 emulator and it does have this in the state machine for in between sector header and sector data:

Code: Select all

                        case 8:
                                // ignore 28 bytes, the fdc can't read them
                                if (pc->datapcnt < 28) {

My best guess is that that's for MFM mode. Could be a conincidence but half of that for FM mode would be 14 == 8 0xFF's + 6 0x00's ?

Anyway, I'll rejig my FSD heuristics and make a Boulderdash that has a better chance of working.


Cheers
Chris

[Rich Talbot-Watkins]

Makes sense, yeah! And, in general, the required lengths of things are double for MFM than for FM, e.g. MFM requires 12 zero bytes in gaps, compared to 6 for FM. So it seems like they've done some pretty thorough research for that 1770 emulator!

[danielj]

I do love some right nerdy low level floppy shenanigans

[scarybeasts]

Wow, that Boulderdash format is weird. I just looked to see if there were any way it could fit, and even reducing gaps 1 and 3 to zero, it still needs 3146 bytes. So is data slightly more squeezed together (written at less than 300rpm?) or is the sector actually incomplete?

I walked through the protected loader and the answer is that for the protection to work, you need 147 bytes of post-sector protection bytes. This is the same length as the ASCII message appears to the human eye. It's used as an EOR "key". Raw notes appended below for posterity.

So it'll fit, but only if we look at shrinking GAP1 / GAP3. As previously determined, best if we leave GAP2 alone!


Cheers
Chris


- 058 BOULDER DASH.FSD
SHA256: a7514a655027bb39a4310976e69d7ac787a955ef341d259ac8a27c4d55b18d98
Interesting disc! Released a little later in the BBC's life.
1) !BOOT runs boot, which executes at $1100. Immediately launches in a VIA + EOR
bit of deobfuscation, using timing and its own program code as a key, with a
bit of self-modification of course. After a few seconds, branches to $1200.
2) At $1200, fiddles with ROMs, looks like it's looking for the infamous REPLAY
ROM! ($1203==REPLAY). Finishes with that at $125A. Writes the first intro
screen at $12C0. Finishes with that at $12CE.
3) At $12DB, does *DISK then *R.Welly
4) What?? If you *R.Welly directly, it loads. Unclear why bother with all the
deobfuscation in the first bit.
5) Ok, Welly executes at $1100 and has the same sort of unpacker. Execution
gets to $1200, REPLAY check. At $1294, does *DISK then *LOAD Welly2 A00. It
is EOR unpacked, at then at $12C4, we have a jump to $0AC1.
6) At $0AD6, JSR $0A38 uses OSWORD $7F to seek to 0. At $0B15, JSR $0A48 reads
track 1, sector 0, size 1024. Result must be $00. At $0B9B, JSR $0A2B reads
track $17, sector 2, size 512, into $1900. Then loop back to $0B6C, $0B9B to
JSR again, read track $10, sector 3, size 256 to $1B00. Back to $0B6C, but this
time branch to $0BA4.
7) At $0BDC, JSR $0A2B, here we go! Read track 0, sector 9, size 512, to $0400.
This is a sector overread, the sector is really 256 bytes, so the post-sector
bytes starting with the checksum and then GAP4 bytes end up at $0500.
At $0BDF, the data read to $1900 is unpacked using data at $0500 as a key.
Routine at $0C9B is called for every byte:
[ITRP] 0C9D: LDY #$00
[ITRP] 0C9F: LDX #$92
[ITRP] 0CA1: EOR $0502,X
[ITRP] 0CA4: EOR $0502,Y
[ITRP] 0CA7: INY
[ITRP] 0CA8: DEX
[ITRP] 0CA9: CPX #$64
[ITRP] 0CAB: BNE $0CA1
So it appears the amount of "overread" data required for correct unpacking is
$92 + $02 + 1 == 149 bytes, or 147 bytes excluding the sector checksum.
There's some check, correct execution branches to $0C26. RTS at $0C3E jumps
to $FF89, a further couple of RTS's land at $8B7D, in BASIC. Later, program
code hits at $1100... another unpacker... zzzzz...

[scarybeasts]

I do love some right nerdy low level floppy shenanigans

Does the attached Boulderdash work at all? (If not, what are the symptoms?)

Warning -- loading Boulderdash is a real nail-biter, it takes a long time and the disc will pause for seconds at a time while it does some CPU intensive unpacking.


Cheers
Chris

[danielj]

Will give it a spin this evening!

[Rich Talbot-Watkins]

Warning -- loading Boulderdash is a real nail-biter, it takes a long time and the disc will pause for seconds at a time while it does some CPU intensive unpacking.

Is that just because of the decryption code, or is it because the disk controller is having to retry multiple times to find sectors (maybe because of shortened gaps)?

[scarybeasts]

Warning -- loading Boulderdash is a real nail-biter, it takes a long time and the disc will pause for seconds at a time while it does some CPU intensive unpacking.

Is that just because of the decryption code, or is it because the disk controller is having to retry multiple times to find sectors (maybe because of shortened gaps)?

It's mostly the decryption code, of which there are many many stages.
But the disc reading also isn't the fastest. The non-protected tracks are all a strange 5 sector format and the sectors are large and read individually (separate 8271 read commands), which introduces plenty of disc rotational latency. The shortened gaps might behave differently on real hardware, but my emulated 8271 doesn't care. When I get a Gotek, I can check its tolerances for short gaps and make it care as appropriate


Cheers
Chris

[danielj]

I'll have to dig out a 5.25" drive and try it on the beeb properly.

Were these definitely all 80 track images btw? Surely lots of stuff was distributed dual format?

d.

[scarybeasts]

I'll have to dig out a 5.25" drive and try it on the beeb properly.

Were these definitely all 80 track images btw? Surely lots of stuff was distributed dual format?

d.

Most of the FSDs are captured 40 track. I currently emulate an 80 track drive so the latter 40 tracks will end up unformatted. The HFEs are likely declaring as 80 track but only populating the first 40. The "double step" property in the header shouldn't be set.


Cheers
Chris

[danielj]

Does the attached Boulderdash work at all? (If not, what are the symptoms?)

Warning -- loading Boulderdash is a real nail-biter, it takes a long time and the disc will pause for seconds at a time while it does some CPU intensive unpacking.

That worked perfectly

d.

[danielj]

Just for added bonus, converted the HFE to SCP, wrote it to a 5.25" disk and it loaded fine

d.

[scarybeasts]

Just for added bonus, converted the HFE to SCP, wrote it to a 5.25" disk and it loaded fine

d.

Oh that's so cool, thanks for trying it! I'm over the moon we guessed the problem and fixed it first time around to be honest.

If we can restore this from the FSD we can probably restore anything. There are a few other known "tough" cases. I'd be happy to post more HFEs if you were interested in trying them.

Additionally, is there any particular game you'd like converted so you can enjoy the original experience? I may have it in my current pile of FSDs.


Cheers
Chris

[danielj]

Can you just bulk convert the lot and I'll work through them?

d.

[scarybeasts]

Can you just bulk convert the lot and I'll work through them?

d.

Ok, will do once I've gotten all the latest ones from billcarr. I've also got a few bugs to fix -- I do seem to have broken The Living Daylights by packing things more correctly

In the interim, attached is Sherston Software's Teddy Bears Picnic. It's very interesting, either uses weak bits or fuzzy bits. I'm not sure which because I need to get my hands on such a disc. I suspect weak bits and these can be modeled in HFEv1 by simply omitting clock bits and data bits in a patch of empty disc surface.

I bet it won't work in Gotek / Flashfloppy, because you'd need to emulate a patch of empty disc surface as noise coming off the drive's peak detector. Maybe it would work as SCP? Depends on if the format / GreaseWeazle can reconstruct a piece of blank disc surface from a file that has a patch with no clock bits and no data bits.


Cheers
Chris

[CMcDougall]

I do seem to have broken The Living Daylights by packing things more correctly

would not worry about that, it's utter pants

how can you break it, I still have the original disc from BITD (cost £1) ,
and it just copies no problem, even with *COPY 0 1 *.* , did not need *BACKUP 0 1 , or anything else....
it was already in the STH archive, so never archived mine :

@BillCarr, is this same as yours?

[danielj]

Fuzzy bits (MFM) from a sherston title:

HFEv1 and v3 can deal with this as it's basically tightly packed fluxes rather than unformatted segments. Flashfloppy can deal with it too (and unformatted segments!)

I can't see anything like that in the HFE you uploaded?

d.

[scarybeasts]

I do seem to have broken The Living Daylights by packing things more correctly

would not worry about that, it's utter pants

Agreed. I earlier commented to someone else, "such good protection for such an awful game"

how can you break it, I still have the original disc from BITD (cost £1) ,
and it just copies no problem, even with *COPY 0 1 *.* , did not need *BACKUP 0 1 , or anything else....
it was already in the STH archive, so never archived mine :
scan0042theLivingDaylightsDisc.jpg
@BillCarr, is this same as yours?

That's interesting. Maybe they de-protected later versions due to problems? The FSD definitely has a really messed up track 2 with 18 sectors and there's a sector overread. My notes from when I first got it working are appended below. (There's a reference to the LOADER code and where / how it executes, maybe that makes it comparable.)


Cheers
Chris

- 358 THE LIVING DAYLIGHTS.FSD
SHA256: c7530cdc837ff0a6ae45ac6a827e91dd352c5fb3e04a5965463b25310860265e
1) Track 2 is interesting. It is 18 sector but looks like 10 sectors of 256
bytes. The logical sector IDs for the first 10 sectors run 09 -> 00 inclusive.
Then, there are 8 sectors with sector headers of 4x 00 bytes.
2) LOADER executes at $1200. It does an EOR based unpack of itself at $1239,
and when execution arrives at $1239, it is JSR $1338.
3) At $1260, a JSR $1340 routine uses OSWORD $7F to seek to 0.
4) At $1263, OSWORD $7F is used to read 10 sector IDs (command $1F) from track
2, with the results going to $2000. A result of 2 is expected for the first
byte, presumably a 40/80 track check?
NOTE: this leaves the head position about half-way through the track.
5) At $12A4, a JSR $1340 routine uses OSWORD $7F to seek to 0.
6) At $12A7, a JSR $134A routine uses OSWORD $7F to seek to 2.
7) At $12AD, OSWORD $7F is used to read track 2, sector 0, 1 sector, 256 byte
sized.
I believe a couple of attempts are allowed but this is sensitive to current
head position and therefore seek timing sensitive! It will only work if the
read starts in the first half of the track, otherwise it'll hit the sector
headers with logical track $00 and bail.
The timing seems to work: the seek to 0 and then back to 2, with default seek
and settle timings, leaves the head position a little over 20% into the track.
Read result is $0E (unchecked?) but the sector read bytes, starting from
offset $80 (i.e. the start of the overread) are expected to be:
(6502db) m 13dc
13DC: 5D 30 FF FF FF FF FF FF FF FF FF 00 00 00 00 00

[scarybeasts]

Fuzzy bits (MFM) from a sherston title:

Oh cool, I was very curious as to what's on the disc surface. It's just MFM? Do you have more details? I'd love a raw flux file to study and learn.

- Is it valid MFM timing? i.e. are the transitions correctly spaced?
- Is it valid MFM encoding? i.e. would an MFM controller read data out of it? (On this question, I'm curious if they just wrote normal MFM and relied on an FM mode controller making a hash out of that, or if the MFM is in some way abnormal.)

HFEv1 and v3 can deal with this as it's basically tightly packed fluxes rather than unformatted segments. Flashfloppy can deal with it too (and unformatted segments!)

I can't see anything like that in the HFE you uploaded?

The HFE has an absence of transitions because that was my incorrect guess as to what was on the disc surface. All the loader needs to see is non-deterministic bits coming out of a certain spot in a sector. Absence of transitions is one way to do that.

I can certainly generate HFEs differently. I'd prefer to keep the HFEv1. I did see that there's a superfluous 0 bit after every FM clock bit and data bit. Do I need to just set those to 1? Something else?


Cheers
Chris

[danielj]

Not sure - I'll share that SCP so you can have a look at it...

Here's what was in your HFE? I'm assuming we're looking at the last sector of track 39?

[scarybeasts]

Not sure - I'll share that SCP so you can have a look at it...

Here's what was in your HFE? I'm assuming we're looking at the last sector of track 39?

Yes. The first few bytes of the sector actually must be valid (they are executed!) and then the loader expects read-to-read variance a few bytes in. My notes from stepping through it below. The loader wouldn't care (or be able to tell?) if the variance came from weak bits or fuzzy bits.


Cheers
Chris

- 186 THE TEDDY BEARS' PICNIC SIDE 1.FSD
SHA256: 7bd415a4ade19250ba34888840a9dfdbe478a3d328a4e11d11bde6a39cdb80ea
Track 39 sector 9, 256 bytes, very special. It gives CRC error $0E when read,
and is expected to contain a few bytes of valid code followed by weak bits.
In the code sequence below, OSWORD &7F is expected to return $0E at &04A0, after
reading track 39 sector 9 into $1100.
Then $1100 is called to check the start of the sector is a few simple
deterministic bytes.
Then at $0539 there's a loop that repeats the same sector read into $1100
until a difference between different reads is detected in the range $1118 to
$111F.
[ITRP] 0536: JSR $0492
...
[ITRP] 0492: LDA #$7F
[ITRP] 0494: LDX $0490
[ITRP] 0497: LDY $0491
[ITRP] 049A: JSR $FFF1
[ITRP] 049D: LDA #$0E
[ITRP] 049F: NOP
[ITRP] 04A0: CMP #$0E
[ITRP] 04A2: BNE $049D
[ITRP] 04A4: LDA #$3D
[ITRP] 04A6: JSR $1100
...
[ITRP] 1100: STA $0737
[ITRP] 1103: RTS
...
[ITRP] 04A9: LDA #$32
[ITRP] 04AB: STA $03A1
[ITRP] 04AE: RTS
...
[ITRP] 0539: LDA $1118
[ITRP] 053C: STA $81
[ITRP] 053E: LDA $1119
[ITRP] 0541: STA $82
[ITRP] 0543: LDA $111A
[ITRP] 0546: STA $83
[ITRP] 0548: LDA $111B
[ITRP] 054B: STA $84
[ITRP] 054D: LDA $111C
[ITRP] 0550: STA $85
[ITRP] 0552: LDA $111D
[ITRP] 0555: STA $86
[ITRP] 0557: LDA $111E
[ITRP] 055A: STA $87
[ITRP] 055C: LDA $111F
[ITRP] 055F: STA $88
[ITRP] 0561: JSR $0492
[ITRP] 0564: LDX #$00
[ITRP] 0566: LDA $81,X
[ITRP] 0568: CMP $1118,X
[ITRP] 056B: BNE $0577
[ITRP] 056D: INX
[ITRP] 056E: CPX #$07
[ITRP] 0570: BNE $0566
[ITRP] 0572: JMP $0536
[ITRP] 0575: BEQ $0536

[danielj]

In the HFE it looks formatted - you can see the fluxes... I think v3 will allow unformatted regions?

d.

[scarybeasts]

In the HFE it looks formatted - you can see the fluxes... I think v3 will allow unformatted regions?

d.

It's tricky to spot but at offset 0xf4a00, there are 32 consecutive 0 bytes representing 8 data bytes where all 16 of the FM clock and data bits are 0. I wonder if that is sufficient to carry something interesting through to the SCP and therefore disc surface?


Cheers
Chris

[danielj]

It's tricky to spot but at offset 0xf4a00, there are 32 consecutive 0 bytes representing 8 data bytes where all 16 of the FM clock and data bits are 0. I wonder if that is sufficient to carry something interesting through to the SCP and therefore disc surface?

Yup, good enough for the SCP - that wrote out properly. the HFE you'll need to use v3 to specify a "no flux" region or else work out a way of confusing the FDC.

[scarybeasts]

Yup, good enough for the SCP - that wrote out properly. the HFE you'll need to use v3 to specify a "no flux" region or else work out a way of confusing the FDC.

Just to make sure I understand -- are you saying the written-out Teddy Bears' Picnic disc actually loaded??!


Cheers
Chris

[danielj]

Yup! (greaseweazle knows how to write no flux regions)

[Rich Talbot-Watkins]

[ITRP] 0492: LDA #$7F
[ITRP] 0494: LDX $0490
[ITRP] 0497: LDY $0491
[ITRP] 049A: JSR $FFF1
[ITRP] 049D: LDA #$0E
[ITRP] 049F: NOP
[ITRP] 04A0: CMP #$0E
[ITRP] 04A2: BNE $049D

^ Looks like you've hacked out the test there so that it always loads. LDA #$0E:NOP should presumably be LDA $04A0!