Archive of FSD files?

[danielj]

The HFE version certainly refused to load from the floppy emulator, sticking on track 39 which is where it would be checking.

[scarybeasts]

[ITRP] 0492: LDA #$7F
[ITRP] 0494: LDX $0490
[ITRP] 0497: LDY $0491
[ITRP] 049A: JSR $FFF1
[ITRP] 049D: LDA #$0E
[ITRP] 049F: NOP
[ITRP] 04A0: CMP #$0E
[ITRP] 04A2: BNE $049D

^ Looks like you've hacked out the test there so that it always loads. LDA #$0E:NOP should presumably be LDA $04A0!

Wasn't me! That's exactly what's on the FSD. The important check is what follows soon after, the check for variance in the sector data across multiple reads. That sort of implies $0E anyway. In fact, if the check for $0E were left in, there's probably a 1/65536 chance the CRC will be accidentally correct on account of the semi-random sector data?


Cheers
Chris

[scarybeasts]

Couple more interesting HFEs attached.

Return To Eden is particularly surprising to me because track 0 -- where the catalog goes! -- is crazy. Aside from being just 5 sectors, those sectors are all declared as 512 bytes in the sector header?!
Of course, DFS doesn't look at that, it just directly attempts 256 byte reads of sectors 0 and 1. These reads are valid data with a valid CRC. I have not studied the 1770 but I thought I read somewhere that it will fail a read request if the requested read size does not match the declared sector size on disc? Perhaps this disc is not 1770 compatible?


Cheers
Chris

[danielj]

In the interim, attached is Sherston Software's Teddy Bears Picnic. It's very interesting, either uses weak bits or fuzzy bits. I'm not sure which because I need to get my hands on such a disc. I suspect weak bits and these can be modeled in HFEv1 by simply omitting clock bits and data bits in a patch of empty disc surface.

Chris, can you post the original FDI of this? Looking at why it's not working with the gotek (Keir thinks it should!)

[Rich Talbot-Watkins]

As far as I know, there wasn't an FDI, only an FSD. And the FSD format has shortcomings here, in that it doesn't model the data properly, but just marks the sector as returning a Data CRC error (&0E) when read, without specifying why. In this case, the fact that there are weak/fuzzy bits requires extra knowledge, as it's just one way of generating that 8271 error.

It would be really good to get a proper raw dump of Teddy Bears' Picnic, or a similarly protected title, to see what's really going on there.

[danielj]

Sorry, I meant FSD - It does work when written to floppy, the extended gap with no clock bits means the FDD cranks up the gain and just spews out spurious fluxes. I suspect it probably was weak bits originally though. It *should* work from the gotek too, Keir's looking into why not.

Ideally, as I've said before, we'd look at recapturing all of these as flux files then we'd be in a position to create whatever was appropriate from them. We could go even further and capture the RF off the head

d.

[scarybeasts]

It would be really good to get a proper raw dump of Teddy Bears' Picnic, or a similarly protected title, to see what's really going on there.

I agree. I've got a setup of suitable drive + scope to see what's actually on the disc surface... but no disc! I would buy one but I haven't found one. If anyone wanted to loan me one "for science", that would be nice and of course I'd report back results and pictures to the community. Perhaps I will also post to Wanted.


Cheers
Chris

[danielj]

Chris, what data did the FSD say was in that region?

- it looks like there's enough variability on a real drive to get a changing CRC in there, but not obviously with a floppy emulator. The emulators can do it but you have to label the region as weak bits which requires HFEv3.

d.

[scarybeasts]

Sorry, I meant FSD

Happy to zip it up and post it along (let me know) but as Rich said it seems unlikely to help you.
My log output gives this for the entire disc:
info:disc:FSD: CRC error sector track 39: $27/$00/$09/$01

It's a normal 256 byte sector (both logical and physical), tagged with a $0E error code. It is up to the interpreter of the FSD file to infer that this means weak bits and which bits are weak. The authoritative heuristics are presumably here:
https://github.com/stardot/beebem-windo ... sc8271.cpp

When I use my converter, 8 bytes of 0x00 data with no clock bits do make it through to the HFE. Hopefully that's a sufficient test case for Keir to debug? I will note it's HFEv1 not v3 so the weak bits are implicit, not explicit, if my reading of HFE is correct.


Cheers
Chris

[danielj]

It's no bug in the emulator, essentially. The original would have been weak bits (as per my MFM example, albeit at multiples of the FM bitcell - 4uS). Writing it to a floppy as a noflux region, there's enough jitter in the spindle that you get a different CRC, but a digital floppy emulator is always going to be running perfectly to time and consequently each read will be identical unless you indicate to the floppy emulator that it needs to be a weak bit area. HFEv3 can do that, but not v1. It's very unlikely that the original was a no-flux

d.

[scarybeasts]

It's no bug in the emulator, essentially. The original would have been weak bits (as per my MFM example, albeit at multiples of the FM bitcell - 4uS).

To make sure we're speaking the same language: I would call this fuzzy bits. I'm not sure terminology is 100% standardized here but I follow this:
https://forum.kryoflux.com/viewtopic.php?t=972

Essentially:
- Weak bits -> no flux
- Fuzzy bits -> borderline timing of flux transitions

Writing it to a floppy as a noflux region, there's enough jitter in the spindle that you get a different CRC, but a digital floppy emulator is always going to be running perfectly to time and consequently each read will be identical unless you indicate to the floppy emulator that it needs to be a weak bit area. HFEv3 can do that, but not v1. It's very unlikely that the original was a no-flux

d.

My mental model is that a noflux region (weak bits as per definition above) would not depend on spindle jitter. In the case of a noflux region, I believe the randomness comes from the drive electronics amplifying noise until ghosts appear.
I'd argue that a full emulation of a floppy drive would also emulate the production of noise from no flux patches of disc.


Cheers
Chris

[vanekp]

Is there a way to convert from/to Kryoflux/FSD formats?

[danielj]

So, with this, we've ruled that out as what happens, I've read the working floppy back via a greaseweazle from a real floppy drive and you still get the "empty" region. Real weak bits would would be an unformatted region of disc = random fluxes, this comes back as just straight no fluxes recorded in that region. Ergo, it's working due to jitter at the starts/ends of the no-flux region rather than random fluxes appearing. The flux dump read back from a real floppy looks the same as the HFEv1. Also read the fluxes back from the floppy emulator.

Just receiving no fluxes isn't enough to confuse the FDC, it needs to be seeing random ones, but you can't express that in HFEv1. Does that make sense?

d.

[danielj]

Is there a way to convert from/to Kryoflux/FSD formats?

No, not directly unless someone writes the software to do it. You can convert kryoflux to HFE for protected or DSD/SSD easily enough though. I think you need to use a real beeb to produce FSDs currently.

[scarybeasts]

Is there a way to convert from/to Kryoflux/FSD formats?

No, not directly unless someone writes the software to do it. You can convert kryoflux to HFE for protected or DSD/SSD easily enough though. I think you need to use a real beeb to produce FSDs currently.

It's good that HFE is a reasonably capable, simple format, and is supported by good open source tooling for converting to and from.

I will attempt to bulk convert billcarr's FSDs to HFEs shortly -- I have a lot of FSDs now! It may also make sense for me to add HFE support to jsbeeb, unless I can hurry up and release my own prototype emulator. Does any other BBC emulator support HFE? MAME perhaps?


Cheers
Chris

[scarybeasts]

So, with this, we've ruled that out as what happens, I've read the working floppy back via a greaseweazle from a real floppy drive and you still get the "empty" region.

That is interesting, and may not match my experimental observations. I've posted about this in other threads but I'll re-post the key details here so people don't have to go ferreting around multiple threads.

The following scope trace tracks the floppy read pin, aka. what GreaseWeazle gets to see, in yellow, vs. actual magnetic flux on the disc, in blue. As you can see, when the flux transitions die, it doesn't take long for the drive to crank up the amplification and start seeing ghosts in the noise.

Real weak bits would would be an unformatted region of disc = random fluxes, this comes back as just straight no fluxes recorded in that region.

I keep some unformatted discs around and they look just the same as the flat blue line -- no flux, not even much going on at the 20mV view. But the read pin is firing pretty randomly.

Ergo, it's working due to jitter at the starts/ends of the no-flux region rather than random fluxes appearing. The flux dump read back from a real floppy looks the same as the HFEv1. Also read the fluxes back from the floppy emulator.

Just receiving no fluxes isn't enough to confuse the FDC, it needs to be seeing random ones, but you can't express that in HFEv1. Does that make sense?

My mental model is that fuzzy bits confuse the FDC. But weak bits confuse the drive electronics. Hence, any differences we are seeing here could come down to different drive behaviors.
What drive are you using? Mine is a Chinon F-051MD. Not much about it online but most importantly, it uses a Motorola MC3470P as the head amplifier. A lovely little chip with the datasheet and pinout still readily findable, nice fat legs, and pre- and post- amplified signal easily tappable. I recently acquired a more modern Teac drive with Teac head amp and will look to check its behavior next week.


Cheers
Chris

[danielj]

Chris, you're completely right - if you look at the blue trace there are still changes in flux, just small and random. If you write 0 flux with a greaseweazle you align all the magnetic domains and write essentially a big string of 0s. You'd have to deliberately write random flux to mimic that, or scramble them with a magnet. The HFE is saying 0 flux, not random small flux. We can be fairly sure it's the variation in rotation speed and lack of clock bits that's enough to give a varying CRC as when we generate the same signal but precisely timed by a digital source (ie the gotek) you get exactly the same behaviour from the fdc's PLL each rotation and fail the protection check.

Upshot of this is 0 in HFE is unchanging flux, but aligned magnetic domains. Real weak bits are unformatted regions where the domains are unaligned. HFEv3 can tell you whether something is 0s (nothing on the read pin) or actually just "blank and unaligned" for which the drive would crank up the gain and produce random signals on the read pin.

Think white noise vs silence. Over the width of the read head all those unaligned domains will average to a flat scope trace but there's noise there, once the head has gone across them and erased the surface, they're all lined up and you have "silence".

This is the reason why HFEv3 was made, to unambiguously encode this sort of thing.

[Pernod]

I will attempt to bulk convert billcarr's FSDs to HFEs shortly -- I have a lot of FSDs now! It may also make sense for me to add HFE support to jsbeeb, unless I can hurry up and release my own prototype emulator. Does any other BBC emulator support HFE? MAME perhaps?

Yes, MAME supports HFE, but currently only revision 0. Your HFE's of Elite and Exile loaded fine, but not Boulderdash which I suspect to be due to other non floppy related timing issues. If you try it then suggest you select a 1770 FDC as it's far more accurate than the current 8271 implementation. The 8271 has been neglected as it's only used in the Beeb and Atom, whereas the WD family were widely used so have seen more improvements over the years.

I've been absent from this topic but have been following with great interest, and hope to eventually port your FSD to HFE conversion into MAME. I'd really like to see HFE being adopted as the archiving format, but it's the owners of the software we need to convince so that they can be re-imaged before it's too late. Converting from FSD to HFE is fine for testing but wouldn't want them widely distributed unless clearly marked as conversions from FSD, as they're approximations due to the limited info available from the FSD.

[danielj]

Totally agree (he says banging the drum again ), we need to get these things into something that can be used to recreate the originals and doesn't leave guesswork in the way As per the discussion, if it's going to go with HFE it needs to be v3 otherwise we lose the weak bits, but I genuinely think that should be in addition to one of the flux formats (i.e. scp). SCP is bigger but is arguably a more accurate raw representation of what came off a disk.

[scarybeasts]

Chris, you're completely right - if you look at the blue trace there are still changes in flux, just small and random. If you write 0 flux with a greaseweazle you align all the magnetic domains and write essentially a big string of 0s. You'd have to deliberately write random flux to mimic that, or scramble them with a magnet. The HFE is saying 0 flux, not random small flux.

That's reasonable. I could argue that its unlikely such a disc surface + drive combo could be created, but that would just make me argumentative
Having the ability to specify no flux in the HFE file actually appeals to me for FDC controller testing on a rainy day.

Anyway, I'll look to emit HFEv3 in the near future -- probably after I've gotten my hands on a Sherston Software disc and reported back what is on it!


Cheers
Chris

[danielj]

Here you go, I used the HxC Floppy software to explicitly mark up that area as "fuzzy", save it as an HFEv3 and lo! it works fine on the gotek...

d.

[vanekp]

Here you go, I used the HxC Floppy software to explicitly mark up that area as "fuzzy", save it as an HFEv3 and lo! it works fine on the gotek...

d.

Hmm it did not work for me on the gotek starts loading (CH."BOOT2") seeks to trk 39 and stops there (drive light constantly on).....

Oh never mind I had a very old version of flash-floppy just updated to the latest Ver. 3.12 and it works.

[scarybeasts]

Here you go, I used the HxC Floppy software to explicitly mark up that area as "fuzzy", save it as an HFEv3 and lo! it works fine on the gotek...

d.

Hmm it did not work for me on the gotek starts loading (CH."BOOT2") seeks to trk 39 and stops there (drive light constantly on).....

That's the protected track and the loader will indeed load that track indefinitely if the bits are constant. Perhaps you need to wait for a stray cosmic ray


Cheers
Chris

[vanekp]

seems none of these FSD or HFE images are publicly available (apart from ones that are posted in the chats here).

[danielj]

seems none of these FSD or HFE images are publicly available (apart from ones that are posted in the chats here).

Well, there aren't any HFE images apart from the ones here, AFAIK, and the FSDs have never been generally released either (although I expect there's nothing preventing that?)

d.

[vanekp]

HFE ones are useful for people that have a gotek, I have converted my few kryoflux one I have to HFE and they seem to work without a problem (Bone Cruncher, Dr Who, Repton3, Repton around the world, Disc Executor).

[CMcDougall]

- ME = "it just copies no problem, even with *COPY 0 1 *.* , did not need *BACKUP 0 1 , or anything else...."

That's interesting. Maybe they de-protected later versions due to problems? The FSD definitely has a really messed up track 2 with 18 sectors and there's a sector overread. My notes from when I first got it working are appended below. (There's a reference to the LOADER code and where / how it executes, maybe that makes it comparable.)

- 358 THE LIVING DAYLIGHTS.FSD

4) At $1263, OSWORD $7F is used to read 10 sector IDs (command $1F) from track
2, with the results going to $2000. A result of 2 is expected for the first
byte, presumably a 40/80 track check?

hi Chris, thanks for indepth details of the must be protected *early* disc
(as BillC has not confirmed from pg2 )
your point 4 gave the game away, as mine is a flippy (40 on sideA, 80 sideB)
so hence does not need to check for a 40 or 80T drive

here is mine copied using UPURS, just checked & still copies easily with *COPY / BACKUP
so did remember rightly

[danielj]

HFE ones are useful for people that have a gotek, I have converted my few kryoflux one I have to HFE and they seem to work without a problem (Bone Cruncher, Dr Who, Repton3, Repton around the world, Disc Executor).

Yes. If you read through this thread you'll see that I'd very much like to work towards that Patience!

d.

[scarybeasts]

hi Chris, thanks for indepth details of the must be protected *early* disc
(as BillC has not confirmed from pg2 )
your point 4 gave the game away, as mine is a flippy (40 on sideA, 80 sideB)
so hence does not need to check for a 40 or 80T drive

Ah interesting. And possibly annoying: I have a Living Daylights disc on its way to me because I wanted to check out the exact copy protection bytes on a real disc. It's hard to tell from the picture I have of it. The label doesn't say "40T this side / 80T other side" or anything like that but it may well have two write protect tabs


Cheers
Chris

[scarybeasts]

hi Chris, thanks for indepth details of the must be protected *early* disc
(as BillC has not confirmed from pg2 )
your point 4 gave the game away, as mine is a flippy (40 on sideA, 80 sideB)
so hence does not need to check for a 40 or 80T drive

Ah interesting. And possibly annoying: I have a Living Daylights disc on its way to me because I wanted to check out the exact copy protection bytes on a real disc. It's hard to tell from the picture I have of it. The label doesn't say "40T this side / 80T other side" or anything like that but it may well have two write protect tabs

Actually there is hope. I just loaded @billcarr's protected FSD in the debugger, hacked the track 2 check to fail, and it says: "Please turn Disc over for 80 Trk version".
So probably they are all flippies? Some protected, some not? I'll report back when I get my hands on another sample of this disc but it'll be a few weeks.


Cheers
Chris