Quadbike Alpha 1

[Diminished]

Might be time I created a thread for this.

If you've been paying attention to this thread you will have seen a little discussion about the possibility of using DSP-type algorithms to capture tapes, rather than the traditional "waveform walking" time-domain based approach.

I've spent the past few weeks experimenting with this idea, pursuing a hunch that a combination of a software phase-locked loop (PLL) for synchronisation and something Fourier-esque to identify frequencies might produce reasonable results. I had intended just to use a full-blooded discrete Fourier transform for the frequency domain portion, but BigEd suggested deploying the Goetzel algorithm instead, so that is what I am using.

I don't have any code to share just yet, but I finally (ten minutes ago) added CSW export to Quadbike, so some results are now starting to emerge:

There are plenty of errors in the capture, but there is still much room for improvement here. In particular, one advantage of this approach is that every bit captured comes with an associated confidence level, which provides various options for automatic (and, indeed manual) correction of bit errors. I have not attempted any error correction yet.

Anyway, I seem to have a total of about six Acorn tapes remaining in my collection. I used to have more, but I don't know what happened to them. Is there a repository of WAV recordings of tapes available online anywhere that I can use to test this? If not, I would be grateful if anyone digitising tapes could keep their WAV or FLAC files, and permit me access to them for testing this.

I will release the code for QB, but I would like to improve its fidelity a little bit first.

Thanks.

D

[BigEd]

Ooh, I'll follow this closely!

[flaxcottage]

This looks to be very interesting.

It would be extremely interesting if I actually understood the maths behind it.

[vanekp]

Might be time I created a thread for this.

If you've been paying attention to this thread you will have seen a little discussion about the possibility of using DSP-type algorithms to capture tapes, rather than the traditional "waveform walking" time-domain based approach.

I've spent the past few weeks experimenting with this idea, pursuing a hunch that a combination of a software phase-locked loop (PLL) for synchronisation and something Fourier-esque to identify frequencies might produce reasonable results. I had intended just to use a full-blooded discrete Fourier transform for the frequency domain portion, but BigEd suggested deploying the Goetzel algorithm instead, so that is what I am using.

I don't have any code to share just yet, but I finally (ten minutes ago) added CSW export to Quadbike, so some results are now starting to emerge:

Screenshot_20210620_074514.png

There are plenty of errors in the capture, but there is still much room for improvement here. In particular, one advantage of this approach is that every bit captured comes with an associated confidence level, which provides various options for automatic (and, indeed manual) correction of bit errors. I have not attempted any error correction yet.

Anyway, I seem to have a total of about six Acorn tapes remaining in my collection. I used to have more, but I don't know what happened to them. Is there a repository of WAV recordings of tapes available online anywhere that I can use to test this? If not, I would be grateful if anyone digitising tapes could keep their WAV or FLAC files, and permit me access to them for testing this.

I will release the code for QB, but I would like to improve its fidelity a little bit first.

Thanks.

D

no real repository of tapes especially not in a wav format, I have put my collection on Google drive not all are original tape wav files (made with makeuef) some come from other sources but the csw and final uef files are also there plus any documentation I have for the programs.
https://drive.google.com/drive/folders/ ... sp=sharing

[Diminished]

It would be extremely interesting if I actually understood the maths behind it.

I am having a similar problem. Fortunately the Internet provides code samples, and also things like this filter designer, which lets you draw the filter response you want and then just spits out the C code needed to implement it.

I have put my collection on Google drive not all are original tape wav files (made with makeuef) some come from other sources but the csw and final uef files are also there plus any documentation I have for the programs.

Ah, excellent. This looks like exactly what I wanted, thanks!

[Diminished]

Does anyone know under what circumstances single cycles of 2400 Hz tone occur (or, to look at it another way, odd numbers of cycles of 2400 Hz tone)? I'm guessing leader sections can have odd numbers of cycles -- do they happen anywhere else? Could a custom protection mechanism create them at will?

[Coeus]

Does anyone know under what circumstances single cycles of 2400 Hz tone occur (or, to look at it another way, odd numbers of cycles of 2400 Hz tone)? I'm guessing leader sections can have odd numbers of cycles -- do they happen anywhere else? Could a custom protection mechanism create them at will?

Could this be behind it (from the Advanced User Guide):

The problem occurred because it was possible for the serial
ULA to get out of sync for a few bits when the 6850 divide bits
were changed. This tended to corrupt the first character of the
first block in a SAVE, or the first character of any block during
sequential access (since the 6850 is reset for each block during
putbytes). The cure is to write a dummy byte to tape at the
start of a SAVE and the start of every block during putbytes.
If the leader of a prerecorded program is played back, a run in
tone followed by a blip (the dummy byte) followed by more
run in tone will be heard. It is necessary to have a run in
period of high tone after the dummy byte. Preferably this
should be done by polling the 6850 to check if the TDR is
empty, since it is difficult to accomplish if the 6850 is
continually interrupting. The 6850 can then be turned on to
interrupt just before starting the block write operation.

See page 393.

[Diminished]

Could this be behind it (from the Advanced User Guide):

I'm not really sure. I think that explanation just talks about writing a dummy byte, whereas I'm talking about a single 2400 Hz cycle (which would be half of a bit). All I know about this is that I have read in some document that it can happen (although I can't remember in which piece of documentation I read it). It may only affect the leader tone sections.

I am pretty sure at this point that the PLL idea is toast. It seemed like the "correct" way to lock on to an incoming carrier signal, but I can't get it to work reliably enough for some reason. I suspect the problem is that the signal from the tape isn't actually a true carrier, because it chops and changes between 1200 Hz and 2400 Hz tone quite a lot.

I had a plan to deal with this, which was to reconstruct a coherent carrier by frequency doubling the signal (to produce 2400 Hz carrier from the 1200 Hz zero-bits). This can be done essentially by squaring the signal, via the identity:

cos(2x) = 2.cos^2(x) - 1

This will obviously double the 2400 Hz carrier up to 4800 Hz, so you then have to mix this with the original signal (possibly with some phase shift) to fill in the 2400 Hz tone for the 1-bit sections. The resulting signal will contain a lot of other harmonic garbage, but it should be out of the frequency range we care about. You can then just use an aggressive bandpass filter to filter off the 2400 Hz carrier, and feed that into the PLL.

This is good enough to keep the PLL locked during blocks, but unfortunately it isn't good enough for accurate synchronous sampling of Goetzel power levels. The PLL has to fight to remain locked to this fake carrier signal, with its output frequency constantly jittering this way and that. This jitter will sometimes delay the sampling of a bit beyond the time at which the bit has already ended, leading to errors. (I can get some traces of this phenomenon if anyone really wants to see them.)

I did try to add an "adjustment mechanism" whereby the algorithm can examine the PLL's state and try to make a decision about whether to expedite or delay each sample based on the phase error exhibited by the PLL sometime shortly afterwards. I've implemented this, and it sometimes helps, but can also make things worse.

I'll probably keep the PLL code in quadbike as an option for the curious, and so I can continue to tinker with it. It has actually proved quite useful as a deliberately unreliable source of data for developing higher-level error correction methods.

Anyway, the summary is that I don't think the PLL is going to provide sufficiently reliable sync timings for accurate power sampling unless I have some sort of breakthrough, so I'll probably have to come up with something else.

Right now I'm trying to work on "multi-source mode", which is based on the idea that you frequently have multiple recordings of the same tape. It should be possible to amalgamate these recordings for maximum accuracy. Most straightforwardly this would be the left and right channels of a single stereo file, but another common scenario would be having a tape where the same software is recorded on both sides. I also have a cycle-level error correction algorithm in its early stages.

I recently integrated libsndfile, which is a relief, as I can finally load WAVs and FLACs instead of 8-bit raw data ...

[BigEd]

How about running a delay locked loop on the output of the Goertzel detectors? (I'm not absolutely sure if DLL is the term I need here: I mean something which resynchronises on each change. Of course, the UART does something a little like this, except it detects the framing and resyncs on each start bit transition - that's what start bits are for! It then has to count up to 8.)

(Edit: spello!)

[Diminished]

How about running a delay locked loop on the output of the Goetzel detectors? (I'm not absolutely sure if DLL is the term I need here: I mean something which resynchronises on each change. Of course, the UART does something a little like this, except it detects the framing and resyncs on each start bit transition - that's what start bits are for! It then has to count up to 8.)

Yes -- this will likely be one of three synchronisation modes offered.

I'll keep the PLL and slap it with a big OUT OF ORDER sign, as the first option.

As you suggest, cycle-level sync derived loosely from the Goetzel output will be the second option. The problem with this is that the sync source evaporates during leader tone runs, and I want to be able to count exact cycles of leader tone with precision, since this is intended for preservation purposes. I'll therefore probably have to label this option with a warning sign, too, although I suspect this "ad-hoc mode" is likely to achieve lower bit error rates than anything else when dealing with severely degraded media. (Incidentally, one of the nice things about the PLL would have been that you can measure gap lengths on the tape without having to do any extra work -- when the PLL loses lock it just continues to oscillate at its centre frequency, so you can continue to count its cycles for time measurement just as you would if it were locked onto leader tone).

The third and default option will probably just be deriving sync by walking the waveform in the time domain, but still enjoying the benefits of frequency measurements in the frequency domain. You still get to judge the "confidence level" of each bit, which will be vital for further error correction work.

I'll try to put together some sound files of outputs taken from various bits of the PLL, since I found those pretty interesting, particularly the unfiltered phase error.

[Diminished]

It's strange how the PLL + jitter correction seems to work very well on some tapes, but miserably on others.

Beebug magazine cassette vol. 5 number 5 works without losing a bit! Most other tapes generate complete rubbish. I'm not sure what's so special about this one. The jitter correction did have to fix over 600 timing errors, though, so it was hardly a flawless performance.

Before I give up on the PLL, here are some audio treats based on a couple of blocks of tape (bunch of FLAC files):

audible-adventures-in-pll-land.zip

(1.6 MiB) Downloaded 79 times

File 1 is the original signal from my Walkman. File 2 is a 2400 Hz (ish!) carrier rebuilt as described above. File 3 is the output from the PLL's oscillator. File 4 is derived directly from 3 -- it's a digital pulse train which dictates exactly when power sampling should be done. You can hear frequency jitter in all of these 2400 Hz signals. File 5 is probably the most interesting -- it's the phase error from the PLL, which doesn't sound like I expected it to. If it weren't for that shortwave radio whistle, it might make a useful sound effect.

[vanekp]

a note here all my wav files I have used an audio filter applied in Goldwave as per this article https://acorn.huininga.nl/pub/unsorted/ ... %20CSW.pdf before using csw and the nmakeuef to make the final uef file.

[Diminished]

"Mode II sync" is tentatively implemented now -- using a traditional wave walking algorithm to recover a clock signal, and then using that clock signal to trigger sampling of power levels in the frequency domain. This clock recovery algorithm collects a bit more data than some others do: It recognises not just zero-crossings or peaks, but both (and then performs a bad heuristic on the gaps between them).

What I find particularly remarkable about this is that it almost works.

I suspect the errors here are caused by sporadically absent sync pulses causing the frequency domain sampling not to happen. The plan is ultimately to correct for missing sync pulses by interpolating between good ones, but I suspect in this case that a bug (rather than a bad tape signal) may be responsible.

[Diminished]

Scrubbed together the code this morning to replace missing sync pulses through interpolation. Elk Boxer and Snapper now load.

Most other tapes are still erratic, but I'm getting there.

[Diminished]

July appears to have become Cassette Month on Stardot.

I have fixed so many awful bugs over the past week that at this point I am flabbergasted that I managed to capture anything at all prior to this point. I even have some hope that it might be possible to resurrect PLL-sync mode and get some useful results out of it, given some of the mistakes I found on the frequency domain side.

Anyway, I have been continuing to work on walk mode sync (mode II), where peaks and zero-crossings generate sync pulses that are then used to take frequency domain samples. Here is a lewd picture of Elk Boxer:

The traces, top to bottom, are as follows:

1. Original signal from tape
2. Signal from (1), normalised, bandpass-filtered, and then shifted back in time to eliminate the filter delay
3. Sync signal generated from a time-domain walk of (2)
4. Oversampled Goertzel 1200-Hz power
5. Oversampled Goertzel 2400-Hz power

(Yes, I have previously been spelling Goertzel wrongly. Some search-and-replace on the source code will be required).

Encouragingly, this now loads scarybeasts' Fortress recording. He described it as needing "a few bits patched by hand", but Quadbike's mode-II decodes it without any fuss, and this is before any higher-level error correction has been done.

The big things that need doing now are:

- walk mode currently expects a specific polarity, so I need to find a way of detecting the input polarity, flipping the wave if necessary
- fix multi-source mode, for automatically aligning and amalgamating multiple recordings of the same tape; this is implemented but it doesn't work properly yet
- sync mode III, the free-running mode, which I expect to give the highest possible chance of producing a loadable image (at the cost of pedantic accuracy)
- possible sync mode IV, which would be a hybrid between modes II and III, and probably the best option if it could be made to work

There are a lot of other things that need sorting out too; the code is very much an experimental mess.

[vanekp]

Following your progress with interest, it looks like its coming along nicely.

[Diminished]

Just a note to say I am still working on this, just with a bit less urgency than when I started it.

I recently decided to change approach from tracking all four wave features (two zero crossings and two turning points) to just tracking the peaks and valleys only (should have listened to Chris in the first place). Then I had to debug those changes, so I've only just got it producing valid CSWs again.

I'll release something eventually!

[Diminished]

I'm back looking at this again.

I just fixed a nasty bug. It turns out that just a slight drop in signal amplitude (say ~50%) causes a very dramatic reduction in the Goertzel power levels (by >90%). This was resulting in the halfbit decoder incorrectly labelling regions as silent when they aren't. I'm guessing this has something to do with power being the square of the amplitude, but I just reduced the silence threshold by a factor of 100 and this problem went away.

Encouragingly, I am getting better results now. I previously pronounced PLL sync as unpromising, but it seems to be back with a vengeance, and often seems to be producing better captures than the walk method at this point. I have now also coded up the frequency-domain-only based sync method, which (after a false start and a subsequent rewrite) also works pretty well.

So there are now three sync strategies to choose from, and interestingly (although I have not done too much testing) there doesn't really seem to be a particularly clear winner between them, with different tapes favouring different algorithms.

Another thing I added recently was a variety of simple cycle-level error correction options. These are not very effective at this time but they do sometimes help. Quadbike detects half-bits, not whole bits, so for example a complete 1-bit is decoded as two independent, discrete cycles of 2400 Hz. Sometimes only one of these cycles is decoded correctly, leading to a bit that's half-true and half-false. There are now half a dozen selectable strategies for correcting this class of error.

Because CSW doesn't provide any way of storing the confidence level of each bit, I have also come up with a very simple text file format as an alternative output (known as a .cycap file, or "cycle capture", because what the world really needs at this point is another tape archival format). Quadbike will also load these files back in, so in theory you can save to a .cycap; inspect, manually edit or externally process the cycap; and then load it back into Quadbike to finally produce a CSW. I have resisted the temptation to define another binary file format because the idea is to make cycaps as easy as possible to parse and manipulate with scripts. There is no reason why emulators could not load cycaps as another tape format, although they often run to hundreds of megabytes as they are just plain text right now. Here's a sample:

Code: Select all

# Quadbike 1
# Raw Cycle Capture
# ------------------------
# Major version: 1
# Minor version: 1
# Input file:    ../bbc-tapes/electron_boxer_sideA.wav
# Sample rate:   44100
# Total samples: 12219904
# Total cycles:  675144
# Timestamp:     FIXME

# | Cycle Num. | Sample Num. | Prev. Len. | Pwr 0 | Pwr 1 | Confidence | Value

|0|23|24|0.00000218042884619371|0.00000088508869785297|0.00000129534014834075|S
|1|42|19|0.00001048687924578770|0.00000020056419896722|0.00001028631504682048|S
|2|60|18|0.00000391333953362832|0.00000119399046336864|0.00000271934907025968|S
|3|73|13|0.00000371710158977956|0.00001081123732894509|0.00000709413573916553|S
|4|87|14|0.00003266930228789993|0.00001824078174732125|0.00001442852054057868|S
|5|104|17|0.00002074083807683021|0.00002421631582039332|0.00000347547774356311|S
|6|121|17|0.00004142374448750493|0.00001037042930494340|0.00003105331518256153|S

<etc...>

(Each one of these lines is a half-bit. A value of 'S' means silence -- other values are '0', '1' and 'L' for leader tone).

There is still a lot of fiddly stuff I need to fix up, and I still feel like the captures produced contain more errors than they really should, but I'm hoping to release something before too much longer.

[BigEd]

So there are now three sync strategies to choose from, and interestingly (although I have not done too much testing) there doesn't really seem to be a particularly clear winner between them, with different tapes favouring different algorithms.

Interesting! And thanks for the detailed update.

[Diminished]

So there are now three sync strategies to choose from, and interestingly (although I have not done too much testing) there doesn't really seem to be a particularly clear winner between them, with different tapes favouring different algorithms.

Interesting!

I think tape speeds might have something to do with it. Decoding tapes always seemed like a frequency domain problem, but I have come to realise that this isn't as bulletproof as it appears at first. You can wait for a frequency domain transition (where a 0-section ends and a 1-section begins) and then start a counter to count the cycles within that section, resetting the counter at the next transition. The problem is that the tape speed is a variable, not a constant, and for long runs without a transition (the worst default case is 18 half-bits) you can lose sync thanks to good old wow and flutter. Most of my recordings have been done with my crappy old Walkman, which is by no means the most high fidelity piece of audio equipment ever manufactured, and if run from batteries will also tend to droop over the course of capturing a tape.

Quadbike now detects the speed of the tape at both the beginning and end of the input, and averages the two to decide how fast to clock the counter. Here are a couple of examples from my droopy Walkman:

Code: Select all

Early: 1216 Hz, late: 1208 Hz, average 1212.0 Hz, tape speed 1.008.

Code: Select all

Early: 1250 Hz, late: 1241 Hz, average 1245.5 Hz, tape speed 1.036.

Even this isn't really good enough, and there is definitely a case to be made for performing speed measurements throughout the source and building up a speed profile across the entire length of the input. The time-domain sync methods seem to have more immunity to this problem, because they effectively resynchronise every cycle, although they will also benefit from knowing the tape speed accurately -- either for knowing the ideal critical cycle length at which a 0-cycle definitively becomes a 1-cycle (for walk mode) or for setting an accurate filter centre frequency (in the case of the PLL).

[Diminished]

Found some more serious bugs and more unexpected PLL weirdness. There are still masses of things wrong with this.

However I am mindful of the fact that it is completely useless on my hard drive, rather than out in the world where it could be usefully capturing tapes, so once I've persuaded it to build for Windows I'll release something.

Code: Select all

Usage:
  ./quadbike [options] <input file>
    -- input file is audio (e.g. WAV, FLAC) or a .cycap file saved by Quadbike

input options:
  -c (L|R)              pick channel from stereo file (left is default)

sync options:
  -s <method>           sync method; <method> may be one of:
                          walk    wave walker; cycle-accurate [default]
                          pll     PLL; also cycle-accurate
                          freq    Goertzel; tolerant but miscounts leader cycles

output options:
  -o <path>             output .CSW file name (default: quadbike.csw)
  -r <path>             output raw cycle capture (.cycap) file name
  --sparse              with -r, create "sparse" cycap lacking redundant fields
  -!                    allow overwriting of output files

options for walk sync mode (-s walk) only:
  -f                    bandpass-filter the input ("-s walk" only)
  -i                    disable sync interpolation ("-s walk" only)
  -p <pmode>            polarity mode (-s walk only); <pmode> may be one of:
                          auto    fixed polarity, autodetected [default]
                          +       force fixed positive polarity
                          -       force fixed negative polarity
                          block   per-block polarity, autodetected

error-correction options:
  -e <policy>           try bad-pair error correction; <policy> may be one of: 
                          singleton, confidence, zero, one, early, late.

debug options:
  --debug-print-feats   print candidate features ("-s walk" only, very verbose)
  --inspect-dir <dir>   save inspection WAV files to directory (useful)

other options:
  -q                    quieter; suppress progress updates
  -h                    show this help text
  

[BigEd]

Good thinking! Always nice to see something out there.

[Diminished]

Here are some data on the current error rates of Quadbike's various modes in a Google spreadsheet. This is pretty much my entire corpus of test cases at this time. The majority of recordings are from the vanekp archive.

https://docs.google.com/spreadsheets/d/ ... MC/pubhtml

Note that a figure of 0 does not mean "no errors". It is much worse than that -- it means "Quadbike failed to produce a CSW".

The results are quite interesting. Walk sync mode has the option of pre-applying a bandpass filter. In some cases this filter is extremely beneficial. Other times it is appallingly deleterious (see the first two columns of figures). I think at least some of the vanekp tapes had a filter applied to them before being uploaded, so that may play a part.

There is clearly enormous room for improvement in all of the sync modes. The walker needs rewriting, and the Goertzel-sync mode is nowhere near as good as it really ought to be. For most half-decent tapes, though, some mode can likely be found which will produce a loadable capture.

Many of my own tapes (listed last) are in a terrible state, so not too much should be read into those figures.

[Diminished]

EDIT: v1 download has been removed.

[Diminished]

First use of Quadbike in anger.

6/10 of these tapes were captured using -s pll and the other four using -f -s walk. It was not necessary to use the frequency sync mode -s freq, which I consider less ideal for archive-quality captures, owing to its miscounting of leader cycles.

Based on my experience with these tapes, I suspect that walk sync mode should use the input bandpass pre-filter by default, with an option to disable it. Right now it works the other way around (the filter must be explicitly enabled with -f). On an unprocessed source, all of these tapes produced better results with the pre-filter than without it.

[Diminished]

vanekp noticed that some of my Beebug captures wouldn't work in BeebEm. I didn't really investigate why, but it seems that BeebEm has narrower timing tolerances for CSW files.

Interestingly, the files that failed were all created with -f -s walk, and those that succeeded all used -s pll, which I found very interesting. Getting a set of CSWs that worked in all emulators therefore involved recapturing the failing tapes until I got recordings that produced error-free CSWs using the PLL.

This has some interesting repercussions for further QB development, but I would suggest that for the time being, anyone using QB should consider using the PLL as the default option, and then trying other sync modes if the PLL fails.

Post-processing the CSW data before it is saved with some sort of temporal smoothing algorithm on the CSW timings might be something that I could add going forwards, which might help with this problem.

[scarybeasts]

vanekp noticed that some of my Beebug captures wouldn't work in BeebEm. I didn't really investigate why, but it seems that BeebEm has narrower timing tolerances for CSW files.

I spent quite a bit of time getting beebjit's CSW loader able to load (most? all?) of vanekp's awesome tape archive.
In essence this involved fiddling with heuristics for differentiating "1"s from "0"s and in particular locking on to the correct phase when transitioning from carrier to data.

If it would help, I could look at adding a beebjit option to convert an incoming CSW to an output CSW with sanitized (i.e. perfect) pulse timings. This would help in this case, and also help convert vanekp's existing CSWs into ones that could be loaded by all emulators.


Cheers
Chris

[Diminished]

vanekp noticed that some of my Beebug captures wouldn't work in BeebEm. I didn't really investigate why, but it seems that BeebEm has narrower timing tolerances for CSW files.

I spent quite a bit of time getting beebjit's CSW loader able to load (most? all?) of vanekp's awesome tape archive.
In essence this involved fiddling with heuristics for differentiating "1"s from "0"s and in particular locking on to the correct phase when transitioning from carrier to data.

If it would help, I could look at adding a beebjit option to convert an incoming CSW to an output CSW with sanitized (i.e. perfect) pulse timings. This would help in this case, and also help convert vanekp's existing CSWs into ones that could be loaded by all emulators.


Cheers
Chris

You've mentioned this mysterious heuristic before, and several times I've wondered how it works.

I suppose it's up to you if you want to add a "CSW cleaner" to beebjit -- it strikes me as rather out-of-scope for an emulator, and it should really be something that QB can do itself. In fact at one point there was some code in QB which did do something like this, but the timings were corrected before sampling the Goertzel power, rather than afterwards. I removed it, because done this way it seemed to cause more harm than good.

I do have a slight philosophical objection to mucking about with CSW timings, because technically what you output isn't what you captured, but QB's performance isn't good enough and I'm starting to realise that in order to get better results I may have to compromise on this principle.

[BigEd]

In the distant future, all signal processsing will be done with a program called 'beebjit' and no-one will remember why there's a microcomputer emulation at the heart of it.

[Diminished]

In the distant future, all signal processsing will be done with a program called 'beebjit' and no-one will remember why there's a microcomputer emulation at the heart of it.