Any summary of WAV to UEF/CSW software (multi-platform)

[CommanderCoder]

Hi,

I dug out some old BBC tapes from my early teenage years and bought a cassette to MP3 'walkman' to sample the content. The first one I did was BEEBMUNCH. I have a strong suspicion this was the first game I bought, but it might have been Philosopher's Quest.

I've converted the mp3 to wav 16bit mono 44100hz. I'm trying to convert the wav to uef with wave2uef.py.

It didn't work and so I change the program to look for peaks and troughs, rather than crossing the zero boundary. That worked better but still had some problems as some peaks have a little wobble.

I've read the forums and I'm going around in circles with this, but I wondered if anyone has compiled a summary of the software and techniques which can do this conversion? I guess it is a bit hit and miss but the wav file looks pretty good and it almost works.

Attached is my mp3 file (wav wouldn't upload) and the output I got by looking at peaks and troughs.

I think spotting the peaks, troughs will help me to make a square wave representation, but I get the feeling all of this has been solved before.

[Diminished]

Quadbike 2 (which is still in development and hasn't been released yet -- and yes, is cross-platform, and FOSS) was able to CSW this, although I had difficulties with the data CRC on one block at first.

Interestingly this tape seemed to have relatively low 2400 Hz power compared to 1200 Hz power, so on a hunch I tried increasing QB_2400_ARBITRARY_POWER_SCALER from 1.1 to 2.0 for this tape, which worked.

This has alerted me that I need to scan across the whole tape test corpus to find an optimal empirical value for this constant (or maybe expose it as a command-line option). I just eyeballed an arbitrary value during development, which may or may not be optimal (probably not).

Here's the CSW. It loads in beebjit and b-em (and verifies 100% OK with cswblks.php). I haven't tested BeebEm. If you want a UEF, you can try running the CSW through MakeUEF. It might work, or it might not. I have come to realise that CSWs are to a certain extent in the eye of the beholder.

beebmunch.csw

(46.42 KiB) Downloaded 26 times

Thank you for the extra audio file, which has been added to my test corpus. If you have any more tapes you fancy making recordings of and posting, please do it -- the audio data is valuable to me. I would suggest trying to record further tapes with a slightly higher volume level if possible, though -- this one was a bit quiet.

The main reason I had wanted to go to the January ABUG was that I was hoping some generous individuals might be prepared to let me go mad with their Acorn tape collections so I could digitise as much test audio as I could manage in one weekend. This would have allowed me to further bolster my collection of audio test data, but thanks to the railways it isn't going to happen.

[CommanderCoder]

Thanks a lot. I have a few tapes which are original and a bunch of tapes which contain my own programs from when I was about 12. I got a disk drive a year or so later as tape were so slow, and I've managed to grab most of those discs! I know the quality of my tape programs will be embarrassing so I'd like to be the first to see them!

If you're interested in adding me to your repository for QuadBike 2, I can run it over some more of my tape recordings. I'm happy to send some to you too once I find a few quiet moments over the next couple of weeks to snap them all. I'll PM you a link to a folder on my web page where I'll put the mp3s: http://commandercoder.com/

I tried using MakeUEF v2.3 but it reported the "incorrect number of pulses by 1" so I couldn't test this on BeebEm.

[Diminished]

I tried using MakeUEF v2.3 but it reported the "incorrect number of pulses by 1" so I couldn't test this on BeebEm.

Ah, yes. Sorry. That's a bug, caused by a recent invasive change to the way output polarity is handled. I don't test with MakeUEF regularly, so I didn't realise it might cause a fatal error. It's literally in the list of things to fix:

Code: Select all

// [ ] W: Pulse count mismatch: 1477513 in header, 1477514 in body. -- prob caused by silent span pulses now being done at cycle level rather than CSW level

I'll fix that bug, and get you another CSW when I can. (This sort of thing is why there is no release yet -- I have ripped up and replaced quite a lot of code over the past month or two, and not done any wide testing on it yet, and this situation is only really just beginning to stabilise).

If you're interested in adding me to your repository for QuadBike 2, I can run it over some more of my tape recordings. I'm happy to send some to you too once I find a few quiet moments over the next couple of weeks to snap them all. I'll PM you a link to a folder on my web page where I'll put the mp3s: http://commandercoder.com/

I could let you have a current copy of the code -- can you build it yourself? What operating system are you using? You'll need libsndfile, and zlib. It should build fine on MacOS and Linux, but fixing it up to work with Microsoft's compiler is generally something I only bother to do prior to a release, so right now I don't have any Windows binaries, or a robust build setup for Visual Studio.

[Diminished]

Frustratingly I don't seem to be able to produce a CSW right now that will work perfectly with everything. I am very tempted to define a new tape file type which is slightly higher-level than CSW but lower-level than UEF (essentially a bitstream, rather than a pulse stream like CSW), which would make all these stupid timing problems go away. Inevitably, though, I would end up having to be the one to add support for it to all the emulators.

beebjit in particular has problems, as it has an odd heuristic for CSW loading and does something weird with phase. Both these CSWs appear to load in b-em, and they might work with MakeUEF.

[CommanderCoder]

I can compile on anything pretty much. I use MacOS / Xcode and Linux / GCC regularly but I have spent many years using Windows / Visual Studio although I'm a bit rusty.

I'm familiar with UEF but not CSW. I'll have to do a little research. This seems useful: https://acorn.huininga.nl/pub/unsorted/ ... CSW2FORMAT

[Diminished]

OK, I'll sort something out.

On the face of it, CSW is very simple. It just measures the pulse lengths (in samples) of positive and negative half-cycles, and then RLE-compresses those pulse lengths into a stream (numbers in red):

You'd think this would be easy to decode -- just set a threshold between the length of a 1200 Hz half-cycle and the length of a 2400 Hz half-cycle, and make a decision about whether the bit is a 1 or a 0 based on those pulse lengths. (I could be wrong, but I think that this simple approach is what b-em does.)

However, things get sticky because the phase of the signal is not guaranteed to be zero. In other words, the switch from one frequency to the other does not necessarily occur at the zero-crossing point, owing to the signal being differentiated an unknown number of times in the process of getting it on and off the tape. For example, if the phase is 90 rather than 0, the pulse length changes at the peak of the wave, not at the zero-crossing, and now things get messy, and unfortunately the results are open to interpretation, particularly if the phase is not known:

Quadbike uses CSW as its output format, but it does not actually measure positive and negative half-cycles at all. Instead it uses frequency domain analysis (Goertzel's algorithm, a close relative of the discrete Fourier transform) to bypass the problem of analysing the waveform entirely. In short, a lovely piece of maths just magically turns input into 1200 Hz power and 2400 Hz power [*] :

In other words, Quadbike decodes 1s and 0s without needing to inspect the waveform at all, so what you get is a bitstream (technically a half-bit stream), not a pulse stream. However, I didn't want to have to go to the trouble of writing a whole UEF output implementation (at least not yet), so CSW was the only sensible choice for an output format. Hence 1-bits are encoded as four synthetic CSW pulses (a pair of 2400 Hz cycles), and 0-bits are encoded as two synthetic CSW pulses (one 1200 Hz cycle). As a result, at the moment Quadbike always produces CSWs with a phase of zero. This seems to confuse beebjit in particular, which I think is expecting "real-world" CSWs with wacky phase issues.

This is why the most suitable output format for Quadbike would actually be something new, which just encodes runs of half-bits, and suffers no ambiguity of interpretation. Or UEF -- but that is a big, complex format, and there would be a million bugs and issues to sort out, and I don't want to cram an entire UEF encoder into Quadbike.

[*] (It's actually trickier than this, because you need to synchronise to the waveform somehow in order to count the bits -- how many 1-bits and 0-bits are there in each of the power1 and power0 pulses shown? Are you sure? Tape speed is a variable, not a constant! QB2 actually has three different schemes for performing this synchronisation, with varying results depending on the tape.)

[Fraser]

The program csw.exe uses Fourier Fast Transform to get the pulselengths. I expect it uses common library code rather than writing all the mathematics code into it.

My program MakeUEF will have a new version soon. I don't see any point in writing another program that does the same job because I know how much hard work it is.

I don't think emulators are emulating the cassette player properly. It's a separate device with a one way serial communication system so it could be emulated by a separate process.

[vanekp]

saving them as mp3 is not recommended as mp3 removes information that may be required, you may have a better chance saving the data as wav files and then try converting them to csw/uef

[CommanderCoder]

Thanks for your description of your method of converting the wave to CSW using Goertzel's algorithm. It seems odd that the cleaner bitstream is not as reliable as the 'real world' bitstream. Maybe it has something to do with the swing positive to negative of the real world signal.

@Fraser - great to hear the new version of MakeUEF will appear soon.

Regarding emulating the cassette player, I did some work on an Acorn Atom emulator and its process for reading tapes was very raw. The OS has a very tight loop reading values directly from a memory address counting changes in the pulse train. The BBC has the serial ULA do this processing from a 'conditioned' signal directly from the tape. I'm not sure what the OpAmps are doing to 'condition' the signal (this helps: viewtopic.php?t=22935) but I can see now why my 11 year old self spent so many hours trying to get tapes to load.

[Diminished]

Thanks a lot. I have a few tapes which are original and a bunch of tapes which contain my own programs from when I was about 12. I got a disk drive a year or so later as tape were so slow, and I've managed to grab most of those discs! I know the quality of my tape programs will be embarrassing so I'd like to be the first to see them!

If you're interested in adding me to your repository for QuadBike 2, I can run it over some more of my tape recordings. I'm happy to send some to you too once I find a few quiet moments over the next couple of weeks to snap them all. I'll PM you a link to a folder on my web page where I'll put the mp3s: http://commandercoder.com/

I tried using MakeUEF v2.3 but it reported the "incorrect number of pulses by 1" so I couldn't test this on BeebEm.

There's a pre-release, not-properly-tested dev copy of the Quadbike code (I've called it 1.9.5) here.

[Diminished]

Thanks a lot. I have a few tapes which are original and a bunch of tapes which contain my own programs from when I was about 12. I got a disk drive a year or so later as tape were so slow, and I've managed to grab most of those discs! I know the quality of my tape programs will be embarrassing so I'd like to be the first to see them!

If you're interested in adding me to your repository for QuadBike 2, I can run it over some more of my tape recordings. I'm happy to send some to you too once I find a few quiet moments over the next couple of weeks to snap them all. I'll PM you a link to a folder on my web page where I'll put the mp3s: http://commandercoder.com/

I tried using MakeUEF v2.3 but it reported the "incorrect number of pulses by 1" so I couldn't test this on BeebEm.

I now have a PHP script that will convert Quadbike's new TIBET format to UEF, so here you go.

beebmunch.uef

(5.29 KiB) Downloaded 19 times