basictool - a command-line tool to tokenise, de-tokenise, pack and analyse BBC BASIC

[SteveF]

I've created a command-line tool which runs on a modern PC and can tokenise, de-tokenise, pack and analyse (6502) BBC BASIC programs. It does this by running the BBC BASIC and Advanced BASIC Editor ROMs on a (hidden) emulated BBC Micro, so these operations are performed by well-tested and reliable code. (Of course, the wrapper I've created around them could still be chock full of bugs.)

The code is plain C99, so it should compile on just about anything. (Famous last words!) I've developed it on Linux, but it should build easily on any Unix-like system. I'm hoping someone will step forward to create the necessary build script/project file to build it on Windows, which I don't think will be that big a job.

De-tokenisation is useful if you have a tokenised program which you want to examine:

Code: Select all

$ xxd test.tok
00000000: 0d00 0a15 f120 2248 656c 6c6f 2c20 776f  ..... "Hello, wo
00000010: 726c 6421 220d ff                        rld!"..
$ basictool test.tok
   10PRINT "Hello, world!"

You can also pretty-print the output (courtesy of ABE):

Code: Select all

$ basictool -f test5.bas
    1 FOR I=1 TO 100
    2   FOR J=1 TO 100
    3     PRINT I*J:
          IF I+J>42
            THEN PRINT "Foo!":
            FOR K=1 TO 100:
            NEXT
    4   NEXT
    5 NEXT

Tokenisation is useful if you're working on BBC software using a modern PC, of course:

Code: Select all

$ cat test.bas
10P. "Hello, world!"
$ basictool -t test.bas test.tok
$ xxd test.tok
00000000: 0d00 0a15 f120 2248 656c 6c6f 2c20 776f  ..... "Hello, wo
00000010: 726c 6421 220d ff                        rld!"..

Note that because the BASIC ROM is doing the tokenising, you can use the standard abbreviations like "P." for PRINT if you wish.

You can also pack the program using ABE:

Code: Select all

$ cat test3.bas
square_root_of_49 = 7
my_variable = 42 + square_root_of_49
PRINT my_variable
$ basictool -p -t test3.bas test3.tok
$ basictool test3.tok
    1s=7:m=42+s:PRINTm

The last example also shows that line numbers are optional in text input; this works exactly like beebasm's PUTBASIC command.

basictool *doesn't* generate .ssd output; you'll need to use something like MMB_Utils to do that, if you need it.

If you're interested, you can read more and download the code from github.

It works for me, I hope it will work for you! If you try it and have problems please post here so I can investigate.

I'll close with a lightly tweaked copy of the credits from the github README; obviously basictool builds on a lot of existing code:

• BBC BASIC was (of course) originally published by Acorn.
• The BASIC editor and utilities were originally published separately by Altra. The Advanced BASIC Editor ROMs are (C) Baildon Electronics. Thanks to Dave Hitchins for his support for developing basictool using these ROMs.
• 6502 emulation is performed using lib6502. This was originally written by Ian Piumarta, but the version of lib6502 used in basictool is taken from PiTubeClient.
• Cross-platform command line parsing is performed using cargs.

[scruss]

Thanks!

It doesn't want to build under gcc 10, for some reason. I've dropped an issue in the repo

[lurkio]

I've created a command-line tool which runs on a modern PC and can tokenise, de-tokenise, pack and analyse (6502) BBC BASIC programs.

Well, you've done it again, Steve! I've only had a quick go, but I think it might already be one of the greatest utilities I've ever used!



Till now, my workflow was fine but a bit faffy:

• write BASIC code in BBEdit,
• copy all,
• paste into BeebEm,
• *B [to invoke the PRES Advanced BASIC Editor],
• press P [for Pack],
• Y,
• Y,
• Y,
• Y,
• Y,
• Y [say yes to all Pack options],
• wait for Pack to finish [it helps to increase emulation speed here!],
• Escape,
• EXIT,
• RENUMBER,
• SAVE,
• RUN.
• Then rinse and repeat...

Even with the emulation speedup at the Pack stage, the procedure was a little tedious!

Now, most of that procedure can be replaced by this:

Code: Select all

./basictool -p -t -r -v -v prog.bas $.prog

Which seems to take less than a second to complete!

Brilliant, brilliant stuff!



Btw, building on macOS Mojave was trouble-free.

[SteveF]

It doesn't want to build under gcc 10, for some reason. I've dropped an issue in the repo

Thanks for giving this a go! I've pushed a fix to master for this, please give it a try. I haven't tagged this as a new release yet (I would prefer to get the fix tested first), but I can if it will make it easier for you to test.

Well, you've done it again, Steve! I've only had a quick go, but I think it might already be one of the greatest utilities I've ever used!

Thanks lurkio, you're too kind!

[scruss]

Thanks, Steve! Your fix worked well.

Does the Advanced BASIC Editor have an option to put spaces between tokens? I find run-together BASIC really hard to read and tiring to port to other systems

[SteveF]

Thanks, Steve! Your fix worked well.

Great, thanks for reporting and testing this. I've tagged up a 0.02 release with this fix so people downloading a .zip won't be caught out by it.

Does the Advanced BASIC Editor have an option to put spaces between tokens? I find run-together BASIC really hard to read and tiring to port to other systems

I can't see any obvious feature to do this, unfortunately, but someone else (lurkio?) may know.

If anyone knows another machine code utility (ROM or otherwise) which can do this, I could maybe add it to basictool.

[daveejhitchins]

Does the Advanced BASIC Editor have an option to put spaces between tokens? I find run-together BASIC really hard to read and tiring to port to other systems

"Unpack" should do this - is this command supported, Steve?

Dave H.

[shifters74]

Thanks for this tool - real handy!!

Builds on Ubuntu 20.04 fine with gcc 9.3 (default compiler)

cheers

shifters

[Deleted User 9295]

these operations are performed by well-tested and reliable code.

It sounds like a really useful utility, but I do question the need to incorporate an emulated BBC Micro! It's not difficult to "tokenise, de-tokenise, pack and analyse" BBC BASIC programs; for example I've got BBC BASIC code to perform all those tasks, in which I have confidence. I don't like to give the impression that there is such 'mystique' attached to these operations that they were only properly understood in the 1980s.

[cardboardguru]

It's a really clever project. The emulator isn't an off the shelf one, It's a very minimal one that doesn't reinvent the wheel of 6502 emulation, but just uses an existing library for that. And then it only emulates the functionality that the specific ROMS use.

It's analogous to Tricky's Asteroids port that, rather than trying to create something that functions like Aseroids, actually runs the original Arcade Asteroids ROM, with the bare minimum emulator needed to make that ROM work on a different platform.

No doubt this project could be expanded or forked to provide other functionality that exists in BBC Micro ROMs or programs from a modern computer shell.

It might also serve as an excellent starting point for someone that wanted to create a fully featured BBC emulator, but didn't know how to start. The emulator here is so minimal it's easy to understand by just skimming the code.

Good job!

[SteveF]

Does the Advanced BASIC Editor have an option to put spaces between tokens? I find run-together BASIC really hard to read and tiring to port to other systems

"Unpack" should do this - is this command supported, Steve?

It is now. Thanks for the suggestion, I've added this in v0.03.

If you try this, you may need to use something like "--unpack --renumber-step 100" when unpacking in order to ensure there are enough free line numbers for the unpack to work.

Thanks for this tool - real handy!!

Builds on Ubuntu 20.04 fine with gcc 9.3 (default compiler)

Thanks! It's always reassuring to get reports of successful builds...

It sounds like a really useful utility, but I do question the need to incorporate an emulated BBC Micro! It's not difficult to "tokenise, de-tokenise, pack and analyse" BBC BASIC programs; for example I've got BBC BASIC code to perform all those tasks, in which I have confidence. I don't like to give the impression that there is such 'mystique' attached to these operations that they were only properly understood in the 1980s.

Thanks Richard! I agree it has an element of "sledgehammer to crack a nut" about it, but lib6502 makes it pretty easy and I was already familiar with lib6502 from my earlier lib6502-jit project. I don't emulate any hardware except ROMSEL at &FE30, everything else is done by implementing a handful of OS entry points in C. Since it doesn't need to run arbitrary code I can get away with very crude implementations as well.

The original motivation for creating this tool was to make ABE's "pack" available at the command line on a modern PC, as I wanted to use it in the build process for Ozmoo. I may be wrong, but I have the idea that crunching a BASIC program both well and safely is more of an art than a science, and I wanted to take advantage of the experience that went into ABE's implementation.

I agree that tokenising and de-tokenising BBC BASIC isn't all that difficult (although bugs like this one suggest there are a few subtleties to be aware of), but once I had the framework in place to run ABE's "pack" I realised it wouldn't be much extra work to run the BBC BASIC ROM as well.

One nice thing about using the BASIC ROM is that the tokenisation/de-tokenisation is arguably correct by definition. In the past I tried adding support for BASIC abbreviations to beebasm's PUTBASIC tokeniser, got bogged down trying to implement the correct behaviour in some corner cases and gave up.

It's a really clever project.
[...]
No doubt this project could be expanded or forked to provide other functionality that exists in BBC Micro ROMs or programs from a modern computer shell.

Thanks! You're right, while I don't want to get too carried away, if there are other toolkit ROMs that provide useful BASIC-manipulating features they could be incorporated into basictool as well. If anyone feels like doing a fork for something other than BASIC manipulation that would be cool!

[Deleted User 9295]

I may be wrong, but I have the idea that crunching a BASIC program both well and safely is more of an art than a science...

I suppose there's a bit of both. Determining, for example, whether it's safe to remove a space or whether it's safe to concatenate two lines is certainly a science - there's only one 'right' answer and the rules aren't that complicated. But deciding how best to abbreviate a variable name, say, has many possibilities and there isn't a single 'perfect' way.

although bugs like this one suggest there are a few subtleties to be aware of

Whether to tokenise a pseudo-variable like PAGE to its 'left' or 'right' token is pretty fundamental to BBC BASIC, I wouldn't call it "subtle". Indeed that's not the only reason why a BBC BASIC tokeniser needs to know whether it's in 'left' or 'right' mode; for example if it encounters a * character when in 'left' mode it needs to disable tokenising for the rest of the line, but in 'right' mode it mustn't. But again the rules governing the transitions between left and right modes aren't exactly complicated.

I do agree that it's not 'trivial', and one does need to have a pretty good understanding of BBC BASIC to get it right in all circumstances, but there must have been dozens of tokenisers written from scratch in the past 40 years that do. Using the tokeniser built into 6502 BASIC for 'reliability' is something of a counsel of despair in my opinion.

You could, with a reasonable degree of confidence, take the tokenising routine from BBC BASIC for SDL 2.0 (lexan starting at line 414 here), although that does of course recognise many keywords not in 6502 BBC BASIC so wouldn't be suitable off-the-shelf.

[cardboardguru]

If anyone feels like doing a fork for something other than BASIC manipulation that would be cool!

One thing that strikes me as possible is to run text adventures this way. It would allow you to put a modern UI and maybe updated graphics on them, whilst not having to understand how the game works beyond a black box that accepts text and outputs responses.

[cardboardguru]

More advanced, someone could emulate Elite, without any BBC graphics emulation, but trap known graphics subroutines and cause them to output to an OpenGL window.

So for example trap the routine that converts surfaces into lines, and have code that outputs a textured polygon to OpenGL instead.

Just a thought. Probably not worth the trouble, unless for the pure challenge of doing it.

[SteveF]

although bugs like this one suggest there are a few subtleties to be aware of

Whether to tokenise a pseudo-variable like PAGE to its 'left' or 'right' token is pretty fundamental to BBC BASIC, I wouldn't call it "subtle". Indeed that's not the only reason why a BBC BASIC tokeniser needs to know whether it's in 'left' or 'right' mode; for example if it encounters a * character when in 'left' mode it needs to disable tokenising for the rest of the line, but in 'right' mode it mustn't. But again the rules governing the transitions between left and right modes aren't exactly complicated.

Thanks for this, Richard! With this hint I've gone and had a look at beebasm's BASIC tokeniser and made what I hope is a correct fix for this.

You could, with a reasonable degree of confidence, take the tokenising routine from BBC BASIC for SDL 2.0 (lexan starting at line 414 here), although that does of course recognise many keywords not in 6502 BBC BASIC so wouldn't be suitable off-the-shelf.

Thanks for the offer - I don't think I'm going to risk ripping out beebasm's existing tokenisation code, but if I do ever need to implement this from scratch I will certainly take a look at that in more detail.

One thing that strikes me as possible is to run text adventures this way. It would allow you to put a modern UI and maybe updated graphics on them, whilst not having to understand how the game works beyond a black box that accepts text and outputs responses.

More advanced, someone could emulate Elite, without any BBC graphics emulation, but trap known graphics subroutines and cause them to output to an OpenGL window.

So for example trap the routine that converts surfaces into lines, and have code that outputs a textured polygon to OpenGL instead.

Just a thought. Probably not worth the trouble, unless for the pure challenge of doing it.

These are pretty cool ideas! I don't plan on doing either myself, but if anyone feels like taking up the challenge I'll certainly be watching with interest.

Mark's annotated disassembly of Elite would obviously help a lot with the second suggestion. I suspect generating polygons would not be trivial as you'd run into the same issues as I did in this thread where I tried to generate 3D models of the ships in a web browser, but I'd love to be proved wrong.

I suspect it would be fairly easy to replace the line plotting routine, so you could draw high resolution anti-aliased lines on a modern PC (albeit with the end points moving at 256x256 resolution). Moving on from there, you could probably peek the Elite data structures to determine what type of object you were drawing and colour the lines for different ship types differently.

Moving up a level, it might be possible to hook into the "plot ship" code, so the emulator is taking an instruction from Elite like "draw a Cobra MkIII at co-ordinates (200,-100,20) rotated (0.3,-0.4)" and the emulator draws an arbitrary 3D model. You could of course patch the Elite binary slightly to help with this, thanks to the annotated disassembly. (I think it wouldn't work quite like that, from what I remember of the disassembly - it would be more like hooking into the code where Elite says "rotate the universe (0.3,-0.4)" and the emulator would then make the necessary adjustments in its 3D model, but something of the sort might be possible, though probably not trivial.)

[scruss]

It is now. Thanks for the suggestion, I've added this in v0.03.

If you try this, you may need to use something like "--unpack --renumber-step 100" when unpacking in order to ensure there are enough free line numbers for the unpack to work.

Thanks! It does seem to be a little selective with the spaces, though:

Code: Select all

$ cat ast.bbc
1L=0:REP.L=L+3:MO.4:DR.1279,0:DR.1279,452:MOVE1279,572:DR.1279,1023:DR.0,1023:F.I=1TOL:V.31,RND(32)+5,RND(31),42,30:N.:P.(L-3)/3:X=0:Y=512:REP.PL.69,X,Y:X=X+4:Y=Y-(INKEY-74+.5)*8:U.PO.X,Y)=1ORX=1280:U.X<1280:V.7:REP.U.INKEY-99:RUN

$ basictool -t ast.bbc ast.tok

$ basictool --unpack --renumber-step=100 ast.tok
   10 L=0
   11 REPEATL=L+3
   12 MODE4
   13 DRAW1279,0
   14 DRAW1279,452
   15 MOVE1279,572
   16 DRAW1279,1023
   17 DRAW0,1023
   18 FORI=1 TOL
   19 VDU31,RND(32)+5,RND(31),42,30
   20 NEXT
   21 PRINT(L-3)/3
   22 X=0
   23 Y=512
   24 REPEAT PLOT69,X,Y
   25 X=X+4
   26 Y=Y-(INKEY-74+.5)*8
   27 UNTIL POINT(X,Y)=1 ORX=1280
   28 UNTILX<1280
   29 VDU7
   30 REPEAT UNTIL INKEY-99
   31 RUN

[SteveF]

Thanks for giving that a try. I believe that's just how ABE's unpack does it, unfortunately, so there's not much I can do to change it. Edit: Of course, if you can recommend another toolkit ROM which does this, I can look at adding support for that to basictool. (Or if you feel keen you could maybe knock up some C code to add spaces and I'd be happy to look at adding that into basictool, although of course if you've written it in C you might just prefer to have it standalone.)

[cardboardguru]

I can report that it builds and works fine using make on macOS Big Sur.

Only comment I'd make is that it would be nice if make gave an error if the ROMs aren't there. As it stands, it fails without error having built bintoinc and one of the zz-editor files.

[hoglet]

I also used this for the first time today (on Ubuntu 18.04) and it worked perfectly.

Why not just check the ROM images into the repository?

Alternatively, have the build script grab them from mdfs.net.

Dave

[SteveF]

I also used this for the first time today (on Ubuntu 18.04) and it worked perfectly.

Why not just check the ROM images into the repository?

Thanks for confirming that Dave, much appreciated. I was feeling a bit paranoid about copyright but I've now gone and checked the ROMs in as you suggested; it's no different than (e.g.) b-em being hosted on github, after all. I've also created a v0.04 simply in order to have a tagged release which includes the ROMs; there aren't any code changes since v0.03.

I can report that it builds and works fine using make on macOS Big Sur.

Only comment I'd make is that it would be nice if make gave an error if the ROMs aren't there. As it stands, it fails without error having built bintoinc and one of the zz-editor files.

Thanks for testing that! I agree this would have been a good change (I'm a bit surprised make didn't give any error, but the failure could definitely have been made more obvious with an explicit check for the ROMs existing) but now I've put the ROMs in the repository this shouldn't be an issue anyway.

[Deleted User 9295]

I was feeling a bit paranoid about copyright but I've now gone and checked the ROMs in as you suggested

How have you convinced yourself that this is legitimate (other than because other people have done it)? This article may not be entirely relevant but it does provide some background.

[cardboardguru]

Nothing is to be served by being jobsworth about copyrights for which there isn't an identifiable copyright holder that objects.

Or are we going to effectively deprive the world of emulators for the next 70 years or whatever? After all, within the letter of the law, you can't even make a copy of a ROM to use in your emulator even if you own that physical ROM. Within the letter of the law, you only have the right to make backups, not working copies.

Of course going down that path you'd also have to object to projects such as the sharing of disassemblies of retro games and the various clever mods people have done as a result. Stopping that kind of behaviour would certainly make the world a better place.

Whilst you're at it, best object to anyone that finds an image on the internet and sticks it in a post to help make a point. If they haven't got the permission of the creator of that image, that's a copyright breach right there. That's just as serious as posting a ROM. Best stop that happening.

Don't go sharing memes either. Unless you know who created the meme, and have explicit permission it's wrong to copy it!

And what about the Complete BBC Micro Games Archive? Do you want that cancelled?

Oh Brave New World that enforces copyrights no one cares about!

[BigEd]

I was feeling a bit paranoid about copyright but I've now gone and checked the ROMs in as you suggested

How have you convinced yourself that this is legitimate (other than because other people have done it)? This article may not be entirely relevant but it does provide some background.

Thanks for the link Richard. I believe this is an area where each person has to make their own judgement, as to what stance to take and how to behave. (And indeed, what their particular reasoning is.)

I feel this is an area where it can be counterproductive to try to rally people into one or other course of action.

There are, of course, much more clear-cut areas where the copyright holder is in evidence, is active, is even still selling the product, and I notice that here on the forum we are much more uniform as to how to act in that case.

[Deleted User 9295]

Wow. Talk about over-reaction to an innocuous comment (I didn't even express my own opinion on the matter, not least because I am as much conflicted as anybody else)! Of course it ultimately comes down to individual judgement, but that needs to be informed by the facts, as far as they can be ascertained.

[Lardo Boffin]

Can anyone recommend a Windows 7 64bit IDE etc. I can download to build this please? I currently don't have any means of building C.

Many thanks!

Edit - think I have one working, now to try and compile the project...

[Lardo Boffin]

Btw, building on macOS Mojave was trouble-free.

Any chance you could post a built version of this please?

My adventure language project kind of stalled due to me being sick of following a similar pattern when building that due to a lack of tokeniser. So I started writing a tokeniser in C# as part of the build process. This resulted in a lot of late nights and subsequent project burnout…

If I can put this in as part of the tool chain it will make life a lot easier! I think my mac is running Sierra or High Sierra if that makes any difference.

Thanks!

[lurkio]

Btw, building on macOS Mojave was trouble-free.

Any chance you could post a built version of this please?

Hope it works! (If not, it shouldn't be hard to build it yourself. I don't think the program has any arcane dependencies.)

[Lardo Boffin]

Btw, building on macOS Mojave was trouble-free.

Any chance you could post a built version of this please?

Hope it works! (If not, it shouldn't be hard to build it yourself. I don't think the program has any arcane dependencies.)

Works a treat thanks! Time to park the tokeniser.

[BeebMaster]

This is great, just installed it on Ubuntu 20.04. For some years, to make a readable text version of a BASIC program stored on the PC, I have used Brandy BASIC, but latterly it no longer works. I can still invoke Brandy BASIC by right-clicking on a file, and that file can be listed with, er well, LIST, when Brandy loads, but for some reason I no longer have any select or copy & paste function, so I can't do anything with the listing except look at it.

Now I've made a little script "btoolclip" which will copy the selected files as de-tokenised text to the clipboard:

Code: Select all

#!/bin/bash
# Btoolclip - copy de-tokenised BASIC program to clipboard using basictool and xclip

exec 2>>/tmp/debug.output
set -x

for i in "$@"
do
  basictool "$i" | xclip -selection clipboard
done

PS credit to SWEH for most of the above in another script elsewhere!

[jgharston]

On a similar theme, I tweeked and updated (and documented) Ben Ryves' BBC BASIC detokeniser and syntax highlighter web page generator to support both Acorn and Russell format BASIC files, single and double extended tokens and implement fuller indentation: http://mdfs.net/Apps/WebTools/WebBasic/

It lets you host actual BBC BASIC files, and also allow it be listed, eg:
Actual BASIC file: http://mdfs.net/System/Library/BLib/IntToTxt
LISTed BASIC file: http://mdfs.net/System/Library/BLib/IntToTxt.bas