basictool - a command-line tool to tokenise, de-tokenise, pack and analyse BBC BASIC

[lurkio]

The ultimate crude hack in an emergency - which would not treat quoted strings specially, but you'd probably get away with it - would just be to do something like:

Code: Select all

sed -e 's/\<REM .*//g' mybigfile.txt | basictool -t - out.tok

Just for the record, the following command works for me. (I deleted the "\<" because it seemed to stop the command having the desired effect, and I wasn't sure what it was meant to be doing. I also had to delete the space after REM because some of my REMs weren't followed by a space. (Naughty, naughty.)):

Code: Select all

sed -e 's/REM.*//g' -e '/^$/d' mybigfile.txt | basictool -p -t -r -v -v - out.tok

That'll delete REMs, and it will also delete blank lines, which might be useful if your program's really big and you don't want to risk going over the limit on the size of a line number (an edge case, obviously).

As you said, this is just a crude hack in case of emergency because, I think, it will delete every instance of "REM", plus everything up to the end of the line, no matter where exactly the string "REM" appears. So it'll play merry hell with this, for example:

Code: Select all

PRINT"Hello" 
PROCTHEOREM
END

DEFPROCTHEOREM:PRINT "Goodbye"
ENDPROC

[SteveF]

I'm glad you got it working! Sorry for taking so long but this has now prompted me to get round to adding a new option (-r, --strip-rems; happy to take suggestions for tweaks to those names) to basiclabel.py, which I've just pushed to github. This has only been lightly tested but seems to work for me, please give it a try and let me know how you get on.

For what it's worth, the intent of the "\<" in the sed command is to say that REM has to appear as the start of a word. It looks like our versions of sed behave differently because it does seem to work for me:

Code: Select all

$ cat test9.txt
   REM test for -r options:PRINT "not executed"
PRINT "Hello ""there"""
PRINT "Goodbye""there"""   :REM well """why not""?
PRINT "REMINDER"
PRINT """REMINDER"" is a word"
foo$="hello":REMark
$ sed -e 's/\<REM .*//g' < test9.txt
   
PRINT "Hello ""there"""
PRINT "Goodbye""there"""   :
PRINT "REMINDER"
PRINT """REMINDER"" is a word"
foo$="hello":REMark

and this extra protection stops it breaking your "THEOREM" example:

Code: Select all

$ cat test10.txt
PRINT"Hello" 
PROCTHEOREM
END

DEFPROCTHEOREM:PRINT "Goodbye"
ENDPROC
$ sed -e 's/\<REM .*//g' < test10.txt
PRINT"Hello" 
PROCTHEOREM
END

DEFPROCTHEOREM:PRINT "Goodbye"
ENDPROC

It's still a crude hack, of course (as your "space after REM" observation shows, for a start - this problem is visible about in the test9.txt output) and hopefully doesn't matter at all if the new basiclabel.py -r option is working correctly.

[lurkio]

a new option (-r, --strip-rems; happy to take suggestions for tweaks to those names) to basiclabel.py, which I've just pushed to github. This has only been lightly tested but seems to work for me, please give it a try and let me know how you get on.

That's great, Steve. Thanks for taking the time to do this.

There's one slight drawback, though, which is that there's no option to "ruthlessly cull" REMs so that they don't leave even a colon behind -- which would be a nice option to have because if your source code isn't numbered in the first place, and if a line begins with a REM (rather than with a label), then clearly the user doesn't intend that line to be the target of a GOTO/GOSUB/RESTORE, so it doesn't really make sense to number it and stick a colon on it.

Also, the colons aren't easy to get rid of after the fact, and if you have quite a long unnumbered program with a lot of REMs, and if you put the program through basiclabel.py -r, you end up with redundant colons littered throughout the output file! If you then pass that file to basictool -p, you get some rather strange-looking results, as even a short example will demonstrate:

Input to basiclabel -r :

Code: Select all

REM dfjkljdkf
PRINT "Hello"
REM dfdsfdf
PRINT "Goodbye"

Output from basiclabel -r :

Code: Select all

1:
2PRINT "Hello"
3:
4PRINT "Goodbye"

If you pass the above as input to basictool -p, you get the following ungainly result:

Code: Select all

10::PRINT"Hello"::PRINT"Goodbye"

So, I wonder if an option to "ruthlessly cull REMs, leaving no trace behind, not even a colon" would be a good addition to the basiclabel.py feature set? (I realise I'm pushing my luck here, so please don't feel any pressure to add this, as I can always tweak and use the sed command instead.)

[Deleted User 9295]

So, I wonder if an option to "ruthlessly cull REMs, leaving no trace behind, not even a colon" would be a good addition to the basiclabel.py feature set?

I appreciate that the primary purpose of this tool is for use with early versions of BBC BASIC that don't have multiline IF...ENDIF clauses, but it may be worth mentioning that you must not 'ruthlessly cull REMs' in versions of BBC BASIC from ARM BASIC V onwards because it can change a single-line IF statement into a multi-line one:

Code: Select all

      IF FNsomething THEN REM this is a dummy IF to discard the result

would become:

Code: Select all

      IF FNsomething THEN

which has an entirely different meaning.

[SteveF]

So, I wonder if an option to "ruthlessly cull REMs, leaving no trace behind, not even a colon" would be a good addition to the basiclabel.py feature set? (I realise I'm pushing my luck here, so please don't feel any pressure to add this, as I can always tweak and use the sed command instead.)

This seems like a reasonable idea and it's a good observation that if there's no line number or label the line presumably isn't going to be referenced. How would it be if -r behaves as it currently does if there is a line number/label and otherwise completely removes lines which have nothing on them but a REM statement? This would keep the user interface simpler by avoiding an extra option. If you think having a separate option is best I'm happy to do that.

I appreciate that the primary purpose of this tool is for use with early versions of BBC BASIC that don't have multiline IF...ENDIF clauses, but it may be worth mentioning that you must not 'ruthlessly cull REMs' in versions of BBC BASIC from ARM BASIC V onwards because it can change a single-line IF statement into a multi-line one

Thanks Richard, that's worth knowing!

[lurkio]

How would it be if -r behaves as it currently does if there is a line number/label and otherwise completely removes lines which have nothing on them but a REM statement?

Yes, I think that would do the trick!

[SteveF]

Thanks lurkio, I've pushed a lightly tested version with this change to github, please give it a try and let me know how you get on.

Edited to add: Nothing to do with the basiclabel.py changes we're discussing, but FWIW: I also labelled up an "official" v0.07 release of basictool earlier: https://github.com/ZornsLemma/basictool ... /tag/v0.07 This is substantially identical to v0.07-pre so there's no point upgrading if you already have that (but no harm either, of course).

[Deleted User 9295]

behaves as it currently does if there is a line number/label and otherwise completely removes lines which have nothing on them but a REM statement?

Again this may be of no relevance or interest, but in my crunchers removing of REM lines completely, if not referenced by a GOTO, GOSUB or RESTORE (yes, RESTORE can legally reference a REM line!) is a function of the line-concatenation mechanism.

In other words if you don't specify that lines should be concatenated, a REM on a line of its own will be removed but the resulting (empty) line will remain. But if you do also specify that lines should be concatenated, the REM line will then disappear completely.

[lurkio]

Thanks lurkio, I've pushed a lightly tested version with this change to github, please give it a try and let me know how you get on.

I tested the updated version of basiclabel.py a few days ago, and it worked well. Many thanks!

(Btw, at the time, the version of basiclabel.py in the full basictool ZIP archive release was out of sync with the version of basiclabel.py that you can download directly from the utils folder in the Code section of the Github repo. Might be worth syncing those two versions up..? Just thought I'd mention it on the off-chance that you weren't aware -- although I'm almost certain you already have it on your to-do list!)

[SteveF]

I tested the updated version of basiclabel.py a few days ago, and it worked well. Many thanks!

Good to know, thanks!

(Btw, at the time, the version of basiclabel.py in the full basictool ZIP archive release was out of sync with the version of basiclabel.py that you can download directly from the utils folder in the Code section of the Github repo. Might be worth syncing those two versions up..? Just thought I'd mention it on the off-chance that you weren't aware -- although I'm almost certain you already have it on your to-do list!)

I am a bit torn here. I'm not saying I'm not open to argument on this - I'm kind of arguing with myself just by writing this - but so far my attitude has been that the stuff in the "utils" directory is not really part of basictool proper; it just happens to share the basictool repository as a matter of convenience. The basictool README.md file's changelog doesn't mention changes to stuff in the "utils" directory and the basictool version number only changes when basictool itself changes.

This means that it's pretty much arbitrary which versions of "utils" get bundled up in a release zip file, which I accept is a bit iffy. On the other hand, I'm reluctant to start bumping the basictool version number up just because one of the utils has been tweaked, particularly when some of the utilities are potentially quite experimental. I could re-use the same version number and just re-build the zip file when one of the utils has been tweaked, but then there are potentially multiple different zip files floating around for the same basictool version. Edit: This isn't good for support purposes - user A says they downloaded basictool 0.21 and basiclabel.py is buggy, user B says *they* downloaded basictool 0.21 as well and they can't reproduce the bug in basiclabel.py, because they downloaded different versions of the 0.21 zip with different utils in.

As I say, I'm torn. I half wonder if the best thing to do would just be to fork the utils off into a basictool-utils repository. They would then *not* be present in the basictool release zip file at all, but at least there wouldn't be this potential confusion about things being out of sync. Edit: If there were frequent updates to basictool itself it wouldn't be such a big deal, since tweaks to the utilities would fairly quickly get bundled up in a new release. But since basictool is mostly stable now, the utils are probably going to be changed more frequently than basictool itself and thus they get "stuck" in github but not bundled up into a numbered basictool release.

I'm open to suggestions on how best to handle this...

[lurkio]

I half wonder if the best thing to do would just be to fork the utils off into a basictool-utils repository. They would then *not* be present in the basictool release zip file at all, but at least there wouldn't be this potential confusion about things being out of sync.

Sounds sensible to me. Maybe replace the contents of the utils folder with a textfile stating that the utils can now be found in a separate repo.

[lurkio]

There seems to be an off-by-one error in basictool. The following code can be copied and pasted into BeebEm, but if you try to pass it to basictool you get a "line too long" error:

Code: Select all

2470 DA. IN A CAVE WITH A STREAM FLOWING THROUGH IT,0,9,0,0,0,0,IN THE EVIL TEMPLE,0,0,18,0,0,0,IN THE SACRIFICE HALL,0,0,0,0,15,0,IN THE HALL OF DISRONDU,0,0,16,0,0,0,IN A SMALL ROBE ROOM,0,11,9,0,0,0,ON A LEDGE IN THE CHASM,0,0,0,0,0,10

If you delete the space after the line number then basictool is happy.

[SteveF]

Thanks lurkio, that's a good find and diagnosis! I've created v0.08-pre with a fix for this, please give it a try and let me know how you get on.

I always get confused about this aspect of OSWORD 0; rather embarrassingly I actually made an edit to BeebWiki clarifying exactly this aspect at a date which suggests I was working on basictool at the time, so I actually made an effort to understand the behaviour correctly and still managed to get it wrong...

[lurkio]

I've created v0.08-pre with a fix for this, please give it a try and let me know how you get on.

Works well, thanks!

[SteveF]

Thanks lurkio! I've now tagged that as an official v0.08 release; there's no change compared to v0.08-pre apart from the version number.

[KenLowe]

I've created a command-line tool which runs on a modern PC and can tokenise, de-tokenise, pack and analyse (6502) BBC BASIC programs. It does this by running the BBC BASIC and Advanced BASIC Editor ROMs on a (hidden) emulated BBC Micro, so these operations are performed by well-tested and reliable code. (Of course, the wrapper I've created around them could still be chock full of bugs.)

The code is plain C99, so it should compile on just about anything. (Famous last words!) I've developed it on Linux, but it should build easily on any Unix-like system. I'm hoping someone will step forward to create the necessary build script/project file to build it on Windows, which I don't think will be that big a job.

I'm not sure I'd call it a build script / project file, but I managed to create a Windows x64 build on my RaspberryPi. A copy of the executable and how I went about creating the file can be found here:

viewtopic.php?p=346349#p346349

[SteveF]

I've just tagged up a release of basictool v0.09 in the github repository. What's changed?

• Add simple cross-compilation support to the makefile. Thanks to Ken Lowe for help with this.
• Fix build error on Ubuntu 22.04. Thanks to shifters67 for reporting this.
• lib6502: fix Z flag after ASL instruction. Thanks to dp11 for the fix.
• lib6502: fix _zpr not displaying zp location. Thanks to dp11 for the fix.
• Fix potential buffer overflow when in mpu_dump(). Thanks to dp11 for the fix.

None of these changes are terribly recent but I suddenly realised I hadn't actually made a release with them in. The lib6502 bugs fixed here are not known to have caused any problems when running BASIC or ABE, but obviously it's good to have them fixed.

I've also generated a PDF copy of the man page and included that in the repository, for the convenience of users who aren't on Unix or who don't want to bother installing the man page somewhere on their MANPATH.

As always, let me know if you have any problems building or using basictool.

[lurkio]

Just submitted an absurdly long post about (i) successfully using (the latest version of) basictool but then running into the limits of PRES ABE Pack; and (ii) a load of other waffle besides.

[SteveF]

Just submitted an absurdly long post about (i) successfully using (the latest version of) basictool but then running into the limits of PRES ABE Pack; and (ii) a load of other waffle besides.

Nice one lurkio, I enjoyed reading the write up!

I wish you hadn't had to work so hard though! As I think I've said before, what I'd like to add to basictool is a facility to pack the code with some other utility which I can build to run at (say) &f000 (the minimally emulated beeb has no real OS so code can live here), and that would allow packing much larger programs. Once crunched like this, you could then re-crunch with ABE if the result is small enough to fit on a normal beeb.

I had hoped to do this with the shrink and lincomp programs from The Micro User, which I remember using BITD and which worked quite nicely, but the versions we found so far seem more complex than the ones I remember. In theory I still have the original magazines and could retype the code, or I guess they *might* be lurking in my own collection of archived Beeb discs (though ISTR seeking and not finding...)

This could also maybe have been done with Toolkit Plus's pack command, if the source was available so it could be rebuilt to run &b800-ish, but that seems to have fallen through for now.

[lurkio]

I seem to have found an unfortunate bug in PRES ABE Pack, which hence also affects basictool.

If you use all Pack options (including Concatenate) and pack the following program...

Code: Select all

10DEFPROCa:PRINT"In PROCa":ENDPROC
20PROCa:PRINT"Outside PROCa"

...then PRES ABE Pack will turn it into this...

Code: Select all

10DEFPROCa:PRINT"In PROCa":ENDPROC:PROCa:PRINT"Outside PROCa"

...which obviously (and undesirably) has different behaviour from the original program!

[BeebMaster]

Yes, it shouldn't concatenate lines starting with a DEF which include lines after the end of the procedure or function. I wonder if all crunchers do that in this situation though.

Everything on a line after a DEF is ignored at run-time unless the named procedure or function is being called. This allows procedures and functions to be nested to use common code, so you can do for example:

Code: Select all

1000DEFFNread(start,length,memory):opcode=&8
1010DEFFNwrite(start,length,memory):opcode=&A
1020=FNosword(&72,opcode,start,length,memory)

with no danger of accidentally writing when calling FNread.

[lurkio]

Yes, what's weird is that I didn't run into this problem until recently, when I added PROCa at the start of G.HAMP01 (click "Display"):

http://bbcmicro.co.uk/explore.php?id=3199

Till then, there were seemingly dozens of PROCs (scroll down that page) which PRES ABE Pack concatenated correctly, without any issues! Obviously I must have hit a particular set of circumstances that triggered the bug with PROCa. (My workaround, btw, is to add a colon followed by an asterisk after the ENDPROC of PROCa! That prevents PRES ABE Pack from trying to concatenate it.)

[lurkio]

I wonder if basictool could check for this buggy ENDPROC-concatenation issue and spit out a warning if it occurs? Maybe basictool could internally generate the ASCII listing for the code being Packed and search each line for ENDPROC followed by optional spaces, a colon, and more code on the same line? (Or maybe it would be easier to search the tokenised form of the program instead -- but I didn't want to suggest that because of the risk of finding false positives...)

Or maybe this idea is just silly..?!

[BeebMaster]

This might be a more general problem with "crunchers", here's what happens with my nested PROC idea, which works OK in its original form:

But breaks down when using the *SQUEEZE command from Beebug BASIC Booster:

So if that was being used in anger, no reads would ever be done, but only writes!!

[lurkio]

This might be a more general problem with "crunchers", here's what happens with my nested PROC idea ...

Yes, it's a pain! Nested PROCs can also be manually "protected" from crunchers if you know what to look for, but it's still a hassle!

Here's what I had to do in one case in BeebScott:

https://github.com/ahope1/BeebScott/blo ... p.bas#L306

[BeebMaster]

Even the Acorn one doesn't do it right:

[SteveF]

I wonder if basictool could check for this buggy ENDPROC-concatenation issue and spit out a warning if it occurs?

That seems reasonable to me. Could you please raise an issue on the basictool github repository to remind me to take a look at this? I might remember anyway, but that would be helpful.

Are there any other things we could check for?

Also, would it be worth documenting all the known bugs/quirks (crunching is hard enough I'm reluctant to call everything a bug!) in ABE pack (and perhaps other crunchers too) somewhere? This would be a useful resource for people using them and anyone who might work on the crunchers in future. Would it be appropriate to put something like this on beebwiki, perhaps?

[lurkio]

Interesting! It seems you can further reduce REDUCE1 —

Code: Select all

168 GOTO 70
3640 IF DF=0 OR RDF=r THEN 3660

— and still get some corruption (albeit not the screen-clear) when you invoke PRES ABE Pack and say yes to all Pack options:

Code: Select all

168GOTO31296IFD=0ORR=rTHEN3660

I was investigating the above test program a bit more (after going to the basictool github repo to report some unrelated Issues), and I noticed that you get an even weirder result if, in BeebEm, you say Yes to all PRES ABE Pack options on the following program:

Code: Select all

70 PRINT"This is line 70"
80 IF TRUE GOTO 70 ELSE GOTO 90
90 IF DF=0 OR RDF=r THEN 70

After packing, if you exit to BASIC and do a LIST, you get this!:

Code: Select all

L.
   70PRINT"This is line 70":IFTRUEGOTO24640ELSEGOTO90
   90IFD=0ORR=rTHEN19776OSCLIZIFD=0ORR=rTHEN19776OSCLI32794F
No FOR at line 70
>_

The Packed program seems to cause LIST itself to throw an error!

This bug in PRES ABE Pack seems to have something to do with the Variables option: if you say No to the Variables option, then the program gets Packed correctly!

[8cup]

Very nice, this makes writing code for the beeb so much faster especially with MMB/SSD Utils.

[lurkio]

Very nice, this makes writing code for the beeb so much faster especially with MMB/SSD Utils.

Yes, here's my handy one-line build-save-boot command for hacking/developing To Hell In A Hamper:

Code: Select all

rm tmp; rm G.HAMP01; basiclabel.py -r 01.bas > tmp; basictool -2tpvv tmp G.HAMP01; beeb delete Disc999-ToHellInAHamper.ssd -y G.HAMP01; beeb putfile Disc999-ToHellInAHamper.ssd G.HAMP01; open Disc999-ToHellInAHamper.ssd