Thanks lurkio. I think I see the problem(s) now - please bear with me as this is perhaps a rather long and pedantic post...
I am able to reproduce your EATOK and EATOK2 files exactly, so I think we can probably rule out any weird differences between basictool running on my machine and running on yours.
I took $.EGYPT and used both an emulated model B and an emulated Master 128 (i.e. BASIC 2 and 4 respectively) to:
- just RENUMBER
- just pack
- RENUMBER then pack
- RENUMBER then pack then RENUMBER
$.EGYPT. I got identical output from both emulated machines for each test, and I was able to reproduce that output byte-for-byte using basictool,
doing everything a step at a time.
When basictool is used to pack and renumber (i.e. -p -r options specified), it does the pack first and the renumber afterwards. [0] This means that if it's necessary to renumber first in order to stop pack corrupting this particular program, using the '-r' option in basictool won't help. You can work around this by doing things in two steps - if $.EGYPT is egypt.orig on your PC (note that we start with the original pre-tokenised BASIC program here):
Code: Select all
basictool -tvvr egypt.orig egypt-renumbered.tok
basictool -tvvp egypt-renumbered.tok egypt-renumbered-then-packed.tok # feel free to add '-r' here too if you want renumbering after packing
The reason it works like this is that I was thinking of renumbering as a purely "cosmetic" operation, so there's no point tidying up *before* the pack as it's probably going to change all the line numbers anyway. Since, as it turns out, renumbering is useful to force BASIC to fix up this program [1],
there's definitely an argument to be made that if you specify the -r option, basictool should renumber before packing as well as after packing. Does that seem like a good idea to you? I don't *think* there's any downside (except a microscopic performance hit) - if the line number references in the program are complex and not RENUMBER-safe, doing a RENUMBER after packing would already break things, so an extra RENUMBER first shouldn't hurt. (Edit: we could add some kind of --pre-renumber option to control this explicitly, but unless there's a downside to just making -r do two renumbers, I'd rather go with that to keep the interface a little bit simpler.)
I think the problems you are having with "decrypted prog Disc999-EgyptianAdventure.bas" (which I have renamed lurkio.bas for ease of reference, having extracted it from the zip you attached first) are caused by your text editor altering the encoding for non-ASCII characters.
Looking at the raw bytes in $.EGYPT:
Code: Select all
$ xxd -g1 lurkio-EGYPT | head -27 | tail -3
00000180: 8a 33 2c 31 30 29 22 84 9d 83 50 6c 65 61 73 65 .3,10)"...Please
00000190: 20 77 61 69 74 2c 20 64 61 74 61 20 6c 6f 61 64 wait, data load
000001a0: 69 6e 67 2e 2e 2e 20 9c 22 3b 3a 44 54 3d 30 3a ing... .";:DT=0:
You can see that the control codes just before "Please wait" are &84, &9D and &83 respectively. Looking at the corresponding raw bytes in lurkio.bas:
Code: Select all
$ xxd -g1 lurkio.bas | head -34 | tail -3
000001f0: 2c 31 30 29 22 c3 91 c3 b9 c3 89 50 6c 65 61 73 ,10)"......Pleas
00000200: 65 20 77 61 69 74 2c 20 64 61 74 61 20 6c 6f 61 e wait, data loa
00000210: 64 69 6e 67 2e 2e 2e 20 c3 ba 22 3b 3a 44 54 3d ding... ..";:DT=
you can see the control codes are now &C3, &91, &C3, &B9, &C3, &89. [2] These have been passed through (reasonably enough, I think) by basictool and are equally present in EATOK:
Code: Select all
$ xxd -g1 lurkio-EATOK | head -32 | tail -3
000001d0: 30 3b 3a f1 8a 33 2c 31 30 29 22 c3 91 c3 b9 c3 0;:..3,10)".....
000001e0: 89 50 6c 65 61 73 65 20 77 61 69 74 2c 20 64 61 .Please wait, da
000001f0: 74 61 20 6c 6f 61 64 69 6e 67 2e 2e 2e 20 c3 ba ta loading... ..
This is why the output when running EATOK is corrupt.
On a different note: lurkio.bas contains REM-ed out lines, with a REM before the line number. basictool is treating these as lines which you didn't give a line number for and is auto-numbering them, rather than discarding them as you perhaps intended. If text input to basictool is:
Code: Select all
10PRINT "Hello ";
REM20PRINT "wurld"
30PRINT "world!"
the output it generates will be:
Code: Select all
10PRINT "Hello ";
11REM20PRINT "wurld"
30PRINT "world!"
This isn't a problem as such, but it does make the basictool output from lurkio.bas look like it has duplicate lines when compared to the $.EGYPT original and confused me a little bit. I appreciate you may have known it worked exactly like this, though. We can discuss changing this if you like; my current preference is not to (because it would mean treating REM as a special case for auto line numbering), but I am open to debate, although probably best to do so after we've cleared up these other issues.
I hope all this makes sense and addresses all the different aspects of this; if I've missed anything or anything I've said seems wrong, please let me know!
[0] Edit: it doesn't make any difference what order -p and -r are on the command line, just FWIW.
[1] I don't think we actually know why this is necessary yet, do we? It would be interesting to understand this, but from a narrow basictool point of view I don't think it's relevant so I'm trying to avoid being distracted by this question.
[2] I tried to see exactly why we ended up with these particular bytes, but they don't seem to be the UTF-8 encoding of the original three bytes interpreted as an ISO-8859-1 encoding as I had half expected. It doesn't really matter - it's obvious from the hex dumps they have been corrupted somehow - but if anyone can explain why we have these particular bytes it would be interesting...
Edit: I've just seen your edit, but I don't think it changes anything substantially - as I said, I was able to reproduce your EATOK and EATOK2 files so if you did get confused at some point I don't think you got confused in an important way.