py8dis - a programmable static tracing 6502 disassembler in Python

[SteveF]

py8dis is a programmable static tracing 6502 disassembler written in Python; the name was chosen out of a) a lack of imagination b) a vague aspiration that it might be extended without too much difficulty to other 8-bit CPUs in the future. It can output beebasm or acme compatible assembly. Although there's some optional support to help with disassembly of Acorn code, it isn't Acorn specific in general.

You can get the code on github.

I suspect there's at least one other project implementing pretty much the same thing out on the net somewhere, but I deliberately haven't looked yet as I wanted the challenge of implementing this for myself without being put off by someone else already having done it better. All I can say is that Google doesn't seem to think the name "py8dis" (or "pydis8") is already taken.

BeebDis has been a big influence on py8dis; it's only disassembler I've used recently, and I've tried to copy some of its command names. Apart from the fact that it's sometimes fun to reinvent the wheel, it was really just the appeal of making the disassembler programmable that motivated me to write py8dis.

As with BeebDis, py8dis is based around the idea of iteratively developing a control file as you learn more about the program being disassembled. The difference is that the control file is itself a Python program and py8dis is used as a kind of library by that Python program.

Let me illustrate with a semi-contrived example - disassembling Acorn DFS 2.26. You can get this at mdfs.net.

To start off with, we create a basic control file, dfs226.py:

Code: Select all

from commands import *
from trace6502 import *
import acorn

load(0x8000, "dfs226.orig", "f083f49d6fe66344c650d7e74249cb96")
set_output_filename("dfs226.rom")

acorn.add_standard_labels()
acorn.is_sideways_rom()

go()

load() loads the ROM image into the disassembler's memory at the correct location. The last argument to load() is optional and is the expected md5sum of the file; if this is provided the disassembly will fail if dfs226.orig doesn't have a matching md5sum.

set_output_filename() is optional but controls the filename given to the beebasm SAVE command; it has no effect if you're using the acme output format.

The two calls to the acorn module define some Acorn OS constants (e.g. oswrch at &FFEE) and automatically interpret the sideways ROM header respectively. If you weren't disassembling a ROM, you'd omit the call to acorn.is_sideways_rom() and replace it with something like:

Code: Select all

entry(0x2000, "start")

to tell py8dis that there's machine code at &2000 which it can start tracing. (For a sideways ROM, we know there are entry points at &8000 and/or &8003.)

To keep things simple, put dfs226.py in the same directory as acorn.py, trace.py etc. (There are some notes in the github README on setting PYTHONPATH to avoid needing to do this.) Running dfs226.py:

Code: Select all

python dfs226.py > dfs226.asm

generates an initial disassembly in beebasm format; you can specify "-a" after dfs226.py to generate acme output instead. The output defaults to lower case mnemonics/labels, but you can specify "-u" if you prefer upper case. ("--help" will show all the options.)

The output (edited for brevity, of course) will look something like this:

Code: Select all

    org &8000
    guard &c000
.pydis_start

; Sideways ROM header
.rom_header
.language_entry
    equb &00, &00, &00                                      ; 8000: ...
.service_entry
    jmp service_handler                                     ; 8003: 4c c8 be
.rom_type
    equb &82                                                ; 8006: .

...

.l80c8
    and #&0f                                                ; 80c8: 29 0f
    cmp #&0a                                                ; 80ca: c9 0a
    bcc l80d0                                               ; 80cc: 90 02
    adc #&06                                                ; 80ce: 69 06
.l80d0
    adc #&30 ; '0'                                          ; 80d0: 69 30
    rts                                                     ; 80d2: 60
    equb &20, &e3, &80, &ca, &ca, &20, &db, &80             ; 80d3:  .... ..
    equb &b1, &b0, &9d, &72, &10, &e8, &c8, &60             ; 80db: ...r...`
    equb &20, &e6, &80, &b1, &b0, &95, &ba, &e8             ; 80e3:  .......
    equb &c8, &60                                           ; 80eb: .`

The output is not necessarily all that useful at this early stage, but it can be reassembled and will re-create the input perfectly. As long as you don't use the expr() or expr_label() features (see below for an example of expr()), the output of py8dis should *always* have this property.

The output includes a hex dump for each line; this can be turned off later, but to start with it's very helpful for getting the hex addresses we'll need as we incrementally add to the control file.

We decide to take a look at the service call handler as a starting point. We see that it is split up a bit; the main code starts at the "service_handler" label at &BEC8 but it does a "JMP &B1B1" fairly early on. To try to keep things straight as we puzzle the code out, let's use a more meaningful label by adding (all additions are towards the end of dfs226.py, just above "go()"):

Code: Select all

label(0xb1b1, "general_service_handler")

Re-running dfs226.py will then apply this label in place of the auto-generate lb1b1 label.

general_service_handler does a JSR to a JSR and it starts to feel a bit confusing. laea9 does "cmp #&09"; does the accumulator still contain the service call number at this point? We inspect the code and decide it does, which means that &09 is the service call number for *HELP. We annotate the disassembly further by adding:

Code: Select all

constant(0x09, "service_help")
expr(0xaeaa, "service_help")

constant() just defines a fixed constant - it's like writing "service_help = &09" in an assembler input file.

The expr() call says that the byte at &AEAA is a reference to the service_help constant. If we re-run dfs226.py the output now has:

Code: Select all

.laea9
    cmp #service_help                                       ; aea9: c9 09
    bne laed7                                               ; aeab: d0 2a

At the moment py8dis does *not* evaluate the expression itself - it is just passed through to the assembler. This means that if you'd set service_help to 0x08 instead of 0x09, the output would no longer re-assemble the input correctly, so you need to be careful. In practice I tend to run py8dis from a shell script (utils/reassemble.sh) which automatically assembles the output and compares it to the original file.

Obviously this is a fairly tedious way to introduce named constants into the assembly. At some point you will have identified all the segments of code and data in the binary, documented them in the control file and you'll then take py8dis's output and start hacking on it in a text editor as you study the code further. The idea behind expr() is just that you can do some initial addition of named constants while you're still iterating with the disassembler.

As we browse the output some more, we spot this weirdness:

Code: Select all

    tya                                                     ; aebc: 98
    beq laed3                                               ; aebd: f0 14
    jsr l8077                                               ; aebf: 20 77 80
    ora l5554                                               ; aec2: 0d 54 55
    equs "BE HOST 2.30"                                     ; aec5: 42 45 20 ...
    equb &0d, &ea                                           ; aed1: ..
.laed3

It looks like &8077 is a subroutine which takes an inline string, but py8dis doesn't know this yet and it's trying to interpret the bytes after the JSR as code.

Studying the code at &8077, we work out what it's doing and add a comment to remind us:

Code: Select all

comment(0x8077,
"""Print (XXX: using l809f, which seems to be quite fancy) an inline string
terminated by a top-bit set instruction to execute after printing the string.
Carry is always clear on exit.""")

We also explain this to py8dis so it can get the disassembly correct:

Code: Select all

hook_subroutine(0x8077, "print_inline_l809f_top_bit_clear", stringhi_hook)

hook_subroutine() assigns a label to the subroutine at &8077 and tells py8dis that it should call the provided "hook" function whenever it disassembles a JSR to that address. Here we use the standard stringhi_hook function which converts everything up to but not including a top-bit-set byte after the JSR into a string and resumes disassembly with the top-bit-set byte.

Re-running the disassembly, we now have this:

Code: Select all

    tya                                                     ; aebc: 98
    beq laed3                                               ; aebd: f0 14
    jsr print_inline_l809f_top_bit_clear                    ; aebf: 20 77 80
    equs &0d, "TUBE HOST 2.30", &0d                         ; aec2: 0d 54 55 ...
    nop                                                     ; aed2: ea
.laed3

and other calls to that same subroutine have also been disassembled correctly.

Staring intently at the disassembly a bit longer, we find the service call 4 (unrecognised * command) handler and realise it's dispatching to different routines for different commands via the subroutine at &8703 and a table at &861C. Rather that try to understand the code properly, we spot the suspicious-looking "lda:pha:lda:pha:rts" code at &873C which suggests the table contains addresses suitable for use with RTS (i.e. one byte before the actual addresses of the corresponding code). Looking further at the table we deduce that each entry seems to have the format:

• command name
• big-endian address of code-1; as this is a ROM the high byte of the address will always be >=&80 so this terminates the command name implicitly with a top-bit-set byte
• some sort of extra byte

Let's get the disassembly to show the table correctly and make sure py8dis knows that some of those bytes point to code it should be disassembling. We add:

Code: Select all

pc = 0x861c
label(pc, "command_table")
for i in range(20):
    pc = stringhi(pc)
    pc = rts_code_ptr(pc + 1, pc)
    pc += 1 # XXX: what are we skipping here?

We loop over the table entries and for each one:

• We call stringhi() to mark the command name as a string and to get the address of the top-bit-set byte terminating it.
• We call rts_code_ptr() to indicate that there's an RTS-style pointer with its low byte at address pc+1 and its high byte at pc.
• We skip over the byte we don't currently understand; it will get labelled as byte data by default.

The disassembly now looks like this:

Code: Select all

.command_table
l861d = command_table+1
    equs "ACCESS"                                           ; 861c: 41 43 43 ...
    equb >(l89e6-1)                                         ; 8622: .
    equb <(l89e6-1)                                         ; 8623: .
    equb &32                                                ; 8624: 2
    equs "BACKUP"                                           ; 8625: 42 41 43 ...
    equb >(la417-1)                                         ; 862b: .
    equb <(la417-1)                                         ; 862c: .

Note that the addresses of the handlers are now being calculated at assembly time.

Examining the disassembly further, we notice there's another subroutine which takes inline data:

Code: Select all

.l9436
    jsr l9ad8                                               ; 9436: 20 d8 9a
    jsr l8048                                               ; 9439: 20 48 80
    ora (l0045),y                                           ; 943c: 11 45
    equs "scape"                                            ; 943e: 73 63 61 ...
    equb &00                                                ; 9443: .

Studying the code at &8048, we realise it's a relatively complex bit of code with two possible behaviours. Let's document them in a comment and tell py8dis how to handle them:

Code: Select all

comment(0x8048,
"""Generate an OS error using inline data. Called as either:
    jsr XXX:equb errnum, "error message", 0
to actually generate an error now, or as:
    jsr XXX:equb errnum, "partial error message", instruction...
to partially construct an error (on the stack) and continue executing
'instruction' afterwards; its opcode must have its top bit set. Carry is
always clear on exit.""")

def generate_error_hook(target, addr):
    # addr + 3 is the error number
    pc = stringhiz(addr + 4)
    if memory[pc] == 0:
        # An OS error will be generated and the subroutine won't return.
        return None
    else:
        # A partial OS error will be constructed on the stack and the subroutine
        # will transfer control to the instruction following the partial error.
        return pc

hook_subroutine(0x8048, "generate_error", generate_error_hook)

Here there isn't a standard hook subroutine like stringhi_hook that will do what we want, so we write our own. generate_error_hook() is called whenever a "JSR &8048" instruction is disassembled; it receives the address of the JSR instruction as "addr". It uses the standard stringhiz() function to mark the following string and find the terminator, then it uses the terminator to decide which instruction (if any) will be executed next.

I'll stop here, but you can see a fuller (but by no means complete; it was just created as a test/demonstration of py8dis) version of the disassembly in examples/dfs226.py. There are also example disassemblies of ANFS 4.18 and BASIC 4r32. You can also see how commands like rts_code_ptr() are implemented in commands.py, which may be helpful in writing your own variants.

I suspect the way this all works might be a little idiosyncratic; since I wrote it and I've been evolving it as I go along it feels relatively natural to me (if a little clunky in places), but I have no idea if anyone else will be able to get along with it or not. Comments, questions (except "why?" ), bug reports and feature requests are welcome!

I think it's somewhat inevitable that you will run up against assertion failures with somewhat scary looking backtraces as a result of the "disassembler as a library" approach; I am open to trying to make things a bit more friendly if possible, but I'd also hope that with a bit of practice these become useful at indicating what's gone wrong. Please post if you give this a try and get stuck and I'll do my best to help.

[TobyLobster]

This looks excellent! It's an interesting approach, I will certainly be trying this out on my next disassembly.

[SteveF]

Thanks Toby! I'm sure that would give py8dis a good workout.

[BigEd]

Sounds great! I've posted over on the 6502 forum (hope that's ok)

[SteveF]

Thanks Ed, that's great!

[davidb]

Looks interesting! Thanks for making it available.

[gfoot]

Very nice! I was about to start disassembling Repton 2, I'll give it a go!

[SteveF]

Thanks guys!

I just hope things don't turn sour once someone actually tries to use this.

[gfoot]

It's working really well so far.

It would be handy to be able to exclude ranges from the output completely, or specify multiple disjoint data ranges as input.

[SteveF]

That's good to hear!

Would allowing multiple load() calls to load different files into different regions of the disassembler's memory be a good solution? If I did that, that would naturally imply disjoint disassembly regions as defined by those load() calls. I could probably implement that fairly easily.

If that wouldn't be a good solution, could you maybe sketch out how this might look in a user disassembly file to give me an idea of what you'd like to see?

Edit: I suppose simply providing a set_disassembly_range() function which takes a list of disjoint ranges (e.g. set_disassembly_range(((0xe00, 0x1000), (0x2000, 0x2430), (0x4123, 0x6650)))) might be the most generic approach - would that work for you?

[gfoot]

Any of the above would work. But for more context, the specific use case is a file that loads at 1D00, and then copies parts of itself to lower addresses - a few dozen bytes to 0380, a few dozen to 0880, and the rest is relocated down from 1D00 to 0D00.

My first try was saving out separate files and using multiple "load" statements, so that's quite intuitive I think. As that didn't work, I used beebjit to let the program run enough to relocate everything, then saved out everything from 0380 through to 6000, which is ok but includes a lot of stuff that's not part of the original file.

The absolute best thing, I think, would be if you could specify ranges of addresses and their relocation addresses, and if the disassembler could then output the right directives to assemble with relocation... That would be awesome as it would truly allow you to generate a source file that recreates the original binary file.

Multiple loads is next best I think though, and as I said I think it would be quite intuitive.

[SteveF]

Thanks, that makes sense. I'll see what I can do, it might be possible to support the relocation without too much difficulty. The current insistence on load() only being allowed once and having a single disassembly range is just an artefact of ROM disassembly being my initial motivation; I thought it might not be adequate but I didn't know exactly what would be required.

Which assembler are you using? The details of emitting relocations might be different for different assemblers and it would perhaps be easier if I focus on just one first.

[gfoot]

I'm not using any assembler at the moment, just using your tool to progressively deconstruct more and more of the code, adding annotations and comments, etc. It is really useful and I like doing it by building up a script that generates the assembly, rather than just editing the assembly directly. It gives me confidence that I'm not wasting my time, because any significant changes (e.g. assembler syntax) won't invalidate all my work.

Something else that could be useful (if you're collecting feature requests!) is annotating labels with a comment saying how many times they are referenced (maybe even a list of referencing addresses). It could help tell whether a routine is a one-off helper or branch target which might be very bespoke, or something more generic that's called from all over the place and probably has a decent API.

[hoglet]

Just catching up on this thread which looks really neat. I had a brief chat with Ed about this before actually fully digesting the first post. Pretty much every tricky case I have encountered with ROM headers, embedded data, jump tables of differing formats you have already covered.

Are you able to support the processor type (6502 vs 65C02) yet?)

I have one very minor suggestion:

Code: Select all

import acorn
...
acorn.add_standard_labels()

It would be nice to accomodate standard label definitions for various Acorn systems, including
- Atom
- BBC Model B/OS 1.20
- BBC Master/MOS 3.20
- BBC 6502 Co Processor (tube regs at a different address)
etc
The Atom one in particular will be quite different.

A couple of further tricky cases to ponder:

1. Overlapping instructions, e.g. the use of opcode &2C (bit absolute) to skip the next 2-byte instruction, saving a byte.

Here's an example from Tube Elite:

Code: Select all

.MT1
     16E9   A9 00      LDA #&00
     16EB   2C
.MT2
     16EC   A9 20      LDA #&20
     16EE   8D 96 24   STA &2496
     16F1   A9 00      LDA #&00
     16F3   8D 9B 24   STA &249B
     16F6   60         RTS

2. Labels that point to instruction operands

One solution is to place the label on the instruction and then add + 1 to the reference.

There's an example in Basic 4r32:
https://github.com/hoglet67/BBCBasic4r3 ... asm#L12147

This is all really excellent work!

Dave

[gfoot]

A couple of further tricky cases to ponder:

1. Overlapping instructions, e.g. the use of opcode &2C (bit absolute) to skip the next 2-byte instruction, saving a byte.

I ran into this in the Repton 2 code, but I think it's actually a bug in the code. The disassembler deals with it pretty well though. What happens at the moment is the disassembler labels the "big" instruction that got jumped/referenced into, then creates another label relative to that to point to the byte that was referenced, and uses that second label for the reference from elsewhere. e.g. this is the example in Repton 2 - the comment is added by me to explain it to myself:

Code: Select all

    asl a                                                   ; 10f2: 0a
    asl a                                                   ; 10f3: 0a
.l10f4
; 0b 0a => ANC #&0a, equiv to AND #&0a in this context - called after screen dissolve?
l10f5 = l10f4+1
    rol zp_ptr_hi                                           ; 10f4: 26 0b
    asl a                                                   ; 10f6: 0a
    rol zp_ptr_hi                                           ; 10f7: 26 0b

The weird calling code then just shows up as a jsr to l10f5. l10f4 was only introduced here to support l10f5 - it's not directly referenced from anywhere else, py8dis did it all automatically.

2. Labels that point to instruction operands

This seems to be dealt with in the same way already. Actually this happens quite a lot for two-byte data - if you declare an address as a data word, then direct references to the second byte of the word will introduce an extra label as above. I think I'd prefer to just use "+1" in this case though. zp_ptr_hi above is a case in point - I'd prefer "zp_ptr+1" rather than using an extra label for the high byte. Maybe that's just my own coding style though.

This is all really excellent work!

It certainly is, fun to use and saving me a lot of time! Thanks Steve!

[SteveF]

Thanks all for your kind words, they are much appreciated! It's good to see there is an interest in this project - I really didn't want to just throw it up on github in a half-baked form once I'd proved to myself the basic idea was workable, but grinding through the last 10-20% of tidying and documenting was a bit of a chore, and this makes it seem worthwhile. And I am very pleased py8dis hasn't simply fallen over the moment someone other than me tries to use it...

I will make another post shortly about some of the other points raised; for this one I just want to comment on gfoot's suggestions re relocation/moving.

I've had a go at adding support for multiple regions and relocations. This is perhaps a bit hacky, so please let me know how you get on. The code is on the new multi-region branch.

It's now possible to use multiple load() commands and py8dis will generate code to re-assemble the contents of all those load()-ed files. examples/split.py is an example of this. This works with acme and beebasm. The output assembly will re-generate a single file spanning the regions populated by all those load() commands - I think this is quite natural to how acme works but with beebasm it might be more natural to add a separate SAVE command to the output for each load() command in the control file. On the other hand, if that's what you want, you could already just disassemble each of those load()-ed files separately, unless I'm missing something.

There is also a new move(dest, src, len) command, which *from the perspective of the disassembler* tells py8dis to move len bytes of memory (previously loaded via a load() command) from src to dest. In the py8dis output, that block of memory will be assembled at "dest" and a copyblock command included to copy the data back to "src" where it started originally in the input binary. examples/move.py is an example of this.

move() currently always generates a copyblock command and so won't generate valid acme output. I don't know if there's a good way to handle this in acme - if you were actually writing the program being disassembled in acme from scratch, you'd probably either "manually relocate" by writing things like:

Code: Select all

    * = $2000
    ... ; cope to copy code_to_copy_to_900_at_run_time down to $900
    lda #65
    sta $70
    ...
    jsr $900
    ...
    rts 
code_to_copy_to_900_at_run_time
    lda #42
    jsr subroutine-code_to_copy_to_900_at_run_time+$900 ; acme is assembling this "high" but it will be copied down to $900 at runtime
    rts
subroutine
    jsr $ffee
    rts

or you'd write code_to_copy_to_900_at_run_time in a separate source file, assemble it at $900 and use !binary to pull that assembled code in at the right place in the "main" executable. I'm not sure either of those approaches is really practical for py8dis to implement. I'd appreciate any thoughts on this from more experienced acme users.

[SteveF]

Actually this happens quite a lot for two-byte data - if you declare an address as a data word, then direct references to the second byte of the word will introduce an extra label as above. I think I'd prefer to just use "+1" in this case though. zp_ptr_hi above is a case in point - I'd prefer "zp_ptr+1" rather than using an extra label for the high byte. Maybe that's just my own coding style though.

You should be able to force this to happen by saying something like:

Code: Select all

expr_label(0xb, "zp_ptr+1")

instead of (as you presumably have):

Code: Select all

label(0xb, "zp_ptr_hi")

By using expr_label() you are telling pydis8 that "zp_ptr+1" will evaluate to &b at assembly time, and it will trust you blindly - if you get it wrong the output will no longer re-assemble the original input binary correctly. (I do hope to add an expression evaluator eventually so py8dis can check this, but it's not there yet.)

I don't think it's feasible to do this completely automatically, unfortunately - py8dis has no understanding of the code and no way to infer that addresses &a and &b are related. (As always, suggestions welcome!)

[SteveF]

It would be nice to accomodate standard label definitions for various Acorn systems, including
- Atom
- BBC Model B/OS 1.20
- BBC Master/MOS 3.20
- BBC 6502 Co Processor (tube regs at a different address)
etc
The Atom one in particular will be quite different.

This is a good idea and not hard to do, of course. My initial thoughts are to provide multiple routines which form two groups. One group declares official OS entry points, vectors, etc for different OS versions:

• acorn.os_120() - OS 1.20
• acorn.os_200() - OS 2.00
• acorn.os_320() - OS 3.20
• acorn.os_100() - OS 1.00 (Electron)
• acorn.os_atom() - I don't know what number (if any) the Atom OS has

and another group declares hardware-specific addresses, such as ROMSEL, tube hardware registers:

• acorn.hardware_bbc_b()
• acorn.hardware_bbc_b_plus()
• acorn.hardware_master()
• acorn.hardware_electron()
• acorn.hardware_atom()
• acorn.hardware_6502_tube() - hardware addresses for code running on 6502 copro

How does that sound? It would be possible to pass arguments to a single function, but I'm not sure it would add that much in terms of readability as you'd end up writing things like acorn.add_standard_os_labels(acorn.OS_120).

I think gfoot has addressed hoglet's other points (apologies for awkward third-person construction!). If there's anything I've managed to overlook in anyone's post please let me know!

Edit: I forgot this:

Are you able to support the processor type (6502 vs 65C02) yet?)

Yes, if you import from trace65c02 instead of trace6502 it will disassemble the CMOS opcodes. examples/basic4.py uses this.

[SteveF]

Just to waffle a bit redundantly: The tracing disassembler will refuse to disassemble an instruction if any of its bytes have been previously classified, whether they have been classified as an instruction, byte, word, whatever. So where instructions overlap it might be a little bit arbitrary which instruction "wins", but the output should always generate a correct (if not necessarily optimal) set of instructions and equb directives to re-generate the input correctly. You then have the fun of figuring it out for yourself from that.

Internally, py8dis associates an arbitrary number of "annotations" with an address (these are things like labels and comments) and attempts to assign a single "classification" to every byte in the load()ed ranges. Classifications are things like "this is a string of length n", "this is a block of 5 2-byte words" or "this is a 2-byte instruction". An address can only have a single classification (and that classification might be "I am part of a multi-byte classification"). Although this could be changed, at the moment classifying an address is irreversible - you'll probably get an assertion if you try to do something like:

Code: Select all

byte(0x900)
byte(0x900) # error

or

Code: Select all

word(0x900)
byte(0x901) # error

I think refusing to modify classifications is fairly reasonable - the tracing disassembler uses this to resolve conflicts, and nothing automated in py8dis should try to modify an existing classification. If the user's control file is saying two contradictory things about the same address, as in the examples above, it's probably an error, not something we want to brush under the carpet.

[SteveF]

Something else that could be useful (if you're collecting feature requests!) is annotating labels with a comment saying how many times they are referenced (maybe even a list of referencing addresses). It could help tell whether a routine is a one-off helper or branch target which might be very bespoke, or something more generic that's called from all over the place and probably has a decent API.

This is a good idea; of course it's something the assembler could potentially do, but in practice it won't, so I've added an experimental implementation of this on the multi-region branch. It counts (barring bugs) references to addresses as targets of branch, jmp or jsr instructions (not references as data) and adds a comment at the target address showing the count and addresses of the instructions referring to it. (Raw addresses are used, since in general the referring instructions won't have labels and it seemed best to be consistent.)

It's currently on by default but can be turned off by saying:

Code: Select all

config.set_label_references(False)

This is a fairly quick hack so let me know if it doesn't work or you have suggestions for improvement...

[TobyLobster]

move() currently always generates a copyblock command and so won't generate valid acme output. I don't know if there's a good way to handle this in acme

Acme has a different approach. You can mark a range of code with e.g. !pseudopc $0900 { ... } and that code will still be assembled at the current location (e.g. $2000) but as if it were already located at $0900 (i.e. any labels defined in that block will start from $0900). This works quite well in practice.

[SteveF]

Thanks Toby, I had no idea you could do that! I will see if I can make py8dis generate this style of output too.

On a different note, I've tweaked the label reference stuff mentioned in my earlier post to also show a table of label references in descending order of frequency at the end of the disassembly, as well as adding comments with the list of references on each individual label.

[gfoot]

I've had a go at adding support for multiple regions and relocations. This is perhaps a bit hacky, so please let me know how you get on. The code is on the new multi-region branch.

Awesome, I'll give it a try when I can.

I see a lot of value in this, especially for things like DFS ROMs where there is embedded code that's intended to be dynamically relocated to RAM before running. Though dealing with multiple code fragments that are all intended to run at the same address, within the disassembler, might not be practical anyway!

It's now possible to use multiple load() commands and py8dis will generate code to re-assemble the contents of all those load()-ed files. examples/split.py is an example of this. This works with acme and beebasm. The output assembly will re-generate a single file spanning the regions populated by all those load() commands - I think this is quite natural to how acme works but with beebasm it might be more natural to add a separate SAVE command to the output for each load() command in the control file. On the other hand, if that's what you want, you could already just disassemble each of those load()-ed files separately, unless I'm missing something.

It's an interesting one because large programs when assembled on a Beeb tend to need to be assembled in parts anyway. My instinctive preference is to disassemble them into the same parts. But I haven't got that far yet.

I think it is still important to analyze all the parts together in the disassembler though because you want to deal correctly with cross-references, where one part calls code in another part - you want the label name to be shared between the two.

There is also a new move(dest, src, len) command, which *from the perspective of the disassembler* tells py8dis to move len bytes of memory (previously loaded via a load() command) from src to dest. In the py8dis output, that block of memory will be assembled at "dest" and a copyblock command included to copy the data back to "src" where it started originally in the input binary. examples/move.py is an example of this.

I think that's right, though many assemblers can dynamically perform the relocation for you, assembling code to run at one address but writing it into their output at another address.

In general I'd expect the runtime code to copy the block to its target address would just show up somewhere else in the code, or would be part of a separate loader. Either way, no need for py8dis to do anything fancy there I think - the author of the code being disassembled already solved that problem somehow and all we should try to do is be ready to present their solution as neatly as we can in the disassembly.

I'm looking forward to trying the other new features too. One more idea I've been toying with is making the automatic label name generation use different styled names for code and data. I always prefix zero-page variable names with "zp_" as they are syntactically different in some instructions, for example. I also wondered about making it detect things like labels that just point at RTS instructions and name them "rts1", "rts2", etc to make that clear. I'm not sure though.

My least favourite feature at the moment is "expr" as it feels quite error-prone. I'm not sure of a way to make it better though. I've wrapped it up a bit in my own code, so that I can declare a constant and all the places it's used in one go, and I think that is useful. I don't want to criticize though because I haven't used it enough yet, and may not have taken the time to fully understand it yet either.

Code: Select all

objtypes = {
	2: ("obj_empty", [0x1533, 0x154c, 0x155e, 0x1584, 0x15b5, 0x1781, 0x19ac, 0x19c1, 0x19f3]),
	6: ("obj_diamond", [0x159b, 0x1ed8]),
	7: ("obj_key", [0x1edf]),
	8: ("obj_skull", [0x1ee6]),
	9: ("obj_end", [0x1eed]),
	11: ("obj_transporter", [0x1efb]),
	13: ("obj_safe", [0x1597]),
	14: ("obj_rock", [0x16c8, 0x1845, 0x1861]),
	15: ("obj_egg", [0x1849, 0x1865]),
	16: ("obj_notsure", [0x16cc]),
}

for k,(name,occurences) in objtypes.items():
	constant(k, name)
	for occurence in occurences:
		expr(occurence, name)

[SteveF]

I think it is still important to analyze all the parts together in the disassembler though because you want to deal correctly with cross-references, where one part calls code in another part - you want the label name to be shared between the two.

I think this is true, but in principle (I haven't tried it) if you are disassembling two related files foo and bar you should be able to have foo.py and bar.py to disassemble each one and share any common symbols by putting them in my_common_symbols.py and doing "from my_common_symbols import *" in both foo.py and bar.py.

I'm looking forward to trying the other new features too. One more idea I've been toying with is making the automatic label name generation use different styled names for code and data. I always prefix zero-page variable names with "zp_" as they are syntactically different in some instructions, for example. I also wondered about making it detect things like labels that just point at RTS instructions and name them "rts1", "rts2", etc to make that clear. I'm not sure though.

This is an interesting idea. It would be possible to provide hooks so a control file can override the default names assigned for labels, which would make it easy to use "zp_" for zero page addresses. Code-vs-data labels would require taking a view once the whole program has been disassembled, so we'd need to go and rename them (probably not a big deal) based on their uses (the "label X referenced by code at points X, Y, Z" feature I added the other day at your suggestion might provide the necessary data) just before emitting the disassembly.

Edited to add: I've had a quick crack at adding code-vs-data labels. Unfortunately it's not trivial to do, because the tracing process generates expressions - which are just strings - which contain label names as it goes, so renaming a label later on (when we realise it's a code label) is a bit fiddly. I still think this is a good idea, but it will probably have to wait for a code tidying/rewriting stage before I implement it.

My least favourite feature at the moment is "expr" as it feels quite error-prone. I'm not sure of a way to make it better though. I've wrapped it up a bit in my own code, so that I can declare a constant and all the places it's used in one go, and I think that is useful. I don't want to criticize though because I haven't used it enough yet, and may not have taken the time to fully understand it yet either.

I like the way you've wrapped this up, it's a lot more readable. It might be a good idea to add something like that code as a standard function (which would just take your objtypes object as its argument) in commands.py.

In terms of improving its safety, py8dis really needs to evaluate the expression itself and check it gives the same value as is present in the input binary. (Similarly for expr_label(), although there we'd check it evaluates to the address.) I have done some playing around this afternoon with pyparsing and it looks like doing this is probably not too difficult, but I haven't finished playing around with it yet.

In the meantime, it occurred to me that we can just emit a series of "assert" directives at the end of the disassembly to cause an assemble-time failure if an expression doesn't have the expected value. ("assert" is a beebasm thing, but for acme we can easily simulate it using !if/!error.) This won't catch errors as early as if py8dis evaluated the expressions, but it *will* ensure they are caught at assembly time.

I've pushed a version of py8dis which will generate the assertions to the multi-region branch.

[gfoot]

Multiregion is working really well for me, it's removed a whole bunch of inlined data which really tidies things up. I haven't tried relocation yet.

The reference comments are incredibly useful, it felt intrusive at first but is well worth it IMHO.

My next feature suggestion is context-specific labels. Especially for zero page, the meaning of an address often varies in different bits of code. In assembler, you'd just define multiple variables with the same address, and use whichever name makes sense in each bit of code. We can use "expr" to override the disassembler of course but it's very tedious. Instead, I wonder if we could somehow choose which label name to emit depending on which segment of code is being processed?

e.g. at the moment in Repton 2 I have:

Code: Select all

label(0x6b, "zp_rock_se__suppressleadingzerosflag")

This zero page location is used in some print routines to suppress leading zeros, and in the rock movement code to track what object is in the cell southeast of the rock. I experimented with just putting both names here but it's really clumsy and these have even more uses in different bits of code. But if I could write something like this, it could work better:

Code: Select all

label(0x6b, "zp_suppressleadingzerosflag", 0x0d00, 0x0e00)   # Between 0d00 and 0e00, use this name
label(0x6b, "zp_rock_se")                                                      # Otherwise use this name

Or maybe defining context regions more formally:

Code: Select all

printing_code = define_contextregion(0xd00, 0xe00)
rock_movement_code = define_contextregion(0x1300, 0x1500)

printing_code.label(0x6b, "zp_suppressleadingzerosflag")

rock_movement_code.label(0x6b, "zp_rock_se")

I haven't thought this through, it's just some ideas. I wonder if context regions could be useful for other things as well?

[SteveF]

Thanks for giving that a test, I'm glad it's working out!

Context-specific labels seems like a sensible feature. I am thinking what I really need to do is rejig the label handling so it postpones converting addresses to labels as long as possible and offers a hook the chance to generate a label name given a) the address we want the label for b) the address we want to use that label at, which would allow you to implement context-specific labels easily, including potentially differentiating code and data labels.

I don't think this will be tremendously difficult but it might involve quite a bit of internal restructuring. I'll give it a try anyway...

[gfoot]

I tried the relocation code as well the other day and it did a good job. I think it would be better though if the final output had all the sections in the order they appeared in the input - right now it seems to do them in the order they appear in relocated memory.

e.g. for Repton 2, it loads at 1d00, then relocates the region from 1d00-7000 down to 0d00-6000; and from 7000-7060 down to 0380-08e0; and from 7060-70a0 down to 0880-08c0. My custom binary also has bootstrap code at 70a0 which runs there and performs all the relocations, as well as some other setup that usually happens in a separate loader for Repton 2.

When setting this up with py8dis, the relocate of 1d00-7000 down to 0d00 didn't work - I think maybe it doesn't like overlapping source and destination ranges? The error was about reclassifications. So as a workaround I loaded the whole file at 0d00, instead of 1d00, and set the following relocations:

Code: Select all

load(0x0d00, "repton2.bin")
move(0x0380, 0x6000, 0x60)
move(0x0880, 0x6060, 0x40)
move(0x70a0, 0x60a0, 0x613e-0x60a0)

It is a little bit backwards, especially the treatment of 70a0, but still it ensures each range is disassembled based on the right address, and that all works great.

The output file then lists the regions in order of their target addresses in memory, i.e. 0380 first, then 0880, then 0d00, then 70a0. It also puts "skips" in the old locations, which may be why it didn't like overlapping relocations.

What I'd prefer is if it just output the regions in the order they were in the original file, i.e. 0d00 first, then 0380, then 0880, then 70a0. I manually moved them around in the generated code, to achieve this, and reassembled it, and got a byte-for-byte identical output to the input, so it does work well this way. This was using Andre Fachat's "xa" assembler, which I added support for as well.

Regarding support for xa - a couple of issues came up. One is that it uses a different comment character, and the hex dump system doesn't respect the assembler's comment character choice - I fixed that code up to do that though. Another was that it doesn't like lines ending with backslashes - it uses it as a continuation character - and so I had to ensure that the hex dump lines that ended with backslashes had a space after them. I'm not sure if that's worth solving for centrally though... I just did a search and replace as a postprocess.

It's really great though that this gave such good results in what I consider to be a short space of time - I have assigned sensible names to all the labels in the 20k program, added extra comments in some places, assigned constants as well to a lot of things, and mostly made the constants get used where appropriate in the code, though it's very hard to be sure you've found all the usages. I have probably missed some cases where the low or high half of a label address is loaded into a register, and I suspect that if I tried to offset the whole program by a few bytes it wouldn't work afterwards due to this. But that's hard to do and hard to detect automatically, so I think it's about as far as a semi-automated process can go.

I also marked up a lot of the data embedded in the binary, even things that aren't directly referenced by the code - and using Python was helpful here as I could use loops to do a lot of this automatically. I also added a function to your code to allow me to query back the value of a label I'd defined earlier on, to help with directly inspecting memory[] to automatically determine where certain things are in the data. For example:

Code: Select all

for i in range(0x4a):
	addr = get_addr_from_label("data_tilegraphics_indices") + i*16
	if i <= 2:
		name = "empty"
	elif i < 6:
		name = "earth"
	elif i >= 32:
		name = "puzzlepiece"
	elif i >= 16:
		name = "wall"
	else:
		name = objtypes[i][0][4:]
	label(addr, "data_tile%02x_%s" % (i, name))
	byte(addr, 16)

I don't know whether get_addr_from_label is a good idea or not. I could have just remembered the address from earlier on, when I originally assigned it.

All in all then, I'm in a really good position now to experiment with my own changes to the Repton 2 code. I think this would have taken a lot longer and been a lot less thorough without your tool!

[SteveF]

Thanks for giving it a try, I'm glad it seems to have worked out well for you overall!

I am currently modifying the code to improve the multi-region handling and make labelling more flexible (roughly following your earlier suggestions) so I won't attempt to reply in detail to your observations yet. I think you are probably right about overlapping regions - the move() support in the last version I posted was decidedly hacky. Some aspects of reshuffling code in the disassembler's memory and the interaction with the different models used by acme and beebasm are a bit confused in my mind at the moment.

Incidentally, how does xa handle code which runs at one address but is originally located elsewhere? Is it like acme's !pseudopc or beebasm's copyblock or something completely different?

It would be good to include xa support once I finish hacking the code about. The current multi-assembler support is definitely a bit hacky, but I didn't want to tie myself in too many knots creating a perfectly generic assembler interface layer straight away. (For example, I just assume "<" and ">" can be used to get lo/hi byte of a 16-bit value, since both acme and beebasm support them.) Likewise, if there isn't some existing function to implement your get_addr_from_label() - at the moment I'm honestly a bit unsure because the code is in a state of flux - that seems like a sensible addition to the standard commands.

Will you be publishing your disassembly anywhere? If not, would you mind sharing your Python control file/modified py8dis code/input binaries with me privately so I can use them for testing? I understand if you'd rather not, and there's no immediate rush because I want to finish the current rework before I would look at them anyway.

[gfoot]

Yes please don't consider my requests anything more than suggestions, there's no urgency behind them. If I'm really blocked by something I can fix it myself

Incidentally, how does xa handle code which runs at one address but is originally located elsewhere? Is it like acme's !pseudopc or beebasm's copyblock or something completely different?

It uses * to define the current program counter, but changing its value only affects the values of labels, it doesn't cause it to leave a gap in the output or anything like that. So given this code:

Code: Select all

    * = $400
label1:
    lda label2
    rts

    * = $800
label2:
    .byt $ff

it will output 5 bytes - one lda instruction, an rts, and the constant $ff. The lda will refer to $800 because that's label2's value. The only effect of changing * is really the values of labels defined after that point.

To actually support this in my custom DFS ROM, I did something like this:

Code: Select all

    ... Other code here ...
    
nmi_handler:

    * = $d00
nmi_start:
    lda $fe80
    sta $ffff
    nmi_addr = *-2
    inc nmi_addr
    bne nmi_rti
    inc nmi_addr+1
nmi_rti:
    rti

    nmi_len = *-nmi_start
    * = nmi_handler+nmi_len
    
    ... more code ...

So nmi_handler is the address within the ROM to copy from, nmi_len is the length of the handler, and the other labels are all in page D where the handler ends up.

I'm sure there are a lot of ways to write this sort of thing, but I found this one led to fairly clear source code.

Will you be publishing your disassembly anywhere? If not, would you mind sharing your Python control file/modified py8dis code/input binaries with me privately so I can use them for testing? I understand if you'd rather not, and there's no immediate rush because I want to finish the current rework before I would look at them anyway.

I'll share it soon I hope on GitHub, I wasn't sure if there were any copyright concerns with this sort of thing though.

[SteveF]

Yes please don't consider my requests anything more than suggestions, there's no urgency behind them. If I'm really blocked by something I can fix it myself

The beauty of open source.

Incidentally, how does xa handle code which runs at one address but is originally located elsewhere? Is it like acme's !pseudopc or beebasm's copyblock or something completely different?

It uses * to define the current program counter, but changing its value only affects the values of labels, it doesn't cause it to leave a gap in the output or anything like that.

That's great, that's pretty much the same as acme's !pseudopc from a py8dis perspective I think, just a slightly different syntax.

Will you be publishing your disassembly anywhere? If not, would you mind sharing your Python control file/modified py8dis code/input binaries with me privately so I can use them for testing? I understand if you'd rather not, and there's no immediate rush because I want to finish the current rework before I would look at them anyway.

I'll share it soon I hope on GitHub, I wasn't sure if there were any copyright concerns with this sort of thing though.

I see you've recently posted this, thanks!

I have pushed a new version of py8dis to the lazy branch in github. This is a bit hacky but I thought it was worth posting before I continue tinkering with it. There are two main user-visible changes:

• move() is hopefully a little more robust and general and more things will probably work. It also outputs the move()d code in the order it appears in the original binary as you suggested; this falls out naturally from adopting the acme !pseudopc model as the "native" model and emulating it on beebasm.
  • What *won't* work is move()ing more than one piece of code to the same address - this would be tempting if you were disassembling (say) a ROM which can copy multiple different routines into RAM depending on context, but it doesn't really fit with the tracing model. I think it might be possible to offer some kind of support for applying a user-specified offset to such code, but I haven't forced myself to think about it yet.
• You can now (as seen in a trivial form in examples/basic4.py) optionally install a "user label maker hook", which will be called whenever py8dis wants to generate a label. It receives the address to be labelled, a "context" (the address the label is going to be used, e.g. the address of a JMP operand) and a suggested label name as a string. The user hook can return None or the suggested label name to use the suggestion, otherwise it can return an arbitrary label name(though if it isn't deterministic things are likely to go badly wrong ). It's probably obvious but note that there's no need to "pre-declare" possible label names via the label() command - basic4.py shows this, creating the "copy_to_stack_loop" label solely in the hook.

This is not terribly polished code as I'm still working on it. I will probably take a look at your Repton 2 disassembly in the near-ish future and also see if I can add proper xa support - I may try to improve the assembler abstration in general a bit while I'm at it.