New emulator: beebjit

[scarybeasts]

Hi,

I'd like to announce a new BBC model B emulator, beebjit.

It's an open source project, hosted on GitHub: https://github.com/scarybeasts/beebjit/
The README, which goes into a broad overview, is here: https://github.com/scarybeasts/beebjit/ ... ter/README
I did a technical blog post overview of how the JIT works, here: https://scarybeastsecurity.blogspot.com ... 15ghz.html

I developed it on Linux and it is not well ported yet. But I know there are a lot of Windows users in the beeb community so I made a Windows build here: https://github.com/scarybeasts/beebjit/ ... _0.9.0.zip
[warning: I think I broke the -fast flag which is a shame, I'll fix it later]

We already have a plethora of great BBC emulators that meet current community needs well. Chances are beebjit may not be the emulator for you, particularly in it's current "tech preview" status, and without support for models other than BBC model B, and with other deficiencies covered in the README above.

On the flipside, in terms of bringing something new to the table, beebjit has:
- Good raw speed. (I'm sure there are some applications of a GHz speed 6502 / BBC that I haven't thought of.)
- Accurate disc emulation.
- An FSD disc loader, which coupled with accurate disc emulation, enables @billcarr2005's original protected disc FSDs to be loaded and played with an experience close to the original.
- Ability to convert FSDs to the more standard HFE. HFE files can be loaded on to a Gotek + FlashFloppy, giving easy access to protected discs on a real beeb.

On the topic of HFEs, I'll be releasing a bunch converted from @billcarr2005's FSDs over the next few days.


Any questions, comments, feedback welcomed!


Cheers
Chris

[Diminished]

I am a big fan of the inherent poetry in the phrase "caches and pipelines and reordering and pain".

I've always had a vague idea of how JITs must work, but I've never really seen any specific details, so the writeup is very helpful in this regard, particularly since the 6502 is so simple. (How they manage to do this kind of thing for something like the IBM 750 in something like the GameCube scrambles my brain.)

I'll look at the emulator soon (currently watching dumb YouTube dreck).

Great work.

[Diminished]

Thought it might be interesting to give CLOCKSP a try on my box -- an aging Intel i7-4790K which is nevertheless overclocked to the same 4.5 GHz as the i9 mentioned on the blog.

This is a strange way of doing benchmarks, but not too shabby!

[scarybeasts]

Thought it might be interesting to give CLOCKSP a try on my box -- an aging Intel i7-4790K which is nevertheless overclocked to the same 4.5 GHz as the i9 mentioned on the blog.

Thanks! It's great to see this running on peoples' machines other than mine. Is that with the command line in the EXAMPLES file?

I suspect beebjit scales fairly linearly with clock speed, because beeb-sized workloads should fit inside the processor's L1 caches and certainly the L2 caches
You're a little lucky beebjit runs at all. I think 4th gen (as you have here) is the first generation to have some of the more esoteric x64 instructions that beebjit uses. (In case anyone were to try beebjit on an even older machine, the command line flag -mode interp will fall back to your typical 6502 interpreter, although it is a reasonably zippy interpreter.)

It's interesting to see that 4th gen -> 9th gen seems to result in only about a 20% IPC gain, at least for this in-cache workload.

If you want to continue to have fun, some of the BASIC "fun" samples posted to stardot over the years are _particularly_ fun at 10GHz. Mandelbrot renders, random circle drawing, PI digit calculation, etc.


Cheers
Chris

[SteveF]

This looks very cool! I haven't tried it yet, but I did enjoy reading about the JIT. My STEM project has a very long-running test suite and it would be handy if this emulator would allow it to be executed super-fast.

[Diminished]

Thanks! It's great to see this running on peoples' machines other than mine. Is that with the command line in the EXAMPLES file?

No, sorry! I confess I didn't spot that! I just ran it with "-fast", and then minimised the window to avoid video overhead.

With the recommended CLI from EXAMPLES it does better still, achieving rather spooky levels of performance.

Yay, overclocking. I wonder whether Intel (knowing that tasks like video and gaming are becoming the dominant performance case) have chosen to target floating-point performance over integer in newer generations of their CPUs (I assume this does basically no floating point work).

Good call on the Mandelbrot idea. I remember getting the BEEBUG magazine cassette for the issue (4.9 ?) with a BASIC implementation by Kate Crennell (?) and watching in impatient awe as it took 15 minutes to draw the initial set. So I think I will have some fun with this tomorrow.

I should just note for science that I had to edit the build shellscripts to remove -Werror in order to get it to build. I'm not sure what my default gcc on here is. EDIT: gcc (Debian 9.3.0-8) 9.3.0.

[SteveF]

I should just note for science that I had to edit the build shellscripts to remove -Werror in order to get it to build. I'm not sure what my default gcc on here is.

Me too! (On Ubuntu 18.04 LTS, gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0)

You're a little lucky beebjit runs at all. I think 4th gen (as you have here) is the first generation to have some of the more esoteric x64 instructions that beebjit uses.

I'm not so lucky, I think. The built-in tests fail:

Code: Select all

unusual:video:horizontal total: 1

Program received signal SIGILL, Illegal instruction.
asm_x64_asm_set_intel_flags_from_scratch () at asm_x64_common.S:362
362	  pext REG_6502_OF_64, REG_SCRATCH1, REG_SCRATCH2

This is on an Intel i5-2500K. It does start up with "-mode interp", as you'd expect if this is the issue. A bit disappointing but I will eventually succumb to the temptation to upgrade my motherboard and CPU. beebjit sounds very cool but probably isn't quite enough to spend a few hundred quid to experience in its full glory though.

[scarybeasts]

Yay, overclocking. I wonder whether Intel (knowing that tasks like video and gaming are becoming the dominant performance case) have chosen to target floating-point performance over integer in newer generations of their CPUs (I assume this does basically no floating point work).

I wish I knew more about this area. My best guess is that because much software is bound by the speed of memory and the larger caches, there is less focus on beebjit like-things.

Good call on the Mandelbrot idea. I remember getting the BEEBUG magazine cassette for the issue (4.9 ?) with a BASIC implementation by Kate Crennell (?) and watching in impatient awe as it took 15 minutes to draw the initial set. So I think I will have some fun with this tomorrow.

There's an awful, terrible regression bug that will hang in -fast mode if you change the BBC MODE, as mandelbrot renders will tend to do. It is fixed on master by https://github.com/scarybeasts/beebjit/ ... 4edaec0b2d, which I would recommend grabbing.

If you want to see how beebjit "is getting on" while running something computationally intensive, there's the flag -log perf:speed, which will report on GHz once per second. My slow laptop is about 7GHz CLOCKSP (reported total) and it cranks through a burning ship BASIC mandelbrot render at about 4.7GHz.

I should just note for science that I had to edit the build shellscripts to remove -Werror in order to get it to build. I'm not sure what my default gcc on here is. EDIT: gcc (Debian 9.3.0-8) 9.3.0.

Sorry about that. I have some testing VMs lying around for this sort of thing so I'll look at what's up.


Cheers
Chris

[scarybeasts]

I should just note for science that I had to edit the build shellscripts to remove -Werror in order to get it to build. I'm not sure what my default gcc on here is. EDIT: gcc (Debian 9.3.0-8) 9.3.0.t

I fixed a few compile warnings (== errors) on my Ubuntu 19.10 install, under both gcc and clang. The fix is on master. Hopefully this addresses the same ones you were hitting

A few extra notes that may help with performance testing:

- The default output of build.sh is to check the tests against the debug build. Sometimes it's nice to have the debug build because it has extra asserts to trap bugs but sometimes it is nice to go as fast as possible! To go as fast as possible just run build_opt.sh and you'll be left with an optimized build instead.
- The -fast flag is nice for BASIC testing. It runs the CPU as fast as possible while ticking the timers and video hardware at wall clock speed. Some games don't mind this (but may run at normal speed on account of syncing to the wall clock speed vsync). Other games won't like this at all.
So there's fast mode #2: run without the -fast flag but just whack Alt+F to toggle speedy vs. normal at any time. Fast mode #2 runs the BBC cycle-for-cycle accurate, with the timers and video hardware keeping pace with the accelerated CPU.



Cheers
Chris

[scarybeasts]

You're a little lucky beebjit runs at all. I think 4th gen (as you have here) is the first generation to have some of the more esoteric x64 instructions that beebjit uses.

I'm not so lucky, I think. The built-in tests fail:

Code: Select all

unusual:video:horizontal total: 1

Program received signal SIGILL, Illegal instruction.
asm_x64_asm_set_intel_flags_from_scratch () at asm_x64_common.S:362
362	  pext REG_6502_OF_64, REG_SCRATCH1, REG_SCRATCH2

This is on an Intel i5-2500K. It does start up with "-mode interp", as you'd expect if this is the issue. A bit disappointing but I will eventually succumb to the temptation to upgrade my motherboard and CPU. beebjit sounds very cool but probably isn't quite enough to spend a few hundred quid to experience in its full glory though.

Yes, "pext" is an exotic instruction that was introduced in Haswell == gen 4?
It's kind of cool actually: https://www.felixcloutier.com/x86/pext

However, I think I may have used the instruction _because it cool_, and not _because it is a good idea_. I don't think it's a good idea because older machines are quite common, and also this instruction is documented as performing terribly on AMD (speaking of emulation, an AMD emulates them instead of having dedicated silicon). (And by terribly we mean: http://i.imgur.com/tQVadpL.png)

If I change beebjit to get rid of them, would you be willing to try again?


Cheers
Chris

[0xC0DE]

What a magnificent (and scary) beast!
I thoroughly enjoyed reading your writeup, well done!

[billcarr2005]

Hi,

I'd like to announce a new BBC model B emulator, beebjit.

..

On the flipside, in terms of bringing something new to the table, beebjit has:
- Good raw speed. (I'm sure there are some applications of a GHz speed 6502 / BBC that I haven't thought of.)
- Accurate disc emulation.
- An FSD disc loader, which coupled with accurate disc emulation, enables @billcarr2005's original protected disc FSDs to be loaded and played with an experience close to the original.
- Ability to convert FSDs to the more standard HFE. HFE files can be loaded on to a Gotek + FlashFloppy, giving easy access to protected discs on a real beeb.

On the topic of HFEs, I'll be releasing a bunch converted from @billcarr2005's FSDs over the next few days.


Any questions, comments, feedback welcomed!


Cheers
Chris

Fantastic!

[BigEd]

Brilliant! And thanks for the detailed writeup!

(If you could gradually chip away at the exotic instructions, so it can run on slightly older harder, or run well on AMD, that would be great. I think I've got a seven year old laptop with 4th generation and one eight years old with 3rd generation.)

Maybe some of the lessons learnt could be applied to jsbeeb, allowing for a fast second processor?

And maybe PiTubeDirect could benefit too...

Neither of the above being x86, of course. But I suspect the discoveries and the architecture are a major leap forward.

[bob147]

I can confirm unsurprisingly that a 2013ish ivy bridge core i5 desktop requires interpreter mode for this to function. I can also confirm that using the windows build linked above, that the castle quest fsd works great! This is very impressive, and I enjoyed reading about all of the weird and wonderful copy protection tricks that various developers used.

[Diminished]

I enjoyed reading about all of the weird and wonderful copy protection tricks that various developers used.

I still love the irony that Kevin Edwards' Nightshade protection system -- which superficially one would think would be something that would make preservation efforts more difficult -- instead ended up accidentally becoming the go-to acid test for emulation accuracy.

[scarybeasts]

I still love the irony that Kevin Edwards' Nightshade protection system -- which superficially one would think would be something that would make preservation efforts more difficult -- instead ended up accidentally becoming the go-to acid test for emulation accuracy.

Rest assured, Kevin Edwards' name was cursed most terribly when I discovered it is possible to pass Rich's VIA tests yet still fail Nightshade unpacking -- on like the 3rd or 4th unpacker in a chain of pain, of course.


Cheers
Chris

[SteveF]

Yes, "pext" is an exotic instruction that was introduced in Haswell == gen 4?
It's kind of cool actually: https://www.felixcloutier.com/x86/pext

However, I think I may have used the instruction _because it cool_, and not _because it is a good idea_. I don't think it's a good idea because older machines are quite common, and also this instruction is documented as performing terribly on AMD (speaking of emulation, an AMD emulates them instead of having dedicated silicon). (And by terribly we mean: http://i.imgur.com/tQVadpL.png)

If I change beebjit to get rid of them, would you be willing to try again?

Absolutely, I'd really appreciate it if you made the effort to do this.

Could you check the CPU at run-time in the compiler and still use pext if it's a recent Intel CPU? Maybe not worth it...

[lurkio]

I'd like to announce a new BBC model B emulator, beebjit.

Great news! But because I don't fully understand all the tech involved, it would be great if you could please explain in simple terms what the advantages of beebjit are over BeebEm, to take the obvious example? (Apart from beebjit's ability to read HFEs -- I get that bit.)

E.g. how come beebjit can run so fast? Is it somehow "closer to the silicon" (whatever that means!) than previous emulators? And I presume it's possible to actually slow beebjit down so it runs at the same speed as an original Beeb..!?

And, as I was bound to ask, will it be possible, at some point, to compile beebjit for macOS?

[Diminished]

Great news! But because I don't fully understand all the tech involved, it would be great if you could please explain in simple terms what the advantages of beebjit are over BeebEm, to take the obvious example? (Apart from beebjit's ability to read HFEs -- I get that bit.)

Basically it can run BBC Micro software in its native environment (i.e. on a properly emulated BBC system) at 5000 times its native speed (give or take). Largely useless for games, but potentially useful for some other cases. (Although I can think of a potential gaming use case for this -- speed run research. On console systems, people have tried building emulator search bots which try all sorts of different input combinations from a particular point in an attempt to brute-force-find glitches in the game, which might ultimately be of use to complete games more quickly, and the faster your emulator goes, the more such potential cases you can examine.)

It's broadly comparable to running an algorithm written in BBC Basic (which is interpreted) versus one that's written in assembly language (which runs natively). Rather than reading each 6502 instruction one-by-one and then choosing to run a corresponding block of code which performs the equivalent function to the 6502 instruction, as a traditional interpreted emulator would, it is instead able to pre-convert 6502 machine code directly to x86 machine code, which is then executed natively on the PC running it.

E.g. how come beebjit can run so fast? Is it somehow "closer to the silicon" (whatever that means!) than previous emulators?

Pretty much.

And I presume it's possible to actually slow beebjit down so it runs at the same speed as an original Beeb..!?

Yes.

And, as I was bound to ask, will it be possible, at some point, to compile beebjit for macOS?

I was wondering this too, but I assume it isn't a trivial job.

[Diminished]

This is the Kate Crennell implementation of Mandelbrot from Beebug 5.1 (not 4.9), if anyone else fancies messing with it. I could only find this on tape (from STH), so I put it on disc. (CHAIN "MANDEL").

Once a plot has completed, use the cursor keys to choose the bottom-left corner of the box you want to zoom in on, then press L. Then use the cursor keys again to choose the top-right corner of the box you want to zoom in on, and press R.

mandel.ssd

(6.5 KiB) Downloaded 102 times

(Builds and runs just fine now with the latest git version -- thanks scarybeasts).

[Coeus]

I was quite excited reading the blog. Trying to build it, though, I get:

Code: Select all

Running built-in unit tests.
info:tape:rewind
info:tape:rewind
info:tape:rewind
unusual:video:horizontal total: 7
unusual:video:vsync pulse width: 16
unusual:video:horizontal total: 1
./build.sh: line 17: 33478 Illegal instruction     (core dumped) ./beebjit -test

Is the processor generation issue mentioned above? From /proc/cpuinfo:

Code: Select all

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 58
model name	: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
stepping	: 9
microcode	: 0x21

[BigEd]

(I think so - i5-3570K matches 3xxx and means a 3rd generation)

[cmorley]

https://en.wikichip.org/wiki/intel/cpuid

Family 6 and model 58 is an Ivy Bridge.

edit: there are die pictures if you click on Ivy Bridge... I like die pictures

[Coeus]

https://en.wikichip.org/wiki/intel/cpuid

Family 6 and model 58 is an Ivy Bridge.

edit: there are die pictures if you click on Ivy Bridge... I like die pictures

The die pictures are cool.

I do find the whole thing about understanding which processor you have rather confusing, probably at least in part because the same marketing names are re-used in successive generations for completely different chips. So looking at that site Family 6, Model 58 is indeed shown against Ivybridge but then in the list of "All Ivy Bridge Chips" only i7 branded devices are shown execpt for one Celeron which may be the one I have in an HP microserver. That also family 6 model 58 but a "Intel(R) Celeron(R) CPU G1610T @ 2.30GHz"

[Coeus]

(I think so - i5-3570K matches 3xxx and means a 3rd generation)

Is there any way to decode these numbers. My work laptop has an i7-6820, not that I can reasonably install beebjit on that.

[BigEd]

I think the numbers are systematic: the leading digit is the generation number
Intel® Processor Numbers: Laptop, Desktop, and Mobile Device


You can lookup a part at
https://ark.intel.com/content/www/us/en ... abel122139

[Coeus]

If I change beebjit to get rid of them, would you be willing to try again?

I'd certainly be interested to try it out. I have no immediate plans to replace the i5 Ivybridge desktop machine I mention further down the thread. Despite the fact a lot seems to have happened since that was new it still feels very fast for most things.

When you say AMD emulate pext, by what mechanism? Do you mean it is being executed in microcode that actually does a succession of shifts and ORs rather than proper parallel execution?

[cmorley]

Ivybridge but then in the list of "All Ivy Bridge Chips" only i7 branded devices are shown execpt for one Celeron which may be the one I have in an HP microserver. That also family 6 model 58 but a "Intel(R) Celeron(R) CPU G1610T @ 2.30GHz"

https://ark.intel.com/content/www/us/en ... ridge.html

[scarybeasts]

When you say AMD emulate pext, by what mechanism? Do you mean it is being executed in microcode that actually does a succession of shifts and ORs rather than proper parallel execution?

Probably...? Hard to explain these numbers otherwise: http://i.imgur.com/tQVadpL.png

Also: https://laatste.info/bb3/viewtopic.php?f=53&t=7980


Cheers
Chris

[Diminished]

I recalled seeing the pext thing mentioned somewhere else recently:

https://dolphin-emu.org/blog/2020/02/07 ... by-techjar