Benchmarks

[SarahWalker]

I seem to have ended up going through a good variety of Arcs during lockdown...

So I thought I'd benchmark them!

I am aware I'm probably the only person who finds this interesting.

Anyway, here are some results. See below for machine & benchmark details. If anyone can think of any other interesting benchmarks, please let me know! I had been thinking of POVRay, but that would probably just show the superiority of FPA.

Code: Select all

			A310	A3000	A3010	A3010*	A3010*	A3020*	A410/1	A310	R260	A5000*	A5000a	R260*	R260*	R260*	A7000	RPC600	RPC600	RPC600	RPC700	RPC700	RPC700	RPC700a	RPCSA	RPCSA	RPCSA*	RPCSA	RPCKi	A7000+	Mico	Omega	Iyonix	Pi3 B+
Year (approx)**		1987	1989	1992	1992	1992	1992	1990	1991	1990	1991	1993	1990	1990	1990	1995	1994	1994	1994	1995	1995	1995	1995	1996	1997	1996	1997	2000	1997	1999	2002	2003	2017
CPU			ARM2	ARM2	ARM250	ARM250	ARM250	ARM250	ARM3	ARM3	ARM3	ARM3	ARM3	ARM3	ARM3	ARM3	ARM7500 ARM610	ARM610	ARM610	ARM700	ARM700	ARM710	ARM710a	SA	SA	SA	SA	SA	7500FE	7500FE	SA	XScale
Clock speed		8MHz	8MHz	12MHz	16.7MHz	24MHz	24.86M	25MHz	36MHz	26+FPA	25/16	33MHz	36MHz	37.5	40	32Mhz	no VRAM 30MHz   33MHz	33MHz	33+FPA	40MHz	40MHz   202MHz	233MHz  287MHz	233MHz	233MHz	48+FPA	56+FPA	302	600
OS version		3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.11	3.6	3.7	3.7	3.7	3.7	3.7	3.7	3.7	3.7	3.7	3.7	4.03	4.03	4.02	4.03	4.03	5.28
Dhrystone/sec		5463	5972	9548	13712	19568	21138	18367	21883	22425	23954	26491	27566	28147	29145	38605	35423	36200	39858	41489		50103	50095	363654	423711	515805	422930	423599	61894	70732   551318	868056	2634352
kWhetstone/sec		45	53	57	64	229	98	228	287	2788	263	310	217	222	230	201	189	195	211	250	4207	295	309	2255	2623	3198	2851	2856	7257	7649	3709	6673	18643
Main memory read MB/s	14.20	15.71	25.16	36.12	51.97	56.25	13.52	14.21	21.99	27.76	23.32	23.21	23.33	23.12	24.70	26.12	28.77	29.56	33.02		34.31	28.02	39.83	40.48	40.81	40.47	104.7	43.98	47.01	108.0
Main memory write MB/s	15.71	15.71	25.16	36.12	51.97	56.25	16.79	16.56	26.31	35.67	25.98	26.24	26.76	26.62	47.09	48.64	54.15	54.01	54.22		53.17	53.03	37.46	37.37	37.36	36.25	47.20	91.70	93.57	125.3
MemBench fetch MB/s	19.62	19.82	31.50	45.63			15.05	14.92				21.03		23.77	27.41		34.57	34.53	39.08		39.21	30.26	40.14	40.13	40.14	40.11	103.45	47.63			149.07	2052
MemBench read MB/s	16.05	16.52	26.25	38.01			14.33	14.65				19.94		22.33	26.87		29.52	29.47	35.68		37.14	28.33	40.32	40.50	40.50	40.42	103.53	46.11			133.64	5447
MemBench write MB/s	16.35	16.52	26.25	38.03			17.49	17.33				25.22		27.57	39.97		45.64	45.48	45.90		45.94	46.13	38.96	38.96	38.93	38.89	62.17	70.24			101.97	6650
CLOCKSP			42.7	49	51.83	60.68		82.60	204.73	242.84	238.18	220.19	284.49	303.43		330.21	298.40	331.46	315.15	318.96	341.46		413.91	469.26	2882.36	3375.49	4114.80	3371.05	3374.15	504.72	568.08	4393	7851.51	21746
CLOCKSP, Rmfaster	57.58	61.46	95.86	148.03		201.54	215.28	283.27	259.11	239.59	313.61	324.23			311.48	335.06	336.58	333.20	346.83		419.05	473.18	2898.58	3815.27	4114.80	3369.85	3374.15	536.14	613.46		7843.78
CLOCKSP, BASIC64									279.64											386.64								575.89	643.00	
Mandelbrot		2626	2214	2025	1816			507.61	409.34	438.44	436.78	365.41	346.41		321.16	363.49	327.14	323.94	294.79	320.1		268.89	251.51	34.18	29.33	24.10	29.34	29.33	225.46	238.41	22.82	15.48	6.19
Mandelbrot, RMfaster	2007	1790	1123	779			498.60	400.28	426.20	420.37	354.18	346.64			352.78	319.26	316.21	287.46	299.31		272.31	228.19	34.19	29.34	24.10	29.34	29.33	220.98	198.41		15.5
Mandelbrot, BASIC64									334.30											213.95								145.72	138.58
Doom, low res		2.49	2.91	4.34			10.08	6.32	6.61	8.65		9.27	10.16	10.35		13.31	12.54	13.09	14.07	15.59		17.87	18.36	39.39	40.47	41.13	40.06	49.54	19.66
Doom			1.82	2.14	3.24			7.63	5.51	5.97	7.25		7.85	8.70			10.10	9.67	9.97	10.78	11.62		13.52	13.84	49.50	52.27	55.11	52.32	58.79	13.72	13.67
Quake																			1.1						10.4	11.4	12.6	11.5	14.9
Reach – Galaxy		6.85	9.53	14.88	21.40		32.73	23.43	28.93	29.05	31.59	33.67	35.86		37.81	39.85	37.68	38.93	41.84	43.08		50.43	50.46	123.16	124.93	127.86	219.09	239.56	64.78	72.55	372.6	65.66	764.9			
Reach – Tunnel		3.71	4.17	6.38	9.08		13.70	10.73	12.86	12.99	13.88	15.19	16.13		17.09	17.75	16.84	17.01	18.34	20.08		24.83	25.28	106.54	116.36	127.45	121.57	141.10	28.94	32.35	161.4	64.83	578.06
povray						325m18s					11m48s		110m16s						105m18s	98m57s	84m53s	7m25s	73m24s	68m9s	10m6s	8m49s   7m30s	8m3s	7m10s	5m50s	5m9s
povylib											9m26s											5m22s								3m30s	4m4s

* Overclocked
** For machines with ARM3 upgrades, this is the approximate year of the CPU upgrade rather than the machine.

Benchmarks :

Dhrystone/Whetstone/Main memory - Synthetics from !SICK V1.28. Run in MODE 12 on non-VRAM machines.
CLOCKSP - jgh's CLOCKSP benchmark under BBC BASIC, combined result reported. I ran with BASIC in both ROM and RMFaster'd in RAM. On the R260, A7000+ & Mico I also ran BASIC64 to test FPA. Run in MODE 12 on non-VRAM machines.
Mandlebrot - from https://github.com/markdryan/basic-benc ... /mandelBAS. Run in MODE 12 on non-VRAM machines.
Doom - Ultimate Doom under Doom+ running with "-timedemo demo1" at 320x240 with 8-bit colour.
Reach - Galaxy - Vast amount of multiplies in geometry part, render part performs read-modify-write on video memory
Reach - Tunnel - Full screen texture mapping, should be even between memory and ALU
povray - POVRay 2.2, command line "povray +Iscenes.level3.car +Oout/tga +W160 +H120". Run in MODE 12 on non-VRAM machines.

Machines :

A3000 - 8 MHz ARM2, 4 MB RAM, Watford A3000 IDE interface, RISC OS 2.00. * result taken from Chris's Acorns

A3010 - Doom and Reach results - 12 MHz ARM250, 4 MB RAM, IanS-made ZIDEFS interface, RISC OS 3.11
Other results - 12 MHz ARM250, 4 MB RAM, Simtec IDEFS interface, RISC OS 3.11 (IanJeffray)
Overclocked to 24 MHz, RMFaster FPEmulator (Phipli)

A3020 - ARM250 Overclocked to 24.86 MHz (trixster)

A410/1 - 25 MHz ARM3, 8 MB RAM @ 8 MHz, ICS ideA interface, RISC OS 3.11

A310/1 - 36 MHz ARM3, 4 MB RAM @ 8MHz, Acorn AKA31 SCSI interface, RISC OS 3.11

R260 - 26 MHz ARM3 + FPA10, 8 MB RAM @ 12 MHz, Acorn AKA31 SCSI interface, RISC OS 3.11

A5000a - 33 MHz ARM3, 8 MB RAM @ 12 MHZ, RISC OS 3.11

RiscPC - 30 MHz ARM610, 33 MHz ARM610, 33 MHz ARM700 + FPA11, 40 MHz ARM710, 40 MHz ARM710a, 202 MHz StrongARM rev K (at both 202 and 287 MHz), 233 MHz StrongARM rev S, 32 MB RAM, 2 MB VRAM, RISC OS 3.7
RiscPC - 233 MHz Kinetic StrongARM, 128 MB SDRAM, 32 MB motherboard RAM, 2 MB VRAM, RISC OS 4.03

A7000 - 32 MHz ARM7500, 34 MB RAM, RISC OS 3.7

A7000+ - 48 MHz ARM7500FE, 8 MB RAM, RISC OS 4.02

Mico - 56 MHz ARM7500FE, 32 MB RAM, RISC OS 4.03

[markdryan]

I am aware I'm probably the only person who finds this interesting.

I'm interested. Particularly, in the FPA performance. Perhaps you could try something like Mandelbrot in normal BASIC and then again in BASIC64 on the machine with the FPA? I'm guessing, we're not getting a true idea of the floating point performance boost provided by the FPA from the ClockSp benchmark as it includes the results for integer, string and function call benchmarks in the final score. Here's the Mandelbrot benchmark I've been using.

https://github.com/markdryan/basic-benc ... /mandelBAS

[SarahWalker]

I'll give that a try, thanks.

In the mean time I've added numbers from a couple of scenes ripped from Reach. In the process of removing VRAM, my current RiscPC got its first taste of blood. I hate that case...

[danielj]

It's quite interesting to see what a bump the 710 was from the 610.

Edit: Although that's probably just the cranked up clock looking at it - I forgot it was 40MHz vs 30.

[sirbod]

You should post the benchmarks.

I have an A310 with MEMC1, an A4000, an A7000+ and a Kinetic. It would be interesting to see what difference a MEMC1a has if someone has a stock A305/A310 with MEMC1a upgrade.

[SarahWalker]

!SICK and CLOCKSP are readily available online, Doom+ can be got from your local dealer...

I've attached the Reach benchmarks. Both will run then print an execution time in centiseconds. Galaxy framerate is 436 / (centisecond execution time / 100), tunnel is 896 / (centisecond execution time / 100).

[IanJeffray]

Faster-clocked ARM3 machine:
A420/1 - 36MHz ARM3, 8MB RAM @ 8MHz, Watford IDE podule, RISC OS 3.10
CLOCKSP: 280.94
CLOCKSP, Rmfaster : 283.41
Dhyrystone/sec : 19768
Whetstone/sec : 280

A3010 results not originally included:
A3010 - 12 MHz ARM250, 4 MB RAM, Simtec IDEFS interface, RISC OS 3.11
CLOCKSP: 51.83
CLOCKSP, Rmfaster: 95.86
Dhyrystone/sec : 8871
Whetstone/sec : 54

But .... how can my A3010 Whetstone be worse than an ARM2 A3000 ? This is !SICK 1.28 I'm using.

[IanJeffray]

But .... how can my A3010 Whetstone be worse than an ARM2 A3000 ? This is !SICK 1.28 I'm using.

ahh hmmm ... RMFaster FPEmulator and Whetstone now comes in at 104 on the A3010.

[SarahWalker]

My A3000 is running RISC OS 2 with an older FPemulator, so the FP results there probably aren't directly comparable with RISC OS 3 machines.

Cheers for the A420 results btw, interesting to see the relative impacts of CPU/memory speed. Just to check, this was in MODE 12?

[SarahWalker]

Updated first post. I also tracked down a RISC OS 3 ARM2 w/MEMC1a Whetstone result to replace the blatantly rigged old-FPemulator-running-from-RAM A3000 result. Given that FPemulator would have been running from ROM it's not entirely surprising that ARM2 and ARM250 give basically the same result on this test, confirming that the best route for optimising an A3010/3020/4000 is to *RMFaster everything in sight.

[IanJeffray]

Cheers for the A420 results btw, interesting to see the relative impacts of CPU/memory speed. Just to check, this was in MODE 12?

They weren't. Oops. That was MODE 27. But on re-reunning in MODE12, I get identical numbers anyway. Except -- I also re-ran them without running UniBoot and the CLOCKSP (non-rmfaster) value is now down to 263. Something in UniBoot must be messing about to affect that. Other timings unaffected.

I notice you've attached Reach - I'll try that sometime too.

[helpful]

Just for fun, a 2GHz Pi4 running BASIC V gives a ClockSp result of 72345MHz

Full results here - https://www.riscosopen.org/forum/forums ... 71?page=19

Bryan.

[trixster]

How is the Doom+ benchmark actually run, Sarah? I'd like to try it on my overclocked A3020. How do you input the command line options? Is it under Miscellaneous Options?

Ah, disregard, I've sussed it. Needed an edit to the Obey file

Arm250 clocked at 24.86Mhz (15.06 MIPS according to !Si)
1710 gametics in 7913 realtics, which i think is 7.563fps (that's with no border, but with the hud showing, so i think the same as Sarah)

--------------------------------------------

As a comparison, here are some figures for a few Amiga's running Ultimate Doom demo3 with two levels of border:
3863 gametics
Amiga 3000 68030 25Mhz - 5.278
Amiga CD32 with 68030 50Mhz - 11.86
Amiga 4000 with 68060 96Mhz - 36.37
A3020 ARM250 24.86 Mhz - 10.76

[SarahWalker]

Yep, that's the correct border size. What's the performance in low detail mode? (press F5)

You shouldn't need to modify any obey files, add '-timedemo demo1' in the 'Others' field under miscellaneous options.

For the curious, Doom timedemo performance is (gametics / realtics) x 35.

For the sake of further comparison, PC Ultimate Doom timedemo demo1 in high res mode and HUD showing but no border gets 10.1 fps on a 486SX/25 with no L2 cache and a reasonable ISA graphics card (Trident TVGA8900D), and 26.5 fps on a 486DX2/66 with 256kb L2 cache and a very fast VLB graphics card (Tseng ET4000/w32p).

[trixster]

low details is 6023 realtics in 1710 gametics = 9.93fps

[SarahWalker]

Added in the O/C A3020 results, fleshed out the Mico a bit, also added !SICK main memory results (though IIRC !SICK applies a fudge factor so these shouldn't be taken as gospel).

[SarahWalker]

!SICK's memory results were annoying me enough for me to do some digging. It's using 12-word LDM/STMs for memory benchmarking, which provoke some suboptimal behaviour in ARM710 and (to a lesser degree) StrongARM, which have 8-word cache lines. For all the other CPUs (which have 4-word cache lines) it should be fine.

[IanJeffray]

ARM710 and (to a lesser degree) StrongARM, which have 8-word cache lines

That's interesting. Does that mean that 710 and SA also need 8-word rather than quad-word alignment of blocks, too? (For the ultimate performance).

[SarahWalker]

Yes, if you're willing to burn the space in cache.


Edit: To be more specific :

If you're not going to use the data again anytime soon, then yes, align to 8 words and try to read 8 words at a time if possible (as this will prevent the CPU being halted during the cache line fetch).

If you _are_ likely to want the data again (and there's a decent change it will still _be_ in the cache by then...) then pack the data as tightly as possible to maximise cache utilisation.

And if you're _storing_, then ARM610/710/7500 will buffer up to 8 words, they will be written in a single burst (unlike on MEMC machines where they will be broken into 4 word bursts), and the alignment doesn't matter unless you cross a 1 kB boundary. SA can buffer up to 32 words, but a quick scan of the datasheet suggests that it probably won't do a burst longer than 4 words from the write buffer. Unless the destination address is in the cache of course...

[SarahWalker]

Ran some POVRay tests yesterday. Results are not at all surprising. If anyone is bored enough to reproduce, I used POVRay 2.2 as that's what I had to hand, with the command line "povray +Iscenes.level3.car +Oout/tga +W160 +H120".

Code: Select all

R260, ARM3 at 26 MHz + FPA10 - 11m48s (ylib version was 9m26s)
A5000a, ARM3 at 33 MHz       - 110m16s
RiscPC, ARM610 at 30 MHz     - 105m18s
RiscPC, ARM710 at 40 MHz     - 68m9s
RiscPC, StrongARM at 233 MHz - 8m49s
Mico, ARM7500FE at 56 MHz    - 5m9s (ylib version was 3m30s)
486DX/33                     - 4m57s

[SarahWalker]

Added numbers for RiscPC w/ARM700 at 33 MHz. Unsurprisingly it's roughly between ARM610 at 30 and ARM710 at 40. ARM610 at 33 would be an interesting comparison. Sadly my board doesn't have the FPA11 installed

[roobarb!]

I love benchmarking fun. I must dig out my A3010 and RiscPC soon, give them a check over and produce some numbers.

[SarahWalker]

Added numbers for RiscPC w/ARM610 at 33 MHz. Most stuff is roughly halfway between ARM610 at 30 and ARM710 at 33, but there are exceptions - CLOCKSP is barely any faster compared to the 30 MHz chip, whereas Mandelbrot clearly hates the ARM7x0 cache configuration as it's slower on ARM700 than ARM610!

That's almost all of the Acorn systems at around 30 MHz covered, with the A7000 the only exception; does anyone who has one feel like helping out? On paper it should be pretty similar to RiscPC 600 without VRAM, but the ARM7500 has a less efficient cache so it would be interesting to see how that affects performance.

[SarahWalker]

Added some A7000+ numbers. Unsurprisingly it's pretty similar to the Mico.

[marcusjambler]

does anyone who has one feel like helping out?

I've got a working A7000 in storage.
I can get it setup if no-one else has there's set-up.
I'd need walking through the run process though I dont have much riscOS experience.

[Phipli]

I just ran SICK 1.28 in MODE 12 on my A3010 at 24MHz and the FPU loaded into RAM.

I got...
Drystone/s 19567.9
kWhetstone/s 229.3825

Main memory r/w both 51.97 MB/s

Ran twice back to back, so reasonably stable. Quite an overclock on the 60ns RAM (running it at 41.67ns).

[SarahWalker]

ARM710 and (to a lesser degree) StrongARM, which have 8-word cache lines

Discovered an interesting quirk here. ARM710 has 8-word cache lines, but ARM710a reverted to 4-word! Acorn appear to have used both ARM710 and ARM710a chips interchangeably in the RiscPC, so different "ARM710" cards may have different performance. Mine is a 710a, which is why the memory performance is noticeably worse than the apparently mostly identical ARM700. I'll have to track down an original 710 to see if there are any noticeable differences outside of memory benchmarks.

ARM700 and StrongARM should always have 8-word cache lines.

[davidb]

Acorn appear to have used both ARM710 and ARM710a chips interchangeably in the RiscPC, so different "ARM710" cards may have different performance. Mine is a 710a, which is why the memory performance is noticeably worse than the apparently mostly identical ARM700.

It would be interesting to know why this was. I'd love to see an ABUG presentation/talk about Acorn 32-bit product development from one of the hardware engineers.

[SarahWalker]

Added some results for ARM700+FPA11. I should probably test the FPA11 in my A5000a at some point as well; maybe that's a weekend job.

[SarahWalker]

Added results for ARM710 (vs ARM710a). Memory read bandwidth is noticeably higher than ARM710a, but the higher latency on cache fills means every other test runs slower.