BBC BASIC test suite?

[Deleted User 9295]

Somebody has asked over at the GitHub project whether a BBC BASIC test suite exists. I'm not aware of one, but it would obviously be very useful for detecting regressions. Does anybody know of such a thing ever having been written, for any version of BBC BASIC?

[jgharston]

Not formally a test suite, but a collection of programs that test a BASIC implementation. They depend on the user knowing what the correct results should be rather than reporting the flaws. I started gathering them together here, but most actually test the host's implementation of calls called by BASIC. Those that do test the functioning of the interpreter depend on my knowledge of the correction method of using the test code.

I have been meaning to tidy things up and make them into a formal test suite. I tend to run every BASIC program I have as a regression test. I was pleasantly surprised last year to find my Teletext Editor works quite fine.

[Deleted User 9295]

I tend to run every BASIC program I have as a regression test.

That would drive me mad (don't say it) with so many platforms to test on: Windows 32-bit, Windows 64-bit, Linux 32-bit, Linux 64-bit, Mac OS, Raspberry Pi, Android (potentially both 32 and 64-bits), iOS and in-browser. That could mean running every program ten times! So if you ever do create an automated suite I would be very interested.

[jgharston]

So if you ever do create an automated suite I would be very interested.

As an interim step, I've dumped all my test programs (after a bit of tidying) into
http://mdfs.net/Software/PDP11/BBCBasic/Tests/
and
http://mdfs.net/Software/BBCBasic/Testing/unsorted

They are raw directory listings, some of the files are tokenised BASIC programs, they can be listed in a browser by manually adding .bas to the filename.

[Deleted User 9295]

As an interim step, I've dumped all my test programs (after a bit of tidying) into
http://mdfs.net/Software/PDP11/BBCBasic/Tests/

To save me ploughing through them, do any of these report pass/fail or are they all of the kind you described which requires the user to know what the output should be?

[Deleted User 9295]

To make a start, I've added a tests directory at the BBCSDL repository. This contains a few non-trivial test programs that I already had lying around, which give pass/fail results. Obviously the overall coverage of BBC BASIC features is negligible, but I am happy to receive programs that might be suitable to be added, perhaps after some tweaking.

(Before anybody comments, I know that convertst.bbc can occasionally fail when testing floating-point-to-string conversion, because of rounding effects).

[jgharston]

As an interim step, I've dumped all my test programs (after a bit of tidying) into
http://mdfs.net/Software/PDP11/BBCBasic/Tests/

To save me ploughing through them, do any of these report pass/fail or are they all of the kind you described which requires the user to know what the output should be?

Most of them depend on the user knowing what the correct output should be, but some prompt for the correct output for the user to visually confirm, a couple have internal tests.
CircleTest tests floating point addition and multiplication by drawing a circle and tests the results against Windows BASIC gives, and reports ERROR where it fails. It relies on the | operator and the floating point format being the same.
The colours tests display the name of the colour the selected colours should be, eg "Red Green Blue..." etc should be those colours. Ut's up to the user to use their eyeballs to confirm the results.
The keyboard tests display what various keypresses should result in, again the Mk 1 Eyeball is needed to confirm the results.
The keyboard and VDU tests are documented as to what the correct results should be, but they're testing the I/O system.

Essentially, if they are listed in http://mdfs.net/Software/BBCBasic/Testing/ with a description and documentation.

I'm working on documenting the BASIC internals tests, and/or making them self-reporting.

[Deleted User 9295]

The colours tests display the name of the colour the selected colours should be, eg "Red Green Blue..." etc should be those colours. Ut's up to the user to use their eyeballs to confirm the results.
The keyboard tests display what various keypresses should result in, again the Mk 1 Eyeball is needed to confirm the results.

I appreciate that graphics and keyboard tests etc. cannot be fully automatic, but I don't currently intend to include that kind of test anyway because it would be testing SDL2 just as much as it is testing BBC BASIC. It's (unfortunately) the case that some versions of SDL2 - including the most recent 'stable' version - have faults such as lines that should be drawn including both end-points not being.

But tests of the internal working of the interpreter ought generally to be of the pass/fail kind. One does have to assume enough basic functionality to run the test program properly, although I remember in one of my early test programs ensuring that a 'passed' test sometimes resulted from an IF statement evaluating to FALSE and other times to TRUE. Otherwise there's a danger that if it always gave the same answer the tests might all seem to pass!

[1024MAK]

Would it be better for a test suite to produce a file, which can then be loaded, tested and checked on a system where the BASIC was known to be good. The file produced should be standardised.

This would then make false results less likely.

Mark

[Deleted User 9295]

Would it be better for a test suite to produce a file, which can then be loaded, tested and checked on a system where the BASIC was known to be good. The file produced should be standardised.

I think it depends on how mature the 'BASIC under test' is. If one was testing a newly created interpreter which might have many faults then, yes, that would be a good approach. But the principal purpose of these tests is to check for regressions in a product which is known to be substantially bug-free, so 'testing itself' is more convenient.