Sending Teletext over Serial

[vela025]

Hi,

I'm trying to get OBBS to display more customized Mode 7 graphic menus in terminal mode (not prestel/viewdata mode), it looks fine on the host and I've had a report from a user that it's working on an A3010 but not so much on a BBC Micro/Master or under emulation. I have created the teletext frames using ABZ Teletext Editor by JGH and then have this procedure to display the frame:

Code: Select all

 
DEFPROCVA(n$):LOCALq,r:q=OPENINn$:IFq=0 THEN VDM%=FALSE
IF VDM% = FALSE ENDPROC
JJ%=q:REPEATr=BGET#q:VDUr:IFr=13VDU10
UNTILEOF#q:JJ%=0:CLOSE#q:PROCW:ENDPROC

On the host everything looks fine, but on the client things get corrupted. Here is a comparison of the outputs http://beebs.ddns.net/jsbeebs.html
And here is the original menu Vs how it's appearing on a BBC Master 128 client.

How it looks on the host when client is browsing

The Teletext Frame

How it's looking on the client

How it looks on A3020

What going wrong?

[BeebMaster]

Can you upload an example Teletext frame which is going wrong?

It looks like it is mainly affecting graphics characters, normal text and colour is coming out OK. Maybe there's something with the serial link which is stopping it correctly sending higher-numbered ASCII codes?

[vela025]

Can you upload an example Teletext frame which is going wrong?

It looks like it is mainly affecting graphics characters, normal text and colour is coming out OK. Maybe there's something with the serial link which is stopping it correctly sending higher-numbered ASCII codes?

I thought the same, but on the Simcity competition frame on JS one does manage to sneek through and they all show on the A3020. Here's the files.

teletext.zip

(1.38 KiB) Downloaded 9 times

[jgharston]

Code: Select all

 
DEFPROCVA(n$):LOCALq,r:q=OPENINn$:IFq=0 THEN VDM%=FALSE
IF VDM% = FALSE ENDPROC
JJ%=q:REPEATr=BGET#q:VDUr:IFr=13VDU10
UNTILEOF#q:JJ%=0:CLOSE#q:PROCW:ENDPROC

Why so complicated? A teletext frame is simply a run of characters:
FOR A%=0 TO 959:VDU mem%?A%:NEXT A%

Any other way of displaying a teletext frame is simply a variation on that, eg: ttxt_display(). For instance, *TTXTYPE does the equivalent of:
DEFPROCttx_type(fn$):in%=OPENIN(fn$):IF in%=0:ENDPROC
n%=0:REPEAT:VDU BGET#in% OR &80:n%=n%+1:UNTIL n%>919 OR EOF#in%
CLOSE#in%:ENDPROC

When saving a teletext screen, the saver should ensure that all the contents are &20-&9F, which all the teletext editors I know do, and certainly the ABZ editor and tools do. Are you "breaking" the frame data up when sending it over a serial stream? Again, it's just a data block, all you need to do is something like FOR A%=0 TO size-1:PROCser_send(mem%?A%):NEXT A%.

[jgharston]

Looking at those two files, they are just a plain 1024-byte teletext frame. All you need to do to display them is the code listed above.

[BeebMaster]

In those Teletext frame files, there are a lot of bytes > &9F, could that be the problem?

[jgharston]

In those Teletext frame files, there are a lot of bytes > &9F, could that be the problem?

Shouldn't be, you'd normally OR everything with &80 anyway*. I'm guessing the problem is the method used to get the data from the source system to the destination system. What we need to see is the code that sends the source data down the serial port, and the code that reads the data from the destination serial port.

*That's ignoring any issue with £#_ but it looks clear that it's nothing to do with that.

[vela025]

In those Teletext frame files, there are a lot of bytes > &9F, could that be the problem?

Shouldn't be, you'd normally OR everything with &80 anyway*. I'm guessing the problem is the method used to get the data from the source system to the destination system. What we need to see is the code that sends the source data down the serial port, and the code that reads the data from the destination serial port.

To send the data over serial it's just using Fx3,5, from there go4retros fork of tcpser sends it over TCP. The user of the screenshot above on the BBC M128 has done a TCP capture of the session if that might help diagnose the problem? At the other end it could be tcpser, wimodem232 or any other tcp solution. The BBC image above was using Commstar I (but it appeared the same in Commstar II and Termulator). However another user has sent me what they're seeing on a BBC M128 and it appears almost correct (top right is a bit wrong).

Corruption Top Right

[iainfm]

That's exactly how teletext used to look. It's working fine. Authentic.

One thought...how many data bits are you using in the serial config? You might need 8 (as in 8N1) for the characters with the upper bit set to transfer? ie If you're using 7E1 or something it might strip 'em.

Disclaimed: What little I knew about serial comms has long been forgotten.

[jgharston]

Is the other end using flow control? (though this would usually result in the tail end of the data being corrupted).

If you are sending exactly 1024 bytes, are exactly 1024 bytes arriving at the other end?

You've provided the source files, can you provide the destination files? Or an FDUMP of them?

You would normally use *FX3,3 to send OSWRCH to the serial port, not *FX3,5 - that sends to the serial port and the VDU driver. Was that a typo? (A more robust method is *FX3,23 which explicitly turns *everything* off *except* the serial port. See the wiki)

[vela025]

It looks like I may have been a bit premature as thanks to some pretty quick work by Lord Hickory http://fish.ccl4.org/js-beebs/ now supports the graphics OBBS is sending out. There still a problem with the bit of corruption in the upper right of the screen on the one M128 that connects and the really corrupted screens on the other (would connecting at a slower baud rate cause this, i.e. if they're connecting at 4800 baud but the system is sending at 9600...or is it a case of so long as the host is quicker or the same as the client it doesn't matter?)

Is the other end using flow control? (though this would usually result in the tail end of the data being corrupted).

If you are sending exactly 1024 bytes, are exactly 1024 bytes arriving at the other end?

You've provided the source files, can you provide the destination files? Or an FDUMP of them?

You would normally use *FX3,3 to send OSWRCH to the serial port, not *FX3,5 - that sends to the serial port and the VDU driver. Was that a typo? (A more robust method is *FX3,23 which explicitly turns *everything* off *except* the serial port. See the wiki)

I'm not sure is the honest answer I will need to ask them. I wouldn't know how to gather an FDUMP of the files at the other end, I can ask the user if they know how. They did provide me with a pcap file which might help, from packet 3 to 19 is the menu being received. OBBS by default was using fx3,5 so I left it as such, I would assume it is so you can monitor the system whilst it is in use and for doing chat with sysop, but I could change it so that it is fx3,23 unless the host presses esc (which enables chat with sysop) at which point it could revert to FX3,5.

I'll make the changes you've suggested to the procedure to send the teletext frames and see if that improves things.

That's exactly how teletext used to look. It's working fine. Authentic.

Yes it's being sent at 9600 baud with 8n1 parity.

[jgharston]

If you are getting corruption at high speeds and not low speeds, then that sounds like the other end *is* not using flow control, and is never saying "woah neddy!" when its buffers are full. But, that would normally be seen as the final bytes getting missed, not corruption early on. Eg: sender sends:
ABCDEFGHIJKLMNOPQRSTUVWXYZ
but receiver only has a 16-byte buffer, and never tells the sender to pause, the receiver would get
ABCDEFGHIJKLMNOP(dropped characters)
What you're seeing is something like:
ABCD(corrupted characters)KLMNOPQRSTUVWXYZ

I wouldn't know how to gather an FDUMP of the files at the other end

Whatever you use at the other end to give you something like:
0000 0D 00 0A 0F F4 20 3E 20 ..... >
0008 00 14 05 3A 0D 00 1E 07 ...:....
0010 20 63 74 72 6C 25 20 33 ctrl% 3
etc.
Or, just upload the files.

[vela025]

Whatever you use at the other end to give you something like:
0000 0D 00 0A 0F F4 20 3E 20 ..... >
0008 00 14 05 3A 0D 00 1E 07 ...:....
0010 20 63 74 72 6C 25 20 33 ctrl% 3
etc.
Or, just upload the files.

Right, it's not me at the other end it's someone else logging in, but they might be able to save the contents of the buffer in Commstar? I do have the tcp dump from their firewall which shows all the data they received while loading the menu:

Main Menu Pcap Dump.txt

(6.48 KiB) Downloaded 4 times

In a teletext related question I'd like to store the length (in lines) that each menu frame is on the top line of each frame. As it is quite slow sending the menu frames (I assume due to the method I'm using), and for small menus it is having to load a lot of blank space. I have played around with OPENIN and BGET to store this information in a variable, however as it gives me the CHR$ number, rather than the numeric value I'm struggling to concatenate these two values in to a variable. For example a 14 gives me 49 and 52, so I've subtracted 48 from each but can't suss out how to get these two values (1 and 4) in to a single variable.

[BeebMaster]

You can use EVAL or VAL to evaluate a string expression, so concatenating the two bytes read from the file representing the two digits making up the number in the file will give a numeric result.

e.g.

Code: Select all

B%=EVAL(CHR$BGET#X+CHR$BGET#X)

[vela025]

You can use EVAL or VAL to evaluate a string expression, so concatenating the two bytes read from the file representing the two digits making up the number in the file will give a numeric result.

e.g.

Code: Select all

B%=EVAL(CHR$BGET#X+CHR$BGET#X)

Screenshot 2024-02-03 22-51-07.png

Oh that's great thank you, just what I needed!

[vela025]

I'm using the method above, which works well however it is a little slow at sending out pages. Is this just how it is/was, or is there a way to speed up the loading of files?

I have some ideas but wanted to know before I embark upon anything if they would actually improve matters. Firstly I wondered if it was the storage method I'm using (1mhz scsi, but could swap back to DataCentre) and pondered (after reading viewtopic.php?t=4227) if it might be possible to store the menu files in sideways ram (if that would mean faster loading times). Failing that, if its possible, would swapping the section of code to 6502 assembly speed things up?

Loading a Mode 7 menu up takes around 5 seconds and a 60 column ANSI menu takes around 7 seconds (I had to reduce from 80 columns as that took too long to load). Here's an example of them loading: http://beebs.ddns.net/videos/ (firefox tells me the videos are corrupt but they open fine in VLC).

[WrightStuff]

Using BGET will slow things down a lot.
You could just *LOAD the file into memory and then loop through memory instead ?

[vela025]

Using BGET will slow things down a lot.
You could just *LOAD the file into memory and then loop through memory instead ?

Ahhh ok yes that sounds like it would work, I'm testing:

Code: Select all

*LOAD MAIN 6000
MEM%=6000
FOR A%=0 TO 959:VDU MEM%?A%:NEXT A

But I just get a pause for a second or two and then it returns to the blinking cursor

[BeebMaster]

It needs to be MEM%=&6000 as *LOAD uses hex addresses.

[BeebMaster]

How does the Teletext frame then get from the "server" screen to the remote machine?

Is it easier in fact to just load the frame file straight into screen memory (ie. *LOAD FRAME 7C00) rather than loading it elsewhere and "copying" it to screen memory with VDU?

[vela025]

How does the Teletext frame then get from the "server" screen to the remote machine?

Is it easier in fact to just load the frame file straight into screen memory (ie. *LOAD FRAME 7C00) rather than loading it elsewhere and "copying" it to screen memory with VDU?

It just uses *fx3,5 to send what ever is sent to the screen over serial. I'm going to test out both these ideas now and get back to you. Thanks they work in BeebEm as just a local host...apart from using *load frame 7c00 with the ANSI files, they instead look like this:

Rather than the sting of control codes:

[BeebMaster]

I'm not sure why it's coming out as a jumble when loaded directly into screen memory. Does the corrected VDU method work when loaded elsewhere?

I think if using FX3 to copy the output stream to serial, direct loading into screen memory isn't going to have the effect of sending the bytes to the serial port anyway.

[vela025]

I'm not sure why it's coming out as a jumble when loaded directly into screen memory. Does the corrected VDU method work when loaded elsewhere?

I think if using FX3 to copy the output stream to serial, direct loading into screen memory isn't going to have the effect of sending the bytes to the serial port anyway.

Yeah worked fine with the main teletext file, but not the ansi. But as you have said this method didn't work with sending over serial (which is a pitty as it was the fastest method). The loading into memory I think is working, although I'm struggling to find a block of memory that isn't being used by the main OBBS program.

Because of the way I have setup the ansi and teletext additions the name of the menu that needs to be loaded is stored inside two strings fld$ and title$ which concatenate to make n$. This was easy with BGET to use OPENINn$, can *LOAD be issued with a string? It doesn't like *LOAD n$ mem% or OSCLI"LOAD";n$;mem%

Edit: problem with using a string with *load sorted viewtopic.php?t=19495

[BeebMaster]

Yes, to concatenate a string with OSCLI you have to use "+" rather than ";" and also convert numbers to strings using STR$ (or STR$~ to give a hex number).

If there isn't 1K spare to load the frame then you could load it in chunks using OSGBPB. The beauty of this is that successive OSGBPB calls update the sequential pointer, so you don't have to remember where you are up to in the file to load the next chunk.

A sensible chunk size to use would be 256 bytes as this is the size of a disc sector. However, if the frames are always 960 bytes then the fourth chunk would be shorter, so you would need a way of knowing when the end has been reached (OSGBPB will tell you this, but it needs extra processing and programming).

Another way, if the files are either always 960 or always an exact multiple of the chunk size, would be to use that instead, e.g. use a buffer size of 240 bytes and then the 960-byte frame will load in 4 equal chunks.

[vela025]

Yes, to concatenate a string with OSCLI you have to use "+" rather than ";" and also convert numbers to strings using STR$ (or STR$~ to give a hex number).

If there isn't 1K spare to load the frame then you could load it in chunks using OSGBPB. The beauty of this is that successive OSGBPB calls update the sequential pointer, so you don't have to remember where you are up to in the file to load the next chunk.

A sensible chunk size to use would be 256 bytes as this is the size of a disc sector. However, if the frames are always 960 bytes then the fourth chunk would be shorter, so you would need a way of knowing when the end has been reached (OSGBPB will tell you this, but it needs extra processing and programming).

Another way, if the files are either always 960 or always an exact multiple of the chunk size, would be to use that instead, e.g. use a buffer size of 240 bytes and then the 960-byte frame will load in 4 equal chunks.

This is all great advice thank you. It's just occurred to me that I'm using a 1mhz rpi scsi which has JIM expansion ram emulation (although I don't fully understand what this is ) https://github.com/dp111/Pi1MHz?tab=rea ... -emulation

It says:

Two types of JIM expansion ram are supported:

Byte mode : 16Mbytes http://www.sprow.co.uk/bbc/ramdisc.htm
Page mode : 480Mbytes for PiZero, 992Mbytes Pi3B+

Byte mode uses the registers &FC02, &FC01, &FC00 to select the byte and FC03 to read and write the memory.

Page mode uses the registers &FCFD &FCFE FCFF to select the page for FD00--FDFF

So I'm thinking I might be able to utilise this space somehow. Or failing that I could revert back to using the Datacentre, which has space from &b3300 to &c0000 and from &F3300 to &FFF00. Would this be as simple as changing the code to:

Code: Select all

*LOAD MAIN &F3300
MEM%=&F3300
FOR A%=0 TO 959:VDU MEM%?A%:NEXT A

[jgharston]

Byte mode uses the registers &FC02, &FC01, &FC00 to select the byte and FC03 to read and write the memory.
Page mode uses the registers &FCFD &FCFE FCFF to select the page for FD00--FDFF

So I'm thinking I might be able to utilise this space somehow. Or failing that I could revert back to using the Datacentre, which has space from &b3300 to &c0000 and from &F3300 to &FFF00. Would this be as simple as changing the code to:

Code: Select all

*LOAD MAIN F3300
MEM%=&F3300
FOR A%=0 TO 959:VDU MEM%?A%:NEXT A

Filing systems don't understand JIM RAM, the address passed via *LOAD is a normal flat addresses. You would have to use something like *XLOAD/*XSAVE which understands JIM RAM and loads in chunks and then copies them over: link.

*XLOAD/*XSAVE uses page-wide RAM, so *XLOAD fred 0F3300 loads to &3300 in bank &0F (or &00 in bank &0F33 depending what nomenclature you're using).

I had started writing the equivalent commands to load/save to byte-wide RAM, but at the moment can't find where I've put them. Probably because I've forgotten what I called them.

Also, BASIC has no concept of memory outside its address space, all BASIC can see is what BASIC can see. When BASIC looks at &F3300 it will look at whatever address that wraps to with the address of the processor it is running on, so on a 6502 which is a 16-bit processor it will see &3300. You'd have to write some code to fetch the data from JIM memory to local memory.

For your purposes, it's probably simplest to have a 256-byte (or 240-byte) buffer and OSGBPB data from the file to the buffer. See the wiki for a primer and sample code.

[vela025]

If there isn't 1K spare to load the frame then you could load it in chunks using OSGBPB. The beauty of this is that successive OSGBPB calls update the sequential pointer, so you don't have to remember where you are up to in the file to load the next chunk.

For your purposes, it's probably simplest to have a 256-byte (or 240-byte) buffer and OSGBPB data from the file to the buffer. See the wiki for a primer and sample code.

Two votes for OSGBPB it is. Although I can't help but feeling I'm missing something, OBBS is from 1984, so one would assume a 32/64kb BBC Micro, and I'm running it on a Master 128Kb (with minimal additions to the code, and where I have made additions I've removed blank lines, PACE modem handling and REM lines elsewhere) and am either getting no room errors...or I'm *LOADing the frames in to memory that's already in use. Would I be correct in thinking that on a Master 128 it should be fine to use the any address space between &7000-&8000 and that this should really be empty?

[BeebMaster]

Even on M128, the main screen memory occupies &8000 downwards just as the BBC B, so for the 20K graphics modes, HIMEM is still set at &3000. In Mode 7, &7C00 upwards is the screen so can't be used for program or data. Also the BASIC stack starts using memory from HIMEM downwards, so overwriting that area will cause problems such as No room.

However the Master 128 has shadow screen memory which can be forced by adding 128 to the mode number, so 135 for Mode 7. This will then set HIMEM to &8000 in all modes.

[jgharston]

Adapting the wiki code, something like this is the bare bones:

Code: Select all

      max%=1024 DIV 4:REM Change this to the block size, eg 960 DIV 4 or whatever
      DIM ctrl% 31:X%=ctrl%:Y%=X%DIV256
      DIM data% max%-1
      :
      INPUT "Input file: "in$
      in%=OPENIN(in$):IF in%=0:PRINT "Not found":END
      FOR pass%=1 TO 4
        PROCgbpb(4,in%,data%,max%,0)
        FOR n%=0 TO max%-1:PROCserial_send(data%?n%):NEXT n%
      NEXT pass%
      CLOSE#in%
      END
      :
      DEFPROCgbpb(A%,chn%,addr%,num%,ptr%)
      ?X%=chn%:X%!1=addr%:X%!5=num%:X%!9=ptr%:CALL &FFD1:ENDPROC
      :
      DEFPROCserial_send(byte%)
      REM whatever to send to serial port
      ENDPROC

[vela025]

Even on M128, the main screen memory occupies &8000 downwards just as the BBC B, so for the 20K graphics modes, HIMEM is still set at &3000. In Mode 7, &7C00 upwards is the screen so can't be used for program or data. Also the BASIC stack starts using memory from HIMEM downwards, so overwriting that area will cause problems such as No room.

However the Master 128 has shadow screen memory which can be forced by adding 128 to the mode number, so 135 for Mode 7. This will then set HIMEM to &8000 in all modes.

Thanks for this, I played around with all sorts of different locations and loading the menus in to memory first was much quicker, unfortunately I just can't find a space to store the pages that doesn't mess up something else. The closest I came to a fully functioning system was using Mode 135 and then loading the pages in to &5200, doing this left enough room to open and close messages and downloads but the search feature would hang. The search feature is loaded as part of machine code before the main OBBS program starts, which is loaded in at &7880 so I wouldn't have thought anything in &5200 would have interfered with it but alas it did.

Adapting the wiki code, something like this is the bare bones:

Thanks for taking the time to do that, it worked well in Mode 135 (but no room errors in mode 7), all features of the system still worked including the aforementioned search facility, however there wasn't really any speed up in the delivery of pages. teletext went from an average load time of 4.7 to 4.3 seconds for the main page and the ANSI pages stayed at 9.7 seconds. To send the file I just used VDUbyte%

Unfortunately running in mode 135 makes the status window I usually have at the top of the screen (containing information about the active session) disappear.

Is there a way to see what memory is being used / is free whilst the program is running? I had a quick look in the debugging section in BeebEm but didn't notice anything. All I'm going on is the Free Space Basic Editor tells me, which prior to these changes was 9320 for the main OBBS program.

For reference all the menu file sizes are <1kb.