Peta Jäger sent me an update for his QPC2 for Mac bundle. It comes in at a hefty 544MB and includes the latest Black Phoenix distribution, QMovie and QBase. Get it from the QPC downloads page.
This will be a highly technical post, but as the last one got some positive feedback, why not 😉 Recently on the QL-users mailing list there was a bug discussed that if you load new Basic extensions using LRESPR within a basic PROCedure or FuNction SBasic would sometimes crash.
Wolfgang did some initial research and found the place where things eventually go wrong but wondered if it’s really worth the effort to try and fix it. And rationally speaking it’s absolutely not: the bug is relatively obscure and the SBasic interpreter code in SMSQ/E is so complex that it takes me huge efforts to barely understand parts of it. But being stubborn I went ahead regardless, starting with the findings Wolfgang already provided.
During the search I found references to bug fixes in the code I made 10 years ago about which I have no recollection whatsoever. The fixes are so deep in the innards of the beast that I apparently once understood what it does, but not anymore. So this time I want to record a few thoughts so maybe I don’t have to relearn everything from scratch next time.
The SBasic main loop starts at sbm_loop (main_asm). It provides the command line interface and it’s the place where I implemented the command line history. Entering a command using the keyboard or loading a program from disc is basically the same: the strings are read from the channel and fed into the parser stage (parse_asm). It does some basic error checking and translates the commands into numerical “parser tokens”. So 5 spaces for example become $8005 and “IF” becomes $8103. The tokens are defined in the file “parser_keys”.
When the program is RUN the action starts at sb_execute (execute_asm). First it calls the compiler at sb_compile (compile_asm) which translates the “parser tokens” into “compiler tokens”. Compiler tokens are similar but not entirely like the parser tokens: “END” and “FOR”, for example, are two tokens in the “parser tokens” space, but when preparing for execution combined into a single “END FOR” token in the compiler token space. These tokens are found in the comp_keys file.
In the next stage the compiler tokens are compiled into operations (compop_asm). This creates the actual stream of operations that is later executed by the interpreter. Pretty much all control structures like GO TO/GO SUB/PROCedure, FuNctions, SELect, FOR/REPeat loops and even IF THEN ELSE are compiled into (conditional) GOTO jumps. At this stage these jumps are still addressed by line- and statement number.
From the operation stream a statement table is built (cmpstt_asm). This table translates line/statement numbers into absolute addresses within the stream.
During the next stage (cmpadd_asm) and using the just built statement table, each line/statement number in the operation stream is replaced by the absolute address within the stream.
Next a table of the DATA statement locations is created, which concludes the compile phase.
After compilation the interpreter (inter_asm) is invoked. It sets up the data structures and uses a jump table to jump to the actual code blocks that executes the operation tokens in a tight loop (sb_iloop). The code snippets that correspond to the operations have the prefix bo_, so the “+” operation for example is executed by bo_add. This is all done by a fairly complex macro, so if you search the source for bo_add you will only find the code itself but no location from where it is called.
Complex structures like PROCedure calls are split into many different operations, like bo_spcall to setup the call, bo_dospr to actually do it and a lot of operations to set up the parameters (bo_formp amongst others). bo_return handles RETurn statements and also things like END DEFines.
SBasic crashed in the bo_return code because it a) wanted to clean up the PROCedure parameters and b) wanted to return to the location following the PROCedure call in the operation stream. So what happened? The LRESPRed extension calls sb.inipr (inipr_asm) to link in its new commands. This just results in an extension of the name table, which in itself is not a problem and which made me really wonder where things go wrong. It took me a long time to see that these three little lines cause all the ruckus:
tas sb_edt(a6) ; edited! to redo name types sf sb_cont(a6) ; do not continue move.w #sb.nact,sb_actn(a6) ; but no action
The LRESPR function that is loading and executing the extension is itself called in sb_icall, which is called from bo_docpr. This checks the sb_cont flag and returns to sb_istop if not set anymore. As the name suggests this stops the interpretation of the program and returns all the way up to the main SBasic loop. The loop checks which action is to be done, which is sb.nact (no action) and then it COMPILES THE PROGRAM AGAIN! This was a bit surprising at first but if you think about it it’s clear that this needs to be done because we’ve just loaded an extension with new SBasic procedures and functions and the programmer might like to start using those now and they weren’t known when we last compiled the program! But now we’re in the middle of a procedure, how can we just continue here? Enter the “return stack”.
As the name implies, the stack holds all data necessary to facilitate returns from PROCedures, FuNctions and GO SUBs. Mainly it holds the return addresses in the operation stream and more data pertaining to the parameters. These are absolute addresses… which is usually not a problem… unless we recompile the program during execution and it so happens that the whole instruction stream data block moves! 😮
What do we always do in SBasic if a block can move? Correct, we just make all pointers relative to the base. Actually understanding the code to a degree to find all those pointers is a wholly different matter, though. And in the end it was all in vain, this did fix most of the crashes but not all, because not only the block as a whole can move but also the operation boundaries within the block, so even relative addresses were not good enough. And as I’m not happy when something only works 50, 80 or even 99% of the time this meant going back to the drawing board.
The final code is curiously both more complex and less invasive at the same time: before the program is re-compiled after the LRESPR the return stack entries are now translated from absolute addresses into Line/Statement numbers (using the previously mentioned statement table). The Line numbers are invariant to the compilation process, so after the compilation they are again translated into absolute addresses which always give the correct result, no matter how the compilation changed the operation stream. So the new code should have zero performance and stability implications for ordinary executions and is only active when you LRESPR things. And neither Wolfgang nor I managed to crash it anymore, yay!
This, quite frankly, was a lot of work for such an obscure bug. Why did I do it? One reason is that I’ve been suffering under the bug for a decade without even realising it: when I was toying around with ProWesS I always loaded it conditionally using a PROCedure, which sometimes crashed and I could never explain why and just assumed ProWesS itself was the culprit… the other reason is that stress levels at my real job are currently very high and these puzzles are actually somewhat relaxing sometimes.
No matter the reason, SMSQ/E 3.31 is out and you can enjoy your crash free BOOTs and whatnot from now on. Have fun!
Update v2.32 (2018-03-04)
The basic problem is that TK2 already didn’t quite fit into 16KB during its last official versions and it became a lot worse once I packed in all the new features available in SMSQ/E. To make it fit again I had to remove something and I chose the network driver code, last having used it myself 20 years ago to prank my dad. Some people weren’t happy with that, which was to be expected, but there is one point which did stick: the network code cannot even be loaded afterwards as the timing needs to be precise and this is (almost) only possible when the code runs from ROM.
Some time later I found a way to free a few more bytes in the ROM and thought about what I could do with them and eventually came up with the idea of splitting the network code: I re-added just the timing critical hardware access routines to the ROM so that the network driver can be loaded later and still work. I think this is a fairly good compromise that would make most people happy.
Martyn Hill kindly beta tested the version for me and later inquired if one could have a full network stack in the ROM and remove some other stuff instead. I was hesitant because for me TK2 is foremost a SuperBasic toolkit and most of the remaining stuff is the basic commands. There is only one part that is even a bit bigger than the network code: the “ED” SuperBasic editor. It includes many improvement over the original TK2 code and thus grew considerably in size. Removing it also has the advantage that you wouldn’t put it into actual SuperBasic code, so the toolkit stays 100% compatible. For me personally “ED” is about a million times more useful than the network code but if you absolutely have to bootstrap a QL without any file systems over the network this is the way to go. Also I do provide a standalone binary for loading later.
Choose your poison
So here are all the different flavours:
TK2 v2.32 ROM version (with separate NET_bin file for full network support)
TK2 v2.32 network ROM version (with separate ED_bin file for the SuperBasic editor)
TK2 v2.32 RESPR version (this version includes everything though the network code will not work unless burned into a ROM or executed from zero waitstate RAM)
Update v2.31 (2017-03-27)
Fixed a bug in CDEC$
Original post for v2.30
As any QL owner will know, the Toolkit 2 from QJUMP/Tony Tebby was THE toolkit without which a QL was almost unusable, arguably its contents should have been in the ROM from the start. And I always thought it was quite a shame that such an important toolkit hasn’t been updated in two decades. This got me thinking that most TK2 source files at one time were incorporated into SMSQ/E and that with a little bit of work it should be possible to re-create something resembling TK2 from them again. Turns out I was right, but also completely wrong in how “little” that work is. But still, too many hours later, I can now present to you a brand new Toolkit II release I arbitrarily labeled version 2.30.
This is now based on the latest SMSQ/E source code, with new SMSQ/E features left intact if it was feasible (e.g. the “LOAD” command now automatically tries to append a “_bas” extension to the filename) plus many commands that have been added in the last 20 years (EXF, EX_M, FET, FEW, FEX, FEX_M, HGET, HPUT, JOBID, LGET, LPUT, UPUT, WGET and WPUT).
Unfortunately there are also parts of the source code that were completely missing, like the extended MDV driver. In this case I completely reverse engineered it from an existing ROM binary in a way that you can’t tell anymore that this isn’t the original source code.
The old network server code is actually still supplied with the SMSQ/E source code even though it’s not actually used there. The problem with that is that it’s just too big to fit into the ROM anymore and it probably doesn’t make much sense to run it from RAM because of the different timing. Therefore I didn’t include it. The ALARM and CLOCK commands had a similar fate.
The ALTKEY code, too, is included in SMSQ/E without seeing any usage. I didn’t want to include it at first, because with the Hotkey System II it’s very much obsolete. But when tinkering with my QL system the HK2 is often not loaded yet and it drives me crazy when ALT+ENTER doesn’t work to recall the last line. So it went back in. Problem then was that the result was about 200 or 300 bytes too big, so I removed the ALTKEY code but left the ALT+ENTER code in. The result was still 20 bytes too big, but with a few tweaks I now actually have 8 bytes left 🙂
So without further ado, here it is:
Recap: latest code, including extended MDV driver (not necessary and therefore disabled on Minerva), all new commands, no ALTKEY, but ALT+ENTER support.
This has been a long time in the making, actually I wanted to do this many years ago, but the day is finally here: QMake is here for everybody to enjoy. QMake is a “make” tool which basically takes a linker file as input, collects all files that make up the executable and assembles any where the source is newer than the relocatable file. You can also add dependencies like “reassemble win1_fu_asm if win1_keys_bar changes”. I’ve used this tool for well over 20 years to build SMSQ/E and all my other projects and consider it to be pretty much essential.
I’ve created a new page for it, check it out here.
Since changing from MGG to the Minerva ROM a few decades ago I’ve never used an unmodified Minerva ROM, I always used one where the German keyboard tables were patched into the binary somehow (by Jochen Hassler I think). This changed when I installed the QL-SD interface in my QL as it comes with a vanilla ROM where the German keyboard support had to be loaded every time, which I found fairly annoying. But now that the Minerva sources are publicly available changing this is fairly easy: take the Minerva source code plus the source of the German language pack, stir a bit and voila, a new ROM. The only thing missing is the printer translation table as that pushes the ROM beyond the 48KB limit.
As probably not everybody can do such a thing I provide the resulting binaries here, in case somebody finds it useful:
As for an EEPROM programmer I ordered one at my currently favourite Chinese store Ali-Express (being so cheap that I spent a truck-load of money there this year…). In this case I got myself the somewhat more expensive TL866A for just 60€, which also includes a lot of adapters:
But the smaller brother TL866 can be had for less than 35€ including shipping. This is an amazing piece of hardware for the price (usually it’s also available on eBay for slightly more) that finally lets me retire Jochen Hassler’s excellent but nowadays slightly outdated QL Eprommer II board:
Mice for the QL always have been kind of a problem. In the age that predated USB by two decades there was simply not the one mouse standard to rule them all. Every system did pretty much its own thing and getting a mouse that actually worked wasn’t that easy to begin with. In fact I have never even owned a mouse for the QL that worked right out of the box, every mouse had its electronics ripped out and replaced by something else to work with my trusty SuperQBoard clone. This posed a problem for me as I have recently resurrected my trusty QL but apparently the mice were thrown away at some point or other. Jochen Hassler kindly gifted me one of his old mice, but it was brown with age, the ergonomics is questionable and I really didn’t remember what a pain the ball-based mice were to use compared to their optical brethren.
This lead to the idea of implementing a converter, like taking a cheap Arduino Pro Mini to interface a modern USB mouse to the old SuperQBoard port. Fortunately before starting such a project I had a look at what’s already there and found the “New MKIV Amiga/Atari USB mouse adapter” for about 16€ plus postage. Atari mice are generally the type used for the QL, too, so I just bought it. The first tests a few days later weren’t that encouraging, nothing worked at all. So I had a look at the schematics of the SuperQBoard and found out that although electronically it’s Atari/QIMI compatible, it used a completely different pin layout!
Long story short, this is the pin mapping I came up with
|8||5||n/c or middle button|
After building a suitable adapter cable the optical mouse began to light up, indicating the power is there, and the buttons worked, too, indicating a working mouse protocol on the USB side (or rather PS/2 side really, the adapter only works with mice that can speak the PS/2 protocol over the USB connector) but the pointer only moved very erratically. So after some hair pulling (as if I still had any) I went back to the schematics and saw that apparently unlike QIMI my SuperQBoard clone only had pull-up resistors on the mouse button lines and not on the lines responsible for X/Y movement! So I decided to integrate them quick&dirty style into the adapter cable:
And what can I say, it worked so smoothly it was a pure joy to use.
So, lessons learned:
- The adapter might be used together with a QIMI interface right out of the box
- SuperQBoards used a pinout different from QIMI
- SuperQBoards might be lacking the necessary pull-up resistors
- Hardware tinkering is still fun
Every year I take the birthday of my daughter as an incentive to get stuff out of the door.
So this year I present to you:
This is mostly a bugfix release. It includes the latest SMSQ/E 3.28. One option was added that instead of a fixed resolution it can use the maximum resolution of the current monitor. Select “Max” as a resolution to do this.
Get it from the Downloads page.
… but there is more
The disclaimer for the following software: you may not sell the software and they come with no support whatsoever.
My old friend Jochen Hassler was over for dinner a few days ago and I asked him if I may release his software. He said yes, so here I present to you DISA, a pretty revolutionary disassembler at its time. V3.04 was the last official release done by Jochen in 1999. I adapted the source code for high colour in 2003, but this release 3.05 has never left my hard drive until today. So here they are:
As I got most of my software directly from the authors I usually lack the original manuals, as it is in this case. If somebody could provide that to me I can add it to the distribution.
Update: Thanks to Albin Hessler (who provided the Word document for v2) and David Westbury (who provided a scan of v3) I can now provide the latest manual here: DISA3E.pdf
I didn’t have the time to sort through the source and binaries I have of the ATR device, but I know for example that Dilwyn has a few ROMs or binaries and those can now be used freely, too.
And finally I had a chat with another old friend of mine, Jochen Merz, and he agreed to officially release QMenu for everybody to use. So here is my latest menu_rext file:
Update: apparently I’m not up to date in regards to QMenu, but 7.66 is the latest version I had the source code to. I will do a follow up when I know more about the later versions.
Was this a great birthday or what? 🙂
Have fun, Marcel
English: HomeMatic home automation articles will be in German as this system is mainly used there.
Den Anfang meiner (zumindest geplanten) HomeMatic Serie macht ein bereits älterer Aufschrieb über den Bau eines Gehäuses für den HM-OU-CM-PCB. Als Ausgangs-Basis dient ein bei Pollin erhältliche “Tragbarer Aktiv-Lautsprecher PULSE”. Anders als bei Pollin üblich ist das Ding auch ein Jahr nach meinem Bastel-Projekt noch erhältlich, mittlerweile sogar schon für wahnsinnige 3,95€!
Der eingebaute Verstärker liefert mit 3 Watt viel Bumms, läuft allerdings mit 5V während der HM-OU-CM-PCB mindestens 8V fordert. Der “richtige” Weg wäre es wohl 8V außen anzuschließen und dann intern auf 5V zu senken, wie es der HM ohnehin macht. Bedeutet aber größere Arbeiten an der Verstärker-Platine wenn man die vorhanden Buchsen des Verstärkers weiter benutzen will und Wegfall der USB Versorgung. Ich hab mich dafür entschieden den Step-Down Wandler auf dem HM zu umgehen und die Platine direkt mit 5V an Pin MP11 zu speisen. Auf eigene Gefahr, man verliert damit den Verpolschutz! Dies stellt dann auch erhöhte Anforderungen an das verwendete Netzteil, die 5V müssen sehr sauber reinkommen sonst gibt’s jede Menge Stör-Geräusche! Ein altes iPhone Netzteil bzw mittlerweile sogar ein billiger 5€ ALDI Zwischenstecker mit USB-Ladeausgängen verrichten hier bei mir aber sehr gute Dienste.
Den internen Verstärker des HM sollte man wie in der Anleitung beschrieben abklemmen und den Ausgang dann mit der Klinken-Buchse am Verstärker verlöten. Dann noch Masse und die 5V rüber und fertig ist die Kiste. Für den physischen Halt habe ich den Batterie-Halter etwas aufgedremelt so dass man die Platine hineinschieben kann. Hält ganz gut und man kommt nach dem Öffnen des Deckels bequem an die SD-Karte.
Den Rest erklären hoffentlich die Bilder.
In my last post I lied that regarding the QMovie player “I’m done here”. OK, to my defense, even I didn’t know that I’m not done yet. So today I present to you QMovie v2.0:
The details and downloads are on a dedicated QMovie page.
This has been superseded by QMovie v2
QMovie is a quick&dirty full colour movie player including sound for QPC. I really didn’t have the time to write this, but sometimes an idea latches itself into my head and I can’t let go until I have done something about it. So the target for this was “be done quickly” and not “be done perfectly”.
You can download it here (135MB). Just start the QPC2.exe and it should run. The movie frames are time synchronised to the audio. No double buffering (neither for the screen nor for the play buffer) nor any other fancy stuff is performed as the performance was already good enough.
Clip: The Iron Maidens recorded at the LKA Stuttgart by me. 10 fps, 20kHz mono
Creation of this video:
ffmpeg -i “IMG_7696.MOV” -r 10 -s 480×270 Frame_%04d.png
ffmpeg -i “IMG_7696.MOV” -f u8 -ar 20k -ac 1 sound.raw
Convert all Frame_%04d.png into _spr files using PNGConv. Settings: “binary”, “.” separator, “solid mask”, “no compression”.
The resulting Frame_xxx.spr and sound.raw files are combined using a quick Perl script called qmovie-concat.pl.
The QMV Format is an interleaved data format with one second of audio, followed by one second (10 frames) of video in Sprite format (fixed size, must not be RLE compressed). No meta data is saved (remember, “quick” was the motto here), all things like resolution and frame rate are hard coded into the qmovie_exe file.
Feel free to improve it, I’m done here 😉