Yearly Archives: 2021

A short history of QDOS file headers

Background

In Germany there was a famous commercial that starts with the sentence “The history of menstruation is a history full of misunderstandings” and the same can be said about QDOS file headers. So what’s it all about?

On today’s PC file systems every file has some meta data associated with it. The main one is the name, obviously. But then there are also the file dates (creation, modification, last access) and a few flags like “file is hidden”, “file is a system file” and so on. Additionally files now have detailed access control lists that define which user can do what with it.

But apart from that little is known about the file itself, even the type information is only encoded more or less by convention in the filename by the “file extension” at the end of the names, like “.txt” or “.exe”.

When QDOS was created they wanted to be a bit more clever and they made room for additional information about the file, mainly the file type and some type dependent data. With the introduction of TK2 even more fields were added, like the file version and backup date.

In the end this meant for the QL that an executable can only be started with EXEC if it has the correct file type encoded in the header and the so called “data space” is also correctly set. “Data space” is basically the amount of RAM the executable needs for its variables and unfortunately it’s not simply included in the file itself but is part of the meta data associated with the file. And this is where things get hairy.

As we all know simpler operating and file systems have won the race and they do not have space for the additional data the QL needs, so a conglomerate of different solutions have been developed to cope with this. Let’s start with ZIP files.

ZIP files

There were many archive formats in the past of the QL, but only ZIP files have survived as they have become some sort of international industry standard. In the late 90s Jonathan Hudson ported the famous InfoZIP package to the QL which is still in use today. Fortunately the ZIP format has defined a so-called “extra field” to save OS specific data, which Jonathan made a good use of and saved the QL header within this field so all all meta data survives being zipped… but, and this is the catch, only as long as it is unzipped on a QL! PC based unzip utilities don’t know what to do with the extra information and in fact have no place to put them anyway on PC based file system, so all meta data is lost.

Emulator style headers

Emulator authors like myself were always thinking about ways to also somehow preserve the header information on PC file systems, but each method has certain advantages and drawbacks. QemuLator was the first to come up with an inline-header style format, meaning that the actual file contents is prepended by a small data block that starts with the the text "]!QDOS File Header", followed by the header data and finally the actual file contents:

I must note here in the strongest term possible that this is a feature exclusive to emulators. This type of file header is recognized only by QemuLator, QPC2 v5 and SMSQmulator so everything works as expected within them, but when zipping the file on a PC, the header information is not magically transferred to the “extra field” mentioned above. This means that when such a file is unzipped on a QL, not only will the header information be missing, the file will also start with garbage from QDOS’ point of view!

So how can one create QL compatible ZIP files in this case? Simply by not zipping the files on the PC but within the emulation. For the emulated system the data block is transparently removed, it doesn’t even know that it’s there, and all information is placed in the fields it expects them to be in, so the ZIP utility can get them as always and again place them in the “extra field” where they belong.

XTcc

Some users may also have heard of the “XTcc” trailer, what’s up with that? I think this was invented by Jonathan Hudson who was developing his software by cross-compiling it on a Unix machine, so he needed a way to preserve the “data space” value of the executables some way. For this he introduced special code into the Unix ZIP executable (not enabled by default, must be specially compiled with the “QLZIP” flag enabled) that scans for the XTcc trailer at the end of files and, if found, creates the standard QL compatible “extra field” information in the ZIP file. The XTC68 C compiler emits the XTcc trailer, so all he had to do was ZIP the files on his Linux box and get a completely QL compatible ZIP in the process.

In conclusion

I hope this clears up some of the confusion around QDOS file headers. As for QPC I fought with myself for many years about how to deal with them, I even had a fully working solution based on NTFS alternative data streams. This is a little known Windows feature where a file can not only have one contents but several at the same time, acting more like a directory in this case. I simply hid the QL header there, invisible to the user, a solution that has many advantages over and is much simpler to implement than the inline header explained above, but of course also has several drawbacks. In the end I opted not to introduce yet another incompatible solution to the world but instead play nice with the existing QemuLator solution so that at least these two systems can talk to each other seamlessly.

Anyway, if there are further questions simply ask them in the comments so that I can amend the article if needed.

QPC2 v5.01 and SMSQ/E v3.38 released

After a long delay, partly because it took quite long to put SMSQ/E v3.38 together, a new QPC2 has finally been released. It took so long that my expensive code signing certificate has expired again, so it was actually only used for a single release 🙁 But I bought another one, so this release is again digitally signed to help with security issues.

Despite the long wait QPC itself is mainly a maintenance release. One added feature is native support for QubIDE image files. These can be used like any .WIN style file, but is only meant for data transfer applications, don’t use it as your main drive. There are two kind of QubIDE image formats, one with big endian and and one with little endian byte order. QPC automatically recognizes the format and adjusts all read/write calls accordingly.

The existing QPC_HOSTOS function was extended, previously it could only differentiate between Windows 95 and Windows NT style operating systems. Now it will return these values:

0DOS (QPC1)
1Win9x/ME
2WinNT/2000
3Wine on Linux or unknown OS
4Wine on Darwin (MacOS)
5Windows XP
6Windows Vista
7Windows 7
8Windows 8
10Windows 10

Also there are two new Basic commands, QPC_FLASHBUTTON and QPC_HASFOCUS. HASFOCUS can be used to check if QPC currently has the input focus. If not FLASHBUTTON can be used to flash the taskbar button to get the attention of the user.

The AYSOUND driver to output FM music can now drive several chips with a single channel. While previously AYSOUND1 already opened a channel to the first chip and AYSOUND2 opened a channel to the second, AYSOUND3 now opens a channel to both chips. In this case the data blocks send to the channel must contain the register data of both chips one after the other. Also, the SOUND basic command has been renamed to AY_SOUND in order not to clash with the Simon N Goodwin’s SOUND device.

Finally, SMSQ/E v3.38 also contains Alain Haoui’s work of implementing index drawing in WMAN (index in the sense of the “A”..”Z” columns and “1”..”9″ rows in Excel), a function that has been part of the WMAN API documentation for 30 years but which was never actually implemented. Details can be found in Wolfgang’s new QPtr manual, along with a few examples.

Links

Get QPC2 here
Get SMSQ/E v3.38 for other platforms here
QPtr manual and WMAN examples

QL-SD ROM news

A while ago I’ve sent pre-production QL-SD ROM samples to a few testers and now Dilwyn Jones has written a review of his, read it here. Thanks Dilwyn!

Unfortunately I don’t currently have much time for QL stuff and what little time I recently had I’ve invested into making a NTSC compatible QL-VGA. But before this quiet phase I had another fun project in the making, which might be of interest: I created an SMSQ/E version that boots directly from QL-SD ROM without having to boot QDOS first. For this to happen I created a new boot loader that employs the software bank switching feature of QL-SD ROM to initialize SMSQ/E bit by bit. This way SMSQ/E occupies 3 to 4 of the 8 available OS slots.

One of the more technically challenging aspects was that I wanted the boot loader to be compatible with stock GoldCards, i.e. it contains a minimal amount of dead code and data that fools the GoldCard ROM into accepting it as a genuine QDOS ROM! Good thing I analyzed the GoldCard ROM some years ago…

GoldCard dummy code

But it’s not all good news, the resulting SMSQ/E only boots correctly 80% of the time, in all other cases it hangs initializing the IPC co-processor. And thus what started as a fun project devolved into a debugging nightmare and I’m still none the wiser. Only after booting QDOS once it starts to work again, so QDOS must do something during boot that SMSQ/E does not… but I haven’t found out what it is, so the project remains in this unsatisfactorily halve-finished state.

Apart from that the hardware seems to have proved itself. There will probably be one more minor tweak to the CPLD code to read the currently selected OS slot, but apart from that things appear to work splendidly. However, I currently don’t have the time to go into production right now. Also, manuals need to be written, maybe 3D printed enclosures evaluated because it IS much more sexy with an enclosure, etc.. I guess all this will happen later this year. As I noted in a previous post, QL-SD ROM is at least designed to be much easier to be build compared to the original QL-SD, so availability once production has started should probably not be an issue.

Speaking of production, there is a batch of 9 internal QL-SDs waiting to be sold. Once the QL-VGAs are updated I will send them to Rich for putting the online.

QL-VGA and the mysterious NTSC QL

After many happy customers Alex from the US was not impressed with the quality of QL-VGA’s picture quality and looking at what he sees this was completely understandable:

QL-VGA on Alex’s QL

It was soon clear that he owns one of the relatively rare original US NTSC QLs and QL-VGA was never tested with one of those. My working theory was that it should work well with it as long as one does not use the JSU QDOS variant that switches the QL into NTSC mode. Unfortunately the assumption turned out wrong, because what I have never seen documented anywhere is that US QLs use a 15.10489 MHz main crystal instead of the usual 15MHz, thus increasing the pixel clock from 10MHz to 10.0699MHz! There is also a daughter board connected to the crystal that is missing in European QLs, but this is just an oscillator circuit presumably to improve the slew rate of the clock signal.

US QL crystal with daughter board

Now QL-VGA was designed to cope with variations in clock frequency as not all QLs are built equally anyway, but 10.069MHz was way out of the band I designed for and thus it fell back to the default 10MHz frequency, resulting in the aliasing of the QL pixel clock and QL-VGAs pixel sampling that can be seen in the picture above.

Fixing this was fairly easy, but then I got ambitious and also wanted to implement full support for the NTSC modes of the QL. So here are a few facts I found out:

US QL facts

The basic “NTSC mode” is also available on European QLs that use the later CLA2345 variant of the ZX8301 chip, the CLA2310 is missing this feature. It can be enabled by setting bit 6 of the MC_STAT register at $18063. This is independent from the mode 4/8 setting, so a JSU QL actually has 4(!) different and distinct display modes.

JSU QLs boot in NTSC mode, which means that the vertical display resolution is reduced to 192 lines. The F1/F2 window is moved up to be in range of the lower resolution:

When the QL is booted using F2 the QL stays in this NTSC mode and it will never leave it, no matter the colour depth. So there you have a 512×192 4-colour mode and a 256×192 8-colour mode. Both with a refresh rate (and thus poll frequency) of 60.05Hz.

Selecting monitor mode using F1 switches the QL to the normal 256 lines modes, albeit with a slightly higher refresh rate of presumably 50.43Hz due to the faster pixel clock. Issuing a “mode 8” command, which is commonly associated with “TV mode”, will however stay in monitor mode and thus result in the normal 256×256 8 colour resolution.

Master clock15.10489 MHz
Pixel clock10.0699 MHz (2/3rd of master clock)
Line length640 pixels, same as PAL QLs
Line frequency10.0699 MHz / 640 = 15.73426 kHz
Front porchTV: 32 lines, monitor: 24 lines
VSYNC4 lines
Back porchTV: 34 lines, monitor: 28 lines
Total linesTV: 262 lines, monitor 312 lines
Refresh rateTV: 15.73426 kHz / 262 = 60.054 Hz, monitor: 15.73426 kHz / 312 = 50.43 Hz
NTSC QL specifications
Front porch/VSYNC/back porch in NTSC mode

Testing it

It’s actually very difficult to get a 15.10489 MHz crystal, the nearest I’ve found is 15.36 MHz which I ordered and then used to run some tests (it works but the QL will hang later in the boot process). Only later I remembered that I’m a master of FPGAs (not really) and those have PLLs that can be tuned to almost any frequency. So I actually created an FPGA project that outputs almost the desired frequency (15.104167 MHz) and tested it with that:

All worked well and thanks to the multi-ROM facility of QL-SD ROM I could easily boot the JSU ROM and verify that it work well, too.

Final verification with actual US hardware is pending as Alex is waiting for a suitable JTAG adapter to update his QL-VGA, but I’m now fairly confident I have removed any kinks left, now really making QL-VGA the best solution for the video needs of all QLs. Once that final verification has been done the remaining boards will be updated and put on sale, after which I might make another production run if demand warrants it.

QL-SD ROM (tiny!)

Hardware revisions

I get asked a lot about the state of my new QL-SD, so here’s an update. The new external QL-SD variant (henceforth called “QL-SD ROM”) went through a lot of revisions:

A sample of QL-SD ROM revisions
A sample of QL-SD ROM revisions

This series also exemplifies my progress as a PCB-designer. I’ve never formally learned to create PCBs, so it’s all learning by doing. With QL-VGA v2 I made my first 4-layer PCB and that experience in turn enabled me to create the final variant on the right, “QL-SD ROM tiny”:

(the case will probably not be part of the final product, but I will offer the files as always).

Features

One of the main features of QL-SD ROM, besides the two MicroSD slots of course, is its 512kB big flash memory. It can hold 8 different QL operating systems (called slot 0 to slot 7) and is bank-switched by software with slot 0 being the default on power up. Switching to another ROM is as easy as entering “ROM_SWITCH 5” into Basic. Also, there is one “Forces ROM 7” switch that can be enabled in case something is wrong with slot 0 so the QL doesn’t boot anymore.

All slots can be re-programmed using a new Basic extension, so updating the QL-SD driver or flashing a new operating system is as easy as typing “ROM_FLASH 2,ram1_minerva_rom” into Basic!

Hardware and software is pretty much finished, so I have sent two units to beta testers and I’m eager to hear what they have to say. When they are satisfied and don’t think major changes are needed a larger batch can be produced.

Bootstrapping

One of the problems with this new variant is how to get the initial operating systems on it in the first place. It cannot be programmed to the flash before it’s being soldered to the board. One way is to put normal ROM chips into the QL and enable the “Disable OS-ROM” switch on the board. This way the QL boots with the internal ROM and one can load everything needed for the flashing from floppy disc. Afterwards one has to switch it back, remove the ROMs from the QL and test the whole board… this takes a lot of time and I already know that I will loath the process very quickly. So you can see how dedicated I am to producing this by the simple fact that I even developed a special flashing station to bootstrap the process:

The Raspberry Pi’s GPIOs are connected to the QL-SD ROM and then basically emit the same pattern a QL would emit when it flashes the device. I actually had to program the software two times because the first version was written in Python and the Python GPIO libraries are incredibly slow (like “5-10 minutes per board” slow). So I rewrote it in C and that does the whole job within 40 seconds.

But I wanted big SDs!

Yeah sorry, no democracy here. I’m in love with the small form factor and the mechanical stability it provides. I did however buy a cheap MicroSD to SD adapter to test it and it works perfectly fine, so people can still enjoy large SDs with this device. Only one will fit of these as the slots are too near each other, but you can still use the second slot with a MicroSD card.

QL-SD ROM tiny with big SD adapter

What now?

When the tests are finished and everything is alright I’ll have a look at making a first run. I just have one request, please restrain from contacting me personally to be informed when it goes on sale. I’m happy that you are interested but I just get too many requests of this kind. I will publish updates in all relevant channels and this being designed to not be too cumbersome to produce I hope I can make enough for everyone.

A sign of the apocalypse? QPC2 v5 is here

As a developer I wouldn’t exactly say that I thrive on pressure but it certainly is an important element in getting stuff done. QPC2 v5 has been in the works for over three years and I wanted to release it for almost as long, but there has always been this one more bug to fix or this one more feature to add to constantly push it back. Besides, writing software is kind of fun, releasing is not. To release software means updating manuals, creating installers, changing web-pages, and after the release hearing all the things that are wrong with it. In the past this would have been a payed for upgrade, giving at least one more incentive, but now that QPC2 is free this went out of the window, too.

On the other hand it’s sad to spend all those many hours, literally hundreds of them, and not make it available for other to enjoy. So I set myself a release date for today, January 24th 2021, which is not coincidentally exactly four years after the last release. I released QPC2 v4.05 on the 3rd birthday of my daughter and today is her 7th, how time flies!

Never thought we’d celebrate during a pandemic, but nonetheless, happy birthday Marla!

By the way, QPC1 went on sale in 1996, so this is QPC’s 25th year!

So, what’s new? Well, the question is more, what stayed the same?

hg churn -s -f "%Y"
2013  58272
2014     73
2015    355
2016   1946
2017  21275
2018   5031
2019   8603
2020     57
2021    116

These are the number of lines changed, by year. 2013 was the initial transfer into the Mercurial versioning system and in 2017, the start of v5 development, we see quite a lot of activity with over 21000 lines changed. This was the time when I rewrote almost all of QPC’s remaining Assembler parts in C. Of course rewrite means that although it’s a lot of work, for the user everything pretty much stays the same. But having the code in an easier to handle language allows to make more complex changes later on.

Direct3D screen driver

The previous screen driver was based on DirectX 3, the one that was current when Windows 95 was still a thing. And, Microsoft not being Apple, it actually still sort of work 25 years later. But support for it became worse and worse over the years, mainly because graphic drivers have to do their part, too, and no manufacturer still cares for these old APIs. What sealed the deal was that my new laptop didn’t do pixel interpolation anymore for DirectDraw surfaces, making the result look really ugly and me very annoyed.

So after dragging my feet for a few years I finally made the plunge and rewrote the whole driver for Direct3D 11. Direct3D is an awfully complex and daunting beast and not really meant for the kind of usage I need for QPC, but it eventually worked fabulously and I couldn’t be happier with it.

DOS device (Windows file access)

Having the DOS device written in C opened up many venues for improvements without going insane by having to write it in x86 assembler. One of the most noticeable changes is that it now supports QemuLator style EXE headers, a choice that also took me many years to decide on. But with this feature the DOS device is now a first class SMSQ/E device that can actually be used in place of a WIN container.

Another feature is that file extensions by default are now translated to PC format, meaning the last “_” in a filename becomes a “.”. This way files can be used equally well from both within Windows and SMSQ/E. Not everybody might like this, so it can be disabled.

QSOUND

When I bought myself a ZX Spectrum 128 I was intrigued with the sound quality of the old AY-3-8912 chip and wanted to support this in QPC, too. The question was how to best provide an API that can abstract the hardware away enough that QPC’s virtual chip can equally well be accessed as one attached to a PAR port or a native solution.

For the QL there was the QSOUND board with the same chip, which wasn’t a huge commercial success and only experienced little circulation. But at least it was a native solution, so I took the existing API and tried to stay as compatible as possible. Not 100% compatible because QPC implements two AY chips for a total of 6 sound channels and two machine code functions needed an additional parameter to support this, but the rest is exactly the same. I also wrote a player application called AYPlay for some common sound formats. It’s included in the QPCDemo.win file.

Unfortunately documentation is a bit lacking at this point. The original QSOUND Basic commands as documented in the QSOUND manual should work though (try typing “explode” into SBASIC…). There is also a new and original QSOUND device that can be used to stream data to the chips. AYPlay uses this device for example. And AYPlay can also be used to create the data stream necessary from all supported formats, with that music can also be played using a simple COPY or SPL command.

Digitally signed

This is the first QPC executable that is actually digitally signed. This might help some people with corporate PCs because due to security concerns it becomes more and more difficult to install unsigned binaries in some environments. Problem is that this costs actual money and must be renewed every year. We’ll see if further version will still have this.

Misc

The floppy driver has been rewritten and got a write cache to drastically improve write performance.

QPC_QLSCREMU now works both ways. Previously it could only transfer writes to the old 512×256 QL screen to the high resolution/colour screen. This was a problem for applications that mixed direct screen writes and OS writes. Now writes to the big screen are reflected back to the old QL screen, too, making the emulation perfect.

There are slight improvements to the BEEP implementation. This was triggered by fellow QL user Lee Privett, who unfortunately didn’t live to see this very delayed release. Still, this one is for you, Lee, rest in peace.

QPC2 now is not compatible to Windows 95 anymore! Say it ain’t so! XP is needed at least, but even that is so old that you shouldn’t really use it anymore, at least not on anything that is connected to the internet.

SMSQ/E v3.37

QPC comes with the new SMSQ/E 3.37, which was also released today. It’s important to know that the two need each other, SMSQ/E 3.37 is not compatible to earlier QPC versions anymore and vice versa.

Get it

If you made it this far you can finally download the new version here. If you enjoy it, consider making a donation or at least tell the world how great it is 😉