Monthly Archives: November 2021

A short history of QDOS file headers

Background

In Germany there was a famous commercial that starts with the sentence “The history of menstruation is a history full of misunderstandings” and the same can be said about QDOS file headers. So what’s it all about?

On today’s PC file systems every file has some meta data associated with it. The main one is the name, obviously. But then there are also the file dates (creation, modification, last access) and a few flags like “file is hidden”, “file is a system file” and so on. Additionally files now have detailed access control lists that define which user can do what with it.

But apart from that little is known about the file itself, even the type information is only encoded more or less by convention in the filename by the “file extension” at the end of the names, like “.txt” or “.exe”.

When QDOS was created they wanted to be a bit more clever and they made room for additional information about the file, mainly the file type and some type dependent data. With the introduction of TK2 even more fields were added, like the file version and backup date.

In the end this meant for the QL that an executable can only be started with EXEC if it has the correct file type encoded in the header and the so called “data space” is also correctly set. “Data space” is basically the amount of RAM the executable needs for its variables and unfortunately it’s not simply included in the file itself but is part of the meta data associated with the file. And this is where things get hairy.

As we all know simpler operating and file systems have won the race and they do not have space for the additional data the QL needs, so a conglomerate of different solutions have been developed to cope with this. Let’s start with ZIP files.

ZIP files

There were many archive formats in the past of the QL, but only ZIP files have survived as they have become some sort of international industry standard. In the late 90s Jonathan Hudson ported the famous InfoZIP package to the QL which is still in use today. Fortunately the ZIP format has defined a so-called “extra field” to save OS specific data, which Jonathan made a good use of and saved the QL header within this field so all all meta data survives being zipped… but, and this is the catch, only as long as it is unzipped on a QL! PC based unzip utilities don’t know what to do with the extra information and in fact have no place to put them anyway on PC based file system, so all meta data is lost.

Emulator style headers

Emulator authors like myself were always thinking about ways to also somehow preserve the header information on PC file systems, but each method has certain advantages and drawbacks. QemuLator was the first to come up with an inline-header style format, meaning that the actual file contents is prepended by a small data block that starts with the the text "]!QDOS File Header", followed by the header data and finally the actual file contents:

I must note here in the strongest term possible that this is a feature exclusive to emulators. This type of file header is recognized only by QemuLator, QPC2 v5 and SMSQmulator so everything works as expected within them, but when zipping the file on a PC, the header information is not magically transferred to the “extra field” mentioned above. This means that when such a file is unzipped on a QL, not only will the header information be missing, the file will also start with garbage from QDOS’ point of view!

So how can one create QL compatible ZIP files in this case? Simply by not zipping the files on the PC but within the emulation. For the emulated system the data block is transparently removed, it doesn’t even know that it’s there, and all information is placed in the fields it expects them to be in, so the ZIP utility can get them as always and again place them in the “extra field” where they belong.

XTcc

Some users may also have heard of the “XTcc” trailer, what’s up with that? I think this was invented by Jonathan Hudson who was developing his software by cross-compiling it on a Unix machine, so he needed a way to preserve the “data space” value of the executables some way. For this he introduced special code into the Unix ZIP executable (not enabled by default, must be specially compiled with the “QLZIP” flag enabled) that scans for the XTcc trailer at the end of files and, if found, creates the standard QL compatible “extra field” information in the ZIP file. The XTC68 C compiler emits the XTcc trailer, so all he had to do was ZIP the files on his Linux box and get a completely QL compatible ZIP in the process.

In conclusion

I hope this clears up some of the confusion around QDOS file headers. As for QPC I fought with myself for many years about how to deal with them, I even had a fully working solution based on NTFS alternative data streams. This is a little known Windows feature where a file can not only have one contents but several at the same time, acting more like a directory in this case. I simply hid the QL header there, invisible to the user, a solution that has many advantages over and is much simpler to implement than the inline header explained above, but of course also has several drawbacks. In the end I opted not to introduce yet another incompatible solution to the world but instead play nice with the existing QemuLator solution so that at least these two systems can talk to each other seamlessly.

Anyway, if there are further questions simply ask them in the comments so that I can amend the article if needed.

QPC2 v5.01 and SMSQ/E v3.38 released

After a long delay, partly because it took quite long to put SMSQ/E v3.38 together, a new QPC2 has finally been released. It took so long that my expensive code signing certificate has expired again, so it was actually only used for a single release 🙁 But I bought another one, so this release is again digitally signed to help with security issues.

Despite the long wait QPC itself is mainly a maintenance release. One added feature is native support for QubIDE image files. These can be used like any .WIN style file, but is only meant for data transfer applications, don’t use it as your main drive. There are two kind of QubIDE image formats, one with big endian and and one with little endian byte order. QPC automatically recognizes the format and adjusts all read/write calls accordingly.

The existing QPC_HOSTOS function was extended, previously it could only differentiate between Windows 95 and Windows NT style operating systems. Now it will return these values:

0DOS (QPC1)
1Win9x/ME
2WinNT/2000
3Wine on Linux or unknown OS
4Wine on Darwin (MacOS)
5Windows XP
6Windows Vista
7Windows 7
8Windows 8
10Windows 10

Also there are two new Basic commands, QPC_FLASHBUTTON and QPC_HASFOCUS. HASFOCUS can be used to check if QPC currently has the input focus. If not FLASHBUTTON can be used to flash the taskbar button to get the attention of the user.

The AYSOUND driver to output FM music can now drive several chips with a single channel. While previously AYSOUND1 already opened a channel to the first chip and AYSOUND2 opened a channel to the second, AYSOUND3 now opens a channel to both chips. In this case the data blocks send to the channel must contain the register data of both chips one after the other. Also, the SOUND basic command has been renamed to AY_SOUND in order not to clash with the Simon N Goodwin’s SOUND device.

Finally, SMSQ/E v3.38 also contains Alain Haoui’s work of implementing index drawing in WMAN (index in the sense of the “A”..”Z” columns and “1”..”9″ rows in Excel), a function that has been part of the WMAN API documentation for 30 years but which was never actually implemented. Details can be found in Wolfgang’s new QPtr manual, along with a few examples.

Links

Get QPC2 here
Get SMSQ/E v3.38 for other platforms here
QPtr manual and WMAN examples