We’ve had some machines stop working with NI-MAX error 42HG08DD. Nothing found online on the error helped. I dug deeper into the binaries and found out it was a simple problem with file access rights. If you experience this, have a look at the access rights of the files under C:\ProgramData\National Instruments\Shared Memory. The MXSEventSharedMemory.tmp file is used by NI-MAX to communicate between the backend and frontend. On normal systems the ACLs should look somewhat like this:
On broken machines it looked like this:
Giving back “Full control” to “Everyone” instantly fixed the problem.
Both my QL-VGA2 and QL-SD ROM hardware use Xilinx chips and to program these I have a Xilinx USB JTAG cable model DLC9LP. After months of inactivity I dusted it off to update a few remaining QL-VGA boards but found that the cable didn’t work anymore with my main laptop: Code 39!? Drat.
Googling that didn’t turn up much useful advice, but the adapter still worked on an older laptop, so it wasn’t the USB device per se. And the next generation USB cable was sufficiently expensive (>300€) that I wasn’t about to just give up, so in the rabbit hole I went.
TLDNR: A quick fix is to disable Windows’ “Device Guard” feature, for a more thorough fix read on. “Device Guard” can for example be disabled by setting the “Enabled” key in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity to 0.
The USB cable has a somewhat unusual boot process. The integrated CY7C68013A microcontroller comes with a default firmware that presents itself as an USB device with the sole purpose of loading the real firmware into RAM (shown as the “Xilinx Platform Cable USB Firmware Loader” device). This is the job of the xusb_xlp.sys driver from Xilinx. Once the firmware is uploaded the cable detaches and comes back as a new USB device called “Xilinx USB Cable”, backed by the more common windrv6.sys Jungo driver.
Reversing the driver
The problem is that the xusb_xlp.sys driver is not compatible with Windows’ “Device Guard” feature. So as a first step I tried to reverse the source code of the driver. I first loaded it into my trusty IDA Pro disassembler, but unfortunately I don’t have a license for the even more expensive decompiler and reversing everything from assembler is a not too enjoyable task. So I tried another thing and also loaded it into NSA’s Ghidra tool, which I find pretty cumbersome to use but is free and includes a decent decompiler.
So this makes things a little bit easier, but it was still a challenge that can be compared to putting together a 10000 pieces puzzle, especially as I’ve never dabbled in Windows driver development before. Here’s a typical example of the process (all pictures display the same piece of code, click the image for a bigger view):
So Ghira helped a lot, though in some cases the decompiled code is actually less readable than the simpler assembler source, like in this example:
Well, in the end I had a fully reversed and re-compilable source code but only then noticed that this is more or less useless: even if I fixed all problems, on the latest Windows 10 releases all kernel drivers must be signed by Microsoft. And not only that, while I do have a valid code signing certificate for Windows I’d need the much more expensive “Extended Validation” certificate (>400€ per year)… so, while this was an interesting challenge, it was also technically a dead end. At least I learned a lot from it. For the curious souls I have attached the code (untested!): xusb_xlp.zip
Fix through firmware mod
The advantage of the double-boot was that the firmware could easily be changed by releasing a new driver. But, seeing that the firmware hasn’t changed for over 8 years I think it’s save to say that this was it. So, after reading a few datasheets, I found another solution: the CY7C68013A can also load its firmware directly from an attached EEPROM, provided it’s big enough. In my case it wasn’t (24LC02 = 256 bytes), but that could easily be remedied by ordering some cheap 24LC64 (= 8kB) chips. After the order I realized that while the raw firmware binary is <8kB, the bootloader form as extracted from the driver has enough overhead to take this to 10kB. But after some recoding of the binary I was able to compress this back down under the 8kB limit, pheeew, otherwise I would have wasted 4€ and have to make another order for some 24LC128 😉
So, when the chips arrived I put one into my trusty TL866A programmer and programmed my firmware binary directly into it (there is probably also a way to program the EEPROM in-circuit through the microcontroller with the Cypress SDK, but this way was easier for me). I then exchanged the 24LC02 with the programmed 24LC64. The only problem is that bigger EEPROMs must have pin A0 set to high according to the CY7C68013A datasheet, so a little tweak was needed in my adapter for pin 1:
And voilà, plugged it into the PC and it went directly to the windrv6.sys driver, which works fine even with “Device Guard” enabled. Bingo, mission accomplished! It was much work and a long journey, but
The QL-SD driver ROM is so full that every byte counts and during my search for superfluous bytes for v1.09 I removed one instruction too much. This results in a wrong value returned from the iof.rhdr call which is often not used anyway but can lead to subtle bugs in applications. Today I’ve uploaded a bug fix release 1.10 which only addresses this one problem and all users of 1.09 should update. If you don’t have the means to update the ROM the resident driver could also be loaded during boot (in case of the GoldCards this can make sense anyway as the driver is executed faster from RAM than from ROM).
On today’s PC file systems every file has some meta data associated with it. The main one is the name, obviously. But then there are also the file dates (creation, modification, last access) and a few flags like “file is hidden”, “file is a system file” and so on. Additionally files now have detailed access control lists that define which user can do what with it.
But apart from that little is known about the file itself, even the type information is only encoded more or less by convention in the filename by the “file extension” at the end of the names, like “.txt” or “.exe”.
When QDOS was created they wanted to be a bit more clever and they made room for additional information about the file, mainly the file type and some type dependent data. With the introduction of TK2 even more fields were added, like the file version and backup date.
In the end this meant for the QL that an executable can only be started with EXEC if it has the correct file type encoded in the header and the so called “data space” is also correctly set. “Data space” is basically the amount of RAM the executable needs for its variables and unfortunately it’s not simply included in the file itself but is part of the meta data associated with the file. And this is where things get hairy.
As we all know simpler operating and file systems have won the race and they do not have space for the additional data the QL needs, so a conglomerate of different solutions have been developed to cope with this. Let’s start with ZIP files.
There were many archive formats in the past of the QL, but only ZIP files have survived as they have become some sort of international industry standard. In the late 90s Jonathan Hudson ported the famous InfoZIP package to the QL which is still in use today. Fortunately the ZIP format has defined a so-called “extra field” to save OS specific data, which Jonathan made a good use of and saved the QL header within this field so all all meta data survives being zipped… but, and this is the catch, only as long as it is unzipped on a QL! PC based unzip utilities don’t know what to do with the extra information and in fact have no place to put them anyway on PC based file system, so all meta data is lost.
Emulator style headers
Emulator authors like myself were always thinking about ways to also somehow preserve the header information on PC file systems, but each method has certain advantages and drawbacks. QemuLator was the first to come up with an inline-header style format, meaning that the actual file contents is prepended by a small data block that starts with the the text "]!QDOS File Header", followed by the header data and finally the actual file contents:
I must note here in the strongest term possible that this is a feature exclusive to emulators. This type of file header is recognized only by QemuLator, QPC2 v5 and SMSQmulator so everything works as expected within them, but when zipping the file on a PC, the header information is not magically transferred to the “extra field” mentioned above. This means that when such a file is unzipped on a QL, not only will the header information be missing, the file will also start with garbage from QDOS’ point of view!
So how can one create QL compatible ZIP files in this case? Simply by not zipping the files on the PC but within the emulation. For the emulated system the data block is transparently removed, it doesn’t even know that it’s there, and all information is placed in the fields it expects them to be in, so the ZIP utility can get them as always and again place them in the “extra field” where they belong.
Some users may also have heard of the “XTcc” trailer, what’s up with that? I think this was invented by Jonathan Hudson who was developing his software by cross-compiling it on a Unix machine, so he needed a way to preserve the “data space” value of the executables some way. For this he introduced special code into the Unix ZIP executable (not enabled by default, must be specially compiled with the “QLZIP” flag enabled) that scans for the XTcc trailer at the end of files and, if found, creates the standard QL compatible “extra field” information in the ZIP file. The XTC68 C compiler emits the XTcc trailer, so all he had to do was ZIP the files on his Linux box and get a completely QL compatible ZIP in the process.
I hope this clears up some of the confusion around QDOS file headers. As for QPC I fought with myself for many years about how to deal with them, I even had a fully working solution based on NTFS alternative data streams. This is a little known Windows feature where a file can not only have one contents but several at the same time, acting more like a directory in this case. I simply hid the QL header there, invisible to the user, a solution that has many advantages over and is much simpler to implement than the inline header explained above, but of course also has several drawbacks. In the end I opted not to introduce yet another incompatible solution to the world but instead play nice with the existing QemuLator solution so that at least these two systems can talk to each other seamlessly.
Anyway, if there are further questions simply ask them in the comments so that I can amend the article if needed.
After a long delay, partly because it took quite long to put SMSQ/E v3.38 together, a new QPC2 has finally been released. It took so long that my expensive code signing certificate has expired again, so it was actually only used for a single release 🙁 But I bought another one, so this release is again digitally signed to help with security issues.
Despite the long wait QPC itself is mainly a maintenance release. One added feature is native support for QubIDE image files. These can be used like any .WIN style file, but is only meant for data transfer applications, don’t use it as your main drive. There are two kind of QubIDE image formats, one with big endian and and one with little endian byte order. QPC automatically recognizes the format and adjusts all read/write calls accordingly.
The existing QPC_HOSTOS function was extended, previously it could only differentiate between Windows 95 and Windows NT style operating systems. Now it will return these values:
Wine on Linux or unknown OS
Wine on Darwin (MacOS)
Also there are two new Basic commands, QPC_FLASHBUTTON and QPC_HASFOCUS. HASFOCUS can be used to check if QPC currently has the input focus. If not FLASHBUTTON can be used to flash the taskbar button to get the attention of the user.
The AYSOUND driver to output FM music can now drive several chips with a single channel. While previously AYSOUND1 already opened a channel to the first chip and AYSOUND2 opened a channel to the second, AYSOUND3 now opens a channel to both chips. In this case the data blocks send to the channel must contain the register data of both chips one after the other. Also, the SOUND basic command has been renamed to AY_SOUND in order not to clash with the Simon N Goodwin’s SOUND device.
Finally, SMSQ/E v3.38 also contains Alain Haoui’s work of implementing index drawing in WMAN (index in the sense of the “A”..”Z” columns and “1”..”9″ rows in Excel), a function that has been part of the WMAN API documentation for 30 years but which was never actually implemented. Details can be found in Wolfgang’s new QPtr manual, along with a few examples.
A while ago I’ve sent pre-production QL-SD ROM samples to a few testers and now Dilwyn Jones has written a review of his, read it here. Thanks Dilwyn!
Unfortunately I don’t currently have much time for QL stuff and what little time I recently had I’ve invested into making a NTSC compatible QL-VGA. But before this quiet phase I had another fun project in the making, which might be of interest: I created an SMSQ/E version that boots directly from QL-SD ROM without having to boot QDOS first. For this to happen I created a new boot loader that employs the software bank switching feature of QL-SD ROM to initialize SMSQ/E bit by bit. This way SMSQ/E occupies 3 to 4 of the 8 available OS slots.
One of the more technically challenging aspects was that I wanted the boot loader to be compatible with stock GoldCards, i.e. it contains a minimal amount of dead code and data that fools the GoldCard ROM into accepting it as a genuine QDOS ROM! Good thing I analyzed the GoldCard ROM some years ago…
But it’s not all good news, the resulting SMSQ/E only boots correctly 80% of the time, in all other cases it hangs initializing the IPC co-processor. And thus what started as a fun project devolved into a debugging nightmare and I’m still none the wiser. Only after booting QDOS once it starts to work again, so QDOS must do something during boot that SMSQ/E does not… but I haven’t found out what it is, so the project remains in this unsatisfactorily halve-finished state.
Apart from that the hardware seems to have proved itself. There will probably be one more minor tweak to the CPLD code to read the currently selected OS slot, but apart from that things appear to work splendidly. However, I currently don’t have the time to go into production right now. Also, manuals need to be written, maybe 3D printed enclosures evaluated because it IS much more sexy with an enclosure, etc.. I guess all this will happen later this year. As I noted in a previous post, QL-SD ROM is at least designed to be much easier to be build compared to the original QL-SD, so availability once production has started should probably not be an issue.
Speaking of production, there is a batch of 9 internal QL-SDs waiting to be sold. Once the QL-VGAs are updated I will send them to Rich for putting the online.
After many happy customers Alex from the US was not impressed with the quality of QL-VGA’s picture quality and looking at what he sees this was completely understandable:
It was soon clear that he owns one of the relatively rare original US NTSC QLs and QL-VGA was never tested with one of those. My working theory was that it should work well with it as long as one does not use the JSU QDOS variant that switches the QL into NTSC mode. Unfortunately the assumption turned out wrong, because what I have never seen documented anywhere is that US QLs use a 15.10489 MHz main crystal instead of the usual 15MHz, thus increasing the pixel clock from 10MHz to 10.0699MHz! There is also a daughter board connected to the crystal that is missing in European QLs, but this is just an oscillator circuit presumably to improve the slew rate of the clock signal.
Now QL-VGA was designed to cope with variations in clock frequency as not all QLs are built equally anyway, but 10.069MHz was way out of the band I designed for and thus it fell back to the default 10MHz frequency, resulting in the aliasing of the QL pixel clock and QL-VGAs pixel sampling that can be seen in the picture above.
Fixing this was fairly easy, but then I got ambitious and also wanted to implement full support for the NTSC modes of the QL. So here are a few facts I found out:
US QL facts
The basic “NTSC mode” is also available on European QLs that use the later CLA2345 variant of the ZX8301 chip, the CLA2310 is missing this feature. It can be enabled by setting bit 6 of the MC_STAT register at $18063. This is independent from the mode 4/8 setting, so a JSU QL actually has 4(!) different and distinct display modes.
JSU QLs boot in NTSC mode, which means that the vertical display resolution is reduced to 192 lines. The F1/F2 window is moved up to be in range of the lower resolution:
When the QL is booted using F2 the QL stays in this NTSC mode and it will never leave it, no matter the colour depth. So there you have a 512×192 4-colour mode and a 256×192 8-colour mode. Both with a refresh rate (and thus poll frequency) of 60.05Hz.
Selecting monitor mode using F1 switches the QL to the normal 256 lines modes, albeit with a slightly higher refresh rate of presumably 50.43Hz due to the faster pixel clock. Issuing a “mode 8” command, which is commonly associated with “TV mode”, will however stay in monitor mode and thus result in the normal 256×256 8 colour resolution.
It’s actually very difficult to get a 15.10489 MHz crystal, the nearest I’ve found is 15.36 MHz which I ordered and then used to run some tests (it works but the QL will hang later in the boot process). Only later I remembered that I’m a master of FPGAs (not really) and those have PLLs that can be tuned to almost any frequency. So I actually created an FPGA project that outputs almost the desired frequency (15.104167 MHz) and tested it with that:
All worked well and thanks to the multi-ROM facility of QL-SD ROM I could easily boot the JSU ROM and verify that it work well, too.
Final verification with actual US hardware is pending as Alex is waiting for a suitable JTAG adapter to update his QL-VGA, but I’m now fairly confident I have removed any kinks left, now really making QL-VGA the best solution for the video needs of all QLs. Once that final verification has been done the remaining boards will be updated and put on sale, after which I might make another production run if demand warrants it.
I get asked a lot about the state of my new QL-SD, so here’s an update. The new external QL-SD variant (henceforth called “QL-SD ROM”) went through a lot of revisions:
This series also exemplifies my progress as a PCB-designer. I’ve never formally learned to create PCBs, so it’s all learning by doing. With QL-VGA v2 I made my first 4-layer PCB and that experience in turn enabled me to create the final variant on the right, “QL-SD ROM tiny”:
(the case will probably not be part of the final product, but I will offer the files as always).
One of the main features of QL-SD ROM, besides the two MicroSD slots of course, is its 512kB big flash memory. It can hold 8 different QL operating systems (called slot 0 to slot 7) and is bank-switched by software with slot 0 being the default on power up. Switching to another ROM is as easy as entering “ROM_SWITCH 5” into Basic. Also, there is one “Forces ROM 7” switch that can be enabled in case something is wrong with slot 0 so the QL doesn’t boot anymore.
All slots can be re-programmed using a new Basic extension, so updating the QL-SD driver or flashing a new operating system is as easy as typing “ROM_FLASH 2,ram1_minerva_rom” into Basic!
Hardware and software is pretty much finished, so I have sent two units to beta testers and I’m eager to hear what they have to say. When they are satisfied and don’t think major changes are needed a larger batch can be produced.
One of the problems with this new variant is how to get the initial operating systems on it in the first place. It cannot be programmed to the flash before it’s being soldered to the board. One way is to put normal ROM chips into the QL and enable the “Disable OS-ROM” switch on the board. This way the QL boots with the internal ROM and one can load everything needed for the flashing from floppy disc. Afterwards one has to switch it back, remove the ROMs from the QL and test the whole board… this takes a lot of time and I already know that I will loath the process very quickly. So you can see how dedicated I am to producing this by the simple fact that I even developed a special flashing station to bootstrap the process:
The Raspberry Pi’s GPIOs are connected to the QL-SD ROM and then basically emit the same pattern a QL would emit when it flashes the device. I actually had to program the software two times because the first version was written in Python and the Python GPIO libraries are incredibly slow (like “5-10 minutes per board” slow). So I rewrote it in C and that does the whole job within 40 seconds.
But I wanted big SDs!
Yeah sorry, no democracy here. I’m in love with the small form factor and the mechanical stability it provides. I did however buy a cheap MicroSD to SD adapter to test it and it works perfectly fine, so people can still enjoy large SDs with this device. Only one will fit of these as the slots are too near each other, but you can still use the second slot with a MicroSD card.
When the tests are finished and everything is alright I’ll have a look at making a first run. I just have one request, please restrain from contacting me personally to be informed when it goes on sale. I’m happy that you are interested but I just get too many requests of this kind. I will publish updates in all relevant channels and this being designed to not be too cumbersome to produce I hope I can make enough for everyone.
As a developer I wouldn’t exactly say that I thrive on pressure but it certainly is an important element in getting stuff done. QPC2 v5 has been in the works for over three years and I wanted to release it for almost as long, but there has always been this one more bug to fix or this one more feature to add to constantly push it back. Besides, writing software is kind of fun, releasing is not. To release software means updating manuals, creating installers, changing web-pages, and after the release hearing all the things that are wrong with it. In the past this would have been a payed for upgrade, giving at least one more incentive, but now that QPC2 is free this went out of the window, too.
On the other hand it’s sad to spend all those many hours, literally hundreds of them, and not make it available for other to enjoy. So I set myself a release date for today, January 24th 2021, which is not coincidentally exactly four years after the last release. I released QPC2 v4.05 on the 3rd birthday of my daughter and today is her 7th, how time flies!
By the way, QPC1 went on sale in 1996, so this is QPC’s 25th year!
So, what’s new? Well, the question is more, what stayed the same?
These are the number of lines changed, by year. 2013 was the initial transfer into the Mercurial versioning system and in 2017, the start of v5 development, we see quite a lot of activity with over 21000 lines changed. This was the time when I rewrote almost all of QPC’s remaining Assembler parts in C. Of course rewrite means that although it’s a lot of work, for the user everything pretty much stays the same. But having the code in an easier to handle language allows to make more complex changes later on.
Direct3D screen driver
The previous screen driver was based on DirectX 3, the one that was current when Windows 95 was still a thing. And, Microsoft not being Apple, it actually still sort of work 25 years later. But support for it became worse and worse over the years, mainly because graphic drivers have to do their part, too, and no manufacturer still cares for these old APIs. What sealed the deal was that my new laptop didn’t do pixel interpolation anymore for DirectDraw surfaces, making the result look really ugly and me very annoyed.
So after dragging my feet for a few years I finally made the plunge and rewrote the whole driver for Direct3D 11. Direct3D is an awfully complex and daunting beast and not really meant for the kind of usage I need for QPC, but it eventually worked fabulously and I couldn’t be happier with it.
DOS device (Windows file access)
Having the DOS device written in C opened up many venues for improvements without going insane by having to write it in x86 assembler. One of the most noticeable changes is that it now supports QemuLator style EXE headers, a choice that also took me many years to decide on. But with this feature the DOS device is now a first class SMSQ/E device that can actually be used in place of a WIN container.
Another feature is that file extensions by default are now translated to PC format, meaning the last “_” in a filename becomes a “.”. This way files can be used equally well from both within Windows and SMSQ/E. Not everybody might like this, so it can be disabled.
When I bought myself a ZX Spectrum 128 I was intrigued with the sound quality of the old AY-3-8912 chip and wanted to support this in QPC, too. The question was how to best provide an API that can abstract the hardware away enough that QPC’s virtual chip can equally well be accessed as one attached to a PAR port or a native solution.
For the QL there was the QSOUND board with the same chip, which wasn’t a huge commercial success and only experienced little circulation. But at least it was a native solution, so I took the existing API and tried to stay as compatible as possible. Not 100% compatible because QPC implements two AY chips for a total of 6 sound channels and two machine code functions needed an additional parameter to support this, but the rest is exactly the same. I also wrote a player application called AYPlay for some common sound formats. It’s included in the QPCDemo.win file.
Unfortunately documentation is a bit lacking at this point. The original QSOUND Basic commands as documented in the QSOUND manual should work though (try typing “explode” into SBASIC…). There is also a new and original QSOUND device that can be used to stream data to the chips. AYPlay uses this device for example. And AYPlay can also be used to create the data stream necessary from all supported formats, with that music can also be played using a simple COPY or SPL command.
This is the first QPC executable that is actually digitally signed. This might help some people with corporate PCs because due to security concerns it becomes more and more difficult to install unsigned binaries in some environments. Problem is that this costs actual money and must be renewed every year. We’ll see if further version will still have this.
The floppy driver has been rewritten and got a write cache to drastically improve write performance.
QPC_QLSCREMU now works both ways. Previously it could only transfer writes to the old 512×256 QL screen to the high resolution/colour screen. This was a problem for applications that mixed direct screen writes and OS writes. Now writes to the big screen are reflected back to the old QL screen, too, making the emulation perfect.
There are slight improvements to the BEEP implementation. This was triggered by fellow QL user Lee Privett, who unfortunately didn’t live to see this very delayed release. Still, this one is for you, Lee, rest in peace.
QPC2 now is not compatible to Windows 95 anymore! Say it ain’t so! XP is needed at least, but even that is so old that you shouldn’t really use it anymore, at least not on anything that is connected to the internet.
QPC comes with the new SMSQ/E 3.37, which was also released today. It’s important to know that the two need each other, SMSQ/E 3.37 is not compatible to earlier QPC versions anymore and vice versa.
If you made it this far you can finally download the new version here. If you enjoy it, consider making a donation or at least tell the world how great it is 😉
Merry Christmas everyone! The first batch of QL-VGA v2 has been produced and sent out today! Here’s the picture the factory has sent me for validation:
The connectors are missing as I’ll just solder them in myself, this shouldn’t take too long. Then only the FPGA must be programmed before a final test can happen, all in all much more manageable compared to v1 production. Not too bad for a pure software guy (despite this board looking fantastic IMHO it’s good to remember that I’m noway near an electronics professional, but at least stuff usually works…).
This leaves the question of distribution. I’ll try to get a contingent shipped to Rich from RWAP to be sold on SellMyRetro. Of course Brexit makes stuff a little bit unpredictable, but he could at least serve the UK marked if not more. If anybody else is also interested in international/EU distribution or has a tip for me in this regard, drop me a line.