MDVRAW tape format
Posted: Sat Jan 11, 2025 8:12 pm
As mentioned in another thread, current file formats for microdrive images seem to be limited to 512-byte sectors and don't support GST/68K microdrives.
I wanted to develop a lower level representation of the tape that can be used by any QL OS and doesn't require knowing what format that OS uses.
Ideally, I would do more research and tests, but time is limited and I don't have all the hardware needed, so I decided to just release what I have. Please feel free to comment!
This is not meant to replace existing formats (it's harder to use), just to enable GST/68K to use microdrive tapes in software emulators and enable playing with non-standard formats in general. It should also be relatively easy to support by hardware emulators if desired as they would just have to play back the byte stream with no special logic related to block structure.
I have collected a few images from physical QDOS, GST/68K (collected by Tony) and OPD cartridges. I have not found a Spectrum cartridge to try, yet.
Physical dump is currently based on attaching a logic analyzer (there are inexpensive ones) to the microdrive port and recording a few loops of the tape. A microdrive cable or other connector is helpful to attach the probe.
A utility (written in C++) converts the signal to MDVRAW.
Not sure whether a QL could collect similar traces in software or not, one of the problems is that the ZX8302 sometimes seems to give up when it encounters errors. It also imposes its own timing, while a utility that interprets the logic trace can be more flexible and hopefully recover tapes that have been stretched or have misaligned tracks or other intermittent errors.
The tape representation is byte-based and for each byte there is an extra bit (in a separate area of the file) to tell whether that byte is actual signal or a 'gap' (= no magnetic transitions). This simplifies access to the tape and makes the images shorter (about 200 KB) compared to using individual bits or raw magnetic transitions ('flux'), but is still harder to use from software compared to block based formats.
The ZX8302 constrains the input to be a stream of bytes at a bit frequency of 100 Kb/sec. The tape has two tracks, at an offset of 4 bits and with bytes alternating between the two tracks, but this file format hides that and just provides a single byte stream.
QL software could still start writing in the middle of an existing byte and this seems to result in OPD tapes having a half-byte of space between the block header and sector header, but I think rounding to the next byte should be good enough.
Writing to microdrives is done entirely in software on the QL, but reading from them is a combination of hardware and software. The ZX8302 is tasked with recognizing the starting 'preamble' of each chunk of valid data on the tape. There is also a gap interrupt as an optimization (to know when a sector is about to start without having to continuously monitor the input stream) and a other hardware registers that are also used by the software.
Unfortunately, this way of reading and writing results in an ambiguous representation on the tape: The same sequence of bytes in a preamble (at least 6 bytes of zeros followed by FF FF) could be present inside the sector data and the only way to tell them apart is the timing of read operations in the OS.
Q-emuLator will probably take this approach and just let the QL OS deal with the file format, while providing precise tape and CPU timings.
Other tools or emulators that just want to access sectors from the image can use heuristics to find all preambles in the byte stream. Most preambles are not far from the last gap. Others need an assumption about what QL OS wrote to the tape. There is a field for the OS type in the header, but it might not always be accurate. It should also be possible to infer the OS from the sizes of data chunks.
I plan to write and release some C code that identifies sector locations inside the MDVRAW.
Here is the current proposed header definition:
The important fields are:
Using the other fields is optional.
The gap bitmap area has one bit for each byte in the data area. 0 means gap and 1 means valid signal.
All offsets are from the start of the file and I'm leaning toward using little-endian byte ordering to simplify all tools.
Thanks,
Daniele
I wanted to develop a lower level representation of the tape that can be used by any QL OS and doesn't require knowing what format that OS uses.
Ideally, I would do more research and tests, but time is limited and I don't have all the hardware needed, so I decided to just release what I have. Please feel free to comment!
This is not meant to replace existing formats (it's harder to use), just to enable GST/68K to use microdrive tapes in software emulators and enable playing with non-standard formats in general. It should also be relatively easy to support by hardware emulators if desired as they would just have to play back the byte stream with no special logic related to block structure.
I have collected a few images from physical QDOS, GST/68K (collected by Tony) and OPD cartridges. I have not found a Spectrum cartridge to try, yet.
Physical dump is currently based on attaching a logic analyzer (there are inexpensive ones) to the microdrive port and recording a few loops of the tape. A microdrive cable or other connector is helpful to attach the probe.
A utility (written in C++) converts the signal to MDVRAW.
Not sure whether a QL could collect similar traces in software or not, one of the problems is that the ZX8302 sometimes seems to give up when it encounters errors. It also imposes its own timing, while a utility that interprets the logic trace can be more flexible and hopefully recover tapes that have been stretched or have misaligned tracks or other intermittent errors.
The tape representation is byte-based and for each byte there is an extra bit (in a separate area of the file) to tell whether that byte is actual signal or a 'gap' (= no magnetic transitions). This simplifies access to the tape and makes the images shorter (about 200 KB) compared to using individual bits or raw magnetic transitions ('flux'), but is still harder to use from software compared to block based formats.
The ZX8302 constrains the input to be a stream of bytes at a bit frequency of 100 Kb/sec. The tape has two tracks, at an offset of 4 bits and with bytes alternating between the two tracks, but this file format hides that and just provides a single byte stream.
QL software could still start writing in the middle of an existing byte and this seems to result in OPD tapes having a half-byte of space between the block header and sector header, but I think rounding to the next byte should be good enough.
Writing to microdrives is done entirely in software on the QL, but reading from them is a combination of hardware and software. The ZX8302 is tasked with recognizing the starting 'preamble' of each chunk of valid data on the tape. There is also a gap interrupt as an optimization (to know when a sector is about to start without having to continuously monitor the input stream) and a other hardware registers that are also used by the software.
Unfortunately, this way of reading and writing results in an ambiguous representation on the tape: The same sequence of bytes in a preamble (at least 6 bytes of zeros followed by FF FF) could be present inside the sector data and the only way to tell them apart is the timing of read operations in the OS.
Q-emuLator will probably take this approach and just let the QL OS deal with the file format, while providing precise tape and CPU timings.
Other tools or emulators that just want to access sectors from the image can use heuristics to find all preambles in the byte stream. Most preambles are not far from the last gap. Others need an assumption about what QL OS wrote to the tape. There is a field for the OS type in the header, but it might not always be accurate. It should also be possible to infer the OS from the sizes of data chunks.
I plan to write and release some C code that identifies sector locations inside the MDVRAW.
Here is the current proposed header definition:
Code: Select all
struct Header
{
char id[8]; // "QLMDVRAW"
uint16_t headerSize;
uint16_t fileFormatMajorVersion;
uint16_t fileFormatMinorVersion;
uint16_t creatorId; // 1 = MdvDecode
uint16_t creatorMajorVersion;
uint16_t creatorMinorVersion;
uint16_t creatorRevision;
uint8_t ulaFamily; // 0 = Unknown, 1 = ZX8302, 2 = Interface 1
uint8_t recognizedFileSystem; // 0 = Unknown, 1 = QDOS, 2 = Spectrum, 3 = OPD, 4 = GST/OK
uint32_t frequency;
uint32_t dataOffset;
uint32_t dataLength;
uint32_t gapBitmapOffset;
uint32_t gapBitmapLength;
uint32_t junctionStartOffset; // Optional unreliable section of the tape
uint32_t junctionLength;
uint32_t extensionOffset; // Future format extension
uint32_t recommendedInitialTapePosition; // Optional initial position to quickly get to sector zero
uint32_t currentTapePosition; // Optional, if we want to restart the tape from the last used position
};
Code: Select all
char id[8];
uint16_t headerSize;
uint32_t dataOffset;
uint32_t dataLength;
uint32_t gapBitmapOffset;
uint32_t gapBitmapLength;The gap bitmap area has one bit for each byte in the data area. 0 means gap and 1 means valid signal.
All offsets are from the start of the file and I'm leaning toward using little-endian byte ordering to simplify all tools.
Thanks,
Daniele