Q_Liberator malaise

[pjw]

I think there may well be self modifying code, but only during initialisation of the runtime, which is perhaps not ideal, but it only means that Qlib_run/Qlib_sys or programs that bake this code in, would not be ROMable. But Qliberated programs are definitely re-entrant, and thus not self modifying. I have for years had qlibbed programs using shared code as Things, running multiple instances simultaneously off one code base with different data. Had the running code been self modifying there would have been artifacts, error, crashes.

If the needle is somewhere in the house, it is pointless looking for it in the garden!

Regarding the disassembly of the runtime code, I did that. The reassembled result of that was compared with the original and found to be identical. I havent done any close analysis of it as, like you, I found it obtuse. Possibly, I thought at the time, this was deliberate to avoid penetration.

Marcel had some clever, expensive, software disassemble one of the other toolkits. I dont know why he chose that one, or whether he did any of the others, but that is better illustrated, thanks to his fancy disassembler.

[tofro]

I think there may well be self modifying code, but only during initialisation of the runtime, which is perhaps not ideal, but it only means that Qlib_run/Qlib_sys or programs that bake this code in, would not be ROMable. But Qliberated programs are definitely re-entrant, and thus not self modifying. I have for years had qlibbed programs using shared code as Things, running multiple instances simultaneously off one code base with different data. Had the running code been self modifying there would have been artifacts, error, crashes.

Per,

self-modifying code can very well be re-entrant (and ROMable), as long as the code being created is unique for every instance of job it's run on and is located in a piece of memory dedicated to this instance. (Note my definition of self-modifying includes code created on the fly)

I.e. this piece of self-modifying (pseudo-)code can be re-entrant and ROMable:

Code: Select all

10 a = ALCHP (2)
20 POKE_W a,$4e75 : REM Opcode for RET
30 CALL a

If it weren't BASIC, you could easily put this stuff in ROM and it'd be a "pure" (in TT sense) program that could be run as a Thing, in multiple instances, whatever. But still wouldn't run properly with caches enabled.

My suspicion is that qlib_run does something along the lines of the above. (I found one piece of code in there that searches for a NOP [$4e71] and wondered what that would be good for...)

[NormanDunbar]

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station. I found stuff like "jump to 3 vectors before a QDOS vector address" or "jump to 6 bytes behind that vector" and similar. Assuming that the disassembly has been properly done, it's a bit of a miracle that code works on multiple platforms at all.

Might be something Simon N Goodwin documented. I can't remember the exact vector but Simon documented and explained a few offsets from that vector which did similar but different things.

Just a thought.

Cheers,
Norm.

[Derek_Stewart]

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station. I found stuff like "jump to 3 vectors before a QDOS vector address" or "jump to 6 bytes behind that vector" and similar. Assuming that the disassembly has been properly done, it's a bit of a miracle that code works on multiple platforms at all.

Might be something Simon N Goodwin documented. I can't remember the exact vector but Simon documented and explained a few offsets from that vector which did similar but different things.

Just a thought.

Cheers,
Norm.

Simon Goodwin is active on the Spectrum Next and QL Core, so maybe interested.

[tofro]

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station. I found stuff like "jump to 3 vectors before a QDOS vector address" or "jump to 6 bytes behind that vector" and similar. Assuming that the disassembly has been properly done, it's a bit of a miracle that code works on multiple platforms at all.

Might be something Simon N Goodwin documented. I can't remember the exact vector but Simon documented and explained a few offsets from that vector which did similar but different things.

Just a thought.

Cheers,
Norm.

Well, the „three vectors before“ where the code ends up jumping seems to be a totally valid and documented one. So, this is not calling an undocumented vector, but just convoluted code.

„Six bytes after vector entry“ I consider extremely dangerous, however

[pjw]

<>
self-modifying code can very well be re-entrant (and ROMable), as long as the code being created is unique for every instance of job it's run on and is located in a piece of memory dedicated to this instance. (Note my definition of self-modifying includes code created on the fly)
<>

The way you describe it, I agree that Qlib could contain "self-modifying
code". But perhaps it should be called something different, like "ad hoc
program generated code", unless there is a ready made technical term for
it.

I can see how the sort of code you describe could mess with cache
operation.

Perhaps Qlib_run and Qlib5_bin would be the most likely candidates to start
with. If your example is typical one might start investigating code related
to jsr instructions that dont involve documented vector addresses..

But as I said before, I dont have the knowhow, the hardware, the time nor
the inclination to be involved in this. Of course Im interested in a
solution since some of the big cheeses here categorically will not use
Qliberated programs which, in my case, represents 100% of my games and
utility output. It saddens me to just be crossed out like that. To me a
program is what a program does, I dont much care about how it was made.
(Like the proverbial sausage.)

Its a pity this business wasnt sorted years ago. It seems to have been a
bugbear since at least the mid-nineties, when many of those involved in the
Qx0, SMSQ/E and Qlib were still around.

[RalfR]

Maybe you would first have to find out (if haven't already done) what QLib is throwing out. Although it shouldn't be so mysterious anymore because of the de-compiler. Apparently it's a kind of P-code (that's what it was called variously) with a part MC so that it can start as a job. And the runtimes do “something” with it, whatever it is.

The entire mechanism of compiled program -->> runtimes would first have to be decrypted.

[NormanDunbar]

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station. I found stuff like "jump to 3 vectors before a QDOS vector address" or "jump to 6 bytes behind that vector" and similar. Assuming that the disassembly has been properly done, it's a bit of a miracle that code works on multiple platforms at all.

Ok, I found it. The documented vector is BV_CHRIX in QDOS speak. Simon found a whole lot more which are offsets from that vector. Documented in sbvector_doc in DIY Toolkit volume A. Attached.


Some of Anders Hartzellius's questions prompted me to investigate SuperBasic storage allocation. In that process I uncovered a further eleven vectors that are reliable in all QL versions, from "AH" to Minerva; they are called in popular programs, yet they have never been documented.

If you have explored the QL Rom with a disassembler you may have noticed that the BV_CHRIX vector points to one entry in a table of routines that check for space in Basic tables and stacks. You can access other entries by adding standard offsets to the BV_CHRIX vector address.

The table at the end of this document sets out offsets to check for D1 bytes in each area. Thus, to reserve N free bytes at the end of the Program File:

CALL PEEK_W(282) + 52 , N

I checked these on Sinclair's "JM" Rom, Minerva 1.64, and on my Thor XVI version 6.41; they should be the same on your Rom, and every other.

The order and offsets of these entry points is fixed, even in Minervae. They are reliable as they are used by Toolkits and disk systems to expand the Channel Table.

The _DEF functions use the SuperBasic Buffer area to hold the index to procedure and function names. This always has at least 128 bytes, allowing 64 names, but is usually larger, and can be expanded by SPACE bytes with CALL PEEK_W(282) + 28 , SPACE . 1000 bytes is usually plenty for MORE and the _DEF functions.

Offset Area Name
------ ---------
+0 RI Stack
+6 Backtrack Stack
+12 Temporary Graph Stack
+18 Name Table
+24 Return Table
+28 Buffer
+32 Token List
+36 Name List
+40 Variable Values
+44 Channel Table
+48 LIST Line Table
+52 Program File

TABLE: BV_CH__X Vector offsets

sbvector_doc.zip

(3.4 KiB) Downloaded 28 times

[Peter]

But as I said before, I dont have the knowhow, the hardware, the time nor the inclination to be involved in this.

I'm in the process of building a Q60 computer for someone more interested. So we might know more in a while. At least how to reproduce the problem.

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station.

Thanks for speaking frankly. Contributes to my gut feeling, that chances for a fix in QLiberator itself are not very high.

[tofro]

First, (excuse the rant) the code is about as "clean" as the average public toilet in Frankfurt Main Station.

Thanks for speaking frankly. Contributes to my gut feeling, that chances for a fix in QLiberator itself are not very high.

Thus my proposal to not try and understand the code fully, that would really be a huge task and not time invested well if whoever does it wouldn't fix the most ugly places in the process - Instead, simply try and find out if some minimal changes in suspicious places might fix the problem. Easily done, and if they don't, nothing is lost, but if they do, we can use the code as-is without investing a lot of time.