Writing $$external extensions with QLib

[HAOUI]

Well, let's return to $$external programming and see how limit/avoid freezing/crashing system.

When the compiled $$external OBJ file is loaded into system (by LRESPR or similar), $$external keywords are registred as machine-code PROCedures or FUNCtions, but they are not. In fact, QLIB has prefixed each external with some machine-code or pointer to, from the QLIB Run time which has the responsabilty to create an independent job and exexcute the external as a normal QLIBerated basic program when the keyword is called from the basic command line.

When control is passed to this created job, the caller job (S*BASIC) is suspended in a way that it cannot be reactived until a normal return from the external job is done. If something gone wrong or job was paused/stoped/killed, the caller will wait undefinitely (Freeze) or corrupt the system (Crash).

To avoid this, it is imperative that the external returns in all cases from the normal end point of the main procedure or function (END DEFine instruction or RETurn). This means you cannot exit from any else place even if an error is encountered or other internal PROCs/FUNCs are called.

Then the first rule will be : Never use a STOP instruction anywhere in the program and manage to unstack calls to internal PROCs/FUNCs in a reverse order that they have been called, no matter what happened.

Alas, this will not be enough. If an uncontrolled error occurs, the QLIB runtime error popup (R/C/A) will STOP/Kill the job without what was planened for return. When this popup window appears, system is nearly crashed. Apparently, this error trap is not compliant with $$external and has to be patched/modified/rewritten to be compatible with $$external use.

In the meantime, the second rule will be : Make your best to catch and control any run time error (true for any kind of porgram ;o).

Some good pratices and tricks:
- when available prefer using FUNCtions form of some QDOS commands that return error you can test; FOPEN() for OPEN, FMAKE_DIR() for MAKE_DIR...
- For others QDOS commands (like COPY, INPUT...), you can use the Q_ERR_ON facility for crucial commands and test Q_ERR returned code after the command. This works fine but only for PROCs type commands.
- You can use a WHEN ERROR clause but it has a limited utility if an error occurs inside lower calls and anyway you shoud exit the clause with a RETURN statement and not a STOP. To be efficient, you have to use ERLIN or GOTO and this is not a good practice.
- Pay attention to arrays indexes evaluated in run time especially for string slicing (index out of range QLIB error).
- Pay attention to "unset" variables used in some expressions or having a value not "coercedable" (variable undefined or wrong errors)
- Dont assume anything about passed parameters to the external, even, that they was passed or they exist.

Next time, we explore the more or less "parameters passing".

Alain

[EmmBee]

Nice article, I enjoyed reading this.

I notice you didn't mention anything about the stack. I can sometimes get better results (less crashing) when I use larger and larger stack space. Also increasing the heap can help. It could well be that it's merely running out of memory and can't keep up and crashes.

I use QPC2, and it's not very helpful when a crash does occur. One never knows what caused it. Sometimes there could be a mistake in the program that has gone unnoticed. All I can do in these circumstances is to reload the system and try again.

EmmBee

[HAOUI]

Thank you EmmBee for your feedback.

I notice you didn't mention anything about the stack. I can sometimes get better results (less crashing) when I use larger and larger stack space. Also increasing the heap can help. It could well be that it's merely running out of memory and can't keep up and crashes.

You are right. I was simply considering that "stack, heap and channel table size" are common problems well known by QLIB programmers and have to be tuned early at test starting. Compiling with option STAT can really help.

What I know : a heap shortage will not cause crash because the program will automatically request more heap slices (by 512 bytes) from the common heap system. This may cause performance problem and common heap fragmentation.
About stack shortage, QLIBerator manual states "If a program runs out of stack, it will stop with QLIB error", but I don't believe it is always true. You should better have correctly (over)sized the stack.
Channel table is more sensible. Attempt to open a channel number higher than the table size, will for sure, have unpredictible behaviour and corrupt something somewhere.

I use QPC2 ...

I am now using QPC2 when programming and I like it enough. I can use very productive IDE tools from windows side, and a simple cycle of modif, load, compil & test can be done in few seconds. When programm crashes only QPC2 is concerned and can be restarted quickely. It's a matter of personal taste...

Alain

[HAOUI]

Hi all,

A little bit wondering about few feedbacks here, given the huge number of users using QLIB and externals facilities which are likely the most exciting feature permitting to share BASIC compiled liberaries. This is the QL life...Isn't that Norman ?

Well as promised, let's have a look this time to what happens after having successfully produced the X_OBJ program file that contains $$external procedures.

In fisrt step when we load and register externals by LRESPR X_OBJ or equivalent methods, following things happen:

• 1) As usual, the whole X_OBJ file is loaded into memory and a call to first instruction in the file is made
  2) As all QLIB compiled programs, this starts by a branch to locate the necessary QLIB RUNTIMES which could be embeded within the X_OBJ or not.
  3) The RUNTIMEs look if they are some externals in the X_OBJ and if yes :
  • a) Registers each external (a linked list is included in the X_OBJ. Thanks Martin/DeLIB) with a classical call to BP.INIT vector utility using a1 register pointing to a defintion table containing the name of the external (as keyword) and an offset code to some location in the X_OBJ which will be used when the new keyword will be called.
    b) This offset location precede the real location of the external by some words containing a branch to the start of the X_OBJ and the offset to the real location of external code moved into register d4.

That is all for this step.

When the external is called by its associated Keyword from S*BASIC, following things happen :

• 1) S*BASIC thinking calling a machine code procedure, copies all parameters to pass on its Name Table
  2) Set registers a3 and a5 pointing to entries in the Name Table to actual passed parameters as usual for assembler extensions. Unfortunately, it doesn't create an entry on its Return Table as for Calls to BASIC PROC/FUNC.
  3) Jump to the absolute address registered with the Keyword.

That is all from the caller S*BASIC point of view.

When the control is passed to the Keyword absolute address, follwing things happen :

• 1) The registered absolute address does a branch to the start of the X_OBJ loaded in memory with register d4 containing the offset to the real external procedure code.
  2) As always for QLIB programs, this starts by a call to RUNTIMEs.
  3) RUNTIMEs will detect that the program is executed for the external assigned in d4 and will do a lot of things before executing the external compiled procedure, Schematicaly :
  • a) It create a new job for the external
    b) Copy/Create the whole data area for the new job (each job has its own data area)
    c) Set the start address of the job to the external program code loaded in memory (all jobs share the same re-entrant code)
    d) Copy (in some obscure way) the passed parameters from caller to the new job
    e) Suspend the caller job (S*BASIC intrepreter)
    f) Execute the job as a normal children QLIBerated program
    g) When external job (happy)ends, RUNTIMEs again copy back (in some obscure way again) altered parameters for the caller.

Next time, with some feedbacks in the meantime I hope, we can have a look to the passing actual parameters process, with some gifts. I promise.

Alain

[RalfR]

Hi all,

A little bit wondering about few feedbacks here, given the huge number of users using QLIB and externals facilities which are likely the most exciting feature permitting to share BASIC compiled liberaries.

Don't bother. The same with the QLib decompiler. There were hundreds of ideas from people to enhance it, the first was to make its window go further than 512x256. But there comes nothing up until now. We have to wait .

Meanwhile I love to read your interesting articles! For me, it would be interesting to know, if a QLib job (not necessarily an external) works in a similar way than SBASIC or totally different. If an _obj is a sum of bytes which are run by the runtime system or something different.

One REMark:

e) Suspend the caller job (S*BASIC intrepreter)

I think, an "!" after the keyword makes the generated QLib job like EX, it does not suspend the Caller.

[tofro]

I think, an "!" after the keyword makes the generated QLib job like EX, it does not suspend the Caller.

That is really a nice idea to build a self-contained multi-threaded S*BASIC program, however:

It should, but it doesn't, - At least not on SMSQ/E.
I used to have an idea why not, but forgot...

Tobias

[pjw]

Hi all,

A little bit wondering about few feedbacks here, given the huge number of users using QLIB and externals facilities

Yup, welcome to the mute world of QL! Its dumb, but appears sometimes to listen!
I, for one, do read what you write, despite the fact that Im not really into Qlib externals. Wherever possible, if I need an
extension I try to write is in assembler. This is because I started out with a 4K computer and still find it traumatic to use
16K to implement an UPPER$ function. But dont mind me. You never know when you might need some extra
kit in your toolbox!

[RalfR]

I think, an "!" after the keyword makes the generated QLib job like EX, it does not suspend the Caller.

That is really a nice idea to build a self-contained multi-threaded S*BASIC program, however:

It should, but it doesn't, - At least not on SMSQ/E.
I used to have an idea why not, but forgot...

I think, you must modify runtimes and/or _obj to make that work under SMSQ/E. Wasn't there any patch program for that? I remember such question from Oliver Fink elsewhere where he wasn't able to use "SEDIT!" from #0 but got an answer.

[EmmBee]

I've tried out the new PARSTR$ keyword from the tkparex.zip. This now works on my DCOPY program, I no longer have to include the quote marks. One thing to notice is that if the result from PARSTR$ is copied back to the parameter, it stops with an error. This is noted in the SBASIC/SuperBASIC Reference Manual. To check whether it is the Interpreter making the call, OJOB(-1) gives the owner job. If that is zero, then S*BASIC has made the call. So now, one can avoid it if called from a compiled job.

[HAOUI]

I think, an "!" after the keyword makes the generated QLib job like EX, it does not suspend the Caller.

That is really a nice idea to build a self-contained multi-threaded S*BASIC program, however:

It should, but it doesn't, - At least not on SMSQ/E.
I used to have an idea why not, but forgot...

Tobias

Indeed, this often crashed my systems. My be related to copying back parameters or multiple versions, mods and patchs.
A simple rule I learned : Have a normal, good & successful return from QLIB program or GOTO reset button ;o)
Alain