No fun funzipping

[pjw]

In some projects I use zip/funzip to archive certain material on the fly via pipes. But what takes zip about a second to do, it takes funzip 10 seconds to undo! The same material can be unzipped with the normal unzip in about a second or two. Im using the latest known versions of Jonathan Hudson's zip suit V2.3 for QDOS/SMSQ/E.

So my questions are: Why? Can I fix it? Can anyone else fix it? Is there something else I could use that does more or less the same? (I dont need individual files, search, encryption etc, just a pure stream of data going in, being compressed, and coming out - and the reverse, of course). Is there a way of fooling unzip to do the same - without the use of intermediary files?

TIA

[tofro]

Per,

did you try "unzip -p" instead of "funzip" (which should basically do the same thing?

Tobias

[pjw]

did you try "unzip -p" instead of "funzip" (which should basically do the same thing?

I was discouraged, as the manual stated "Archives read from standard input are not yet supported,
except with funzip (and then only the first member of the archive can be extracted)."

I did try anyway, but no arrangement of parameters or (pre-opened) channels produced the desired
effect:

Code: Select all

500 id = FEX_M('win2_arc_zip_funzip', fnm$, 'pipe_in_2048')
510 cpi = FOP_IN("pipe_in")
..

BTW: My unzip version was probably 5.2

[pjw]

Ok, over to plan B: Does anyone have a current email address for Jonathan Hudson? The last contact I had with him was in 1998, and that address no longer works. Please PM it to me if you can.

[XorA]

Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!

[pjw]

Is this a streaming speed issue, because zip algorithm needs to read the end of archive first!

You cracked it! Thanks a lot!
Increasing the pipe size to 32768 reduced the time to a fraction, even for archives many times this size. (I tried pipe sizes beyond that but it didnt appear to make any difference..)

[mk79]

Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.

[pjw]

Have a look at bzcat/bzip2, it's specifically for compressing/decompressing streams of data. I've used it to stream huge amounts using SBasic before.

Thanks for the tip
I checked out the version currently on Dilwyn's site. This is what I found:
Sample: 64 files @ 32792b == 2Mb to be squashed into a single archive, tmp$
:
bzip2:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_bzp_bzip2', 'pipe_out', 'nul', tmp$)
<stuff data into pipe>
..
and
..
id = FEX_M('win2_arc_bzp_bzcat', tmp$, 'nul', 'pipe_in_32768')
<get data out of pipe>
..

pipe_out 32k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 32k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 10k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 5k, t 1s
pipe_out 2k, zip t= 17s, unzip pipe_in 2k, t >5min or choked

Compressed size: 183440

zip:

Code: Select all

..
cpo = FOPEN("pipe_out_2048")
id = FEX_M('win2_arc_zip_exe', 'pipe_out', 'nul', 'ram1_tst_zip')
..
and
..
id = FEX_M('win2_arc_zip_funzip', tmp$, 'nul', 'pipe_in_32768')
..

pipe_out 32k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 32k, t 2s
pipe_out 2k, zip t= 2s, unzip pipe_in 10k, t 4s
pipe_out 2k, zip t= 2s, unzip pipe_in 5k, t 10s

Compressed size: 176478

183440 - 176478 = 6962 in zip's favour

Conclusion: For the kind of data I tested, zip compresses faster and meaner,
but is slow at decompression. bzip is super fast at decompressing. Sadly bzcat
doesnt decompress zipped files!

Different settings may give different results! If you or anyone knows how any of
this may be improved, please post it here! For now Ill stick with zip/funzip