Hype Compressor

Here is a Hype generated file JavaScript compressor based on Hype Document Loader (for PHP).
It has the most impact on files using regular symbols. Give it a try and report back if it worked for you and how much compression you gained from it. I don't store any information on the server and the script takes the uploaded JavaScript and echos it in a processed manner back directly from the temporary folder. As it is not part of the preview workflow yet (as an export script for example) I recommend using this helper as the final step before deploying bigger files.

Symbol-Compression:
https://playground.maxziebell.de/Hype/Compressor/

Symbol-Compression with additional ZIP-Encoding (Base64-Encoded):
https://playground.maxziebell.de/Hype/CompressorZipEdition/

Read me:

Repository for the JavaScript-Decompressor part:


This only compresses the generated JavaScript file found in the resources folder. It is not for any runtime or asset compression. Use ImageAlpha, ImageOptim and SVGO for such things on your local computer (and online checkout SVGOMG). As much as I would like to offer a full OAM optimization including images and more that is beyond the scope of my time and server capabilities. Also, remember to test your files before deployment. I haven’t had any reports of it breaking anything so far but as always use at your own risk (specially for legal reasons).

7 Likes

In a nutshell it compresses the hype generated file though what is it doing behind the scenes when its compressing? Meaning, is it cleaning up the code or minifying it? Or it just makes symbol lines of code condensed in some way?

This is so great - coming from Flash I always wondered, why using symbols doesn´t minimize the file size (Movie-Clip -> Instance principle) - quite the contrary, it increased it a little bit. For my understanding this is, what the compresser does, @petester - compressed hype generated files treat symbols as references to the original symbol without writing the original code over and over to the hype file.

1 Like

@ktewes yes, that is right. Exactly how it works and on top it does a string lookup in the above Hype Compressor. I just completed a MVP also doing a ZIP compress with an browser based decompression routine (only 7kb big and I might even squeeze that to 5kb). The ZIP-Edition converts the entire document loader into a ZIP string but then adds a little overhead back in because we are in ASCII land and therefor use base encoding. In the browser I reverse that compression and if you stack it all up you get a 1 to 10 compression. My example file started with 1,9 mb and when I drop it here it comes back as 206kb and works flawlessly!

Still needs progress indication and a success feedback:
Moved link into first post!

Thank you for making it clear @ktewes

@MaxZieb I’m just thinking if I wanted to use this on the fly without dragging and dropping can you create a folder action (is it even a possibility) for a converted hype generated file?

Thanks...( I'm newbie in code..) but I though I take a chance...I throw the .js in the compressor...and yes, it shrinks 90% (from 93 to 7 kb) :wink:

I've send it to our advertise agent of Ebay and says its good!!

i zipped it afterwards...and the different was something like 25kb, but still a lot for me, thanks!

little question, is every .js the same? I made banners for different countries...fo now I have a German and Belgum banner, both for the size 300x250... but the look exactly the same...

This would be easier to just change all of the .js with the .js version which comes from this composer.

I also have other sizes...I don't see any text in this .js which is connected to the size of the banner.

In general, I would love to have the compression work on an export script basis or as @petester mentioned as some sort of app. Beware, that the compressor was just release yesterday. I am still looking for feedback and bug reports. I am glad to hear that your ad works as expected. I had a report that the optional string optimization broke an inner HTML-based animation, and I am investigating that as we speak. But that is a special case. Without string optimization the innerHTML animations work, so I am leaving the tool online but disabled that additional compression feature for now. Still yields good results, though. In the ZipEdition we load a HypeCompressor.decompress function and that could also be used for the runtime. I am looking into that too.

@everybody reports like these and feedback help me to refine the tool!
@cartimundi Thanks a lot! I just saw that you are trying to compress the runtime. That is not the intention of the compressor, but I will look into it. This is currently meant for compressing the *_hype_generated_script.js file. I will add some feedback if the tool detects wrong JS files dropped and look into Runtime ZIP compression.

I was a bit aggressive in the string substitution department and at one point I wanted to save some quote characters around IDs that could very well also work as numbers, so I converted them. The runtime doesn't like that, so I now stop doing that particular conversion while running the string substitution. This comes at the cost a 1-2% performance. So, everything is back to normal with the one bug squashed that has been reported so far. Enjoy the tools and let me know what you think and experience (positive and bugs).

One thing I saw while creating some small files for testing. The Symbol-Compression doesn't really add any benefit if a symbol is only used once and the ZIP-Edition adds the decompressor neglecting any gains on very small files < 12-14KB. I will add some sort of statistic to the tool and only if an action yields benefit will I apply it. Makes sense! For now just compare before and after and don't use it if the result isn't smaller :slight_smile: Early days.

2 Likes

Morning...sorry but I have to say it...I thought it was working, but when I try it in

https://h5validator.appspot.com/dcm/asset?result=4645636731437056

it doesn't work ;-(
in the zip 2 versions...the one with _2 is with you're version .js

Have you converted since I did the fix yesterday or is the file from before? I’ll have a look today.

O okay...I will try it later today...

The one I used was from 18 hours ago..

16 European time (Netherlands)

@cartimundi please read the message I wrote for you two posts previously. You are trying to compress the runtime. That is not what the compressor is meant for currently. It is meant for the hype_generated_script and not viable for your ads because you are including the generated script in your HTML file. Hence, the compressor in its current form isn't for your type of file distribution. I'll let you know once the compressor supports runtime compression and/or compression of inlined generated script.

Okay thanks...please let me know, when composer 2.0 is ready :wink:

There are two versions of the compressor
Some thought and hints:

:green_circle: The first one is useful in most cases (reducing the overall size of symbols and strings by using a lookup. This is a simple and trivial rearrangement of the data structure describing your Hype file. Putting such a symbol compressed file in a Zip-Container results in a smaller file than the unprocessed file (overall project ZIP size and also mod_deflate benefit). So, this kind of optimization makes nearly always sense but relies on regular symbols or similar text strings being used across scences to show results.

:orange_circle: The second compressor currently called "ZIP-EDITION" needs some more evaluating and this is what this sections is mainly about: The ZIP-EDITION compressor can now also handle any JS library (in those case it doesn't add the decompressor (4,4kb) and only returns the JS in a GZIP compressed and Base64 encoded manner. The logic thus far is that if you need smaller files the ZIP-EDITION compressor can produce them through additional compression. The result is a binary file that can't be embedded into an ASCII file (as half of that ASCII syntax is reserved). Hence, we need to encode the full byte array into an ASCII compatible realm. The most common used and MIME compatible way of doing this is using Base64. This adds 33% overhead as we need more Bytes to describe the binary data. The compressor also respects the RFC2045 standard that demands splitting lines into 500 char max chunks. This adds 3% of newline and quote data. Then we need the decompressor at around 4,4kb. All this adds up but in most cases we are still smaller than the original file at face value. Meaning if you need a smaller file this is for you but zipping this file produces a bigger file then zipping the original. Hence, if you need smaller individual files to pass some test or client demand or your goal is reduced storage or some added obfuscation is needed this compression is for you. On the other hand ... if overall ZIP size of your entire project is the optimization goal refrain from using ASCII file base compression.

:red_circle: If your server uses mod_deflate (as mentioned by @jonathan) I would also only use the symbol compression version as the on-the-fly zipping applied to the additional Base64 overhead most probably doesn't yield better results. The symbol compression is respected by mod_deflate, so you gain benefits. Another approach currently not implemented is using a more efficient Binary to ASCII encoding. There is ASCII85 (only adds 25% overhead) and yEnc (adds only 7-10% overhead). Either of these isn't MIME certified, but I will try to add them in the future anyway as an option if viable (they are used in USENET protocols and elsewhere). Hence, they are broadly used and would bring the compressor much closer to native zip values but because of the missing certification I mentioned this under the red dot.


Eventually, I will consolidate the compressor versions and add the different optimizations as toggle switches (if time allows it).

1 Like

This project can now be found on GitHub

Other great news! I just ran a simple test of loading a binary ZIP-File using a XMLHttpRequest and decoding it. I reduced the decoder to only single files, so no folders are supported but for the purpose of encoding hype_generated_script and the runtime or arbitrary JavaScript that is sufficient!


var zipReq = new XMLHttpRequest();
zipReq.open("GET", "${resourcesFolderName}/someCompressedJS.zip", true);
zipReq.responseType = "arraybuffer";
zipReq.onload = function (oEvent) {
  var arrayBuffer = zipReq.response;
  if (arrayBuffer) {
    var byteArray = new Uint8Array(arrayBuffer);
    // decode JS to console for this binary file test
    console.log(HypeCompressor.decompressArray(byteArray));
  }
};
zipReq.send(null);

it works flawless… hence native ZIP data directly from the binary source will now become an option. No ASCII overhead. HypeCompressor will be updated very soon with new API similar to...

v1.0.4 preview:

  • decompressArray
  • runBinary (files could end in .dat)

:smiley:

3 Likes

Whoa, that's an awesome technique!

2 Likes

It gets even better… one can ZIP the resource folder and read from it. For example, I use HypeResourceLoad to return images, I reinstated the recursive nature of Util.Unzip for that in the HypeCompressor.js but that only costs 1kb extra data on the library, so worth it:

function HypeResourceLoad(hypeDocument, element, event) {
	if(!data) return;
	var file = event.url.split('/').pop();
	var type = file.toLowerCase().split('.').pop();		
	for(var i=0; i<data.length; i++){
		if (data[i][1].substr(0, 1)=='.') continue;
		if (data[i][1] == file){
			switch (type){
				case 'jpg':
				case 'png':
					return 'data:image/'+type+';base64,'+window.btoa(data[i][0]);
					break;
			}
		}
	}
}

if("HYPE_eventListeners" in window === false) { window.HYPE_eventListeners = Array(); }
window.HYPE_eventListeners.push({"type":"HypeResourceLoad", 

Now, I ask myself what resources can be handled directly from a ZIP given that approach.

There is also another Idea, but it requires a modern browser: to fill the cache with the files from the ZIP, but caches doesn't support IE11. I hate that browser so much… and it probably should/could be disregarded.

1 Like

Oh wow, you're right that could be very interesting. I would imagine that ultimately making an optimum export boils down to trying different strategies and determining the winner; it is nearly impossible to know ahead of time how items will compress beyond generalities. A video may rarely make sense to compress this way (they are large, can't take advantage of HTTP offset requests, etc.) but if it is a single small video in a sea of compressible SVGs then it may make sense to lump together...

1 Like

I updated the JavaScript on GitHub to 1.0.4

Super!

is the browser link the same? ( I'm a zero in code ;-(
don't know what to do with the GitHub

https://playground.maxziebell.de/Hype/Compressor/