Hype Data Parser

Allows to process data and presently supports the CSV format. It offers ways to convert CSV data from text to list in array format, object format and now also to a named key nested object. This can be used together with Hype Data Magic, but that is totally optional. Also, loading data isn't currently part of this extension, as the browsers native fetch mechanism is pretty well established and easy to use.

Examples files demonstrating loading and parsing CSV / TSV strings

Here are some sample showing the usage of Hype Data Parser in conjunction with Hype Data Magic, but the latter is only to bind the parsed and loaded data to the view and totally optional.


GitHub repository

Documentation

3 Likes

↑ look at project
1.0.6 Added grouped option to CSV to object by key and csvToArrayByKey

↑ look at project
1.0.7 Fixed some regressions on defaults, thanks to @h_classen

Can this also work with online Google sheets?

In the examples above, you can find the following example:
https://forums.tumult.com/uploads/short-url/w6zvu8FfaWncUZ5F9R1EiMuGFEA.zip

↑ look at project
1.0.8 Remove leading and trailing whitespaces on CSV string in csvToArray, added filter option for csvToObject and csvToObjectByKey

↑ look at project
1.0.9 Added aliases to all functions to support TSV format without needing to always specify the field separator (fSep) as tab each time

↑ look at project
1.1.0 Added getTables to extract multiple tables from a single CSV/TSV string with the option to parse the tables directly as data, added includesTables


This update allows reading multiple tables from a single CSV/TSV file using HypeDataParser.getTables. This kind of multi sheet / table export is offered by Apple Numbers (not sure about Excel, though). The getTables function always create a flat object lookup with indexed keys, but additionally a named lookup (sheet → table) is created, if the CSV/TSV was exported with "Include table names".

Dialog found in Numbers → File → Export To → CSV or TSV:

  1. Enable the second select option "Combine tables into a single file"
  2. This step is optional and depends on if the table names are important to you
2 Likes

I was just playing with Hype Data Parser in conjunction with Hype Compressor.

Example code demonstrating using the parser with compression

It is, effortless to just ZIP your CSV/TSV data file(s) and then load them all in one go. You can even just rename .zip to .dat and making it not so obvious that ZIP encoding is being used.

In your Head HTML include the following, either from the CDN or download them (around 8kb in total):

<script src="https://cdn.jsdelivr.net/gh/worldoptimizer/HypeCompressor/HypeCompressor.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/worldoptimizer/HypeDataParser/HypeDataParser.min.js"></script>

In data loading (Hype function or external callback) use:

HypeCompressor.load("${resourcesFolderName}/application.dat",function(files){
	// use first unpacked file assuming it is an CSV / TSV (with tables)
	// files[0][1] contains the file name if you want to check
	var tables = HypeDataParser.getTables(files[0][0], true);

	// do what ever you want with the data, we only log it
	console.log(tables);
});

Example file demonstrating using the parser with compression

Here is an example, but it is slightly different compared to the above code, as it also uses Hype Data Magic to bind the received data to some rectangles on stage. This is of course totally optional. The example file is rather here to demonstrate how compression can be combined with the parser and the tables feature (either is also optional).

fetch_hype_data_magic_advanced_CSV_parser_using_compression_and_tables.hype.zip (47,9 KB)

1 Like

↑ look at project
1.1.1 Fixed some issues when reusing previously converted data from csvToArray

↑ look at project
1.1.2 Added rowsToColumns to transpose data in an array if needed, added redirection through getTables in csvToArray if includesTables is true

Quick an dirty, but - yes - you can do all this sort of things...

actionsByCSV_1.hype.zip (50,4 KB)

2 Likes

Awesome, going to check it out

Putting this here as it is a bit more "exotic", but might still be useful. Just patch this into Hype Data Parser if you need to support JSONL.

https://jsonlines.org/

/**
 * Converts a JSONL file to a JavaScript object.
 * @param {string} jsonl - The JSONL file.
 * @param {object} options - Options for the conversion.
 * @param {boolean} options.trim - Trim whitespace
 * @param {boolean} options.head - Remove head
 * @param {string} options.rSep - Row separator
 * @returns {object} - The JavaScript object.
 */
function jsonlToObject(jsonl, options) {
	
	if (typeof(jsonl) !== 'string') return;
	if(includesTables(jsonl)) return getTables(jsonl,  true);

	options = options || {};
	options.trim = options.trim || options.trimWhitespace || false;
	options.head = options.head || options.removeHead || false;
	options.rSep = options.rSep || options.rowSeparator || (getLineBreakChar(jsonl));

	jsonl = jsonl.replace(/^\s\s*/, '').replace(/\s\s*$/, '');
	
	var a = {};

	var rows = jsonl.split(options.rSep);
	if (options.head) {
		rows.shift()
	}
	rows.forEach(function(row, index){
		a[index] = JSON.parse(row);
	});

	return a;
}

Usage:

var jsonl = `{"id":1,"first_name":"John","last_name":"Smith","email":"john.smith@example.com","gender":"Male","ip_address":"1.0.0.0"}
{"id":2,"first_name":"Jane","last_name":"Doe","email":"jane.doe@example.com","gender":"Female","ip_address":"1.0.0.1"}
{"id":3,"first_name":"Bob","last_name":"Smith","email":"bob.smith@example.com","gender":"Male","ip_address":"1.0.0.1"}`;

console.log(jsonlToObject(jsonl));

There is also a version to convert to object key

/**
 * Converts a JSONL file to an object with a key
 * @param {string} jsonl - JSONL file
 * @param {string} key - Key to use as object key
 * @param {object} options - Options
 * @param {boolean} options.trim - Trim whitespace
 * @param {boolean} options.head - Remove head
 * @param {string} options.rSep - Row separator
 * @returns {object}
 */
function jsonlToObjectByKey(jsonl, key, options) {

	if (typeof(jsonl) !== 'string') return;
	if (includesTables(jsonl)) return getTables(jsonl, true);

	options = options || {};
	options.trim = options.trim || options.trimWhitespace || false;
	options.head = options.head || options.removeHead || false;
	options.rSep = options.rSep || options.rowSeparator || (getLineBreakChar(jsonl));

	jsonl = jsonl.replace(/^\s\s*/, '').replace(/\s\s*$/, '');

	var a = {};

	var rows = jsonl.split(options.rSep);
	if (options.head) {
		rows.shift()
	}
	rows.forEach(function(row, index) {
		var obj = JSON.parse(row);
		a[obj[key]] = obj;
	});

	return a;
}

Usage:

console.log(jsonlToObjectByKey(jsonl, 'first_name'));