JSON file containing single 81 MB object won't load in Zui

[philrz]

tl;dr

A community zync user reported failure to load the JSON file stored at s3://zui-issues/3025/mun.json (it's on S3 because it's too big to attach here in the issue) into Zui. Closer study shows evidence of multiple bugs, so this issue walks through the high-level behaviors. Other issues or PRs will be spawned from here as necessary.

Details

Repro is with Zui commit a7c3aed which uses Zed commit 38763f8.

The community zync user originally asked:

any idea why the following file mun.json would not load in the latest Zui?
I am guessing it may be the (large) size of some geometries, but not sure ...
I should mention this data is public, so it can be shared.

The JSON file is 81 MB in size and contains a single object. A summary:

$ ls -l mun.json 
-rw-r--r--  1 phil  staff  85114393 Mar  7 08:10 mun.json

$ zq -version
Version: v1.14.0-16-g38763f82

$ zq 'count()' mun.json 
1(uint64)

$ head -5 mun.json 
{
"type": "FeatureCollection",
"name": "mun",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::4269" } },
"features": [

The attached video shows my own walk through the repro. A summary of what we see:

1. zq has no problem reading the file via auto-detect at the shell
2. When dragged into the app, the "Loading..." messages in the Preview screen never disappear no matter how long we wait
3. Two zq processes of the head 100 variety seem to stay open indefinitely as long as we remain at the Preview screen
4. If we go forward with clicking Load, it fails and shows auto-detect failure messages
5. Even after we've canceled out of the failed load and deleted the pool, a zq process of the zq -i auto * variety seems to stay open indefinitely
6. I never see any apparent error messages on the DevTools console

Repro.mp4

[philrz]

A summary of what's been spawned from this issue:

• While zq can read the JSON without complaint, there's some curious challenges getting the same JSON to successfully zed load. This is described in "buffer exceeded max size" when reading JSON array via auto-detect zed#3865 (comment).
• The main symptom of not being able to load the JSON into Zui should be fixed by Specify explicit ZNG format from file loader #3026
• I can't yet explain what's cited in bullets 2, 3, & 5 above, so I'm hoping @jameskerr can look into those at some point.

[philrz]

As shown in the attached video, the main symptom of this issue has been addressed by the changes in #3026. The mun.json file can now be loaded into Zui without complaint.

Verify.mp4

This issue will remain open as a reminder for us to look into the zq processes that were left behind when the load was failing.