How to Open Large JSON Files Without Crashing Your Editor

Two cheap commands tell you what you are dealing with, without parsing anything:

$ ls -lh orders.json | awk '{print $5}'
121M

$ head -c 160 orders.json
[{"id":"ord_0000000","status":"shipped","customer":{"id":"u_19773","country":"US"},"items":[{"sku":"SKU-50","qty":1,"price":164.79},{"sku":"SKU-97","qty":3,"pri

The first characters show the shape: a top-level array of order objects, written without whitespace. That decides the approach. A top-level array can be streamed one element at a time; a single giant object with one huge key is harder.

Parsing the whole document turns compact text into objects, and objects take far more memory than their JSON text. Peak memory for one full parse of the 121 MB file:

jq length orders.json                  8.0 s    1,277 MB
python3 json.load                      5.1 s      927 MB
node JSON.parse(readFileSync(...))     1.8 s      599 MB

That is 5 to 10 times the file size. Node also has a hard ceiling: it cannot create a string longer than about 512 MB, so readFileSync(path, "utf8") on a larger file fails with ERR_STRING_TOO_LONG before JSON.parse even starts. An editor or browser tab that renders the parsed tree needs memory on top of all this.

jq's --stream mode reads the document as a sequence of small events instead of building the whole tree. Combined with fromstream it hands you one array element at a time. To see the first three orders:

jq -cn --stream 'limit(3; fromstream(1|truncate_stream(inputs)))' orders.json

That returned in 0.01 seconds using 3 MB, because jq stops reading as soon as it has three elements. The expression looks cryptic, but it is a fixed recipe: 1|truncate_stream(inputs) removes the outer array level, and fromstream rebuilds each element.

For repeated exploration, convert the array into newline-delimited JSON, one order per line. Every line-based tool then works on it. There are two ways, with opposite costs:

jq -cn --stream 'fromstream(1|truncate_stream(inputs))' orders.json > orders.ndjson
                                            76 s      3 MB
jq -c '.[]' orders.json > orders.ndjson     16 s  1,269 MB

Both produce identical output. Use the fast one when the machine has memory to spare, and the streaming one when it does not, for example inside a small container. After the conversion, everything is quick and light:

# Count matching records: 2.1 s, 1 MB
grep -c '"status":"refunded"' orders.ndjson

# Filter on several fields, line by line: 4.4 s, 3 MB
jq -c 'select(.status == "refunded" and .customer.country == "IN") | .id' orders.ndjson

# Split into files of 100,000 records each: 2.6 s
split -l 100000 orders.ndjson part-

A grep count is only safe when the text you search for cannot appear inside other fields. Here status occurs once per order, so it counts orders. When in doubt, use the jq version. Each split file holds 100,000 complete records, small enough for the tree viewer or an editor.

When the exploration turns into a script, use a streaming parser rather than json.load. In Python, ijson yields one array element at a time:

import ijson

with open("orders.json", "rb") as f:
    refunded = sum(1 for order in ijson.items(f, "item") if order["status"] == "refunded")

On the same file that took 4.6 seconds and 18 MB, against 927 MB for json.load. The prefix "item" means each element of the top-level array; for a nested array, use a path such as "data.orders.item".

Use the tree viewer for documents up to a few tens of megabytes, and for any slice you have extracted from a bigger file: one record, one split file, or the output of a jq filter. The terminal is for finding the part worth looking at; the tree view is for reading it.