jsonl.dump_fork¶

Write data to multiple JSON Lines files simultaneously. This is useful when you need to split data across different files based on some criteria, while minimizing memory usage through incremental writing.

Function Signature¶

jsonl.dump_fork(
    paths,
    *,
    opener=None,
    text_mode=True,
    dump_if_empty=True,
    max_open_files=64,
    cls=None,
    **kwargs,
)

Parameters¶

Parameter	Type	Default	Description
`paths`	`Iterable[tuple[str \| PathLike[str], Iterable[Any]]]`	(required)	Iterable of `(filepath, items)` tuples
`opener`	`Callable` or `None`	`None`	Custom function to open the given file paths
`text_mode`	`bool`	`True`	If `False`, write bytes instead of text
`dump_if_empty`	`bool`	`True`	If `False`, don't create empty files
`max_open_files`	`int` or `None`	`64`	Maximum number of files kept open; `None` disables the limit
`cls`	`type[json.JSONEncoder]` `Callable` or `None`	`json.JSONEncoder`	Custom encoder
`**kwargs`			Additional keyword arguments passed to the `cls` encoder

Behavior¶

If the same filepath appears multiple times, subsequent data is appended to the file.
At most max_open_files destinations remain open. When the limit is reached, the least recently used writer is closed and reopened in append mode if that destination appears again. Pass None to preserve the previous unbounded behavior.
Raw bytes paths and PathLike objects returning bytes are rejected; decode them with os.fsdecode() first.
Files can use compression extensions (.gz, .bz2, .xz, and .zst Python ≥ 3.14 ) and will be compressed accordingly. Reopened compressed destinations use concatenated streams supported by their corresponding readers; frequent reopening can reduce compression efficiency.
Custom openers must support both write (wt or wb) and append (at or ab) modes when destinations are reopened.
When dump_if_empty=False, files with no data are not created.

Examples¶

Split data into separate files¶

import jsonl


def generate_player_files():
    """Yield (filepath, records) tuples — one file per player."""

    data = [
        {"name": "Gilbert", "wins": [{"hand": "straight", "card": "7♣"}]},
        {"name": "May", "wins": [{"hand": "two pair", "card": "9♠"}]},
        {"name": "Gilbert", "wins": [{"hand": "three of a kind", "card": "A♦"}]},
    ]
    for player in data:
        yield (f"{player['name']}.jsonl", player["wins"])


jsonl.dump_fork(generate_player_files())
# Creates: Gilbert.jsonl (with 2 entries), May.jsonl (with 1 entry)

Write to multiple files with static data¶

import jsonl

data = [
    ("users.jsonl", [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]),
    ("orders.jsonl", [{"id": 1, "total": 99.90}, {"id": 2, "total": 45.00}]),
    ("users.jsonl", [{"name": "Eve", "age": 28}]),  # Appends to users.jsonl
]

jsonl.dump_fork(data)

Custom serialization¶

import orjson
import jsonl


def worker():
    yield ("numbers.jsonl", ({"value": 1}, {"value": 2}))
    yield ("strings.jsonl", iter(({"a": "1"}, {"b": "2"})))
    yield ("numbers.jsonl", [{"value": 3}])


# Using orjson for faster serialization
jsonl.dump_fork(worker(), cls=orjson.dumps, text_mode=False)