#827 Add implementation for the new JSON connector

[lmolotii]

JSON Connector specification

Overview

Implements the basic JSON connector that supports reading & writing JSON and JSONL (JSON Lines) files, it extends the existing implementation of the file connector, providing more options to configure the connector.

Usage Examples

Reading JSON Files

import wrangles

# Basic JSON read
df = wrangles.connectors.json.read('data.json')

# With column selection
df = wrangles.connectors.json.read('data.json', columns=['col1', 'col2'])

# With specific orientation
df = wrangles.connectors.json.read('data.json', orient='records')

# With encoding
df = wrangles.connectors.json.read('data.json', encoding='utf-8')

Reading JSONL Files

# Basic JSONL read
df = wrangles.connectors.json.read('data.jsonl')

# With nrows (only works for JSONL!)
df = wrangles.connectors.json.read('data.jsonl', nrows=100)

# Explicit lines parameter
df = wrangles.connectors.json.read('data.jsonl', lines=True, nrows=50)

Using in Recipes

# Read JSON
read:
  - json:
      name: input.json
      orient: records

# Read JSONL with row limit
read:
  - json:
      name: input.jsonl
      nrows: 1000

# Write JSON with pretty printing
write:
  - json:
      name: output.json
      indent: 2

# Write JSONL
write:
  - json:
      name: output.jsonl

Writing JSON/JSONL Files

import pandas as pd
import wrangles

df = pd.DataFrame({'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']})

# Write JSON with indentation
wrangles.connectors.json.write(df, 'output.json', indent=2)

# Write JSONL
wrangles.connectors.json.write(df, 'output.jsonl')

# Write with column selection
wrangles.connectors.json.write(df, 'output.json', columns=['col1'])

# Write with specific orientation
wrangles.connectors.json.write(df, 'output.json', orient='records')

Explicit Connector Parameters

All parameters are explicitly documented and validated:

Read Parameters:

• name - File path (required)
• columns - Subset of columns to read
• orient - JSON format ('split', 'records', 'index', 'columns', 'values')
• encoding - File encoding (default: 'utf-8')
• nrows - Number of rows to read (JSONL only!)
• lines - Whether file is line-delimited (auto-detected)
• compression - Compression type ('infer', 'gzip', 'bz2', 'zip', 'xz')

Write Parameters:

• name - File path (required)
• columns - Subset of columns to write
• orient - JSON format
• indent - Indentation level for pretty-printing (JSON only, not JSONL)
• lines - Write as line-delimited (auto-detected)
• compression - Compression type

[ebhills]

In you test examples your recipe looks like:

read:
  json: ...

read / wrangles / write objects contain lists so each read / write must be a list element like:

read:
  - json:

[lmolotii]

@ebhills, yes, the example description is not correct. I've fixed it. However, the implementation is ok. Please check for the 'test_write_json_basic' unit test to verify.

@ebhills, yes, the example description is not correct. I've fixed it. However, the implementation is ok. Please check for the 'test_write_json_basic' unit test to verify.

[ebhills]

@lmolotii - the test syntax is still wrong. Even if it works, I would prefer to document / test using the correct syntax.

[lmolotii]

@lmolotii - the test syntax is still wrong. Even if it works, I would prefer to document / test using the correct syntax.

I've adjusted the tests to be compliant with the syntax remarks. Please check.

[thomasstvr]

@lmolotii we should add tests with multiple parameters used at the same time.

Also, the massive amount of if statements seems a bit hacky. Would be nice to condense those in some way.

[lmolotii]

I've added multiple tests to cover multi-parameter scenarios. However, I don't think that we can do something about multiple if's. Basically, there, we try to configure the pandas_kwargs, and if an argument is present, then we set the config in kwargs. How to avoid it, without adding the complexity for the implementation, is a good question.

I had several options in mind:

1. Build the dict in one statement and filter out default values
2. Parameterized mapping approach - to define a configuration that maps parameters to their defaults, then iterate
3. Helper function approach -where we can create a reusable utility to handle conditional kwargs building

But, it is just adds more code, without adding any value... I would prefer to keep it simple and in one place.

[lmolotii]

I've updated the initialization of the pandas_kwargs. Please review it and let me know if you are ok with this approach.

[thomasstvr]

Users should not be allowed to set this parameter. This just opens the door for something like below, which there is no error handling for.

read:
  - json:
      name: tests/samples/data.json
      lines: True

[thomasstvr]

Instead of setting so many parameters to None, lets just set them to the default in the first place. This will clean up all of the if statements, and also be much easier to quickly tell what the default is set to.

[thomasstvr]

The file connector uses a small handful of the same files to read and write, let's mimic that here. It just makes it easier to actually check the file being read/written if need be. For an example, see test_write_jsonl in test_file.py.

[thomasstvr]

Please add comments where any ambiguity might exist, like here.

[ebhills]

Queue triage (2026-07-27)

• Disposition: Draft — delivery-owner action
• Delivery owner: @lmolotii
• Next action: Decide whether issue new connector: JSON #827 is still wanted. If yes, update from main, resolve the merge conflict and four open review threads, and narrow the 1,379-line change before requesting a reviewer.

GitHub is the status record; update this PR rather than the external spreadsheet.