Skip to content

feat(csv): choose comma, semicolon or tab delimiters - #760

Merged
cevheri merged 2 commits into
libredb:mainfrom
2160039878-cyber:feat/csv-delimiter-688
Sep 10, 2026
Merged

feat(csv): choose comma, semicolon or tab delimiters#760
cevheri merged 2 commits into
libredb:mainfrom
2160039878-cyber:feat/csv-delimiter-688

Conversation

@2160039878-cyber

@2160039878-cyber 2160039878-cyber commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Description

CSV import and result export now offer comma, semicolon and tab delimiters. Comma remains the default, so existing exports and callers keep their current output.

Closes #688.

Type of Change

  • New feature (non-breaking)
  • Documentation and test updates

Changes Made

  • Parameterize the shared csvRow / toCsv writer and quote values containing the chosen separator. Keep quote escaping, CR/LF handling, formula neutralization and value serialization unchanged.
  • Add an explicit delimiter picker to the CSV import preview. Reparse the original file text when the delimiter or header setting changes, preserve column mappings and reset to comma for a new upload.
  • Pass the export choice from the result menu through standalone and embedded downloads to buildResultExport; keep agent artifact provenance and display masking on their existing paths.
  • Keep the CSV file extension, UTF-8 MIME type and download BOM. Update README and the feature guide.

Testing

  • TDD: six writer/parser tests and nine component tests failed before the implementation.
  • bun run test:unit --isolate --pass-with-no-tests -t 'csvRow|toCsv|parseCSV|buildResultExport': 154 passed, 0 failed.
  • Component checks cover Studio, StudioWorkspace, BottomPanel and DataImportModal, including real export menus, downloads and generated SQL. Final focused bun run test:components --pass-with-no-tests -t 'DataImportModal': 50 passed, 0 failed, after correcting a test's button label. The other 433 matching component tests passed in the broader component run.
  • Covers quoted delimiters, escaped quotes, decimal commas, export newlines, headerless imports, picker reset, hidden CSV options for JSON, and agent/embedded export forwarding.
  • Passed locally: format, lint, typecheck, knip, readme:check, chart:check, channels:showcase:check, security:check, production build, build:lib and attw.
  • Review follow-up: clarified the NEEDS_QUOTING docstring to distinguish fixed quoting characters from the parameterized delimiter check. The targeted Biome format check passed; this follow-up changes comments only.
  • Full local bun run test / coverage and E2E were not completed: this Windows host lacks Helm/chart dependencies, Docker is unavailable, and existing SQLite cleanup tests encounter Windows file-lock errors. Official Linux CI must verify the full suite and 100% line-coverage gate.

Environment: Windows, Node.js 24.18.1, Bun 1.4.2. Both builds ran from a clean checkout of 9e054c574707bf46a648fab2a7f6828b2a8429a7 with real local dependencies, before the comment-only review follow-up.

Checklist

  • Reviewed the diff, added regression tests and updated documentation.
  • Required CI test job passes the 100% line-coverage gate.

Additional Notes

AI-assisted implementation and test execution using Codex. No dependencies or provider changes. This parameterizes the existing import parser; it does not change its pre-existing line-based parsing behavior.

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@cevheri cevheri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the best-shaped PR of the batch, and the hard part of #688 is the part you got right.

Its third item, "leave the RFC 4180 quoting untouched, only the separator becomes a parameter", is in tension with itself: the quoting predicate has to start testing the CHOSEN delimiter, or a semicolon inside a field in a semicolon file breaks the row. You resolved it the right way, by taking , out of NEEDS_QUOTING and adding text.includes(delimiter), and then you pinned the consequence: toCsv([{ amount: "1,5" }], ..., ";") asserts the comma stays BARE. That is the whole point of the issue, the European decimal comma, and it is the assertion most people would have missed. Putting , back into NEEDS_QUOTING turns two cases red, so it holds.

The parse side matches it: "Snow;""quote""" covers a quoted field carrying both the delimiter and an escaped quote, with CRLF, and the headerless column_1 path from #712 still composes. You also kept #712's mapping-retention logic intact while refactoring that handler into updateCsvPreview, which is the kind of thing that usually gets lost in a refactor.

Measured: 339 cases pass across the five affected suites, all twenty checks green, and seven of seven mutants I tried are killed, including hardcoding the comma back into the parse loop, dropping the reset, showing the picker for JSON files, and having updateCsvPreview ignore the delimiter it is handed. Both shells are wired, and _hydrated is the honest way to reach a third parameter in a shell that never had a second one.

One change, and it is the only thing.

NEEDS_QUOTING's docstring now says something false. It reads "The characters RFC 4180 says force a field to be quoted. A field is left bare otherwise" while the delimiter, the single most important character in a CSV, has moved out of the constant and into a separate test beside it. This is the file whose comments are the specification for the format, so please say there that the delimiter is checked separately because it is now a parameter. One or two lines is enough.

@cevheri

cevheri commented Sep 9, 2026

Copy link
Copy Markdown
Member

This is done. Merging shortly.

The docstring says the right thing now, and I checked the thing I would have worried about: the clause you dropped, "a field is left bare otherwise, so a numeric column stays numeric to a spreadsheet", is not lost. PLAIN_NUMBER's own docstring carries that argument in more detail than the sentence did, so the file still explains why the pattern is narrow.

Two lines of comment, nothing else: 197 unit cases still pass, and removing text.includes(delimiter) still turns five red, so the behaviour and its pinning are untouched. typecheck and format clean.

Thanks for the batch. Six of your PRs went through tonight and the pattern across them was that each round got smaller, which is the part that actually costs a maintainer time.

@cevheri
cevheri merged commit 9f0e419 into libredb:main Sep 10, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV import and export both hardcode the comma delimiter

2 participants