feat(registry): configurable driver registries via TOML config by zeroshade · Pull Request #360 · columnar-tech/dbc

zeroshade · 2026-04-17T16:16:47Z

Summary

Adds support for configuring driver registries via TOML config files, allowing users to add custom registries at both the global (~/.config/columnar/dbc/config.toml) and project (dbc.toml) level.

What's New

Global registry config (~/.config/columnar/dbc/config.toml): loaded on every command via ConfigureRegistries() called in main()
Project registry config (dbc.toml): [[registries]] section respected by sync and add commands
Priority order: project → global → built-in defaults (first-match-wins, URL-deduplicated)
replace_defaults = true: opt-in flag to suppress built-in registries entirely
DBC_BASE_URL env var: still overrides everything (unchanged behavior)
Backward compatible: existing dbc.toml files without [[registries]] work unchanged

Config Format

Global (~/.config/columnar/dbc/config.toml):

[[registries]]
url = "https://my-registry.example.com"
name = "my-registry"

# replace_defaults = true  # omit built-in registries

Project (dbc.toml):

[[registries]]
url = "https://project-registry.example.com"
name = "project"

[drivers]
[drivers.my-driver]
version = ">=1.0.0"

Implementation Notes

URL validation requires http/https scheme and non-empty host
defaultRegistries is snapshotted eagerly in init() (after drivers.go init runs); env-sensitivity documented in comments
add.go decodes dbc.toml once (single file open) for both registry config and driver list
remove intentionally omits registry wiring — it never queries a registry

Add RegistryEntry and GlobalConfig structs with TOML tags, and loadGlobalConfig() function that parses config.toml from a directory, validates URLs with url.Parse (not mustParseURL), and returns nil,nil for missing files. 7 subtests cover valid configs, missing file, invalid URL, empty URL, and empty registries.

…faults

- Add SetProjectRegistries() to dbc package for project-level registry merging - Snapshot defaultRegistries in ConfigureRegistries for proper re-merge ordering - Inject SetProjectRegistries in sync driversListMsg handler before getDriverRegistry - Pre-load dbc.toml in add Init() to set project registries before getDriverRegistry - Add tests: TestSyncWithProjectRegistries, TestSyncWithProjectRegistriesBackwardCompat - Add tests: TestAddWithProjectRegistries, TestAddWithProjectRegistriesBackwardCompat, TestAddWithInvalidProjectRegistryURL

…leanup - Add u.Host == "" check after url.Parse in loadGlobalConfig and SetProjectRegistries to catch hostless URLs like "my-registry" - Add TestSetProjectRegistries with 6 subtests covering merge precedence, replace_defaults, DBC_BASE_URL no-op, and error cases - Update TestConfigureRegistries subtests to save/restore defaultRegistries and globalConfig alongside registries - Use defer preF.Close() in add.go instead of manual close in two branches

…t improvements

…tries snapshot

…ment toRegistry invariant

…ional SetProjectRegistries, integration test

…egration test baseline

…ad toRegistry check, strengthen test

…y state propagation

…rop test

…h test invariant

…est cleanup patterns

…hen wiring tests, fix test isolation

… wiring tests, fix origReg copy

…assumption in add/sync

…location in ConfigureRegistries

…origReg/origDefault in test

amoeba

I did a quick code review over the new pieces and left some comments. Even if we don't have an immediate use case for this, I think having it in dbc is useful.

Some other more general thoughts:

Would we want CLI integration for this? I think it would be useful for dbc to be able to do all the work to show/add/remove registries from the project and user config.
I didn't see any tests for how dbc install will work with this. Presumably dbc install will respect the new fields in the user config.
Do we need to define a default priority order for registries to handle name conflicts? We have a lot of the same concerns as conda does with channels. I'm wishing for more tests in this PR for the interaction between the default, user, and project registries around name conflicts.

amoeba · 2026-04-27T17:39:16Z

+
+func SetProjectRegistries(entries []RegistryEntry, replaceDefaults *bool) error {
+	if os.Getenv("DBC_BASE_URL") != "" {
+		return nil


I think we should warn the user here.

amoeba · 2026-04-27T17:44:43Z

+		return nil
+	}
+	if replaceDefaults != nil && *replaceDefaults && len(entries) == 0 {
+		return fmt.Errorf("replace_defaults = true requires at least one [[registries]] entry; omit replace_defaults or add a registry entry")


I feel like this error would be better surfaced earlier, such as when we initially parse config.toml or dbc.toml. How this is now, I think this would error like this,

error configuring project registries: replace_defaults = true requires at least one [[registries]] entry; omit replace_defaults or add a registry entry

As a user, it's unclear what I should do to fix it. Which file is my problem in?

If we validated this earlier, I bet we'd have enough context to know which file has the issue and we could tell them.

amoeba · 2026-04-27T17:47:35Z

+	}
+	for _, e := range entries {
+		if e.URL == "" {
+			return fmt.Errorf("registry entry has empty url")


Are these checks needed or is this the best place to handle this? I see loadGlobalConfig already does validation and that seems like a better place. i.e., I much prefer being able to know that any []RegistryEntry I might get is always valid so I'd rather error before creating a RegistryEntry.

amoeba · 2026-04-27T17:55:06Z

 	})
 }

+func boolPtr(b bool) *bool { return &b }


is this the best place for this def?

amoeba · 2026-04-27T17:57:19Z

 	}

+	if configDir, err := internal.GetUserConfigPath(); err != nil {
+		fmt.Fprintf(os.Stderr, "warning: failed to locate config directory: %v\n", err)


Do we really want to emit a warning? Won't this warn if the folder isn't found? Seems like we should be okay with that.

The warning when the registry config fails to load once found is fine.

amoeba · 2026-04-27T18:00:09Z

+
+# Sisyphus workflow artifacts (plans, evidence, notepads)
+.sisyphus/


is this just a tool you use? can we revert this?

amoeba · 2026-04-27T18:05:54Z

+	if configDir, err := internal.GetUserConfigPath(); err != nil {
+		fmt.Fprintf(os.Stderr, "warning: failed to locate config directory: %v\n", err)
+	} else if err := dbc.ConfigureRegistries(configDir); err != nil {
+		fmt.Fprintf(os.Stderr, "warning: failed to load registry config: %v\n", err)


I don't think the error here includes the file with the issue. Whenever we show an error about a TOML file, we need to show the user the path to it.

amoeba · 2026-04-27T18:18:36Z

+		if u.Scheme != "http" && u.Scheme != "https" {
+			return Registry{}, false


Is the validation in here even needed? In the parent call, we already error when a RegistryEntry is invalid.

zeroshade temporarily deployed to snapshot April 17, 2026 16:16 — with GitHub Actions Inactive

zeroshade requested a review from amoeba April 17, 2026 16:17

zeroshade force-pushed the feat/configurable-registries branch from 5a8bf36 to 5ac1ddd Compare April 17, 2026 16:19

zeroshade temporarily deployed to snapshot April 17, 2026 16:19 — with GitHub Actions Inactive

zeroshade requested review from eitsupi, esadek and ianmcook April 17, 2026 18:34

zeroshade force-pushed the feat/configurable-registries branch from 5ac1ddd to 0813c82 Compare April 17, 2026 22:14

zeroshade added 10 commits April 22, 2026 17:33

feat(driver-list): add optional [[registries]] section to DriversList

20bb910

feat(registry): add registry merge logic with priority and replace_de…

6a42588

…faults

feat(registry): wire ConfigureRegistries into CLI init path

c9acbf9

fix(registry): add scheme validation, fix add.go double-read, and tes…

95ba6d9

…t improvements

chore: add .sisyphus/ to .gitignore

1092f88

fix(registry): snapshot defaultRegistries in init() instead of lazily

c95ef03

docs(registry): document DBC_BASE_URL env-sensitivity of defaultRegis…

1834c57

…tries snapshot

zeroshade force-pushed the feat/configurable-registries branch from 0813c82 to 79e6f5c Compare April 22, 2026 21:33

zeroshade temporarily deployed to snapshot April 22, 2026 21:33 — with GitHub Actions Inactive

zeroshade added 10 commits April 22, 2026 17:38

fix(registry): always reset globalConfig in ConfigureRegistries, docu…

cb1ee19

…ment toRegistry invariant

fix(registry): guard global replace_defaults, omitempty tag, uncondit…

02652f9

…ional SetProjectRegistries, integration test

fix(registry): remove over-aggressive replace_defaults guard, fix int…

b766d6f

…egration test baseline

fix(registry): reset registries on nil cfg, boolPtr helper, remove de…

16e0482

…ad toRegistry check, strengthen test

fix(registry): restore toRegistry scheme check, add nil-cfg reset test

a4f3f98

fix(registry): fix integration test snapshot ordering to prevent dirt…

4a2bd1a

…y state propagation

fix(registry): reset registry state in TearDownTest, add ftp scheme d…

db662fa

…rop test

fix(registry): defensive copy for registries reset, clarify error-pat…

5969e2f

…h test invariant

fix(registry): capture origReg separately in integration test cleanup

896a483

fix(registry): resolve relative URLs in AllVersions, add tests, fix t…

4f37276

…est cleanup patterns

zeroshade added 6 commits April 22, 2026 17:39

fix(registry): reorder DriversList fields, add GetRegistries, strengt…

a1f897c

…hen wiring tests, fix test isolation

fix(registry): defensive copy in GetRegistries, guard DBC_BASE_URL in…

10f6ff1

… wiring tests, fix origReg copy

fix(registry): defensive copy in saveAndRestore helper

97dbdbb

fix(registry): defensive copy in test setup, document single-process …

090f628

…assumption in add/sync

fix(registry): defensive copy for origDefault, eliminate throwaway al…

3a14489

…location in ConfigureRegistries

fix(registry): defensive copies for base in SetProjectRegistries and …

40e5ad2

…origReg/origDefault in test

zeroshade force-pushed the feat/configurable-registries branch from 79e6f5c to 40e5ad2 Compare April 22, 2026 21:39

zeroshade temporarily deployed to snapshot April 22, 2026 21:39 — with GitHub Actions Inactive

zeroshade requested a review from kentkwu April 26, 2026 18:21

amoeba reviewed Apr 27, 2026

View reviewed changes

Merge branch 'main' into feat/configurable-registries

a41a2ad

zeroshade temporarily deployed to snapshot April 28, 2026 17:51 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(registry): configurable driver registries via TOML config#360

feat(registry): configurable driver registries via TOML config#360
zeroshade wants to merge 27 commits intomainfrom
feat/configurable-registries

zeroshade commented Apr 17, 2026

Uh oh!

amoeba left a comment

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

amoeba Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		# Sisyphus workflow artifacts (plans, evidence, notepads)
		.sisyphus/

		if u.Scheme != "http" && u.Scheme != "https" {
		return Registry{}, false

Conversation

zeroshade commented Apr 17, 2026

Summary

What's New

Config Format

Implementation Notes

Uh oh!

amoeba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants