Skip to content

fix(test): make external model fixtures hermetic - #154

Open
Rishabh060105 wants to merge 3 commits into
accordproject:mainfrom
Rishabh060105:Rishabh060105/fix-hermetic-tests
Open

fix(test): make external model fixtures hermetic#154
Rishabh060105 wants to merge 3 commits into
accordproject:mainfrom
Rishabh060105:Rishabh060105/fix-hermetic-tests

Conversation

@Rishabh060105

@Rishabh060105 Rishabh060105 commented May 30, 2026

Copy link
Copy Markdown
Contributor

Closes #153

Summary

This change makes the test suite hermetic for the remaining external-model paths.

It replaces live model downloads in the affected tests with cached @models... fixtures, following the same naming pattern already used in Concerto tests and existing Accord Project archive fixtures.

Changes

  • add cached external Concerto model fixtures under test/models
  • add a shared test helper to load vendored external models into a ModelManager
  • mock external model fetches for the archive-based test path using cached local fixtures
  • update TemplateMarkInterpreter.test.ts to stop calling updateExternalModels() at runtime
  • update TemplateArchiveProcessor.test.ts to resolve external model fetches from local cached fixtures

Test

npm test -- --runInBand --coverage=false

Results:

  • 11 test suites passed
  • 63 tests passed
  • 38 snapshots passed

@Rishabh060105
Rishabh060105 requested a review from a team May 30, 2026 11:29
@Rishabh060105

Rishabh060105 commented May 30, 2026

Copy link
Copy Markdown
Contributor Author

Hi @mttrbrts, the fix follows the Concerto-style cached external model pattern.

What changed:

  • vendored the required external models as cached @models... fixtures under test/models
  • replaced the runtime updateExternalModels() dependency in TemplateMarkInterpreter.test.ts with local cached model loading
  • added a shared helper for offline external model resolution in tests
  • updated TemplateArchiveProcessor.test.ts to resolve external model fetches from cached local fixtures instead of live downloads

I also verified the full suite locally after the change:

npm test -- --runInBand --coverage=false

Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
@Rishabh060105
Rishabh060105 force-pushed the Rishabh060105/fix-hermetic-tests branch from 61df9ad to f675fe7 Compare June 4, 2026 04:57
@github-actions

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open with no activity. Remove the stale label or comment to keep it active. Only items with maintainer engagement are auto-closed.

@github-actions github-actions Bot added the Stale label Jun 20, 2026
@Rishabh060105

Copy link
Copy Markdown
Contributor Author

@github unstale

@github-actions github-actions Bot removed the Stale label Jul 22, 2026
@devanshi00

Copy link
Copy Markdown
Contributor

Hey @Rishabh060105, thanks for the PR!

One suggestion before merge: take a look at how cicero-template-library handles this same problem in test/render.test.mjs — it might be a cleaner pattern than fetch-mocking here.

Instead of intercepting fetch and vendoring copies of the external model files, it forces offline: true at the loader boundary: Template.fromDirectory(path, { offline: true }). That routes cicero-core to mm.validateModelFiles() (a local-only check) instead of mm.updateExternalModels() (the network call), so no HTTP request is ever made in the first place. It goes a step further and globally monkeypatches Template.prototype.validate to always pass offline: true, which protects against any internal validate() call elsewhere in the chain, not just the one call site you know about.

Might be worth considering here instead of (or alongside) the mockExternalModelFetches approach , curious what you think.

@Rishabh060105
Rishabh060105 force-pushed the Rishabh060105/fix-hermetic-tests branch from 4104bd3 to f675fe7 Compare August 3, 2026 07:05
Rishabh Jain added 2 commits August 3, 2026 12:54
Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
@Rishabh060105

Rishabh060105 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Hi @devanshi00, TemplateArchiveProcessor.test.ts now calls Template.fromDirectory(..., { offline: true }) and temporarily patches Template.prototype.validate to preserve offline: true for internal validation calls. The original method is restored afterward to avoid leaking the patch into other tests.
This removes the fetch interception entirely and ensures archive models are validated locally through validateModelFiles() without making HTTP requests.
For tests that create a ModelManager directly, offline: true cannot be used. Instead, the required external model files are loaded locally before the template model. This keeps the tests offline and preserves validation without adding dependencies. Currently, only the money model is needed.
Thank you for pointing it out!
Hi @mttrbrts, please review the pr, thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests are not hermetic: several suites require live model downloads

2 participants