Skip to content

Test integrity: tests that pass without testing #8

Description

@Ugbot

The suite is large and mostly green (770 tests pass in the main module), but three mechanisms let it report green on untested code.

  1. Fixture-missing skips hide the flagship parity proof. The cross-runtime parity tests resolve examples/pipelines/banking.yaml via a CWD-relative path guarded by assumeTrue. Under the documented build invocation the path misses and they silently skip — measured: 3/3 and 5/6 skipped. Re-run with the correct working directory, all 21 parity/replay/CEP tests pass — so the parity claim is real but unverified by default. The same fail-open pattern exists in the Pekko and Clojure parity tests.
  2. H2 stands in for Postgres, which is precisely why the invalid-SQL bug in Correctness bugs found by audit #5 shipped. Needs Testcontainers.
  3. *IT.java files are unrunnable in every profile — no failsafe plugin, and surefires filename filter excludes them before the tag filter applies. Net effect: all Redis/Fluss integration tests have zero executed coverage, and no IT exists at all for Postgres, pgvector, Qdrant or Milvus (~2,300 LOC of untested vector-store drivers).

Also worth a sweep: a set of tests assert nothing meaningful — one re-declares the parser regexes locally instead of importing the class under test, one constructs a function and never invokes it, and the config test never sets an env var despite the class existing to resolve them.


Roadmap epic. Detailed findings, file:line citations and sub-tasks live in the AI-grind tracker (project AGS), which is the source of truth for this work. Local dev — build, test, tracking, profiling — is managed with AI-grind.


Tracked as AGS-8 in the devtools-mcp tracker.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions