Skip to content

Plan and wire 10TB scale operating layer #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SonAIengine merged 5 commits into from
Jul 3, 2026
Merged

Plan and wire 10TB scale operating layer #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
297fcec
Plan 10TB scale operating layer
SonAIengine Jul 2, 2026
4d48c1b
Wire in-process index router into search
SonAIengine Jul 2, 2026
b4c2ad9
Add OpenSearch candidate provider
SonAIengine Jul 2, 2026
c6d9274
Add Qdrant candidate provider
SonAIengine Jul 3, 2026
2b1db24
Persist ingestion jobs and index health reports
SonAIengine Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
405 changes: 405 additions & 0 deletions docs/PLAN-v0.31-10tb-scale-operating-layer.md
View file
Open in desktop

Large diffs are not rendered by default.

91 changes: 91 additions & 0 deletions examples/ablation/diagnostics/agent_loop_20260703_000851.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Agent Loop Retrieval Benchmark — Synaptic

- Run at: 2026-07-03 00:08:51 KST
- Dataset path: tests/benchmark/data/msmarco_passage_full.json
- SQLite DB path: tests/benchmark/data/msmarco_full.db
- Subset: 50
- Corpus limit: 8841823
- LLM base URL: https://api.deepseek.com/v1
- Model: deepseek-v4-flash
- Max turns: 5
- Sufficiency gate: yes
- Force first tool: yes
- Incremental JSONL: examples/ablation/diagnostics/agent_loop_deepseek_v4_flash_50_post_scale_plan.jsonl
- SQLite FTS AND-first threshold: 20
- SQLite FTS lexical rerank pool: 500

This measures LLM-planned exploration. The agent can change follow-up queries and tool choices based on evidence from earlier turns. The main metric is document reach, not ranked MRR, because the agent loop returns a cumulative evidence set.

## Summary

- Reach: 31/50 (0.620)
- Mean turns: 4.18
- Mean tool calls: 6.18
- Mean first relevant turn: 1.16
- Mean first relevant tool calls: 1.32
- Mean elapsed: 57.2s
- P50/P90 elapsed: 57.7s / 74.3s
- Mean prompt tokens: 21487
- Mean completion tokens: 1122
- Mean unique tools: 2.38
- Mean unique search targets: 5.84
- Mean query rewrites: 4.48
- Queries with >1 tool type: 48/50
- Queries with query rewrites: 46/50
- Duplicate tool calls: 0
- Empty tool calls: 6

## Per Query

| QID | Reach | Turns | Calls | Tools | Targets | Rewrites | First Rel Turn | First Rel Calls | Found Relevant | Elapsed | Query |
|-----|:-----:|------:|------:|------:|--------:|---------:|---------------:|----------------:|----------------|--------:|-------|
| 300674 | yes | 3 | 3 | 2 | 2 | 2 | 1 | 1 | 7067032 | 37.3s | how many years did william bradford serve as governor of plymouth colony? |
| 125705 | no | 5 | 7 | 2 | 7 | 7 | - | - | - | 54.6s | define preventive |
| 94798 | yes | 4 | 6 | 2 | 5 | 4 | 1 | 1 | 7067181 | 56.6s | color overlay photoshop |
| 9083 | yes | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 7067274 | 30.2s | ____________________ is considered the father of modern medicine. |
| 174249 | no | 5 | 4 | 3 | 4 | 4 | - | - | - | 48.1s | does xpress bet charge to deposit money in your account |
| 320792 | yes | 5 | 7 | 3 | 7 | 7 | 2 | 3 | 7067677 | 70.7s | how much is a cost to run disneyland |
| 1090270 | yes | 3 | 3 | 2 | 3 | 0 | 1 | 1 | 7067796 | 39.1s | botulinum definition |
| 1101279 | no | 5 | 12 | 3 | 12 | 12 | - | - | - | 78.4s | do physicians pay for insurance from their salaries? |
| 201376 | yes | 4 | 6 | 3 | 6 | 1 | 1 | 1 | 7068066 | 56.4s | here there be dragons comic |
| 54544 | yes | 5 | 8 | 3 | 8 | 2 | 1 | 1 | 7068203 | 63.2s | blood diseases that are sexually transmitted |
| 118457 | no | 3 | 2 | 2 | 2 | 1 | - | - | - | 37.0s | define bona fides |
| 178627 | no | 5 | 10 | 3 | 10 | 10 | - | - | - | 77.4s | effects of detox juice cleanse |
| 1101278 | yes | 4 | 6 | 2 | 6 | 1 | 1 | 1 | 7068907 | 55.6s | do prince harry and william have last names |
| 68095 | yes | 4 | 6 | 2 | 6 | 5 | 1 | 1 | 7069266 | 54.8s | can hives be a sign of pregnancy |
| 87892 | yes | 5 | 9 | 3 | 8 | 7 | 1 | 1 | 7069601 | 65.1s | causes of petechial hemorrhage |
| 257309 | no | 4 | 5 | 2 | 5 | 5 | - | - | - | 59.2s | how long does it take to get your bsrn if you already have a bachelors degree |
| 1090242 | yes | 5 | 12 | 3 | 12 | 11 | 1 | 1 | 7070556 | 76.1s | symptoms of ptsd in vietnam veterans |
| 211691 | no | 5 | 5 | 3 | 5 | 3 | - | - | - | 53.1s | how coffee works quote |
| 165002 | yes | 3 | 3 | 2 | 3 | 3 | 1 | 1 | 7070877 | 49.4s | does contraction of the ciliary muscles shorten the lens |
| 1101276 | yes | 2 | 1 | 1 | 1 | 0 | 1 | 1 | 7070950 | 31.9s | do spiders eat other animals |
| 264827 | yes | 3 | 2 | 2 | 2 | 2 | 1 | 1 | 7071066 | 35.6s | how long is the flight from chicago to cairo |
| 342285 | no | 5 | 11 | 2 | 11 | 1 | - | - | - | 65.4s | how titanic facts |
| 372586 | no | 4 | 9 | 2 | 8 | 7 | - | - | - | 64.6s | how to play blu ray discs |
| 89786 | yes | 3 | 3 | 3 | 3 | 2 | 1 | 1 | 7071501 | 44.8s | central city definition |
| 118448 | no | 3 | 3 | 2 | 2 | 1 | - | - | - | 43.7s | define body muscular endurance |
| 92542 | yes | 3 | 4 | 2 | 4 | 3 | 1 | 1 | 7072003 | 49.0s | circulation money definition |
| 206117 | yes | 5 | 11 | 3 | 8 | 8 | 1 | 1 | 7072155, 7072160 | 64.0s | hotels in thornton co |
| 141472 | yes | 4 | 4 | 2 | 4 | 0 | 1 | 1 | 7072290 | 48.3s | derriere definition |
| 293992 | no | 5 | 6 | 3 | 5 | 5 | - | - | - | 57.5s | how many product lines does coca cola have |
| 196232 | no | 5 | 8 | 3 | 8 | 8 | - | - | - | 75.5s | government does do |
| 352818 | no | 5 | 7 | 3 | 6 | 5 | - | - | - | 60.1s | how to cook string beans |
| 45924 | no | 5 | 8 | 3 | 7 | 6 | - | - | - | 71.5s | average temperatures las vegas by month |
| 208145 | yes | 5 | 9 | 3 | 9 | 9 | 1 | 1 | 7072843 | 82.0s | how bicycle tire tubes are sized |
| 79891 | no | 4 | 5 | 2 | 5 | 5 | - | - | - | 57.8s | can you substitute chocolate chips for semi-sweet |
| 208494 | yes | 4 | 5 | 3 | 5 | 5 | 1 | 1 | 7073272 | 57.1s | how big do newfypoo's get |
| 319564 | yes | 4 | 6 | 2 | 6 | 6 | 1 | 1 | 7073381 | 63.8s | how much fiber is in carrots |
| 155234 | yes | 3 | 4 | 2 | 4 | 3 | 1 | 1 | 502713 | 57.1s | do bigger tires affect gas mileage |
| 14151 | yes | 5 | 8 | 2 | 8 | 7 | 1 | 1 | 7073772 | 69.5s | age requirements for name change |
| 67802 | yes | 3 | 6 | 2 | 6 | 5 | 1 | 1 | 7074071 | 54.4s | can green tea cause stomach problems |
| 1090184 | yes | 5 | 4 | 2 | 4 | 3 | 1 | 1 | 7074235 | 42.9s | synonym of subordinate |
| 323382 | no | 5 | 10 | 2 | 10 | 10 | - | - | - | 67.5s | how much is the stamp to send a card |
| 323998 | yes | 3 | 3 | 2 | 2 | 2 | 1 | 1 | 7074377 | 41.4s | how much magnesium in kidney beans |
| 91711 | yes | 5 | 7 | 2 | 4 | 3 | 1 | 1 | 1956185 | 63.9s | child psychiatrist salary 2016 |
| 125898 | yes | 5 | 8 | 3 | 6 | 6 | 3 | 6 | 7074710 | 62.3s | define prosthetic device |
| 289812 | no | 3 | 3 | 2 | 3 | 1 | - | - | - | 34.1s | how many mm is a nickel coin |
| 333486 | yes | 5 | 6 | 3 | 6 | 6 | 3 | 4 | 7075218 | 63.9s | how old do you have to be to get a job in idaho |
| 1090171 | yes | 5 | 8 | 2 | 8 | 0 | 1 | 1 | 7075317 | 58.6s | synonyms for the word discipline |
| 73257 | no | 5 | 8 | 3 | 8 | 3 | - | - | - | 70.6s | can seizure meds cause low sodium? |
| 1090170 | no | 5 | 11 | 3 | 11 | 11 | - | - | - | 74.3s | synonyms for the word, decline |
| 237373 | yes | 4 | 6 | 2 | 6 | 5 | 1 | 1 | 7075449 | 64.2s | how is soil created from rocks |
37 changes: 37 additions & 0 deletions examples/ablation/diagnostics/agent_loop_deepseek_auto_rewrites_20260702.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,43 @@ Current deterministic `deep_search` also showed that several misses were caused
| 155234 | do bigger tires affect gas mileage | tire size factors influence gas mileage; tire width versus gas mileage | 1 | reach=yes, first relevant turn 1 / call 1 |
| 208145 | how bicycle tire tubes are sized | bicycle tire tube size sidewall ETRTO metric imperial; bicycle tire sidewall tube size printed raised numbers | 1 | reach=yes, first relevant turn 1 / call 1 |

## Current Main 50-Query Baseline

After merging the process cleanup, blood-borne transmission rewrite, and answer-shaped rewrite hints, the current `main` DeepSeek 50-query run reached `31/50`, up from the original `23/50` and the earlier post-rewrite `25/50`.

| Metric | Original 50-query run | Current main run |
| --- | ---: | ---: |
| Reach | 23/50 | 31/50 |
| Mean turns | 4.14 | 4.18 |
| Mean tool calls | 5.78 | 6.18 |
| Mean first relevant turn | 1.70 | 1.16 |
| Mean first relevant tool calls | 2.22 | 1.32 |
| Empty tool calls | 11 | 6 |
| Duplicate tool calls | 0 | 0 |

Net new hits versus the original 50-query run:

- `1101278` - do prince harry and william have last names
- `125898` - define prosthetic device
- `14151` - age requirements for name change
- `155234` - do bigger tires affect gas mileage
- `208145` - how bicycle tire tubes are sized
- `237373` - how is soil created from rocks
- `319564` - how much fiber is in carrots
- `320792` - how much is a cost to run disneyland
- `54544` - blood diseases that are sexually transmitted
- `91711` - child psychiatrist salary 2016

Regressed versus the original 50-query run:

- `178627` - effects of detox juice cleanse
- `45924` - average temperatures las vegas by month

Artifacts:

- `examples/ablation/diagnostics/agent_loop_20260703_000851.md`
- `examples/ablation/diagnostics/agent_loop_deepseek_v4_flash_50_post_scale_plan.jsonl`

## Interpretation

Prompt-visible hints alone were not enough: DeepSeek often generated nearby but non-gold rewrites. Running the deterministic rewrite hints inside `deep_search` removes that planning variance for cheap, bounded patterns such as dropping noisy numeric years and rewriting "created from" process questions into answer-shaped phrases.
Loading
Loading