Make provenance and evidence traceability first-class for mocktopus (multi-armed mocks llm apps drop-in)

[haasonsaas]

Summary

Carry source, decision, and output provenance through the main workflow so downstream agents can audit and cite it.

This issue was generated from an org-wide EvalOps mining pass on 2026-05-10 07:57 UTC. It combines live GitHub repo signals with a per-repo arXiv search. Treat the research links as grounding for a concrete implementation, not as a request for a literature review.

Repo Evidence

• Repository description: 🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing
• Tree signals: 1 docs files, 1 workflows, 0 proto files, 3 test-like files.
• README.md:98 includes latent-spec language: ### Record Mode (Coming Soon) Proxy and record real API calls for later replay:
• README.md:105 includes latent-spec language: ### Replay Mode (Coming Soon) Replay previously recorded API interactions:
• ROADMAP.md:114 includes latent-spec language: ## Future Ideas (v2.0+) - WebSocket Support: Real-time streaming applications
• tests/test_integration.py:348 includes latent-spec language: # Should work twice for i in range(2):
• tests/test_integration.py:357 includes latent-spec language: # Should not work third time matched, _ = scenario.find_llm(
• tests/test_server.py:109 includes latent-spec language: # First use - should work assert rule.ok_to_use() is True

Research Grounding

Repo axes: memory, evaluation, tooling, desktop

Search keywords: mocktopus, yaml, openai, api, scenario, llm, response, testing, true, https, replay, tests

• arXiv:2509.19209v1 A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent (Olalekan K. Akindele, Bhupesh Kumar Mishra, Kenneth Y. Wertheim), 2025.
• arXiv:2502.06864v1 Knowledge Graph-Guided Retrieval Augmented Generation (Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, Wei Hu), 2025.
• arXiv:2506.21556v3 VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation (Hyeongcheol Park, Jiyoung Seo, MinHyuk Jang, Hogun Park, Ha Dam Baek, Gyusam Chang), 2025.
• arXiv:2510.14271v1 Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generation (Yilun Zheng, Dan Yang, Jie Li, Lin Shang, Lihui Chen, Jiahao Xu), 2025.
• arXiv:2512.20626v2 MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation (Chi-Hsiang Hsiao, Yi-Cheng Wang, Tzung-Sheng Lin, Yi-Ren Yeh, Chu-Song Chen), 2025.
• arXiv:2603.20309v1 BubbleRAG: Evidence-Driven Retrieval-Augmented Generation for Black-Box Knowledge Graphs (Duyi Pan, Tianao Lou, Xin Li, Haoze Song, Yiwen Wu, Mengyi Deng), 2026.
• arXiv:2504.05163v2 Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness (Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Yuan He, Jiaoyan Chen, Steffen Staab), 2025.
• arXiv:2504.08893v1 Knowledge Graph-extended Retrieval Augmented Generation for Question Answering (Jasper Linders, Jakub M. Tomczak), 2025.
• arXiv:2507.16826v1 A Query-Aware Multi-Path Knowledge Graph Fusion Approach for Enhancing Retrieval-Augmented Generation in Large Language Models (Qikai Wei, Huansheng Ning, Chunlong Han, Jianguo Ding), 2025.
• arXiv:2603.05698v2 Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis (Hazem Amamou, Stéphane Gagnon, Alan Davoust, Anderson R. Avila), 2026.

What To Build

• Add stable identifiers for source records, derived decisions, and emitted outputs.
• Thread those identifiers through logs/events/API responses without leaking secrets.
• Provide a query or debug surface that reconstructs the chain for one completed workflow.

Acceptance Criteria

• A short design note names the repo-specific workflow, threat or correctness model, and the research assumptions being adopted.
• A runnable check, fixture, or verifier exercises the new contract in CI or an equivalent local command documented in the repo.
• The implementation emits or stores enough evidence for a downstream agent/operator to cite inputs, decisions, and outputs.
• At least one negative/degraded-mode case is covered so failures are observable rather than silently accepted.
• Documentation links the new behavior to the relevant EvalOps platform primitive or explicitly records why this repo remains standalone.

Notes

• Generated issue 2/5 for evalops/mocktopus by evalops_org_miner.py.
• Before implementation, confirm the sampled latent-spec snippets still match main; this issue intentionally cites exact file paths/lines where the mining pass saw them.