DevEval Agents

Read this in other languages: English (this file) · 한국어.

A model-agnostic, multi-agent framework that supports OECD-DAC / KOICA-style evaluation of ODA (Official Development Assistance) projects. It applies a set of multi-agent design principles — role = authority, evidence gates, rules injection, parallel multi-angle review, verification, completion enforcement, and a human gate — to the ODA evaluation domain. Criteria, scales, and rules are grounded in the KOICA Evaluation Guidelines (2024) and Project Evaluation Regulation No. 536 (digests in reference/).

The system is portable Markdown: agent instructions plus a shared knowledge base and one shell hook. The same agents already run on three independent stacks — Claude Code, Codex, and a fully open-weight model — so the tool is not locked to any single vendor.

---

🌍 Relevance to the Sustainable Development Goals

Better, evidence-grounded evaluation of development aid is part of the machinery that makes aid accountable and effective. This tool strengthens that machinery:

• SDG 16 — Peace, Justice and Strong Institutions, esp. target 16.6 ("develop effective, accountable and transparent institutions"). The system improves the quality, consistency, and transparency of ODA project evaluation — with cited evidence, balanced strengths/weaknesses, and a mandatory human gate — reinforcing accountable institutional practice in development cooperation.
• SDG 17 — Partnerships for the Goals, esp. the targets on development effectiveness and capacity. By lowering the effort and raising the consistency of monitoring & evaluation (M&E), it supports the evaluation capacity on which effective, learning-oriented partnerships depend.

Because it evaluates ODA projects across sectors (health, energy, water, education, public administration, …), it indirectly supports the SDGs those projects target by improving the learning-and-accountability feedback loop. It is an enabler for the M&E function, not a direct service-delivery tool.

✅ Digital Public Good readiness

This repository is prepared against the DPG Standard's nine indicators:

#	Indicator	Where
1	SDG relevance	this section, and docs/dpg-application.md
2	Approved open license	LICENSE (MIT) + LICENSE-CONTENT (CC BY 4.0)
3	Clear ownership	MAINTAINERS.md
4	Platform independence	docs/platform-independence.md
5	Documentation	this README + docs/ + English translations in docs/en/
6	Data extraction / portability	open Markdown/plain-text only — PRIVACY.md
7	Privacy & applicable laws	PRIVACY.md (PIPA / GDPR)
8	Standards & best practices	docs/standards.md
9	Do no harm by design	docs/do-no-harm.md

Governance: CONTRIBUTING.md · CODE_OF_CONDUCT.md · SECURITY.md · CHANGELOG.md.

🎛️ System — 12 agents, two evaluation types, three harnesses

KOICA evaluation comes in different types; the system handles two of them distinctly.

① Final Evaluation — rate a project on 6 criteria → A–F grade

An orchestrator injects the KOICA criteria/rules, delegates the criteria to read-only evaluators in parallel, has a verifier check the evidence, sums the scores, and hands a draft grade to a human. Standard 5 criteria (Relevance, Coherence, Effectiveness, Efficiency, Sustainability) sum to 20 points → A–F; CTS/technology-innovation projects add Validity as a 6th criterion.

The evaluation criteria (OECD-DAC, KOICA 2024)

DAC criterion	What it asks	Scored?
Relevance	Does the design fit the partner country's / beneficiaries' real needs & priorities?	✅
Coherence	Does it complement & harmonise with other interventions — internal (ROK gov't, other KOICA projects) and external (other donors, partner gov't), avoiding duplication?	✅
Effectiveness	Did it achieve (or is it expected to achieve) its objectives & outputs, including for vulnerable groups?	✅
Efficiency	Were results delivered economically and on time relative to inputs?	✅
Impact	Did it produce (or is it likely to produce) long-term, transformative effects?	➖ ex-post only
Sustainability	Are the financial, institutional & social capacities in place for benefits to last after close-out?	✅
Validity	(CTS / tech-innovation projects only) technical validity — a non-standard add-on	⭐ CTS only

• Composite score = the 5 scored criteria (all but Impact), each 1–4 points = 20 max → A–F. Impact is an ex-post criterion, excluded from the final-evaluation composite; CTS projects add Validity and are graded on the 6-criteria average.
• 4-point scale — 1 clear negative effect · 2 partial shortfall · 3 largely achieved as planned · 4 fully achieved + beyond expectations.
• Grades — ≥18 A (highly successful) · 16–18 B · 14–16 C (successful) · 12–14 D · 10–12 E (partially successful) · <10 F (unsatisfactory).

Source: reference/KOICA-평가지침-2024-다이제스트.md (§1–2). The framework is the DAC six, but the final-evaluation composite is scored on five (Impact excluded).

② Impact Evaluation — causal-method review → project triage label (no grade)

A different type entirely (causal effect via PSM/DiD/RCT, no A-F grade). It is reviewed against six proposed operating principles and ten DevEval operational questions; planning-stage evaluability uses three elements and seven items. Adequate / Conditional / Inadequate is a project triage label, not an official KOICA/KIEP verdict. The 6-criteria frame is deliberately not applied here.

Agent catalog (12)

Role	Agent	Access
Final-eval DAC criteria (6)	dac-{relevance,coherence,effectiveness,efficiency,sustainability,impact}-evaluator — Impact is ex-post, excluded from the 20-pt composite	read
CTS Validity add-on (CTS only)	cts-validity-evaluator	read
Evidence verification	quality-verifier	read
Report composition	report-composer	write
Narrative verification (hallucination/consistency)	narrative-verifier	read
Report quality inspection (24-item / A–D)	report-quality-inspector	read
Impact-evaluation review (6 proposed principles / 10 operational questions)	impact-evaluation-reviewer	read

Plus a completion engine (hooks/boulder.sh, a Stop hook) that drives long/multi-project evaluations to completion, with stagnation and attempt-cap guards.

Design principles → this system

Design principle	Implementation
Parallel multi-angle	six criteria evaluated in parallel, each from its own angle
Role = authority	evaluators/verifiers are read-only; only report-composer writes
Evidence gate	no evidence → no grade / no text (fabrication prohibited)
Distrust completion claims	quality-verifier / narrative-verifier check evidence & consistency
Completion enforcement	Stop hook with stagnation / cap guards
Rules injection	skills inject the KOICA 2024 guidelines + Regulation No. 536 (reference/ digests) into delegation prompts
Human gate (public-sector, new)	AI cannot finalize a grade — that is a human's job

▶️ Quick start

How it works, in three steps — ① prepare the material to evaluate (project plan, PDM, completion report, …) as Markdown / plain text → ② start a harness and ask for an evaluation, pointing it at the file → ③ the system returns per-criterion scores + evidence → verification → a draft composite grade. A human (the evaluation officer) sets the final grade — the AI stops at an evidence-backed draft (the human gate).

Three ways to ask:

Ask it this	You get	Handled by
evaluate this project against the DAC criteria	6-criteria scores + a draft A–F grade	final-eval team
review this impact-evaluation report	causal-method review → project triage label + source status	impact-evaluation-reviewer
inspect the quality of this evaluation report	24-item meta-review → quality grade A–D	report-quality-inspector

New here? Run the bundled fictional sample samples/sample-evaluation-report.md — some result indicators are deliberately left blank, so you can watch the "no evidence → no grade" gate fire.

Install as a plugin

Install the plugin once, then use it from the folder that contains your evaluation material rather than from inside this repository.

Codex (terminal):

codex plugin marketplace add amnotyoung/dev-eval-agents --ref main
codex plugin add deveval@deveval-agents

Start a new Codex task after installation. Codex reviews bundled hooks separately; inspect and trust the completion hook with /hooks if you want it enabled.

Claude Code (inside an interactive session):

/plugin marketplace add amnotyoung/dev-eval-agents
/plugin install deveval@deveval-agents
/reload-plugins

Then invoke the same four workflows with the syntax for your host:

Workflow	Codex	Claude Code	What it does
Evaluate	$deveval:evaluate	/deveval:evaluate	5–6 independent criteria → composite score + draft grade
Quality review	$deveval:quality-review	/deveval:quality-review	24-item / 100-point / A–D report-quality inspection
Impact review	$deveval:impact-review	/deveval:impact-review	6 proposed principles / 10 operational questions
Write report	$deveval:write-report	/deveval:write-report	draft → numeric check → narrative verification → human

Your working files (.omo/eval-plan.md, .omo/draft-report*.md) stay in your folder; the plugin directory is read-only knowledge. Claude Code exposes the bundled deveval-consistency-check command on PATH; the shared skills use the same checker by absolute path on hosts that do not expose plugin bin/ entries.

To try the Claude plugin without installing it (or to develop it):

git clone https://github.com/amnotyoung/dev-eval-agents
claude --plugin-dir ./dev-eval-agents

To run the cloned repository directly in Codex without installing the plugin, use the AGENTS.md fallback:

cd dev-eval-agents
codex exec "samples/sample-evaluation-report.md 이 사업을 DAC 기준으로 평가해줘"

Open-weight model (no proprietary API — Ollama + open weights):

ollama pull qwen2.5:14b
python3 scripts/open_runner.py --out docs/open-model-demo-output.md

Or reproduce it on free Google Colab: notebooks/open-model-demo.ipynb.

🔌 Optional companion — ODA Intelligence plugin

DevEval's evidence gate says no evidence → no grade. The optional ODA Intelligence plugin (same maintainer) widens what counts as checkable evidence: 30 read-only tools over one public MCP gateway — country indicators and other donors' activities (IATI, World Bank, …) for relevance/coherence, the Korean ODA project map for duplication checks, KOICA regulation full text, and verify_citation, which catches report citations to regulation articles that do not exist.

/plugin marketplace add amnotyoung/oda-intelligence-plugin
/plugin install oda-intelligence@oda-intelligence-plugin

Strictly optional — without the connector every workflow runs unchanged and the output simply notes that external context was not augmented (platform independence is unaffected). With it, the same evidence discipline applies: status tools first, a missing source is not a zero, and gateway evidence augments — never replaces — the project documents. Details: docs/oda-intelligence-integration.md.

✅ Validation

Records of the system actually running and agreeing with real KOICA evaluations: docs/validation-log.md — 5 end-to-end runs (Claude Code ×3, Codex ×1, open-weight ×1) and 4 real-report comparisons (Cambodia: grade match). ⚠️ Self-validation of a learning/research project (small sample); expert cross-validation is ongoing.

🔌 Platform independence

The mandatory dependency is a capable LLM agent-harness — a category, not a product. The same agents run on Claude Code, Codex, and an open-weight model (Qwen2.5, Apache-2.0) with no change to the core product. Full evidence: docs/platform-independence.md.

📜 License

Dual-licensed with DPGA-approved open licenses:

• Software (shell hook, config, runner scripts, notebooks) — MIT, see LICENSE.
• Documentation & content (Markdown agents, reference/ digests, templates, samples, docs) — CC BY 4.0, see LICENSE-CONTENT.

The reference/ digests are the project's own descriptions of publicly documented KOICA/KIEP evaluation practice, cited to their sources; original PDF/HWP documents are not redistributed. See MAINTAINERS.md.

📚 reference/ (official norms + specialist proposal + methods)

Original PDFs/HWP are excluded for copyright (.gitignore); only the project's own-word digests are kept.

The official normative layer governs criteria, scales, and obligations: the 2024 KOICA guidance, Regulation No. 536, and quality-review guideline v2. A separate specialist proposal layer digests the KIEP 2025 policy study's proposed Impact Evaluation guidelines; it is not presented as an enacted rule or binding verdict table. The methods layer adds three selectively loaded modules: evaluation design (front-end analysis, ToC, questions, design matrix, validity), data and analysis (measurement, sampling, bias, mixed methods, triangulation), and management/reporting/ethics (TOR, QA, recommendations, independence).

The methods modules are the project's own synthesis of the World Bank's The Road to Results (2009; KOICA Korean translation, 2016). Because that source predates DAC Coherence and current grading, it never defines current criteria, scores, or grades. Precedence is official norms > specialist proposal > methods.

For complex evaluations, templates/evaluation-design-matrix-template.md connects questions, criteria, sources, samples, analysis, triangulation, limits, and the claim-evidence register in one working artifact.

🗺️ Roadmap

Slices 1–8 are complete (see CHANGELOG.md): first evaluator on KOICA 2024 → parallel 6-criteria team + CTS validity → completion engine → report quality inspector → report composition + narrative verifier → impact-evaluation review + Regulation No. 536 → quality inspector v2 → multi-harness (Codex).

📌 Attribution & status

Evaluation criteria and rules are grounded in official KOICA materials (digests only, originals excluded). An independent, unofficial learning/research project — not affiliated with or endorsed by KOICA. Design-lineage acknowledgment, maintainer & ownership: MAINTAINERS.md. Formerly named oh-my-oda-agent (the repository was renamed; old links redirect).