From b7304af1483270712dd41cec6835815d7acadd85 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 18:38:33 +0000 Subject: [PATCH 01/13] docs(challenge): experimental caveat, honesty review, lives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify that CodeDNA is still experimental per agent×language (issues/PRs welcome), require truthful metrics with optional Meet verification, and note public lives on submitted tests. AI-Agent: composer-2.5 AI-Provider: cursor AI-Session: s_20260908_001 AI-Visited: docs/challenge.md,docs/challenge.it.md,docs/challenge.html,challenge/SUBMISSION_TEMPLATE.md,challenge/SUBMISSION_TEMPLATE.it.md,README.md,README-it.md AI-Message: Honesty + Meet review + lives added; experimental agent/language note mirrored to HTML Co-authored-by: Fabrizio Corpora --- .codedna | 16 ++++++++-------- README-it.md | 2 +- README.md | 2 +- challenge/SUBMISSION_TEMPLATE.it.md | 2 ++ challenge/SUBMISSION_TEMPLATE.md | 2 ++ docs/challenge.html | 19 +++++++++++++++++-- docs/challenge.it.md | 26 ++++++++++++++++++++++++-- docs/challenge.md | 28 +++++++++++++++++++++++++--- 8 files changed, 80 insertions(+), 17 deletions(-) diff --git a/.codedna b/.codedna index 7daf86b..ee8416a 100644 --- a/.codedna +++ b/.codedna @@ -30,14 +30,6 @@ packages: cross_cutting_patterns: {} agent_sessions: - - agent: "gpt-5" - provider: "openai" - date: "2026-08-21" - session_id: "s_20260821_gitbook_docs" - task: "Redesign documentation like GitBook" - changed: ["docs/install.html","docs/site.css",".codedna"] - visited: ["docs/install.html","docs/site.css",".codedna"] - message: "Reworked the documentation shell with a GitBook-inspired three-column layout, searchable hierarchical sidebar, keyboard shortcut, breadcrumb, page metadata, sticky outline, pagination, feedback, and responsive spacing. HTML/CSS checks, Ruff, and 284 tests pass." - agent: "gpt-5" provider: "openai" date: "2026-08-21" @@ -70,3 +62,11 @@ agent_sessions: changed: ["codedna_tool/languages/php.py","codedna_tool/languages/base.py","codedna_tool/languages/vbnet.py","codedna_tool/languages/__init__.py","codedna_tool/cli.py","integrations/.roorules","integrations/install.sh","integrations/README.md","docs/languages.md","README.md","README-it.md","CHANGELOG.md","tests/test_issues_4_6_7.py","tests/test_docs.py"] visited: ["codedna_tool/languages/php.py","codedna_tool/languages/base.py","codedna_tool/languages/csharp.py","codedna_tool/languages/__init__.py","codedna_tool/cli.py","integrations/.clinerules"] message: "Validated #4/#6/#7 still open. Fixed PHP mixed-HTML injection + full-file has_codedna_header; added VbNetAdapter; added Roo --tools roo. 306 tests pass." + - agent: "composer-2.5" + provider: "cursor" + date: "2026-09-08" + session_id: "s_20260908_001" + task: "Challenge honesty + experimental notes" + changed: ["docs/challenge.md","docs/challenge.it.md","docs/challenge.html","challenge/SUBMISSION_TEMPLATE.md","challenge/SUBMISSION_TEMPLATE.it.md","README.md","README-it.md"] + visited: ["docs/challenge.md","docs/challenge.it.md","docs/challenge.html","challenge/SUBMISSION_TEMPLATE.md","challenge/SUBMISSION_TEMPLATE.it.md","README.md","README-it.md"] + message: "Documented experimental agent×language caveat + honesty/Meet review + public lives in challenge rules." diff --git a/README-it.md b/README-it.md index f7bc644..5c36030 100644 --- a/README-it.md +++ b/README-it.md @@ -48,7 +48,7 @@ Nessuna infrastruttura. Nessuna pipeline di retrieval. Nessuna memoria esterna. ## CodeDNA Challenge — €200 -Valutazione pubblica sul **tuo** progetto (non SWE-bench). Almeno 10 task con/senza CodeDNA, stack L0/L1/L2 a pari livello, PR con metriche — anche se vanno contro CodeDNA. +Valutazione pubblica sul **tuo** progetto (non SWE-bench). Almeno 10 task con/senza CodeDNA, stack L0/L1/L2 a pari livello, PR con metriche — anche se vanno contro CodeDNA. Affermazioni oneste: il team può chiedere una review Meet; live sui test quando possibile. - Bacheca: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Regolamento: [`docs/challenge.it.md`](docs/challenge.it.md) (IT) · [`docs/challenge.md`](docs/challenge.md) (EN) diff --git a/README.md b/README.md index 1642ae5..7312378 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ No infrastructure. No retrieval pipeline. No external memory. The code carries i ## CodeDNA Challenge — €200 -Public **own-project** evaluation (not SWE-bench). ≥10 tasks with/without CodeDNA, fair L0/L1/L2 stack parity, metrics PR — even if results go against CodeDNA. +Public **own-project** evaluation (not SWE-bench). ≥10 tasks with/without CodeDNA, fair L0/L1/L2 stack parity, metrics PR — even if results go against CodeDNA. Claims must be honest: the team may request a Meet review; live sessions on submitted tests when possible. - Board: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Rules: [`docs/challenge.md`](docs/challenge.md) (EN) · [`docs/challenge.it.md`](docs/challenge.it.md) (IT) diff --git a/challenge/SUBMISSION_TEMPLATE.it.md b/challenge/SUBMISSION_TEMPLATE.it.md index 2e8c30b..29b7ad0 100644 --- a/challenge/SUBMISSION_TEMPLATE.it.md +++ b/challenge/SUBMISSION_TEMPLATE.it.md @@ -53,3 +53,5 @@ - [ ] Nessun secret / sorgente proprietaria in questa PR - [ ] Solo log redatti - [ ] Modalità + parity di stack dichiarate nel JSON (`mode`, `stack`) +- [ ] Metriche e narrativa sono veritiere (non inventate); posso ripercorrerle in una call di review (Meet / simile) se richiesto +- [ ] Capisco che claim inventati = squalifica diff --git a/challenge/SUBMISSION_TEMPLATE.md b/challenge/SUBMISSION_TEMPLATE.md index 71b2333..4474782 100644 --- a/challenge/SUBMISSION_TEMPLATE.md +++ b/challenge/SUBMISSION_TEMPLATE.md @@ -53,3 +53,5 @@ - [ ] No secrets / proprietary source in this PR - [ ] Redacted logs only - [ ] Mode + stack parity declared in JSON (`mode`, `stack`) +- [ ] Metrics and narrative are truthful (not invented); I can walk through them on a review call (Meet / similar) if asked +- [ ] I understand fabricated claims = disqualification diff --git a/docs/challenge.html b/docs/challenge.html index c9e57a9..60eba1e 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -212,6 +212,13 @@

CodeDNA Challenge
€200

Full rules (EN) Regolamento (IT) +

+ Experimental. Results can vary by AI coding agent and language. + If annotations or the CLI misbehave on your stack, open a + GitHub issue or a fix + pull request so we can harden support. + That feedback is part of the challenge. +

@@ -276,6 +283,10 @@

Open the PR

Graph / structural memory — parity required.

+

+ Honesty. Do not invent results. Fabrizio or the review team may request a live review (e.g. Google Meet) to verify your runs. + When possible we will also host public lives discussing submitted tests. +

@@ -333,6 +344,7 @@

Public board

h1:'CodeDNA Challenge
€200', lead:'Test CodeDNA on your own project. Open a metrics PR to enroll — no separate signup. We update this board when submissions are merged.', ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', + experimentalNote:'Experimental. Results can vary by AI coding agent and language. If annotations or the CLI misbehave on your stack, open a GitHub issue or a fix pull request so we can harden support. That feedback is part of the challenge.', statusTitle:'Challenge status', statusLead:'Maintainer-updated board. Source file: docs/challenge-board.json.', metaPrize:'Prize', metaPrizeSub:'paid after review', @@ -346,11 +358,12 @@

Public board

l0:'In-source headers — the layer under test.', l1:'LLM wiki, skill packs, instruction files — parity required.', l2:'Graph / structural memory — parity required.', + honestyNote:'Honesty. Do not invent results. Fabrizio or the review team may request a live review (e.g. Google Meet) to verify your runs. When possible we will also host public lives discussing submitted tests.', boardTitle:'Public board', boardLead:'Language + framework, project size and file count are required so comparisons stay honest.', emptyTitle:'No submissions on the board yet', emptyBody:'When a metrics PR is reviewed and merged, we add a row here by editing challenge-board.json.', - boardNote:'This board is static and curated. Opening a PR enrolls you automatically; appearing here means maintainers accepted the submission format.', + boardNote:'This board is static and curated. Opening a PR enrolls you automatically; appearing here means maintainers accepted the submission format. Agent × language gaps are expected — file issues or fix PRs.', thHandle:'Entrant', thStack:'Language / framework', thSize:'Size', thFiles:'Files', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', unlock:'min {n} to unlock', windowFallback:'dates TBD' @@ -361,6 +374,7 @@

Public board

h1:'CodeDNA Challenge
€200', lead:'Prova CodeDNA sul tuo progetto. Apri una PR con le metriche per iscriverti — niente signup separato. Aggiorniamo questa bacheca quando le submission vengono mergiate.', ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', + experimentalNote:'Sperimentale. I risultati possono variare per agente AI di coding e linguaggio. Se annotazioni o CLI si comportano male sul tuo stack, apri una issue o una pull request di fix così possiamo rafforzare il supporto. Quel feedback fa parte della challenge.', statusTitle:'Stato challenge', statusLead:'Bacheca aggiornata dai maintainer. File sorgente: docs/challenge-board.json.', metaPrize:'Premio', metaPrizeSub:'pagato dopo review', @@ -374,11 +388,12 @@

Public board

l0:'Header in-source — layer sotto test.', l1:'Wiki LLM, skill pack, instruction file — parity obbligatoria.', l2:'Memoria graph / strutturale — parity obbligatoria.', + honestyNote:'Onestà. Non inventare i risultati. Fabrizio o il team di review possono chiedere una review live (es. Google Meet) per verificare le run. Quando possibile faremo anche live pubbliche sui test inviati.', boardTitle:'Bacheca pubblica', boardLead:'Linguaggio + framework, dimensione progetto e numero file sono obbligatori per confronti onesti.', emptyTitle:'Ancora nessuna submission in bacheca', emptyBody:'Quando una PR metrics viene accettata e mergiata, aggiungiamo una riga modificando challenge-board.json.', - boardNote:'Bacheca statica e curata. Aprire una PR ti iscrive automaticamente; comparire qui significa che i maintainer hanno accettato il formato.', + boardNote:'Bacheca statica e curata. Aprire una PR ti iscrive automaticamente; comparire qui significa che i maintainer hanno accettato il formato. Gap agente × linguaggio sono attesi — apri issue o PR di fix.', thHandle:'Partecipante', thStack:'Linguaggio / framework', thSize:'Size', thFiles:'File', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', unlock:'min {n} per sbloccare', windowFallback:'date TBD' diff --git a/docs/challenge.it.md b/docs/challenge.it.md index 49789a7..4dd57e4 100644 --- a/docs/challenge.it.md +++ b/docs/challenge.it.md @@ -12,7 +12,7 @@ Questa challenge pone una sola domanda: > CodeDNA aiuta (o no) quando fai sviluppo reale assistito da AI — e possiamo misurarlo in modo onesto? -I bug trovati durante la challenge sono benvenuti: apri una GitHub issue e li correggiamo upstream. +**Sperimentale.** La qualità di CodeDNA dipende ancora dall’**agente AI di coding** e dal **linguaggio / framework** che usi. Alcune combinazioni funzionano meglio di altre. Se install, annotation, hook o refresh non si comportano bene sul tuo stack, **apri una GitHub issue o una PR di fix** — fa parte della challenge, e sistemiamo upstream a partire dai tuoi report. --- @@ -36,15 +36,25 @@ I bug trovati durante la challenge sono benvenuti: apri una GitHub issue e li co | Submission valide minime | **5** per sbloccare il premio (obiettivo stretch: 10) | | Se ci sono meno di 5 PR valide | I risultati si pubblicano comunque; il premio **non** viene assegnato (o passa all’edizione successiva) | | Vincitore | Scelto da Fabrizio Corpora + team di review (non voto community) | +| Onestà | Le affermazioni devono essere vere e supportate da evidenze — metriche o narrative inventate = squalifica | +| Verifica | Fabrizio o il team di review possono chiedere una review live (Google Meet / simile) per ripercorrere run, log e setup del repo | +| Live | Quando possibile ospiteremo live pubbliche sui test inviati (con consenso del partecipante dove serve) | ### Cosa valutiamo (in ordine) -1. **Onestà del protocollo** — stack equo, modalità dichiarata, note riproducibili +1. **Onestà del protocollo** — stack equo, modalità dichiarata, note riproducibili; **niente risultati inventati** 2. **Qualità dei task** — mix reale facili/medi/difficili su una codebase vera 3. **Evidenze** — metriche + breve narrativa; a favore **o** contro CodeDNA va bene 4. **Bug report** — issue actionable aperte upstream contano positivamente 5. **Chiarezza** — un altro engineer può ripetere il confronto +### Onestà, call di review e live + +- **Non puoi inventare** esiti pass/fail, tempi, dettagli di stack o claim narrativi. Se non riesci a riprodurre un numero, marca il task come inconclusive e dillo. +- Fabrizio Corpora e/o il team di review **possono chiedere una video call** (es. Google Meet) per verificare la submission: screen-share del progetto, riesecuzione di un task campione, walkthrough di `metrics.json` / note. +- Rifiutare senza motivo valido una richiesta ragionevole di verifica può annullare l’eleggibilità al premio per quella entry. +- Quando possibile faremo anche **live pubbliche** sui test della challenge (metodologia, sorprese, gap agente × linguaggio). Partecipare a una live è opzionale, salvo finalisti chiamati in verifica. + --- ## Timeline (da compilare prima della pubblicazione) @@ -194,6 +204,16 @@ codedna init . --no-llm # oppure con LLM per le rules: Solo domande (opzionale): GitHub Discussions / Discord — il vecchio template “entry issue” non è obbligatorio. +### Se qualcosa si rompe sul tuo agente o linguaggio + +CodeDNA è ancora sperimentale tra agenti e linguaggi. Percorso consigliato: + +1. Riproduci una volta (agente + linguaggio/framework + comando). +2. Apri una **issue** (bug) o una **PR** con fix minimo / test di regressione. +3. Continua la challenge se puoi; annota l’incidente in `metrics.json` → `bugs_reported` e in `notes.md`. + +Tooling rotto su un certo stack **non** ti squalifica — segnalarlo è utile. + --- ## Cosa non è questa challenge @@ -209,6 +229,8 @@ Solo domande (opzionale): GitHub Discussions / Discord — il vecchio template - Issue / bug: GitHub Issues - Q&A challenge: GitHub Discussions (Announcements / Q&A) - Community: Discord (vedi badge in README) +- Call di verifica: Google Meet (o simile) su richiesta dei maintainer +- Live pubbliche: annunciate su Discussions / Discord quando programmate - Maintainer: Fabrizio Corpora --- diff --git a/docs/challenge.md b/docs/challenge.md index 36dd9c3..65a5c9d 100644 --- a/docs/challenge.md +++ b/docs/challenge.md @@ -12,7 +12,7 @@ This challenge asks one question: > Does CodeDNA help (or not) when you do real AI-assisted development work — and can we measure it honestly? -Bugs found during the challenge are welcome: open a GitHub issue and we will fix them upstream. +**Experimental.** CodeDNA quality still depends on the **AI coding agent** and the **language / framework** you use. Some combinations work better than others. If install, annotation, hooks, or refresh misbehave on your stack, **open a GitHub issue or a fix PR** — that is part of the challenge, and we will harden upstream from your reports. --- @@ -36,14 +36,24 @@ Bugs found during the challenge are welcome: open a GitHub issue and we will fix | Minimum valid submissions | **5** to unlock the prize (stretch goal: 10) | | If fewer than 5 valid PRs | Challenge still publishes results; prize is **not** awarded (or rolled to the next edition) | | Winner | Selected by Fabrizio Corpora + review team (not community vote) | +| Honesty | Claims must be true and evidence-backed — fabricated metrics or narratives = disqualification | +| Verification | Fabrizio or the review team may request a live review (Google Meet / similar) to walk through runs, logs, and repo setup | +| Lives | When possible we will host public live sessions discussing submitted tests (with entrant consent where needed) | ### What we score (in order) -1. **Protocol honesty** — fair stack, declared mode, reproducible notes +1. **Protocol honesty** — fair stack, declared mode, reproducible notes; **no invented results** 2. **Task quality** — real easy/medium/hard mix on a real codebase 3. **Evidence** — metrics + short narrative; pro **or** against CodeDNA is fine 4. **Bug reports** — actionable issues filed upstream count positively -5. **Clarity** — another engineer can re-run your comparison +5. **Clarity** — another engineer can re-run your comparison + +### Honesty, review calls, and lives + +- You **must not invent** pass/fail outcomes, timings, stack details, or narrative claims. If you cannot reproduce a number, mark the task inconclusive and say so. +- Fabrizio Corpora and/or the review team **may ask for a video call** (e.g. Google Meet) to verify your submission: screen-share the project, rerun a sample task, and walk through `metrics.json` / notes. +- Refusing a reasonable verification request without a good reason can void prize eligibility for that entry. +- When feasible we will also run **public live sessions** covering interesting challenge runs (methodology, surprises, agent × language gaps). Participation in a live is optional unless you are a finalist asked for verification. --- @@ -194,6 +204,16 @@ codedna init . --no-llm # or with an LLM for rules: Questions only (optional): GitHub Discussions / Discord — the old “entry issue” template is not required. +### If something breaks on your agent or language + +CodeDNA is still experimental across agents and languages. Prefer this path: + +1. Reproduce once (agent + language/framework + command). +2. Open an **issue** (bug) or a **PR** with a minimal fix / regression test. +3. Continue the challenge if you can; note the incident in `metrics.json` → `bugs_reported` and in `notes.md`. + +Broken tooling on a given stack does **not** disqualify you — reporting it is valuable. + --- ## What this challenge is not @@ -209,6 +229,8 @@ Questions only (optional): GitHub Discussions / Discord — the old “entry iss - Issues / bugs: GitHub Issues - Challenge Q&A: GitHub Discussions (Announcements / Q&A) - Community: Discord (see README badge) +- Verification calls: Google Meet (or similar) when requested by maintainers +- Public lives: announced on Discussions / Discord when scheduled - Maintainer: Fabrizio Corpora --- From 60924d8a59fa81a180b695d4c3a0e2eca9be5f05 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 20:34:37 +0000 Subject: [PATCH 02/13] docs(challenge): real projects, install path, Meet review, live board MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Require declared tech stack and CodeDNA install steps, reject toy/throwaway sites (≥25 source files), mandate same-task control vs CodeDNA runs, require bug reports when experimental agent/language gaps appear, and clarify that the public ranking updates as valid PRs arrive with optional Meet presentation. AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: docs/challenge.md,docs/challenge.it.md,docs/challenge.html,challenge/metrics.schema.json,challenge/metrics.example.json,challenge/SUBMISSION_TEMPLATE.md AI-Message: Tightened eligibility; dates still TBD; draft #18 content included on this branch Co-authored-by: Fabrizio Corpora --- CHANGELOG.md | 2 +- README-it.md | 2 +- README.md | 2 +- challenge/README.md | 8 +- challenge/SUBMISSION_TEMPLATE.it.md | 41 +++++++---- challenge/SUBMISSION_TEMPLATE.md | 41 +++++++---- challenge/metrics.example.json | 12 ++- challenge/metrics.schema.json | 54 +++++++++++++- docs/challenge-board.README.md | 10 ++- docs/challenge.html | 70 +++++++++--------- docs/challenge.it.md | 109 +++++++++++++++++++--------- docs/challenge.md | 109 +++++++++++++++++++--------- 12 files changed, 315 insertions(+), 145 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9aa3478..22eec31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to CodeDNA will be documented in this file. ### Added -- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files), docs site page + static board (`docs/challenge.html`, `docs/challenge-board.json`). PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. +- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files ≥25 + `install` + `bugs_reported`), real-project bar (no toy sites), same-task control vs CodeDNA, Meet presentation verification, docs site page + live public board updated as valid PRs arrive. PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. ### Added diff --git a/README-it.md b/README-it.md index 5c36030..826c4c4 100644 --- a/README-it.md +++ b/README-it.md @@ -48,7 +48,7 @@ Nessuna infrastruttura. Nessuna pipeline di retrieval. Nessuna memoria esterna. ## CodeDNA Challenge — €200 -Valutazione pubblica sul **tuo** progetto (non SWE-bench). Almeno 10 task con/senza CodeDNA, stack L0/L1/L2 a pari livello, PR con metriche — anche se vanno contro CodeDNA. Affermazioni oneste: il team può chiedere una review Meet; live sui test quando possibile. +Valutazione pubblica sul **tuo** progetto (non SWE-bench). Progetto reale funzionante (≥25 file), stack tecnologico + install CodeDNA dichiarati, ≥10 **stessi** task con/senza CodeDNA, parity L0/L1/L2, PR con metriche — anche se vanno contro CodeDNA. I bug vanno segnalati (sperimentale tra agenti). Possibile presentazione Meet; la bacheca pubblica si aggiorna quando arrivano PR valide. - Bacheca: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Regolamento: [`docs/challenge.it.md`](docs/challenge.it.md) (IT) · [`docs/challenge.md`](docs/challenge.md) (EN) diff --git a/README.md b/README.md index 7312378..801989e 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ No infrastructure. No retrieval pipeline. No external memory. The code carries i ## CodeDNA Challenge — €200 -Public **own-project** evaluation (not SWE-bench). ≥10 tasks with/without CodeDNA, fair L0/L1/L2 stack parity, metrics PR — even if results go against CodeDNA. Claims must be honest: the team may request a Meet review; live sessions on submitted tests when possible. +Public **own-project** evaluation (not SWE-bench). Real working project (≥25 files), declared tech stack + CodeDNA install, ≥10 **same** tasks with/without CodeDNA, fair L0/L1/L2 parity, metrics PR — even if results go against CodeDNA. Bugs must be reported (experimental across agents). Meet presentation may be required; public board updates as valid PRs arrive. - Board: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Rules: [`docs/challenge.md`](docs/challenge.md) (EN) · [`docs/challenge.it.md`](docs/challenge.it.md) (IT) diff --git a/challenge/README.md b/challenge/README.md index 09a75e3..3101fb8 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -4,19 +4,21 @@ ```text challenge// - metrics.json # REQUIRED + metrics.json # REQUIRED (stack + install + same-task control/codedna results) README.md # optional notes.md # optional ``` -No official signup. Public board (maintainer-updated): [`docs/challenge.html`](../docs/challenge.html) · data [`docs/challenge-board.json`](../docs/challenge-board.json) +No official signup. Public board / ranking (updated as valid PRs arrive): [`docs/challenge.html`](../docs/challenge.html) · data [`docs/challenge-board.json`](../docs/challenge-board.json) **Metrics format** - Example: [`metrics.example.json`](./metrics.example.json) - Schema: [`metrics.schema.json`](./metrics.schema.json) -Required project fields: `languages`, `frameworks`, `approx_source_files`, `size_band` (`S`/`M`/`L`/`XL`). +Required: `languages`, `frameworks`, `approx_source_files` (≥25), `size_band` (`S`/`M`/`L`/`XL`), `install` (agent + steps), `bugs_reported` (array; empty OK). + +Real working projects only — toy sites / hello-world demos are rejected. Same tasks must be run with and without CodeDNA. Meet presentation may be required. **Rules** diff --git a/challenge/SUBMISSION_TEMPLATE.it.md b/challenge/SUBMISSION_TEMPLATE.it.md index 29b7ad0..82adfbf 100644 --- a/challenge/SUBMISSION_TEMPLATE.it.md +++ b/challenge/SUBMISSION_TEMPLATE.it.md @@ -7,12 +7,29 @@ ### Partecipante - Handle: -- Issue di iscrizione collegata: - Modalità challenge: - [ ] Parity (stessi L1/L2 su Control e CodeDNA) - [ ] Dichiarata `codedna-only` vs stack superiore -### Stack sotto test +### Stack tecnologico (obbligatorio) + +- Linguaggio/i: +- Framework (obbligatorio — non solo il linguaggio): +- Agente / modello: +- File sorgente approx. (≥ **25**): +- Size band (`S`/`M`/`L`/`XL`): +- Note stack (DB, monorepo, infra — opzionale): +- URL pubblico (opzionale) / “privato — solo metriche”: +- [ ] È un **progetto reale e funzionante** (non sito giocattolo / hello-world / demo throwaway) + +### Install CodeDNA (obbligatorio) + +- Agente usato con CodeDNA: +- Passi esatti di install / init (incolla i comandi): +- Valore di `codedna install --tools` (se usato): +- Install + annotation ok? sì / no — note: + +### Stack sotto test (L0/L1/L2) | Livello | Tool / file | Presente in Control? | Presente in CodeDNA? | |---|---|---|---| @@ -20,22 +37,16 @@ | L1 | | | | | L2 | | | | -### Progetto - -- Linguaggio/i: -- Framework (obbligatorio — non solo il linguaggio): -- File sorgente approx.: -- Size band (`S`/`M`/`L`/`XL`): -- URL pubblico (opzionale) / “privato — solo metriche”: - ### Metrics JSON (obbligatorio) - [ ] Presente `challenge//metrics.json` - [ ] Partito da [`metrics.example.json`](./metrics.example.json) / conforme a [`metrics.schema.json`](./metrics.schema.json) - [ ] `schema_version` = `"1.0"` -- [ ] Compilati `project.languages` + `project.frameworks` + `approx_source_files` + `size_band` +- [ ] Compilati `project.languages` + `project.frameworks` + `approx_source_files` (≥25) + `size_band` +- [ ] Compilati `install.agent` + `install.steps` +- [ ] Presente `bugs_reported` (array vuoto OK se nessuno) - [ ] ≥10 task con mix `difficulty` (easy ≥3, medium ≥3, hard ≥2) -- [ ] Ogni task ha risultati `control` e `codedna` +- [ ] Gli **stessi task** hanno risultati `control` e `codedna` - [ ] `summary.favors` impostato (`codedna` | `control` | `tie` | `inconclusive`) - [ ] I risultati possono favorire **oppure** sfavorire CodeDNA (onestà OK) @@ -47,11 +58,13 @@ ### Bug trovati in CodeDNA - Link (oppure “nessuno”) — elencarli anche in `metrics.json` → `bugs_reported`: +- [ ] Se qualcosa si è rotto su agente/linguaggio, ho aperto issue o PR di fix ### Checklist - [ ] Nessun secret / sorgente proprietaria in questa PR - [ ] Solo log redatti - [ ] Modalità + parity di stack dichiarate nel JSON (`mode`, `stack`) -- [ ] Metriche e narrativa sono veritiere (non inventate); posso ripercorrerle in una call di review (Meet / simile) se richiesto -- [ ] Capisco che claim inventati = squalifica +- [ ] Metriche e narrativa sono veritiere (non inventate) +- [ ] Posso presentare progetto e processo di test in una call di review (Meet / simile) se richiesto +- [ ] Capisco che claim inventati o progetti finti = squalifica diff --git a/challenge/SUBMISSION_TEMPLATE.md b/challenge/SUBMISSION_TEMPLATE.md index 4474782..6d1e8b9 100644 --- a/challenge/SUBMISSION_TEMPLATE.md +++ b/challenge/SUBMISSION_TEMPLATE.md @@ -7,12 +7,29 @@ ### Entrant - Handle: -- Linked entry issue: - Challenge mode: - [ ] Parity (same L1/L2 on Control and CodeDNA) - [ ] Declared `codedna-only` vs higher stack -### Stack under test +### Tech stack (required) + +- Language(s): +- Framework(s) (required — not language alone): +- Agent / model: +- Approx. source files (≥ **25**): +- Size band (`S`/`M`/`L`/`XL`): +- Tech stack notes (DB, monorepo, infra — optional): +- Public URL (optional) / “private — metrics only”: +- [ ] This is a **real working project** (not a toy site / hello-world / throwaway demo) + +### CodeDNA install (required) + +- Agent used with CodeDNA: +- Exact install / init steps (paste commands): +- `codedna install --tools` value (if any): +- Did install + annotation work? yes / no — notes: + +### Stack under test (L0/L1/L2) | Level | Tools / files | Present in Control? | Present in CodeDNA? | |---|---|---|---| @@ -20,22 +37,16 @@ | L1 | | | | | L2 | | | | -### Project - -- Language(s): -- Framework(s) (required — not language alone): -- Approx. source files: -- Size band (`S`/`M`/`L`/`XL`): -- Public URL (optional) / “private — metrics only”: - ### Metrics JSON (required) - [ ] `challenge//metrics.json` present - [ ] Copied from [`metrics.example.json`](./metrics.example.json) / matches [`metrics.schema.json`](./metrics.schema.json) - [ ] `schema_version` = `"1.0"` -- [ ] `project.languages` + `project.frameworks` + `approx_source_files` + `size_band` filled +- [ ] `project.languages` + `project.frameworks` + `approx_source_files` (≥25) + `size_band` filled +- [ ] `install.agent` + `install.steps` filled +- [ ] `bugs_reported` present (empty array OK if none) - [ ] ≥10 tasks with `difficulty` mix (easy ≥3, medium ≥3, hard ≥2) -- [ ] Each task has both `control` and `codedna` results +- [ ] **Same tasks** each have both `control` and `codedna` results - [ ] `summary.favors` set (`codedna` | `control` | `tie` | `inconclusive`) - [ ] Results may favor **or** disfavor CodeDNA (honesty OK) @@ -47,11 +58,13 @@ ### Bugs found in CodeDNA - Links (or “none”) — also list them in `metrics.json` → `bugs_reported`: +- [ ] If something broke on my agent/language, I filed an issue or fix PR ### Checklist - [ ] No secrets / proprietary source in this PR - [ ] Redacted logs only - [ ] Mode + stack parity declared in JSON (`mode`, `stack`) -- [ ] Metrics and narrative are truthful (not invented); I can walk through them on a review call (Meet / similar) if asked -- [ ] I understand fabricated claims = disqualification +- [ ] Metrics and narrative are truthful (not invented) +- [ ] I can present the project and test process on a review call (Meet / similar) if asked +- [ ] I understand fabricated claims or fake projects = disqualification diff --git a/challenge/metrics.example.json b/challenge/metrics.example.json index fb036ff..38a1e81 100644 --- a/challenge/metrics.example.json +++ b/challenge/metrics.example.json @@ -44,7 +44,8 @@ "NestJS", "FastAPI" ], - "size_band": "M" + "size_band": "M", + "tech_stack_notes": "NestJS API + FastAPI workers, Postgres, monorepo apps/api + apps/worker" }, "success_definition": "Task passes if the agent produced a correct change without the human rewriting the core logic.", "tasks": [ @@ -313,5 +314,12 @@ "favors": "codedna", "one_line_verdict": "CodeDNA reduced wrong-file mistakes on hard tasks; still honest about one medium task needing a steer." }, - "bugs_reported": [] + "bugs_reported": [], + "install": { + "agent": "cursor", + "tools_flag": "cursor", + "steps": "pipx install git+https://github.com/Larens94/codedna.git\ncodedna install --path . --tools cursor\ncodedna init . --no-llm", + "worked": true, + "notes": "Install OK on Cursor; no adapter issues on this NestJS/FastAPI stack." + } } diff --git a/challenge/metrics.schema.json b/challenge/metrics.schema.json index 61e5692..a1eaf0e 100644 --- a/challenge/metrics.schema.json +++ b/challenge/metrics.schema.json @@ -10,8 +10,10 @@ "mode", "stack", "project", + "install", "tasks", - "summary" + "summary", + "bugs_reported" ], "properties": { "schema_version": { @@ -154,8 +156,8 @@ }, "approx_source_files": { "type": "integer", - "minimum": 1, - "description": "Approximate number of source files in the project under test." + "minimum": 25, + "description": "Approximate number of source files. Minimum 25. Toy landing pages / hello-world demos are rejected; must be a real working project (small is OK)." }, "visibility": { "type": "string", @@ -193,6 +195,13 @@ "XL" ], "description": "S=<50 files, M=50-199, L=200-999, XL=1000+" + }, + "tech_stack_notes": { + "type": [ + "string", + "null" + ], + "description": "Optional free-text stack notes (DB, infra, monorepo layout, etc.)." } } }, @@ -334,7 +343,44 @@ } } }, - "default": [] + "default": [], + "description": "Bugs found in CodeDNA during the challenge. Empty array if none. If you hit agent/language issues you MUST file issues or fix PRs and list them here." + }, + "install": { + "type": "object", + "additionalProperties": false, + "required": [ + "agent", + "steps" + ], + "description": "How CodeDNA was installed for this run (required \u2014 publish your install path).", + "properties": { + "agent": { + "type": "string", + "minLength": 1, + "description": "AI coding agent used (e.g. cursor, claude-code, opencode, codex)." + }, + "tools_flag": { + "type": [ + "string", + "null" + ], + "description": "Value passed to `codedna install --tools`, if any." + }, + "steps": { + "type": "string", + "minLength": 1, + "description": "Exact install/init commands and steps you ran." + }, + "worked": { + "type": "boolean", + "description": "Whether install + annotation completed successfully on this agent/stack." + }, + "notes": { + "type": "string", + "description": "Problems, workarounds, agent-specific gaps." + } + } } }, "$defs": { diff --git a/docs/challenge-board.README.md b/docs/challenge-board.README.md index a3478b5..87a6d1e 100644 --- a/docs/challenge-board.README.md +++ b/docs/challenge-board.README.md @@ -1,11 +1,12 @@ # Updating the public challenge board -The live board is [`docs/challenge.html`](../docs/challenge.html). -It reads [`docs/challenge-board.json`](../docs/challenge-board.json) (static — no backend). +The live board / ranking is [`docs/challenge.html`](../docs/challenge.html). +It reads [`docs/challenge-board.json`](../docs/challenge-board.json) (static — no backend). +**Update it as soon as valid PRs arrive** so the ranking stays publicly visible. ## When a metrics PR is accepted -1. Merge the PR (`challenge//metrics.json`). +1. Merge the PR (`challenge//metrics.json`) after checking: real project (≥25 files), stack declared, `install` present, same-task control/codedna pairs, bugs filed when relevant. 2. Append a row to `docs/challenge-board.json` → `submissions` using fields from that JSON: ```json @@ -29,4 +30,5 @@ It reads [`docs/challenge-board.json`](../docs/challenge-board.json) (static — 3. Bump `updated_at` (ISO date). 4. Commit + push `main` so GitHub Pages refreshes. -Enrollment does **not** require an issue — the metrics PR is enough. +Enrollment does **not** require an issue — the metrics PR is enough. +Optional: invite the entrant to a Meet presentation before or after the board row if verification is needed. diff --git a/docs/challenge.html b/docs/challenge.html index 60eba1e..d943874 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -206,17 +206,17 @@
Public evaluation · €200

CodeDNA Challenge
€200

-

Test CodeDNA on your own project. Open a metrics PR to enroll — no separate signup. We update this board when submissions are merged.

+

Test CodeDNA on your own real project. Declare stack + install. Same tasks with vs without CodeDNA. Open a metrics PR to enroll — the public board updates as valid PRs arrive.

- Experimental. Results can vary by AI coding agent and language. - If annotations or the CLI misbehave on your stack, open a + Experimental. Results can vary by AI coding agent and language (Claude Code, OpenCode, Cursor, Codex, …). + Publish how you installed CodeDNA. If annotations or the CLI misbehave, open a GitHub issue or a fix - pull request so we can harden support. + pull request. That feedback is part of the challenge.

@@ -251,22 +251,22 @@

Challenge status

How it works

-

Own project · ≥10 tasks · with vs without CodeDNA · fair L0/L1/L2 stack · metrics.json PR.

+

Real working project (≥25 files) · declare languages/frameworks/agent · ≥10 same tasks with vs without CodeDNA · publish install · report bugs · metrics.json PR.

01
-

Run on your repo

-

Declare languages, frameworks, file count and size band. Easy / medium / hard mix.

+

Real project + stack

+

No toy sites. Declare languages, frameworks, file count, size band, and how you installed CodeDNA.

02
-

Keep the stack fair

-

If you use wiki, skills or Graphify, both conditions get them — unless you declare codedna-only mode.

+

Same tasks, fair stack

+

Run the same ≥10 tasks with and without CodeDNA. Keep L1/L2 parity unless you declare codedna-only mode.

03
-

Open the PR

-

Add challenge/<handle>/metrics.json. That PR enrolls you. We merge and update this board.

+

PR → live board

+

Open a metrics PR to enroll. The public ranking updates as valid PRs arrive. We may ask for a Meet presentation.

@@ -284,14 +284,14 @@

Open the PR

- Honesty. Do not invent results. Fabrizio or the review team may request a live review (e.g. Google Meet) to verify your runs. + Honesty. Do not invent results or fake projects. Fabrizio or the review team may require a conference call + presentation to verify your project and test process. When possible we will also host public lives discussing submitted tests.

Public board

-

Language + framework, project size and file count are required so comparisons stay honest.

+

Language + framework + install path and project size are required. Ranking updates as valid PRs arrive and stays public.

updated —
@@ -342,28 +342,28 @@

Public board

navHome:'Home', navChallenge:'Challenge', navInstall:'Install', eyebrow:'Public evaluation · €200', h1:'CodeDNA Challenge
€200', - lead:'Test CodeDNA on your own project. Open a metrics PR to enroll — no separate signup. We update this board when submissions are merged.', + lead:'Test CodeDNA on your own real project. Declare stack + install. Same tasks with vs without CodeDNA. Open a metrics PR to enroll — the public board updates as valid PRs arrive.', ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', - experimentalNote:'Experimental. Results can vary by AI coding agent and language. If annotations or the CLI misbehave on your stack, open a GitHub issue or a fix pull request so we can harden support. That feedback is part of the challenge.', + experimentalNote:'Experimental. Results can vary by AI coding agent and language (Claude Code, OpenCode, Cursor, Codex, …). Publish how you installed CodeDNA. If annotations or the CLI misbehave, open a GitHub issue or a fix pull request. That feedback is part of the challenge.', statusTitle:'Challenge status', - statusLead:'Maintainer-updated board. Source file: docs/challenge-board.json.', + statusLead:'Public ranking updated as valid PRs arrive. Source file: docs/challenge-board.json.', metaPrize:'Prize', metaPrizeSub:'paid after review', metaSubs:'Submissions', metaStatus:'Status', metaEnroll:'Enrollment', metaEnrollVal:'PR = in', metaEnrollSub:'no official signup', howTitle:'How it works', - howLead:'Own project · ≥10 tasks · with vs without CodeDNA · fair L0/L1/L2 stack · metrics.json PR.', - s1t:'Run on your repo', s1p:'Declare languages, frameworks, file count and size band. Easy / medium / hard mix.', - s2t:'Keep the stack fair', s2p:'If you use wiki, skills or Graphify, both conditions get them — unless you declare codedna-only mode.', - s3t:'Open the PR', s3p:'Add challenge//metrics.json. That PR enrolls you. We merge and update this board.', + howLead:'Real working project (≥25 files) · declare languages/frameworks/agent · ≥10 same tasks with vs without CodeDNA · publish install · report bugs · metrics.json PR.', + s1t:'Real project + stack', s1p:'No toy sites. Declare languages, frameworks, file count, size band, and how you installed CodeDNA.', + s2t:'Same tasks, fair stack', s2p:'Run the same ≥10 tasks with and without CodeDNA. Keep L1/L2 parity unless you declare codedna-only mode.', + s3t:'PR → live board', s3p:'Open a metrics PR to enroll. The public ranking updates as valid PRs arrive. We may ask for a Meet presentation.', l0:'In-source headers — the layer under test.', l1:'LLM wiki, skill packs, instruction files — parity required.', l2:'Graph / structural memory — parity required.', - honestyNote:'Honesty. Do not invent results. Fabrizio or the review team may request a live review (e.g. Google Meet) to verify your runs. When possible we will also host public lives discussing submitted tests.', + honestyNote:'Honesty. Do not invent results or fake projects. Fabrizio or the review team may require a conference call + presentation to verify your project and test process. When possible we will also host public lives discussing submitted tests.', boardTitle:'Public board', - boardLead:'Language + framework, project size and file count are required so comparisons stay honest.', + boardLead:'Language + framework + install path and project size are required. Ranking updates as valid PRs arrive and stays public.', emptyTitle:'No submissions on the board yet', - emptyBody:'When a metrics PR is reviewed and merged, we add a row here by editing challenge-board.json.', - boardNote:'This board is static and curated. Opening a PR enrolls you automatically; appearing here means maintainers accepted the submission format. Agent × language gaps are expected — file issues or fix PRs.', + emptyBody:'As soon as a valid metrics PR is reviewed, we add a row here by editing challenge-board.json.', + boardNote:'Live ranking, curated by maintainers. Opening a PR enrolls you; appearing here means the submission format and stack declaration were accepted. Agent × language gaps are expected — file issues or fix PRs.', thHandle:'Entrant', thStack:'Language / framework', thSize:'Size', thFiles:'Files', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', unlock:'min {n} to unlock', windowFallback:'dates TBD' @@ -372,28 +372,28 @@

Public board

navHome:'Home', navChallenge:'Challenge', navInstall:'Install', eyebrow:'Valutazione pubblica · €200', h1:'CodeDNA Challenge
€200', - lead:'Prova CodeDNA sul tuo progetto. Apri una PR con le metriche per iscriverti — niente signup separato. Aggiorniamo questa bacheca quando le submission vengono mergiate.', + lead:'Prova CodeDNA sul tuo progetto reale. Dichiara stack + install. Stessi task con vs senza CodeDNA. Apri una PR metrics per iscriverti — la bacheca pubblica si aggiorna quando arrivano PR valide.', ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', - experimentalNote:'Sperimentale. I risultati possono variare per agente AI di coding e linguaggio. Se annotazioni o CLI si comportano male sul tuo stack, apri una issue o una pull request di fix così possiamo rafforzare il supporto. Quel feedback fa parte della challenge.', + experimentalNote:'Sperimentale. I risultati possono variare per agente AI e linguaggio (Claude Code, OpenCode, Cursor, Codex, …). Pubblica come hai installato CodeDNA. Se annotazioni o CLI si comportano male, apri una issue o una pull request di fix. Quel feedback fa parte della challenge.', statusTitle:'Stato challenge', - statusLead:'Bacheca aggiornata dai maintainer. File sorgente: docs/challenge-board.json.', + statusLead:'Classifica pubblica aggiornata quando arrivano PR valide. File sorgente: docs/challenge-board.json.', metaPrize:'Premio', metaPrizeSub:'pagato dopo review', metaSubs:'Submission', metaStatus:'Stato', metaEnroll:'Iscrizione', metaEnrollVal:'PR = dentro', metaEnrollSub:'niente signup ufficiale', howTitle:'Come funziona', - howLead:'Progetto tuo · ≥10 task · con vs senza CodeDNA · stack L0/L1/L2 equo · PR metrics.json.', - s1t:'Sul tuo repo', s1p:'Dichiara linguaggi, framework, numero file e size band. Mix facili / medi / difficili.', - s2t:'Stack equo', s2p:'Se usi wiki, skill o Graphify, ci devono essere in entrambe le condizioni — salvo modalità codedna-only dichiarata.', - s3t:'Apri la PR', s3p:'Aggiungi challenge//metrics.json. Quella PR ti iscrive. Merge e aggiornamento bacheca a cura nostra.', + howLead:'Progetto reale funzionante (≥25 file) · dichiara linguaggi/framework/agente · ≥10 stessi task con vs senza CodeDNA · pubblica install · segnala bug · PR metrics.json.', + s1t:'Progetto reale + stack', s1p:'Niente siti giocattolo. Dichiara linguaggi, framework, numero file, size band e come hai installato CodeDNA.', + s2t:'Stessi task, stack equo', s2p:'Esegui gli stessi ≥10 task con e senza CodeDNA. Tieni la parity L1/L2 salvo modalità codedna-only dichiarata.', + s3t:'PR → bacheca live', s3p:'Apri una PR metrics per iscriverti. La classifica pubblica si aggiorna quando arrivano PR valide. Possiamo chiedere una presentazione Meet.', l0:'Header in-source — layer sotto test.', l1:'Wiki LLM, skill pack, instruction file — parity obbligatoria.', l2:'Memoria graph / strutturale — parity obbligatoria.', - honestyNote:'Onestà. Non inventare i risultati. Fabrizio o il team di review possono chiedere una review live (es. Google Meet) per verificare le run. Quando possibile faremo anche live pubbliche sui test inviati.', + honestyNote:'Onestà. Non inventare risultati o progetti finti. Fabrizio o il team di review possono richiedere una call + presentazione per verificare progetto e processo di test. Quando possibile faremo anche live pubbliche sui test inviati.', boardTitle:'Bacheca pubblica', - boardLead:'Linguaggio + framework, dimensione progetto e numero file sono obbligatori per confronti onesti.', + boardLead:'Linguaggio + framework + percorso di install e dimensione progetto sono obbligatori. La classifica si aggiorna quando arrivano PR valide e resta pubblica.', emptyTitle:'Ancora nessuna submission in bacheca', - emptyBody:'Quando una PR metrics viene accettata e mergiata, aggiungiamo una riga modificando challenge-board.json.', - boardNote:'Bacheca statica e curata. Aprire una PR ti iscrive automaticamente; comparire qui significa che i maintainer hanno accettato il formato. Gap agente × linguaggio sono attesi — apri issue o PR di fix.', + emptyBody:'Appena una PR metrics valida viene reviewata, aggiungiamo una riga modificando challenge-board.json.', + boardNote:'Classifica live, curata dai maintainer. Aprire una PR ti iscrive; comparire qui significa che formato e dichiarazione di stack sono stati accettati. Gap agente × linguaggio sono attesi — apri issue o PR di fix.', thHandle:'Partecipante', thStack:'Linguaggio / framework', thSize:'Size', thFiles:'File', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', unlock:'min {n} per sbloccare', windowFallback:'date TBD' diff --git a/docs/challenge.it.md b/docs/challenge.it.md index 4dd57e4..220dedb 100644 --- a/docs/challenge.it.md +++ b/docs/challenge.it.md @@ -18,13 +18,15 @@ Questa challenge pone una sola domanda: ## In sintesi -1. Lavori sul **tuo** repo (non sui nostri vecchi benchmark). -2. Esegui **almeno 10 task** (mix facili / medi / difficili). -3. Confronti **con CodeDNA** vs **senza CodeDNA**. +1. Lavori sul **tuo progetto reale e funzionante** (non sui nostri vecchi benchmark — non un sito giocattolo). +2. Dichiari lo **stack tecnologico** completo (linguaggi, framework, agente, percorso di install). +3. Esegui **almeno 10 task** (mix facili / medi / difficili) — gli **stessi task due volte**: una **senza** CodeDNA e una **con** CodeDNA. 4. Tieni lo stack AI **equo** (vedi Livelli sotto). -5. Apri una **Pull Request** con `metrics.json` — anche se CodeDNA risulta peggiore. **Quella PR è la tua iscrizione.** -6. Fabrizio + team valutano complessità, protocollo ed evidenze; la bacheca pubblica si aggiorna a mano. -7. Il premio si sblocca solo se si raggiunge il **minimo di partecipanti**. +5. Documenti **come hai installato CodeDNA**; segnali bug / apri PR di fix quando qualcosa si rompe (atteso — è ancora sperimentale). +6. Apri una **Pull Request** con `metrics.json` — anche se CodeDNA risulta peggiore. **Quella PR è la tua iscrizione.** +7. La **bacheca / classifica pubblica si aggiorna quando arrivano PR valide** e resta visibile sul sito docs. +8. Il progetto può essere chiamato a una **review live / presentazione** (call) così verifichiamo che i test siano reali. +9. Il premio si sblocca solo se si raggiunge il **minimo di partecipanti**. --- @@ -36,24 +38,30 @@ Questa challenge pone una sola domanda: | Submission valide minime | **5** per sbloccare il premio (obiettivo stretch: 10) | | Se ci sono meno di 5 PR valide | I risultati si pubblicano comunque; il premio **non** viene assegnato (o passa all’edizione successiva) | | Vincitore | Scelto da Fabrizio Corpora + team di review (non voto community) | -| Onestà | Le affermazioni devono essere vere e supportate da evidenze — metriche o narrative inventate = squalifica | -| Verifica | Fabrizio o il team di review possono chiedere una review live (Google Meet / simile) per ripercorrere run, log e setup del repo | +| Onestà | Le affermazioni devono essere vere e supportate da evidenze — metriche inventate, progetti finti o narrative false = squalifica | +| Verifica | Fabrizio o il team di review possono richiedere una **call + presentazione** (Google Meet / simile): mostrare il progetto reale, spiegare come hai fatto i test, ripercorrere install + `metrics.json` | +| Bacheca pubblica | La classifica su [challenge.html](challenge.html) si aggiorna **appena arrivano PR valide** (review maintainer → riga in bacheca) e resta visibile | | Live | Quando possibile ospiteremo live pubbliche sui test inviati (con consenso del partecipante dove serve) | ### Cosa valutiamo (in ordine) -1. **Onestà del protocollo** — stack equo, modalità dichiarata, note riproducibili; **niente risultati inventati** -2. **Qualità dei task** — mix reale facili/medi/difficili su una codebase vera -3. **Evidenze** — metriche + breve narrativa; a favore **o** contro CodeDNA va bene -4. **Bug report** — issue actionable aperte upstream contano positivamente +1. **Onestà del protocollo** — progetto reale, stack equo, modalità dichiarata, note riproducibili; **niente risultati inventati** +2. **Qualità dei task** — mix reale facili/medi/difficili su una codebase funzionante +3. **Evidenze** — metriche + percorso di install + breve narrativa; a favore **o** contro CodeDNA va bene +4. **Bug report / PR di fix** — obbligatori quando CodeDNA si rompe sul tuo agente o linguaggio; le issue actionable contano positivamente 5. **Chiarezza** — un altro engineer può ripetere il confronto ### Onestà, call di review e live - **Non puoi inventare** esiti pass/fail, tempi, dettagli di stack o claim narrativi. Se non riesci a riprodurre un numero, marca il task come inconclusive e dillo. -- Fabrizio Corpora e/o il team di review **possono chiedere una video call** (es. Google Meet) per verificare la submission: screen-share del progetto, riesecuzione di un task campione, walkthrough di `metrics.json` / note. +- **Progetti finti o throwaway non sono ammessi.** Possiamo richiedere una **conferenza live + breve presentazione** in cui: + - dimostri che il progetto è reale e funzionante; + - spieghi lo stack tecnologico; + - mostri come hai installato CodeDNA; + - mostri come hai eseguito gli **stessi task** con e senza CodeDNA; + - rispondi su `metrics.json` / note. - Rifiutare senza motivo valido una richiesta ragionevole di verifica può annullare l’eleggibilità al premio per quella entry. -- Quando possibile faremo anche **live pubbliche** sui test della challenge (metodologia, sorprese, gap agente × linguaggio). Partecipare a una live è opzionale, salvo finalisti chiamati in verifica. +- Quando possibile faremo anche **live pubbliche** sui test della challenge (metodologia, sorprese, gap agente × linguaggio). Partecipare a una live è opzionale, salvo richiesta di verifica. --- @@ -62,10 +70,11 @@ Questa challenge pone una sola domanda: | Fase | Quando | |---|---| | Annuncio + video | **TBD** | -| Apertura iscrizioni | **TBD** | +| Apertura submission (PR = iscrizione) | **TBD** | | Finestra challenge | **1 mese** dalla data di inizio | | Scadenza submission | fine della finestra (le PR devono essere aperte) | -| Review | ~1–2 settimane dopo la scadenza | +| Bacheca / classifica | aggiornata in continuo quando arrivano PR valide | +| Review | ~1–2 settimane dopo la scadenza (più eventuali presentazioni Meet) | | Annuncio vincitore | **TBD** | --- @@ -80,20 +89,26 @@ Questa challenge pone una sola domanda: ## Protocollo centrale (obbligatorio) -### 1. Il tuo progetto +### 1. Il tuo progetto (deve essere reale) -Usa un repo che **mantieni tu** (lavoro o personale). Linea guida dimensione: abbastanza superficie perché 10 task siano significativi (indicativamente ≥20 file sorgente). +Usa un repo che **mantieni tu** (lavoro o personale). Deve essere un **progetto reale e funzionante** — anche piccolo va bene; una pagina vetrina, un hello-world o una demo throwaway **no**. -In `metrics.json` **devi** dichiarare: +**Esempi rifiutati:** sito vetrina monostrato, CRUD vuoto senza logica di dominio, repo sintetici costruiti solo per la challenge. + +**Esempi accettati:** un servizio che usi o shippi davvero, un tool interno, una libreria con caller reali, un prodotto piccolo ma completo (≥ **25** file sorgente come soglia dura). + +In `metrics.json` **devi** dichiarare lo **stack tecnologico**: | Campo | Esempio | |---|---| | `languages` | `["TypeScript", "Python"]` | -| `frameworks` | `["NestJS", "FastAPI"]` — il solo linguaggio non basta | -| `approx_source_files` | `120` | +| `frameworks` | `["NestJS", "FastAPI"]` — il solo linguaggio **non** basta | +| `approx_source_files` | `120` (minimo **25**) | | `size_band` | `S` (<50) · `M` (50–199) · `L` (200–999) · `XL` (1000+) | +| `tech_stack_notes` | opzionale — DB, layout monorepo, infra, lib principali | +| `install.agent` + `install.steps` | quale agente AI + comandi esatti di install/init CodeDNA | -### 2. Almeno dieci task +### 2. Almeno dieci task — gli stessi in entrambe le condizioni Definisci **≥10** task **prima** delle run cronometrate (o congela l’elenco nella PR). @@ -107,9 +122,11 @@ Mix suggerito: Per ogni task registra: obiettivo, difficoltà, agente/tool, successo/fallimento, note. +**Critico:** ogni task va eseguito in **entrambe** le condizioni. Non inventare liste di task diverse per Control vs CodeDNA. + ### 3. Due condizioni: senza vs con CodeDNA -Per ogni task (o batch accoppiati) esegui: +Per **ciascuno degli stessi task** esegui: | Condizione | Setup | |---|---| @@ -155,12 +172,29 @@ Obbligatorio per ogni task (Control + CodeDNA): | `human_interventions` | Quante volte hai dovuto guidare | | `confidence_1_to_5` | Opzionale ma utile | -Compila anche `summary.favors`: `codedna` | `control` | `tie` | `inconclusive`. +Compila anche: + +- `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` +- `install` — **obbligatorio**: agente + passi esatti di install/init (e se ha funzionato) +- `bugs_reported` — array **obbligatorio** (vuoto se nessuno); se qualcosa si è rotto, apri issue o PR di fix e elencala qui I numeri grezzi possono favorire CodeDNA **oppure no**. L’onestà batte il tifo. `notes.md` è narrativa opzionale — **non sostituisce** `metrics.json`. -### 6. Submission = Pull Request +### 6. Percorso di install + bug report (obbligatori quando rilevanti) + +Pubblica **come hai installato CodeDNA** in `metrics.json` → `install` (comandi, flag `--tools`, agente). +CodeDNA è ancora **sperimentale**: può fallire o comportarsi male con alcuni tool agentici (Claude Code, OpenCode, Cursor, Codex, …) o linguaggi. + +Se install, annotation, hook o refresh si comportano male: + +1. Riproduci una volta (agente + linguaggio/framework + comando). +2. Apri una **GitHub issue** o una **PR di fix**. +3. Elencala in `bugs_reported` e menzionala in `notes.md`. + +Tooling rotto su un certo stack **non** ti squalifica — **nasconderlo** sì. + +### 7. Submission = Pull Request Apri una PR su `Larens94/codedna`. Copia la checklist nel body della PR: @@ -184,13 +218,21 @@ challenge: — CodeDNA Challenge submission ``` **Perché JSON:** ogni partecipante ha la sua cartella, le PR si mergiano senza conflitti, e a fine challenge aggreghiamo tutti i `metrics.json`. + +### 8. Bacheca / classifica pubblica + +La classifica live è [challenge.html](challenge.html) (dati: [`challenge-board.json`](challenge-board.json)). + +- Appena una PR metrics valida viene reviewata, aggiungiamo/aggiorniamo una riga — la bacheca resta **visibile pubblicamente**. +- Aprire una PR ti iscrive; comparire in bacheca significa che i maintainer hanno accettato formato e dichiarazione di stack. + --- ## Come entrare (niente signup ufficiale) **Non c’è iscrizione separata.** Quando apri una PR valida di challenge con `metrics.json`, sei iscritto. -1. Installa CodeDNA e annota il progetto: +1. Installa CodeDNA e annota il progetto (registra i passi esatti in `install`): ```bash pipx install git+https://github.com/Larens94/codedna.git @@ -198,21 +240,21 @@ codedna install --path . --tools codedna init . --no-llm # oppure con LLM per le rules: ``` -2. Esegui i ≥10 task (Control vs CodeDNA, stack equo). -3. Apri una PR con `challenge//metrics.json`. -4. Facciamo review, merge e aggiorniamo la [bacheca pubblica](challenge.html). +2. Esegui i ≥10 **stessi** task due volte (Control vs CodeDNA, stack equo). +3. Apri una PR con `challenge//metrics.json` (includi `install` + `bugs_reported`). +4. Facciamo review, aggiorniamo la [bacheca pubblica](challenge.html) e possiamo invitarti a una presentazione Meet. Solo domande (opzionale): GitHub Discussions / Discord — il vecchio template “entry issue” non è obbligatorio. ### Se qualcosa si rompe sul tuo agente o linguaggio -CodeDNA è ancora sperimentale tra agenti e linguaggi. Percorso consigliato: +CodeDNA è ancora sperimentale tra agenti e linguaggi (Claude Code, OpenCode, Cursor, Codex, …). Percorso consigliato: 1. Riproduci una volta (agente + linguaggio/framework + comando). 2. Apri una **issue** (bug) o una **PR** con fix minimo / test di regressione. 3. Continua la challenge se puoi; annota l’incidente in `metrics.json` → `bugs_reported` e in `notes.md`. -Tooling rotto su un certo stack **non** ti squalifica — segnalarlo è utile. +Tooling rotto su un certo stack **non** ti squalifica — **nasconderlo** sì. --- @@ -220,7 +262,8 @@ Tooling rotto su un certo stack **non** ti squalifica — segnalarlo è utile. - Non è un rerun di SWE-bench / delle nostre tabelle F1 storiche - Non è “annota solo i nostri repo di fixture” -- Non è consulenza gratis per noi — tieni l’IP del tuo progetto; noi reviewiamo solo la PR delle metriche +- Non è un sito throwaway o hello-world costruito solo per il premio +- Non è consulenza gratis per noi — tieni l’IP del tuo progetto; noi reviewiamo solo la PR delle metriche (+ eventuale presentazione Meet) --- @@ -229,7 +272,7 @@ Tooling rotto su un certo stack **non** ti squalifica — segnalarlo è utile. - Issue / bug: GitHub Issues - Q&A challenge: GitHub Discussions (Announcements / Q&A) - Community: Discord (vedi badge in README) -- Call di verifica: Google Meet (o simile) su richiesta dei maintainer +- Call di verifica / presentazione: Google Meet (o simile) su richiesta dei maintainer - Live pubbliche: annunciate su Discussions / Discord quando programmate - Maintainer: Fabrizio Corpora diff --git a/docs/challenge.md b/docs/challenge.md index 65a5c9d..1d4fed8 100644 --- a/docs/challenge.md +++ b/docs/challenge.md @@ -18,13 +18,15 @@ This challenge asks one question: ## TL;DR -1. Work on **your** repo (not our old benchmarks). -2. Run **at least 10 tasks** (easy / medium / hard mix). -3. Compare **with CodeDNA** vs **without CodeDNA**. +1. Work on **your real, working project** (not our old benchmarks — not a toy site). +2. Declare your full **tech stack** (languages, frameworks, agent, install path). +3. Run **at least 10 tasks** (easy / medium / hard mix) — the **same tasks twice**: once **without** CodeDNA, once **with** CodeDNA. 4. Keep your AI stack **fair** (see Levels below). -5. Open a **Pull Request** with `metrics.json` — even if CodeDNA looks worse. **That PR is your enrollment.** -6. Fabrizio + team review complexity, protocol compliance, and evidence; the public board is updated manually. -7. Prize unlocks only if the **minimum participant count** is met. +5. Document **how you installed CodeDNA**; report bugs / open fix PRs when something breaks (expected — still experimental). +6. Open a **Pull Request** with `metrics.json` — even if CodeDNA looks worse. **That PR is your enrollment.** +7. The **public board / ranking updates as valid PRs arrive** and stays visible on the docs site. +8. Your project may be called to a **live review / presentation** (conference call) so we can verify the tests are real. +9. Prize unlocks only if the **minimum participant count** is met. --- @@ -36,24 +38,30 @@ This challenge asks one question: | Minimum valid submissions | **5** to unlock the prize (stretch goal: 10) | | If fewer than 5 valid PRs | Challenge still publishes results; prize is **not** awarded (or rolled to the next edition) | | Winner | Selected by Fabrizio Corpora + review team (not community vote) | -| Honesty | Claims must be true and evidence-backed — fabricated metrics or narratives = disqualification | -| Verification | Fabrizio or the review team may request a live review (Google Meet / similar) to walk through runs, logs, and repo setup | +| Honesty | Claims must be true and evidence-backed — fabricated metrics, fake projects, or invented narratives = disqualification | +| Verification | Fabrizio or the review team may require a **conference call + presentation** (Google Meet / similar): show the real project, explain how you ran the tests, walk through install + `metrics.json` | +| Public board | Ranking on [challenge.html](challenge.html) is updated **as soon as valid PRs arrive** (maintainer review → board row) and remains publicly visible | | Lives | When possible we will host public live sessions discussing submitted tests (with entrant consent where needed) | ### What we score (in order) -1. **Protocol honesty** — fair stack, declared mode, reproducible notes; **no invented results** -2. **Task quality** — real easy/medium/hard mix on a real codebase -3. **Evidence** — metrics + short narrative; pro **or** against CodeDNA is fine -4. **Bug reports** — actionable issues filed upstream count positively +1. **Protocol honesty** — real project, fair stack, declared mode, reproducible notes; **no invented results** +2. **Task quality** — real easy/medium/hard mix on a real working codebase +3. **Evidence** — metrics + install path + short narrative; pro **or** against CodeDNA is fine +4. **Bug reports / fix PRs** — required when CodeDNA breaks on your agent or language; actionable upstream issues count positively 5. **Clarity** — another engineer can re-run your comparison ### Honesty, review calls, and lives - You **must not invent** pass/fail outcomes, timings, stack details, or narrative claims. If you cannot reproduce a number, mark the task inconclusive and say so. -- Fabrizio Corpora and/or the review team **may ask for a video call** (e.g. Google Meet) to verify your submission: screen-share the project, rerun a sample task, and walk through `metrics.json` / notes. +- **Fake or throwaway projects are not allowed.** We may require a **live conference + short presentation** where you: + - show that the project is real and working; + - explain your tech stack; + - walk through how you installed CodeDNA; + - show how you ran the **same tasks** with and without CodeDNA; + - answer questions on `metrics.json` / notes. - Refusing a reasonable verification request without a good reason can void prize eligibility for that entry. -- When feasible we will also run **public live sessions** covering interesting challenge runs (methodology, surprises, agent × language gaps). Participation in a live is optional unless you are a finalist asked for verification. +- When feasible we will also run **public live sessions** covering interesting challenge runs (methodology, surprises, agent × language gaps). Participation in a live is optional unless you are asked for verification. --- @@ -62,10 +70,11 @@ This challenge asks one question: | Phase | When | |---|---| | Announcement + video | **TBD** | -| Signup opens | **TBD** | +| Submissions open (PR = enroll) | **TBD** | | Challenge window | **1 month** from start date | | Submission deadline | end of challenge window (PRs must be open) | -| Review | ~1–2 weeks after deadline | +| Board / ranking | updated continuously as valid PRs arrive | +| Review | ~1–2 weeks after deadline (plus any Meet presentations) | | Winner announced | **TBD** | --- @@ -80,20 +89,26 @@ This challenge asks one question: ## Core protocol (mandatory) -### 1. Your project +### 1. Your project (must be real) -Use a repo **you** maintain (work or personal). Size guideline: enough surface that 10 tasks are meaningful (roughly ≥20 source files). +Use a repo **you** maintain (work or personal). It must be a **real, working project** — small is fine; a tiny marketing page, hello-world, or throwaway demo is **not**. -In `metrics.json` you **must** declare: +**Rejected examples:** one-page “sito vetrina”, empty CRUD scaffold with no domain logic, synthetic repos built only for the challenge. + +**Accepted examples:** a service you actually use or ship, an internal tool, a library with real callers, a small but complete product (≥ **25** source files as a hard floor). + +In `metrics.json` you **must** declare your **tech stack**: | Field | Example | |---|---| | `languages` | `["TypeScript", "Python"]` | -| `frameworks` | `["NestJS", "FastAPI"]` — language alone is not enough | -| `approx_source_files` | `120` | +| `frameworks` | `["NestJS", "FastAPI"]` — language alone is **not** enough | +| `approx_source_files` | `120` (minimum **25**) | | `size_band` | `S` (<50) · `M` (50–199) · `L` (200–999) · `XL` (1000+) | +| `tech_stack_notes` | optional — DB, monorepo layout, infra, major libs | +| `install.agent` + `install.steps` | which AI agent + exact CodeDNA install/init commands | -### 2. Ten tasks minimum +### 2. Ten tasks minimum — same tasks both ways Define **≥10** tasks **before** you start the timed runs (or freeze the list in the PR). @@ -107,9 +122,11 @@ Suggested mix: Record for each task: goal, difficulty, agent/tool, success/fail, notes. +**Critical:** every task runs in **both** conditions. Do not invent a different task list for Control vs CodeDNA. + ### 3. Two conditions: without vs with CodeDNA -For each task (or paired batches), run: +For **each of the same tasks**, run: | Condition | Setup | |---|---| @@ -155,12 +172,29 @@ Required per task (Control + CodeDNA): | `human_interventions` | How often you had to steer | | `confidence_1_to_5` | Optional but useful | -Also fill `summary.favors`: `codedna` | `control` | `tie` | `inconclusive`. +Also fill: + +- `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` +- `install` — **required**: agent + exact install/init steps (and whether it worked) +- `bugs_reported` — **required** array (empty if none); if something broke, file an issue or fix PR and list it here Raw numbers can favor CodeDNA **or not**. Honesty beats cheerleading. Markdown notes (`notes.md`) are optional narrative — **do not replace** `metrics.json`. -### 6. Submission = Pull Request +### 6. Install path + bug reports (mandatory when relevant) + +Publish **how you installed CodeDNA** in `metrics.json` → `install` (commands, `--tools` flag, agent). +CodeDNA is still **experimental**: it may fail or behave poorly with some agentic tools (Claude Code, OpenCode, Cursor, Codex, …) or languages. + +If install, annotation, hooks, or refresh misbehaves: + +1. Reproduce once (agent + language/framework + command). +2. Open a **GitHub issue** or a **fix PR**. +3. List it in `bugs_reported` and mention it in `notes.md`. + +Broken tooling on a given stack does **not** disqualify you — **hiding** it does. + +### 7. Submission = Pull Request Open a PR against `Larens94/codedna`. Copy the checklist into the PR body: @@ -184,13 +218,21 @@ challenge: — CodeDNA Challenge submission ``` **Why JSON:** each entrant lands in their own folder, PRs merge cleanly, and we can aggregate all `metrics.json` files after the window closes. + +### 8. Public board / ranking + +The live ranking is [challenge.html](challenge.html) (data: [`challenge-board.json`](challenge-board.json)). + +- As soon as a valid metrics PR is reviewed, we add/update a row — the board stays **publicly visible**. +- Opening a PR enrolls you; appearing on the board means maintainers accepted the submission format and stack declaration. + --- ## How to enter (no official signup) There is **no separate registration**. When you open a valid challenge PR with `metrics.json`, you are enrolled. -1. Install CodeDNA and annotate your project: +1. Install CodeDNA and annotate your project (record the exact steps in `install`): ```bash pipx install git+https://github.com/Larens94/codedna.git @@ -198,21 +240,21 @@ codedna install --path . --tools codedna init . --no-llm # or with an LLM for rules: ``` -2. Run your ≥10 tasks (Control vs CodeDNA, fair stack). -3. Open a PR adding `challenge//metrics.json`. -4. We review, merge, and update the [public board](challenge.html). +2. Run your ≥10 **same** tasks twice (Control vs CodeDNA, fair stack). +3. Open a PR adding `challenge//metrics.json` (include `install` + `bugs_reported`). +4. We review, update the [public board](challenge.html), and may invite you to a Meet presentation. Questions only (optional): GitHub Discussions / Discord — the old “entry issue” template is not required. ### If something breaks on your agent or language -CodeDNA is still experimental across agents and languages. Prefer this path: +CodeDNA is still experimental across agents and languages (Claude Code, OpenCode, Cursor, Codex, …). Prefer this path: 1. Reproduce once (agent + language/framework + command). 2. Open an **issue** (bug) or a **PR** with a minimal fix / regression test. 3. Continue the challenge if you can; note the incident in `metrics.json` → `bugs_reported` and in `notes.md`. -Broken tooling on a given stack does **not** disqualify you — reporting it is valuable. +Broken tooling on a given stack does **not** disqualify you — **hiding** it does. --- @@ -220,7 +262,8 @@ Broken tooling on a given stack does **not** disqualify you — reporting it is - Not a rerun of SWE-bench / our historical F1 tables - Not “annotate our fixture repos only” -- Not unpaid consulting for us — you keep your project IP; we only review the metrics PR +- Not a tiny throwaway site or hello-world built only for the prize +- Not unpaid consulting for us — you keep your project IP; we only review the metrics PR (+ optional Meet presentation) --- @@ -229,7 +272,7 @@ Broken tooling on a given stack does **not** disqualify you — reporting it is - Issues / bugs: GitHub Issues - Challenge Q&A: GitHub Discussions (Announcements / Q&A) - Community: Discord (see README badge) -- Verification calls: Google Meet (or similar) when requested by maintainers +- Verification / presentation calls: Google Meet (or similar) when requested by maintainers - Public lives: announced on Discussions / Discord when scheduled - Maintainer: Fabrizio Corpora From c38fc02ec50c9cac1ef9c527e093d53707206cbf Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 20:38:17 +0000 Subject: [PATCH 03/13] docs(challenge): polish public ranking page for launch video MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename board to ranking, add #ranking deep link + OG tags, improve empty state for social screenshots, and soften window note to “coming soon” until official dates are set. AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: docs/challenge.html,docs/challenge-board.json,README.md AI-Message: Ranking page ready for video; merge #19 to refresh GitHub Pages Co-authored-by: Fabrizio Corpora --- README-it.md | 2 +- README.md | 2 +- docs/challenge-board.json | 2 +- docs/challenge.html | 73 +++++++++++++++++++++++++++------------ 4 files changed, 54 insertions(+), 25 deletions(-) diff --git a/README-it.md b/README-it.md index 826c4c4..77b4abb 100644 --- a/README-it.md +++ b/README-it.md @@ -50,7 +50,7 @@ Nessuna infrastruttura. Nessuna pipeline di retrieval. Nessuna memoria esterna. Valutazione pubblica sul **tuo** progetto (non SWE-bench). Progetto reale funzionante (≥25 file), stack tecnologico + install CodeDNA dichiarati, ≥10 **stessi** task con/senza CodeDNA, parity L0/L1/L2, PR con metriche — anche se vanno contro CodeDNA. I bug vanno segnalati (sperimentale tra agenti). Possibile presentazione Meet; la bacheca pubblica si aggiorna quando arrivano PR valide. -- Bacheca: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) +- Classifica / bacheca: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Regolamento: [`docs/challenge.it.md`](docs/challenge.it.md) (IT) · [`docs/challenge.md`](docs/challenge.md) (EN) - Submission: `challenge//metrics.json` ([esempio](challenge/metrics.example.json)) — **aprire la PR ti iscrive** diff --git a/README.md b/README.md index 801989e..c7ad48b 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ No infrastructure. No retrieval pipeline. No external memory. The code carries i Public **own-project** evaluation (not SWE-bench). Real working project (≥25 files), declared tech stack + CodeDNA install, ≥10 **same** tasks with/without CodeDNA, fair L0/L1/L2 parity, metrics PR — even if results go against CodeDNA. Bugs must be reported (experimental across agents). Meet presentation may be required; public board updates as valid PRs arrive. -- Board: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) +- Ranking / board: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Rules: [`docs/challenge.md`](docs/challenge.md) (EN) · [`docs/challenge.it.md`](docs/challenge.it.md) (IT) - Submission: `challenge//metrics.json` ([example](challenge/metrics.example.json)) — **opening the PR enrolls you** diff --git a/docs/challenge-board.json b/docs/challenge-board.json index a62d9bf..dcdb6ca 100644 --- a/docs/challenge-board.json +++ b/docs/challenge-board.json @@ -7,7 +7,7 @@ "window": { "start": null, "end": null, - "note": "Dates TBD — set before public launch" + "note": "Coming soon — dates announced with the launch video" }, "submissions": [] } diff --git a/docs/challenge.html b/docs/challenge.html index d943874..56563a9 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -3,8 +3,14 @@ -CodeDNA Challenge — €200 - +CodeDNA Challenge — €200 · Public ranking + + + + + + + @@ -150,10 +156,21 @@ .favors-control { color: #d97706; font-weight: 600; } .favors-tie, .favors-inconclusive { color: var(--gray-600); font-weight: 600; } .empty { - padding: 3rem 1.5rem; text-align: center; color: var(--gray-600); + padding: 3.25rem 1.75rem; text-align: center; color: var(--gray-600); border: 1px dashed var(--gray-200); border-radius: var(--radius); + background: + linear-gradient(180deg, #fafafa 0%, #fff 100%); +} +.empty strong { display: block; color: var(--black); font-size: 1.05rem; margin-bottom: 0.55rem; } +.empty span { display: block; max-width: 420px; margin: 0 auto 1.25rem; line-height: 1.65; } +.empty-meta { + display: inline-flex; flex-wrap: wrap; gap: 0.5rem; justify-content: center; + font-family: 'JetBrains Mono', monospace; font-size: 11px; color: var(--gray-600); +} +.empty-meta span { + display: inline-block; margin: 0; padding: 0.35rem 0.65rem; + border: 1px solid var(--gray-200); border-radius: var(--radius); background: var(--white); } -.empty strong { display: block; color: var(--black); margin-bottom: 0.4rem; } .note { margin-top: 1.25rem; font-size: 0.85rem; color: var(--gray-600); @@ -208,7 +225,8 @@

CodeDNA Challenge
€200

Test CodeDNA on your own real project. Declare stack + install. Same tasks with vs without CodeDNA. Open a metrics PR to enroll — the public board updates as valid PRs arrive.

@@ -224,7 +242,7 @@

CodeDNA Challenge
€200

Challenge status

-

Maintainer-updated board. Source file: docs/challenge-board.json.

+

Public ranking updated as valid PRs arrive. Source file: docs/challenge-board.json.

Prize
@@ -239,7 +257,7 @@

Challenge status

Status
open
-
dates TBD
+
coming soon
Enrollment
@@ -289,8 +307,8 @@

PR → live board

-
-

Public board

+
+

Public ranking

Language + framework + install path and project size are required. Ranking updates as valid PRs arrive and stays public.

@@ -315,12 +333,17 @@

Public board

- No submissions on the board yet - When a metrics PR is reviewed and merged, we add a row here by editing challenge-board.json. + Ranking ready — waiting for first PRs + This is the live public leaderboard. As soon as valid metrics PRs arrive and are reviewed, rows appear here. +
-

This board is static and curated. Opening a PR enrolls you automatically; appearing here means maintainers accepted the submission format.

+

Live ranking, curated by maintainers. Opening a PR enrolls you; appearing here means the submission format and stack declaration were accepted. Agent × language gaps are expected — file issues or fix PRs.

@@ -343,7 +366,7 @@

Public board

eyebrow:'Public evaluation · €200', h1:'CodeDNA Challenge
€200', lead:'Test CodeDNA on your own real project. Declare stack + install. Same tasks with vs without CodeDNA. Open a metrics PR to enroll — the public board updates as valid PRs arrive.', - ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', + ctaRanking:'View ranking →', ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', experimentalNote:'Experimental. Results can vary by AI coding agent and language (Claude Code, OpenCode, Cursor, Codex, …). Publish how you installed CodeDNA. If annotations or the CLI misbehave, open a GitHub issue or a fix pull request. That feedback is part of the challenge.', statusTitle:'Challenge status', statusLead:'Public ranking updated as valid PRs arrive. Source file: docs/challenge-board.json.', @@ -359,21 +382,21 @@

Public board

l1:'LLM wiki, skill packs, instruction files — parity required.', l2:'Graph / structural memory — parity required.', honestyNote:'Honesty. Do not invent results or fake projects. Fabrizio or the review team may require a conference call + presentation to verify your project and test process. When possible we will also host public lives discussing submitted tests.', - boardTitle:'Public board', + boardTitle:'Public ranking', boardLead:'Language + framework + install path and project size are required. Ranking updates as valid PRs arrive and stays public.', - emptyTitle:'No submissions on the board yet', + emptyTitle:'Ranking ready — waiting for first PRs', emptyBody:'As soon as a valid metrics PR is reviewed, we add a row here by editing challenge-board.json.', boardNote:'Live ranking, curated by maintainers. Opening a PR enrolls you; appearing here means the submission format and stack declaration were accepted. Agent × language gaps are expected — file issues or fix PRs.', thHandle:'Entrant', thStack:'Language / framework', thSize:'Size', thFiles:'Files', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', - unlock:'min {n} to unlock', windowFallback:'dates TBD' + unlock:'min {n} to unlock', windowFallback:'coming soon' }, it: { navHome:'Home', navChallenge:'Challenge', navInstall:'Install', eyebrow:'Valutazione pubblica · €200', h1:'CodeDNA Challenge
€200', lead:'Prova CodeDNA sul tuo progetto reale. Dichiara stack + install. Stessi task con vs senza CodeDNA. Apri una PR metrics per iscriverti — la bacheca pubblica si aggiorna quando arrivano PR valide.', - ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', + ctaRanking:'Vedi classifica →', ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', experimentalNote:'Sperimentale. I risultati possono variare per agente AI e linguaggio (Claude Code, OpenCode, Cursor, Codex, …). Pubblica come hai installato CodeDNA. Se annotazioni o CLI si comportano male, apri una issue o una pull request di fix. Quel feedback fa parte della challenge.', statusTitle:'Stato challenge', statusLead:'Classifica pubblica aggiornata quando arrivano PR valide. File sorgente: docs/challenge-board.json.', @@ -389,14 +412,14 @@

Public board

l1:'Wiki LLM, skill pack, instruction file — parity obbligatoria.', l2:'Memoria graph / strutturale — parity obbligatoria.', honestyNote:'Onestà. Non inventare risultati o progetti finti. Fabrizio o il team di review possono richiedere una call + presentazione per verificare progetto e processo di test. Quando possibile faremo anche live pubbliche sui test inviati.', - boardTitle:'Bacheca pubblica', + boardTitle:'Classifica pubblica', boardLead:'Linguaggio + framework + percorso di install e dimensione progetto sono obbligatori. La classifica si aggiorna quando arrivano PR valide e resta pubblica.', - emptyTitle:'Ancora nessuna submission in bacheca', - emptyBody:'Appena una PR metrics valida viene reviewata, aggiungiamo una riga modificando challenge-board.json.', + emptyTitle:'Classifica pronta — in attesa delle prime PR', + emptyBody:'Questa è la leaderboard pubblica live. Appena arrivano PR metrics valide e vengono reviewate, le righe compaiono qui.', boardNote:'Classifica live, curata dai maintainer. Aprire una PR ti iscrive; comparire qui significa che formato e dichiarazione di stack sono stati accettati. Gap agente × linguaggio sono attesi — apri issue o PR di fix.', thHandle:'Partecipante', thStack:'Linguaggio / framework', thSize:'Size', thFiles:'File', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', - unlock:'min {n} per sbloccare', windowFallback:'date TBD' + unlock:'min {n} per sbloccare', windowFallback:'in arrivo' } }; @@ -449,7 +472,13 @@

Public board

const root = document.getElementById('board-root'); if (!rows.length) { - root.innerHTML = `
${dict.emptyTitle}${dict.emptyBody}
`; + root.innerHTML = `
${dict.emptyTitle}${dict.emptyBody} + +
`; return; } From 71de2bab9e7ed3f97b81a593b2cca1413a3e21f2 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 20:38:28 +0000 Subject: [PATCH 04/13] docs(challenge): point headers at public ranking page AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: docs/challenge.md,docs/challenge.it.md AI-Message: Header labels match ranking page for video launch Co-authored-by: Fabrizio Corpora --- docs/challenge.it.md | 2 +- docs/challenge.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/challenge.it.md b/docs/challenge.it.md index 220dedb..a98f80a 100644 --- a/docs/challenge.it.md +++ b/docs/challenge.it.md @@ -6,7 +6,7 @@ > **Durata:** 1 mese dalla data ufficiale di inizio > **Iscrizione:** aprire la PR con le metriche = sei iscritto (niente signup ufficiale) > **Non è un rerun di SWE-bench.** Provi CodeDNA sul **tuo** progetto. -> **Bacheca pubblica:** [challenge.html](challenge.html) sul sito docs +> **Classifica pubblica:** [challenge.html](challenge.html) sul sito docs Questa challenge pone una sola domanda: diff --git a/docs/challenge.md b/docs/challenge.md index 1d4fed8..15e1eaa 100644 --- a/docs/challenge.md +++ b/docs/challenge.md @@ -6,7 +6,7 @@ > **Duration:** 1 month from the official start date > **Enrollment:** opening a metrics PR = you are enrolled (no separate signup) > **Not a SWE-bench rerun.** You test CodeDNA on **your own project**. -> **Public board:** [challenge.html](challenge.html) on the docs site +> **Public ranking:** [challenge.html](challenge.html) on the docs site This challenge asks one question: From cd2ef8aeb74232344458b001ff14d24bb6b889b6 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 20:38:39 +0000 Subject: [PATCH 05/13] docs(challenge): align EN emptyBody copy on ranking page AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: docs/challenge.html AI-Message: EN empty-state I18N matches static HTML for video screenshots Co-authored-by: Fabrizio Corpora --- docs/challenge.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/challenge.html b/docs/challenge.html index 56563a9..5c89ef2 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -385,7 +385,7 @@

Public ranking

boardTitle:'Public ranking', boardLead:'Language + framework + install path and project size are required. Ranking updates as valid PRs arrive and stays public.', emptyTitle:'Ranking ready — waiting for first PRs', - emptyBody:'As soon as a valid metrics PR is reviewed, we add a row here by editing challenge-board.json.', + emptyBody:'This is the live public leaderboard. As soon as valid metrics PRs arrive and are reviewed, rows appear here.', boardNote:'Live ranking, curated by maintainers. Opening a PR enrolls you; appearing here means the submission format and stack declaration were accepted. Agent × language gaps are expected — file issues or fix PRs.', thHandle:'Entrant', thStack:'Language / framework', thSize:'Size', thFiles:'Files', thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', From 0b621ae91a5e97df9d987a9c53647f0287f8ac84 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 20:42:42 +0000 Subject: [PATCH 06/13] fix(docs): challenge page mobile padding and nav spacing site.css forced .section horizontal padding to 0; restore side gutters on the challenge page and separate the mobile GitHub CTA from the language toggle. AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: docs/challenge.html,docs/site.css AI-Message: Mobile gutters + GitHub/IT gap fixed via challenge-page overrides Co-authored-by: Fabrizio Corpora --- docs/challenge.html | 69 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 3 deletions(-) diff --git a/docs/challenge.html b/docs/challenge.html index 5c89ef2..9132c2c 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -189,17 +189,80 @@ .meta-cell { border-right: none; border-bottom: 1px solid var(--gray-200); } } @media (max-width: 720px) { - .nav-links { display: none; position: absolute; top: 60px; left: 0; right: 0; flex-direction: column; gap: 0; background: var(--white); border-bottom: 1px solid var(--gray-200); padding: 0.75rem 1rem 1rem; } + .nav-links { display: none; position: absolute; top: 68px; left: 0; right: 0; flex-direction: column; gap: 0.5rem; background: var(--white); border-bottom: 1px solid var(--gray-200); padding: 0.85rem 1.1rem 1.15rem; } .nav-links.open { display: flex; } - .nav-links a { padding: 0.65rem 0; } + .nav-links a { padding: 0.7rem 0; } + .nav-links a.nav-cta { margin-top: 0.35rem; margin-bottom: 0.35rem; } + .nav-links .lang-toggle { margin-top: 0.15rem; align-self: stretch; } .nav-hamburger { display: flex; } .meta-strip, .steps, .levels { grid-template-columns: 1fr; } .page-hero h1 { max-width: none; } + .page-hero, .section, footer.site-footer { padding-left: 20px; padding-right: 20px; } } + - +
Regolamento (IT) Rules (EN) Copia metrics.json → +

Sperimentale. I risultati possono variare per agente AI e linguaggio (Claude Code, OpenCode, Cursor, Codex, …). Pubblica come hai installato CodeDNA. Se annotazioni o CLI si comportano male, apri una @@ -332,7 +338,7 @@

Stato challenge

Come funziona

-

Progetto reale funzionante (≥25 file) · dichiara linguaggi/framework/agente · ≥10 stessi task con vs senza CodeDNA · pubblica install · segnala bug · PR metrics.json.

+

Metodologia uguale per tutti: progetto reale · due branch/checkout/progetti · ≥10 stessi task con vs senza CodeDNA · copia prompt per l’agente · pubblica install · segnala bug.

01
@@ -341,13 +347,13 @@

Progetto reale + stack

02
-

Stessi task, stack equo

-

Esegui gli stessi ≥10 task con e senza CodeDNA. Tieni la parity L1/L2 salvo modalità codedna-only dichiarata.

+

Due branch o due progetti

+

Stessi ≥10 task su Control e CodeDNA. Separa con due branch, due checkout o due progetti gemelli — dichiara `setup.layout`.

03
-

PR → bacheca live

-

Apri una PR metrics per iscriverti. La classifica pubblica si aggiorna quando arrivano PR valide. Possiamo chiedere una presentazione Meet.

+

Copia prompt → PR

+

Copia il prompt per il tuo agente, esegui la lista attività, apri la PR metrics. La classifica si aggiorna quando arrivano PR valide.

@@ -428,8 +434,8 @@

Classifica pubblica

navHome:'Home', navChallenge:'Challenge', navInstall:'Install', eyebrow:'Public evaluation · €200', h1:'CodeDNA Challenge
€200', - lead:'Test CodeDNA on your own real project. Declare stack + install. Same tasks with vs without CodeDNA. Open a metrics PR to enroll — the public board updates as valid PRs arrive.', - ctaRanking:'View ranking →', ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', + lead:'Test CodeDNA on your own real project. Same methodology for everyone: two branches or twin projects, same tasks with vs without CodeDNA. Copy the agent prompt, then open a metrics PR.', + ctaRanking:'View ranking →', ctaCopyAgent:'Copy for your agent', ctaExample:'Copy metrics.json →', ctaRules:'Full rules (EN)', ctaRulesIt:'Regolamento (IT)', ctaCopied:'Copied — paste into your AI agent', ctaCopyFail:'Could not copy — open challenge-agent-prompt.md', experimentalNote:'Experimental. Results can vary by AI coding agent and language (Claude Code, OpenCode, Cursor, Codex, …). Publish how you installed CodeDNA. If annotations or the CLI misbehave, open a GitHub issue or a fix pull request. That feedback is part of the challenge.', statusTitle:'Challenge status', statusLead:'Public ranking updated as valid PRs arrive. Source file: docs/challenge-board.json.', @@ -437,10 +443,10 @@

Classifica pubblica

metaSubs:'Submissions', metaStatus:'Status', metaEnroll:'Enrollment', metaEnrollVal:'PR = in', metaEnrollSub:'no official signup', howTitle:'How it works', - howLead:'Real working project (≥25 files) · declare languages/frameworks/agent · ≥10 same tasks with vs without CodeDNA · publish install · report bugs · metrics.json PR.', + howLead:'Same methodology for everyone: real project · two branches/checkouts/projects · ≥10 same tasks with vs without CodeDNA · copy agent prompt · publish install · report bugs.', s1t:'Real project + stack', s1p:'No toy sites. Declare languages, frameworks, file count, size band, and how you installed CodeDNA.', - s2t:'Same tasks, fair stack', s2p:'Run the same ≥10 tasks with and without CodeDNA. Keep L1/L2 parity unless you declare codedna-only mode.', - s3t:'PR → live board', s3p:'Open a metrics PR to enroll. The public ranking updates as valid PRs arrive. We may ask for a Meet presentation.', + s2t:'Two branches or twin projects', s2p:'Same ≥10 tasks on Control and CodeDNA. Split with two branches, two checkouts, or twin projects — declare setup.layout.', + s3t:'Copy prompt → PR', s3p:'Copy the agent prompt, run the activity list, open the metrics PR. Ranking updates as valid PRs arrive.', l0:'In-source headers — the layer under test.', l1:'LLM wiki, skill packs, instruction files — parity required.', l2:'Graph / structural memory — parity required.', @@ -458,8 +464,8 @@

Classifica pubblica

navHome:'Home', navChallenge:'Challenge', navInstall:'Install', eyebrow:'Valutazione pubblica · €200', h1:'CodeDNA Challenge
€200', - lead:'Prova CodeDNA sul tuo progetto reale. Dichiara stack + install. Stessi task con vs senza CodeDNA. Apri una PR metrics per iscriverti — la bacheca pubblica si aggiorna quando arrivano PR valide.', - ctaRanking:'Vedi classifica →', ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', + lead:'Prova CodeDNA sul tuo progetto reale. Metodologia uguale per tutti: due branch o progetti gemelli, stessi task con vs senza CodeDNA. Copia il prompt per l’agente, poi apri la PR metrics.', + ctaRanking:'Vedi classifica →', ctaCopyAgent:'Copia per il tuo agente', ctaExample:'Copia metrics.json →', ctaRules:'Rules (EN)', ctaRulesIt:'Regolamento (IT)', ctaCopied:'Copiato — incollalo nel tuo agente AI', ctaCopyFail:'Copia non riuscita — apri challenge-agent-prompt.it.md', experimentalNote:'Sperimentale. I risultati possono variare per agente AI e linguaggio (Claude Code, OpenCode, Cursor, Codex, …). Pubblica come hai installato CodeDNA. Se annotazioni o CLI si comportano male, apri una issue o una pull request di fix. Quel feedback fa parte della challenge.', statusTitle:'Stato challenge', statusLead:'Classifica pubblica aggiornata quando arrivano PR valide. File sorgente: docs/challenge-board.json.', @@ -467,10 +473,10 @@

Classifica pubblica

metaSubs:'Submission', metaStatus:'Stato', metaEnroll:'Iscrizione', metaEnrollVal:'PR = dentro', metaEnrollSub:'niente signup ufficiale', howTitle:'Come funziona', - howLead:'Progetto reale funzionante (≥25 file) · dichiara linguaggi/framework/agente · ≥10 stessi task con vs senza CodeDNA · pubblica install · segnala bug · PR metrics.json.', + howLead:'Metodologia uguale per tutti: progetto reale · due branch/checkout/progetti · ≥10 stessi task con vs senza CodeDNA · copia prompt per l’agente · pubblica install · segnala bug.', s1t:'Progetto reale + stack', s1p:'Niente siti giocattolo. Dichiara linguaggi, framework, numero file, size band e come hai installato CodeDNA.', - s2t:'Stessi task, stack equo', s2p:'Esegui gli stessi ≥10 task con e senza CodeDNA. Tieni la parity L1/L2 salvo modalità codedna-only dichiarata.', - s3t:'PR → bacheca live', s3p:'Apri una PR metrics per iscriverti. La classifica pubblica si aggiorna quando arrivano PR valide. Possiamo chiedere una presentazione Meet.', + s2t:'Due branch o due progetti', s2p:'Stessi ≥10 task su Control e CodeDNA. Separa con due branch, due checkout o due progetti gemelli — dichiara setup.layout.', + s3t:'Copia prompt → PR', s3p:'Copia il prompt per il tuo agente, esegui la lista attività, apri la PR metrics. La classifica si aggiorna quando arrivano PR valide.', l0:'Header in-source — layer sotto test.', l1:'Wiki LLM, skill pack, instruction file — parity obbligatoria.', l2:'Memoria graph / strutturale — parity obbligatoria.', @@ -590,8 +596,29 @@

Classifica pubblica

document.getElementById('filter-favors').addEventListener('change', () => boardData && renderBoard(boardData)); document.getElementById('filter-size').addEventListener('change', () => boardData && renderBoard(boardData)); + +async function copyAgentPrompt() { + const dict = I18N[currentLang]; + const status = document.getElementById('copy-status'); + const file = currentLang === 'it' ? 'challenge-agent-prompt.it.md' : 'challenge-agent-prompt.md'; + try { + const res = await fetch(file, { cache: 'no-store' }); + if (!res.ok) throw new Error(res.statusText); + const text = await res.text(); + await navigator.clipboard.writeText(text); + status.hidden = false; + status.textContent = dict.ctaCopied; + } catch (e) { + status.hidden = false; + status.textContent = dict.ctaCopyFail; + window.open(file, '_blank'); + } +} +document.getElementById('copy-agent-prompt').addEventListener('click', copyAgentPrompt); + applyI18n(); loadBoard(); + diff --git a/docs/challenge.it.md b/docs/challenge.it.md index a98f80a..895d371 100644 --- a/docs/challenge.it.md +++ b/docs/challenge.it.md @@ -20,13 +20,14 @@ Questa challenge pone una sola domanda: 1. Lavori sul **tuo progetto reale e funzionante** (non sui nostri vecchi benchmark — non un sito giocattolo). 2. Dichiari lo **stack tecnologico** completo (linguaggi, framework, agente, percorso di install). -3. Esegui **almeno 10 task** (mix facili / medi / difficili) — gli **stessi task due volte**: una **senza** CodeDNA e una **con** CodeDNA. +3. Usi la **stessa metodologia per tutti**: **≥10 stessi task** due volte — senza CodeDNA e con CodeDNA — tramite **due branch**, **due checkout** o **due progetti gemelli**. 4. Tieni lo stack AI **equo** (vedi Livelli sotto). 5. Documenti **come hai installato CodeDNA**; segnali bug / apri PR di fix quando qualcosa si rompe (atteso — è ancora sperimentale). -6. Apri una **Pull Request** con `metrics.json` — anche se CodeDNA risulta peggiore. **Quella PR è la tua iscrizione.** -7. La **bacheca / classifica pubblica si aggiorna quando arrivano PR valide** e resta visibile sul sito docs. -8. Il progetto può essere chiamato a una **review live / presentazione** (call) così verifichiamo che i test siano reali. -9. Il premio si sblocca solo se si raggiunge il **minimo di partecipanti**. +6. Opzionale: incolla il **[prompt agente](challenge-agent-prompt.it.md)** nel tuo agente AI così segue protocollo + lista attività. +7. Apri una **Pull Request** con `metrics.json` — anche se CodeDNA risulta peggiore. **Quella PR è la tua iscrizione.** +8. La **bacheca / classifica pubblica si aggiorna quando arrivano PR valide** e resta visibile sul sito docs. +9. Il progetto può essere chiamato a una **review live / presentazione** (call) così verifichiamo che i test siano reali. +10. Il premio si sblocca solo se si raggiunge il **minimo di partecipanti**. --- @@ -126,15 +127,33 @@ Per ogni task registra: obiettivo, difficoltà, agente/tool, successo/fallimento ### 3. Due condizioni: senza vs con CodeDNA -Per **ciascuno degli stessi task** esegui: +**La metodologia è uguale per tutti.** Per **ciascuno degli stessi task** esegui: | Condizione | Setup | |---|---| | **A — Control** | Il tuo workflow AI normale **senza** annotation CodeDNA / senza affidarti agli header CodeDNA | | **B — CodeDNA** | Stesso workflow **con** CodeDNA installato e annotato (`codedna init` / header mantenuti) | +#### Come separare A e B (scegline uno — dichiaralo in `metrics.json` → `setup.layout`) + +| Layout | `setup.layout` | Esempio | +|---|---|---| +| Due branch nello stesso repo | `two_branches` | `challenge/control` + `challenge/codedna` | +| Due checkout / cartelle | `two_checkouts` | due clone locali dello stesso progetto | +| Due progetti gemelli | `two_projects` | due repo/cartelle dallo stesso codice di partenza | + Mantieni **agente, modello e layer superiori identici** tra A e B, tranne CodeDNA stesso (salvo modalità dichiarata — vedi sotto). +### 3b. Copia per il tuo agente (consigliato) + +Incolla il prompt pronto nel tuo agente AI così riceve regole + scheletro attività: + +- Italiano: [`challenge-agent-prompt.it.md`](challenge-agent-prompt.it.md) +- English: [`challenge-agent-prompt.md`](challenge-agent-prompt.md) +- Checklist task: [`../challenge/TASKS_TEMPLATE.it.md`](../challenge/TASKS_TEMPLATE.it.md) · [EN](../challenge/TASKS_TEMPLATE.md) + +Sulla pagina pubblica: **Copia per il tuo agente** copia il prompt nella lingua corrente. + ### 4. Stack equo — Livelli (critico) Se già usi layer extra per lo sviluppo AI, CodeDNA va testato **sopra lo stesso stack**, non al posto di quelle cose per sbaglio. @@ -176,6 +195,7 @@ Compila anche: - `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` - `install` — **obbligatorio**: agente + passi esatti di install/init (e se ha funzionato) +- `setup.layout` — **obbligatorio**: `two_branches` | `two_checkouts` | `two_projects` - `bugs_reported` — array **obbligatorio** (vuoto se nessuno); se qualcosa si è rotto, apri issue o PR di fix e elencala qui I numeri grezzi possono favorire CodeDNA **oppure no**. L’onestà batte il tifo. diff --git a/docs/challenge.md b/docs/challenge.md index 15e1eaa..a74765c 100644 --- a/docs/challenge.md +++ b/docs/challenge.md @@ -20,13 +20,14 @@ This challenge asks one question: 1. Work on **your real, working project** (not our old benchmarks — not a toy site). 2. Declare your full **tech stack** (languages, frameworks, agent, install path). -3. Run **at least 10 tasks** (easy / medium / hard mix) — the **same tasks twice**: once **without** CodeDNA, once **with** CodeDNA. +3. Use the **same methodology for everyone**: **≥10 same tasks** twice — without CodeDNA and with CodeDNA — via **two branches**, **two checkouts**, or **two twin projects**. 4. Keep your AI stack **fair** (see Levels below). 5. Document **how you installed CodeDNA**; report bugs / open fix PRs when something breaks (expected — still experimental). -6. Open a **Pull Request** with `metrics.json` — even if CodeDNA looks worse. **That PR is your enrollment.** -7. The **public board / ranking updates as valid PRs arrive** and stays visible on the docs site. -8. Your project may be called to a **live review / presentation** (conference call) so we can verify the tests are real. -9. Prize unlocks only if the **minimum participant count** is met. +6. Optional: paste the **[agent prompt](challenge-agent-prompt.md)** into your AI agent so it follows the protocol + activity list. +7. Open a **Pull Request** with `metrics.json` — even if CodeDNA looks worse. **That PR is your enrollment.** +8. The **public board / ranking updates as valid PRs arrive** and stays visible on the docs site. +9. Your project may be called to a **live review / presentation** (conference call) so we can verify the tests are real. +10. Prize unlocks only if the **minimum participant count** is met. --- @@ -126,15 +127,33 @@ Record for each task: goal, difficulty, agent/tool, success/fail, notes. ### 3. Two conditions: without vs with CodeDNA -For **each of the same tasks**, run: +**Methodology is the same for everyone.** For **each of the same tasks**, run: | Condition | Setup | |---|---| | **A — Control** | Your normal AI workflow **without** CodeDNA annotations / without relying on CodeDNA headers | | **B — CodeDNA** | Same workflow **with** CodeDNA installed + annotated (`codedna init` / maintained headers) | +#### How to separate A and B (pick one — declare in `metrics.json` → `setup.layout`) + +| Layout | `setup.layout` | Example | +|---|---|---| +| Two branches in your repo | `two_branches` | `challenge/control` + `challenge/codedna` | +| Two checkouts / folders | `two_checkouts` | two local clones of the same project | +| Two twin projects | `two_projects` | two repos/folders from the same starting code | + Keep the **agent, model, and higher stack layers identical** between A and B except for CodeDNA itself (unless you declare an explicit mode — see below). +### 3b. Copy for your agent (recommended) + +Paste the ready-made prompt into your AI agent so it gets rules + activity skeleton: + +- English: [`challenge-agent-prompt.md`](challenge-agent-prompt.md) +- Italiano: [`challenge-agent-prompt.it.md`](challenge-agent-prompt.it.md) +- Task checklist: [`../challenge/TASKS_TEMPLATE.md`](../challenge/TASKS_TEMPLATE.md) · [IT](../challenge/TASKS_TEMPLATE.it.md) + +On the public page: **Copia per il tuo agente** copies the prompt for the current language. + ### 4. Fair stack — Levels (critical) If you already use extra AI-dev layers, CodeDNA must be tested **on top of the same stack**, not instead of it by accident. @@ -176,6 +195,7 @@ Also fill: - `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` - `install` — **required**: agent + exact install/init steps (and whether it worked) +- `setup.layout` — **required**: `two_branches` | `two_checkouts` | `two_projects` - `bugs_reported` — **required** array (empty if none); if something broke, file an issue or fix PR and list it here Raw numbers can favor CodeDNA **or not**. Honesty beats cheerleading. From d8d9282aca300981f6eeb1a67507bbd2d7e27cc2 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 21:03:38 +0000 Subject: [PATCH 11/13] docs: note agent prompt methodology in README IT and changelog AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: README-it.md,CHANGELOG.md AI-Message: IT README + changelog mention agent copy-pack Co-authored-by: Fabrizio Corpora --- CHANGELOG.md | 2 +- README-it.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 22eec31..e494b2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to CodeDNA will be documented in this file. ### Added -- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files ≥25 + `install` + `bugs_reported`), real-project bar (no toy sites), same-task control vs CodeDNA, Meet presentation verification, docs site page + live public board updated as valid PRs arrive. PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. +- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files ≥25 + `install` + `setup.layout` + `bugs_reported`), real-project bar (no toy sites), same-task control vs CodeDNA via two branches/checkouts/twin projects, copy-paste agent prompt + task list, Meet presentation verification, docs site page + live public board (IT default) updated as valid PRs arrive. PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. ### Added diff --git a/README-it.md b/README-it.md index 77b4abb..9f0ba38 100644 --- a/README-it.md +++ b/README-it.md @@ -48,7 +48,7 @@ Nessuna infrastruttura. Nessuna pipeline di retrieval. Nessuna memoria esterna. ## CodeDNA Challenge — €200 -Valutazione pubblica sul **tuo** progetto (non SWE-bench). Progetto reale funzionante (≥25 file), stack tecnologico + install CodeDNA dichiarati, ≥10 **stessi** task con/senza CodeDNA, parity L0/L1/L2, PR con metriche — anche se vanno contro CodeDNA. I bug vanno segnalati (sperimentale tra agenti). Possibile presentazione Meet; la bacheca pubblica si aggiorna quando arrivano PR valide. +Valutazione pubblica sul **tuo** progetto (non SWE-bench). Progetto reale funzionante (≥25 file), stack tecnologico + install CodeDNA dichiarati, stessa metodologia per tutti (due branch / checkout / progetti gemelli), ≥10 **stessi** task con/senza CodeDNA, [prompt da copiare per l’agente](docs/challenge-agent-prompt.it.md), PR con metriche — anche se vanno contro CodeDNA. I bug vanno segnalati (sperimentale tra agenti). Possibile presentazione Meet; la bacheca pubblica si aggiorna quando arrivano PR valide. - Classifica / bacheca: [larens94.github.io/codedna/challenge.html](https://larens94.github.io/codedna/challenge.html) - Regolamento: [`docs/challenge.it.md`](docs/challenge.it.md) (IT) · [`docs/challenge.md`](docs/challenge.md) (EN) From 3829cbf99a49eb179b16289caaaa7c40c39bc729 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 21:13:32 +0000 Subject: [PATCH 12/13] feat(challenge): optional file metrics + judge-agent comparison Keep files_expected / precision / recall / F1 optional when ground truth is unknown. Add judge prompt and schema fields so entrants can compare Control vs CodeDNA sessions with a separate judge agent after both runs. AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: challenge/metrics.schema.json,docs/challenge.md,docs/challenge-judge-prompt.it.md,docs/challenge.html AI-Message: File metrics optional; judge agent path added for post-run comparison Co-authored-by: Fabrizio Corpora --- challenge/README.md | 4 +- challenge/SUBMISSION_TEMPLATE.it.md | 2 + challenge/SUBMISSION_TEMPLATE.md | 2 + challenge/metrics.example.json | 92 ++++++- challenge/metrics.schema.json | 372 ++++++++++++++++++++++++++++ docs/challenge-agent-prompt.it.md | 14 +- docs/challenge-agent-prompt.md | 14 +- docs/challenge-board.README.md | 6 + docs/challenge-judge-prompt.it.md | 81 ++++++ docs/challenge-judge-prompt.md | 81 ++++++ docs/challenge.html | 14 +- docs/challenge.it.md | 21 ++ docs/challenge.md | 21 ++ 13 files changed, 710 insertions(+), 14 deletions(-) create mode 100644 docs/challenge-judge-prompt.it.md create mode 100644 docs/challenge-judge-prompt.md diff --git a/challenge/README.md b/challenge/README.md index f33f0c6..a617008 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -15,10 +15,12 @@ No official signup. Public ranking (updated as valid PRs arrive): [`docs/challen - Italiano: [`docs/challenge-agent-prompt.it.md`](../docs/challenge-agent-prompt.it.md) - English: [`docs/challenge-agent-prompt.md`](../docs/challenge-agent-prompt.md) +- Judge (optional, after both sessions): [`docs/challenge-judge-prompt.it.md`](../docs/challenge-judge-prompt.it.md) · [EN](../docs/challenge-judge-prompt.md) - Task lists: [`TASKS_TEMPLATE.it.md`](./TASKS_TEMPLATE.it.md) · [`TASKS_TEMPLATE.md`](./TASKS_TEMPLATE.md) - Pointer: [`AGENT_PROMPT.md`](./AGENT_PROMPT.md) -Methodology (same for everyone): same ≥10 tasks with vs without CodeDNA via `two_branches` / `two_checkouts` / `two_projects`. +Methodology (same for everyone): same ≥10 tasks with vs without CodeDNA via `two_branches` / `two_checkouts` / `two_projects`. +File-localization fields (`files_expected`, F1, …) are **optional** — use when known; otherwise a judge agent can still compare sessions. **Metrics format** diff --git a/challenge/SUBMISSION_TEMPLATE.it.md b/challenge/SUBMISSION_TEMPLATE.it.md index 27a0962..b529cf7 100644 --- a/challenge/SUBMISSION_TEMPLATE.it.md +++ b/challenge/SUBMISSION_TEMPLATE.it.md @@ -58,6 +58,8 @@ - [ ] ≥10 task con mix `difficulty` (easy ≥3, medium ≥3, hard ≥2) - [ ] Gli **stessi task** hanno risultati `control` e `codedna` - [ ] `summary.favors` impostato (`codedna` | `control` | `tie` | `inconclusive`) +- [ ] (Opzionale) `files_expected` / file F1 solo se conosciuti — non inventati +- [ ] (Opzionale) giudice usato: [`docs/challenge-judge-prompt.it.md`](../docs/challenge-judge-prompt.it.md) → `judge` / `tasks[].judge` - [ ] I risultati possono favorire **oppure** sfavorire CodeDNA (onestà OK) ### Opzionale diff --git a/challenge/SUBMISSION_TEMPLATE.md b/challenge/SUBMISSION_TEMPLATE.md index e89271a..8745274 100644 --- a/challenge/SUBMISSION_TEMPLATE.md +++ b/challenge/SUBMISSION_TEMPLATE.md @@ -58,6 +58,8 @@ - [ ] ≥10 tasks with `difficulty` mix (easy ≥3, medium ≥3, hard ≥2) - [ ] **Same tasks** each have both `control` and `codedna` results - [ ] `summary.favors` set (`codedna` | `control` | `tie` | `inconclusive`) +- [ ] (Optional) `files_expected` / file F1 only when known — never invented +- [ ] (Optional) judge used: [`docs/challenge-judge-prompt.md`](../docs/challenge-judge-prompt.md) → `judge` / `tasks[].judge` - [ ] Results may favor **or** disfavor CodeDNA (honesty OK) ### Optional diff --git a/challenge/metrics.example.json b/challenge/metrics.example.json index 6da9603..7f3be9e 100644 --- a/challenge/metrics.example.json +++ b/challenge/metrics.example.json @@ -61,7 +61,30 @@ "tool_calls": 12, "wrong_file_or_module": false, "human_interventions": 0, - "confidence_1_to_5": 4 + "confidence_1_to_5": 4, + "outcome": "pass", + "files_opened": [ + "apps/api/src/billing/formatAmount.ts", + "apps/api/src/billing/invoice.service.ts", + "apps/api/src/users/user.service.ts", + "apps/api/src/billing/invoice.service.spec.ts" + ], + "files_edited": [ + "apps/api/src/billing/formatAmount.ts", + "apps/api/src/billing/invoice.service.ts" + ], + "files_missed": [ + "apps/api/src/billing/invoice.service.spec.ts" + ], + "files_extra": [], + "file_precision": 1.0, + "file_recall": 0.667, + "file_f1": 0.8, + "first_relevant_file_turn": 3, + "opened_before_first_edit": 2, + "tests_green": false, + "rules_followed": true, + "human_rewrote_core": false }, "codedna": { "passed": true, @@ -70,9 +93,46 @@ "tool_calls": 8, "wrong_file_or_module": false, "human_interventions": 0, - "confidence_1_to_5": 5 + "confidence_1_to_5": 5, + "outcome": "pass", + "files_opened": [ + "apps/api/src/billing/formatAmount.ts", + "apps/api/src/billing/invoice.service.ts", + "apps/api/src/billing/invoice.service.spec.ts" + ], + "files_edited": [ + "apps/api/src/billing/formatAmount.ts", + "apps/api/src/billing/invoice.service.ts", + "apps/api/src/billing/invoice.service.spec.ts" + ], + "files_missed": [], + "files_extra": [], + "file_precision": 1.0, + "file_recall": 1.0, + "file_f1": 1.0, + "first_relevant_file_turn": 1, + "opened_before_first_edit": 1, + "tests_green": true, + "rules_followed": true, + "human_rewrote_core": false }, - "notes": "" + "notes": "", + "files_expected": [ + "apps/api/src/billing/formatAmount.ts", + "apps/api/src/billing/invoice.service.ts", + "apps/api/src/billing/invoice.service.spec.ts" + ], + "judge": { + "agent": "cursor", + "prompt_ref": "docs/challenge-judge-prompt.it.md", + "favors": "codedna", + "score_control_1_to_5": 3, + "score_codedna_1_to_5": 5, + "navigation_notes": "Control opened an unrelated user service file first.", + "correctness_notes": "Control missed updating the unit test.", + "files_assessment": "CodeDNA edited all three expected files; Control missed the spec.", + "raw_verdict": "CodeDNA clearer navigation and complete caller/test updates." + } }, { "id": "T02", @@ -312,7 +372,18 @@ "control_total_turns": 212, "codedna_total_turns": 125, "favors": "codedna", - "one_line_verdict": "CodeDNA reduced wrong-file mistakes on hard tasks; still honest about one medium task needing a steer." + "one_line_verdict": "CodeDNA reduced wrong-file mistakes on hard tasks; still honest about one medium task needing a steer.", + "control_pass_rate": 0.7, + "codedna_pass_rate": 0.9, + "delta_pass_rate": 0.2, + "control_file_f1_avg": 0.8, + "codedna_file_f1_avg": 1.0, + "delta_file_f1": 0.2, + "control_wrong_file_rate": 0.3, + "codedna_wrong_file_rate": 0.0, + "control_total_interventions": 8, + "codedna_total_interventions": 2, + "favors_basis": "judge + pass_rate + file_f1" }, "bugs_reported": [], "install": { @@ -327,5 +398,18 @@ "control_ref": "challenge/control", "codedna_ref": "challenge/codedna", "notes": "Same commit base; CodeDNA branch has codedna init annotations" + }, + "judge": { + "used": true, + "agent": "cursor", + "model": "claude-sonnet", + "prompt_ref": "docs/challenge-judge-prompt.it.md", + "method": "per_task", + "favors": "codedna", + "tasks_judged": 10, + "codedna_wins": 7, + "control_wins": 1, + "ties": 2, + "notes": "Judge compared diffs/logs after both sessions; did not need files_expected on every task." } } diff --git a/challenge/metrics.schema.json b/challenge/metrics.schema.json index 2f588e1..8f9e3a3 100644 --- a/challenge/metrics.schema.json +++ b/challenge/metrics.schema.json @@ -251,6 +251,90 @@ }, "notes": { "type": "string" + }, + "files_expected": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Optional ground-truth files for this task. Omit if unknown before/during the run. Needed only to compute precision/recall/F1." + }, + "judge": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "description": "Optional judge-agent comparison of Control vs CodeDNA for this task.", + "properties": { + "agent": { + "type": "string", + "description": "Judge agent/model used." + }, + "prompt_ref": { + "type": [ + "string", + "null" + ], + "description": "Link or path to the judge prompt used." + }, + "favors": { + "type": "string", + "enum": [ + "codedna", + "control", + "tie", + "inconclusive" + ] + }, + "score_control_1_to_5": { + "type": [ + "integer", + "null" + ], + "minimum": 1, + "maximum": 5 + }, + "score_codedna_1_to_5": { + "type": [ + "integer", + "null" + ], + "minimum": 1, + "maximum": 5 + }, + "navigation_notes": { + "type": [ + "string", + "null" + ] + }, + "correctness_notes": { + "type": [ + "string", + "null" + ] + }, + "files_assessment": { + "type": [ + "string", + "null" + ], + "description": "Judge notes on right/wrong/missed files even without a predeclared expected list." + }, + "raw_verdict": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "favors" + ] } } } @@ -324,6 +408,89 @@ }, "one_line_verdict": { "type": "string" + }, + "control_pass_rate": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1 + }, + "codedna_pass_rate": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1 + }, + "delta_pass_rate": { + "type": [ + "number", + "null" + ] + }, + "control_file_f1_avg": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Optional average file_f1 over tasks where measurable." + }, + "codedna_file_f1_avg": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1 + }, + "delta_file_f1": { + "type": [ + "number", + "null" + ] + }, + "control_wrong_file_rate": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Optional share of tasks with missed/extra or wrong_file_or_module." + }, + "codedna_wrong_file_rate": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1 + }, + "control_total_interventions": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "codedna_total_interventions": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "favors_basis": { + "type": [ + "string", + "null" + ], + "description": "Optional note: human / judge / pass_rate+file_f1 / etc." } } }, @@ -418,6 +585,90 @@ "type": "string" } } + }, + "judge": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "description": "Optional overall judge-agent review across sessions (Control vs CodeDNA).", + "properties": { + "used": { + "type": "boolean" + }, + "agent": { + "type": "string" + }, + "model": { + "type": [ + "string", + "null" + ] + }, + "prompt_ref": { + "type": [ + "string", + "null" + ] + }, + "method": { + "type": "string", + "enum": [ + "per_task", + "batch", + "spot_check" + ], + "description": "How the judge reviewed the runs." + }, + "favors": { + "type": "string", + "enum": [ + "codedna", + "control", + "tie", + "inconclusive" + ] + }, + "tasks_judged": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "codedna_wins": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "control_wins": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "ties": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "notes": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "used", + "favors" + ] } }, "$defs": { @@ -468,6 +719,127 @@ ], "minimum": 1, "maximum": 5 + }, + "outcome": { + "description": "Optional finer outcome; if set, passed should match outcome==pass.", + "anyOf": [ + { + "type": "string", + "enum": [ + "pass", + "partial", + "fail", + "inconclusive" + ] + }, + { + "type": "null" + } + ] + }, + "files_opened": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Optional. Files opened/read during the run." + }, + "files_edited": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Optional. Files actually modified." + }, + "files_missed": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Optional. Expected files not edited (only if files_expected known)." + }, + "files_extra": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + }, + "description": "Optional. Edited files not in expected (only if files_expected known)." + }, + "file_precision": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Optional. |edited \u2229 expected| / |edited|." + }, + "file_recall": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Optional. |edited \u2229 expected| / |expected|." + }, + "file_f1": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Optional. Harmonic mean of file_precision and file_recall." + }, + "first_relevant_file_turn": { + "type": [ + "integer", + "null" + ], + "minimum": 0, + "description": "Optional. Turn when the first relevant/expected file was opened." + }, + "opened_before_first_edit": { + "type": [ + "integer", + "null" + ], + "minimum": 0, + "description": "Optional. Files opened before the first useful edit." + }, + "tests_green": { + "type": [ + "boolean", + "null" + ], + "description": "Optional. Relevant tests green after the change." + }, + "rules_followed": { + "type": [ + "boolean", + "null" + ], + "description": "Optional. Applicable CodeDNA Rules: constraints followed." + }, + "human_rewrote_core": { + "type": [ + "boolean", + "null" + ], + "description": "Optional. Human had to rewrite core logic after the agent." } } } diff --git a/docs/challenge-agent-prompt.it.md b/docs/challenge-agent-prompt.it.md index 5eea746..fafaf3b 100644 --- a/docs/challenge-agent-prompt.it.md +++ b/docs/challenge-agent-prompt.it.md @@ -79,17 +79,24 @@ Usa questa checklist come scheletro (sostituisci i titoli con task **reali** del Per **ogni** task registra Control e CodeDNA: `passed`, `minutes`/`turns`/`tool_calls` se disponibili, `wrong_file_or_module`, `human_interventions`, note. +**Opzionale (non obbligatorio):** + +- `files_expected` solo se li conosci; altrimenti ometti (niente precision/recall inventate) +- `files_opened` / `files_edited` se riesci a traccarli +- Dopo entrambe le sessioni: usa un **agente giudice** con [`challenge-judge-prompt.it.md`](challenge-judge-prompt.it.md) e salva `tasks[].judge` + eventuale `judge` top-level + ## Deliverable 1. Congela la lista task **prima** delle run cronometrate. 2. Esegui tutti i task in **A**, poi gli **stessi** in **B** (o interleaved, ma stessi ID). -3. Compila `challenge//metrics.json` da +3. (Consigliato) Fai giudicare le due sessioni da un agente separato col prompt giudice. +4. Compila `challenge//metrics.json` da https://github.com/Larens94/codedna/blob/main/challenge/metrics.example.json (schema: `metrics.schema.json`). -4. Apri PR su `Larens94/codedna` con titolo: +5. Apri PR su `Larens94/codedna` con titolo: `challenge: — CodeDNA Challenge submission` Checklist: `challenge/SUBMISSION_TEMPLATE.it.md` -5. Onestà: niente risultati inventati. Possibile Meet di verifica. +6. Onestà: niente risultati inventati. Possibile Meet di verifica. ## Cosa fare ora @@ -97,3 +104,4 @@ Per **ogni** task registra Control e CodeDNA: `passed`, `minutes`/`turns`/`tool_ 2. Chiedimi quale setup uso: **due branch** / **due checkout** / **due progetti**. 3. Prepara i comandi di install per la condizione B. 4. Solo dopo conferma mia: esegui i task e compila le metriche. +5. Dopo le run: proponi il passaggio giudice (prompt già pronto) se voglio confrontare le sessioni. diff --git a/docs/challenge-agent-prompt.md b/docs/challenge-agent-prompt.md index 575cb85..30ee114 100644 --- a/docs/challenge-agent-prompt.md +++ b/docs/challenge-agent-prompt.md @@ -79,17 +79,24 @@ Use this checklist as a skeleton (replace titles with **real** tasks from my pro For **each** task record Control and CodeDNA: `passed`, `minutes`/`turns`/`tool_calls` when available, `wrong_file_or_module`, `human_interventions`, notes. +**Optional (not required):** + +- `files_expected` only if known; otherwise omit (do not invent precision/recall) +- `files_opened` / `files_edited` if you can track them +- After both sessions: run a **judge agent** with [`challenge-judge-prompt.md`](challenge-judge-prompt.md) and save `tasks[].judge` + optional top-level `judge` + ## Deliverable 1. Freeze the task list **before** timed runs. 2. Run every task in **A**, then the **same** ones in **B** (or interleaved, but same IDs). -3. Fill `challenge//metrics.json` from +3. (Recommended) Have a separate agent judge both sessions with the judge prompt. +4. Fill `challenge//metrics.json` from https://github.com/Larens94/codedna/blob/main/challenge/metrics.example.json (schema: `metrics.schema.json`). -4. Open a PR on `Larens94/codedna` titled: +5. Open a PR on `Larens94/codedna` titled: `challenge: — CodeDNA Challenge submission` Checklist: `challenge/SUBMISSION_TEMPLATE.md` -5. Honesty: no invented results. A Meet verification may be requested. +6. Honesty: no invented results. A Meet verification may be requested. ## What to do now @@ -97,3 +104,4 @@ For **each** task record Control and CodeDNA: `passed`, `minutes`/`turns`/`tool_ 2. Ask which setup I will use: **two branches** / **two checkouts** / **two projects**. 3. Prepare install commands for condition B. 4. Only after I confirm: run the tasks and fill the metrics. +5. After the runs: offer the judge pass (prompt ready) if I want a structured comparison. diff --git a/docs/challenge-board.README.md b/docs/challenge-board.README.md index 87a6d1e..be93b12 100644 --- a/docs/challenge-board.README.md +++ b/docs/challenge-board.README.md @@ -22,11 +22,17 @@ It reads [`docs/challenge-board.json`](../docs/challenge-board.json) (static — "tasks_total": 10, "control_passed": 7, "codedna_passed": 9, + "control_file_f1_avg": 0.72, + "codedna_file_f1_avg": 0.91, + "delta_file_f1": 0.19, + "judge_favors": "codedna", "favors": "codedna", + "favors_basis": "judge + pass_rate", "merged": true } ``` +Optional board fields (`control_file_f1_avg`, `codedna_file_f1_avg`, `delta_file_f1`, `judge_favors`, `favors_basis`) are shown when present — file lists / F1 are **not** required from entrants. 3. Bump `updated_at` (ISO date). 4. Commit + push `main` so GitHub Pages refreshes. diff --git a/docs/challenge-judge-prompt.it.md b/docs/challenge-judge-prompt.it.md new file mode 100644 index 0000000..f36dc0a --- /dev/null +++ b/docs/challenge-judge-prompt.it.md @@ -0,0 +1,81 @@ +# Prompt giudice — CodeDNA Challenge + +> Usa questo prompt **dopo** le due sessioni (Control e CodeDNA), non durante. +> Serve a confrontare le run in modo strutturato. +> English: [challenge-judge-prompt.md](challenge-judge-prompt.md) + +--- + +## Ruolo + +Sei un **agente giudice imparziale**. Confronti due esecuzioni dello **stesso task** (o dello stesso set di task): + +- **A — Control:** senza CodeDNA +- **B — CodeDNA:** con CodeDNA + +Non inventare file o esiti. Se l’evidenza manca, marca `inconclusive`. + +## Input che ti fornisco + +Per ogni task: + +1. Titolo / descrizione del task +2. Diff / file modificati della sessione Control +3. Diff / file modificati della sessione CodeDNA +4. (Opzionale) log turni, tool calls, interventi umani +5. (Opzionale) `files_expected` — **solo se li conosco**; altrimenti valuta senza ground truth + +## Cosa valutare (in ordine) + +1. **Completezza** — ha risolto il task? (pass / partial / fail) +2. **File toccati** — ha modificato i posti giusti? Ha dimenticato caller/test? Ha toccato file irrilevanti? +3. **Navigazione** — ha girato a vuoto o è andato subito ai file rilevanti? +4. **Vincoli** — ha rispettato regole di dominio / `Rules:` se evidenti dal diff o dal contesto +5. **Intervento umano** — quanto ha dovuto correggere la persona + +Se **non** c’è `files_expected`: + +- Non inventare precision/recall/F1 +- Descrivi comunque `files_assessment` in linguaggio naturale (completo / incompleto / fuori target) +- Puoi confrontare le due liste `files_edited` tra loro (quale sembra più coerente col task) + +Se **c’è** `files_expected`: + +- Calcola o stima missed / extra / precision / recall / F1 per entrambe le condizioni + +## Output richiesto (per task) + +```json +{ + "task_id": "T01", + "favors": "codedna | control | tie | inconclusive", + "score_control_1_to_5": 1, + "score_codedna_1_to_5": 1, + "navigation_notes": "", + "correctness_notes": "", + "files_assessment": "", + "raw_verdict": "" +} +``` + +## Output complessivo (opzionale) + +```json +{ + "used": true, + "method": "per_task | batch | spot_check", + "favors": "codedna | control | tie | inconclusive", + "tasks_judged": 10, + "codedna_wins": 0, + "control_wins": 0, + "ties": 0, + "notes": "" +} +``` + +## Regole + +- Non favorire CodeDNA a prescindere: se Control è meglio, dillo. +- Non usare conoscenza esterna inventata sul repo. +- Se confronti solo un campione di task, usa `method: spot_check` e dichiaralo. +- Il giudice **non sostituisce** `metrics.json`: arricchisce `tasks[].judge` e/o `judge` top-level. diff --git a/docs/challenge-judge-prompt.md b/docs/challenge-judge-prompt.md new file mode 100644 index 0000000..ddb2123 --- /dev/null +++ b/docs/challenge-judge-prompt.md @@ -0,0 +1,81 @@ +# Judge prompt — CodeDNA Challenge + +> Use this prompt **after** both sessions (Control and CodeDNA), not during them. +> It structures a comparison of the two runs. +> Italiano: [challenge-judge-prompt.it.md](challenge-judge-prompt.it.md) + +--- + +## Role + +You are an **impartial judge agent**. Compare two executions of the **same task** (or task set): + +- **A — Control:** without CodeDNA +- **B — CodeDNA:** with CodeDNA + +Do not invent files or outcomes. If evidence is missing, mark `inconclusive`. + +## Inputs I will provide + +Per task: + +1. Task title / description +2. Diff / edited files from Control +3. Diff / edited files from CodeDNA +4. (Optional) turn logs, tool calls, human interventions +5. (Optional) `files_expected` — **only if known**; otherwise judge without ground truth + +## What to score (in order) + +1. **Completeness** — did it solve the task? (pass / partial / fail) +2. **Files touched** — right places? missed callers/tests? irrelevant files? +3. **Navigation** — wandered vs went to relevant files quickly? +4. **Constraints** — domain / `Rules:` respected when visible from diff/context +5. **Human steering** — how much did the person have to fix + +If there is **no** `files_expected`: + +- Do **not** invent precision/recall/F1 +- Still write a natural-language `files_assessment` (complete / incomplete / off-target) +- You may compare the two `files_edited` lists for coherence with the task + +If there **is** `files_expected`: + +- Compute or estimate missed / extra / precision / recall / F1 for both conditions + +## Required per-task output + +```json +{ + "task_id": "T01", + "favors": "codedna | control | tie | inconclusive", + "score_control_1_to_5": 1, + "score_codedna_1_to_5": 1, + "navigation_notes": "", + "correctness_notes": "", + "files_assessment": "", + "raw_verdict": "" +} +``` + +## Optional overall output + +```json +{ + "used": true, + "method": "per_task | batch | spot_check", + "favors": "codedna | control | tie | inconclusive", + "tasks_judged": 10, + "codedna_wins": 0, + "control_wins": 0, + "ties": 0, + "notes": "" +} +``` + +## Rules + +- Do not bias toward CodeDNA: if Control is better, say so. +- Do not invent external repo knowledge. +- If you only sample tasks, use `method: spot_check` and declare it. +- The judge **does not replace** `metrics.json`: it fills `tasks[].judge` and/or top-level `judge`. diff --git a/docs/challenge.html b/docs/challenge.html index 6cb754a..8f9b2ef 100644 --- a/docs/challenge.html +++ b/docs/challenge.html @@ -457,7 +457,7 @@

Classifica pubblica

emptyBody:'This is the live public leaderboard. As soon as valid metrics PRs arrive and are reviewed, rows appear here.', boardNote:'Live ranking, curated by maintainers. Opening a PR enrolls you; appearing here means the submission format and stack declaration were accepted. Agent × language gaps are expected — file issues or fix PRs.', thHandle:'Entrant', thStack:'Language / framework', thSize:'Size', thFiles:'Files', - thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', + thMode:'Mode', thScore:'Pass C / DNA', thF1:'File-F1 C / DNA', thFavors:'Favors', thJudge:'Judge', thPr:'PR', unlock:'min {n} to unlock', windowFallback:'coming soon' }, it: { @@ -487,7 +487,7 @@

Classifica pubblica

emptyBody:'Questa è la leaderboard pubblica live. Appena arrivano PR metrics valide e vengono reviewate, le righe compaiono qui.', boardNote:'Classifica live, curata dai maintainer. Aprire una PR ti iscrive; comparire qui significa che formato e dichiarazione di stack sono stati accettati. Gap agente × linguaggio sono attesi — apri issue o PR di fix.', thHandle:'Partecipante', thStack:'Linguaggio / framework', thSize:'Size', thFiles:'File', - thMode:'Mode', thScore:'Pass C / DNA', thFavors:'Favors', thPr:'PR', + thMode:'Mode', thScore:'Pass C / DNA', thF1:'File-F1 C / DNA', thFavors:'Favors', thJudge:'Giudice', thPr:'PR', unlock:'min {n} per sbloccare', windowFallback:'in arrivo' } }; @@ -556,6 +556,12 @@

Classifica pubblica

const langs = (s.languages || []).map(x => `${escapeHtml(x)}`).join(''); const fws = (s.frameworks || []).map(x => `${escapeHtml(x)}`).join(''); const pr = s.pr_url ? `#${escapeHtml(String(s.pr_number || 'PR'))}` : '—'; + const f1c = s.control_file_f1_avg; + const f1d = s.codedna_file_f1_avg; + const f1 = (f1c != null || f1d != null) + ? `${f1c != null ? Number(f1c).toFixed(2) : '—'} / ${f1d != null ? Number(f1d).toFixed(2) : '—'}` + : '—'; + const judge = s.judge_favors || '—'; return ` ${escapeHtml(s.handle || '—')} ${langs}
${fws || ''}
@@ -563,7 +569,9 @@

Classifica pubblica

${s.approx_source_files != null ? escapeHtml(String(s.approx_source_files)) : '—'} ${escapeHtml(s.mode || '—')} ${escapeHtml(String(s.control_passed ?? '—'))} / ${escapeHtml(String(s.codedna_passed ?? '—'))} + ${escapeHtml(f1)} ${escapeHtml(s.favors || '—')} + ${escapeHtml(String(judge))} ${pr} `; }).join(''); @@ -571,7 +579,7 @@

Classifica pubblica

root.innerHTML = `
- + ${body}
${dict.thHandle}${dict.thStack}${dict.thSize}${dict.thFiles}${dict.thMode}${dict.thScore}${dict.thFavors}${dict.thPr}${dict.thMode}${dict.thScore}${dict.thF1}${dict.thFavors}${dict.thJudge}${dict.thPr}
`; diff --git a/docs/challenge.it.md b/docs/challenge.it.md index 895d371..04b4335 100644 --- a/docs/challenge.it.md +++ b/docs/challenge.it.md @@ -191,16 +191,37 @@ Obbligatorio per ogni task (Control + CodeDNA): | `human_interventions` | Quante volte hai dovuto guidare | | `confidence_1_to_5` | Opzionale ma utile | +Opzionali (consigliati quando disponibili — **non obbligatori**): + +| Campo | Note | +|---|---| +| `files_expected` (a livello task) | Solo se conosci/congeli i file; ometti se sconosciuti | +| `files_opened` / `files_edited` | Impronta di navigazione e edit | +| `files_missed` / `files_extra` / `file_precision` / `file_recall` / `file_f1` | Solo se esiste `files_expected` | +| `first_relevant_file_turn` / `opened_before_first_edit` | Efficienza di navigazione | +| `outcome` / `tests_green` / `rules_followed` / `human_rewrote_core` | Qualità più fine | +| `tasks[].judge` + `judge` top-level | Opzionale **agente giudice** che confronta Control vs CodeDNA dopo entrambe le sessioni — vedi [`challenge-judge-prompt.it.md`](challenge-judge-prompt.it.md) | + Compila anche: - `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` - `install` — **obbligatorio**: agente + passi esatti di install/init (e se ha funzionato) - `setup.layout` — **obbligatorio**: `two_branches` | `two_checkouts` | `two_projects` - `bugs_reported` — array **obbligatorio** (vuoto se nessuno); se qualcosa si è rotto, apri issue o PR di fix e elencala qui +- Extra summary opzionali: `*_file_f1_avg`, `delta_file_f1`, `*_wrong_file_rate`, `favors_basis` (`human` / `judge` / misto) I numeri grezzi possono favorire CodeDNA **oppure no**. L’onestà batte il tifo. `notes.md` è narrativa opzionale — **non sostituisce** `metrics.json`. +### 5b. Agente giudice opzionale (consigliato) + +**Non** serve avere `files_expected` per confrontare le run. Dopo entrambe le sessioni puoi incollare il [prompt giudice](challenge-judge-prompt.it.md) in un agente separato con i due diff/log e registrare: + +- per task `tasks[].judge` (favors, score, assessment file in linguaggio naturale) +- opzionale summary top-level `judge` (`method`: `per_task` | `batch` | `spot_check`) + +Il giudice arricchisce la storia in classifica; non sostituisce i campi obbligatori. + ### 6. Percorso di install + bug report (obbligatori quando rilevanti) Pubblica **come hai installato CodeDNA** in `metrics.json` → `install` (comandi, flag `--tools`, agente). diff --git a/docs/challenge.md b/docs/challenge.md index a74765c..611bfd0 100644 --- a/docs/challenge.md +++ b/docs/challenge.md @@ -191,16 +191,37 @@ Required per task (Control + CodeDNA): | `human_interventions` | How often you had to steer | | `confidence_1_to_5` | Optional but useful | +Optional (recommended when available — **not required**): + +| Field | Notes | +|---|---| +| `files_expected` (task-level) | Only if you know/ground the files beforehand or after; omit if unknown | +| `files_opened` / `files_edited` | Navigation & edit footprint | +| `files_missed` / `files_extra` / `file_precision` / `file_recall` / `file_f1` | Only meaningful when `files_expected` exists | +| `first_relevant_file_turn` / `opened_before_first_edit` | Navigation efficiency | +| `outcome` / `tests_green` / `rules_followed` / `human_rewrote_core` | Finer quality signals | +| `tasks[].judge` + top-level `judge` | Optional **judge agent** comparing Control vs CodeDNA after both sessions — see [`challenge-judge-prompt.md`](challenge-judge-prompt.md) | + Also fill: - `summary.favors`: `codedna` | `control` | `tie` | `inconclusive` - `install` — **required**: agent + exact install/init steps (and whether it worked) - `setup.layout` — **required**: `two_branches` | `two_checkouts` | `two_projects` - `bugs_reported` — **required** array (empty if none); if something broke, file an issue or fix PR and list it here +- Optional summary extras: `*_file_f1_avg`, `delta_file_f1`, `*_wrong_file_rate`, `favors_basis` (`human` / `judge` / mixed) Raw numbers can favor CodeDNA **or not**. Honesty beats cheerleading. Markdown notes (`notes.md`) are optional narrative — **do not replace** `metrics.json`. +### 5b. Optional judge agent (recommended) + +You do **not** need `files_expected` to compare runs. After both sessions you may paste the [judge prompt](challenge-judge-prompt.md) into a separate agent with the two diffs/logs and record: + +- per-task `tasks[].judge` (favors, scores, file assessment in natural language) +- optional top-level `judge` summary (`method`: `per_task` | `batch` | `spot_check`) + +The judge enriches the ranking story; it does not replace required fields. + ### 6. Install path + bug reports (mandatory when relevant) Publish **how you installed CodeDNA** in `metrics.json` → `install` (commands, `--tools` flag, agent). From 574b705c9a66a19c78786f78249adc62e7b05629 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 21:13:44 +0000 Subject: [PATCH 13/13] docs: changelog note for optional file metrics and judge AI-Agent: composer AI-Provider: cursor AI-Session: s_20260908_challenge_rules AI-Visited: CHANGELOG.md AI-Message: Changelog mentions optional F1 + judge agent Co-authored-by: Fabrizio Corpora --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e494b2a..fe6ba70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to CodeDNA will be documented in this file. ### Added -- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files ≥25 + `install` + `setup.layout` + `bugs_reported`), real-project bar (no toy sites), same-task control vs CodeDNA via two branches/checkouts/twin projects, copy-paste agent prompt + task list, Meet presentation verification, docs site page + live public board (IT default) updated as valid PRs arrive. PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. +- **CodeDNA Challenge (€200)** — bilingual rules, required `metrics.json` (languages + frameworks + size/files ≥25 + `install` + `setup.layout` + `bugs_reported`), optional file-localization metrics + optional judge-agent comparison, real-project bar (no toy sites), same-task control vs CodeDNA via two branches/checkouts/twin projects, copy-paste agent/judge prompts + task list, Meet presentation verification, docs site page + live public board (IT default) updated as valid PRs arrive. PR opens enrollment (no separate signup). Separate from historical SWE-bench benchmarks. ### Added