Skip to content

fix: align task contracts with observable verification - #3

Draft
zfj1998 wants to merge 1 commit into
AIoT-MLSys-Lab:mainfrom
zfj1998:agent/task-contract-quality-pr
Draft

fix: align task contracts with observable verification#3
zfj1998 wants to merge 1 commit into
AIoT-MLSys-Lab:mainfrom
zfj1998:agent/task-contract-quality-pr

Conversation

@zfj1998

@zfj1998 zfj1998 commented Jul 28, 2026

Copy link
Copy Markdown

Summary

This draft PR proposes task-contract and verifier-quality repairs found while
studying whether skills learned on T1--T3 transfer to changed T4--T6 inputs.

The core concern is attribution: when a semantically correct implementation is
rejected by an unpublished label, exact phrase, arbitrary tie-break, or
source-code spelling check, apparent "skill learning" can actually be verifier
adaptation.

The full rationale and per-task findings are in
docs/task_contract_quality_audit.md.

Scope

Across the 90 held-out T4--T6 tasks, this patch:

  • repairs 27 outcome or visible task-contract issues;
  • rewrites process-only checks for 27 additional tasks so they exercise
    observable behavior instead of private source spelling/layout;
  • leaves 36 held-out tasks unchanged;
  • replaces two scheduling instances whose original scoring depended on hidden
    arbitrary choices (one had no positive-length feasible intersection); and
  • retains all 30 skill-family capability targets.

The patch includes only benchmark task assets and focused quality regressions.
It intentionally excludes model harnesses, experiment-platform integration,
watchers, and analysis/reporting infrastructure.

Main defect classes

  1. Instruction/verifier mismatch — output paths, allowed null markers,
    reply obligations, assignee forms, or result cardinality differed.
  2. Hidden arbitrary contracts — exact labels, raw list order, slot IDs,
    fixed phrases, and one-reference-token checks were not published.
  3. Impossible fixture — an E6 scheduling task required a common window that
    did not exist.
  4. Source-shaped process checks — tests searched for variable names,
    helper-file locations, regex tokens, or comments instead of running the
    promised behavior.
  5. Cross-task policy drift — six inbox-triage tasks used inconsistent
    priority/deadline boundaries and occasionally omitted the decisive blocker
    from visible fixtures.

Validation

  • python scripts/validate_assets.py: PASS, 30 families / 180 tasks
    (the local review environment emitted expected warnings because Docker/Harbor
    were unavailable).
  • pytest tests -q: 15 passed.
  • Modified JSON files parse successfully.
  • Modified Python files compile successfully.
  • Ruff and git diff --check pass.
  • Checked-in reference solutions were also executed in real task containers:
    90/90 held-out T4--T6 tasks passed strict verification.
  • A fresh live-agent family smoke on the final fixtures passed all six T1--T6
    tasks.

The reference audit establishes internal solvability, not that every task is
perfect or that a particular model should pass. Live runs remain useful because
they can expose valid output surfaces that the reference solution does not
exercise.

Review requested

This is opened as a draft because the change is intentionally audit-sized.
Feedback would be especially useful on:

  1. whether the published policies match the intended latent skills;
  2. whether process verifiers should constrain implementation structure at all;
  3. whether semantic equivalence or a published deterministic tie-break is
    preferable where several answers are valid;
  4. whether the replacement scheduling instances preserve the intended
    difficulty; and
  5. whether maintainers would prefer the patch split by environment or defect
    class before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant