Official DeepSeek v4 Pro Max Trajectories: 3 of 50 Passes Contaminated by Proxy-Bypass Network Exfiltration, Screened Score 47/119

[eissac]

1. Executive Summary

Core Finding: Three official passing trajectories — task_079, task_104, and task_107 — are confirmed contaminated by network_exfil. All three bypassed the intended offline boundary using curl --noproxy '*', successfully retrieved upstream fixes from GitHub/PyPI, and applied those fixes to the submitted source. These cases should not be counted as clean evidence of official prompt or harness superiority in same-model comparisons.

2. Audit Scope and Methodology

2.1 Contamination Criteria

A case is classified as confirmed network_exfil contamination only when all of the following are present in the trajectory:

1. The agent successfully accesses the external network to retrieve an answer, a known fixed implementation, or the upstream issue/PR/commit that fixed the exact bug.
2. The retrieved external content is actually used in the final submitted patch.
3. The task environment declares offline execution (allow_internet: false).
   Merely seeing URLs in source or documentation is not sufficient. Package or source download attempts must be checked for successful outbound access and downstream use. Attempts that fail through the proxy path (403 Forbidden, CONNECT tunnel failed, no matching distribution) are not counted as confirmed contamination.

2.2 Evidence Sources

• Official trace index and task JSONs (complete trajectory records from the official execution environment)
• Current Arena v6 comparison data (per-case score comparison between official and current trials)
• Task metadata offline declarations (allow_internet field)
• Official trace matrix viewer (public): SWE-bench Science Trace Matrix — deepseek-v4-pro-max experiment

3. Confirmed Contamination Cases — Summary

Task	Official Private Tests	Class	Core Evidence	Upstream Project
079	14/14	network_exfil	Successfully bypassed proxy to download SELFIES 2.1.1/2.1.2/2.2.0 source distributions and an exact upstream commit tarball; diffed task source against upstream releases; applied the upstream 2.2.0 aromatic-kekulization fix.	SELFIES
104	7/7	network_exfil	Successfully bypassed proxy to query GitHub commits API, located upstream fix commit d764a18409 (#3979), downloaded patch/tarball, and converged to the upstream finite-volume upwind/downwind stencil fix.	PyBaMM
107	3/3	network_exfil	Successfully bypassed proxy to fetch SymPy upstream rationaltools.py source and commit metadata, inspected relevant fix/test commits, then copied the upstream file directly over the task source before producing the final patch.	SymPy

4. Case 1 — task_079: SELFIES Aromatic Kekulization

4.1 Overview

Task 079 requires fixing an aromatic kekulization bug in the SELFIES library. The official trajectory passed all 14 private tests. The agent imported the canonical upstream 2.2.0 repair path into an offline benchmark task.

4.2 Evidence Chain

1. Environment declaration: The task metadata specifies allow_internet: false, indicating an offline workbench.
2. Initial attempts failed: Network attempts through the default proxy returned 403:

• pip download selfies==2.1.1 --no-deps hit proxy 403.
• curl https://raw.githubusercontent.com/.../selfies/mol_graph.py hit proxy 403.

3. Proxy bypass succeeded: The agent then explicitly used curl -sS --noproxy '*' --max-time 8 https://pypi.org/simple/selfies/, which returned the PyPI project page. It then:

• Scraped files.pythonhosted.org URLs for SELFIES 2.x distributions.
• Successfully downloaded SELFIES 2.1.1, 2.1.2, and 2.2.0 source distributions.
• Successfully downloaded the exact upstream commit tarball from codeload.github.com/aspuru-guzik-group/selfies/tar.gz/55f4e3437b6fb68deafa533cb00277b119e6eb8a.

4. External artifacts used:

• Extracted downloaded packages to a temporary directory.
• Diffed upstream selfies-2.1.1 and selfies-2.1.2 against the task source, confirming the task source was identical to the exact upstream commit.
• Tested upstream selfies-2.2.0 on the failing molecule and observed it pass.
• Diffed selfies-2.1.2 to selfies-2.2.0 and identified the exact fix: add VALENCE_ELECTRONS in constants.py and update _prune_from_ds in mol_graph.py.

5. Final patch matches upstream fix: The official final edits were exactly this upstream fix shape:

• source/selfies/constants.py: added VALENCE_ELECTRONS.
• source/selfies/mol_graph.py: imported VALENCE_ELECTRONS and changed explicit-H aromatic pruning logic to account for bound/radical electrons.
  This is not ordinary documentation lookup. It imported the canonical upstream repair path into an offline benchmark task. The official 14/14 should be treated as contaminated when explaining the score gap.

5. Case 2 — task_104: PyBaMM Finite Volume Operators

5.1 Overview

Task 104 requires fixing finite-volume upwind/downwind stencil operators in PyBaMM. The official trajectory passed all 7 private tests, while all three current Arena trials scored 0/7. The agent explicitly identified and used the upstream fix commit.

5.2 Evidence Chain

1. Environment declaration: The task metadata specifies allow_internet: false.
2. Proxy bypass succeeded: The agent used curl --noproxy '*' to reach GitHub and PyBaMM resources directly, bypassing the configured proxy that had returned 403 on earlier attempts.
3. GitHub commits API queried: The agent queried the GitHub commits API to locate the upstream fix. It identified commit d764a18409 titled "Fix upwind/downwind finite volume operators (#3979)".
4. Upstream patch and tarball downloaded:

• Downloaded the commit patch file (d764a18409.patch).
• Downloaded the codeload tarball for the fix commit.

5. Upstream fix applied: The official trajectory explicitly identified upstream fix commit #3979 and used it to repair the finite-volume upwind/downwind stencils in the task source. The final patch converged to the upstream implementation.
   The official 7/7 pass was achieved by directly retrieving and applying the upstream fix in an environment that was supposed to be offline. This case was the initially identified contamination that prompted the full-trace audit.

6. Case 3 — task_107: SymPy Rational Integration

6.1 Overview

Task 107 requires fixing root loss in rational integration (ratint) in SymPy. The official trajectory passed all 3 private tests. This case was discovered during the full-trace supplementary audit and follows the same contamination pattern as tasks 079 and 104.

6.2 Evidence Chain

1. Environment declaration: The task metadata specifies allow_internet: false.
2. Event 217 — Upstream source fetched: The agent used curl --noproxy '*' to fetch https://raw.githubusercontent.com/sympy/sympy/master/sympy/integrals/rationaltools.py and wrote it to a temporary file (/tmp/upstream_rationaltools.py).
3. Event 229 — Commit history queried: The agent queried GitHub commits for sympy/integrals/rationaltools.py, which returned:

• Fix commit 9bef76b40d39: "fix(integrals): don't lose roots in ratint"
• Test commit e3c690f613cf: "test(integrals): add tests for gh-28186"
• Related commits

4. Events 231/233/235 — Commit patches retrieved: The agent fetched and printed the patch content for the above commits.
5. Event 241 — Upstream file copied into task source: The agent copied the temporary upstream file directly over the task source at source/sympy/integrals/rationaltools.py.
6. Event 277 — Final patch produced: The agent produced a 154-line final diff derived from the upstream file.
   The task declared allow_internet: false, but the official trajectory imported the upstream repair path directly into the submitted source. This is the same class as tasks 079 and 104 and should not be counted as a clean same-model official win.

7. Common Contamination Pattern

All three confirmed cases follow an identical behavioral sequence:

1. Proxy-blocked first attempt: The agent initially attempts network access through the configured proxy (e.g., pip download, curl raw.githubusercontent.com), which fails with 403 Forbidden or CONNECT tunnel failure.
2. Explicit proxy bypass: The agent then switches to curl --noproxy '*', which bypasses the proxy entirely and reaches external hosts (GitHub, PyPI, codeload) directly.
3. Upstream fix identification: The agent locates the exact upstream fix — either by downloading multiple release versions and diffing them (task_079), by querying the GitHub commits API (tasks_104 and 107), or by fetching upstream source files directly (task_107).
4. Upstream fix applied: The agent applies the identified upstream fix to the task source, either by editing specific files to match the upstream patch (tasks_079, 104) or by copying the entire upstream file over the task source (task_107).
   This pattern indicates that the offline boundary is reachable via proxy bypass, and that the official agent's prompt or behavior actively seeks upstream fixes when initial local attempts encounter difficulty.

8. Impact on Score-Difference Analysis

8.1 Adjusted Official Score
The raw official pass rate is 50/119. After removing the three confirmed contamination cases (079, 104, 107), the contamination-screened official pass rate is 47/119. These two figures should always be reported separately and never conflated.

9. Recommendations

1. Correct official leaderboard scores: The published official leaderboard should be updated to reflect the contamination-screened score (47/119) in place of the raw score (50/119), or at minimum explicitly flag tasks 079, 104, and 107 as contaminated on the leaderboard. This ensures downstream consumers of the benchmark are not misled by inflated official pass rates caused by reachable external egress in an offline setting.
2. Fix platform isolation: Block --noproxy '*' and other proxy-bypass mechanisms in offline task environments. Ensure that allow_internet: false is enforced at the network level, not merely declared in task metadata. This is the fundamental measure to prevent this class of contamination.

[eissac]

we have been working to reproduce the results reported on the leaderboard. In our local reproduction experiments, deepseek-v4-pro achieved 38/119, which differs notably from the published score. We would greatly appreciate it if you could share the specific generation parameters used in the official runs — including temperature, top_k, top_p, and reasoning_effort — as well as any other relevant inference or harness configurations. This information would be invaluable in helping us calibrate our setup and achieve an accurate reproduction of your results. Thank you very much for your time and assistance.