SWE-Gate

This package contains the SWE-Gate instance-construction scripts, 303 released instances, a mini_swe_agent runtime, and deterministic patch evaluation tools. Run all commands from the package root. Internal paths and identifiers retain the historical swe_if name for compatibility with the released artifacts.

Setup

Requirements:

• Python 3.10-3.12
• Node.js 18 or later for decoding the compressed model predictions
• Git
• Docker Engine and access to the Docker daemon
• Network access to GitHub, container registries, and Python package indexes
• Model credentials for instance construction or agent execution
• An authenticated codex CLI for instance construction stages 4-5

Install the Python dependencies:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Released Artifact Layout

The main released artifacts are:

data/dataset/swe_if_instances.json       canonical 303-instance dataset
data/instances/<instance_id>/            per-instance tests, patches, and metadata
data/docker_contexts/                     99 reproducible base-image contexts
data/model_predictions/                   eight final model-condition JSONL files
data/dataset_info.json                    release counts and optional-file coverage

The prediction archive contains four models under with_constraints and no_constraints, with 303 records in every JSONL file. Patches use lossless Brotli plus RFC 1924 Base85 encoding. Decode and integrity-check one patch with:

node data/model_predictions/decode_patch.js \
  data/model_predictions/with_constraints/gpt-5.5.jsonl \
  INSTANCE_ID > prediction.patch

Of the 303 instances, 255 include a standalone validation_matrix.json. The remaining 48 are listed under missing_optional_files in data/dataset_info.json; their files have not been reconstructed or fabricated. See docs/validation_coverage.md for the scope of this limitation.

Third-party code notices and license copies are provided in THIRD_PARTY_NOTICES.md and third_party_licenses/.

1. Construct New Instances

config/seed_repos.txt lists repositories whose pull-request discussions are used to extract constraints. config/instance_repos.txt lists repositories in which new instances can be constructed. Edit copies of these lists when using a different repository set.

First clone the target repositories and write their local path mapping:

export PACKAGE_ROOT="$(pwd)"
export WORK_ROOT="${PACKAGE_ROOT}/runs/instance_generation"
export TARGET_REPOS="${PACKAGE_ROOT}/runs/target_repositories"

mkdir -p "${WORK_ROOT}"
python scripts/swe_if_pipeline/00_prepare_local_repos.py \
  --repos-file config/instance_repos.txt \
  --target-dir "${TARGET_REPOS}" \
  --output-repo-paths "${WORK_ROOT}/repo_paths.txt" \
  --status-json "${WORK_ROOT}/repo_prepare_status.json" \
  --depth 0

Provide a GitHub token file and an API key for an OpenAI-compatible chat completion endpoint. Both files must remain outside this package. Then run the construction pipeline until at least one instance passes validation:

python scripts/swe_if_pipeline/run_pipeline_until_target.py \
  --repos-file "${PACKAGE_ROOT}/config/seed_repos.txt" \
  --repo-paths-file "${WORK_ROOT}/repo_paths.txt" \
  --work-dir "${WORK_ROOT}" \
  --github-token /secure/path/github_token.txt \
  --llm-api-key-file /secure/path/model_key.txt \
  --llm-base-url https://api.deepseek.com \
  --llm-model deepseek-chat \
  --llm-cache-dir "${WORK_ROOT}/.llm_cache" \
  --target-instances 1 \
  --target-seeds 10 \
  --instances-per-seed 2 \
  --max-rounds 1 \
  --workers 1 \
  --build-docker

The endpoint must implement /v1/chat/completions; the script appends that path to --llm-base-url. Stages 4-5 invoke codex exec to generate and validate candidate artifacts. Model or validation failures stop the pipeline; the wrapper does not enable heuristic generation fallback.

Important outputs under ${WORK_ROOT} are:

rawdata/                         fetched pull-request and issue records
constraint_seeds.json           selected construction seeds
instances/<instance_id>/        generated instance artifacts
validated_instances.json        instances that passed validation
rejected_instances.json         rejected candidates and reasons
docker/                          generated Docker contexts and image mapping
dataset/swe_if_instances.json    generated dataset metadata

Use --resume to continue the same work directory. Prompt templates used by the construction stages are stored in prompt/.

2. Run the Released Instances

The canonical dataset is data/dataset/swe_if_instances.json. Each row maps an instance to one of the 99 base Dockerfiles through its docker_build object. Each instance also stores the same mapping in data/instances/<instance_id>/docker_build.json.

Run a host and Docker preflight without calling an LLM:

python scripts/swe_if_pipeline/13_run_swefactory_queue.py \
  --output-dir runs/preflight \
  --preflight-only

Run one released instance with mini_swe_agent:

python scripts/swe_if_pipeline/13_run_swefactory_queue.py \
  --dataset-json data/dataset/swe_if_instances.json \
  --context-root data \
  --output-dir runs/mini_swe_agent \
  --instance-ids 02506978e8943df4_nltk_weighted_choice_bins \
  --llm-provider deepseek \
  --llm-model deepseek-chat \
  --api-key-file /secure/path/model_key.txt \
  --workers 1 \
  --build-slots 1 \
  --base-workers 1

The runner uses LiteLLM provider names. Use --api-base-url for a custom OpenAI-compatible endpoint, or --api-key-env VARIABLE_NAME to read a key from the environment. Run python scripts/swe_if_pipeline/13_run_swefactory_queue.py --help for all resource and model options.

For every selected instance, the runner builds the mapped base and instance images, validates the environment with the gold patch, runs mini_swe_agent, exports the predicted patch, and evaluates it without an LLM. Remove --instance-ids to process all 303 instances.

Results are written under the selected output directory:

report.json
report.md
predictions.jsonl
instances/<instance_id>/patch.diff
instances/<instance_id>/prediction.json
instances/<instance_id>/eval_result.json
instances/<instance_id>/done.json

Completed instances are skipped when the same output directory is resumed. Use --rerun-completed to run them again.

3. Evaluate Prediction Patches

Evaluation uses Docker and does not call an LLM. Build and gold-validate the required image first. The following command handles one selected instance:

python scripts/swe_if_pipeline/12_rebuild_docker_images_from_contexts.py \
  --dataset-json data/dataset/swe_if_instances.json \
  --context-root data \
  --instance-ids 02506978e8943df4_nltk_weighted_choice_bins \
  --output-json runs/docker_build.json

Prediction input is JSONL with one object per line:

{"instance_id":"INSTANCE_ID","model_name":"MODEL_ID","model_patch":"diff --git ..."}

Evaluate the predictions:

python scripts/swe_if_pipeline/eval_swe_if.py \
  --predictions-jsonl runs/predictions.jsonl \
  --dataset-json data/dataset/swe_if_instances.json \
  --output-json runs/evaluation.json \
  --output-jsonl runs/evaluation.jsonl

The result records patch application, functional-test status, constraint-test status, and evaluator errors for each instance. Test-file changes are removed from model patches by default. Gold patches can be checked with:

python scripts/swe_if_pipeline/eval_swe_if.py \
  --self-test-gold \
  --dataset-json data/dataset/swe_if_instances.json \
  --output-json runs/gold_evaluation.json \
  --output-jsonl runs/gold_evaluation.jsonl

To regenerate the concise execution report from an existing agent run without calling an LLM or rerunning evaluation:

python scripts/swe_if_pipeline/14_generate_agent_run_report.py \
  --run-dir runs/mini_swe_agent

The bug-injecting artifact is consistently named mutant.patch throughout the released instances and scripts.