Graphectory transforms agent execution traces into structured graphs that capture the problem-solving patterns of AI software engineering agents. By modeling agent actions as directed graphs with phase classification (localization, patching, validation), this tool enables systematic analysis of how agents solve software engineering tasks.
Graphectory is very easy to adopt (please see "Supporting New Agents" and "Supporting New SWE Agent Tools" in the ReadMe). If you have any question or need help, please post on the issue tracker with a sample of your trajectory and we would be happy to assist.
New: Beyond the two agent frameworks studied in the paper (SWE-agent and OpenHands), the repository additionally supports mini-swe-agent (v2.0.0, trajectory_format version mini-swe-agent-1.1; and trajectory_format version mini-swe-agent-1), a widely used scaffold in agentic research with over 3.3k GitHub stars.
Pre-computed Graphs: Full dataset (2 agents × 4 models) available under data/{OpenHands|SWE-agent}/graphs
Raw Trajectories: Hosted on Zenodo due to file size: https://zenodo.org/records/17364210
git clone git@github.com:Intelligent-CAT-Lab/Graphectory.git
cd GraphectoryWe provide a Dockerfile which includes the pre-computed graphs and installs all necessary dependencies to reproduce the results of Graphectory. Please download Docker, and then build and run:
docker build -t graphectory .
docker run -it graphectory bashMulti-platform: Works on Intel and Apple Silicon. For Apple Silicon, use:
docker build --platform linux/arm64 -t graphectory .For Docker workflows, see DOCKER.md. If interested in interactive graph construction, see graph_construction/README.md.
Requires Python ≥ 3.12. We recommend using conda or virtual environments:
conda create -n graphectory python=3.12 && conda activate graphectory
python -m pip install -e .PyGraphviz Note (Required for Live Visualization): On Windows, standard pip install often fails due to missing Graphviz C-libraries. Using conda is recommended:
conda install -c conda-forge pygraphvizOtherwise, install Graphviz system binaries manually before python -m pip install -e .
Graphectory provides two tools for working with agent trajectories:
- generatejson.py: Batch export graphs to JSON files
- live_graph_server.py: Interactive browser-based graph visualization
For detailed usage and configuration options, see graph_construction/README.md.
Pre-computed analysis results for the full dataset are available under data/{OpenHands|SWE-agent}/analysis, including Graphectory metrics.
python -m graph_analysis.batch_runnerpython -m graph_analysis.batch_runner --data-dir ./my_graphs --output-dir ./my_outputResults are saved to trajectory_metrics.csv.
Precomputed graphs are provided under data/{OpenHands|SWE-agent}/graphs. The reproduction pipeline has three optional stages:
Requires precomputed analysis in data/{agent}/analysis/{model}/:
bash scripts/reproduce.sh # Generate figures in figures/
bash scripts/reproduce.sh -o ./my_output # Custom output directoryAll paper figures (RQ1-RQ3):
- RQ1:
figures/median_iqr_trajectory_heatmap.png(Figure 3) - RQ2:
figures/sankey_grid.png(Figure 7),figures/end_phase_donuts.png(Figure 8),figures/phase_transition_overview.png(Figure 9) - RQ3:
figures/inefficiency_venn/*.pdf(Figures 14-15)
To generate graphs from raw trajectories (requires Zenodo data https://zenodo.org/records/17364210 or data/samples/):
# Generate graphs from trajectories
bash scripts/construct.sh <trajectories_path> <eval_report.json> [output_dir] [model]
# Analyze graphs and compute metrics
bash scripts/analyze.sh <data_dir> [output_dir] [--agent AGENT] [--model MODEL]
# Generate figures
bash scripts/reproduce.shFor script usage and options:
bash scripts/construct.sh -h # Show construct options
bash scripts/analyze.sh -h # Show analyze options
bash scripts/reproduce.sh -h # Show reproduce optionsFor interactive graph visualization, see graph_construction/README.md.