AgentFAIR

AgentFAIR combines deterministic metadata extraction, specialized LLM agents, and LangGraph workflows to assess dataset landing pages against the 13 FAIR sub-principles. It includes a local FastAPI backend and a React/Vite dashboard.

Screenshots

Landing page	Dataset input	Evaluation results

Features

• Hybrid extraction from rendered HTML, meta tags, JSON-LD, RDF/XML, DataCite, microdata, and RDFa.
• Dedicated evaluators for F1–F4, A1.1–A2, I1–I3, and R1.1–R1.3.
• Critic feedback loops for evidence sufficiency and scoring consistency.
• Markdown/JSON reports, SQLite evidence storage, and optional local/LangSmith traces.
• A browser dashboard backed by a local-only API.

Architecture

The evaluator renders the supplied landing page with Playwright, extracts a unified metadata record, and runs four FAIR-dimension workflows concurrently. Each workflow evaluates its sub-principles, optionally retries after critic feedback, and emits structured scores and evidence. Results are stored locally and exposed to the dashboard through FastAPI.

Path	Purpose
fair_agents/agents/	Principle-specific evaluators and critic agents
fair_agents/orchestration/	LangGraph workflows for F, A, I, and R
fair_agents/tools/	Crawling, parsing, identifier, vocabulary, and trace utilities
fair_agents/api/	Local FastAPI service
fair_agents/storage/	SQLite access code
fair_agents/prompts/	Prompt templates
fair_agents/reports/	Generated reports (ignored except .gitkeep)
fair_agents/traces/	Generated local traces (ignored except .gitkeep)
frontend/fair-evaluation-frontend/	React/Vite dashboard

Quick start

The launcher targets Python 3.11 and uses the checked-in uv.lock for reproducible dependencies. It can bootstrap curl, uv, Python, and Playwright where supported; review the script before running it on a managed machine because Playwright's --with-deps setup may install operating-system packages.

git clone https://github.com/MingCHEN-Github/AgentFAIR.git
cd AgentFAIR

cp .env.example .env
# Add OPENAI_API_KEY to .env, then:
./run.sh

The backend starts on http://127.0.0.1:8000, and the launcher opens the hosted dashboard. Stop it with Ctrl+C.

For local frontend development:

cd frontend/fair-evaluation-frontend
npm ci
npm run dev

Configuration and credential safety

Use a local .env file for API keys. .env and common variants are ignored by git. The dashboard also accepts keys for a single browser session, but .env is preferred: code served by any hosted frontend can read values entered into that page.

Important variables are documented in .env.example:

• OPENAI_API_KEY for the primary model provider.
• OPEN_ROUTES_API_KEY or OPENROUTER_API_KEY for the optional fallback provider.
• BRAVE_API_KEY for search enrichment.
• LANGSMITH_* and LANGCHAIN_* for opt-in remote tracing. Tracing is off by default because prompts and dataset metadata may be sensitive.
• DATABASE_URL for the local SQLite file.
• API_HOST, API_PORT, FRONTEND_URL, and CORS_ORIGINS for local deployment.

The API intentionally binds to loopback and rejects dataset URLs that resolve to private or non-public IP addresses. It is not hardened for public or multi-user deployment: do not expose port 8000 to a network without adding authentication, rate limiting, redirect-aware SSRF protections, and isolated per-request credentials.

CLI usage

# One URL
uv run fair-evaluate --url "https://zenodo.org/records/3940430"

# Multiple URLs
uv run fair-evaluate --urls "https://zenodo.org/records/3940430" "https://doi.org/10.1594/PANGAEA.971421"

# API without the launcher
uv run fair-api

Generated runtime data is written to:

• fair_agents/reports/evaluation_<timestamp>/
• fair_agents/logs/
• fair_agents/traces/
• storage/fair_evaluator.db

These paths are ignored by git because they may contain retrieved dataset content, model inputs/outputs, and other sensitive evidence.

Development checks

uv sync --extra dev
uv run python -m compileall -q fair_agents
uv run ruff check fair_agents

cd frontend/fair-evaluation-frontend
npm ci
npm run lint
npm run build
npm audit --omit=dev

The repository has only a small URL-safety unit test, not an automated benchmark or comprehensive test suite. The manuscript's aggregate results therefore cannot be regenerated from this checkout alone; raw benchmark outputs and analysis notebooks are not included.

Paper and citation

@misc{chen2026agentfairmultiagentcollaborativeframework,
      title={AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets}, 
      author={Ming Chen and Pranav Pai},
      year={2026},
      eprint={2607.15781},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.15781}, 
}

License

The software is licensed under AGPL-3.0-only; see LICENSE. Manuscript and research-artifact licensing is separate from this software license.