Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion docs/adapters.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# Multi-Framework Durability Adapter Suite

LetItLoop provides native, drop-in durability adapters across the 4 major Python AI agent ecosystems:
LetItLoop provides native, drop-in durability adapters across the major Python AI agent ecosystems and web stacks:
1. **CrewAI** (`CrewAIDurabilityHandler`)
2. **Hugging Face Smolagents** (`SmolagentsWALCallback`)
3. **Microsoft AutoGen 0.4 / Magentic-One** (`AutoGenStateSerializer`)
4. **LangGraph** (`LetItLoopCheckpointSaver`)
5. **FastAPI / Starlette** (`DurableBackgroundTasks`)

All adapters feature **zero mandatory runtime dependencies** (lazy-loaded optional imports), ensuring that `pip install letitloop` stays ultralight while providing full crash resilience.

Expand Down Expand Up @@ -124,3 +125,48 @@ app = builder.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": "session_001"}}
result = app.invoke({"input": "query"}, config=config)
```

---

## 5. ⚡ FastAPI / Starlette: `DurableBackgroundTasks`

FastAPI's built-in `BackgroundTasks` run inside the web-server process and are **lost** if the worker restarts, redeploys, or is killed mid-task. `DurableBackgroundTasks` is a drop-in dependency that records every task to a fsync'd Write-Ahead Log **before** it runs, so an interrupted task is transparently re-run on the next startup — no Redis, no Celery, no separate worker process.

Two layers of durability:
- **At-least-once (task level):** a `TASK_PENDING` record is written before the response is returned, and `TASK_COMPLETED` once the task finishes. On startup, any `PENDING`-without-`COMPLETED` task is resumed.
- **Skip-completed-work (step level):** each task runs inside a per-task `@durable` context, so tasks that call `step()` / `async_step()` fast-forward already-completed steps when resumed.

Because Python callables can't be serialized, a task is referenced by a stable key: the name from `@durable_task(...)`, or an auto-derived `"module:qualname"`. On resume the key is looked up in the registry first, then imported dynamically.

### Quickstart
```python
from fastapi import FastAPI
from letitloop.adapters.fastapi import (
DurableBackgroundTasks,
durable_task,
install_durable_background_tasks,
)

app = FastAPI()

# 1. Attach the WAL-backed manager and wire resume-on-startup.
install_durable_background_tasks(app) # optional: wal_dir="..." / manager=...

# 2. (Optional) give the task a stable key so resume survives renames.
@durable_task()
async def run_durable_report(report_id: str) -> None:
... # use step()/async_step() inside for step-level resume

# 3. Use it exactly like FastAPI's BackgroundTasks — via dependency injection.
@app.post("/generate-report/{report_id}")
async def generate_report(report_id: str, background_tasks: DurableBackgroundTasks):
background_tasks.add_task(run_durable_report, report_id)
return {"status": "queued"}
```

If the process is killed after the response is sent but before `run_durable_report`
finishes, the task is re-run automatically the next time the app starts.

> **Custom lifespans:** `install_durable_background_tasks` wraps the app's existing
> lifespan, so any `lifespan=` you pass to `FastAPI(...)` still runs. To wire resume
> manually instead, call `await manager.resume_pending()` inside your own lifespan.
116 changes: 116 additions & 0 deletions examples/fastapi_durable_background.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
"""FastAPI durable background tasks — survive worker restarts / SIGKILL (issue #93).

FastAPI's built-in `BackgroundTasks` run in the web-server process and are lost if
the worker restarts, redeploys, or is killed mid-task. `DurableBackgroundTasks`
records each task to a fsync'd WAL *before* it runs, so an interrupted task is
transparently re-run on the next startup — zero daemon, zero Redis.

This file shows two things:
1. `build_app()` — the canonical FastAPI wiring from the issue.
2. `demo_crash_recovery()` — a runnable proof that a task recorded but not
completed (a crash) is resumed by a fresh manager (a restart), without a
live server so it runs anywhere.

Usage:
python examples/fastapi_durable_background.py # run the crash-recovery demo
uvicorn examples.fastapi_durable_background:app --reload # serve the real app
"""

from __future__ import annotations

import asyncio
import os
import pathlib
import shutil
import sys
import tempfile

# Ensure workspace root on path when run as `python examples/...`
ROOT = pathlib.Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))

from letitloop.adapters.fastapi import ( # noqa: E402
DurableTaskManager,
durable_task,
install_durable_background_tasks,
)

WAL_DIR_DEFAULT = str(ROOT / ".bench_wal" / "fastapi_demo")


# --- the durable task -------------------------------------------------------


@durable_task("reports.generate")
def run_durable_report(report_id: str, out_path: str) -> dict:
"""Pretend to generate a report, writing a file so we can observe execution."""
with open(out_path, "w", encoding="utf-8") as f:
f.write(f"report:{report_id}")
return {"report_id": report_id, "path": out_path}


# --- 1. the canonical FastAPI app ------------------------------------------


def build_app():
"""Build the FastAPI app exactly as issue #93 describes."""
from fastapi import FastAPI
from letitloop.adapters.fastapi import DurableBackgroundTasks

app = FastAPI()
install_durable_background_tasks(app, wal_dir=WAL_DIR_DEFAULT)

@app.post("/generate-report/{report_id}")
async def generate_report(report_id: str, background_tasks: DurableBackgroundTasks):
out_path = str(pathlib.Path(WAL_DIR_DEFAULT) / f"report_{report_id}.txt")
background_tasks.add_task(run_durable_report, report_id, out_path)
return {"status": "queued"}

return app


# Importable ASGI app for `uvicorn examples.fastapi_durable_background:app`.
try: # pragma: no cover - only when fastapi is installed
app = build_app()
except Exception: # pragma: no cover - fastapi not installed
app = None


# --- 2. runnable crash-recovery proof --------------------------------------


def demo_crash_recovery() -> None:
"""Record a task, simulate a crash before it runs, then resume on 'restart'."""
wal_dir = tempfile.mkdtemp(prefix="letitloop_fastapi_demo_")
out_path = os.path.join(wal_dir, "report_42.txt")
try:
print(f"[demo] wal_dir={wal_dir}")

# Run 1: a request arrives — the task is written to the WAL *before* running.
print("[demo] 1) Request received: recording task to WAL, then 'crashing'...")
manager = DurableTaskManager(wal_dir=wal_dir)
key = manager.key_for(run_durable_report)
manager.record_pending(key, [42, out_path], {})
assert not os.path.exists(out_path), "task must not have run yet"
print(f"[demo] pending tasks on disk: {len(manager.pending_tasks())} (report not generated)")

# 2) Server restarts: a fresh manager reads the same WAL.
print("[demo] 2) Server restarts: new manager reads the WAL and resumes...")
recovered = DurableTaskManager(wal_dir=wal_dir)
resumed = asyncio.run(recovered.resume_pending())

# 3) The interrupted task ran to completion.
assert resumed == 1, f"expected 1 resumed task, got {resumed}"
assert os.path.exists(out_path), "report should exist after resume"
with open(out_path, encoding="utf-8") as f:
content = f.read()
assert recovered.pending_tasks() == [], "no tasks should remain pending"
print(f"[demo] resumed {resumed} task(s); report content = {content!r}")
print("[demo] SUCCESS — task interrupted by a crash was resumed on restart, 0 data loss")
finally:
shutil.rmtree(wal_dir, ignore_errors=True)


if __name__ == "__main__":
demo_crash_recovery()
14 changes: 13 additions & 1 deletion letitloop/adapters/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
"""letitloop/adapters — Official Multi-Framework Durability Adapter Suite.

Provides drop-in, zero-daemon WAL durability handlers and checkpointers across
major multi-agent frameworks:
major multi-agent frameworks and web stacks:
- CrewAI (`CrewAIDurabilityHandler`)
- Hugging Face Smolagents (`SmolagentsWALCallback`)
- Microsoft AutoGen 0.4 / Magentic-One (`AutoGenStateSerializer`)
- LangGraph (`LetItLoopCheckpointSaver`)
- FastAPI / Starlette (`DurableBackgroundTasks`, `DurableTaskManager`)

All adapters support zero mandatory runtime dependencies with optional lazy loading.
"""
Expand All @@ -16,6 +17,12 @@

from .autogen import AutoGenStateSerializer
from .crewai import CrewAIDurabilityHandler
from .fastapi import (
DurableBackgroundTasks,
DurableTaskManager,
durable_task,
install_durable_background_tasks,
)
from .langgraph import LetItLoopCheckpointSaver
from .smolagents import SmolagentsWALCallback

Expand All @@ -24,6 +31,10 @@
"SmolagentsWALCallback",
"AutoGenStateSerializer",
"LetItLoopCheckpointSaver",
"DurableBackgroundTasks",
"DurableTaskManager",
"durable_task",
"install_durable_background_tasks",
"get_available_adapters",
]

Expand All @@ -35,4 +46,5 @@ def get_available_adapters() -> Dict[str, bool]:
"smolagents": SmolagentsWALCallback.is_available(),
"autogen": AutoGenStateSerializer.is_available(),
"langgraph": LetItLoopCheckpointSaver.is_available(),
"fastapi": DurableTaskManager.is_available(),
}
Loading