Skip to content
BranchesTags

Repository files navigation

AI Image Understanding & Content Matching Engine

I built a FastAPI service that looks at a small licensed-free image library, writes validated tags and captions, embeds those captions next to blog posts I authored, and refuses a wrong pairing (a wolf on a red-fox article) with an explanation.

Good suggestions when confident. Safe rejection when not.

Top-1 precision (labeled family match): 100% (10/10) on 2026-08-15 with AI_STUB=1 after seed + pipeline. I re-run python scripts/run_eval.py after a live Ollama vision/embed pass — that measured number is the demo closer.

Suggested public repo name: flyrank-capstone-image-relevance.

Interactive API docs (once the server is running): http://127.0.0.1:8000/docs

Goal, in plain language

This is not an image search engine. It is a small trustworthy matcher:

A blog post about red foxes should get a red-fox photo. A similar-looking wolf must be refused. If nothing is good enough, say so instead of guessing.

I own two separate datasets:

What I created What I do not assume
~50 photos on disk (Pexels) That a file named brown-bear-….jpg is actually a bear until vision looks at it
~10 short articles (fox, wolf, dog, bear, deer, coffee) That the article is the source of the photo’s tags

Then three steps happen:

  1. Understand the photo (slow, background) — vision AI writes {subject, category, attributes, caption, confidence}. Invalid JSON is thrown away.
  2. Understand the article (slow, background) — llama3.2 extracts expected subject; both caption and article become embeddings.
  3. Match with a safety gate (fast, on the API) — rank by meaning, then the mismatch guard accepts or rejects with a reason.

If I only start uvicorn and never run a job, posts and image pointers exist, but there are no tags and no rankings yet.

What happens, step by step

I do                 System                         Result in Postgres
────────────────     ──────────────────────────     ─────────────────────────
download photos  →   files on disk                  (not in git)
seed.py          →   posts + image file paths       no vision tags yet
POST /jobs  or   →   background pipeline            tags, embeddings, costs
worker.py
GET /posts/id/images → cosine rank + guard          suggestions (accept/reject)
POST .../approve →   human review row               audit trail

Background work (yes, I added it). Details are in Background jobs below.

Background jobs

Vision, subject extract, and embeddings are slow. They never run inside GET /posts/{id}/images. That GET only ranks what a job already stored.

If I only start uvicorn and never enqueue a job, posts and image paths exist, but there are no tags and no rankings.

Job types

type What it does
vision Send each pending / failed image to llava, validate JSON, store tags
extract llama3.2 writes each post’s expected subject
embed Turn captions and post text into vectors
pipeline All three, in that order (this is what I run for the demo)

Vision only looks at images with status pending or failed. Already processed photos are skipped — seed does not reset that.

Status on GET /jobs/{id}

status Meaning
queued Row exists; nobody is working it yet
running Someone is calling Ollama right now
succeeded Finished. Reusing the same Idempotency-Key will not run it again
failed Error (timeout, schema, budget). CLI --once can claim it

Example while llava is tagging:

{
  "id": 8,
  "type": "pipeline",
  "status": "running",
  "progress": { "done": 1, "total": 16, "failed": 0, "flagged": 0 },
  "attempts": 1
}

done / total is pending images only, not the whole library. Llava on this laptop is often 1–3+ minutes per photo. I poll until status is succeeded. I do not Ctrl+C uvicorn while it is running.

Pick one runner — never both

POST /jobs (while uvicorn is up) already starts FastAPI BackgroundTasks. The job goes queued → running in a second.

Situation What I run
Uvicorn is running POST /jobs only, then GET /jobs/{id}
Uvicorn is off python scripts/worker.py --pipeline or --once
First seed, before the API python scripts/worker.py --pipeline

If I POST /jobs and then python scripts/worker.py --once, the CLI prints no queued jobs. That is correct: uvicorn already set the job to running.

--once only claims queued or failed. It does not create a job. --pipeline creates key cli-pipeline; if that already succeeded, it prints replayed=True and does nothing — I use a new Idempotency-Key on POST /jobs instead.

Start and watch (uvicorn already up)

curl -s -X POST http://127.0.0.1:8000/jobs \
  -H "X-API-Key: dev-local-key" \
  -H "Idempotency-Key: vision-retry-2" \
  -H "Content-Type: application/json" \
  -d '{"type":"pipeline"}'

# use the id from the response (e.g. 8)
curl -s http://127.0.0.1:8000/jobs/8 -H "X-API-Key: dev-local-key"

Same Idempotency-Key twice → same job (replayed: true), not a second vision run. A new key is required after photos change.

After I replace photos

Seed only updates file paths. Old tags stay until I reset and run a new job:

UPDATE images SET status = 'pending';
DELETE FROM image_embeddings;
DELETE FROM image_metadata;

Then POST /jobs with a new Idempotency-Key.

Timeouts and Ctrl+C

ReadTimeout means llava did not answer within OLLAMA_TIMEOUT_S (default 120s). Raise it in .env (e.g. 300) and restart uvicorn.

Ctrl+C on uvicorn prints Waiting for background tasks to complete. A second Ctrl+C kills the job and can leave status: running. Unstick:

UPDATE jobs SET status = 'failed' WHERE status = 'running';

Then, with uvicorn off: python scripts/worker.py --once.

APIs I run

Auth: mutating routes need header X-API-Key: dev-local-key (from .env API_KEY). That is not a Pexels or Ollama key.

Order Call What I should see
0 Open /docs Clickable list of every route
1 GET /health Server is up
2 POST /jobs body {"type":"pipeline"} + Idempotency-Key 202 Job queued — this is the background job
3 GET /jobs/{id} status: succeeded and progress counts
4 GET /posts 10 articles; I note the fox, wolf, and coffee ids
5 GET /images Tags/captions after the job; at least one flagged if confidence was low
6 GET /posts/{fox_id}/images Fox photo on top; wolf rejected or ranked far below
7 POST /posts/{fox_id}/candidates/{wolf_image_id}/evaluate decision: reject + subject-mismatch explanation
8 GET /posts/{coffee_id}/images match_status: no_confident_match
9 POST /suggestions/{id}/approve or /reject Human review saved
10 GET /costs Every vision/embed/extract call attributed (cost_usd is 0 locally)

Copy-paste (I replace IDs from steps 4–5):

# 1. Health — no key
curl -s http://127.0.0.1:8000/health

# 2–3. Background pipeline (wait until GET /jobs/ID shows succeeded)
curl -s -X POST http://127.0.0.1:8000/jobs \
  -H "X-API-Key: dev-local-key" \
  -H "Idempotency-Key: demo-pipeline-1" \
  -H "Content-Type: application/json" \
  -d '{"type":"pipeline"}'

curl -s http://127.0.0.1:8000/jobs/1 \
  -H "X-API-Key: dev-local-key"

# 4–6. Browse, then match a fox article
curl -s http://127.0.0.1:8000/posts
curl -s http://127.0.0.1:8000/images
curl -s http://127.0.0.1:8000/posts/1/images

# 7. Force the wolf onto the fox post (the demo moment)
curl -s -X POST http://127.0.0.1:8000/posts/FOX_ID/candidates/WOLF_ID/evaluate \
  -H "X-API-Key: dev-local-key"

# 8–10. No-match, review, cost log
curl -s http://127.0.0.1:8000/posts/COFFEE_ID/images
curl -s -X POST http://127.0.0.1:8000/suggestions/SUGGESTION_ID/approve \
  -H "X-API-Key: dev-local-key" -H "Content-Type: application/json" \
  -d '{"note":"fox is correct"}'
curl -s http://127.0.0.1:8000/costs -H "X-API-Key: dev-local-key"

Keys I may need outside the API:

Key / tool Where Used for
PEXELS_API_KEY pexels.com/api (free, no card) scripts/download_pexels.py only
API_KEY I set it in .env (default dev-local-key) X-API-Key on jobs, evaluate, review, costs
Ollama No key Local llava + nomic-embed-text + llama3.2

Root-level diagram

flowchart LR
  Client[HTTP client]
  API[FastAPI]
  PG[(Postgres 16)]
  Ollama[Ollama]
  Pexels[Pexels API]
  Disk[data/images]

  Client --> API
  API --> PG
  API --> Ollama
  Seed[scripts/seed.py] --> PG
  Download[scripts/download_pexels.py] --> Pexels
  Download --> Disk
  Worker[scripts/worker.py] --> Ollama
  Worker --> PG
  Worker --> Disk
Loading

Ollama is three roles, not one model:

Role Model (env) On this machine?
Vision VISION_MODEL (llava or llama3.2-vision) I pull this
Embeddings EMBED_MODEL (nomic-embed-text or all-minilm) I pull this
Post subject TEXT_MODEL (llama3.2) Yes

Until I pull the extra models, I set AI_STUB=1. Stubs still exercise schema validation, ranking, and the fox/wolf guard.

Activity diagram

flowchart TB
  subgraph ingest [Image stream]
    Img[Pexels file on disk]
    Vision[Vision model]
    Schema[Pydantic ImageMetadata]
    StoreMeta[image_metadata]
    EmbImg[embed caption]
    Img --> Vision --> Schema
    Schema -->|invalid| Retry[Retry or flag failed]
    Schema -->|valid| StoreMeta --> EmbImg
  end

  subgraph posts [Post stream]
    Post[Authored article]
    Extract[llama3.2 subject extract]
    EmbPost[embed title + body]
    Post --> Extract --> EmbPost
  end

  EmbImg --> Rank[Cosine ranking]
  EmbPost --> Rank
  StoreMeta --> Guard[Mismatch guard]
  Extract --> Guard
  Rank --> Guard
  Guard -->|accept| Suggest[Ranked suggestion]
  Guard -->|reject| Explain[No confident match + reason]
  Suggest --> Review[Approve or reject]
  Explain --> Review
Loading

Two validations stay separate: schema (immediately after vision) vs guard (at match time).

Architectural decisions

A1 — Vision is untrusted I/O. app/schemas/ai.py is the trust boundary. confidence must be a float in [0,1]. "hello" is a validation failure, not a tag.

A2 — Two embedding streams, one space. I embed captions and post text with the same model. I do not embed pixels.

A3 — The guard is rules. llama3.2 only extracts expected subject. app/services/subject_map.py maps red fox / Vulpes vulpes → fox and gray wolf / Canis lupus → wolf. Similarity cannot override a family mismatch.

A4 — float8[] in Postgres, no pgvector. Assignment allows in-DB arrays at ~50 images. Ranking happens in app/services/ranker.py. Reuses postgres:16 already on the machine.

A5 — Metadata after vision only. scripts/seed.py writes posts and image pointers. image_metadata / embeddings are written by the batch job after validation.

A6 — Batch jobs own model calls. POST /jobs enqueues; FastAPI BackgroundTasks + scripts/worker.py run vision/extract/embed with retries. The request path never blocks on Ollama.

A7 — Cost is metered at $0. Every call is a row in ai_cost_events (model, tokens, duration, image_id or post_id). Budget = MAX_CALLS_PER_JOB.

A8 — Pexels IDs are pinned. data/corpus/images.json lists 50 IDs tied to post slugs. Images are gitignored.

A9 — Layered FastAPI. HTTP / services / AI adapters / DB. I can swap Ollama or Postgres without touching the guard.

A10 — Default tenant. tenant_id on domain rows. Demo tenant slug: demo. Mutating routes require X-API-Key.

First-time setup

cp .env.example .env
# I set PEXELS_API_KEY if I will download photos.
# I set AI_STUB=1 until I have pulled vision + embedding models.

docker compose up -d
python scripts/download_pexels.py   # optional when AI_STUB=1
python scripts/seed.py
python scripts/worker.py --pipeline   # CLI background pipeline (or use POST /jobs later)
uvicorn main:app --reload --port 8000

Then I follow APIs I run above (or open /docs).

When I have pulled models:

ollama pull llava
ollama pull nomic-embed-text
# keep llama3.2 as-is

Then I set AI_STUB=0 in .env and run the pipeline job again so tags come from real vision.

Eval:

python scripts/run_eval.py
pytest -q

Layout

app/           HTTP, services, AI adapters, DB
data/corpus/   posts.json, images.json
data/eval/     labels.json
data/images/   gitignored downloads
db/migrations/ 001_init.sql
prompts/       vision-v1.md, extract-v1.md
scripts/       download_pexels.py, seed.py, worker.py, run_eval.py
tests/         schema, guard, matching, parse

Limitations

  • Scope is ~50 images and 10 posts. Not an image search engine.
  • No frontend; review is API-only.
  • Local vision quality depends on the model I pull. The guard is what makes fox/wolf safe when the embedder thinks they are similar.
  • Stub embeddings are keyword axes, not a neural space. I use them for tests and a first demo, then switch AI_STUB=0.
  • Some historical Pexels IDs 404; the download script falls back to query search.
  • Isolated tenants are a tenant_id column plus a default slug, not a full multi-tenant product.

License

MIT. Photos remain under the Pexels license.

About

I built a FastAPI service that looks at a small licensed-free image library, writes validated tags and captions, embeds those captions next to blog posts I authored, and refuses a wrong pairing (a wolf on a red-fox article) with an explanation.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages