Transcoder

Transcoder is a preset-driven internal video transcoding platform built around:

• Go API (upload/status/job orchestration)
• Go gRPC worker (doing all of the ffmpeg transcoding)
• Redis stream queue
• MinIO object storage
• Postgres metadata store
• Vue frontend for upload + job monitoring The idea is to have it as a service where companies that require transcoding of their videos (cctv cams, private recordings and videos gotten from lower quality phones) can be transcoded to quality or standards that are easy to translate for editing or privavte work. Everything will be done in your ptivate network. Everything can be configured by the administrator or user.

Reasoning

The transcoder is an attempt at leveraging Go's concurrency primitives and ideas while producing something genuinely useful. This project is home to various ideas and tooling: ffmpeg, concurrency patterns, parallelism, object storage, containerization, web development, queuing system, locks and mutexes, databases, grpc, testing, benchmarking e.t.c

It is simple - A user wants certain videos to be transcoded to a different resolution or format (presets are included, albeit simple). The video is uploaded from device and then the magic happens.

What happens after upload ?

Once the web UI has been accessed and the file correctly uploaded, an acknowledgement will be sent back by the server to say that the video file has been successfully received and processing is being carried out in the background (asynchronously). What happens behind the scenes is classic system design:

• POST /upload/initiate creates a job row in Postgres, starts a multipart upload in MinIO, and returns one presigned PUT URL per part. The client then divides the file into chunks and uploads each part directly to MinIO - the file bytes never touch the API server, eliminating the I/O bottleneck it would otherwise become. Multipart is used because a single PUT is capped at 5GB per object, and it also enables parallel part upload plus retrying failed parts.
• POST /upload/complete stitches the parts together in MinIO, stores the video metadata (preset/format/codec/resolution) in Postgres, pushes the job ID onto a Redis stream, and transitions the job from pending to queued. The worker pool (consumer group on that stream) picks it up and runs the transcode asynchronously.
• Status is reported by polling GET /status/:id/update; the frontend never needs stored presigned URLs for that. When processing is done, a fresh presigned GET URL is generated on demand (GET /jobs/:id/output-url / GET /jobs/:id/download) so the client can fetch the transcoded file.

Once the job ID is in the Redis stream, the worker pool picks it up and transcodes it asynchronously: it fetches the transcode profile from the web API (metadata ultimately sourced from Postgres), streams the source from MinIO, runs ffmpeg, and uploads the output back to MinIO. Concurrency follows a fan-out/fan-in pattern: a dispatcher reads jobs from the stream and feeds a bounded worker pool (one goroutine per worker), each transcoding a different video in parallel. When a worker finishes, it aggregates its result into a shared result channel, which a single handler drains to release the in-flight reservation and ACK the stream message. Synchronization uses channels, sync.WaitGroups, and context cancellation (ctx.Done()) for graceful shutdown. Reliability is handled two ways: dependency failures (network timeouts, upload errors) are retried in-process with exponential backoff plus jitter, while jobs whose worker crashes leave their stream message un-ACKed so the Redis consumer-group auto-claim re-delivers them after a 35-minute idle window. Deterministic ffmpeg failures are not retried.

Highlights

• Multipart upload to MinIO with presigned part URLs
• Job lifecycle status machine (pending -> queued -> downloading -> processing -> uploading -> completed|failed|cancelled)
• Preset catalog with validated overrides
• Async worker pool with retries/backoff on dependency failures
• Health + readiness + basic metrics endpoints

Quickstart (Docker)

1. Copy env file:
   • cp example.env .env
2. Start the full stack:
   • docker compose up --build
3. Verify service endpoints:
   • GET http://localhost:8084/health
   • GET http://localhost:8084/ready
   • GET http://localhost:8084/metrics

Core API Endpoints

• POST /upload/initiate
• POST /upload/complete
• POST /jobs (preset-based job creation)
• GET /jobs/:id/source-url
• GET /jobs/:id/output-url
• GET /jobs/:id/download
• GET /jobs/:id/transcode-profile
• POST /status/:id/update
• GET /status/:id/update
• GET /presets
• GET /presets/:id

Run frontend locally

cd frontend
npm install
npm run dev

Frontend dev server defaults to http://localhost:3000 and proxies API requests to backend.

Frontend request flow

1. POST /upload/initiate
2. Upload parts directly to MinIO with presigned URLs
3. POST /upload/complete with preset_id and optional overrides
4. Poll GET /status/:id/update until terminal state
5. Use GET /jobs/:id/output-url or GET /jobs/:id/download

Preset Submission Example

curl -X POST http://localhost:8084/jobs \
  -H 'Content-Type: application/json' \
  -d '{
    "video_name": "sample.mp4",
    "description": "demo transcode",
    "preset_id": "web-h264-v1",
    "overrides": {
      "bitrate": 2500,
      "resolution": "720"
    }
  }'

Error Contract

API errors include structured error codes in error_code:

• VALIDATION_ERROR
• DEPENDENCY_ERROR
• INVALID_STATE
• INTERNAL_ERROR
• PRESET_NOT_FOUND
• PRESET_OVERRIDE_INVALID
• TRANSCODE_FAILED

Notes

• gRPC protobuf regeneration requires protoc-gen-go and protoc-gen-go-grpc to be installed.
• Repository module path: github.com/franzego/transcoder.