Saltbox Docker Controller (SDC)

A dependency-aware Docker container orchestrator for Saltbox, written in Go.

Overview

Saltbox Docker Controller manages Docker container startup and shutdown based on dependency labels. It starts one lifecycle job at a time, prevents dependency failures from cascading into unsafe work, and handles shutdown in reverse dependency order.

Features:

• Dependency-aware orchestration using Docker labels
• Topological sort for optimal startup/shutdown order
• Parallel execution of independent containers
• Health check polling bounded by the operation timeout
• Job-based API for async operations
• Block/unblock operations for maintenance windows
• REST API server mode
• Helper mode for Docker daemon lifecycle integration
• Graceful shutdown handling
• Deterministic lifecycle, API, client, and Docker-adapter tests

Building

Build using the Makefile:

make build

The binary will be created at build/sdc.

For all available targets:

make help

Testing

Run all tests:

make test

Run tests with coverage:

make test-coverage

Usage

Server Mode

Start the REST API server:

./build/sdc server --host 127.0.0.1 --port 3377

Or use the Makefile:

make run-server

Helper Mode

Run the helper daemon for automatic lifecycle management:

./build/sdc helper --controller-url http://127.0.0.1:3377

Or use the Makefile:

make run-helper

Version Information

./build/sdc --version

Architecture

Package Structure

sdc/
├── cmd/controller/         # Main entry point (server/helper commands)
├── internal/
│   ├── api/               # HTTP handlers, middleware, and router
│   ├── client/            # HTTP client for helper mode
│   ├── config/            # Configuration management
│   ├── docker/            # Moby adapter for the local Docker daemon
│   └── lifecycle/         # Planning, execution, jobs, blocking, and shutdown
└── pkg/logger/            # Key-value logging over the standard library logger

How It Works

1. Label-based dependencies: Containers declare dependencies via com.github.saltbox.depends_on labels
2. Lifecycle planning: SDC builds and validates a deterministic dependency plan
3. Safe execution: Failed or unavailable dependencies block affected work rather than failing open
4. Bounded parallelism: Independent containers in a batch run concurrently, while complete jobs remain FIFO
5. State-aware ignores: Ignored running containers protect dependencies needed to keep them running
6. Job tracking: Jobs retain legacy result arrays and add per-container typed outcomes

Docker Labels

Saltbox Docker Controller uses Saltbox Docker labels to define container management and dependencies:

labels:
  com.github.saltbox.saltbox_managed: "true"              # Required: Enable SDC management
  com.github.saltbox.saltbox_controller: "true"           # Optional: Enable/disable controller (default: true)
  com.github.saltbox.depends_on: "postgres,redis"         # Optional: Comma-separated dependencies
  com.github.saltbox.depends_on.delay: "5"                # Optional: Startup delay in seconds
  com.github.saltbox.depends_on.healthchecks: "true"      # Optional: Wait for healthchecks (default: false)

Example docker-compose.yml:

services:
  postgres:
    image: postgres:15
    labels:
      com.github.saltbox.saltbox_managed: "true"
      com.github.saltbox.depends_on.delay: "10"

  redis:
    image: redis:7
    labels:
      com.github.saltbox.saltbox_managed: "true"
      com.github.saltbox.depends_on.delay: "5"

  app:
    image: myapp:latest
    labels:
      com.github.saltbox.saltbox_managed: "true"
      com.github.saltbox.depends_on: "postgres,redis"
      com.github.saltbox.depends_on.delay: "2"
      com.github.saltbox.depends_on.healthchecks: "true"

Saltbox Docker Controller will ensure postgres and redis start first (in parallel), wait for health checks and startup delays, then start app.

SDC connects only to Saltbox's rootful local Docker socket. DOCKER_HOST and remote Docker TLS configuration are intentionally not part of the controller contract.

API Endpoints

Container Operations

• POST /start - Start containers in dependency order
  • Query params: ?timeout=600 (optional, default: 600)
  • Response: {"job_id": "uuid", "status": "pending"}
  • Returns HTTP 503 if operations are blocked
• POST /stop - Stop containers in reverse dependency order
  • Query params: ?timeout=300&ignore=container1&ignore=container2 (optional, default timeout: 300)
  • JSON body: {"shutdown": true} makes this request the helper shutdown gate
  • Response: {"job_id": "uuid", "status": "pending"}
  • Returns HTTP 503 if operations are blocked

Block/Unblock Operations

• POST /block/{duration} - Block start/stop operations temporarily
  • duration parameter in minutes (default: 10)
  • Response: {"message": "Operations are now blocked for N minutes"}
  • Auto-unblocks after the specified duration
• POST /unblock - Manually unblock operations
  • Response: {"message": "Operations are now unblocked"}

Job Status

• GET /job_status/{job_id} - Get job details and status
  • Response: Full job object with status, timing, legacy arrays, and outcomes
  • Outcomes use changed, unchanged, ignored, protected, blocked, failed, or canceled, with a stable reason and optional error
  • Returns {"status": "not_found"} with HTTP 404 if job doesn't exist

Health Check

• GET /ping - Health check endpoint
  • Response: {"status": "healthy"}
• GET /ready - Operational readiness endpoint
  • Returns 200 only while the lifecycle controller accepts work and the local Docker daemon responds
  • Returns 503 with a machine-readable reason otherwise

Helper Mode Details

The helper mode is designed to run as a systemd service for automatic container lifecycle management:

Lifecycle:

1. Wait for /ready, falling back to /ping when communicating with a v1 controller
2. Apply configured startup delay (default: 5 seconds)
3. Submit start job for all managed containers
4. Wait for job completion
5. Run until receiving SIGTERM/SIGINT; startup waits are cancelable
6. Submit a priority shutdown stop job that cancels pending startup work
7. Wait for stop completion and exit gracefully

Blocked Operations Handling: When start/stop operations are blocked (HTTP 503 response), the helper will:

• Log an INFO message: "Container start/stop operation is currently blocked, skipping"
• Continue running without failing
• This allows the helper to gracefully handle maintenance windows

Options:

./build/sdc helper \
  --controller-url http://127.0.0.1:3377 \  # Controller API URL (default: http://127.0.0.1:3377)
  --startup-delay 5s \                       # Delay before starting containers (default: 5s)
  --timeout 600 \                            # Job timeout in seconds (default: 600)
  --poll-interval 5s                         # Status polling interval (default: 5s)

License

GNU General Public License v3.0