light-oss

A lightweight object storage and static site hosting MVP with a Go API, React console, MySQL metadata, and local filesystem object storage.

English | Chinese

Overview

light-oss is designed for internal tools, prototypes, lightweight private deployments, and systems that need a simple object storage base that is easy to run and modify.

It is not a full S3-compatible service. The current implementation focuses on a clear MVP: bucket management, object upload/download, directory browsing, signed downloads, static site hosting, and a web console.

Components

• backend/: Go + Gin API for buckets, objects, folders, signed URLs, and site bindings.
• frontend/: React + TypeScript + Vite management console.
• gateway/: Nginx gateway for console, API, and hosted-site routing.
• docker-compose.yml: local stack for MySQL, backend, frontend, and gateway.

Features

• Bucket create/list/delete, including cascading object and site cleanup.
• Object upload, download, metadata lookup, listing, deletion, and public / private visibility.
• Bearer Token authentication for private APIs and private object access.
• Signed download paths for private objects and signed single-object uploads that reuse the ordinary upload pipeline.
• Folder tree, folder creation/deletion, directory entries, search, pagination, and ZIP folder download.
• Batch folder upload with multipart/form-data and a manifest.
• Static site hosting from bucket + root_prefix, with custom domains, index document, error document, and SPA fallback.
• Health checks, upload size limits, basic rate limiting, request_id, and structured error responses.

Tech Stack

• Backend: Go 1.22+, Gin, GORM, golang-migrate
• Frontend: React, TypeScript, Vite, TanStack Query, Axios
• UI: Tailwind CSS, Radix / shadcn-style components
• Database: MySQL 8.x
• Gateway: Nginx

Project Layout

.
├─ backend/
│  ├─ cmd/server
│  ├─ docs/openapi.apifox.json
│  ├─ internal/
│  └─ migrations/
├─ frontend/
├─ gateway/
├─ docker-compose.yml
├─ Makefile
└─ .env

Quick Start

Prerequisites

• Go 1.22+
• Node.js 20+
• npm
• MySQL 8.x
• Docker Desktop or Docker Engine, if you use Compose

Local Development

The simplest local setup is to run MySQL with Docker, then run the backend and frontend directly on your machine.

1. Prepare the root .env or .env.personal.

   Local backend and frontend commands prefer .env.personal when it exists. If .env.personal is found, .env is not merged for missing values, so keep .env.personal complete.

   Recommended local values:

   APP_ENV=development
   APP_ADDR=:8080
   APP_STORAGE_MODE=local
   APP_RATE_LIMIT_BACKEND=local
   APP_STORAGE_ROOT=./light-oss-data/storage
   APP_BEARER_TOKENS=light-oss
   APP_SIGNING_SECRET=change-me-in-local-dev
   
   DB_DSN=root:112233ss@tcp(localhost:3306)/light-oss?charset=utf8mb4&parseTime=True&loc=UTC&multiStatements=true
   
   VITE_DEFAULT_API_BASE_URL=http://localhost:8080
   VITE_DEFAULT_BEARER_TOKEN=light-oss

2. Start MySQL.

   docker compose up -d mysql

3. Start the backend.

   cd backend
   go test ./...
   go run ./cmd/server

   The API listens on http://localhost:8080 by default.

4. Start the frontend in another terminal.

   cd frontend
   npm install
   npm test
   npm run dev

   The console runs at http://localhost:3000 by default.

5. Open /settings in the console and confirm the API Base URL and Bearer Token.

Docker Compose with Gateway

Use this mode when you want to verify the full gateway and hosted-site domain flow.

1. Adjust the root .env for gateway mode.

   APP_STORAGE_MODE=local
   APP_RATE_LIMIT_BACKEND=local
   APP_STORAGE_ROOT=/data/storage
   VITE_DEFAULT_API_BASE_URL=http://api.localhost
   VITE_DEFAULT_BEARER_TOKEN=light-oss

2. Add local hosts entries if needed.

   127.0.0.1 console.localhost
   127.0.0.1 api.localhost
   127.0.0.1 demo.localhost
   

3. Start the stack.

   docker compose up --build

   Or:

   make up

4. Visit the services.

   • Console: http://console.localhost
   • API: http://api.localhost
   • MySQL: localhost:3306
   • Example hosted site: http://demo.localhost

Gateway routing:

• console.localhost -> frontend
• api.localhost -> backend API
• other hostnames -> backend static site resolver

Custom site domains must point to the gateway through DNS or hosts. The gateway currently handles HTTP only; HTTPS, certificate management, and production DNS automation are outside this MVP.

Configuration Notes

• Root .env is used by Docker Compose. Compose pins the in-container listener and storage root to :8080 and /data/storage so host paths cannot leak into container configuration.
• Root .env.personal is preferred by local backend/frontend commands when it exists.
• Frontend settings saved in browser localStorage override VITE_DEFAULT_API_BASE_URL and VITE_DEFAULT_BEARER_TOKEN.
• Signing endpoints return relative API paths. Signed uploads also return the exact request headers that the uploader must send unchanged, each exactly once. Clients must preserve additional returned headers.
• Signing JSON and multipart uploads accept raw filenames. Only the single-object PUT X-Original-Filename header is percent-decoded, once at the HTTP boundary; literal percent sequences are preserved in stored metadata.
• Gateway access logs retain request paths, status codes, and timings but omit query strings and Referer headers to keep signed URLs out of access logs.
• APP_STORAGE_MODE defaults to local, in which case the service creates APP_STORAGE_ROOT as needed. Use a host path for direct local runs; the default Compose setup uses /data/storage.
• APP_STORAGE_MODE=shared-filesystem selects an operator-provided shared volume. APP_STORAGE_ROOT must already be mounted because the service will not create a missing shared root. The volume must provide read-write-many access, coherent cross-instance visibility, atomic rename within one filesystem, and read/write/delete permissions. A root-level .storage-id sentinel is bound to MySQL so an instance mounted to a different volume is rejected.
• Object quota is coordinated by the MySQL used_bytes / reserved_bytes ledger. APP_CHUNK_SIZE_BYTES controls streaming reservation increments, and APP_STORAGE_STAGING_TTL_SECONDS controls recovery of abandoned staging blobs.
• DB_CONNECT_TIMEOUT_SECONDS, DB_READ_TIMEOUT_SECONDS, and DB_WRITE_TIMEOUT_SECONDS enforce finite MySQL network waits (defaults: 5/300/30 seconds), including an upper bound on an uncertain transaction commit response.
• APP_BEARER_TOKENS is the Bearer Token allowlist. Multiple tokens are comma-separated.
• APP_RATE_LIMIT_BACKEND defaults to local. Set it to mysql to coordinate every IP and authenticated route bucket through the shared rate_limit_buckets table; all replicas must use identical rate and burst settings. A database decision error fails closed with HTTP 503 instead of bypassing the limit.
• Rate limiting has two levels: APP_RATE_LIMIT_IP_RPS / APP_RATE_LIMIT_IP_BURST apply a coarse client-IP budget before authentication, while APP_RATE_LIMIT_MANAGEMENT_RPS / APP_RATE_LIMIT_MANAGEMENT_BURST are the authenticated management API budget.
• Public object/site downloads, uploads, signed-link creation, and health checks use independent budgets. Override them with APP_RATE_LIMIT_PUBLIC_*, APP_RATE_LIMIT_UPLOAD_*, APP_RATE_LIMIT_SIGN_*, and APP_RATE_LIMIT_HEALTH_*; each route class has its own explicit default.
• APP_RATE_LIMIT_CACHE_TTL_SECONDS controls local-cache eviction and shared MySQL bucket expiry; APP_RATE_LIMIT_CACHE_MAX_ENTRIES is a hard cap for both the local cache and shared MySQL buckets. At shared capacity, a new key receives 429 while existing keys retain their buckets; expired rows release capacity. APP_TRUSTED_PROXIES is a comma-separated allowlist of exact gateway IP addresses or CIDRs; leave it empty to ignore forwarded client-IP headers, and never use an all-address range.
• shared-filesystem is covered by a two-instance test against one MySQL database and shared root, including cross-instance upload/read/overwrite/cleanup and lease/staging takeover. The MySQL limiter is also covered by a two-Router concurrency test that proves one shared burst. The backend must remain at replicas: 1 when APP_STORAGE_MODE=local or APP_RATE_LIMIT_BACKEND=local; horizontal scaling requires both shared modes and validation against the actual shared volume described in the runbook.
• Do not commit real production passwords, signing secrets, tokens, or domain configuration.

API and Documentation

• OpenAPI document: backend/docs/openapi.apifox.json
• Upload performance smoke baseline: backend/docs/upload-performance-baseline.zh-CN.md
• Blob ledger, reconciliation, and migration runbook: backend/docs/storage-lifecycle-operations.zh-CN.md
• Legacy schema retirement note: backend/docs/legacy-upload-session-tables.zh-CN.md
• Backend code organization: backend/docs/code-organization.zh-CN.md
• Main authenticated API prefix: /api/v1
• Liveness: GET /livez; readiness: GET /readyz
• Authenticated health check: GET /api/v1/healthz
• Authenticated runtime metrics: GET /api/v1/system/metrics
• Object API path keys are full object paths, so nested / segments act as directory-like prefixes.
• Static sites only serve public objects.
• Successful JSON responses usually use {"request_id":"...","data":...}.
• Failed JSON responses usually use {"request_id":"...","error":{"code":"...","message":"..."}}.

Upgrade note: signed object paths now use one opaque token query parameter. Previously issued download links using expires and signature become invalid after this release, so coordinate multi-instance rollout and reissue links. The old public GET /healthz alias and APP_RATE_LIMIT_RPS / APP_RATE_LIMIT_BURST variables have also been removed. Use /livez, /readyz, and APP_RATE_LIMIT_MANAGEMENT_RPS / APP_RATE_LIMIT_MANAGEMENT_BURST. Migration 000012_remove_legacy_schema drops unused upload-session tables and objects.is_deleted; back up any historical session data before upgrading and do not run old and new backend versions together.

For manual API testing, set:

BASE_URL=http://localhost:8080
TOKEN=light-oss

Then call authenticated endpoints with:

curl "$BASE_URL/api/v1/buckets" \
  -H "Authorization: Bearer $TOKEN"

Development

make test
make lint

cd backend && go test ./...
cd backend && go test -race ./...
cd backend && go vet ./...
cd backend && gofmt -l .
cd frontend && npm test
cd frontend && npm run lint
cd frontend && npm run build

Run managed-blob reconciliation and exit (orphans are registered and reported, never deleted automatically):

cd backend && go run ./cmd/server -reconcile-storage-only

Useful logs in Compose mode:

docker compose logs -f mysql
docker compose logs -f backend
docker compose logs -f frontend
docker compose logs -f gateway

Known Limits

• Object content is stored on a local or operator-provided shared filesystem, not an object-storage service.
• Static site hosting only serves public objects.
• Gateway mode currently supports HTTP only.
• The project is a lightweight OSS MVP, not a full S3-compatible implementation.