Dev-container images on UBI 8

Two image families live in this repo:

1. The dev-container family — a common base (base-dev) plus thin per-toolchain images derived from it (dev-jdk*, dev-py*). Built on an internet-connected machine, then pushed to the internal registry.
2. The air-gapped build images (jdk*-build.dockerfile + airgap/) — the older, fully offline lineage: every artifact is pre-vendored into airgap/bundle*/ and the image builds with zero network access. See airgap/README.md.

Repo layout

File	What it is
base.dockerfile	Common dev-container base → base-dev:1.0
dev-jdk8.df	JDK 8 (latest el8, 8u502) + Maven 3.6.3 → dev-jdk8:1.0
dev-jdk11.df	JDK 11 + Maven 3.6.3 + Thrift 0.13 → dev-jdk11:1.0
dev-jdk25.df	Temurin JDK 25 + Maven 3.6.3 + Thrift 0.13 → dev-jdk25:1.0
dev-py3.12.df	Python 3.12 (el8 pkg) + Thrift 0.13 → dev-py3.12:1.0
dev-py3.7.9.df	Python 3.7.9 (source build) + Thrift 0.13 → dev-py3.7.9:1.0
dev-py3.14.df	Python 3.14.x (source build) + Thrift 0.13 → dev-py3.14:1.0
dev-py3.14-nogil.df	Python 3.14.x free-threaded (--disable-gil) + Thrift 0.13 → dev-py3.14-nogil:1.0
docker-entrypoint.sh	Runtime docker-socket GID alignment (shared by ALL images)
dev-entrypoint.sh	Home-dir seeding/ownership for the bind-mounted /local (base family only)
jdk11-build.dockerfile, jdk8-build.dockerfile	Air-gapped lineage (see airgap/)

Naming convention: dockerfile dev-X.df ⇒ image dev-X:1.0.

The base image (base.dockerfile)

Everything every dev container shares:

• Non-root user (USERNAME build arg, default alabanou, UID/GID 1000) with passwordless sudo (NOPASSWD:SETENV).
• HOME = /local — designed to be a host bind mount (vscode-server lives there and survives container recreation).
• zsh + oh-my-zsh: theme ys, plugins git docker docker-compose kubectl. The framework is in /opt/oh-my-zsh (deliberately outside /local so a mount can't shadow it); the .zshrc is seeded via /etc/skel.
• docker CLI + compose plugin with runtime socket-GID alignment (below) — docker info / docker ps work against the mounted host socket on any host.
• tini as PID 1 (signal forwarding, zombie reaping — needed for Testcontainers).
• Locale en_US.UTF-8 (glibc-langpack-en).
• Tooling: curl wget zip unzip vim nano less tree file procps-ng iproute net-tools lsof strace jq yq git ca-certificates.
• tree is the one tool absent from all UBI 8 repos — its RPM comes from the CentOS Stream 8 vault (binary-compatible with UBI 8).

Build

Always from the repo root (COPY paths resolve from here):

docker buildx build --load --platform linux/amd64 --provenance=false --sbom=false \
    -f base.dockerfile -t base-dev:1.0 .

Then any derived image, e.g.:

docker buildx build --load --platform linux/amd64 --provenance=false --sbom=false \
    -f dev-jdk11.df -t dev-jdk11:1.0 .

All derived files take --build-arg BASE_IMAGE=... (default base-dev:1.0) — point it at the internal-registry copy once the base is pushed.

Derived-image notes:

• dev-jdk8: "latest JDK 8" per el8 repos = 1.8.0.502 (Red Hat numbering runs ahead of Temurin's 8uXXX). No Thrift in this one.
• dev-jdk25: el8 repos stop at JDK 21, so this is an Eclipse Temurin tarball fetched from the Adoptium API (/latest/25/ga → currently 25.0.3+9).
• dev-py3.12: the only Python of the series packaged for el8.
• dev-py3.7.9 / 3.14 / 3.14-nogil: compiled from source in a throwaway builder stage; installed under /opt/python/<ver>; PGO (--enable-optimizations) is OFF on purpose — under emulation it multiplies build time for a few % of runtime. The nogil build installs as python3.14t, with python3/python symlinks, and asserts sys._is_gil_enabled() == False at build time.
• Maven and the Thrift compiler binary are reused from airgap/bundle/ (local, checksummed). Python images also pip install thrift==0.13.0.

Runtime

Typical run (dev container):

docker run -it \
    -v /path/on/host/home:/local \
    -v /var/run/docker.sock:/var/run/docker.sock \
    dev-jdk11:1.0

Entrypoint chain (do not override)

tini (PID 1) → dev-entrypoint.sh → docker-entrypoint.sh → CMD (zsh)

• dev-entrypoint.sh — the /local mount shadows the image's home, so on start it fixes the mount's top-level ownership (never recurses into host files) and seeds any dotfile missing from /etc/skel (never overwrites existing ones — your customizations survive).
• docker-entrypoint.sh — reads the GID of the mounted docker socket, aligns the container's docker group to it, re-execs as the user so the membership applies. No DOCKER_GID build arg, no --group-add, no compose/devcontainer tweaks needed — the host's docker GID is a host property and is handled at runtime. The DOCKER_GID=999 build arg is only the placeholder the group is created with.
• In devcontainer.json use "overrideCommand": false; in compose set command:, never entrypoint:. VS Code opens terminals via docker exec, which skips the entrypoint — that still works because the group fix persists in /etc/group after the first start.
• Hardened runtimes that set no-new-privileges or drop CAP_SETUID/CAP_SETGID break the sudo re-exec; there (and only there) fall back to --group-add $(stat -c %g /var/run/docker.sock).

Hard-won gotchas (read before touching the Dockerfiles)

1. Building amd64 images on Apple Silicon

The target is always linux/amd64 (there is no "darwin" image; containers are Linux). On an arm64 Mac both of these are required, and they do different jobs:

• FROM --platform=${TARGET_PLATFORM} in the Dockerfile pins the base layers;
• --platform linux/amd64 on the CLI stamps the output index.

Miss the CLI flag and you get an image whose content is amd64 but whose manifest says arm64 — docker image inspect looks fine, while Rancher Desktop / containerd / k8s reject it with manifest errors. Add --provenance=false --sbom=false or buildx appends an unknown/unknown attestation manifest that older tooling also chokes on.

Verifying the arch: docker image inspect reads the config and can disagree with the index; docker buildx imagetools inspect can serve stale results after a rebuild. The ground truth is what a transfer would carry:

docker save IMG | tar -xO index.json   # one manifest, no arm64 stamp, no unknown/unknown

A bad platform stamp lives in the build cache — rebuilding with corrected flags but a warm cache re-exports the same bad index. --no-cache (or docker builder prune) is required to re-stamp.

2. sudo silently clobbers PATH (secure_path)

The socket-GID re-exec goes through sudo, and sudo resets PATH to secure_path (/sbin:/bin:/usr/sbin:/usr/bin) even with -E. That drops /usr/local/bin (thrift, tini, yq), /opt/maven/bin, /usr/lib/jvm/*/bin, /opt/python/*/bin — i.e. mounting the docker socket used to break the whole toolchain, while everything worked without the socket. The fix is --preserve-env=PATH on the sudo call (allowed by the sudoers SETENV tag). It's in docker-entrypoint.sh; don't remove it again.

Corollary for testing: always verify toolchains WITH the socket mounted — that's the mode the container actually runs in, and it's the mode that broke.

3. Rosetta poisons $HOME during emulated builds

When building amd64 on Apple Silicon, Rosetta drops a root-owned cache dir into $HOME of any emulated root step. After ENV HOME=/local that means a root-owned /local/.cache, which breaks oh-my-zsh (and anything else writing to home) for the non-root user. Rules baked into the Dockerfiles:

• in base.dockerfile the user-creation RUN is the last RUN executed as root, ends with rm -rf /local/.cache && chown -R ... /local;
• every derived .df ends its last root RUN with rm -rf /local/.cache.

Keep it that way when adding steps.

4. UBI 8 package gaps

UBI's repos are a subset of RHEL's. Missing packages found so far: tree, readline-devel. The pattern for filling gaps: take the RPM from the CentOS Stream 8 vault (https://vault.centos.org/8-stream/...) — binary-compatible, same trick as bison/flex in airgap/vendor.dockerfile. dnf accepts the RPM URL directly in a normal install transaction.

5. Docker Desktop disk pressure

Emulated source builds inflate the build cache fast (three CPython builds ≈ 29 GB). Symptoms are misleading: dnf dies with "Transaction test error", shells fail with "No space left on device", and runtime verification of perfectly good images fails. Check docker system df, clean with docker builder prune -af. Run heavy emulated builds sequentially, not in parallel.

6. docker pull --platform on an existing tag lies

If the tag already exists locally, docker pull --platform linux/amd64 may say "up to date" while keeping the other-arch copy, and docker image inspect of a pulled base image reports the host-matching variant. Don't pre-pull bases when cross-building — let the build resolve them itself.

Verification checklist

Run per image, with the socket mounted (see gotcha 2):

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock IMG sh -c '
  echo "PATH: $PATH"                      # must contain /usr/local/bin + toolchain dirs
  cat /proc/1/comm                        # tini
  id -Gn                                  # includes the socket group
  docker ps --format x >/dev/null && echo "docker ps OK"
  java -version 2>&1 | head -1 || true
  mvn -v 2>/dev/null | head -1 || true
  python3 --version 2>/dev/null || true
  thrift --version 2>/dev/null || true'
# arch/manifest:
docker save IMG | tar -xO index.json
# home seeding (empty host dir):
docker run --rm -v "$(mktemp -d)":/local IMG zsh -ic 'echo "theme=$ZSH_THEME"'

Air-gapped lineage

The pre-dev-container images (alexlaban/jdk11-mvn3.6.3-thrift0.13, alexlaban/jdk8-mvn3.6.3) build fully offline from vendored bundles — flow, transfer and rebuild instructions: airgap/README.md. They share docker-entrypoint.sh with the dev family. Images of that lineage tagged before the --preserve-env=PATH fix (jdk11 ≤ 0.4, jdk8 ≤ 0.1) carry the PATH-clobbering bug from gotcha 2 — with the socket mounted, mvn/thrift vanish from PATH. Rebuild them from the current docker-entrypoint.sh before shipping.