procd: add support for OCI runtime spec 1.3.0 - #37
Open
dangowrt wants to merge 82 commits into
Open
Conversation
Member
Author
|
@dhewg would appreciate your review! |
Contributor
|
Nice, will take a look! |
|
@dangowrt While this PR does indeed fix part of my issue in #35, the part I included in the edit does this happen. When actually trying to run a OCI container UXC will throw The directory in the uxc json file is present and filled with files: The command is just |
Member
Author
@dangowrt Your fix does indeed resolve the EPERM issue however it still does not work. Instead of error 1 EPERM I now get error 22 Invalid Argument. UXC stills ays
and this in the listing:
config file/command is the same. Note that this also happens when running uxc start manually, so it's not an invalid argument to the uxc command itself (which I originally thought) The issue for this happening seems to be seccomp. Since when disabling seccomp the error disappears when using UXC directly. Now it can run the container. However when trying to attach via Another point is, it really only works when using uxc directly. When it is execut by podman I get the following error still: Also note that despite what's being said, the container is NOT removed from uxc. Edit: here's the full podman log:Edit 2: After recompiling OpenWRT with seccomp support and rerunning the error still persists but I get a new error message:
However inspecting it shows that it is indeed present just with a slightly different name
It looks like a) podman is giving out bas paths that UXC can't find or b) UXC just uses a bad name. Could this be a cgroup v1/v2 thing? According to strings and some googling ujail searches for memory.swap_max which is cgroup v1 only yet the cgroup version that's running is v2 which uses memory.swap.max instead. |
dangowrt
force-pushed
the
oci-1.3.0
branch
2 times, most recently
from
May 26, 2026 13:01
ed1ba35 to
14c4b4a
Compare
Member
Author
|
@Juff-Ma I was wrongly assuming that podman, runc and crun would implement the CLI as defined in the spec, but turns out that part of the spec is abandonned and it become more of just a convention. Now regarding the seccomp problem you were seeing: I've reworked the seccomp support for now also work with "foreign" libc containers (ie. OpenWrt host being musl, container being eg. glibc or bionic). This needs a small change to the procd package Makefile as well, see below diff --git a/package/system/procd/Makefile b/package/system/procd/Makefile
index cf730a3c0a..c508a447c6 100644
--- a/package/system/procd/Makefile
+++ b/package/system/procd/Makefile
@@ -30,12 +30,14 @@ PKG_CONFIG_DEPENDS:= \
include $(INCLUDE_DIR)/package.mk
include $(INCLUDE_DIR)/cmake.mk
+include $(INCLUDE_DIR)/kernel.mk
ifeq ($(DUMP),)
STAMP_CONFIGURED:=$(strip $(STAMP_CONFIGURED))_$(shell echo $(CONFIG_TARGET_INIT_PATH) | $(MKHASH) md5)
endif
-CMAKE_OPTIONS += -DEARLY_PATH="$(TARGET_INIT_PATH)"
+CMAKE_OPTIONS += -DEARLY_PATH="$(TARGET_INIT_PATH)" \
+ -DNOLIBC_INCLUDE_DIR="$(LINUX_DIR)/tools/include/nolibc"
define Package/procd/Default
SECTION:=base |
|
@dangowrt I've tested your build and it has the same problem I reported in Edit 2 of my previous message. The seccomp error (while I appreciate it being fixed) was easily solvable by just compiling OpenWRT with seccomp support. Through a bit of grepping and googling I eventually made this simple one line patch but still got an error. I could try combining my change with your patchset and see if that fixes anything. |
I can not report success. The error about the missing cgroup file is gone but (once again) I still receive a |
|
@dangowrt I have dug deeper and created a script that logs the arguments passed to uxc. Finding out that the commands to UXC absolute are fine and DO WORK when manually running them there's dafinitely something weird at play. So far I know the following: There is a race condition between UXC and Podman. Particularly in creating the pidfile. When the UXC command exits the pidfile is not yet created for some reason and Podman WILL fail early. Introducing 1s delay into the UXC execution via my logging script fixes this issue. (Podman reports the correct PID) With that in place Podman now tries to actually start the container only for that to immediately fail with a new error:
Which is... weird? If it is Podmans job to create this file why isn't it doing so. And if it is UXCs job to do so why isn't podman providing ANY command line param. Now, this could be related to this debian issue which claims this can be the result of a conmon/podman version mismatch. This certainly could be the case. Podmans dependencies in the repos are not as thoroughly updated as podman itself. Conmon is 2 years out of date for example. But still, I continued. I ran the command with -d and... It worked? Kinda at least. The container started (and showed as running) but podman immediately regards it as crashed and kills the container (note podman regards it as having exited with -1, not UXC, when it receives the kill command the container is still running). I don't know why podman kills it. Podman seems to execute a cleanup command. but I don't know why. Trying to get logs fails with a |
|
I'm making one small step after the other. Upgrading all podman dependencies helped a bit. I needed to patch a build error in conmon but that was easily fixable. (I'm still using the 1s delay for UXC to work around the race condition) Using the latest tools I now am able to start a container and it actually runs. Well the container does. The app inside not so much. When trying to get logs I get the error Note that /docker-entrypoint.sh is not static. It is the entrypoint specified by the Dockerfile and therefore changes with the container used. Note that I am not able to recover from this error. My entire shell session freezes and I can't use |
Member
Author
|
There is an architectural problem with how seccomp is applied which break foreign-libc containers (ie. OpenWrt with musl and container eg. with glibc). I've resolved that and tons of other issues, currently last mile of testing, going to push to this branch in the next hours with validated podman working. Meanwhile, please open a PR to update podman in case you didn't do that already and there isn't any existing PR for that (I've opened the PR to update conmon earlier this morning) |
I see. Glad to hear that. I haven't actually updated Podman itself. It is up to date. In addition to conmon I've update the netavark/aardvark-dns stack, catatonit and crun (since it was also out of date and I was at it) I'll look into opening PRs P.S.: Your PR does not include a second patch conmon required in my build env at least. I got a GCC false positive on |
Member
Author
|
@Juff-Ma Please retry with the changes I've pushed now. Don't forget to also apply the patch for procd's Makefile I've posted in #37 (comment) |
@dangowrt IT IS ALIVE. Oh my god this is amazing. I still included my cgroups patch just to be sure but it absolutely works now. I can run containers and their services are reachable. The work you've done to get this working is incredible. I found 2 little issues (that do not impact day to day operations, at least for me, but I still wanna name them).
P.S.: The error really does occur when killing or deleting containers. I don't know why though. Killing never works and rm-ing will error out the first time. Still not an issue for me. |
Member
Author
|
@Juff-Ma console issues with |
If you mean the commits up to 280a8bb I wasn't really able to test them (I already tried when they were pushed) since I've been encountering a new issue. I though that I had left a comment but it seems it didn't go through (why github?) On some containers I get a Here are the two containers I used:
|
Contributor
|
Some testing feedback: create+start+attach doesn't have that issue, but now escape sequences get printed (prompt displays as Sometimes create hangs too: attach on a created but not yet started container is just an echo chamber. In an attempt to move a broken board's debian install to uxc (so a full distro with systemd), I whipped up the attached patch and used the attached config. That worked prior to this PR, including 0001-jail-add-config-to-not-lock-securebits.patch |
Contributor
|
I see some of the above mentioned issues only on my openwrt one, and not on riscv |
Contributor
|
On riscv64, adding a memory limit to config.json: yields a crash: Happens reproducible with the limit, doesn't happen without it. |
dangowrt
force-pushed
the
oci-1.3.0
branch
4 times, most recently
from
August 21, 2026 00:43
90d7d07 to
6212139
Compare
put_namespace() stats /proc/self/ns/<name> and only advertises namespaces the running kernel actually provides, so guarding the "time" entry with #ifdef CLONE_NEWTIME cannot enable anything the runtime check would not. The guard tests the toolchain headers at build time, and the guarded code does not even use CLONE_NEWTIME, so its only possible effect is to hide time namespace support from the features reply when procd was built against headers predating the flag. Drop the guard together with the <sched.h> include which was added solely to provide it. Fixes: 47a9f0d ("service: add method to query available container features") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
_add_mount() keyed mounts by target alone and silently skipped any second registration for an already-known target. With OCI bundles a target can legitimately be registered twice from independent sources (for instance an implicit mount and an explicit one), and the two may carry conflicting parameters that the old code discarded without notice. Compare the full descriptor instead: an exact duplicate (same source, filesystemtype, optstr, flags, error and inner flags) is accepted as a no-op, whereas a genuine conflict on the same target now returns EEXIST so the caller can fail loudly rather than honour whichever registration happened to win. Fixes: 71e75f4 ("jail: refactor mount support to cover OCI spec") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
conmon learns a container's exit status by waiting for the pid it reads from the runtime's pid file, but with ujail that process is a child of procd and never of conmon, so conmon never obtains a status at all: podman reports 0 for a container that exited 42, and podman exec fails even when the command succeeded. A shim would give conmon something to wait for at the price of putting a process other than the container's own in the pid file, which tooling needs for introspecting namespaces, cgroups and seccomp state. Write the decoded status, WEXITSTATUS or 128 plus the terminating signal, to an exit_status file in the directory the pid file lives in. The value goes to a temporary file and is renamed into place, so a reader either sees the previous value or the whole new one, and it is written before the SIGCHLD that ends conmon's loop. Exec sessions do the same with their own pid file, so podman exec has a status to report as well. A stale value from an earlier run is removed when a pid file is written, and nothing is written when no pid file was asked for, which leaves the behaviour of a caller that does not want any of this unchanged. The status file on its own ends no wait: the container is procd's child, so its death raises no SIGCHLD in conmon, and a detached container's streams never reach end of file, leaving conmon to sit until its own timeout. The runtime therefore signals the process that asked for the container. conmon execs the runtime, so uxc runs as conmon's child and its parent is the waiting process; uxc opens a pidfd on getppid() with pidfd_open(), called through syscall() since musl wraps neither it nor pidfd_send_signal(), and sends the descriptor along with the create and exec requests. procd keeps it on the instance beside the stdio descriptors and lets ujail inherit it across the execve() that starts the jail, -a naming the descriptor number, and ujail sends SIGCHLD through it with pidfd_send_signal() once the container is gone, after the status has been written. An exec session signals through the descriptor its own request carried once the session has been reaped, and a respawned instance carries the same descriptor into the next ujail, so the invoker also learns when a later incarnation dies. This holds for any invoker instead of leaning on the conmon.pid filename, which is merely podman's default for --conmon-pidfile and silently defeated by overriding it. A pid would identify the invoker only for as long as it lives. Callers such as uxc.init and interactive shells routinely exit long before their container does, leaving behind a number the kernel may hand out again, and any scheme that re-checks the pid before the kill still leaves a window between the check and the signal. A pidfd pins the identity at the instant it is opened: from that instant on it is the only process a signal through it can ever reach, however much later it is sent. One window remains, and it is not one a descriptor can close: getppid() is read before the descriptor exists, so an invoker that died first yields the reaper instead, and a number already recycled by then names a stranger. uxc therefore re-reads getppid() after opening and sends nothing if it changed, which leaves only the two adjacent syscalls in between, against the whole lifetime of a container in the old scheme. ujail still polls the descriptor for POLLIN first, the same way exec_jail() watches the pidfd of its own parent, and stays silent for an invoker that has already gone. A caller that supplies no descriptor is never signalled; when pidfd_open() fails uxc warns and sends none rather than falling back to a pid, since a caller which then waits for a wake-up that never comes deserves to see why. The descriptor set now carries its own count, so procd and uxc must be upgraded together: the previous revision of this commit accepted exactly three descriptors and nothing else. Losing the carrier now loses the notification with it, where the pid had travelled separately in the request. Two limits are worth naming. The signal is sent as ujail begins tearing down, before cgroups and the network are dismantled, so a woken manager can observe a container whose traces have not all gone yet. And an exec session still in flight when the container itself dies is never signalled, because the supervisor exits first; such a session is left to end on stream EOF, as it was before. ubus carries a single descriptor per request, and the three standard descriptors already travel as SCM_RIGHTS over a socket pair whose receiving end goes to ubus_invoke_fd(). That carrier now takes a counted set instead of a fixed trio, the count riding in the payload byte, because the stdio descriptors are only sent when pass-through is wanted while the notification descriptor is wanted independently of that: a create sends stdio and, when the invoker could be named, the pidfd; an exec session with a terminal sends the pidfd alone. The receiver tells the layouts apart by the count, which stays unambiguous because stdio is all or nothing. The invoker's identity is thereby no longer configuration. procd used to compare the notifypid attribute like the other jail attributes, restarting a running instance when a re-add named a different invoker; the descriptor is runtime state like the stdio descriptors, so a re-add replaces the stored descriptor for the next start while the running jail keeps the one it inherited. The descriptor stays close-on-exec everywhere except across the one execve() that starts the instance's own ujail, and ujail marks it close-on-exec again as soon as it parses the option, so neither hooks nor the container itself ever inherit it. conmon ends its loop when the streams it handed the runtime reach end of file and only then looks for the status, so an exec session's descriptors stay with the session and are closed once its status has been written. Recording it from the process that waits for the session is no alternative: that one has joined the container's mount namespace, where the path the status belongs at does not exist. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
07e0f7d ("jail: fix /proc,/sys mounting under CLONE_NEWUSER") did two things: it replaced the hardcoded MS_NOATIME on the procfs and sysfs mounts with detect_atime_flag(), which is what actually made them mountable and which is kept, and it moved the creation of the user namespace to after the mounts were built, which left the pid, network, ipc, uts and cgroup namespaces owned by the initial user namespace. ujail itself, being privileged, never had a problem with that; the container at runtime did. The kernel resolves a container's privileges against the owner of the namespace being used, so a container holding CAP_NET_BIND_SERVICE could not bind a privileged port, __inet_bind() asking ns_capable(net->user_ns, CAP_NET_BIND_SERVICE), and it could mount neither its own procfs, which wants CAP_SYS_ADMIN in the user namespace owning the pid namespace, nor its own sysfs, which wants the same in the one owning the network namespace. runc and crun both establish the user namespace before everything else for this reason. Undo only the reordering half of that commit. Let clone() create it together with the rest, which the kernel attributes to the new user namespace since the credentials are copied before the namespaces, and have the child wait for its uid and gid maps before it does anything that needs privilege. Creating it late is kept for the one case crun also keeps it for, a container joining existing namespaces, whether an OCI bundle names them by path or -j on the command line names them by pid, because entering those needs privilege in the user namespace owning them. The time namespace is not yet handed over: CLONE_NEWTIME stays masked out of the clone3() flags and the parent still unshares it before the clone, so it remains owned by the initial user namespace and the title only fully holds once a following commit moves it into the child as well. Redefining what defers the user namespace moves work between the two phases without changing the end state. remask_after_unshare(), remount_proc_sys_after_unshare(), the oci_deferred_* bookkeeping and the JAIL_NOAFILE bind now serve only the deferred path; on the common path the default masks and the read-only /proc/sys hack are applied while the mount list is built. The deferred path in turn drops privileges with setregid(), setreuid() and setgroups() before its second unshare(CLONE_NEWNS) rather than after. The inherited-mount detach introduced by a1c5633 ("jail: detach inherited mounts under /proc,/sys before mounting own") and guarded by 3fc9d11 ("jail: run inherited-mount detach before joining an external userns") now runs in two cases only, the deferred user namespace and the join of an external one. In a mount namespace owned by our own user namespace the inherited mounts are locked and cannot be detached; for mounts sitting on the kernel's permanently empty mount points it is not needed either, since mount_too_revealing() ignores those, but a locked mount covering an ordinary path, which our own masking and OCI maskedPaths create, still disqualifies the reference mount and can no longer be detached there. The command line could combine -j <pid>:user with -f, which the old code tolerated because CLONE_NEWUSER was always stripped from the clone flags and the join simply won. Now that the flag reaching clone3() creates a namespace of its own, -f leaves it unset when a user namespace has already been joined; an OCI bundle cannot express the combination, as parseOCIlinuxns() rejects the duplicate in both directions. Comments left over from the late-creation scheme, now stating the opposite of what the code does, are dropped. One detail follows from the new order: the gid 5 the standard /dev/pts options carry cannot be resolved by a mapping that holds a single id, and devpts refuses a mount whose gid does not map, so an unmapped gid is dropped from the options. crun likewise omits gid=5 for its rootless containers, though it does not filter bundle-supplied options this way. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
6212139 ("jail: give the container's namespaces to its own user namespace") left one namespace behind: the time namespace was still created by the parent, with unshare(CLONE_NEWTIME) before the clone, and therefore stayed owned by the initial user namespace. clone3() cannot simply take the flag either, since a child created with CLONE_NEWTIME sits in the namespace from birth, which freezes its offsets before anyone can write them. Create it in the child instead, once the uid and gid maps are in place. A namespace made by unshare() is owned by the user namespace of its creator, so it now belongs to the container's user namespace, where mapped root holds CAP_SYS_TIME, exactly what writing /proc/self/timens_offsets asks for. unshare(CLONE_NEWTIME) does not move the caller, and the offsets of an inhabited namespace are sealed, so the child writes the offsets first and then enters through /proc/self/ns/time_for_children with setns(), which the kernel allows while the process is still single-threaded. A bundle that defers its own user namespace creates the time namespace in enter_userns() for the same ownership reason; /proc is present there, as a new time namespace can only be configured through OCI and an OCI jail always mounts it. The createContainer hooks of such a bundle now run before the time namespace exists; everything else runs inside it as before. A time namespace joined by path moves to the child as well, alongside the other setns() joins and before any user namespace is created or joined, because entering needs privilege in the user namespace owning the target. The parent keeps only the probe for kernels without time namespace support, and no longer switches its own time namespace around the clone: the old join path setns()'d the supervisor into the container's time namespace and back, briefly running it on shifted clocks. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
uxc-net now honours the org.openwrt.network.proto, proto6 and ip6ifaceid annotations to pick the in-jail IPv4 and IPv6 protocols and to pin the IPv6 interface identifier inside a delegated prefix. Let a stack definition set them per member through the new spec keys proto, proto6 and ip6ifaceid, named after the annotation leaves just like the egress, ingress and host access options, and copy them into the member's annotations sidecar alongside the existing network annotations. A key set by the stack definition overrides the same annotation in the member's own bundle, since uxc-net merges the sidecar over the bundle's config.json annotations; a key the stack leaves unset is omitted from the sidecar, so the bundle's own value stays effective. This matches the behaviour of the existing attach, egress, ingress and host annotations, whose precedence is unchanged. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The helper jail mounts /lib/netifd wholesale, so its netifd registers the dhcpv6 protocol handler whenever the odhcp6c package is installed on the host, but /usr/sbin/odhcp6c itself is missing from the mount list. A container can therefore never run DHCPv6 or accept router advertisements through its private netifd, while IPv4 DHCP works because udhcpc is mounted. Add odhcp6c next to udhcpc; read-only mount entries are skipped silently when the source is absent, so hosts without the odhcp6c package are unaffected. Also mount /bin/sed: proto_dhcpv6_setup() unconditionally pipes the prefix request hint through sed to extract an explicit IAID suffix, so without it every DHCPv6 interface setup logs "sed: not found". tr and hexdump stay out on purpose. Their only users are hexdump_2hex() in /lib/functions.sh and the vendorid encoding in dhcp.sh, reached via option clientid, option vendorid or a global dhcp_default_duid, and no configuration the jail can see contains any of those: the jail's /etc/config/network is compiled exclusively by uxc-net, which emits neither these options nor a globals section, the compiled config is bind-mounted read-only, and the private ubus socket is mounted only into the helper jails, so the container cannot inject dynamic interfaces carrying such options either. Both proto handlers then take the branch where clientid is empty and the dhcp_default_duid lookup returns empty before hexdump_2hex() is called. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
…tate
uxc list cannot show a container's addresses, nor tell a container with
its own network namespace apart from one sharing the host's. Only ujail
can provide that data: a bridged container obtains its address by DHCP
inside its network namespace, and the in-jail netifd sits on a private
ubus, so nothing on the host bus ever learns it.
Extend the state reply with a reverse-DNS top-level object:
"org.openwrt.network": {
"namespace": "private",
"attach": "routed",
"interfaces": [
{ "name": "eth0", "mac": "9a:...", "addresses": ["10.7.3.2/31", "fe80::1/64"] }
]
}
"namespace" is host, joined or private depending on whether the
container requests no network namespace, joins one by path or gets its
own. "attach" is the effective attachment mode: the bundle annotation
merged with the sidecar file uxc-stack leaves for uxc-net, the sidecar
taking precedence just as uxc-net applies it; a private namespace
without any annotation is reported as none since uxc-net treats it
that way, and the field is omitted when the mode is genuinely unknown.
"interfaces" lists the interfaces in the container's network
namespace, excluding loopback, with their MAC and their IPv4 and IPv6
addresses in CIDR form, gathered by RTM_GETLINK and RTM_GETADDR dumps
over a netlink socket created inside the container's namespace; the
parent enters the namespace just long enough to create the socket and
returns at once, leaving the container undisturbed. The dump is only
attempted while the container process is alive, so a query racing
teardown simply omits the list.
The runtime spec allows additional state properties and runc, crun and
ujail already ship some; a reverse-DNS key can never collide with a
future spec property. The object is added in handle_state() only,
after oci_state_fill(), so OCI hooks keep receiving an unchanged
spec-shaped document on stdin and no netlink work happens per hook
invocation.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The in-jail addressing was hardcoded per attachment mode: bridged containers always received a DHCP interface and routed containers always received the static /31 configuration, while the documented org.openwrt.network.proto annotation was read nowhere. Implement the annotation. Replace the hardcoded configuration blobs with a renderer parameterised on the requested protocol, composed through append_injail(), and regenerate the in-jail file from scratch on every bring-up so a stale file from an earlier run cannot leak into the composition. Accepted values are 'dhcp' and 'static'. The defaults preserve today's behaviour exactly: bridged defaults to 'dhcp', routed to 'static'. An unknown value is rejected with a diagnostic instead of silently falling back. Requesting 'dhcp' for a routed container is refused as well, because routed builds a /31 point-to-point link whose gateway end runs no DHCP server, so the request could never be served. A bridged container with proto 'static' takes its address from org.openwrt.network.address in address/prefix notation, with the optional org.openwrt.network.gateway and org.openwrt.network.dns annotations rendered verbatim; the backhaul section now goes through the same renderer. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The in-jail interface was IPv4 only; nothing in the contract could ask for an IPv6 address. Add org.openwrt.network.proto6, accepting 'dhcpv6', 'slaac', 'static' and 'none'. Where IPv6 is requested a second in-jail interface section is rendered on the moved device: 'dhcpv6' runs the full odhcp6c cycle, 'slaac' renders proto 'dhcpv6' with reqaddress 'none' and reqprefix 'no' so odhcp6c only processes router advertisements, and 'static' takes org.openwrt.network.address6 in address/prefix notation with an optional org.openwrt.network.gateway6. Add org.openwrt.network.ip6ifaceid, an address with a zero network part such as '::1234', rendered as option ip6ifaceid on the dhcpv6 section; /lib/netifd/proto/dhcpv6.sh hands it to odhcp6c through -i, giving the container a deterministic interface identifier within whatever prefix the link advertises. The default is 'none': no IPv6 section is rendered, preserving the existing behaviour for containers that do not ask for IPv6. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
A routed container had no IPv6 path at all: the host gateway interface carried only the /31 IPv4 address, and nothing on the host answered router solicitations on the link, so a container asking for proto6 could never configure itself. Give the gateway interface created through network.add_dynamic an ip6assign of 64 and ip6ifaceid '::1'. netifd hands the whole blob to interface_alloc(), which parses every interface attribute, so netifd carves a /64 for the link out of the upstream delegated prefix and the host end takes the deterministic ::1 address within it; no netifd change is needed. Prefix assignment alone does not make the link work: router advertisements must actually be emitted on the gateway interface for the container to learn the prefix and its default route. That takes an odhcpd section, which is new host-side state: bring-up commits a per-container section named after the gateway interface to the persistent dhcp configuration, with ra and dhcpv6 in server mode, and reloads it through the config.change service event. Bring-down removes exactly that section again, following the lifecycle the per-container firewall sections already use. dnsmasq ignores the section because its dhcpv4 option defaults to disabled. fw_reload() becomes pkg_reload() so the firewall and dhcp reload paths share one helper. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
ensure_network() commits persistent sections to network, dhcp and firewall when a bridged attachment names a network that does not exist, and bridged_down() never removed them, so a single container run left a network, a DHCP pool and a firewall zone in /etc/config forever. Removing the sections on bring-down is the right resolution rather than not committing them in the first place, because the DHCP pool can only exist as persistent configuration: dnsmasq reads UCI alone and offers no ubus path to hand it a pool, so an ephemerally created network would be one without addressing. The per-container firewall sections already follow this lifecycle for the same reason, stock fw4 ignoring zone data delivered over ubus: create at start, idempotent, delete on teardown. Record each auto-created network in a marker under the uxc state directory and name the created sections deterministically, the device section as <net>_dev and the firewall zone section as <net>. On bring-down of a bridged container the network is removed again when the marker exists and its bridge has no members left, deleting exactly the four sections creation wrote, and the marker with them. A network the operator defined is never touched, because it has no marker, and a network still carrying other containers keeps its configuration until the last member goes down. The marker lives on the uvol metadata volume, so a firmware re-image that wipes /etc/config but keeps the container registrations simply leads to the network being recreated on the next start. A marker can go stale if an auto-created network is never brought down and the operator later defines the same network by hand; the next last-member bring-down would then remove the hand-written sections. The window is narrow and the alternative is refcounting operator intent, which UCI cannot express. With this, bring-down restores /etc/config for bridged containers as well; while a container is up, its container-lifetime sections exist, exactly as the routed firewall and router advertisement sections do. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
run_uxc_net() reports whether the helper succeeded and the caller threw the answer away, so a container whose networking the helper refused, for instance a routed one asking for DHCP where the point-to-point link has no server, came up regardless with nothing but loopback inside its namespace. The diagnostic went to the log and the container looked healthy, which is the worst of both. Treat it as the setup failure it is. A network the runtime was asked for and could not build is not something to paper over, and a container that starts without it would have to be diagnosed from the inside. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
ujail's container state reply now carries an "org.openwrt.network" object describing the network namespace mode and the interfaces the container ended up with. Surface it in 'uxc list' so a glance at the table answers the most common question about a container: how is it reachable. The JSON output passes the object through verbatim, next to the annotations, so runc-style consumers see exactly what the jail reported. The table gains a NET column: "host" for a container sharing the host network namespace, "-" when there is nothing to show, otherwise the addresses. A container with a single interface prints a bare comma-separated address list, the common case staying compact; with several interfaces each address is prefixed with its interface name as "eth0=10.7.3.2/31", since a bare list from several interfaces cannot be told apart. The column is appended after OWNER rather than inserted: the package hook library /lib/functions/uxc.sh greps the table with '^$name[[:space:]].*[[:space:]]running', which an appended column leaves matching, and podman and conmon never parse this table. 'uxc state' needs no change as it prints the jail's blob verbatim. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
'uxc list' only understood its own --json/-j spelling, while runc and crun both spell it '--format json' and both offer -q/--quiet. Accept all of them so tooling written against runc works unmodified: '--format table' and '--format json' select the output, an unknown format is rejected, and --json keeps working so nothing in-tree breaks. --quiet prints one container id per line and takes precedence over the format selection, matching runc's behaviour. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
runc and crun both carry rootfs and created in the document their state operation returns, and tooling built against them expects to find them there. ujail reported neither, so uxc had nothing to print and showed a placeholder where every other runtime shows a timestamp. The runtime is the only honest source for both. It learns the resolved rootfs when it parses the bundle, and it knows the instant the container reached created state, which is the moment the OCI lifecycle calls its creation. Take the timestamp there rather than deriving it later from a process start time, which would answer a slightly different question and would oblige the caller to go reading /proc on the runtime's behalf. Both are added where the network object is added, after oci_state_fill(), so the document handed to OCI hooks on stdin keeps the shape the specification describes. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The CREATED column was a hyphen for every container and the JSON carried neither rootfs nor created, both of which runc emits and tooling reads. The runtime now reports them, so take them from the state reply and print them. OWNER stays "root" deliberately. Containers here are created by procd, which runs as root, so the field is already truthful; inventing a per-container owner would mean uxc reading registration files, and uxc talks to procd and ujail over ubus and to nothing else. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Recursively remove the state directory with nftw() instead of spawning /bin/rm, whose exit status was discarded. uxc is meant to reach the system only through ubus, and this exec had nothing to do with volume management in the first place: it merely borrowed the uvol helper. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The result of activating the container's image volume was discarded, so a volume left write-only by an interrupted upgrade, or one that vanished entirely, was only noticed later when ujail could not read the bundle. Report it where it happens and abort the creation. Fixes: d1fe47f ("uxc: provision and reap per-container volumes") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
uvol now publishes its volume operations through rpcd, so use that instead of forking the CLI whenever the object is there, and keep the exec path for systems whose uvol predates it. The reply carries exactly the exit codes the callers already interpret, so create-versus-resize and "already larger, kept" keep working unchanged. Two properties of rpcd's exec plugins shape this. Its worker is killed once the exec timeout expires, and the caller then sees a reply-less failure rather than a timeout status, so treat any call that produced no reply as failure and never match on a particular status. The client timeout is derived from rpcd's own, read over ubus like the fstab lookup in uxc_boot() already is, so raising one raises the other. A create killed that way leaves the volume write-only or write-pending with the orphaned formatter still holding the locks, and retrying the identical create blocks behind it and then reclaims it. Boot gating moves to the readiness query: containers with data volumes or an overlay wait for the .meta volume, containers whose bundle lives on a uvol volume wait for the backend itself, which also stops a backend that is merely still coming up from being reported as an interrupted upgrade. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Containers live in their own avl tree, so neither data method could ever reach them: get_data walked the services tree alone and set_data looked up the name there alone, returning "not found" for every container. Consumers already fold procd data into their runtime configuration, fw4 reading type "firewall" and odhcpd type "dhcp", and a container is exactly the publisher that wants it, because the data is dropped when the instance goes away and the host configuration it describes goes with it. Walk both trees when dumping, and fall back to the container tree when attaching. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Every container brought a bridge, an interface, a firewall zone with its forwardings and rules, and a DHCP section into /etc/config, committed at ten sites across three packages. That is a flash write to the overlay on every start and stop, and whatever a power cut interrupts stays behind, describing a container that no longer exists. Stopping a container also left the deletions staged in /tmp/.uci rather than applied, so the next "Save & Apply" decided what the host config would be. Publish it as procd data instead. A container's zone, forwardings, redirects, rules and DHCP section go into its own instance data, which procd drops when the instance goes away, so the host configuration cannot outlive what it describes. fw4 already folds data of type "firewall" into its ruleset and odhcpd now does the same for "dhcp", and both are told to reload with the config.change event they already trigger on, because attaching data to an instance emits no event of its own. Networks created on demand for bridged containers are shared, so they cannot hang off one container's instance. They live in the data of a service of their own, and the published set is the state: the list is read back to add or drop a member, which is why no state file is needed and why nothing is lost when the volume holding it is not mounted. The zone a container gets is closed, so the services the host offers into it are opened explicitly, the way /etc/config/firewall opens them for wan rather than fw4 inferring anything: DNS always, ICMPv6 whenever the container has any IPv6 at all because neighbour discovery is ICMPv6 over IP and a static-IPv6 container behind a closed zone cannot resolve its gateway, and DHCPv6 only when the container asks for it. Creating the interface no longer waits for a reload to bring it up, so the thirty second wait for the interface object to appear is gone. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Volume names reach uvol, which builds shell command lines from them in its backends, so a name carrying a quote and a semicolon would run as root. uxc composes names from registration data, which comes from package Makefiles, stack templates and hand-written files rather than from uxc itself, so check them before either the ubus or the exec path is taken. Names are restricted to alphanumerics, dot, underscore and hyphen, which still admits the content-addressed image names and the internal .meta volume. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
…g netifd Describe the host side of a container's network as procd service data and let netifd materialise it on config load, rather than creating the objects one by one through the runtime ubus API (network create_device, network add_dynamic, network.interface.X add_device/remove_device/remove). Only the read-only queries (interface status, device status, interface dump, service get_data) remain. Everything a single container needs is attached to its own instance in one set_data call: the veth pair as a network-device entry, the jailed end (and the routed /31 gateway) as network-interface entries, the port membership in a user-owned UCI bridge as a bridge-port entry, plus the firewall and dhcp sections that were already published this way. The blob is assembled by a single function that always emits all five type keys, because set_data replaces the whole instance blob and a partial publish would silently withdraw the container's link. The lifecycle of these objects is therefore bound to the instance record: withdrawing the data on down, or deleting the instance, removes them on the next reload without any explicit teardown call. Bridges that uxc-net owns (on-demand br-<net> autonets and bhr-<seg> backhaul segments) are shared between containers and live at service level on the uxc-net data service, since netifd replaces a bridge's ports list on every publish and exactly one publisher must hold the full union. The ports list of each owned bridge is the reference count: on every up and down the list is pruned to the veth names currently published by some instance, an owned bridge whose ports run empty is dropped together with its interface, zone and dhcp entries, and the service is deleted once nothing is left. The read-modify-write of the shared data, and the instance publish that follows it, are serialised with an exclusive flock so that concurrent container starts cannot lose an update or prune each other's pending port. Subnet selection for a new autonet also takes the published interfaces into account, not only netifd's runtime state. netifd re-reads procd data only during config load, so after each publish or withdrawal a synchronous network reload is issued before the firewall and dhcp config.change events. On up a failed reload is fatal, because the jailed interface and its veth must exist by the time ujail hands the network namespace over with netns_updown. The veth blob is byte-stable across republishes: MAC addresses come from the persisted store and are never regenerated, so a reload never restarts the pair and never destroys the container's eth0. The in-jail UCI rendering, annotation parsing and IPv6 handling are unchanged. The downstream-only persistent attribute is gone from all published interfaces. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The OCI runtime-spec default device symlinks give /dev/ptmx the target "pts/ptmx", relative to /dev. ujail created it with an absolute target, so a bundle that validates the symlink target sees a mismatch. Use the relative target the spec and the conformance suite expect. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
applyOCIlinuxseccomp() set PR_SET_NO_NEW_PRIVS unconditionally before installing the filter, so a bundle with process.noNewPrivileges left false still ended up with NoNewPrivs set once it carried a seccomp profile. The runtime-tools default configuration is exactly that, and runtimetest reads the flag back from /proc/self/status. The kernel accepts SECCOMP_SET_MODE_FILTER either under no-new-privs or from a task holding CAP_SYS_ADMIN in its user namespace, so the flag is not needed to install a filter, only the ordering matters. Follow runc: when noNewPrivileges is true, set the flag after the capability drop and install the filter last, so the filter need not permit the remaining setup syscalls; when it is false, install the filter in the container init before the capability drop, while the init still holds CAP_SYS_ADMIN in its own user namespace. The filter then has to permit the syscalls used up to execve (capset, prctl, setresuid, setgroups, umask, chdir, the landlock calls, execve), which is the same requirement runc imposes with that ordering. The ptrace injection path cannot serve the noNewPrivileges=false case at all: it acts after execve, when the bounding set of a typical bundle no longer contains CAP_SYS_ADMIN, which is why seccomp-inject sets the flag itself. Enforcing bundle profiles without no-new-privs therefore take the in-process route already used for SCMP_ACT_NOTIFY and flag-carrying profiles, where the linker-extended profile stays in place for the lifetime of the process instead of being narrowed at the entry point. The early install is confined to OCI bundles: a plain jail given a profile with -S also lands in opts.ociseccomp, and keeps the injection path whether or not procd passed -c, exactly as before. The CLOSE_RANGE_CLOEXEC sweep moves in front of the early install so that marking the inherited descriptors close-on-exec does not depend on the profile permitting close_range; the flag only takes effect at execve, and nothing opened between the two points is meant to survive it. Containers with noNewPrivileges=true are unaffected: they keep the flag, drop capabilities and get the filter last, whether injected or in-process. Fixes: ea7a790 ("jail: add support for running OCI bundle") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The field has been in the linux policy since OCI support arrived, but nothing consumed it: the value was parsed and dropped, and the root of the container kept whatever propagation the private remounts left it with. Map shared, slave, private and unbindable to their MS_* flags, all recursive, and apply the result to "/" as the last mount step once the rootfs is in place, from post_jail_fs(), which every path reaches after pivot_root() and, for a deferred user namespace, after the second unshare(CLONE_NEWNS) and the deferred masks. The MS_REC|MS_PRIVATE isolation applied earlier is untouched; it keeps the masks from leaking while the tree is built, and only the final propagation type of the container root changes, and only when the bundle asks for one. An empty string is treated as unset, as runc does; any other unknown value is rejected at parse time. The mount is skipped without a mount namespace, where "/" is the host root. Fixes: ea7a790 ("jail: add support for running OCI bundle") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The runtime spec makes process optional in config.json and requires start to fail when it is absent, while create is expected to succeed. parseOCI() refused the whole bundle with ENODATA instead, so such a container never reached the created state and start.t could not get to its assertion. Let create go ahead without a process section and have handle_start() refuse to run a container whose bundle defines none, logging the reason and answering UBUS_STATUS_INVALID_ARGUMENT like the other state rejections. A bundle without a process combined with -i, where ujail would start the container on its own right after create, is still rejected at parse time since there is nothing it could exec. uxc start now prints the ubus error of a refused start; it returned the status code silently before, and the operator saw nothing. Fixes: ea7a790 ("jail: add support for running OCI bundle") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The host side of a container network is dynamic: uxc-net publishes it as procd service data and netifd materialises the devices from that. A netifd restart drops those objects, and the container keeps running with its veth gone and nothing left to bring it back. Re-running uxc-net up is not enough, it re-plumbs the host end but never repeats the namespace hand-over done at create time by jail_network_attach() and the netns handling, so the container stays without its interface. Watch ubus.object.add for the network.interface object once the container is running, per container and only when it owns a network namespace, so the first appearance a jail sees is a genuine restart of netifd. On that event ujail stops its container the way uxc kill does, SIGTERM with the usual SIGKILL escalation, and lets the normal shutdown run: poststop withdraws the published network data through uxc-net down and runs the poststop hooks. Instead of exiting afterwards, the supervisor re-executes itself with its original arguments plus -i, so the fresh instance creates the container again, uxc-net up and the namespace hand-over included, and starts it right away. The pid and therefore procd's view of the instance do not change; procd has no restart operation for an instance and never respawns a container on its own, and uxc-managed instances created through create and start carry no -i, so a procd-side respawn would have stopped at the created state. The arguments are copied before option parsing, since getopt and the mount and namespace parsers split their operands in place, and the notify descriptor procd hands over with -a for the original create is left out of the copy handed to the new instance: its invoker returned long ago, and close-on-exec takes the descriptor with it. For the same reason the invoker is not signalled about an exit while the restart is under way. A restart is refused while a stop is already in flight, and a stop requested during the teardown wins over the pending restart, so a container being taken down by uxc kill or by procd is not revived. The handler is only registered once the container runs and the new instance registers its own only after it is up again, when the netifd object already exists, so a container cannot re-trigger itself. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
uxc delete failed with EIO when procd answered NOT_FOUND to the instance removal, and delete -f aborted when the kill it issued first found no container process any more. Both happen when the container exits between the runtime state being read and the request being made, and both leave the configuration behind although the instance is gone, which is the state delete is meant to reach. Treat NOT_FOUND from the procd delete as done, skip the removal wait in that case since no instance is left to report it, and let a forced delete continue when its kill reports the container as already gone. uxc_kill now distinguishes that case as -ENOENT instead of folding it into -EIO; kill itself keeps failing for a stopped container, as the spec requires. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
A bridged container joined exactly one VLAN of a VLAN-filtering bridge: bridge_of() turned the br-X.N device of the attach network into a single "N:*" element on the bridge-port entry, so the host end of the veth was an untagged access port and the container could reach no other VLAN of that bridge. Add the annotation org.openwrt.network.vlans, a comma-separated list of <vid>[:t] entries that has a meaning only on top of bridged:<net> where <net> sits on a VLAN-filtering bridge. The attach network's own VLAN stays the untagged PVID member exactly as before; a bare <vid> adds the port as a further untagged member and <vid>:t adds it as a tagged trunk member. Without the annotation nothing changes, neither for a plain bridge nor for a single VLAN. On the host the bridge-port entry keeps its one "vlans" array, which now carries every requested element, for example ["20:*","30:t","40:t"]. netifd merges each element into the ports list of the UCI bridge-vlan section with that VID, so a single entry puts the veth into several VLANs with the right flag on each. VIDs must lie within 1 to 4094, identical entries are folded, and an entry is rejected when it is listed both tagged and untagged, when it names the attach network's own VLAN, or when the bridge has no such VLAN according to network.device status. The last check matters because netifd merges only into bridge-vlan sections that exist and a port of a VLAN-filtering bridge without any membership forwards nothing, so accepting the VID would leave a dead interface in the container. Networks created on demand are plain bridges owned by uxc-net and refuse the annotation. Inside the container a tagged VLAN is reachable only through an 802.1Q sub-interface, so for every <vid>:t the in-jail network file gains a device section of type 8021q named eth0.<vid> on top of eth0 and an interface 'vlan<vid>' on that device. The same netifd runs inside the jail and creates the device when the interface claims it, given the 8021q module on the host kernel. Untagged members need nothing inside: their frames are the untagged traffic on eth0. The VLAN interfaces default to proto none and switch to static when org.openwrt.network.vlan.<vid>.address supplies an address/prefix. A container on a trunk typically uses its extra VLANs for plain L2 or for a service with a fixed address; a DHCP client on every VLAN is unusual and would compete for the default route that eth0 already provides. Teardown is unchanged: withdrawing the instance data removes the bridge-port entry with all of its memberships on the next reload, and the in-jail file is per container. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series adds OCI runtime-spec 1.3.0 support to procd's process jail (
ujail) and builds OpenWrt's native container manager,uxc, on top of it, so containers run as ordinary procd services rather than under a separate daemon stack.Main areas:
ujailas an OCI 1.3.0 runtime: namespaces including the user namespace, cgroup v2 resources, capabilities (empty by default), OCI prestart/createRuntime/poststart hooks, devices, sysctl, rlimits, masked and read-only paths, rootfs propagation, and rootless containers via idmapped mounts. Seccomp is applied by ptrace syscall injection, dropping theLD_PRELOADhelper, with enforce, trace, audit and complain modes and a floor that is deliberately tighter than runc/crun.uxc: a runc/crun-compatible CLI plus procd boot integration, usable both standalone and as the low-level OCI runtime under Podman (caller stdio pass-through, exec, pause/resume, and exit-status hand-off through conmon)./etc/configwrites:uxc-netpublishes the host-side setup (network devices, interfaces, firewall zones and forwardings, and dhcp) as ephemeral procd service data attached to each container's own instance, consumed by netifd, firewall4 and odhcpd on reload, so the state dies with the container. Bridged, routed and multi-VLAN attachment are supported, anduxc-stackcomposes multi-container stacks.The consumer side lives in companion PRs:
bridge-portprocd data type plus jail/bridge fixes)Should fix #35.
This is a large series carrying the whole container-runtime stack rather than a minimal change, and the most recent networking work (full reconvergence of running containers across a netifd restart) is still being hardened.