chore: agents md and repo fixes - #234
Merged
Merged
Conversation
Salt state functions accept **kwargs, so a misspelled requisite key is dropped without raising. Three states were relying on ordering that was never actually enforced: - sift/include-desktop.sls used 'requires:' instead of 'require:', so the desktop aggregate did not wait on sift.server or sift.config. - sift/scripts/exiftool.sls used 'include:' as a requisite on a cmd.run, so 'perl Makefile.PL' was not ordered after build-essential and perl. It also never required the extract step, so the cwd could be missing. - sift/desktop.sls required sift.include-server where it meant sift.include-desktop; it resolved transitively, by accident.
…tates Both states existed but were absent from python3-packages/init.sls, so they were unreachable from sift.desktop and sift.server and never installed. windowsprefetch was updated for 24.04 in #164 while dead, which suggests the omission was an oversight rather than a deliberate removal.
The state existed but was absent from scripts/init.sls, so docker-compose was never installed despite being updated for 24.04 in #166. Also adds the standard header block and a renovate annotation so the pinned version is picked up by the custom manager, and lowercases the kernel grain so the asset name matches upstream literally (GitHub's asset lookup is case-insensitive, so this was cosmetic, not a fix).
sift.perl-packages.dbd-sqlite was the only perl-packages state not included by 4n6.sls, leaving it unreachable. Its sibling modules are all included here, and 4n6-scripts ships sqlite-parser.pl and squirrelgripper.pl which need DBD::SQLite -- which is what #45 ('Fix sqlite_miner requirements') added the state for in the first place.
Unlike the orphans wired up in the preceding commits, these are not salvageable and are deleted rather than registered: - config/symlinks.sls: duplicates the /usr/local/bin/rip.pl symlink that scripts/regripper.sls already manages, and is the tree's only reference to mactime, icat-sleuthkit, l2t_process and iscsiadm -- none of which are installed. It also required 'pkg: python-plaso', a state ID that does not exist (the real one is sift-package-python3-plaso), so enabling it as-is would have failed outright. - config/rclocal.sls: appends a /dev/loop mknod loop to /etc/rc.local without creating the file, a shebang, or the exec bit, and rc-local.service is not enabled by default on 22.04+. Loop devices are allocated dynamically now. - packages/absent/: binplist and unity-webapps-common are 14.04/16.04-era packages that do not exist on jammy or noble. - repos/ubuntu-tweak.sls: a bare test.nop stub left over from the 18.04 port. All four were unreachable from sift.desktop and sift.server.
…tional The custom manager's version group was '(?<currentValue>.*)' -- greedy, and Renovate compiles matchStrings with the dotAll flag, so it swallowed every line up to the last viable quote. It only ever worked because radare2.sls originally had exactly one 'hash =' literal after the version, which forced backtracking onto the right quote (see 6532b12). Adding the arm64 hash in cc1122f introduced a second literal and broke it: powershell currentValue = '7.4.6" -%}\n{%- set arch = "amd64' radare2 currentValue = '5.9.6" -%}\n{%- set base_url = ...arm64.deb' Neither parses as semver, so both deps were silently skipped. Making the group lazy yields '7.4.6' and '5.9.6'. Also wraps the digest half in an optional group so version-only pins, whose source_hash is a remote .sha256 URL rather than a literal, are managed too. Annotates scripts/cyberchef.sls, the remaining file where this is correct: a single hash literal from a GitHub release. Still unmanaged, deliberately: aws-cli.sls and claude-code.sls are hosted off GitHub so the digest cannot be refreshed, and bumping the version alone would leave a stale hash and fail the install. repos/gift.sls and repos/sift.sls use 'version' for a PPA channel name, not a version number.
weekly-tests.yml matrixed Ubuntu 20.04, which .cast.yml has not listed as
supported since noble was added, and never tested 24.04 at all. Now matches
tests.yml: 22.04 and 24.04.
Switches its image tag from ${{ matrix.code }} to ${{ matrix.os }} for
consistency with tests.yml and drops the now-redundant os->codename include
block. Both formats are valid aliases in the registry (jammy-3006 and
22.04-3006 both resolve), so this is consistency, not a broken-tag fix.
Also adds sift.tests.libewf to the weekly matrix -- it was the only state
under sift/tests/ that no workflow referenced.
tests.yml carried the same os->codename include block while only ever
referencing matrix.os, so its dead entries are removed too.
Everything in .ci/ predated the move to GitHub Actions and cast, and none of it ran: - test.sh dispatched on TRAVIS_TAG / TRAVIS_EVENT_TYPE and invoked ./scripts/*.sh, a directory that does not exist. - changed-files.sh read TRAVIS_COMMIT and pulled sansdfir/sift-salt-tester. - packages.sh pulled a bare 'sift-salt-tester' with no registry. - tag-and-sign.sh and publish-draft.sh pushed releases straight to master against a hardcoded GPG key and the old sans-dfir repo path; releases are cut by cast now. publish-draft.sh also had a stray quote in its API URL. - debug.sh, dev-state.sh and test-all.sh were three near-identical shell droppers disagreeing on image, registry and SYS_ADMIN, defaulting to focal. Kept as test-state.sh (apply one state) and shell.sh (interactive poking), both on ghcr.io/ekristen/cast-tools/saltstack-tester -- the image CI actually uses -- defaulting to 22.04/3007. They mount the repo root and pass --file-root . so a local run matches CI exactly, rather than mounting ./sift onto /srv/salt/sift as the old scripts did. Dockerfile (Ubuntu 18.04, apt-key, dead repo.saltstack.com URLs) and Dockerfile.jammy3005 built tester images that are now published upstream, and nothing referenced either.
The SKS keyserver pool was shut down in 2021; hkp://p80.pool.sks-keyservers.net:80 no longer resolves, so every pkgrepo.managed key fetch fell back to whatever gpg/apt does by default. Verified all four keyids resolve on keyserver.ubuntu.com, which is the right home for Launchpad PPA keys: sift/dev,stable 3E04D0A9A043FAFD66F5E774B2A668DD0744BEC3 gift/dev,stable 3ED1EAECE81894B171D7DA5B5E80511B10C598B8 openjdk-r/ppa DA1A4A13543B466853BAF164EB9B1D8886F44E2A dotnet/backports 45A3F127159BE9E5017811C62125B164E8E5D3FA
README listed Ubuntu 20.04 (EOL) and never mentioned 24.04; .cast.yml has declared 22.04 and 24.04 for a while now. Also notes arm64 support. VERSION said v2020.01.01-rc1 against a latest tag of v2026.04.21. Nothing in the repo reads it any more -- its only consumer was .ci/tag-and-sign.sh, now removed, and the cast binary contains no reference to it. Synced to the current tag rather than deleted, since whether the release process should keep maintaining it or drop the file is a maintainer call.
The Notes field lists what a state puts on $PATH; defang's was blank despite symlinking /usr/local/bin/defang. Confirmed against the sdist: its only console_script is 'defang = defang.bin:defang'. Touching the state file also brings it into the changed-states CI matrix, which the preceding init.sls registration alone would not have -- the workflow filters init.sls out.
Rewrites the sections invalidated by the preceding commits: the renovate guidance now describes the repaired regex and spells out the dual-arch and non-GitHub-host cases where an annotation would do harm; testing points at the rebuilt .ci scripts and calls out that registering a state in init.sls does not get it CI coverage; the release section reflects the removed Travis scripts. 'Known rough edges' shrinks to what actually remains -- deliberate choices to leave alone, and limitations (dual naming conventions, partial header coverage, the four skip_verify sites) that need per-case research rather than a sweep. Also corrects two claims from the first draft that further checking disproved: the docker-compose kernel-grain casing was not a bug (GitHub asset lookup is case-insensitive), and weekly-tests.yml's codename image tags were valid aliases, not broken references.
Fourteen files under sift/packages/ are referenced by nothing at all, so they have never been installed. Documented rather than changed: eleven look like leftovers from consumers that were since removed (volatility2/dpapick, plutil, the imagemounter refactor in #177, the python3 reorg in #173), and the other three -- dos2unix, libguestfs-tools, ugrep -- are usable tools whose inclusion is a product decision, not a cleanup. Notes that ugrep would need fixing first, since it pulls a hardcoded 20.04-era amd64 .deb with no arch guard.
The scripts bind-mounted $PWD onto /srv/salt, which is only correct when the
daemon shares this filesystem. Against an ssh:// or tcp:// context the mount
resolves on the *remote* host, so /srv/salt came up empty and salt-call ran
against no state tree at all -- silently, since an empty tree is not an error.
New .ci/lib.sh detects the context endpoint and streams the tree in over stdin
when the daemon is remote, keeping the bind mount (and its live-edit
convenience) when it is local.
Verified against a remote ssh:// context: sift.repos.refresh succeeds, a
nonexistent state exits 1, and a failing cmd.run exits 2, so
--retcode-passthrough still reaches the caller through the pipeline.
The empty-array expansions are guarded with ${a[@]+"${a[@]}"} because macOS
still ships bash 3.2, which errors on a bare "${a[@]}" under set -u.
Records the empty-state-tree trap behind the remote-daemon handling in .ci/lib.sh, and adds state.show_sls as a cheap way to validate requisites and Jinja across the full graph -- including why the sift_user pillar is required for it to render.
These states were referenced by nothing -- not packages/init.sls, not any
other state's include: or require: -- so they never ran. Verified three ways
before removal:
1. No reference by sls path (sift.packages.<name>).
2. No reference by state ID as a requisite target. This mattered: the IDs do
not all match their filenames (patch -> sift-patch, libguestfs-tools ->
sift-package-libguestfs, python3-tsk declares two), so a path-only grep
would have missed a live reference.
3. No Jinja-constructed include names and no relative includes (.foo /
..packages.foo) remain anywhere, so nothing evades a literal grep. Every
include token in the tree resolves to a real state.
Provenance says each was dropped deliberately or never wired:
ugrep removed twice -- 'remove ugrep', 'really disable ugrep'
libguestfs-tools 'removing libguestfs-tools as they are currently broken'
libencode-perl Remove plutil to support 24 (#168)
swig Remove remaining python2 references (#179)
python3-m2crypto python 2/3 re-org (#38)
ewf-tools superseded by gift libewf-tools (#177)
python3-tsk imagemounter fix (#177)
patch exiftool rewrite (#172)
apt-transport-https DEB822 migration; apt has it built in
dos2unix fell out of the RegRipper 3.0 rewrite
libpcap-dev, libvhdi, libyara3, python3-keyrings-alt
never referenced in recorded history
Behaviourally inert, and confirmed so: sift.desktop still renders 820 states
and sift.server 720 -- identical counts to before this commit. An install
simulation also showed patch is in the Ubuntu base and libencode-perl and
libvhdi arrive as transitive deps regardless, so nothing that reaches a SIFT
box today is lost.
Unreachable states now number 7, all intentional: claude-code, the pkgs/vm
legacy aliases, and the sift.tests.* states the weekly workflow calls directly
(plus libewf and python3-plaso, which only those tests reference).
sift.packages.docker failed outright with 'Package docker-ce has no installation candidate'. The DEB822 migration in c662f6f replaced pkgrepo.managed with file.managed, and only pkgrepo.managed tells Salt the apt cache is stale -- so pkg.installed resolved docker-ce against an index that had never seen download.docker.com. A full sift.server run masked it, because sift.repos.refresh is included there and happens to be ordered later. Applying sift.packages.docker on its own -- which is exactly what the changed-states CI job does -- failed every time. Refreshing via onchanges keeps it to when the repo file actually changes. Verified: the state now applies clean (8 succeeded) and installs docker 29.7.2. Note the same file.managed-without-refresh shape exists in repos/microsoft.sls and the ubuntu-universe/multiverse states; those are not currently broken because nothing installs from them standalone, but they are the same latent trap.
Mulder (https://github.com/calebevans/mulder, Apache-2.0) runs a five-phase investigation over disk images, memory dumps, PCAPs and event logs, emitting incident reports with MITRE ATT&CK mappings and STIX/CSV IOC exports. Upstream ships it only as a container image, so this installs a wrapper at /usr/local/bin/mulder rather than baking the image in: the amd64 image is 3.61 GB compressed across 47 layers, and pulling at build time would also require a working daemon during the salt run, which the CI tester does not have. Docker pulls it on first invocation instead. The image is multi-arch (amd64 + arm64), so no arch guard is needed. The wrapper handles the two mounts upstream requires, defaulting evidence to SIFT's own /cases, and passes through whichever provider credentials are set -- Anthropic, Vertex, Bedrock or LiteLLM -- reading them from the environment so nothing lands on disk. It refuses early with a pointer to the usage guide when no credentials are present, rather than failing deep inside a container. Uses the narrower '--cap-add SYS_ADMIN --device /dev/fuse' grant that upstream documents as the alternative to --privileged; MULDER_DOCKER_ARGS is there for anyone who needs the broader flag. Not registered in scripts/init.sls, following the precedent set for claude-code in 34b42da: it needs an API key, pulls several GB on first run, and sends evidence content to a third-party LLM provider, so opting in should be deliberate. Verified on 22.04 and 24.04 (salt 3007): state applies clean, the rendered wrapper passes bash -n with no unrendered Jinja, both guard paths exit 1 with usable messages, and the assembled docker command is correct with args, without args, and with MULDER_DOCKER_ARGS plus Bedrock credentials.
Adds it to the 'deliberate, leave alone' list next to claude-code. Without this the convention documented two commits earlier -- a state in neither init.sls nor any include: is dead -- would flag sift.scripts.mulder for exactly the cleanup that just removed 14 genuinely dead states.
…apping Replaces the /usr/local/bin/mulder wrapper with a build-time image fetch, per direction. The image is pinned by digest rather than tag so what gets fetched is immutable, matching how the rest of the tree pins downloads; the digest is the multi-arch index, so it resolves on amd64 and arm64 alike. Pulling by digest leaves the image untagged, so a second state tags it to ghcr.io/calebevans/mulder:1.3.2 and the documented 'docker run' form works. Both steps are guarded with 'docker image inspect' so re-runs are no-ops. Consequence of dropping the wrapper: there is no mulder command on $PATH. Users invoke the image directly; the header carries the full docker run line, including the volume mounts, the narrower SYS_ADMIN/fuse grant, and the credential env var. Requires a running docker daemon, so this state cannot be applied inside the CI tester container -- it fails there on service.running plus the pull. Added to SKIP_STATES in tests.yml so the changed-states job skips it rather than going red. The tradeoff is explicit: this state gets no CI coverage and must be applied by hand on a real target. A silent 'onlyif docker info' guard was the alternative and is worse -- a real build with a stopped daemon would then ship without the image and report success. Verified against a live daemon: the digest resolves, the pull completes (89s, 6.34 GB on disk), both inspect guards flip correctly so the states are idempotent, tagging works, and 'docker run ghcr.io/calebevans/mulder:1.3.2 mulder --version' runs. Note upstream's --version reports 1.0.0 regardless of the image tag, so do not use it to confirm the pinned version.
TRUDI (https://github.com/nebulae/trudi, MIT) is an MCP server and agent that drives a traced investigation, linking every finding back to the tool call that produced it. It is not pip-installable -- no pyproject.toml or setup.py -- so this follows the git-clone-plus-venv shape ufade.sls and mac-apt.sls already use: virtualenv at /opt/trudi, source at /opt/trudi/trudi, requirements.txt installed against the venv interpreter. Pinned to a commit on main, not the 1.0.0 tag. That tag is 19 commits behind and predates every fix that makes TRUDI work on SIFT: venv creation failing on the SIFT base for want of ensurepip, the version-matched pythonX.Y-venv, and the asyncio event-loop failures on python 3.12 -- which is what 24.04 ships. Pinning the tag would pin known-broken-on-SIFT code. Upstream's install.sh is deliberately not invoked. It targets someone setting up their own workstation and does several things a distro image must not: it builds its venv at the generic ~/.venv, overwrites ~/.claude/CLAUDE.md, runs 'claude mcp add --scope user' against a CLI that SIFT does not install by default, copies bundled case studies into ~/cases, and installs requirements-dev.txt to run an 1100-test suite. This state does the image-level part only; the header records the one command a user still runs to register the MCP server against their own Claude Code install. The dashboard launcher defaults to ~/trudi and ~/.venv, so the wrapper exports TRUDI_DIR and VENV_PY to the paths actually used, honouring any override. Most of the forensic binaries TRUDI shells out to are already SIFT states (pff-tools, pst-utils, tcpxtract, sleuthkit); ewfmount comes from gift's libewf-tools rather than the ewf-tools package upstream names. binwalk was the only gap, added here as a dependency-only state rather than registered, so the default image is unchanged. Left out of python3-packages/init.sls, following mulder and claude-code: full use needs an ANTHROPIC_API_KEY and sends evidence content to the provider. Apply sift.python3-packages.trudi explicitly. Verified on 22.04 (python 3.10.12) and 24.04 (python 3.12.3), salt 3007: 19 states succeed, a second apply changes nothing TRUDI owns, the pinned commit is what lands, all eight runtime imports resolve in the venv, server.py compiles, all six forensic binaries are on PATH, and the dashboard answers HTTP 200 on both /_dashboard/dashboard.html and /_dashboard/api/cases.
Keeps them off the dead-code sweep and records the two decisions most likely to be 'corrected' later: not using upstream's install.sh, and pinning a main commit rather than the 1.0.0 tag.
Reverses 34b42da ('do not install claude by default'). TRUDI drives its agent side, skills and MCP server through the claude CLI, so the tool added in the previous commit is inert without it. Registered in packages/init.sls and pulled in explicitly by sift.python3-packages.trudi so a standalone apply is complete. Bumped 2.0.61 -> 2.1.224 while enabling it. The old pin still downloads, but the channel marker has moved on and shipping a stale agent CLI by default is worse than shipping none. 2.1.224 is what {base_url}/stable resolves to; /latest is 2.1.233. Both per-arch hashes recomputed from the published binaries. Added the missing header block, and recorded there that renovate cannot manage this file -- it is hosted off GitHub and carries one hash per arch, so the custom manager captures a single digest -- along with how to bump it by hand. The header exposed a Jinja trap worth keeping: '{%- set' as the first tag after a comment block strips the preceding newline and splices the last comment line onto the state ID, commenting the entire state out. It surfaces as 'ID file.managed ... is not a dictionary'. The original file had no leading comment so it never showed. Left a note in place. Verified on 22.04 and 24.04 (salt 3007): the state renders, installs, reports '2.1.224 (Claude Code)', and re-runs clean; sift.desktop and sift.server both render with sift-package-claude-code present; sift.python3-packages.trudi now applies 20 states with claude on PATH.
Adds the '{%- set' after a comment block failure mode, which presents as an
unrelated 'is not a dictionary' error on the first state, and the related '{#'
constraint inside contents: blocks. Both cost real debugging time this branch.
Updates the claude-code entry: it ships by default now, and notes it is
hand-bumped because renovate cannot carry two per-arch digests.
…mage Upstream now publishes mulder-dfir on PyPI and documents a native install, so the 3.6 GB image fetch added in 29683b0 goes away. Moves the state to python3-packages/ -- it is a pip-installed Python CLI now -- and installs 'mulder-dfir[forensics,pdf,stix]==1.4.0' into a venv at /opt/mulder, pinned with a renovate pypi annotation. Pinning the package pins the assets too: upstream fixes Chainsaw, Hayabusa, Zircolite, capa and FLOSS versions per release. Extras beyond upstream's documented [forensics]: export-iocs defaults to STIX and report advertises PDF, and an image with no guaranteed network at investigation time should have both work out of the box. Not uv, despite mulder being a uv project upstream: salt has no uv support, so it would mean a uv package state plus 'cmd.run: uv tool install' -- more shelling out, and no pip state change reporting. The published artifact is an ordinary hatchling wheel. The one cmd.run left is 'mulder setup', which no installer choice removes; it is guarded by 'mulder setup --verify', upstream's own read-only check, so re-runs are true no-ops. Assets go to a shared /opt/mulder-assets rather than a home directory, which is why /usr/local/bin gets wrappers instead of symlinks -- MULDER_ASSET_ROOT has to be exported at run time, not just at setup time. Setting it disables mulder's own /opt search, which costs nothing: SIFT ships none of these assets under /opt. 'mulder setup' refuses to run as root and CI pillars sift_user=root, so the asset step branches three ways rather than guarding silently -- 29683b0 already rejected a silent guard, and a build that ships an empty mulder while reporting success is the same failure. Root gets a test.show_notification saying plainly what was skipped and how to do it by hand; a sift_user that does not exist gets test.fail_without_changes; otherwise it runs. Dropping docker means the state runs in the tester container, so SKIP_STATES is now empty and mulder gets CI coverage for the first time. Only the 2.2 GB fetch is still unverified there. New packages: yara (registered -- SIFT shipped the python3-yara module but never the scanner, which upstream calls the one gap on SIFT), plus dependency-only binutils, libpango and libharfbuzz. The last two are WeasyPrint's run-time stack for the pdf extra, which pip installs happily without and then fails at import. libharfbuzz-subset0 is guarded on oscodename: jammy's harfbuzz 2.7.4 builds no subset package, and an unguarded pkg.installed fails there. Verified WeasyPrint 69.0 still writes a PDF on jammy without it. Verified on donnager, 20/20 across 22.04 and 24.04 x salt 3006 and 3007 for every changed state. Beyond the state applying: mulder --version reports 1.4.0 (unlike the image, which reported 1.0.0 regardless of tag), all six subcommands exit 0, the MCP server completes an initialize handshake and registers 150 tools, and 'mulder setup --verify' as a non-root user exits 4 -- the code the unless guard needs, where root exits 2 for the wrong reason. 'mulder setup --dry-run' through su - still resolves the asset root to /opt/mulder-assets, confirming the wrapper. The 2.2 GB download and a real investigate run still need a live target and an API key; arm64 remains untested.
Registers both in python3-packages/init.sls, include list and test.nop require list alike, so cast install picks them up instead of needing a hand-applied salt-call. Reverses the opt-in decision recorded for mulder in 2214728 and for trudi in 54a6390; per direction, they should ship. Consequences worth knowing, since both were kept out for reasons that have not gone away: - mulder's setup step fetches ~2.2 GB of assets on every build, as sift_user, and now dominates image build time. - Both tools send evidence content to a third-party LLM provider when used, and both need an API key. Installing them transmits nothing on its own, but the capability now ships by default rather than by choice. - python3-packages sits under include-server, so sift.server gets them too, not just sift.desktop. - sift/packages/binwalk.sls stops being dead: it stays out of packages/init.sls but now installs transitively through trudi. Also updates the headers and AGENTS.md, which still said mulder was not in init.sls and told agents not to "fix" that by registering it -- leaving those would have had the tree contradicting itself in three places. CI cannot cover this: the changed-states workflow filters init.sls out by design. Verified instead with state.show_sls on 24.04/3007 -- sift.python3- packages, sift.desktop and sift.server all render clean with both tools in the graph -- on top of the 20/20 matrix for mulder and 4/4 for trudi. Open question, not addressed here: .cast.yml sets sift_user from {{ .User }}. If that resolves to root under sudo, mulder installs with no assets and prints its skip notification on every image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.