feat(#126/#127): mlisp-bugs filter hooks + subscriber ask command#136
Open
denzuko wants to merge 6 commits into
Open
feat(#126/#127): mlisp-bugs filter hooks + subscriber ask command#136denzuko wants to merge 6 commits into
denzuko wants to merge 6 commits into
Conversation
## #130 -- Streaming base64 encoder base64-encode-file-streaming writes to a stream in 57-byte input / 76-char output chunks (RFC 2045 line length). Never holds more than one chunk in memory -- safe for arbitrarily large files. base64-encode-file (legacy) now delegates to the streaming version via with-output-to-string. distrib-get also updated to use streaming. ## #131 -- yEnc multipart chunking distrib-file no longer rejects files above max-file-size-kb. Instead: - Files at/below :segment-size-kb (default 750KB): single message, base64 encoded (streaming). Unchanged from before for small files. - Files above :segment-size-kb: yEnc multipart segments, one message per segment, subject '[list-id] fname (N/total)'. New functions (all exported for test suite): base64-encode-file-streaming (path out-stream) yenc-encode-byte (byte) -- (byte+42)%256, escapes 214/224/227/19 yenc-encode-bytes (bytes) -- vector->string, 128-char line wrap yenc-decode-bytes (str) -- inverse (for roundtrip verification) compute-segments (path bytes) -- list of (offset size) pairs segment-subject (list fname n total) -- '[list] fname (N/total)' yenc-segment-header (...) -- =ybegin/=ypart fields (1-indexed) yenc-segment-footer (...) -- =yend size/pcrc32 yEnc encoding note: the bytes to escape (214/224/227/19) are the INPUT bytes whose encoded form (input+42)%256 is a critical char (NUL/LF/CR/=). NOT the raw bytes 0/10/13/61 themselves. The escaped output is '=' followed by (encoded+64)%256, giving =@/=J/=M/=} respectively. The spec also mentions dot (.) line-stuffing but that is handled by the MTA transport layer, not the encoder. ## Spec fixes YENC-1: corrected from testing raw bytes 0/10/13/61 (wrong) to testing input bytes 214/224/227/19 (correct). YENC-2: updated to use bytes 65/214/224/66, expects k/=@/=J/l. CHUNK-5: yEnc is 1-indexed (begin=1 not begin=0 for first segment). DYENC-4: 1600KB / 750KB = ceil(2.13) = 3 segments, not 2. 478/478 (400 BATS + 78 FiveAM, was 468/468 -- +10 BATS DSTRM/DYENC).
## #126 -- mlisp-bugs pre/post-filter hooks on bugs-submit src/bugs.lisp: bugs-set-option (pkg key value): set arbitrary plist keys on a bugs package entry. Supported: :pre-filter :post-filter :ai-ask. bugs-invoke-filter (filter-programs headers body-lines): delegates to invoke-filter-chain from src/filters.lisp. Same exit-code contract as mlisp list pre/post-filters (0=pass 1=reject 3=discard). bugs-process-submit: pre-filter runs before submission; non-zero exit aborts with exit 1. post-filter runs after archival, before distribution (annotation-only pattern: the post-filter modifies the distributed copy, not the archive). Also fixed: bugs-process-submit was returning bug-id (e.g. 1 for the first bug) which bugs-main.lisp used as the exit code, causing mlisp-bugs to exit 1 (error) on every first submission. Now returns 0 on success. BF-1 test caught this pre-existing bug. src/admin.lisp: bugs-set-option <pkg> <key> <value>: new admin subcommand. Registered in the dispatch table alongside bugs-report. src/package.lisp: exports bugs-set-option, bugs-invoke-filter. ## #127 -- subscriber ask command src/commands.lisp: :ask dispatch added. Matches 'ask' or 'ask <question>' in Subject or first body line. src/requests.lisp: handle-ask-command: if :ai-ask is configured on the list (via mlisp-admin set-option <list> ai-ask <cmd>), pipes the question through pipe-through-command and replies with the output. Falls back to handle-ask-fallback (list info + command list) when :ai-ask is not configured or the command produces no output. handle-ask-fallback: plain-text fallback reply listing available subscriber commands. src/main.lisp: :ask case added to the request command dispatch. src/package.lisp: exports handle-ask-command. ## Specs (13/13 BATS) BF-1..6: bugs-submit without filter, pre-filter allow/reject/annotate, post-filter invocation, bugs-set-option config storage. ASK-1..7: ask dispatch (no AI), with neural stub, off by default, reply addressed to From:, subject echoed, Subject: dispatch, body-first-line dispatch. 491/491 (413 BATS + 78 FiveAM, was 478/478 on develop -- +13 BF/ASK specs +2 FLT specs from cherry-picked distrib yEnc work that was missing from this branch).
bugs-process-submit now returns 1 (integer) on pre-filter rejection instead of always returning the bug-id. bugs-main.lisp's submit branch was discarding the return value and always returning 0. Fix: use the return value as the exit code when it's numeric.
This was referenced Jun 19, 2026
CLAUDE.md: project-wide internal knowledge base for Claude sessions. Covers: core architectural philosophy (email as actor-based pub/sub), GitFlow/BDD workflow rules, system architecture (subgroups, filter pipeline, distrib, microservice pattern), key design constraints (zero-dep mlisp.asd, neural.sh scope, GPG out of scope), future roadmap (Q3 2026 anon networks, RT4-TWG, Watchers series). docs/skills/email-pubsub-architecture.md: reusable skill document covering the actor model over email, mapping to conventional pub/sub primitives, historical lineage (BITNET/FidoNet/LISTSERV/debbugs), why this beats Redis+Celery (supply chain, no broker, 40yr operational history), mlisp-specific architecture, when to use/not use the pattern, and future work directions. Key insight documented: email is inherently async -- latency inside the SMTP transaction is invisible to users. Do not import web/chat latency expectations into this domain. Processing time is not a bottleneck concern at any realistic mailing list traffic level (floor: NYCbug ~50 msgs/week; ceiling: Linux kernel ~2000 msgs/day).
denzuko
commented
Jun 19, 2026
| exit 0=pass 1=reject 3=discard | ||
| Space-separated list for a filter chain. | ||
| post-filter <path> Filter run after archival, before distribution. | ||
| ai-ask <cmd> Command for subscriber ask queries (neural.sh etc.)." |
Owner
Author
There was a problem hiding this comment.
neural.sh is to be implemented as a pre/post filter add-onif a milter/filter is just a script or program then if neural.sh is used in chain or on is own, why does need its own key?
| - P2P mesh networks (FidoNet-style store-and-forward) | ||
| - Anonymous remailers (Type I/II) with mlisp as list management layer | ||
|
|
||
| **Intended audiences:** |
Owner
Author
There was a problem hiding this comment.
this is internal notes not for public release; the tool is to be generic and flexible in use case not specifically targeted as an internal tool.
PR #136 feedback: 1. src/requests.lisp / src/admin.lisp: :ai-ask key removed. The ask handler now reads :ask-filter from list options -- a neutral key naming any program (not AI-specific). Using :pre-filter was also wrong: pre-filter runs on all inbound messages at delivery time; ask needs to invoke the filter directly against the question string only. :ask-filter is distinct from :pre-filter and invoked only within handle-ask-command via pipe-through-command (same mechanism as bugs-report --summarize). The operator configures any program: neural.sh, a lookup script, a KB query, etc. No AI-specific framing. 2. CLAUDE.md: Removed Watchers series reference from roadmap (internal project name). Removed deployment context section (operator-specific infrastructure). Tool described as generic and flexible. 3. docs/skills/email-pubsub-architecture.md: Removed operator attribution from mlisp entry. Generalised RT4-TWG reference to a generic standardisation note. 4. test/bats/test_mlisp_126_127.bats: ASK-2 updated: set-option ask-filter (not pre-filter, not ai-ask). ASK-3 description updated to reference ask-filter. 491/491, CI pending.
Corrected architectural error from previous iteration.
The ask command is a -request command. The user talks to mlisp.
mlisp's job is to recognise the command and let the filter pipeline
handle it. The filter is a complete self-contained actor:
#!/bin/sh
BODY=$(cat)
REPLY=$(neural.sh "$BODY")
( echo "To: ..."; echo ""; echo "$REPLY" ) | sendmail -t
exit 3 # discard: mlisp should not send a second reply
mlisp's pre-filter already runs before command dispatch (main.lisp
step 3x). If the list owner configures a pre-filter on -request, it
handles the ask interaction entirely -- no additional key, no special
integration point, no pipe-through-command in mlisp code.
handle-ask-command is now minimal:
- Dispatched by -request when Subject/body starts with 'ask'
- Sends a plain-text fallback (list info + commands) when no
pre-filter is configured
- When a pre-filter IS configured, it runs before handle-ask-command
is even reached (main.lisp 3x), handles the response itself, and
exits 3 (discard) so mlisp stops
Removed: :ask-filter key (unnecessary), pipe-through-command call
in handle-ask-command (wrong layer), ask-filter documentation in
bugs-set-option (not needed there either).
ASK-2 test updated: pre-filter is a complete actor that sends its
own reply and exits 3. mlisp never touches the response content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two features split out from #100 use cases 2 and 1 respectively.
#126 — mlisp-bugs pre/post-filter hook
Adds the same
:pre-filter/:post-filterhook pattern that mlisp lists already have, now available for bug packages.New functions:
bugs-set-option pkg key value— sets config on a bugs package (:pre-filter,:post-filter,:ai-ask)bugs-invoke-filter filter-programs headers body-lines— delegates toinvoke-filter-chain(same exit-code contract: 0=pass, 1=reject, 3=discard)Updated:
bugs-process-submit— pre-filter runs before submission (non-zero aborts); post-filter runs after archival, before distribution (annotation pattern).Also fixed:
bugs-process-submitwas returningbug-idas an exit code —mlisp-bugsexited 1 on every first bug submission because bug-id=1 looked like an error. Now returns 0 on success.New admin command:
bugs-set-option <pkg> <key> <value>#127 — subscriber ask command
:askcommand on-requestlists:Subject: ask <question>or first body lineask <question>.If
:ai-askis configured (mlisp-admin set-option <list> ai-ask <neural-cmd>), pipes the question throughpipe-through-commandand replies with the output. Falls back to a helpful plain-text reply listing available subscriber commands when:ai-askis not set or produces no output (e.g. neural endpoint unreachable).Specs: 13/13 BATS (BF-1..6, ASK-1..7)
491/491 locally (413 BATS + 78 FiveAM).