Merge upstream v0.7.0 into the fork - #11
Merged
Merged
Conversation
Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
- emit_product_log imported LogRecord from opentelemetry.sdk._logs, a private class recent opentelemetry-sdk releases renamed to ReadWriteLogRecord/ReadableLogRecord. Every emit hit ImportError, printing a traceback on every lc command. Import LogRecord from the stable opentelemetry._logs API module instead. - shutdown_otel() cleared logger/_PROVIDER immediately without draining the async telemetry queue first. CLI/MCP shutdown paths enqueue final events (session_end) via non-blocking emit_product() then call shutdown_otel() in a finally block, racing the background worker thread. When the worker saw logger=None it re-ran init_otel() -> Resource.create()'s internal ThreadPoolExecutor right as the interpreter was exiting, raising 'cannot schedule new futures after interpreter shutdown'. shutdown_otel() now flushes the queue before tearing down state.
Only symlink ~/.local/bin/lemoncrow -> $LEMONCROW_BIN_DIR/lemoncrow when the real binary is present, matching the existing lc/lcd guard right below it. Closes the residual gap in lemoncrow-lab#41 after ff482f730 already fixed the silent bundle.sh failure (curl|bash leaving a broken symlink).
…lobs Unbounded greedy classes before required literals (email @, JWT dots, PEM key END, </think>) caused catastrophic backtracking on large base64/PDF tool-output blobs, hanging lc import 9-12+ min and holding a write lock on lemoncrow_history.db. Fixes lemoncrow-lab#38
…oncrow-lab#37) - _claude_additional_dirs read only the legacy top-level additionalDirectories key; Claude Code writes it nested under permissions.additionalDirectories, so real settings.json files were silently ignored, blocking edits e.g. under ~/.claude/plans/. - read both keys, both home and workspace settings.json. - document LEMONCROW_ADDITIONAL_DIRS in settings_registry.py so it shows up in `lc settings show` instead of only the error string. - clarify in the error message and docs that the settings.json route live-reloads (mtime-checked) while the env var route needs a restart (inherent to subprocess env vars).
…row-lab#41) - bundle.sh: a wheel-less distribution no longer passes just because some\n foreign lemoncrow is on PATH; only a LemonCrow-owned binary counts. - install.sh: assert the binary exists after bundle.sh (child->parent bin-dir\n handshake), propagate tar's real status out of the process substitution,\n base the success banner on the installed binary rather than PATH, and\n repair a dangling ~/.local/bin link instead of skipping it. - install.sh: stop redirecting stderr to /dev/null for the whole run, and\n keep the checksum grep from aborting under pipefail -- both hid every\n error message the installer tried to print. - lib/common.sh: find/grep exiting non-zero killed 'source lib/common.sh'\n under set -euo pipefail.
…row-lab#40) - root-logger logging.exception in the telemetry package fell through to\n logging.lastResort -> stderr, so any failure reprinted the lemoncrow-lab#40 tracebacks;\n route them to a named debug logger behind a NullHandler. - add an SDK compat shim for opentelemetry-sdk 1.30-1.36, where Logger.emit\n does not convert API log records and export silently dies in the batch\n thread (letta pins sdk==1.30, so the floor cannot be raised). - fence lazy re-init after shutdown_otel so a late emit cannot resurrect an\n un-shutdown provider; explicit re-init still re-arms.
…ry (lemoncrow-lab#37) - retrieval.additional_dirs shared LEMONCROW_ADDITIONAL_DIRS with the edit\n gate, so a search/indexing knob silently granted write access; give it its\n own env var and add a registry test for env_var uniqueness. - read settings.local.json (home and workspace) alongside settings.json. - reject relative entries (previously resolved against the server CWD) and\n refuse blanket grants of / or the whole home dir; warn once per file on\n malformed settings instead of swallowing them. - document the routes in docs/, including which live-reload.
…xes (lemoncrow-lab#38) - bound the unbounded lazy-DOTALL patterns still on the import path (<task>/<prompt>/<user_query>, agent-settings scrape) and add closing-literal guards so a missing close tag skips the scan entirely. - cap oversized records in the SQLite-backed importers (opencode, lemoncode, cursor) by eliding the huge payload rather than dropping the tail, so step-finish token accounting survives; artifact sha256/byte counts now describe the true original bytes and truncation is marked. - truncation never emits a partial record.
Workflow fan-out writes agents under subagents/workflows/<id>/agent-*.jsonl; only the direct subagents/*.jsonl level was priced. Journals stay excluded.
Trigger-shaped so they fire on a condition, not on virtue: a bound added for speed changed what matched (lemoncrow-lab#38 redaction leak), and a rewrite kept happy-path parity while diverging elsewhere. Plus: subagent green is not verification.
Data-flow review of a design before it is built: derivability, wiring, contracts, citations, lifecycle closure, failure paths. Optional public skill (--include-skills), so it ships only where asked for.
…SONL - bounding <think>/PEM spans to a fixed window (lemoncrow-lab#38 follow-up) traded a hang for a leak: a >16KB key body or >64KB reasoning block stopped matching and was written verbatim. Replaced the span regexes with a two-pointer scan over opener/closer positions: linear, and with no size ceiling. - redact_jsonl(): redacting serialized JSONL as raw text corrupts it -- a match can straddle a JSON escape, and the credential rule masks to end-of-line, i.e. the rest of the record. Decode first, redact string values. - escape U+2028/U+2029/VT/FF/NEL when serializing records: legal inside a JSON string, but splitlines() treats them as breaks and halves the record.
- opencode/codex/claude/_common redacted serialized JSONL as raw text, so stored artifacts had unparseable records that every reader silently skipped. Measured on a real 12GB opencode.db: 335/690 session artifacts affected, 1661 corrupt lines; after this change, 0 of 169966. - cursor built its payload in memory and only had the whole-session cap, so an oversized assistant record took every later record -- and its token accounting -- with it. Now uses the same per-record eliding as opencode via serialize_capped_events().
The banner interpolated `lemoncrow --version` with `|| echo ''`, so a binary that was -x but died on every run (wrong interpreter, broken venv, missing dep) still printed "ready!" and the installer exited 0 -- the GH lemoncrow-lab#41 end state with a working-looking symlink. The version is now captured once, its failure aborts the install with the real error, and the banner reuses it.
…crow-lab#43) lc import held one transaction for the whole run, so a concurrent import died on the 30s busy_timeout. Four things were needed, each measured: - scope batch_mode() to one host, and run the read-only reconstruction audit outside it (via the new read_scope(), which pins a connection without holding a transaction -- dropping the shared connection cost ~2x). - BEGIN IMMEDIATE instead of the deferred BEGIN: a deferred transaction takes a read snapshot and fails the write upgrade with SQLITE_BUSY *immediately*, since busy_timeout does not apply to snapshot conflicts. - checkpoint the batch every second, and stay out of the lock briefly after each commit: re-acquiring straight away hands the lock back to the same process (a waiter still failed after 41s with 1s chunks). - raise busy_timeout to 120s, since one huge record can hold the lock ~20s. Verified on a real 12GB opencode.db: a cursor import (289 sessions) now runs to completion alongside a 690-session opencode import, both exit 0.
redact_jsonl ran all ~11 patterns per line, turning a 170k-line session into 1.5M regex calls: import went 1m53s -> 4m0s. Profiled, then fixed in three steps -- skip lines with no literal anchor, skip the same way per JSON value, and stop redacting the whole serialized line before redacting its values (pure duplicate work). str.translate for line-separator escaping is now guarded by a C-level search; it alone was 7.6s of a 21.5s profile. Import is back to 2m28s, with a reference-equivalence test pinning the fast path to the always-parse implementation for every pattern in the module.
…ling 720f2464e put persona sources at 4451 tokens, over the 4400 ceiling test_personas_stay_telegraphic enforces. Same three contracts, fewer words; regenerated host files via make sync-agent-context. 4398/4400.
Released 2026-09-01: same $10/$50 input/output and cache-write rates as Fable 5, but cache reads drop to a quarter of base input ($0.25 vs $1.00). Also maps the "fable 5.1"/"mythos 5.1" display names to the new model ids for cost tracking.
Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
Telemetry conflicts resolved in favor of upstream: their GH lemoncrow-lab#40 fix supersedes our otel/emit patch, and their _sdk_log_record_cls covers the sdk >=1.43 LogRecord rename our workaround handled. Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges upstream
v0.6.9,v0.6.10, andv0.7.0(26 commits) into the fork.Conflicts: both telemetry, both resolved in upstream's favor
Upstream independently fixed the OTel bug our fork patched in #6/#9 (their GH lemoncrow-lab#40, commits
3ff89e7a+1bae9588). Their fix is a strict superset of ours, so our patch is dropped rather than merged:ImportErrorfallback for the sdk >=1.43LogRecordrename_sdk_log_record_cls()feature-detects the whole 1.30 / 1.37 / 1.43 matrixsys.is_finalizing()guards_SHUTDOWNfence + aflush_product_telemetry()drain insideshutdown_otellogging.exception->_logger.debugNullHandleron the telemetry root so records never reachlogging.lastResort_INIT_ARGSreplay so a rebuilt exporter keeps its endpoint and auth headersemit.pyandexporters/otel.pyare now byte-identical to upstream, so this conflict will not recur on the next sync.Verified, not assumed. Re-ran the A/B probe that validated our original patch, against the merged tree on opentelemetry-sdk 1.43.0:
Matches what our patch produced (
True / 1 / 0), against a pre-fix baseline ofFalse / 0 / 6.Fork deltas preserved
The
portableextra,.gitignore,code_intel/**, and the mcpbash.py/session_state.py/smart_state.py/mcp_server.pywork all merged cleanly — no conflicts outside telemetry.Checked while resolving: upstream still carries the unbounded
fcntl.flock(fd, LOCK_EX)insmart_state.pyand still returns a bareunknown shell session: {id}, so the fixes in #7 and #10 remain novel and are still worth submitting upstream. Only the telemetry fix drops off that list.Tests: no regressions
Full scoped suite, same ignore set, run on both branches:
main(pre-merge)The merge fixes 13 failures (the
test_bootstrap_optional_extras.pyblock andtest_installer_records_version_before_replacing_cli) and adds 118 passing tests.The high absolute count on both sides is the known compiled-engine ceiling on a public-mirror checkout (no
src/lemoncrow/prosource; mypyc runtime type strictness rejectsMagicMock), not defects in this diff.Three tests fail only on this branch. All three were run down, and none is a regression:
test_native_search.py::...catastrophic_single_linetest_agent_cli_install_artifacts.py::test_pi_is_default_selected...scripts/install_pi.shexists; that file is absent fromupstream/mainitself (mirror strips it)test_telegraphic_budget.py::test_personas_stay_telegraphicThe last two were confirmed by checking out
upstream/maindetached, with no fork code present, where both fail identically. They are upstream defects in the v0.7.0 mirror that we inherit, not something this merge introduced.Also excluded from both runs (they segfault the pytest process inside compiled
pro.capabilities.tool_supervision.symbol_edit, identically onmain, so they are engine artifacts):test_mcp_jsonrpc_e2e.pyandtest_rich_edit_symbol.py, alongside the previously-knowngit_history/,cross_lang/, andtest_ann_symbol_index.py.Note for local checkouts
v0.7.0 needs two top-level mypyc group modules vendored, not one — and it reuses the v0.6.8 hash name
ab9f1a693248adfed5e2for a different 96MB binary, so compare sizes rather than names. Upstream also moved_redact_json_valuesintocore/foundation/redaction.py, which a stale group module shadows with no per-module.soon disk; test for shadowing by importing and reading__file__, not by looking for a file.