Repository navigation
Living Weights: a fact that only answers an earlier fact's question no longer replaces it; a told fact's answer is never its subject - #910
Conversation
…a chat fact replace it A chat fact replaced an earlier one when it answered any of the earlier fact's questions, before the model was asked whether it changes anything (ashhart#678). Two different facts about the same kind of thing answer each other's questions: "My sister's GitHub name is ana77." answers "What is my sister's name?", so it was taken for a correction of "My sister is called Ana.", whose neurons were retired before its lesson. The same held for a name and a GitHub or forum name, and for "I work in Bath." after "I live in Bristol.". That check is gone. An earlier fact now gives way when the new one is not the same thing said again and the model says it changes or takes back what the earlier one says, as before, or says the earlier one is no longer true (the new still_prompt). The second question is needed: the model says "My favourite colour is red." changes nothing in "I like blue.", yet that "I like blue." no longer holds, and ashhart#678's corrections came through the removed check. The verdict from the replies is a function of its own, with a test.
…ect keys on the reply
For "I contribute to TensorFold." the subject phrase came back as "TensorFold", the fact's answer. Only the yes-or-no
questions say it ("Do I contribute to TensorFold?"), and those are dropped as asking about the fact itself, so every
open question ("What do I contribute to?", "Which project do I contribute to?") was dropped as not naming the subject
and the lesson ended with too few clean answers. A told fact's subject that fewer than half of its questions name,
not counting the yes-or-no ones, is now no subject: the fact keys on the reply, as a fact about the user does. A
document's facts are unchanged. The test fails before this change (the subject TensorFold is kept) and passes after;
a subject most open questions name ("My dog is called Ginger.") stays.
|
Thank you, @ZackSample, for separating a genuine correction from a new fact that happens to answer an earlier question. We checked both commits through 7f903c6. The narrower replacement rule and the answer-key fallback make sense. One string check needs tightening: Please require a complete yes/no answer and keep an uncertain or empty answer conservative. We can fold that parser change in at landing if you prefer. We'd add those strings to the decision tests, then run independent name/GitHub facts, an actual city correction, repeated facts and overlapping questions, checking retained answers, save/reload and deletion after each lesson. Posted by Ash's AI Assistant Synthena |
|
Thanks, @ashhart. Agreed: a complete yes/no answer only, and an uncertain or empty reply keeps the earlier fact. Please fold the parser change and the extra decision-test strings in at landing. |
|
@ashhart @ZackSample, a single-file patch with CPU regressions is ready to fold into the agreed parser change. The three behavioral counterexamples fail on the original decision logic and pass with the patch: uncertain no-prefix replies, yes-prefix replies containing more text, and empty or uncertain repeat judgments. Whole Yes/No answers keep the existing formatting support; an unknown repeat judgment preserves the earlier fact. The patch also checks I replayed both of Zack's commits unchanged on current main Zig 0.17.0: all six decision/response tests plus their wrapper pass in Debug and Fast. The full server suite, all No-GPU workflow commands, whole-tree formatting and lean (zero new findings) pass. Astra reviewed the patch without a remaining material source blocker. The patch has local validation only. This supplies parser and completion evidence. Learned-answer retention, retirement, save/reload and deletion still need their model gates; the reported recall regression has not been retested. |
…ollow-ups named as ours After review of the fourteen lines that overstated: #710's fresh-quantization gain is one measured case; #902's stop and group cleanup hold where the watch lines are open and through the shared helper; #887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; #899 records the ring start and the wider refusal; #802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; #836's equalities are the GB10's and its Windows build the author's; #888 says a failed restore's pinned memory is kept, never freed; #734's tree rounds are checked against the host's, not float64; the Living Weights lines (#876, #877, #878, #891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; #946 and #910 read plainer.
|
Thank you, @ZackSample, for distinguishing a changed fact from one that merely answers an earlier question. We merged your fix for 1.1.0 in 8ca156f. Our follow-up accepts only whole Yes/No judgments from replies that ended normally. An uncertain or truncated repeat answer keeps the earlier fact; definite answers are required before replacement. The CPU controls cover prefixes, empty answers and token-limited replies. This existing-learner correction is separate from the larger Living Weights series moving to 1.1.1. Posted by Ash's AI Assistant Synthena |
…one changes it or it no longer holds, not for answering the same kind of question
…ollow-ups named as ours After review of the fourteen lines that overstated: ashhart#710's fresh-quantization gain is one measured case; ashhart#902's stop and group cleanup hold where the watch lines are open and through the shared helper; ashhart#887 says its chunks change the attention bits below 8,192, on M5 too, with no larger error against float64; ashhart#899 records the ring start and the wider refusal; ashhart#802 keeps the author's B70 receipts apart from our host-checked lifetime fixes; ashhart#836's equalities are the GB10's and its Windows build the author's; ashhart#888 says a failed restore's pinned memory is kept, never freed; ashhart#734's tree rounds are checked against the host's, not float64; the Living Weights lines (ashhart#876, ashhart#877, ashhart#878, ashhart#891) say which runs their numbers come from, the ten-fact costs, and the M3 Ultra's four-fact result; ashhart#946 and ashhart#910 read plainer.
3 participants
Two lesson-writer bugs from #896's lesson run on Flash Next. Both are in the shared server code (
server/slide_*.zig)and show on Nemotron too, so this PR fixes them on
mainwith Nemotron receipts. It touches no model code.The problems
Zack." (
... replaces "My name is Zack." (fact 1), whose neurons are taken out first). The name fact's neurons wereretired and the graph marked it
replaced by fact 3. The corrections rule from Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678 (slide_replace.zig:37-40)counts a new fact that answers any of an earlier fact's questions as a replacement, before it asks the model whether
anything changed. Two different facts about the same kind of thing answer each other's questions: a GitHub name
answers "What is my name?".
Only the yes-or-no questions say it ("Do I contribute to TensorFold?"), and those are dropped as asking about the
fact itself. Every open question ("What do I contribute to?", "Which project do I contribute to?") was dropped as
not naming the subject. On Flash Next one question was left and the lesson ended
too few clean answers to learn fromin 14 s. On Nemotron, 5 questions were left, all of them naming TensorFold (receipt below).What this changes
A replacement needs the earlier fact to stop holding (
e84e35a4). The answers-a-question check is gone. Anearlier fact now gives way when the new one is not the same thing said again (
again_prompt, as before), and themodel says it changes or takes back the earlier one (
replace_prompt, as before) or that the earlier one is nolonger true (new
still_prompt: "Is what they told you earlier still true?"). The verdict from the three replies isgivesWay, with a test.The second question is needed. Without it, Nemotron says "My favourite colour is red." changes nothing in "I like
blue.", and Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678's own sequence stops working: a first try of this PR that dropped the check and kept only
replace_promptleft red unlearned and blue in place (receipt below). Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678's corrections came through theanswers-a-question check. On the same pair, the model says "I like blue." no longer holds.
A replacement check now asks two or three short prompts per earlier fact, against two plus one per stored question
before.
A told fact's subject that its open questions do not name is no subject (
7f903c6c). InfactSubject, a toldfact's subject words that fewer than half of its questions name, not counting the yes-or-no ones, are dropped. The
fact then keys on the reply, as a fact about the user does. A document's facts are unchanged. The test fails on
main(expected 0, found 1: the subject TensorFold is kept) and passes after. A subject most open questions name("My dog is called Ginger.") stays.
Receipt
Environment:
lw-e-writer7f903c6 onmainc7137bb (v1.0.11 + 19), Zig 0.17.0; NVIDIA GB10 (DGX Spark,sm_121), driver 580.178.04, CUDA 13.0.88, Ubuntu 24.04.5; checkpoint
TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bitrevision d9d758fb, served from a freshcp -RLcopy foreach run:
tensorfold-native serve <copy> --no-thinking --temperature 0 --slide --slide-graph <graph.json>. Eachfact is one learn request (
"source": "chat"). After each lesson, every fact so far is asked two fixed wordings andits stored held-out questions, greedy, 60 tokens. A reply counts when it names the fact's value.
The false correction,
mainand this branch (the same four facts in turn):mainit kept answering questions about something else, last "What is my brother's name on GitHub?"); Ana put backOn
mainthe ana77 lesson ran with Ana's 10 neurons retired. It failed here, so they were put back and Ana stillanswers. Had it been kept, Ana would have stayed retired, as the name fact did in Living Weights on Qwen3.8 Flash Next (CUDA, GB10): the pocket in every pass, the trainer, and --slide #896's run (0/3 after the lesson).
A second
mainrun gave the same replacement and the same take-out. Lessons 3 and 4 differ only in what thepocket held before them; I do not claim the rule caused
main's misses there.A real correction still replaces. "I like blue." then "I don't like blue anymore, now I like red." On both builds:
"I don't like blue anymore, now I like red." replaces "I like blue." (fact 1), whose neurons are taken out first.The red lesson was not recalled after 5 steps on either build, and blue was put back. The shards after each lesson
are identical between the builds (sha256 of every file).
On
mainwith Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678's three messages ("I like blue.", "My favourite colour is red.", "I do not like blue anymore. NowI like red."), the red fact replaced blue, and "Now I like red." then also replaced "My favourite colour is red.",
which says the same thing (
again_promptsaid no, the answers-a-question check said yes). The first try of this PR,which kept only
replace_prompt, ran the same sequence: nothing replaced blue at lesson 4, the red fact was notrecalled after 5 steps with blue's fact still in place (its neurons refitted to stay quiet on red's questions), and
"What colour do I like?" still said blue at the end. That run is why the
still_promptis there.The three prompts, asked of the untaught model (greedy, 4 tokens) over 9 corrections and 12 pairs that are not:
main: not said again, and answers an old question orreplace_promptyesreplace_promptyes onlyreplace_promptyes orstill_promptnoThe non-corrections
mainreplaces: a name and a GitHub name, a name and a forum username, a sister's name and herGitHub name, a dog's name and its Instagram name, "I work in Bath." after "I live in Bristol.", and "Now I like red."
after "My favourite colour is red.". The one correction this PR misses is "Now I like red." after "I like blue.",
where the model says blue still holds.
still_promptsays yes on all 12 non-corrections.The subject fix. On
main, the lesson for "I contribute to TensorFold." (subject "TensorFold") dropped 23questions as
does not name the fact's subjectand kept 5, all of which name TensorFold ("What am I doing forTensorFold?"). It was not recalled after 5 steps. With the fix (the first try's build, whose subject commit is this
one), the subject is dropped, 18 questions are kept, among them "What do I contribute to?" and "Which project do I
contribute to?", and it keys on the reply. On Nemotron it was still not recalled after 5 steps: its held-out pair came
out vague ("What do I do?", "Do you know what I do?"). On Flash Next, where the writer refused it, it is learned with
the fix (next item). Across 19 told facts, the subject prompt named the answer for 6 ("TensorFold", "Siemens",
"Hollow Pines", "Leeds University", "Riverside Choir", "The Rustls library"), so the case is not rare.
Flash Next (Living Weights on Qwen3.8 Flash Next (CUDA, GB10): the pocket in every pass, the trainer, and --slide #896's sequence, on the comparison build with Living Weights: a quiet refit holds an earlier fact's neuron on its own questions as read now #876, Living Weights: every earlier fact asked again after a lesson, and a take-out that puts the pocket back bit for bit #877, Living Weights: facts learned and checked in conversation, not only asked cold #891 and these two commits; not part of
this PR's code path on
main, quoted to show the fix on the model it was found on): "My GitHub name is ZackSample."no longer replaces "My name is Zack.", and "I contribute to TensorFold." keeps 20 questions instead of 1 and is
learned in 3 steps. Held-out probes after the five lessons: 9 of 12, against 4 of 12 without these two commits.
Upstream's ten facts (1.0.8's receipt, "I like blue." to "I am allergic to peanuts."; 20 fixed wordings):
mainit made up something about the user it was never told)it kept answering questions about something else, last "What's my home address?")This is a regression on this sequence and I do not want to hide it. The two builds are byte-identical until lesson 8.
There the subject fix turns Volvo's lesson from 3 yes-or-no questions into 21 open ones. Both builds miss it, but the
branch's lesson refitted an earlier fact's neuron (layer 27, neuron 2) and a missed lesson on
maindoes not put sucha refit back (Living Weights: every earlier fact asked again after a lesson, and a take-out that puts the pocket back bit for bit #877 makes the take-out exact). Every earlier fact still answered after lesson 8 on both builds. The
whole difference is lesson 10, whose lesson on the branch started from that pocket. I have not run this sequence with
Living Weights: every earlier fact asked again after a lesson, and a take-out that puts the pocket back bit for bit #877, so I cannot say yet whether the exact take-out removes the difference.
main's Volvo "held-out 2/2" in theprobes is its yes-or-no questions, which say Volvo.
Tests:
zig build teston Linux: 745/750 pass, 5 skipped (main: 743/748, 5 skipped; the 2 more are the two newtests).
zig build test-server-cpu: 209/209 (main: 207/207). Both commits build and pass.zig fmt --check build.zig zigis clean, andpython3 tools/zig/lean_check.pygives 527 problems against 527 onmain(0 new).Exactness, decode and prompt speed: n/a. Nothing in the forward or in serving changes. Where both builds decide the
same way, a lesson writes the same shards as
main, byte for byte (receipts above).Not run: Metal (no Mac here). The change is server-side lesson writing, shared by both backends.
Not in this PR, said plainly
whole fact never counts, and yes-or-no questions about the model's own facts) found no case on Nemotron's user facts,
so they are left out.
still_promptmakes two prompts agree where one did not, but amodel that calls a non-correction "no longer true" would still replace.
builds) are not touched.
Checklist
change.)
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Nottouched.)
change.)
fails on
main. The replacement verdict is a new function: its test pins the verdict, and the before and after isthe Nemotron run above.)
Questions
still_promptintoreplace_prompt'swording? I kept Living Weights: conflicting facts, updates and forgetting (measurements + proposal) #678's prompt as it is, so every replacement it already caught still happens.