Repository navigation
platform-crossgram: replace unstorable text in the named fields at the entry point - #14
Merged
std-microblock merged 1 commit intoOct 18, 2026
Conversation
QQNT hands the relay text QQ copied from a person: a group title that is a contact's remark, a sender's nickname and remark, the message body, a forward title, a request note. PostgreSQL stores neither a NUL (`U+0000`) nor a UTF-16 surrogate without its partner in a `text`, `json` or `jsonb` column, and the event checkpoint advances only after the handler commits, so a single such unit failed the whole event and every later event replayed behind it forever. Clean exactly the fields that carry that text, once the wire data has been received and before any other step reads it: - `sanitize.ts` names the fields per wire object (message sender name/alias, text and markdown bodies, forward title/preview, card title/description, sticker title, media name/transcript, service notice wording and the names it carries, conversation title, request requester/group name and note, profile name/alias/signature). Ids, peer ids, URLs, MIME types, locators, enums and entity offsets are protocol values and stay untouched. - The event stream cleans each event at reception, before its filters, mapping and dispatch; the conversation/message/request mappers clean whatever a history, dialog, search or request read returns, and the profile reads clean a user's name and signature before they are persisted. Each unstorable code unit becomes `.` rather than disappearing, which keeps the replaced string exactly as long as it was: a text part's entities measure `offset`/`length` in code units of that text, so every mention range QQ sent still selects the same text and nothing has to be re-measured. A valid surrogate pair (an emoji) is one code point and is left alone. Tests: - `sanitize.test.ts` covers each named field, the length guarantee, a valid pair surviving, and a mention range still selecting `@alice` after a NUL. - `platform.test.ts` maps a message, a group title, a request and a profile with unstorable units and checks the mention entity slices. - `message-order.e2e.test.ts` replays a NUL message and a NUL group-join request over the WebSocket and asserts the persisted conversation title, user name, message text, mention range, sender remark and request note, plus the checkpoint reaching the second event.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
platform-crossgram: 在入口点名字段替换 PostgreSQL 存不下的字符
重做 #5(原作者 ZianTT,已关闭)。#5 的方向对(PostgreSQL 存不了
U+0000和孤立代理码元,只能在写库前清),但按 #5 的讨论 意见改成:只处理点名的字段、在收到数据之后所有步骤之前处理、替换成.且长度不变。问题
QQNT 把 QQ 里人写的东西原样递过来:群名/备注、昵称、群名片、正文、转发标题、申请附言。PostgreSQL 的
text/json/jsonb列都存不下 NUL 和孤立 UTF-16 代理码元(本机 PG 17.11 实测:text是 22021/08P01,jsonb是 22P05/22P02)。事件检查点只在 handler 提交之后才推进,所以一个字段存不下 → 整条事件失败 → 重放同一条事件 → 后面所有消息按序卡住。#5 的实测缺口:
request事件(好友申请/入群申请)的requester.name、group.name、附言没有经过清洗,合并后照样能卡死检查点。改法
只点名字段,不递归
新增
packages/platform-crossgram/src/sanitize.ts,按 wire 对象点名字段:WireMessage.sendername(昵称)、alias(备注/群名片)mtproto_im_user.firstName、senderTitle、contentWireMessage.partstext、markdowncontent、multi-forwardtitle/preview、cardtitle/description、stickertitle、medianame/transcriptmtproto_im_message.text、contentWireMessage.serviceActiontext、members[].name、actor.namecontentWireConversationtitle(群名/备注)mtproto_im_conversation.titleWireRequestrequester.name、group.name、message(附言)mtproto_im_request.requestname、alias、signaturemtproto_im_user.firstName、about不动的:id、
peerUid/peerUin、URL、MIME 类型、locator、枚举、实体offset/length——这些是协议值,不是人写的文本;内联键盘是机器人下发的协议载荷,同理由。递归遍历整个对象的问题:适配器被迫知道「PostgreSQL 能存什么」,谁也说不清哪层还有漏网字段(#5 就漏了 request 路径),而且会顺手改写协议值。
在入口处一次做完
subscribeLoop的client.subscribe回调第一行就清洗整个事件,过滤、映射、投递、写库、检查点都在它之后,且覆盖request事件(platform-crossgram: sanitize invalid QQ message strings #5 的缺口)。mapConversation/mapMessage/mapRequest以及用户资料映射处再清一次(幂等),覆盖 dialog/history/search/request 这些不走事件流的入口。用
.,且长度不变一个不可存的 code unit 换成一个
.,字符串长度完全不变。text part 的entities的offset/length是以该 text 的 code unit 计的,等长替换后 QQ 发来的每个@范围仍然选中同一段文字,不需要重算 offset,也不会出现「实体边界正好落在代理对中间而漂一位」的问题。删掉字符则要让后面所有 offset 前移,这正是 #5 里stripNulTextPart那段反推要解决的问题。合法的代理对(emoji)是一个 code point,原样保留。验证
yarn vitest run packages/platform-crossgram/src/sanitize.test.ts packages/platform-crossgram/src/platform.test.ts --config vitest.config.mts yarn vitest run packages/platform-crossgram/src/message-order.e2e.test.ts --config vitest.mtproto-e2e.config.mts --maxWorkers=1 yarn typecheck --pretty falsesanitize.test.ts(11 项):每个点名字段、长度不变、合法代理对保留、NUL 之后@alice范围仍选中@alice、干净事件原样返回。platform.test.ts(143 项):带 NUL 的昵称/备注/正文映射后firstName/senderTitle/text是.,mention 实体切片仍是mention:@alice;群标题、申请、用户资料同样覆盖。message-order.e2e.test.ts(21 项):走真实 WebSocket + SQLite +MessageStore,重放一条带 NUL 的消息事件和一条带 NUL 的入群申请事件,断言落库的群标题Gr.oup、昵称Bad.Name、正文@alice. hello、mentionoffset 0 / length 6、备注re.mark、申请附言let me in.,以及检查点推进到第二条事件(platform-crossgram: sanitize invalid QQ message strings #5 的现象)。