Skip to content

platform-crossgram: replace unstorable text in the named fields at the entry point - #14

Merged
std-microblock merged 1 commit into
std-microblock:mainfrom
std-external:fix/qqnt-unstorable-text
Oct 18, 2026
Merged

std-microblock merged 1 commit into
std-microblock:mainfrom
std-external:fix/qqnt-unstorable-text

Conversation

@std-external

Copy link
Copy Markdown
Contributor

platform-crossgram: 在入口点名字段替换 PostgreSQL 存不下的字符

重做 #5(原作者 ZianTT,已关闭)。#5 的方向对(PostgreSQL 存不了 U+0000 和孤立代理码元,只能在写库前清),但按 #5 的讨论 意见改成:只处理点名的字段、在收到数据之后所有步骤之前处理、替换成 . 且长度不变。

问题

QQNT 把 QQ 里人写的东西原样递过来:群名/备注、昵称、群名片、正文、转发标题、申请附言。PostgreSQL 的 text / json / jsonb 列都存不下 NUL 和孤立 UTF-16 代理码元(本机 PG 17.11 实测:text 是 22021/08P01,jsonb 是 22P05/22P02)。事件检查点只在 handler 提交之后才推进,所以一个字段存不下 → 整条事件失败 → 重放同一条事件 → 后面所有消息按序卡住。

#5 的实测缺口:request 事件(好友申请/入群申请)的 requester.name、group.name、附言没有经过清洗,合并后照样能卡死检查点。

改法

只点名字段,不递归

新增 packages/platform-crossgram/src/sanitize.ts,按 wire 对象点名字段:

对象 字段 落库位置
WireMessage.sender name(昵称)、alias(备注/群名片) mtproto_im_user.firstName、senderTitle、content
WireMessage.parts text text、markdown content、multi-forward title/preview、card title/description、sticker title、media name/transcript mtproto_im_message.text、content
WireMessage.serviceAction text、members[].name、actor.name content
WireConversation title(群名/备注) mtproto_im_conversation.title
WireRequest requester.name、group.name、message(附言) mtproto_im_request.request
用户资料 name、alias、signature mtproto_im_user.firstName、about

不动的:id、peerUid/peerUin、URL、MIME 类型、locator、枚举、实体 offset/length——这些是协议值,不是人写的文本;内联键盘是机器人下发的协议载荷,同理由。

递归遍历整个对象的问题:适配器被迫知道「PostgreSQL 能存什么」,谁也说不清哪层还有漏网字段(#5 就漏了 request 路径),而且会顺手改写协议值。

在入口处一次做完

  • 事件流:subscribeLoop 的 client.subscribe 回调第一行就清洗整个事件,过滤、映射、投递、写库、检查点都在它之后,且覆盖 request 事件(platform-crossgram: sanitize invalid QQ message strings #5 的缺口)。
  • 读请求:mapConversation / mapMessage / mapRequest 以及用户资料映射处再清一次(幂等),覆盖 dialog/history/search/request 这些不走事件流的入口。

用 .,且长度不变

一个不可存的 code unit 换成一个 .,字符串长度完全不变。text part 的 entities 的 offset/length 是以该 text 的 code unit 计的,等长替换后 QQ 发来的每个 @ 范围仍然选中同一段文字,不需要重算 offset,也不会出现「实体边界正好落在代理对中间而漂一位」的问题。删掉字符则要让后面所有 offset 前移,这正是 #5 里 stripNulTextPart 那段反推要解决的问题。合法的代理对(emoji)是一个 code point,原样保留。

验证

yarn vitest run packages/platform-crossgram/src/sanitize.test.ts packages/platform-crossgram/src/platform.test.ts --config vitest.config.mts
yarn vitest run packages/platform-crossgram/src/message-order.e2e.test.ts --config vitest.mtproto-e2e.config.mts --maxWorkers=1
yarn typecheck --pretty false
  • sanitize.test.ts(11 项):每个点名字段、长度不变、合法代理对保留、NUL 之后 @alice 范围仍选中 @alice、干净事件原样返回。
  • platform.test.ts(143 项):带 NUL 的昵称/备注/正文映射后 firstName/senderTitle/text 是 .,mention 实体切片仍是 mention:@alice;群标题、申请、用户资料同样覆盖。
  • message-order.e2e.test.ts(21 项):走真实 WebSocket + SQLite + MessageStore,重放一条带 NUL 的消息事件和一条带 NUL 的入群申请事件,断言落库的群标题 Gr.oup、昵称 Bad.Name、正文 @alice. hello、mention offset 0 / length 6、备注 re.mark、申请附言 let me in.,以及检查点推进到第二条事件(platform-crossgram: sanitize invalid QQ message strings #5 的现象)。
  • 反向验证:把清洗改成 no-op 后,该 e2e 失败。

QQNT hands the relay text QQ copied from a person: a group title that is a
contact's remark, a sender's nickname and remark, the message body, a forward
title, a request note. PostgreSQL stores neither a NUL (`U+0000`) nor a UTF-16
surrogate without its partner in a `text`, `json` or `jsonb` column, and the
event checkpoint advances only after the handler commits, so a single such
unit failed the whole event and every later event replayed behind it forever.

Clean exactly the fields that carry that text, once the wire data has been
received and before any other step reads it:

- `sanitize.ts` names the fields per wire object (message sender name/alias,
  text and markdown bodies, forward title/preview, card title/description,
  sticker title, media name/transcript, service notice wording and the names
  it carries, conversation title, request requester/group name and note,
  profile name/alias/signature). Ids, peer ids, URLs, MIME types, locators,
  enums and entity offsets are protocol values and stay untouched.
- The event stream cleans each event at reception, before its filters, mapping
  and dispatch; the conversation/message/request mappers clean whatever a
  history, dialog, search or request read returns, and the profile reads clean
  a user's name and signature before they are persisted.

Each unstorable code unit becomes `.` rather than disappearing, which keeps
the replaced string exactly as long as it was: a text part's entities measure
`offset`/`length` in code units of that text, so every mention range QQ sent
still selects the same text and nothing has to be re-measured. A valid
surrogate pair (an emoji) is one code point and is left alone.

Tests:

- `sanitize.test.ts` covers each named field, the length guarantee, a valid
  pair surviving, and a mention range still selecting `@alice` after a NUL.
- `platform.test.ts` maps a message, a group title, a request and a profile
  with unstorable units and checks the mention entity slices.
- `message-order.e2e.test.ts` replays a NUL message and a NUL group-join
  request over the WebSocket and asserts the persisted conversation title,
  user name, message text, mention range, sender remark and request note, plus
  the checkpoint reaching the second event.
@std-microblock
std-microblock merged commit 7850203 into std-microblock:main Oct 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants