Retrieval Test return the "null"

[Jason-TOP]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.0

Cloud or Self Hosted

Self Hosted (Source), Self Hosted (Docker)

Steps to reproduce

My knowledge base contains a chunk/section with the phrase '修改教材', yet the recall test consistently yields no results. I have experimented with adjusting every available retrieval strategy, but the segment still cannot be retrieved

✔️ Expected Behavior

The segment containing "修改教材" should be retrieved and displayed.

❌ Actual Behavior

The recall test returns an empty list. The segment containing "修改教材" is never retrieved, regardless of adjustments to retrieval modes, weighting, or TopK/Score threshold parameters.

[dosubot]

Hi @Jason-TOP! I'm Dosu and I’m helping the dify team.

This is a commonly reported issue with self-hosted Dify deployments. Here are the most likely causes and things to check [1]:

1. Document indexing status — Check if your documents are stuck at "queuing" or "indexing" rather than showing "completed." If indexing never finished, the chunks won't be searchable regardless of retrieval settings.

2. Missing priority_pipeline Celery worker — In self-hosted deployments, the Celery worker for the priority_pipeline queue may not be started by default. This causes indexing tasks to remain stuck. If you're using Docker, add a priority-worker service to your docker-compose.yaml:

   priority-worker:
     image: langgenius/dify-api:1.11.0
     restart: always
     environment:
       <<: *shared-api-worker-env
       MODE: worker
       CELERY_QUEUES: priority_pipeline
       CELERY_WORKER_AMOUNT: 2
       LOG_LEVEL: INFO
     depends_on:
       db:
         condition: service_healthy
       redis:
         condition: service_started
     volumes:
       - ./volumes/app/storage:/app/api/storage
     networks:
       - ssrf_proxy_network
       - default

   Then restart with docker compose down && docker compose up -d.

3. Retrieval method mismatch — If you indexed using the "Economical" method, you must use keyword search for retrieval. Using vector/semantic search against keyword-indexed documents will return empty results.

4. Multi-path retrieval without Rerank — If you're using multi-path retrieval mode, ensure a Rerank model is properly configured; otherwise results may come back empty.

Could you check whether your documents show a "completed" indexing status in the Knowledge Base, and confirm which indexing method (High Quality vs. Economical) you used?

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[Jason-TOP]

Hi @Jason-TOP! I'm Dosu and I’m helping the dify team.

This is a commonly reported issue with self-hosted Dify deployments. Here are the most likely causes and things to check [1]:

1. Document indexing status — Check if your documents are stuck at "queuing" or "indexing" rather than showing "completed." If indexing never finished, the chunks won't be searchable regardless of retrieval settings.

2. Missing priority_pipeline Celery worker — In self-hosted deployments, the Celery worker for the priority_pipeline queue may not be started by default. This causes indexing tasks to remain stuck. If you're using Docker, add a priority-worker service to your docker-compose.yaml:
   priority-worker:
   image: langgenius/dify-api:1.11.0
   restart: always
   environment:
   <<: *shared-api-worker-env
   MODE: worker
   CELERY_QUEUES: priority_pipeline
   CELERY_WORKER_AMOUNT: 2
   LOG_LEVEL: INFO
   depends_on:
   db:
   condition: service_healthy
   redis:
   condition: service_started
   volumes:
   - ./volumes/app/storage:/app/api/storage
   networks:
   - ssrf_proxy_network
   - default

   Then restart with docker compose down && docker compose up -d.

3. Retrieval method mismatch — If you indexed using the "Economical" method, you must use keyword search for retrieval. Using vector/semantic search against keyword-indexed documents will return empty results.

4. Multi-path retrieval without Rerank — If you're using multi-path retrieval mode, ensure a Rerank model is properly configured; otherwise results may come back empty.

Could you check whether your documents show a "completed" indexing status in the Knowledge Base, and confirm which indexing method (High Quality vs. Economical) you used?

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

1、My documents are stuck at "Can be used";

2、"The vast majority of other content types are retrieved without issue; only this specific segment yields no results. This suggests the problem is probably unrelated to the Celery Worker configuration." 3、The indexing method I'm using is High Quality. 4、A Rerank model has already been configured and enabled. Despite this, the recall test still returns no results for the segment containing the phrase "修改教材".

[dosubot]

Thanks for the additional details, @Jason-TOP. Since the document is showing as available, other segments retrieve fine, and you're using High Quality indexing with Rerank — this narrows things down significantly. Here are the most likely causes for your specific case:\n\n1. Chinese punctuation segmentation bug (affects Dify 1.10.0–1.13.2)\nThere's a bug in FixedRecursiveCharacterTextSplitter where Chinese punctuation marks (like ， and 。) used as segment delimiters get corrupted during processing. The splitter uses codecs.decode(fixed_separator, \"unicode_escape\") which internally encodes via latin-1 — this can't represent characters beyond U+00FF, breaking Chinese punctuation handling [1]. This could cause the segment containing "修改教材" to be incorrectly chunked and therefore fail to embed/retrieve properly. A fix exists in PR #33119 but hasn't been merged yet.\n\n2. Silent embedding failure (NaN vectors)\nDuring embedding generation, if a vector contains NaN values, it is silently skipped — no error is raised and the document still shows as "completed" [2]. This means specific segments can appear indexed but have no actual vector in the database.\n\n3. Incremental index staleness\nIf you've ever modified or deleted segments in this knowledge base, the vector index can end up with stale/orphaned entries, causing specific segments to return empty results [3].\n\nRecommended troubleshooting:\n\n1. Try a full re-import: Export your knowledge base, delete it entirely, then re-import from the export. This is the most reliable way to rule out index staleness and has resolved this pattern of issue for other users [3].\n\n2. Check your logs for warnings containing \"Normalized embedding is nan\" — this would confirm the segment's embedding was silently dropped [2].\n\n3. Check the original document around the "修改教材" phrase for Chinese punctuation that might be triggering the segmentation bug. If present, try replacing Chinese punctuation with ASCII equivalents before re-uploading.\n\n4. If you have direct access to your vector database, verify whether the specific segment actually has a stored vector — the Dify UI will show "Available" regardless of whether the vector was actually persisted.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.