Skip to content

fix(dfm-search): extend semantic filetype suffix coverage #354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
deepin-bot merged 1 commit into from
Jul 16, 2026
Merged

fix(dfm-search): extend semantic filetype suffix coverage #354

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 79 additions & 55 deletions autotests/dfm-search-tests/tst_chinese_nlp.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,16 @@
// SPDX-License-Identifier: GPL-3.0-or-later

#include <QCoreApplication>
#include <QDate>

Check warning on line 6 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QDate> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 6 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QDate> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QDateTime>

Check warning on line 7 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QDateTime> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 7 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QDateTime> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QDir>

Check warning on line 8 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QDir> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 8 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QDir> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QFile>

Check warning on line 9 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QFile> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 9 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QFile> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QHash>

Check warning on line 10 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QHash> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 10 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QHash> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QJsonArray>

Check warning on line 11 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QJsonArray> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 11 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QJsonArray> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QJsonDocument>

Check warning on line 12 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QJsonDocument> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 12 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QJsonDocument> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QJsonObject>

Check warning on line 13 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QJsonObject> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 13 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QJsonObject> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QSet>

Check warning on line 14 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QSet> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 14 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QSet> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QStandardPaths>

Check warning on line 15 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / cppcheck 

Include file: <QStandardPaths> not found. Please note: Cppcheck does not need standard library headers to get proper results.

Check warning on line 15 in autotests/dfm-search-tests/tst_chinese_nlp.cpp

View workflow job for this annotation

GitHub Actions / static-check / static-check 

Include file: <QStandardPaths> not found. Please note: Cppcheck does not need standard library headers to get proper results.
#include <QTest>

#include "semantic/intentparser.h"
Expand All @@ -28,38 +33,73 @@

namespace {

// Keep these lists in sync with filetype_rules.json extensions.
QHash<QString, QStringList> loadFiletypeExtensions()
{
const QString rulesPath = QDir(rulesDir()).filePath(QStringLiteral("filetype_rules.json"));
QFile file(rulesPath);
if (!file.open(QIODevice::ReadOnly)) {
qFatal("Failed to open filetype rules: %s", qPrintable(rulesPath));
}

QJsonParseError parseError;
const QJsonDocument doc = QJsonDocument::fromJson(file.readAll(), &parseError);
if (parseError.error != QJsonParseError::NoError || !doc.isObject()) {
qFatal("Failed to parse filetype rules %s: %s",
qPrintable(rulesPath), qPrintable(parseError.errorString()));
}

QHash<QString, QStringList> extensionsByRuleId;
const QJsonArray groups = doc.object().value(QStringLiteral("groups")).toArray();
for (const QJsonValue &groupValue : groups) {
const QJsonObject groupObj = groupValue.toObject();
if (groupObj.value(QStringLiteral("name")).toString() != QLatin1String("filetype")) {
continue;
}

const QJsonArray rules = groupObj.value(QStringLiteral("rules")).toArray();
for (const QJsonValue &ruleValue : rules) {
const QJsonObject ruleObj = ruleValue.toObject();
const QString ruleId = ruleObj.value(QStringLiteral("id")).toString();
const QJsonArray extensions = ruleObj.value(QStringLiteral("metadata"))
.toObject()
.value(QStringLiteral("extensions"))
.toArray();

QStringList values;
values.reserve(extensions.size());
for (const QJsonValue &extValue : extensions) {
values.append(extValue.toString());
}
extensionsByRuleId.insert(ruleId, values);
}
}

return extensionsByRuleId;
}

QStringList expectedExtsForRule(const char *ruleId)
{
static const QHash<QString, QStringList> kExtensionsByRuleId = loadFiletypeExtensions();
const auto it = kExtensionsByRuleId.constFind(QLatin1String(ruleId));
if (it == kExtensionsByRuleId.constEnd()) {
qFatal("Missing filetype rule in test helper: %s", ruleId);
}
return it.value();
}

QStringList imageExpectedExts()
{
return QStringList {
"ani", "avif", "avifs", "bmp", "bw", "dci", "eps", "epsf", "epsi", "exr", "gif",
"heic", "heif", "heis", "heix", "icns", "ico", "jpe", "jpeg", "jpg", "kra", "mng",
"ora", "pcx", "pic", "png", "psd", "raf", "ras", "raw", "rgb", "rgba", "sgi", "svg",
"svgz", "tga", "tif", "tiff", "wbmp", "webp", "wmf", "xcf", "cr2", "crw", "nef", "dng",
"arw", "orf", "rw2", "pef", "srw", "mrw", "x3f", "dcr", "kdc", "3fr", "nrw", "mef", "iiq",
"sr2", "erf", "mos", "rwl"
};
return expectedExtsForRule("filetype_image");
}

QStringList videoExpectedExts()
{
return QStringList {
"3g2", "3gp", "3gp2", "3gpp", "amr", "amv", "asf", "asx", "avi", "bdmv", "bik", "d2v",
"divx", "drc", "dsa", "dsm", "dss", "dsv", "evo", "f4v", "flc", "fli", "flic", "flv",
"hdmov", "ifo", "ivf", "m1v", "m2p", "m2t", "m2ts", "m2v", "m4b", "m4p", "m4v", "mkv",
"mp2v", "mp4", "mp4v", "mpe", "mpeg", "mpg", "mpls", "mpv2", "mpv4", "mov", "mts", "ogm",
"ogv", "pss", "pva", "qt", "ram", "ratdvd", "rm", "rmm", "rmvb", "roq", "rpm", "smil", "smk",
"swf", "tp", "tpr", "vob", "vp6", "webm", "wm", "wmp", "wmv"
};
return expectedExtsForRule("filetype_video");
}

QStringList audioExpectedExts()
{
return QStringList {
"aac", "ac3", "aif", "aifc", "aiff", "au", "cda", "dts", "fla", "flac", "it", "m1a",
"m2a", "m3u", "m4a", "mid", "midi", "mka", "mod", "mp2", "mp3", "mpa", "ogg", "opus",
"ra", "rmi", "spc", "snd", "umx", "voc", "wav", "wma", "xm", "ape"
};
return expectedExtsForRule("filetype_audio");
}

} // namespace
Expand Down Expand Up @@ -570,30 +610,24 @@
{
ParsedIntent intent;
m_parser->parse(QStringLiteral("文档"), intent);
const QStringList expectedExts = { "rtf", "odt", "ods", "odp", "odg",
"docx", "xlsx", "et", "pptx", "ppsx",
"md", "xls", "xlsb", "doc", "dot", "wps",
"ppt", "pps", "txt", "pdf", "dps", "sh",
"json", "yaml", "ini", "bat", "sql", "uof",
"ofd", "pot" };
const QStringList expectedExts = expectedExtsForRule("filetype_document_general");
QVERIFY(setEquals(intent.fileExtensions(), expectedExts));
}

void tst_ChineseNLP::fileType_general_spreadsheet()
{
ParsedIntent intent;
m_parser->parse(QStringLiteral("表格"), intent);
QVERIFY(intent.fileExtensions().contains("xls"));
QVERIFY(intent.fileExtensions().contains("xlsx"));
QVERIFY(intent.fileExtensions().contains("csv"));
const QStringList expectedExts = expectedExtsForRule("filetype_spreadsheet_general");
QVERIFY(setEquals(intent.fileExtensions(), expectedExts));
}

void tst_ChineseNLP::fileType_general_presentation()
{
ParsedIntent intent;
m_parser->parse(QStringLiteral("幻灯片"), intent);
QVERIFY(intent.fileExtensions().contains("ppt"));
QVERIFY(intent.fileExtensions().contains("pptx"));
const QStringList expectedExts = expectedExtsForRule("filetype_presentation_general");
QVERIFY(setEquals(intent.fileExtensions(), expectedExts));
}

// ===== Keyword Tests =====
Expand Down Expand Up @@ -1064,12 +1098,7 @@
QStringLiteral("文章"), QStringLiteral("方案"), QStringLiteral("文本"),
QStringLiteral("资料"), QStringLiteral("笔记"), QStringLiteral("稿件")
};
const QStringList expectedExts = { "rtf", "odt", "ods", "odp", "odg",
"docx", "xlsx", "et", "pptx", "ppsx",
"md", "xls", "xlsb", "doc", "dot", "wps",
"ppt", "pps", "txt", "pdf", "dps", "sh",
"json", "yaml", "ini", "bat", "sql", "uof",
"ofd", "pot" };
const QStringList expectedExts = expectedExtsForRule("filetype_document_general");
for (const QString &input : inputs) {
ParsedIntent intent;
m_parser->parse(input, intent);
Expand All @@ -1087,15 +1116,13 @@
QStringLiteral("表格"), QStringLiteral("统计表"), QStringLiteral("报表"),
QStringLiteral("名单"), QStringLiteral("数据表"), QStringLiteral("明细")
};
const QStringList expectedExts = expectedExtsForRule("filetype_spreadsheet_general");
for (const QString &input : inputs) {
ParsedIntent intent;
m_parser->parse(input, intent);
QVERIFY2(intent.fileExtensions().contains("xls"),
qPrintable(QStringLiteral("Missing xls for: ") + input));
QVERIFY2(intent.fileExtensions().contains("xlsx"),
qPrintable(QStringLiteral("Missing xlsx for: ") + input));
QVERIFY2(intent.fileExtensions().contains("csv"),
qPrintable(QStringLiteral("Missing csv for: ") + input));
QVERIFY2(setEquals(intent.fileExtensions(), expectedExts),
qPrintable(QStringLiteral("Failed for input: ") + input
+ QStringLiteral(" got: ") + intent.fileExtensions().join(",")));
}
}

Expand All @@ -1106,13 +1133,13 @@
QStringLiteral("幻灯片"), QStringLiteral("演示文稿"), QStringLiteral("汇报"),
QStringLiteral("课件"), QStringLiteral("宣讲")
};
const QStringList expectedExts = expectedExtsForRule("filetype_presentation_general");
for (const QString &input : inputs) {
ParsedIntent intent;
m_parser->parse(input, intent);
QVERIFY2(intent.fileExtensions().contains("ppt"),
qPrintable(QStringLiteral("Missing ppt for: ") + input));
QVERIFY2(intent.fileExtensions().contains("pptx"),
qPrintable(QStringLiteral("Missing pptx for: ") + input));
QVERIFY2(setEquals(intent.fileExtensions(), expectedExts),
qPrintable(QStringLiteral("Failed for input: ") + input
+ QStringLiteral(" got: ") + intent.fileExtensions().join(",")));
}
}

Expand Down Expand Up @@ -2290,12 +2317,9 @@
m_parser->parse(QStringLiteral("图片目录中的文档"), intent);
QCOMPARE(intent.searchDirectories().size(), 1);
QCOMPARE(intent.searchDirectories().first(), picsPath);
// Document extensions present, but NOT image extensions (e.g. "jpg")
QVERIFY(!intent.fileExtensions().isEmpty());
QVERIFY(!intent.fileExtensions().contains("jpg"));
QVERIFY(!intent.fileExtensions().contains("png"));
QVERIFY(intent.fileExtensions().contains("doc"));
QVERIFY(intent.fileExtensions().contains("pdf"));
QVERIFY2(setEquals(intent.fileExtensions(), expectedExtsForRule("filetype_document_general")),
qPrintable("Expected only document extensions, got: "
+ intent.fileExtensions().join(", ")));
QVERIFY(intent.keywords().isEmpty());
}

Expand Down
6 changes: 3 additions & 3 deletions src/dfm-search/dfm-search-lib/semantic/rules/zh_CN/filetype_rules.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@
"enabled": true,
"priority": 100,
"metadata": {
"extensions": ["rtf","odt","ods","odp","odg","docx","xlsx","et","pptx","ppsx","md","xls","xlsb","doc","dot","wps","ppt","pps","txt","pdf","dps","sh","json","yaml","ini","bat","sql","uof","ofd","pot"],
"extensions": ["rtf","odt","ods","odp","odg","docx","xlsx","et","pptx","ppsx","md","xls","xlsb","doc","dot","wps","ppt","pps","txt","pdf","dps","sh","json","yaml","ini","bat","sql","uof","ofd","pot","docm","dotx","dotm","wpt","uot","uott","uot3","uott3","ott","eig","otl"],
"fileTypes": ["doc"],
"general": true
}
Expand All @@ -67,7 +67,7 @@
"enabled": true,
"priority": 100,
"metadata": {
"extensions": ["xls", "xlsx", "csv", "ods", "et"],
"extensions": ["xls", "xlsx", "csv", "ods", "et", "xlsm", "xlt", "xltx", "xltm", "ett", "uos", "uost", "uos3", "uost3", "ots", "tsv", "dbt"],
"fileTypes": ["doc"],
"general": true
}
Expand All @@ -79,7 +79,7 @@
"enabled": true,
"priority": 100,
"metadata": {
"extensions": ["ppt", "pptx", "dps", "odp"],
"extensions": ["ppt", "pptx", "dps", "odp", "pptm", "ppsm", "potx", "potm", "dpt", "uop", "uopt", "uop3", "uopt3", "otp"],
"general": true
}
},
Expand Down
Loading