MEDomics uses MongoDB as its main storage and querying system, which works well so far for small-scale datasets. However, it can run into trouble when datasets get really large, like when doing heavy aggregations or large scans; hence, leading to crashes, as I understand it.
DuckDB (https://duckdb.org/) is a modern, easy-to-use database engine that can be a good candidate to embed directly into MEDomics (no need for a server). It reads from common formats like CSV, Parquet, and JSON and can handle multi-gigabyte datasets without any trouble on a standard laptop. So, I reckon it might be a good idea to combine it with MongoDB to make things easier when users are working with large-scale datasets.
Here are three clients to look into: the Python client (https://duckdb.org/docs/lts/clients/python/overview); the Node.js client (https://duckdb.org/docs/lts/clients/nodejs/overview) for the Electron side; and DuckDB-WASM (https://duckdb.org/docs/current/clients/wasm/overview) for handling everything on the client side.
Before all though, it seems worth validating the approach outside the platform entirely. @StRavary's INSPECT dataset is a good candidate: load it through each and one of the above mentionned client, run a handful of representative (SQL) queries and (Python-side) lookups, and confirm the engine handles it without / or with less memory issues.
Runtime-Extensible Parsers (CIDR 2025)
Robust External Hash Aggregation in the Solid State Age (ICDE 2024)
These Rows Are Made for Sorting and That's Just What We'll Do (ICDE 2023)
Join Order Optimization with (Almost) No Statistics (Master thesis, 2022)
DuckDB-Wasm: Fast Analytical Processing for the Web (VLDB 2022 demo)
Data Management for Data Science - Towards Embedded Analytics (CIDR 2020)
DuckDB: An Embeddable Analytical Database (SIGMOD 2019 demo)
Notes
Happy to contribute on this one, just let me know 👌
MEDomics uses MongoDB as its main storage and querying system, which works well so far for small-scale datasets. However, it can run into trouble when datasets get really large, like when doing heavy aggregations or large scans; hence, leading to crashes, as I understand it.
DuckDB (https://duckdb.org/) is a modern, easy-to-use database engine that can be a good candidate to embed directly into MEDomics (no need for a server). It reads from common formats like CSV, Parquet, and JSON and can handle multi-gigabyte datasets without any trouble on a standard laptop. So, I reckon it might be a good idea to combine it with MongoDB to make things easier when users are working with large-scale datasets.
Here are three clients to look into: the Python client (https://duckdb.org/docs/lts/clients/python/overview); the Node.js client (https://duckdb.org/docs/lts/clients/nodejs/overview) for the Electron side; and DuckDB-WASM (https://duckdb.org/docs/current/clients/wasm/overview) for handling everything on the client side.
Before all though, it seems worth validating the approach outside the platform entirely. @StRavary's INSPECT dataset is a good candidate: load it through each and one of the above mentionned client, run a handful of representative (SQL) queries and (Python-side) lookups, and confirm the engine handles it without / or with less memory issues.
Peer-Reviewed Papers and Thesis Works
Runtime-Extensible Parsers (CIDR 2025)
Robust External Hash Aggregation in the Solid State Age (ICDE 2024)
These Rows Are Made for Sorting and That's Just What We'll Do (ICDE 2023)
Join Order Optimization with (Almost) No Statistics (Master thesis, 2022)
DuckDB-Wasm: Fast Analytical Processing for the Web (VLDB 2022 demo)
Data Management for Data Science - Towards Embedded Analytics (CIDR 2020)
DuckDB: An Embeddable Analytical Database (SIGMOD 2019 demo)
Notes
Happy to contribute on this one, just let me know 👌