Skip to content

README.md

Latest commit

 

History

History
191 lines (137 loc) · 6.56 KB

README.md

File metadata and controls

191 lines (137 loc) · 6.56 KB
Raw

DuckDBArray Benchmark Scripts

These scripts provide benchmarks for DuckDBArray that are directly comparable to the HDF5Array performance vignette.

Quick Start: Side-by-Side Comparison

Copy and paste this into R to compare DuckDBArray vs HDF5Array on your machine:

# Install packages (if needed)
BiocManager::install(c("DuckDBArray", "HDF5Array", "ExperimentHub", "RSpectra"))

# Download data (~2 GB, only needed once)
library(ExperimentHub)
hub <- ExperimentHub()
hub[["EH1039"]]

# Create a working directory for benchmark output
benchmark_dir <- file.path(tempdir(), "bioc_benchmarks")
dir.create(benchmark_dir, showWarnings = FALSE)
setwd(benchmark_dir)

# Run the comparison (200K cells, ~30-60 min)
scripts_dir <- system.file(package = "DuckDBArray", "scripts")
system(paste(file.path(scripts_dir, "run_comparison.sh"), "200000 1000"))

For a quicker test (~5-10 min), use 50K cells instead:

system(paste(file.path(scripts_dir, "run_comparison.sh"), "50000 1000"))

Results are saved to your working directory (check getwd()).

Note: These shell scripts require macOS or Linux. On Windows, run the R scripts directly (see normalize_and_PCA.R below).

Key Differences from HDF5Array

Aspect HDF5Array DuckDBArray
Block size tuning Required (4 sizes tested) Not needed
Format variations 3 formats (s, D, Ds) 1 format (Parquet)
Realization step Required between steps Not needed
Total benchmark runs 240 (5 × 2 × 3 × 4 × 2) 10 (5 × 2)
Expected runtime 20-55 hours 1-3 hours

Scripts

normalize_and_PCA.R

Runs DuckDBArray normalization and PCA on the 10x Genomics 1.3M Brain Cell Dataset.

# From R: run DuckDBArray benchmark only
scripts_dir <- system.file(package = "DuckDBArray", "scripts")
script_path <- file.path(scripts_dir, "normalize_and_PCA.R")
system(paste("Rscript", script_path, "200000 1000"))  # 200K cells

Arguments: <ncells> <num_var_genes> [part_block_size] [part_block_shape]

  • ncells: Number of cells to process
  • num_var_genes: Number of variable genes to select
  • part_block_size: Partition block size in Mb (default: 1000 = 1GB)
  • part_block_shape: Partition block shape, "scale" or "first-dim-grows-first" (default: "scale")

run_benchmarks.sh

Runs the full DuckDBArray benchmark suite (all cell counts: 12.5K to 200K).

# From R: run the full benchmark suite
benchmark_dir <- file.path(tempdir(), "bioc_benchmarks")
dir.create(benchmark_dir, showWarnings = FALSE)
setwd(benchmark_dir)

scripts_dir <- system.file(package = "DuckDBArray", "scripts")
system(file.path(scripts_dir, "run_benchmarks.sh"))

run_comparison.sh

Runs both DuckDBArray and HDF5Array on the same machine for direct comparison.

# From R: run the side-by-side comparison
benchmark_dir <- file.path(tempdir(), "bioc_benchmarks")
dir.create(benchmark_dir, showWarnings = FALSE)
setwd(benchmark_dir)

scripts_dir <- system.file(package = "DuckDBArray", "scripts")
system(paste(file.path(scripts_dir, "run_comparison.sh"), "200000 1000"))

The script automatically:

  1. Runs DuckDBArray with default settings (no tuning needed)
  2. Runs HDF5Array with optimal settings (250 Mb blocks, sparse format)
  3. Prints a side-by-side timing comparison

Output is saved to comparison_200000_1000/ in your working directory:

  • duckdb.log and hdf5.log - Full output from each run
  • duckdb_timings.dcf and hdf5_timings.dcf - Machine-readable timings

compare_to_hdf5array.R

Compares your DuckDBArray results to HDF5Array's published vignette timings (useful if you don't want to run HDF5Array yourself).

# From R: compare your results to published HDF5Array timings
scripts_dir <- system.file(package = "DuckDBArray", "scripts")
script_path <- file.path(scripts_dir, "compare_to_hdf5array.R")
system(paste("Rscript", script_path, "timings.dcf"))

Why DuckDBArray is Simpler

  1. No block size tuning: DuckDB's query planner handles memory management automatically. HDF5Array requires testing 4 different block sizes to find optimal performance.

  2. No format variations: DuckDBArray uses Parquet/COO format exclusively. HDF5Array tests 3 formats (TENxMatrix sparse, HDF5Matrix dense, HDF5Matrix as sparse).

  3. No realization step: HDF5Array's workflow requires writing the normalized matrix back to disk before PCA. DuckDBArray's normalized matrix remains efficiently queryable without an intermediate write.

DuckDB-Specific Optimizations

The simple_normalize() function in this benchmark uses a key optimization that differs from HDF5Array's approach:

Statistics First, Filter Last

DuckDBMatrix excels at SQL aggregations (rowSums, colSums, rowVars, rowSds), which run as efficient GROUP BY queries. However, row subsetting creates inefficient NOT IN queries with potentially thousands of values.

Our approach:

  1. Compute all statistics on the full matrix (fast SQL operations)
  2. Apply the column normalization sweep on the full matrix
  3. Select highly variable genes at the very end (single subsetting operation)

The results are mathematically equivalent because zero-sum rows have zero variance and would never be selected as HVGs anyway.

Expected Results

Based on the DuckDBArray vignette benchmarks, you should see:

  • Normalization: 15-30x faster than HDF5Array
  • PCA: Similar to HDF5Array (matrix multiplication is backend-agnostic)
  • Total workflow: 5-20x faster than HDF5Array

The exact speedup depends on your hardware, but the key observation is that DuckDBArray achieves these results without any tuning.

Comparison with HDF5Array Vignette

The HDF5Array performance vignette reports these timings for 200,000 cells on a DELL XPS 15 laptop (8 cores, 32 GB RAM) using TENxMatrix:

Operation HDF5Array (best block size) Block size
Normalization ~643 s 250 Mb
Realization ~69 s 40 Mb
PCA ~692 s 40 Mb
Total ~1,404 s (23 min)

DuckDBArray typically completes the equivalent workflow in under 2 minutes on similar hardware, representing a ~10-30x speedup.

Hardware Requirements

  • RAM: 8 GB minimum, 32 GB recommended
  • Storage: 10 GB free space for benchmark data
  • The 10x Genomics 1.3M Brain Cell Dataset will be downloaded via ExperimentHub

Dependencies

Required packages:

  • DuckDBArray
  • ExperimentHub
  • HDF5Array (for data loading and optional memory tracking)
  • MatrixGenerics
  • RSpectra (for PCA)