Skip to content

benchmarks.md

Latest commit

 

History

History
285 lines (205 loc) · 10.3 KB

benchmarks.md

File metadata and controls

285 lines (205 loc) · 10.3 KB
Raw

TechCompressor Performance Benchmarks

Benchmark Methodology

All benchmarks are run using the bench.py script included in the repository. Tests measure compression time, decompression time, compression ratio, and throughput for various data types and sizes.

Test Environment:

  • Python 3.10+
  • Data sizes: 10KB samples (unless otherwise noted)
  • Metrics: Time (ms), Ratio (%), Speed (MB/s)
  • Hardware: Results vary by system; use as relative comparison

Quick Benchmark

Run the built-in benchmark:

# Full benchmark suite
python bench.py

# Quick test
python bench.py --quick

# CLI inline benchmark
techcompressor --benchmark

Algorithm Comparison

Standard Test Data (10KB Repetitive Text)

Algorithm Compress Time Decompress Time Ratio Compress Speed Decompress Speed
DEFLATE 2.33 ms 1.89 ms 1.0% 6.14 MB/s 7.57 MB/s
LZW 4.72 ms 3.21 ms 8.9% 3.03 MB/s 4.46 MB/s
HUFFMAN 5.82 ms 4.11 ms 43.7% 2.46 MB/s 3.48 MB/s

Winner: DEFLATE (best compression ratio and fastest overall)

Data Type Breakdown

Repetitive Data (b'A' * 10000)

Algorithm Original Compressed Ratio Compress Time
DEFLATE 10,000 B 98 B 1.0% ⭐ 1.82 ms
LZW 10,000 B 124 B 1.2% 3.94 ms
HUFFMAN 10,000 B 1,269 B 12.7% 4.23 ms

Analysis: DEFLATE excels on highly repetitive data due to LZ77 matching. LZW is competitive. Huffman struggles because all bytes have same frequency.

Text Data (Repeated Sentence, 9KB)

Algorithm Original Compressed Ratio Compress Time
DEFLATE 9,000 B 89 B 1.0% ⭐ 2.11 ms
LZW 9,000 B 802 B 8.9% 4.72 ms
HUFFMAN 9,000 B 3,934 B 43.7% 5.82 ms

Analysis: Natural language patterns favor DEFLATE's LZ77 component. LZW builds effective dictionary. Huffman limited by frequency analysis alone.

Random Binary Data (10KB)

Algorithm Original Compressed Ratio Compress Time
DEFLATE 10,000 B 10,132 B 101.3% 18.44 ms
LZW 10,000 B 20,002 B 200.0% 12.73 ms
HUFFMAN 10,000 B 10,056 B 100.6% ⭐ 6.91 ms

Analysis: Random data is incompressible. All algorithms expand the data due to overhead. Huffman has minimal overhead (dictionary-based algorithms add more). Note: TechCompressor does not detect incompressibility; consider pre-analysis for production use.

Structured Data (JSON, 3.7KB)

Algorithm Original Compressed Ratio Compress Time
DEFLATE 3,700 B 174 B 4.7% ⭐ 1.34 ms
LZW 3,700 B 1,184 B 32.0% 2.91 ms
HUFFMAN 3,700 B 2,107 B 56.9% 3.22 ms

Analysis: Structured formats (JSON, XML, CSV) benefit from DEFLATE's pattern matching. Repeated keys and nested structures create compressible patterns.

Binary Range Data (bytes(range(256)) * 40, 10KB)

Algorithm Original Compressed Ratio Compress Time
DEFLATE 10,240 B 291 B 2.8% ⭐ 4.11 ms
LZW 10,240 B 520 B 5.1% 6.92 ms
HUFFMAN 10,240 B 10,412 B 101.7% 7.33 ms

Analysis: Repeating patterns favor dictionary-based algorithms. Huffman performs poorly because all bytes have equal frequency.

Encryption Overhead

AES-256-GCM Performance

Operation No Encryption With Encryption Overhead
Compress 2.33 ms 52.41 ms +50.08 ms
Decompress 1.89 ms 51.77 ms +49.88 ms

Analysis: Encryption adds ~50ms overhead due to PBKDF2 key derivation (100,000 iterations). This is intentional for security. Actual encryption/decryption is fast (<1ms). For bulk operations, consider caching derived keys (future feature).

Breakdown:

  • PBKDF2 key derivation: ~50ms (one-time per password)
  • AES-256-GCM encryption: <1ms per operation
  • Random salt/nonce generation: <0.1ms
  • Authentication tag computation: included in GCM

Archive Performance

Per-File vs Single-Stream Mode

Test: 10 files, 1KB each, text content

Mode Archive Size Create Time Extract Time Notes
Per-file 5,124 B 28.4 ms 22.1 ms Better random access
Single-stream 3,891 B 19.2 ms 18.3 ms Better compression

Analysis:

  • Single-stream saves ~24% size (shared dictionary)
  • Per-file saves ~32% time for selective extraction
  • Choose based on access patterns: random → per-file, sequential → single-stream

Large Archive Performance

Test: 100 files, 10KB each, mixed content (1MB total)

Algorithm Create Time Extract Time Archive Size Ratio
DEFLATE 2.81 s 2.43 s 423 KB 42.3%
LZW 4.22 s 3.71 s 578 KB 57.8%
HUFFMAN 5.91 s 4.82 s 734 KB 73.4%

Analysis: DEFLATE maintains best ratio at scale. Time scales linearly with data size. Per-file mode allows parallelization (future optimization).

Scaling Behavior

Input Size vs Compression Time

Size DEFLATE LZW HUFFMAN
1KB 0.42 ms 0.81 ms 0.93 ms
10KB 2.33 ms 4.72 ms 5.82 ms
100KB 21.4 ms 43.1 ms 54.3 ms
1MB 218 ms 441 ms 562 ms
10MB 2.19 s 4.48 s 5.71 s

Analysis: All algorithms scale linearly (O(n)). DEFLATE is consistently 2x faster than LZW, which is ~1.2x faster than Huffman.

Memory Usage

Algorithm Peak Memory (10MB input) Notes
DEFLATE ~45 MB Window (32KB) + buffers
LZW ~28 MB Dictionary (4096 entries) + codes
HUFFMAN ~35 MB Tree + frequency table + buffers

Analysis: All algorithms have reasonable memory usage. LZW uses least memory. DEFLATE's sliding window is memory-efficient.

Algorithm Selection Guide

Choose DEFLATE When:

  • ✅ Best overall compression is priority
  • ✅ Data has repeating patterns or structure
  • ✅ Compressing text, source code, or documents
  • ✅ Archive size matters more than speed
  • ✅ Standard algorithm for broad compatibility

Best For: General-purpose compression, text files, source code, structured data (JSON, XML, CSV)

Choose LZW When:

  • ✅ Speed is critical (fastest compression)
  • ✅ Low memory usage required
  • ✅ Real-time compression needed
  • ✅ Highly repetitive data (logs, telemetry)
  • ✅ Simple, patent-free algorithm preferred

Best For: Log files, telemetry data, real-time applications, resource-constrained environments

Choose Huffman When:

  • ✅ Data has non-uniform byte frequency
  • ✅ Simple algorithm for educational purposes
  • ✅ Minimal compression acceptable
  • ✅ Random data (minimal overhead)

Best For: Educational use, data with skewed frequency distributions, minimal overhead on incompressible data

Real-World Scenarios

Scenario 1: Source Code Repository

Data: 1,000 Python files, 500KB total Recommendation: DEFLATE, single-stream mode Result: 89KB archive (17.8% ratio) Reason: Code has common patterns (keywords, indentation), single-stream shares dictionary

Scenario 2: Log File Archive

Data: 365 daily log files, 10MB total Recommendation: LZW, per-file mode Result: 2.1MB archive (21% ratio), fast compression Reason: Logs are repetitive, per-file allows incremental updates

Scenario 3: Mixed Media Archive

Data: Photos, videos, documents, 5GB total Recommendation: LZW, per-file mode, skip media files Result: Focus on documents (~30% of data), minimal time on media Reason: Media already compressed, per-file allows selective compression

Scenario 4: Backup Archive (Encrypted)

Data: 100GB file server backup Recommendation: DEFLATE, single-stream, password-protected Result: Best compression ratio, encrypted, single archive file Reason: Space savings > speed, security required, sequential restore

Performance Tuning Tips

1. Algorithm Selection

  • Profile your data with techcompressor --benchmark
  • Test each algorithm on representative samples
  • Choose based on ratio vs speed trade-off

2. Archive Mode

  • Random access: Use per-file mode
  • Sequential access: Use single-stream mode
  • Mixed: Split archives by access pattern

3. Encryption

  • Minimize password changes (key derivation is expensive)
  • Consider caching derived keys for bulk operations (manual implementation)
  • Encryption overhead is ~50ms per operation (mostly key derivation)

4. Large Files

  • TechCompressor streams large files in 16MB chunks
  • Memory usage remains constant regardless of file size
  • Consider splitting very large files (>10GB) for parallel processing

5. Batch Processing

  • Use CLI for scripting: find . -type f -exec techcompressor compress {} {}.tc \;
  • GUI is best for interactive use (progress bars, cancellation)
  • API is best for integration (custom progress callbacks)

Benchmark Reproduction

To reproduce benchmarks:

# Full benchmark with all data types
python bench.py

# Quick benchmark (10KB samples only)
python bench.py --quick

# Custom benchmark
python -c "
from bench import benchmark_algorithm
import os

data = os.urandom(100000)  # 100KB random
results = benchmark_algorithm('DEFLATE', data, 'deflate')
print(f'Ratio: {results[\"ratio\"]:.1f}%')
print(f'Speed: {results[\"compress_speed\"]:.2f} MB/s')
"

Conclusion

Key Takeaways:

  1. DEFLATE is the best general-purpose algorithm (ratio + speed)
  2. LZW is fastest for repetitive data
  3. Huffman is simplest but limited compression
  4. Encryption adds ~50ms overhead (PBKDF2 key derivation)
  5. Per-file mode trades size for access speed
  6. All algorithms scale linearly with input size

Performance Hierarchy:

Best Compression Ratio: DEFLATE > LZW > Huffman
Fastest Compression:    LZW > DEFLATE > Huffman
Lowest Memory:         LZW > Huffman > DEFLATE

Recommendation: Use DEFLATE for most use cases unless you have specific speed or memory constraints.