Skip to content

performance.md

Latest commit

 

History

History
168 lines (127 loc) · 5.07 KB

performance.md

File metadata and controls

168 lines (127 loc) · 5.07 KB

kernelradar - Performance

All numbers below were collected on the lowest-spec hardware kernelradar is officially supported on - the worst-case floor. Throughput on bigger hardware will be higher, but the project does not yet have measurements on Xeon/Threadripper/Ampere class boxes to publish here.

Hardware

Component Spec
CPU Intel Celeron J4125 @ 2.0 GHz (4 cores, no SMT, no AVX-512)
Memory 8 GB DDR4
Kernel 6.13.9-zabbly+ (Debian 12)

Headline numbers

Metric Value
Sustained event rate (BPF tracepoint) 321 000 events/sec
Userspace processed rate ~25 000 events/sec
Idle RSS 65 to 80 MB
RSS peak (HWM under 100k flood) 136 MB
Memory growth after 100k flood 0 bytes
Graceful shutdown time 641 ms (12 BPF programs detached)
Threads at runtime 5 (tokio + 4 detector tasks)

Methodology

All tests use the production daemon installed via make install, running as a systemd service in --format=journald mode. No artificial helpers in BPF or userspace code paths.

BPF tracepoint throughput

Configured rate limiter to a no-op (window_max = 1_000_000) so we measure raw kernel-side observation rate.

# Generate 100 000 setuid(0) calls from a non-root user (uid=1000) across 4 workers
for w in 1 2 3 4; do
  sudo -u testuser python3 -c '
    import os
    for _ in range(25_000):
        try: os.setuid(0)
        except: pass
  ' &
done
wait

Result:

Quantity Value
Wall time 0.311 sec
Events fired (kernel) 100 000
BPF observed 100 000 (every event captured)
Ring buffer drops 92 161 (userspace lagged behind kernel)
Userspace processed ~7 800
Effective fired-rate 321 001 / sec

The 92 161 drops are not a bug: at this synthetic rate, the kernel is producing events 40× faster than journald can consume them. In a real attack scenario you'd see at most 100s of events/sec, all of which userspace handles comfortably.

The drops are visible via kernelradar_*_dropped_total Prometheus counters - this is operationally important: an admin can distinguish "no attacks detected" from "we're losing events".

Graceful shutdown

$ time systemctl stop kernelradar

real    0m0.641s

After shutdown:

$ bpftool prog show | grep -c kr_tp_
0

Aya's Drop impl detaches every program before the userspace process exits. No leaked BPF programs, no leaked maps.

Memory profile

$ cat /proc/$(pidof kernelradar)/status | grep -E '^Vm'
VmSize: 286812 kB    # virtual address space (mostly tokio)
VmHWM:  136768 kB    # peak resident set during 100k flood
VmRSS:   80344 kB    # current resident set (post-flood, idle)

The 137 MB peak under a 320k events/sec flood is dominated by:

  1. The 256 KB BPF ring buffer per detector (8 detectors = 2 MB)
  2. Aya's program/map metadata (~10 MB)
  3. tokio worker stacks (4 × 8 MB = 32 MB)
  4. Allocations during burst (<100 MB)

After the flood subsides, RSS returns to its 80 MB baseline. There is no leak under load.

CPU profile

Under steady-state idle (no events): kernelradar consumes <0.1% of one core. Under 100k/sec flood: ~28% of one core. Profiling shows the hot path concentrated in:

  1. Ring buffer reader loop (~40% of cycles)
  2. JSON serialisation for journald output (~25%)
  3. comm/exe path resolution from /proc (~15%)
  4. Everything else < 5%

For deployments that don't need exe paths, setting RUST_LOG=...,kernelradar.alert=warn removes ~15% of CPU work on the alert hot path (informational events skipped).

Soak observations

Daemon ran for >1 hour across multiple flood tests, configuration reloads, baseline saves, and SIGHUP cycles. Memory remained at 65 to 80 MB. CPU returned to <1% within seconds of any load event.

For longer soaks (24+ hours) you'd want to watch:

  • systemctl show kernelradar -p MemoryCurrent over time
  • kernelradar_alerts_total rate (no monotonic growth pattern)
  • BPF program count = 12 (not increasing)

Comparison

Tool Idle RSS BPF programs Adaptive baseline
kernelradar 80 MB 12 yes
Falco ~200 MB 50+ no
Tetragon ~500 MB 30+ no
Tracee ~300 MB 25+ no

(Numbers from each project's published docs; kernelradar measured directly. Comparisons only meaningful as orders of magnitude.)

Tuning

If you need higher throughput under attack flood:

[ratelimit]
# Increase window_max so userspace doesn't suppress legitimate volume
window_max  = 1000

[detectors.privesc]
# Disable detectors whose surface isn't relevant to your threat model
enabled = false

Or if you want lower memory:

# Increase BPF ringbuf size? - currently a compile-time 256 KB.
# Disable heavy detectors (cred, fim) on systems that don't need them.