English | 中文
FOAMS: Efficient Anonymized Network Sensing using Compact Matrix Format and Multi-Stage Pipeline Parallelism
FOAMS is a GPU-accelerated framework developed for the IEEE HPEC 2026 Anonymized Network Sensing Graph Challenge. It extracts IPv4 source and destination addresses from classic PCAP files, represents traffic matrices in a compact Dictionary-Offset Format (DOF), performs GPU-based formatting, dictionary-level IP anonymization, and compression, and then reloads and aggregates the data to compute the network statistics defined by the challenge.
The Anonymized Network Sensing Graph Challenge transforms raw network packets into anonymized source-to-destination traffic matrices that can be shared and analyzed collaboratively. Its end-to-end workflow contains six timed stages:
- Read or stream PCAP input.
- Extract source and destination IP addresses.
- Anonymize IP addresses consistently across files.
- Construct one sparse traffic matrix for every
$N_v=2^{17}$ valid address pairs. - Store the traffic matrices.
- Reload, globally aggregate, and analyze the matrices.
The reference implementation uses GraphBLAS, but GraphBLAS is not required. FOAMS targets the PCAP ingestion, redundant anonymization, and intermediate matrix movement costs that remain significant after matrix computation is accelerated on a GPU.
Official resources:
- GraphChallenge website and HPEC 2026 submission information
- Challenge specification, reference code, and examples
- Official datasets
- Anonymized Network Sensing Graph Challenge paper
- End-to-end PCAP-to-statistics Graph Challenge workflow
- IPv4 source-to-destination pair extraction from classic PCAP
- Ethernet, single-level 802.1Q VLAN, and Raw IP link types
- CUDA, Thrust, and CUB based data processing and aggregation
- DOF representation that separates IP dictionaries from packet-level offsets
- Crypto-PAn invocation only for distinct IPv4 addresses
- Delta encoding, variable-length integer encoding, and nvCOMP Zstandard compression
- Parallel chunked file I/O
- All nine source, destination, link, and packet statistics required by the challenge
PCAP
-> partition valid address pairs into Nv = 2^17 windows
-> extract and sort IPv4 source/destination pairs
-> build DOF offset arrays (R and C)
-> anonymize unique address dictionaries (R_value and C_value)
-> Delta + Varint + nvCOMP Zstd compression and storage
-> reload and decompress all blocks
-> reconstruct anonymized address pairs
-> GPU aggregation and analysis
R and C correspond to src_off and dst_off in the paper. R_value and C_value correspond to the anonymized src_dict and dst_dict. Pair i is reconstructed as R_value[R[i]] and C_value[C[i]]. DOF changes only the intermediate physical representation and preserves the logical traffic information required for aggregation.
The complete design described in the paper organizes Host Input, Device Processing, and Host Output as a three-stage producer-consumer pipeline. The current public code overlaps the main-thread PCAP scan with worker threads processing large batches. The fully queue-based three-stage scheduler is still being consolidated.
- Linux
- CMake 3.20 or later
- A C++17 and CUDA C++17 toolchain
- NVIDIA CUDA Toolkit
- NVIDIA nvCOMP with the Zstandard Manager API
- An NVIDIA GPU supporting the selected CUDA architecture
CMake currently fixes CMAKE_CUDA_ARCHITECTURES to 90 for Hopper/SM90 GPUs. Change this value in CMakeLists.txt before building for another GPU architecture.
Set the CUDA and nvCOMP paths:
export CUDA_HOME=/usr/local/cuda
export NVCOMP_ROOT=/path/to/nvcompWhen NVCOMP_ROOT is unset, CMake falls back to:
$BASE_DIR/install/nvcomp-install
Configure and build:
cmake -S . -B build
cmake --build build -jThe executable is generated at build/run.
./build/run <input.pcap> <output_dir/>Example:
mkdir -p output
./build/run ./capture.pcap ./output/The current implementation appends block_/ directly to the output directory string, so output_dir must end with a slash.
The program prints the following statistics:
======== PCAP Analysis Results ========
max_destination_packets = ...
max_fan_in = ...
max_fan_out = ...
max_packets = ...
max_source_packets = ...
n_destinations = ...
n_links = ...
n_packets = ...
n_sources = ...
=======================================
| Metric | Meaning |
|---|---|
| max_destination_packets | Maximum number of packets received by one destination |
| max_fan_in | Maximum number of distinct sources connected to one destination |
| max_fan_out | Maximum number of distinct destinations connected to one source |
| max_packets | Maximum packet count on one source-to-destination link |
| max_source_packets | Maximum number of packets sent by one source |
| n_destinations | Number of distinct destinations |
| n_links | Number of distinct source-to-destination links |
| n_packets | Number of packets included in aggregation |
| n_sources | Number of distinct sources |
Each processing batch is stored in a separate directory:
output/
└── block_0/
├── R_000.lz4 ... R_015.lz4
├── C_000.lz4 ... C_015.lz4
├── R_value_000.lz4 ... R_value_015.lz4
└── C_value_000.lz4 ... C_value_015.lz4
- R and C are the relabeled source and destination dictionary offsets.
- R_value and C_value are the anonymized source and destination dictionaries.
- Each array is split into THREAD_NUMBER files; the default is 16.
Despite their .lz4 extension, the current files contain an nvCOMP Zstandard native bitstream. They are not ordinary LZ4 files and must be read with the decompression path provided by this project.
Compile-time settings are defined in config.h:
| Setting | Default | Purpose |
|---|---|---|
| CHUNK_SIZE | 1 << 17 | Graph Challenge traffic-matrix window size |
| FOLDER_SIZE | CHUNK_SIZE * 1024 | Maximum address pairs in one pipeline/output batch |
| THREAD_NUMBER | 16 | Number of parallel file chunks per array |
| BLOCK_SIZE | 256 | CUDA kernel block size |
| ENABLE_TIME_MEASURE | enabled | Enables timing and CSV logging |
With timing enabled, results are appended to the path currently hard-coded in config.h:
/home/maijun/data/ANS/results.csv
Change logFile or disable ENABLE_TIME_MEASURE before running on another machine.
.
├── main.cu # PCAP input, batch scheduling, and main workflow
├── config.h # Compile-time configuration
├── processing/
│ ├── extract.cpp # IPv4 pair extraction
│ ├── matrix.cu # Window sorting and DOF construction
│ ├── ip_anonymize.cu # GPU Crypto-PAn anonymization
│ ├── compress.cu # Encoding and nvCOMP compression
│ └── write.cpp # Parallel chunked output
├── analysis/
│ ├── read.cpp # Parallel chunked input
│ ├── decompress.cu # Decompression and decoding
│ ├── aggregate.cu # GPU aggregation
│ └── analysis.cu # Analysis workflow
└── utils/ # Crypto-PAn, timing, and helper utilities
- Only classic PCAP is supported; PCAPNG is not supported.
- Only IPv4 is processed; IPv6 is not supported.
- Ethernet parsing covers plain IPv4 and one 802.1Q VLAN level.
- PCAP byte order and timestamp variants are not fully selected from the magic number.
- Frames that cannot be parsed currently produce a zero address pair; validate results when using mixed-protocol captures.
- ip_anonymize.cu contains a fixed default Crypto-PAn key. It is suitable only for development and experiments. Replace it with a securely generated and managed key before processing sensitive data.
- The program allocates CPU and GPU memory in large batches. Adjust FOLDER_SIZE for the available hardware.
If this project is useful in your research, please cite both the FOAMS paper and the Graph Challenge specification. The final FOAMS publication details and BibTeX entry will be added after publication.
FOAMS: Efficient Anonymized Network Sensing using Compact Matrix Format
and Multi-Stage Pipeline Parallelism.
Graph Challenge specification:
@inproceedings{jananthan2024anonymized,
title = {Anonymized Network Sensing Graph Challenge},
author = {Hayden Jananthan and others},
booktitle = {IEEE High Performance Extreme Computing Conference (HPEC)},
year = {2024}
}This repository does not currently contain a license file. All rights are reserved until an explicit license is added.