Skip to content

Commit 1f96ecf

Browse files
committed
build(python): faster copy of cpp files for the python build
1 parent c185d08 commit 1f96ecf

5 files changed

Lines changed: 39 additions & 27 deletions

File tree

examples/README.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,12 @@ We use [`uv`](https://docs.astral.sh/uv/) for fast Python environment and depend
2828
Navigating into any example directory and running `uv sync` or `uv run` will automatically set up a isolated Python virtual environment, build the local `deglib` C++ bindings, and install all required dependencies:
2929

3030
```bash
31-
cd examples/paper_reproduction
31+
cd examples/knng
3232
uv sync
3333
```
3434

3535
## Projects
3636

37-
- [`paper_reproduction`](./paper_reproduction/): Reproduce DEG paper search benchmarks (Recall vs QPS) on datasets mentioned in the root `readme.md` (`Audio`, `Enron`, `SIFT1M`, `DEEP1M`, `GloVe-100`).
37+
- [`knng`](./knng/): k-Nearest Neighbor Graph (k-NNG) construction benchmark using EVP quantization and FP16 reranking (SISAP 2026 Challenge Task 1).
38+
- [`static_data`](./static_data/): DEG paper search benchmark reproduction (Recall vs. QPS) on static datasets (`sift1m`, `deep1m`, `glove-100`, `audio`, `enron`).
39+
- [`dynamic_data`](./dynamic_data/): DEG dynamic data streaming benchmark (AddHalf, AddHalfRemoveAndAddOneAtATime, AddAllRemoveHalf).

examples/dynamic_data/README.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,12 +53,19 @@ uv run python main.py sift1m --graph-dir /data/deg_graphs/sift1m/dynamic
5353
uv run python main.py all --graph-dir /data/deg_graphs --threads 4
5454
```
5555

56-
## Setup
56+
## Prerequisites: Building the Python Library (`deglib`)
57+
58+
The `deglib` Python library contains C++ pybind11 bindings (`deglib_cpp`).
59+
60+
Since `pyproject.toml` references `deglib` locally (`path = "../../python"`), running `uv sync` automatically copies the C++ sources from `cpp/`, invokes CMake, and compiles/installs the latest `deglib` C++ bindings into the local environment:
5761

5862
```bash
63+
# Install dependencies & compile latest deglib C++ bindings
5964
uv sync
6065
```
6166

67+
> **Note:** A C++ compiler (MSVC on Windows, GCC/Clang on Linux/macOS) and CMake must be available in your system `PATH`. `setup.py` handles copying C++ files, running CMake, and compiling the extension during the `uv sync` step.
68+
6269
## Key Differences from `static_data`
6370

6471
- Iterates over 3 `DataStreamType`s instead of a single `AddAll`

examples/knng/README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,19 @@ The approach demonstrates **Mode 4 (`evp-rerank`)**, which achieves state-of-the
1313
3. **Graph Exploration**: Exploration for vertex $i$ walks the DEG graph neighborhood using fast EVP bit-level inner product distances to collect candidates (`evpK = 50`).
1414
4. **FP16 Candidate Reranking**: Exact inner-product distances are computed using `deglib_cpp.floats_to_fp16` and `deglib_cpp.fp16_to_floats` for candidate sets to produce final $k$-nearest neighbor edges.
1515

16+
## Prerequisites: Building the Python Library (`deglib`)
17+
18+
The `deglib` Python library contains C++ pybind11 bindings (`deglib_cpp`).
19+
20+
Since `pyproject.toml` references `deglib` locally (`path = "../../python"`), running `uv sync` automatically copies the C++ sources from `cpp/`, invokes CMake, and compiles/installs the latest `deglib` C++ bindings into the local environment:
21+
22+
```bash
23+
# Install dependencies & compile latest deglib C++ bindings
24+
uv sync
25+
```
26+
27+
> **Note:** A C++ compiler (MSVC on Windows, GCC/Clang on Linux/macOS) and CMake must be available in your system `PATH`. `setup.py` handles copying C++ files, running CMake, and compiling the extension during the `uv sync` step.
28+
1629
## Running the Benchmark with `uv`
1730

1831
### 1. Run Full Benchmark (200K Wikipedia BGE-M3 vectors)

examples/static_data/README.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,18 @@ This example project demonstrates how to download paper datasets from `readme.md
1010
- `deep1m` (96D, 1M base vectors, L2 distance)
1111
- `glove-100` (100D, 1.18M base vectors, Angular / InnerProduct distance)
1212

13-
## Prerequisites & UV Setup
14-
15-
1. **Install `uv`** (if not already installed):
16-
- **Linux/macOS**: `curl -LsSf https://astral.sh/uv/install.sh | sh`
17-
- **Windows (PowerShell)**: `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
18-
- **Pip**: `pip install uv`
19-
20-
2. **Sync Dependencies**:
21-
Inside this directory, initialize the environment and install dependencies (including building local `deglib` bindings):
22-
```bash
23-
uv sync
24-
```
13+
## Prerequisites: Building the Python Library (`deglib`)
14+
15+
The `deglib` Python library contains C++ pybind11 bindings (`deglib_cpp`).
16+
17+
Since `pyproject.toml` references `deglib` locally (`path = "../../python"`), running `uv sync` automatically copies the C++ sources from `cpp/`, invokes CMake, and compiles/installs the latest `deglib` C++ bindings into the local environment:
18+
19+
```bash
20+
# Install dependencies & compile latest deglib C++ bindings
21+
uv sync
22+
```
23+
24+
> **Note:** A C++ compiler (MSVC on Windows, GCC/Clang on Linux/macOS) and CMake must be available in your system `PATH`. `setup.py` handles copying C++ files, running CMake, and compiling the extension during the `uv sync` step.
2525
2626
## Running the Benchmark
2727

python/setup.py

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -50,26 +50,16 @@ def finalize_options(self):
5050
pass
5151

5252
def run(self):
53+
ignore_dirs = shutil.ignore_patterns('external', 'cmake-build*', 'build', 'benchmark', '.venv', '.git*')
5354
copy_dirs = [
5455
(os.path.join('..', 'cpp'), 'lib')
5556
]
5657

5758
for src, dst in copy_dirs:
5859
print(f"Copying {src} to {dst}")
59-
shutil.copytree(src, dst, dirs_exist_ok=True)
60+
shutil.copytree(src, dst, dirs_exist_ok=True, ignore=ignore_dirs)
6061
print("Files copied successfully.")
6162

62-
rm_dirs = [
63-
os.path.join('lib', 'external'),
64-
os.path.join('lib', 'cmake-build-default'),
65-
os.path.join('lib', 'build'),
66-
os.path.join('lib', 'benchmark'),
67-
]
68-
for rm_dir in rm_dirs:
69-
if os.path.isdir(rm_dir):
70-
print(f'Removing "{rm_dir}"')
71-
shutil.rmtree(rm_dir)
72-
7363

7464
class CopySDist(sdist_class):
7565
def run(self):

0 commit comments

Comments
 (0)