Skip to content

Commit 1e8545b

Browse files
kerthcetclaude
andauthored
docs: bring the docs in line with the code (#44)
* docs: bring the docs in line with the code - DEVELOP.md: fix the project layout (protocol/ crate, python.rs, snapshot/), drop references to files that do not exist (test_build.sh, simple_test.py, agent_example.py, protocol.rs, STATUS.md), and document the real make targets, test markers and release flow. - QUICKSTART.md: use make targets, and list_daemons() returning DaemonInfo. - ARCHITECTURE.md: stop claiming daemons authenticate; describe the planned Keycloak-based auth instead. - README.md: file upload is not implemented on the daemon; list snapshots and labels/broadcast; add auth and file upload to the roadmap. - PROTOCOL.md: point at protocol/src/lib.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kerthcet <kerthcet@gmail.com> * docs: mark file transfer as not supported yet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kerthcet <kerthcet@gmail.com> * docs: add a Security section to the README State plainly that daemons are not authenticated yet, what direct and tunnel mode do and do not protect, and that the tunnel authkey is a Headscale key rather than a SandD credential. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kerthcet <kerthcet@gmail.com> * docs: drop the CPython linker troubleshooting entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kerthcet <kerthcet@gmail.com> --------- Signed-off-by: kerthcet <kerthcet@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 96c97b2 commit 1e8545b

5 files changed

Lines changed: 125 additions & 138 deletions

File tree

‎README.md‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,12 @@ Rust-powered WebSocket server with Python API for remote command execution and i
1818

1919
- **Command Execution** - Run shell commands on remote machines with timeout control
2020
- **Interactive Sessions** - Full PTY sessions with bash for manual work
21-
- **File Transfer** - Upload/download files between controller and workers
21+
- **Snapshots** - Save, tag and restore workspace state on a worker
22+
- **Labels & Broadcast** - Filter workers by label and run a command on many at once
2223
- **Tunnel Mode (VPN)** - Secure mesh networking with WireGuard encryption via Tailscale
2324
- **High Performance** - Rust async runtime handles high-concurrency workloads
2425
- **Auto Reconnection** - Workers reconnect automatically on network failures
25-
- **Cross-Platform** - Linux, macOS, Windows support
26+
- **Cross-Platform** - Linux and macOS release binaries; Windows builds from source
2627

2728
## Architecture
2829

@@ -171,6 +172,20 @@ server = Server(connect="tunnel", tunnel_config=tunnel_config_from_env())
171172

172173
See [Tunnel Mode Guide](./docs/proposals/TUNNEL.md) for setup instructions.
173174

175+
## Security
176+
177+
SandD does not authenticate daemons yet. The controller accepts any client that
178+
speaks the `sandd.v1` protocol, and a daemon can register under any id, so anyone who
179+
can reach the controller's port can join and receive commands meant for that id.
180+
181+
- **Direct mode:** run the controller only on a trusted network.
182+
- **Tunnel mode:** only nodes admitted to your Headscale mesh can reach the
183+
controller, and traffic is encrypted with WireGuard. The `authkey` is a Headscale
184+
key for joining the mesh, not a SandD credential.
185+
186+
Daemon authentication and authorization with Keycloak is planned (see
187+
[Architecture](./docs/ARCHITECTURE.md#future-authentication)).
188+
174189
## Documentation
175190

176191
- [Quick Start Guide](./docs/QUICKSTART.md)
@@ -182,6 +197,8 @@ See [Tunnel Mode Guide](./docs/proposals/TUNNEL.md) for setup instructions.
182197

183198
## Roadmap
184199

200+
- [ ] **Authentication & Authorization** - Keycloak-issued tokens for daemons (see [Security](#security))
201+
- [ ] **File Transfer** - Upload and download files between controller and workers
185202
- [ ] **Audit Logging** - Track all commands, sessions, and file transfers
186203
- [ ] **Metrics** - Prometheus-compatible metrics for monitoring
187204
- [ ] **Resource Limits** - CPU/memory/timeout controls per daemon

‎docs/ARCHITECTURE.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
- ✅ No ports exposed on execution plane (security)
1717
- ✅ Works through NAT/firewalls
1818
- ✅ Easy to add/remove daemons dynamically
19-
- ✅ Agent controls access (daemon authenticates to agent)
19+
- ✅ Agent decides what runs (daemons only execute what the agent sends)
2020

2121
### Why Rust Server?
2222

@@ -56,12 +56,15 @@ Forward raw SSH traffic through WebSocket to daemon's sshd.
5656

5757
### Future: Authentication
5858

59-
```rust
60-
Message::Register {
61-
daemon_id: String,
62-
auth_token: String, // JWT or pre-shared key
63-
metadata: DaemonMetadata,
64-
}
65-
```
59+
Today the controller admits any client that speaks `sandd.v1`, and a daemon can
60+
register under any id. Run it on a trusted network, or in tunnel mode, where only mesh
61+
members can reach it.
62+
63+
The plan is to use Keycloak for authentication and authorization:
6664

67-
Agent validates token before accepting connection.
65+
- The daemon gets an access token from Keycloak and sends it as
66+
`Authorization: Bearer <token>` on the WebSocket upgrade.
67+
- The controller verifies the token against Keycloak's JWKS endpoint before upgrading,
68+
and rejects a bad one with HTTP 401.
69+
- The token's claims decide which daemon id the connection registers as, and what it
70+
is allowed to do.

‎docs/DEVELOP.md‎

Lines changed: 80 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -6,98 +6,90 @@ This guide is for developers working on SandD itself.
66

77
- **Rust**: Install from https://rustup.rs/
88
- **Python 3.8+**: With pip
9-
- **Maturin**: `pip install maturin`
9+
- **Docker**: Only for the e2e tests and images
10+
11+
The Makefile creates `.venv/` and installs maturin, pytest and ruff into it as needed.
1012

1113
## Project Structure
1214

1315
```
1416
SandD/
15-
├── server/ # Rust server with PyO3 bindings
17+
├── protocol/ # Shared message types (sandd-protocol crate)
18+
│ └── src/lib.rs
19+
│
20+
├── server/ # Controller: Rust WebSocket server + PyO3 bindings
1621
│ ├── src/
17-
│ │ ├── lib.rs # Python API bindings
18-
│ │ ├── server.rs # WebSocket server (axum)
22+
│ │ ├── lib.rs # Crate root
23+
│ │ ├── server.rs # WebSocket server (axum): /ws, /stats, /health
1924
│ │ ├── registry.rs # Daemon connection registry
20-
│ │ └── protocol.rs # Message protocol
25+
│ │ └── python.rs # Python API bindings (behind the `python` feature)
2126
│ └── Cargo.toml
2227
│
23-
├── sandd/ # Rust daemon binary
28+
├── sandd/ # Daemon binary (runs on workers)
2429
│ ├── src/
25-
│ │ ├── main.rs # Daemon entry point
30+
│ │ ├── main.rs # Entry point, connection loop, tunnel setup
2631
│ │ ├── executor.rs # Command execution
2732
│ │ ├── session.rs # Interactive sessions (PTY)
28-
│ │ └── protocol.rs # Message protocol
33+
│ │ └── snapshot/ # Workspace snapshots
34+
│ ├── benches/ # Criterion benchmarks
2935
│ └── Cargo.toml
3036
│
31-
├── python/sandd/ # Python package wrapper
32-
│ └── __init__.py
37+
├── python/
38+
│ ├── sandd/ # Python package (Server, models, tunnel keys; AsyncServer is a stub)
39+
│ └── tests/ # Unit, integration and e2e tests
3340
│
41+
├── hack/ # Dockerfiles, compose stacks, install script
3442
└── examples/ # Usage examples
3543
```
3644

3745
## Building
3846

39-
### Quick Build (Development)
40-
4147
```bash
42-
# Build everything
43-
./test_build.sh
48+
make dev # Build and install the Python package (debug, editable)
49+
make daemon-build # Build the daemon (debug): ./target/debug/sandd
4450

45-
# Or step by step:
46-
make dev # Build Python package (debug)
47-
make daemon-build # Build daemon (debug)
48-
```
49-
50-
### Release Build
51-
52-
```bash
5351
make release # Python package (optimized)
54-
make daemon-release # Daemon binary (optimized)
52+
make daemon-release # Daemon binary (optimized): ./target/release/sandd
5553
```
5654

57-
### Manual Build
55+
Run `make help` for the full list of targets.
5856

59-
```bash
60-
# Python package
61-
maturin develop --release -m server/Cargo.toml
62-
63-
# Daemon
64-
cargo build --package sandd --release
65-
```
57+
The `python` feature of `server/` is off by default, so plain `cargo build` and
58+
`cargo test` work without a Python interpreter. Maturin turns it on when building the
59+
wheel (see `[tool.maturin]` in `pyproject.toml`).
6660

6761
## Development Workflow
6862

6963
### 1. Make Changes
7064

71-
Edit files in `server/src/`, `sandd/src/`, or `python/sandd/`
65+
Edit files in `protocol/src/`, `server/src/`, `sandd/src/`, or `python/sandd/`.
7266

7367
### 2. Rebuild
7468

7569
```bash
76-
# If you changed server/
70+
# If you changed protocol/, server/ or python/sandd/
7771
make dev
7872

79-
# If you changed daemon/
73+
# If you changed protocol/ or sandd/
8074
make daemon-build
81-
82-
# If you changed Python wrapper only
83-
# (no rebuild needed, it's just a wrapper)
8475
```
8576

86-
### 3. Test
77+
### 3. Try It
8778

8879
```bash
89-
# Terminal 1: Start test agent
90-
python3 examples/simple_test.py
91-
92-
# Terminal 2: Start daemon
93-
./target/debug/sandd --server-url ws://127.0.0.1:8765/ws
80+
# Terminal 1: start a controller
81+
python3 -c "
82+
from sandd import Server
83+
server = Server()
84+
server.wait_for_daemon('dev-1', timeout=60)
85+
print(server.exec('dev-1', 'uname -a').stdout)
86+
"
87+
88+
# Terminal 2: start a daemon
89+
./target/debug/sandd --server-url ws://127.0.0.1:8765/ws --daemon-id dev-1
9490
```
9591

96-
### 4. Run Examples
97-
98-
```bash
99-
python3 examples/agent_example.py
100-
```
92+
See [examples/](../examples) for sessions, snapshots and tunnel mode.
10193

10294
## Architecture
10395

@@ -130,39 +122,35 @@ file_transfers: Vec<Vec<u8>> // Chunked buffering
130122

131123
## Testing
132124

133-
### Unit Tests
134-
135125
```bash
136-
cargo test --workspace
126+
make test # Lint + Rust tests + Python tests (no Docker)
127+
make test-e2e # Direct-mode e2e against daemons in Docker (slow)
128+
make test-e2e-tunnel # Tunnel-mode e2e with a Headscale mesh in Docker (slow)
129+
make lint # ruff check
130+
make format # ruff fix + format
131+
make benchmark # Daemon benchmarks (criterion)
137132
```
138133

139-
### Integration Tests
140-
141-
```bash
142-
pytest tests/ # (when tests are added)
143-
```
144-
145-
### Manual Testing
146-
147-
Use `examples/agent_example.py` to test all features.
134+
Python tests are selected by marker: `e2e` for Docker-based tests, `tunnel` for the
135+
tunnel stack. `make test` runs `-m "not e2e"`.
148136

149137
## Common Tasks
150138

151-
### Adding a New Command Type
139+
### Adding a New Message Type
152140

153-
1. Add to `protocol.rs` (both server and daemon):
141+
1. Add the variant to `Message` in `protocol/src/lib.rs` (shared by server and daemon):
154142
```rust
155-
Message::MyNewCommand { field: String }
143+
MyNewCommand { field: String }
156144
```
157145

158-
2. Handle in `server/src/server.rs`:
146+
2. Handle it in `server/src/server.rs`:
159147
```rust
160148
Message::MyNewCommand { field } => {
161149
// Forward to daemon or handle
162150
}
163151
```
164152

165-
3. Handle in `sandd/src/main.rs`:
153+
3. Handle it in `sandd/src/main.rs`:
166154
```rust
167155
Message::MyNewCommand { field } => {
168156
// Execute and respond
@@ -171,7 +159,7 @@ Use `examples/agent_example.py` to test all features.
171159

172160
### Adding Python API
173161

174-
1. Add method to `Server` in `server/src/lib.rs`:
162+
1. Add a method to `Server` in `server/src/python.rs`:
175163
```rust
176164
#[pymethods]
177165
impl Server {
@@ -181,7 +169,7 @@ Use `examples/agent_example.py` to test all features.
181169
}
182170
```
183171

184-
2. Add wrapper in `python/sandd/__init__.py`:
172+
2. Add a wrapper in `python/sandd/server.py`:
185173
```python
186174
def my_method(self, arg: str) -> None:
187175
"""User-friendly docstring"""
@@ -195,30 +183,21 @@ Use `examples/agent_example.py` to test all features.
195183
RUST_LOG=debug ./target/debug/sandd --server-url ws://127.0.0.1:8765/ws
196184
```
197185

198-
**Python side:**
199-
```python
200-
import logging
201-
logging.basicConfig(level=logging.DEBUG)
202-
```
203-
204-
**Check WebSocket traffic:**
205-
```bash
206-
# In server
207-
RUST_LOG=server=debug python3 examples/simple_test.py
208-
```
209-
210-
## Known Issues & Limitations (MVP)
186+
**Controller logs:** `Server(verbose=True)` (the default) logs at INFO; set
187+
`RUST_LOG=sandbox_server=debug` for more.
211188

212-
### Not Implemented
189+
## Known Issues & Limitations
213190

214-
1. **File Transfer**: Protocol defined, daemon just logs
215-
- Reason: Deferred for MVP
216-
- Fix: Implement actual file I/O in daemon
191+
1. **File transfer is not supported yet.** `Server.upload_file` and
192+
`Server.download_file` exist, and the protocol messages are defined, but the
193+
feature is unfinished: don't rely on either call.
194+
2. **No daemon authentication.** Any client that speaks `sandd.v1` can register as any
195+
daemon id. Run the controller on a trusted network or in tunnel mode. See
196+
[ARCHITECTURE.md](ARCHITECTURE.md#future-authentication).
217197

218198
### Warnings
219199

220200
- PyO3 `non_local_definitions` warning: Safe to ignore (PyO3 macro limitation)
221-
- Unused imports/variables: Run `cargo fix` to clean up
222201

223202
## Performance
224203

@@ -232,21 +211,22 @@ At 200 concurrent daemons:
232211

233212
### Before Submitting PR
234213

235-
1. Run tests: `cargo test --workspace`
236-
2. Check formatting: `cargo fmt --all`
237-
3. Check lints: `cargo clippy --all`
238-
4. Test manually with examples
239-
5. Update docs if adding features
214+
1. Run `make test`
215+
2. Check formatting: `cargo fmt --all` and `make format`
216+
3. Check lints: `cargo clippy --workspace --all-targets`
217+
4. Update docs if adding features
240218

241219
### Commit Style
242220

243221
```
244-
Add feature: brief description
222+
feat: brief description
245223
246224
More detailed explanation if needed.
247225
Include motivation and context.
248226
```
249227

228+
Use a `feat:`, `fix:` or `chore:` prefix, as in the existing history.
229+
250230
### Adding Dependencies
251231

252232
- Keep dependencies minimal
@@ -255,23 +235,23 @@ Include motivation and context.
255235

256236
## Release Process
257237

258-
1. Update version in `pyproject.toml` and `Cargo.toml` files
259-
2. Update `CHANGELOG.md` (when added)
260-
3. Build release: `make release && make daemon-release`
261-
4. Test release build
262-
5. Tag: `git tag v0.x.0`
263-
6. Build wheel: `maturin build --release -m server/Cargo.toml`
264-
7. Publish: `maturin publish` (when ready)
238+
1. Update the version in `pyproject.toml`, `protocol/Cargo.toml`, `server/Cargo.toml`
239+
and `sandd/Cargo.toml` (and the `sandd-protocol` dependency version in
240+
`sandd/Cargo.toml`)
241+
2. Tag and push: `git tag v0.x.y && git push upstream v0.x.y`. The `Release` workflow
242+
builds the daemon binaries and creates the GitHub release.
243+
3. Publish packages: `make publish` (wheels to PyPI, needs `INFTYAI_PYPI_TOKEN`; crates
244+
to crates.io)
265245

266246
## Troubleshooting
267247

268248
### User Issues
269249

270250
**Daemon won't connect:**
271-
- Check agent URL is reachable: `curl -v ws://agent-host:8765/ws`
251+
- Check the controller is up: `curl http://controller-host:8765/health`
272252
- Verify firewall allows outbound WebSocket connections
273-
- Check agent server is running: `ps aux | grep python`
274253
- Check daemon logs: `RUST_LOG=info ./target/release/sandd ...`
254+
- Check what the controller sees: `curl http://controller-host:8765/stats`
275255

276256
**Commands timing out:**
277257
- Increase `timeout` parameter in `exec()` (in seconds)
@@ -323,5 +303,4 @@ For complete protocol specification, see [proposals/PROTOCOL.md](proposals/PROTO
323303

324304
- Check [ARCHITECTURE.md](ARCHITECTURE.md) for design details
325305
- Check [proposals/PROTOCOL.md](proposals/PROTOCOL.md) for protocol specification
326-
- Check [STATUS.md](STATUS.md) for implementation status
327306
- Check [QUICKSTART.md](QUICKSTART.md) for usage examples

0 commit comments

Comments
 (0)