Columnar analytics and graph traversal in one fused pipeline.
Rayforce is a pure C zero-dependency embeddable engine where columnar analytics and graph traversals share a single operation DAG, pass through a multi-pass optimizer, and execute as morsel-driven bytecode compiled at execution time. No malloc.
Homebrew (macOS & Linux):
brew install rayforcedb/tap/rayforceDebian / Ubuntu (.deb, x86-64) — grab the .deb from the
latest release:
curl -LO https://github.com/RayforceDB/rayforce/releases/download/vX.Y.Z/rayforce_X.Y.Z_amd64.deb
sudo dpkg -i rayforce_X.Y.Z_amd64.debPrebuilt tarball (Linux x86-64 / macOS arm64) is also attached to each
release. Or build from source below — zero dependencies, just
make.
make # debug build (ASan + UBSan)
make release # optimized build
make test # run full test suite
./rayforce # start the Rayfall REPLmake compiles with clang when CC is not set; make CC=gcc (or CC=gcc
in the environment) builds with gcc.
Rayforce ships with Rayfall — a Lisp-like query language with a rich set
of builtins. The REPL prompt is ‣:
‣ (set t (table [Symbol Side Qty]
(list [AAPL GOOG MSFT AAPL GOOG]
[Buy Sell Buy Sell Buy]
[100 200 150 300 250])))
‣ (select {from:t by: Symbol Qty: (sum Qty)})
+--------+----------------------------+
| Symbol | Qty |
| SYM | I64 |
+--------+----------------------------+
| AAPL | 400 |
| GOOG | 450 |
| MSFT | 150 |
+-------------------------------------+
| 3 rows (3 shown) 2 columns (2 shown)|
+-------------------------------------+
‣ (pivot t 'Symbol 'Side 'Qty sum)
+--------+-----+----------------------+
| Symbol | Buy | Sell |
| SYM | I64 | I64 |
+--------+-----+----------------------+
| AAPL | 100 | 300 |
| GOOG | 250 | 200 |
| MSFT | 150 | 0Nl |
+-------------------------------------+
| 3 rows (3 shown) 3 columns (3 shown)|
+-------------------------------------+
Headers: include/rayforce.h (types, memory, atoms,
vectors, tables, symbols), src/ops/ops.h (DAG construction, opcodes,
optimizer, executor, graph algorithms), src/mem/heap.h (allocator lifecycle).
#include <rayforce.h>
#include "mem/heap.h"
#include "ops/ops.h"
int main(void) {
ray_heap_init();
ray_sym_init();
/* Build a table */
int64_t regions[] = {0, 0, 1, 1, 2, 2};
int64_t amounts[] = {100, 200, 150, 300, 175, 225};
ray_t* reg = ray_vec_from_raw(RAY_I64, regions, 6);
ray_t* amt = ray_vec_from_raw(RAY_I64, amounts, 6);
ray_t* tbl = ray_table_new(2);
tbl = ray_table_add_col(tbl, ray_sym_intern("region", 6), reg);
tbl = ray_table_add_col(tbl, ray_sym_intern("amount", 6), amt);
ray_release(reg); ray_release(amt);
/* Group by region, sum amounts */
ray_graph_t* g = ray_graph_new(tbl);
ray_op_t* keys[] = { ray_scan(g, "region") };
uint16_t agg_ops[] = { OP_SUM };
ray_op_t* agg_ins[] = { ray_scan(g, "amount") };
ray_op_t* grp = ray_group(g, keys, 1, agg_ops, agg_ins, 1);
ray_t* result = ray_execute(g, grp);
if (result && !RAY_IS_ERR(result)) ray_release(result);
ray_graph_free(g);
ray_release(tbl);
ray_sym_destroy();
ray_heap_destroy();
}make lib builds the engine as a static library, librayforce.a, with no
dependencies beyond libc, libm and pthreads. The build compiles library
objects with -ffunction-sections -fdata-sections (the Makefile's
LIB_CFLAGS flavour, applied automatically to the .lib.o objects that go
into the archive), so no extra flags are needed at this step.
To discard everything your code never reaches, pass -Wl,--gc-sections to
the linker on the final link, and -s to strip the result. A DAG-route
client, meaning one that builds and executes query plans through the
ray_graph_*, ray_scan and ray_execute C API and never calls
ray_eval_str or loads .rfl files, never touches the Rayfall parser,
compiler, REPL or IPC server, and section GC drops that whole surface
(mixing the two routes pulls the parser back in):
make clean && make lib
cc -O2 -Iinclude -Isrc examples/analytics.c librayforce.a \
-lm -lpthread -Wl,--gc-sections -s -o analyticsThe command above is the Linux one. On macOS, use -Wl,-dead_strip in
place of -Wl,--gc-sections (the Makefile's platform block does the same
for its own release links). On MinGW, link the winsock libraries as well:
-lws2_32 -lmswsock -lkernel32 -ladvapi32, with the same section-GC flag
the toolchain accepts.
make install-headers copies the headers a C client needs into a staging
root, so a tree staged from one machine can be carried to a build host
that has no checkout. The staged set is include/rayforce.h (public)
plus the internals that bindings include directly - src/ops/ops.h,
src/io/parquet.h, src/mem/heap.h - and the two those internals reach
through their own quoted includes, src/store/hnsw.h (from ops.h) and
src/core/platform.h (from heap.h). They are copies rather than
symlinks, staged read-only, mirroring the source layout: the internals
land under install/include/rayforce/, and rayforce.h is staged
twice - inside that tree and at the stage root
(install/include/rayforce.h), which is the copy that lets
-Iinstall/include resolve <rayforce.h>. DESTDIR and
INSTALL_INCLUDE move the stage root, and make clean removes what the
same overrides staged.
The staged layout needs two include roots: the internals include
their siblings by quoted, src/-relative path, so -Iinstall/include
alone does not resolve "ops/ops.h":
cc -O2 -Iinstall/include -Iinstall/include/rayforce/src \
examples/analytics.c librayforce.a -lm -lpthread -o analyticsThe target fails loudly: any failed copy aborts it, so a run that returns success has left a complete tree, and one that cannot stage says so rather than reporting success over a partial one.
Staging is a convenience for building against one engine revision, not
an SDK contract. What it promises is the header contents at the commit
that staged them - no ABI stability, and no API stability either. The
staged set carries internal types and layouts that can change between
revisions, so stage from the same commit you built librayforce.a
from.
A checkout at the current commit links analytics.c to a 3.47 MB
stripped binary, from a 10.22 MB archive; sizes move with the tree, so
this is just one measurement, not a bound.
Memory follows consumed-argument conventions documented per function in
the headers: arguments marked consumes are released by the call, and
returns marked borrowed stay owned by the container. On error paths the
library releases the references it holds. Values are COW, managed with
ray_retain/ray_release; error objects are owned and freed with
ray_error_free.
Build — Construct a lazy DAG: scans, filters, joins, aggregations, window functions, graph traversals. Nothing executes yet.
Optimize — Multi-pass rewriting: type inference → constant folding → idiom rewrite → SIP → factorize → predicate pushdown → filter reorder → projection pushdown → partition pruning → DCE.
Execute — Element-wise subtrees are compiled to flat bytecode at execution time and evaluated over 1024-element morsels that stay L1-resident, with no intermediate heap allocation. Radix-partitioned hash joins size partitions to fit L2. Thread pool dispatches morsels in parallel.
Execution engine
- Lazy operation DAG — nothing runs until
ray_execute - Multi-pass optimizer with sideways information passing
- Expression compilation at execution time — element-wise subtrees compiled to flat bytecode, evaluated in 1024-row morsel batches with no intermediate materialization
- Radix-partitioned hash joins sized for L2 cache
- Thread pool with parallel morsel dispatch
Graph engine
- Double-indexed CSR storage (forward + reverse), mmap support
- BFS, DFS, Dijkstra, A*, PageRank, Louvain, Betweenness, LFTJ, and more
- Factorized execution avoids materializing cross-products
- SIP propagates selection bitmaps backward through expand chains
Rayfall language
- Arithmetic, string, aggregation, joins, higher-order, I/O builtins
- Lambdas compile lazily to bytecode, run in computed-goto VM
select/update/pivotbridge to the DAG optimizer at runtime
Memory
- Buddy allocator with slab cache — O(1) for ~90% of allocations
- Thread-local arenas, lock-free allocation, COW ref counting
- No system allocator —
ray_alloc/ray_freefor everything
Vector search
- Multi-metric HNSW index (cosine / L2 / inner-product) with save/load
- Rayfall builtins:
cos-dist/l2-dist/inner-prod/norm/knnand the HNSW lifecyclehnsw-build/ann/hnsw-save/hnsw-load/hnsw-free/hnsw-info - Filter-aware ANN via
select ... where ... nearest (ann handle query) take k - Iterative streaming scan: the
wherepredicate is pushed into HNSW's beam loop so rejected candidates don't consume result slots
Storage
- Columnar files with mmap, splayed tables, date-partitioned tables
- CSV reader with parallel mmap parse, type inference, null handling
include/rayforce.h Single public header
src/mem/ Buddy allocator, slab cache, arena, COW
src/core/ Type system, platform abstraction, runtime
src/vec/ Vector, list, string, selection bitmap ops
src/table/ Table, symbol intern table
src/store/ Column files, CSR, splayed/parted tables, HNSW
src/ops/ DAG, optimizer, fused executor, LFTJ
src/io/ CSV reader/writer (parallel mmap)
src/lang/ Rayfall parser, evaluator, bytecode VM
src/app/ REPL, terminal, pretty-printer
test/ Test suites
examples/rfl/ Rayfall example scripts
examples/ C API examples
website/ Documentation site (GitHub Pages)
Full docs: rayforcedb.github.io/rayforce
- Quick Start — build, REPL, first query
- Rayfall Language — syntax and builtins
- Data Types — types and collections
- Queries — select, joins, pivot, window
- C API — full API reference
- Graph Engine — algorithms
- Architecture — DAG, optimizer, memory
Rayforce has Python bindings at rayforce-py — contributions welcome.
Contributions are welcome. You can help by:
- Reporting bugs and requesting features via GitHub Issues
- Submitting pull requests
- Creating example scripts and use cases
- Improving documentation
Rayforce is jointly developed with and sponsored by Lynx.
This partnership has been instrumental in making Rayforce a mature, production-ready engine. Lynx's active involvement in development and their commitment to innovative open-source technologies in the financial sector has enabled Rayforce to reach its full potential.