Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -382,13 +382,25 @@ jobs:
- name: Unit tests
run: cargo test --manifest-path crates/tinymemory-module/Cargo.toml --lib

- name: Linked module exports
run: |
cargo clippy --locked --manifest-path crates/tinymemory-module/Cargo.toml \
--all-targets --features static-link -- -D warnings
cargo test --locked --manifest-path crates/tinymemory-module/Cargo.toml \
Comment thread
senamakel marked this conversation as resolved.
Comment thread
senamakel marked this conversation as resolved.
Comment thread
senamakel marked this conversation as resolved.
Comment thread
senamakel marked this conversation as resolved.
--features static-link --test static_link

# The module is excluded from the root workspace, so the root coverage
# gate cannot see it. Hold its production code to the same floor here.
# gate cannot see it. Keep both profiles in one coverage report: default
# mode runs the dynamic loader E2E, while all-features runs the linked
# host test. The loader cannot open a static-link artifact, so these must
# be separate test passes over the same coverage target.
- name: Enforce module production-source coverage
run: |
set -euo pipefail
cargo llvm-cov --manifest-path crates/tinymemory-module/Cargo.toml \
--workspace --all-features \
--workspace --no-report
cargo llvm-cov --manifest-path crates/tinymemory-module/Cargo.toml \
--workspace --all-features --no-clean \
--ignore-filename-regex '(^|/)(tests|vendor)/|(^|/)(test|tests|test_helpers|test_support|test_seams)\.rs$|(_test|_tests|_test_support)\.rs$|/crates/tinymemory-core/src/(engine/parity|tree/retrieval/benchmarks)\.rs$|/crates/tinymemory-conformance/src/reference/full\.rs$' \
--fail-under-lines 80 --summary-only \
| tee "$GITHUB_STEP_SUMMARY"
Expand Down
6 changes: 3 additions & 3 deletions crates/tinymemory-module/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 9 additions & 5 deletions crates/tinymemory-module/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ publish = false
# without going through the loader; the `cdylib` is what a release ships.
crate-type = ["rlib", "cdylib"]

[features]
# Link this module into a host and use Rust-addressable ABI entry points.
# The default keeps the existing loadable cdylib exports.
static-link = []

[dependencies]
# The contract. Every type crossing the bus is one of these, and all of them
# already carry serde impls — which is why this module needs no `wire` module of
Expand Down Expand Up @@ -123,12 +128,11 @@ tinyinference-core = { path = "../../vendor/tinyinference/crates/tinyinference-c
tinyinference-embeddings = { path = "../../vendor/tinyinference/crates/tinyinference-embeddings" }
tinyinference-llm = { path = "../../vendor/tinyinference/crates/tinyinference-llm" }

# Mirrors the root package's set. `unsafe_code = "forbid"` holds even though
# `module_export!` emits `unsafe extern "C"` symbols: the macro's expansion
# carries its own hygiene context, so the C ABI surface does not trip the lint
# here. Verified against `tinywallet-module`, which forbids it too and builds.
# The linked-host integration test must call TinyBus's unsafe `attach_raw` ABI
# and borrow the generated manifest bytes. Keep unsafe denied everywhere else;
# the test documents its narrow lifetime and pointer invariants at each call.
[lints.rust]
unsafe_code = "forbid"
unsafe_code = "deny"
missing_docs = "warn"
missing_debug_implementations = "warn"
unreachable_pub = "warn"
Expand Down
23 changes: 17 additions & 6 deletions crates/tinymemory-module/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
//! Loadable `TinyBus` module adapter for `TinyMemory`.
//!
//! This private workspace crate keeps the vendored `TinyBus` dependency out of
//! the published `tinymemory` crates. Its `cdylib` output is the
//! target-specific binary distributed in GitHub releases.
//! the published `tinymemory` crates. Its default `cdylib` output is the
//! target-specific binary distributed in GitHub releases. With `static-link`,
//! a host can instead reference the descriptor, manifest, and initializer by
//! Rust path without colliding with another module's C symbols.
//!
//! # What this module is for, stated honestly
//!
Expand Down Expand Up @@ -83,6 +85,10 @@ pub use embedding::{
pub use host::{RUNTIME_HOST_BUS_NAME, RUNTIME_HOST_INTERFACE, RUNTIME_HOST_OBJECT_PATH};
pub use service::{BUS_NAME, OBJECT_PATH};

/// Rust-addressable TinyBus ABI entries for an in-process linked host.
Comment thread
senamakel marked this conversation as resolved.
#[cfg(feature = "static-link")]
pub use exports::{tinybus_module_init_v1, tinybus_module_manifest_v1, TINYBUS_MODULE_ABI_V1};

use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, OnceLock};
Expand Down Expand Up @@ -498,16 +504,21 @@ fn setup_error(message: impl Into<String>) -> BusError {
}
}

// Isolate the generated public C symbols so the lint exception cannot hide
// undocumented Rust API. Their contract is TinyBus ABI v1, and none is a
// Rust-callable export from this crate.
// Isolate the generated ABI symbols so the lint exception cannot hide
// undocumented Rust API. The static-link feature exports these by Rust path;
// the default gives them the established dynamic C symbol names.
#[allow(
missing_docs,
unreachable_pub,
reason = "generated C ABI symbols are documented by the TinyBus module SDK"
)]
mod exports {
tinybus_module::module_export! {
#[cfg(not(feature = "static-link"))]
use tinybus_module::module_export as export_module;
#[cfg(feature = "static-link")]
use tinybus_module::module_export_static as export_module;

export_module! {
setup = super::setup,
config = super::ModuleConfig,
// Eight, derived rather than picked. Two are the floor this module has
Expand Down
3 changes: 3 additions & 0 deletions crates/tinymemory-module/tests/module_e2e.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
//! `--ignored` alone runs them all in one process and the second will hang. This
//! is the same constraint the `tinywallet` module's loader tests carry.

// All-features coverage builds the linked mode, whose ABI entries are Rust
// symbols. This suite loads the default cdylib by its C symbols instead.
#![cfg(not(feature = "static-link"))]
Comment thread
senamakel marked this conversation as resolved.
#![allow(
clippy::expect_used,
clippy::unwrap_used,
Expand Down
63 changes: 63 additions & 0 deletions crates/tinymemory-module/tests/static_link.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
//! Public linked-module entry points remain callable by a Rust host.

#![cfg(feature = "static-link")]
Comment thread
senamakel marked this conversation as resolved.

use tinybus::broker::Broker;
use tinybus::module::abi::{TbModuleInit, TbSlice, ABI_MAGIC};
use tinybus::module::manifest::ModuleManifest;
use tinybus::module::ModuleHost;
use tinybus::transport::memory::MemoryBus;
use tinybus::Connection;
use tinymemory_module::{
tinybus_module_init_v1, tinybus_module_manifest_v1, BUS_NAME, OBJECT_PATH,
TINYBUS_MODULE_ABI_V1,
};

#[test]
fn linked_module_exposes_its_descriptor_manifest_and_initializer() {
assert_eq!(TINYBUS_MODULE_ABI_V1.magic, ABI_MAGIC);
let manifest: extern "C" fn() -> TbSlice = tinybus_module_manifest_v1;
let slice = manifest();
assert!(!slice.ptr.is_null());
assert!(slice.len > 0);
let initialize: TbModuleInit = tinybus_module_init_v1;
assert_ne!(initialize as usize, 0);
}

#[allow(unsafe_code)]
Comment thread
senamakel marked this conversation as resolved.
#[tokio::test]
async fn linked_module_serves_memory_calls_through_tinybus(
) -> Result<(), Box<dyn std::error::Error>> {
let workspace = tempfile::tempdir()?;
let bus = MemoryBus::new();
let broker = Broker::new();
let _broker_task = broker.spawn(bus.clone());
let host = ModuleHost::new(broker);

let slice = tinybus_module_manifest_v1();
// SAFETY: the generated manifest owns its bytes in a process-lifetime
// OnceLock; its non-null pointer and length remain valid during parsing.
let bytes = unsafe { std::slice::from_raw_parts(slice.ptr, slice.len) };
let manifest: ModuleManifest = serde_json::from_slice(bytes)?;
let config = serde_json::json!({ "workspace_dir": workspace.path() });

// SAFETY: these three entries come from the linked module in this process.
// They remain mapped and callable until exit, including all callbacks the
// host retains after initialization.
let loaded = unsafe {
host.attach_raw_with_config(
Comment thread
senamakel marked this conversation as resolved.
Comment thread
senamakel marked this conversation as resolved.
"linked-tinymemory",
TINYBUS_MODULE_ABI_V1,
manifest,
tinybus_module_init_v1,
config,
)
}?;
assert_eq!(loaded.manifest.bus_name.as_str(), BUS_NAME);

let client = Connection::connect(bus.connect().await?).await?;
let proxy = client.proxy(BUS_NAME, OBJECT_PATH, "ai.tinyhumans.tinymemory.Memory")?;
let driver_id: String = proxy.call("DriverId", ()).await?;
assert_eq!(driver_id, "tinycortex");
Ok(())
}
8 changes: 8 additions & 0 deletions docs/specs/tinybus-module.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
`crates/tinymemory-module` is a `cdylib` speaking the TinyBus module ABI. A host
loads it and gets a bound memory driver without compiling the engine.

The default build exports the TinyBus v1 C symbols for dynamic loading. A host
that compiles the module into its own executable can enable the module crate's
`static-link` feature and pass `tinymemory_module::TINYBUS_MODULE_ABI_V1`,
`tinymemory_module::tinybus_module_manifest_v1`, and
`tinymemory_module::tinybus_module_init_v1` to the linked TinyBus module host.
Both modes use the same manifest declaration; the linked entry points have no
unmangled global C symbol names.

## What it buys, and what it does not

**It sheds no dependencies.** This is measured, not assumed, and it is stated
Expand Down
Loading