Skip to content

metadata-service.mdx

Latest commit

 

History

History
275 lines (210 loc) · 10.1 KB

metadata-service.mdx

File metadata and controls

275 lines (210 loc) · 10.1 KB
Raw
title Metadata Service
description The unified engine for loading, saving, and managing configuration metadata across the platform

import { Database, FileCode, Layers, RefreshCw, Save } from 'lucide-react';

The Metadata Service is the backbone of ObjectStack configuration management. It decouples the definition of metadata (Objects, Views, Flows) from its runtime delivery path: local development reads files or dist/objectstack.json, while production ObjectStack reads an immutable artifact from the control plane. Project databases store business rows only.

Architecture

The Metadata Service uses a Provider/Loader architecture. The core MetadataManager acts as a registry and orchestrator, delegating actual I/O operations to registered Loaders.

} title="Registry Pattern" description="The Manager accepts any number of Loaders. Runtime ObjectStack uses file/artifact loaders; control-plane services can opt into database loaders explicitly." /> } title="Format Agnostic" description="Automatically handles JSON, YAML, and TypeScript formats. Developers write TS, Machines write JSON." /> } title="Hot Reload" description="Built-in file watching. Edit a file in VS Code, and the runtime updates immediately." /> } title="Atomic Persistence" description="Safe save API with backup generation and atomic writes to prevent data corruption." />

Core Concepts

1. The Manager

The MetadataManager is the single entry point for all metadata operations. It handles caching, serialization, and correct routing of requests to loaders.

import { NodeMetadataManager } from '@objectstack/metadata/node';

// NodeMetadataManager wires up the FilesystemLoader and file watcher.
// The browser-safe base `MetadataManager` (from '@objectstack/metadata')
// registers no loaders on its own — callers must provide them.
const manager = new NodeMetadataManager({
  rootDir: './src',
  watch: true // Enable hot reload
});

// Load an Object Definition
const accountObj = await manager.load('object', 'account');

2. Loaders (Data Sources)

Loaders adapt different storage backends to a common interface.

Loader Type Description
FilesystemLoader filesystem Reads/Writes .json, .yaml, .ts files from disk. Primary for development.
MemoryLoader memory Hydrates compiled artifact items into the runtime metadata registry.
DatabaseLoader database Stores metadata revisions in control-plane services that explicitly configure it. Not auto-enabled by ObjectStack.
RemoteLoader http Fetches metadata from a remote HTTP API.

3. Granular Capabilities

Each Loader declares its capabilities via a Contract:

{/* os:check */}

export interface MetadataLoaderContract {
  name: string;
  protocol: 'file:' | 'http:' | 's3:' | 'datasource:' | 'memory:';
  capabilities: {
    read: boolean;  // Can read items?
    write: boolean; // Can save modifications?
    watch: boolean; // Can push real-time updates?
    list: boolean;  // Can enumerate items?
  };
  supportedFormats: ('json' | 'yaml' | 'typescript' | 'javascript')[];
}

Internal Persistence API

The Metadata Service supports full CRUD operations for authoring tools and control-plane services. Runtime ObjectStack treats metadata as read-only after artifact/file hydration; MetadataPlugin does not automatically bridge ObjectQL to DatabaseLoader.

Saving Metadata (Node.js)

// 1. Simple Save (Auto-detects loader)
await manager.save('object', 'customer_profile', data);

// 2. Targeted Save (Specific Source)
await manager.save('object', 'customer_profile', data, {
  loader: 'filesystem',
  format: 'yaml', // Convert to YAML on save
  atomic: true    // Use write-rename strategy for safety
});

// 3. Create Backup
await manager.save('object', 'workflow_config', data, {
  backup: true // Creates 'workflow_config.json.bak' before writing
});

Loading Strategy

When load() is called, the Manager iterates through registered loaders. In ObjectStack runtime boot this means "artifact/file first"; database-backed metadata is reserved for explicitly configured control-plane services.

graph LR
    A[Request: load('object', 'user')] --> B{Metadata Manager}
    B --> C[Check Artifact / Memory Loader]
    C -- Not Found --> D[Check Filesystem Loader]
    D -- Found --> E[Return & Cache]
    C -- Found --> E
Loading

Watching & Hot Reload

The Service emits events when metadata changes, allowing the runtime to reconfigure itself without restarting.

const unsubscribe = manager.subscribe('object', (event) => {
  console.log(`Object ${event.name} changed!`);
  console.log(`New Schema:`, event.data);
  
  // Trigger system update
  SchemaEngine.refresh(event.name);
});

Remote API Access

ObjectStack metadata is artifact-backed at runtime. Read/write metadata APIs belong to authoring tools and the control plane.

The system provides standard REST and TypeScript APIs for interacting with metadata, including full read/write support.

REST API

  • GET /api/v1/meta: List all metadata types.
  • GET /api/v1/meta/:type: List items of a type.
  • GET /api/v1/meta/:type/:name: Get the definition of an item.
  • PUT /api/v1/meta/:type/:name: Save or update a metadata item.

Client SDK

// Save metadata from the browser
const result = await client.meta.saveItem('object', 'customer', {
  label: 'Customer',
  fields: {
    name: { type: 'text', required: true }
  }
});

Server-Side Protocol

// Access from Protocol Service (Server-side)
const protocol = kernel.getService('protocol');
await protocol.saveMetaItem({ 
  type: 'object', 
  name: 'lead', 
  item: { ... } 
});

Runtime Controls (v3.1+)

Four orthogonal levers shape how MetadataManager and MetadataPlugin behave at boot and at request time. All four default to "dev-friendly" so existing code keeps working.

1. Bootstrap Modes

MetadataPluginConfigSchema.bootstrap decides what the plugin does on start():

Mode When to use Behavior
eager (default) Local dev, traditional servers Scans filesystem (or hydrates from artifactSource) at boot.
lazy Cold-start sensitive runtimes, on-demand workloads Skips the FS prime; reads flow through registered loaders (incl. DatabaseLoader cache). Honors artifactSource when set.
artifact-only Edge / serverless / read-only production Refuses to touch the filesystem. Requires artifactSource (mode: 'local-file'; path may be an http(s) URL). 
new MetadataPlugin({
  config: { bootstrap: 'artifact-only' },
  artifactSource: { mode: 'local-file', path: './dist/objectstack.json' },
});

Artifact source

artifactSource: { mode: 'local-file', path, fetchTimeoutMs? } is the single artifact source, honored by every bootstrap mode above. path is a filesystem path or an http(s) URL fetched verbatim — the control plane's public artifact route (/pub/v1/environments/:id/artifact[?commit=<id>]) serves exactly such URLs, so a sealed runtime can boot straight off a published revision:

new MetadataPlugin({
  config: { bootstrap: 'artifact-only' },
  artifactSource: {
    mode: 'local-file',
    path: 'https://cloud.example.com/pub/v1/environments/env_42/artifact?commit=cmt_1a2b',
  },
});

Only non-URL paths get the artifact-file HMR watcher (artifactWatch); remote reads honor fetchTimeoutMs (and OS_ARTIFACT_FETCH_TIMEOUT_MS, default 60 s). Outside artifact-only, an absent local file is "nothing compiled yet", not a fault.

A second artifact-api mode (a Bearer-authenticated control-plane pull) was removed while resolving #4246: it had no consumers — the cloud runtime uses its own ArtifactApiClient, and package distribution into a running OSS instance goes through @objectstack/cloud-connection (os package install). A still-configured artifact-api source fails loudly at start() rather than silently falling back to the filesystem scan.

2. Persistence Write Gate

MetadataManagerConfigSchema.persistence is a runtime freeze. The flag defaults to true.

Flag Effect when false
persistence.writable register() becomes a no-op (or throws when validation.throwOnError). 
new MetadataManager({
  persistence: { writable: false },
});

(persistence.overlayWritable was removed in #13135 with the paper metadata-customization protocol — the saveOverlay() it gated was never reachable from any served surface. Authoring it is now a compile-time and parse-time error carrying the prescription.)

3. DatabaseLoader Read-Through Cache

DatabaseLoader wraps load / loadMany / list / stat results in a generic LRU cache (lazy TTL, promote-on-get, write invalidation). Reads always observe writes performed through the same loader instance; out-of-band SQL writes are honored within ttl milliseconds.

new MetadataManager({
  datasource: 'default',
  cache: {
    enabled: true,
    databaseLoader: { enabled: true, maxSize: 500, ttl: 60_000 },
  },
});

The cache exposes LRUCache.stats() (size / hits / misses / hitRate) for metrics.

4. Single-Source Schema Discipline

The canonical MetadataManagerConfigSchema and MetadataFallbackStrategySchema live in kernel/metadata-loader.zod.ts. The persistence-side module system/metadata-persistence.zod.ts re-exports them, so consumers observe a single TypeScript type regardless of import path. Drift between kernel-side runtime config and persistence-side typing is no longer possible.

CLI & Tooling Integration

The Metadata Service is designed to power the CLI.

  • os compile: Validates TypeScript metadata and emits dist/objectstack.json.
  • os package publish: Uploads the compiled dist/objectstack.json artifact to the control plane as a versioned package.
  • os dev: Boots ObjectStack from local files or the compiled artifact without a control-plane connection.