diff --git a/reflexio/server/README.md b/reflexio/server/README.md index 7a34a301..5dd5d0ad 100644 --- a/reflexio/server/README.md +++ b/reflexio/server/README.md @@ -40,6 +40,7 @@ Description: FastAPI backend server that processes user interactions to generate - **Extension Registry**: `extensions.py` - Capability and service registry for optional OSS/enterprise integrations - **Core Service**: `services/generation_service.py` - Main orchestrator - **Durable Learning**: `services/durable_learning/` - background queue worker for deferred post-publish extraction +- **Scheduler infrastructure**: `scheduling.py`, `callback_executor.py`, `org_fanout.py` - shared daemon lifecycle, bounded callback dispatch, and bounded org-walking helpers ## Cache @@ -214,7 +215,7 @@ python -m reflexio.server.scripts.manage_invitation_codes list --show-used - **Evaluation**: `agent_success_evaluation/service.py`, `agent_success_evaluation/runner.py`, `agent_success_evaluation/scheduler.py`, `agent_success_evaluation/components/evaluator.py`, `shadow_comparison/`, and `evaluation_overview/` handle session grading, per-turn shadow verdicts, regeneration jobs, and dashboard-facing rollups. - **Durable learning queue**: `durable_learning/scheduler.py` and `durable_learning/worker.py` drain `learning_jobs` after deferred publishes and report coverage through `GET /api/learning_status`. - **Async clarification**: `extraction/` and `reflection/` manage resumable agent runs, pending tool calls, prior-answer search, and long-horizon reflection updates. -- **Search preparation**: `pre_retrieval/` and `unified_search_service.py` handle query reformulation, document expansion, embeddings, and cross-entity search orchestration. +- **Search preparation**: `pre_retrieval/` and `unified_search_service.py` handle query reformulation, document expansion, embeddings, vector backfill repair, and cross-entity search orchestration. - **Optimization/integrations**: `playbook_optimizer/` and `braintrust/` run candidate playbook optimization, rollout support, and Braintrust export/sync. - **Lineage**: `lineage/` resolves active records across superseded chains and schedules tombstone garbage collection for profile/playbook storage. - **Governance**: `governance/` defines subject-reference contracts and retention/barrier policy helpers used by storage and lineage paths. @@ -489,6 +490,7 @@ Key files: |------|---------| | `resolve.py` | Helpers for resolving current records across supersede chains and status transitions. | | `gc_scheduler.py` | Tombstone garbage-collection scheduler for lineage-aware storage cleanup. | +| `vector_backfill_sweep.py` | Opt-in per-org sweep (`REFLEXIO_MISSING_VECTOR_BACKFILL_ENABLED`, cap via `REFLEXIO_MISSING_VECTOR_BACKFILL_CAP`) that re-embeds interactions saved without vector rows. | **Pattern**: profile/playbook update paths preserve lineage metadata in storage; service code asks lineage helpers to find current records rather than walking superseded chains ad hoc. diff --git a/reflexio/server/services/README.md b/reflexio/server/services/README.md index 8b7502b6..6b46f3fe 100644 --- a/reflexio/server/services/README.md +++ b/reflexio/server/services/README.md @@ -24,6 +24,9 @@ strings before deleting old import paths in the same PR. |------|---------| | `generation_service.py` | `GenerationService` — saves interactions, runs/defer-runs profile + playbook generation, schedules deferred evaluation when `session_id` is present, and exposes `run_deferred_learning()` for queue workers. | | `publish_learning_worker.py` | Deferred post-persist publish learning worker — queues async publish learning after durable interaction writes and requeues under publish limiter pressure. | +| `../scheduling.py` | `ThreadedScheduler` base for process-local daemons; owns start/stop/idempotent thread lifecycle and optional leader gates. | +| `../callback_executor.py` | Process-wide bounded callback executor for debounce scheduler fires; fixed worker pool, bounded queue, drop-oldest overflow telemetry. | +| `../org_fanout.py` | Bounded per-tick org iteration helper for org-walking daemons; classifies starved vs genuinely timed-out org work. | | `base_generation_service.py` + `base_generation/` | `BaseGenerationService` stable import surface plus mixins for batch progress, config filtering, extraction lifecycle, should-run prechecks, status transitions, and usage billing. Per-extractor timeout `EXTRACTOR_TIMEOUT_SECONDS = 300`. | | `operation_state_utils.py` | `OperationStateManager` — all `_operation_state` access (progress, concurrency locks, extractor/aggregator bookmarks, cluster fingerprints, cancellation). | | `extractor_config_utils.py`, `extractor_interaction_utils.py` | Filter extractors by source / `allow_manual_trigger` / names; per-extractor stride + window + bookmark handling. | @@ -53,7 +56,7 @@ strings before deleting old import paths in the same PR. | `evaluation_overview/` | Dashboard/read-side rollups: `service.py` entry point, `components/` aggregation helpers, and root `eval_sampler.py` shared with regenerate jobs. See [README](evaluation_overview/README.md). | | `playbook_optimizer/` | Scenario-based playbook optimization: mature flat package with `optimizer.py`, `scheduler.py`, `rollout.py`, `judge.py`, `models.py`, `scenario_resolver.py`, `gepa_adapter.py`, and `assistant_webhook.py`. See [README](playbook_optimizer/README.md). | | `braintrust/` | Braintrust export/sync: `service.py`, `client.py`, `_cron.py`, `_encryption.py`. | -| `lineage/` | Current-record resolution and tombstone GC: `resolve.py`, `gc_scheduler.py`. | +| `lineage/` | Current-record resolution, tombstone GC, and opt-in missing-vector repair: `resolve.py`, `gc_scheduler.py`, `vector_backfill_sweep.py` (`REFLEXIO_MISSING_VECTOR_BACKFILL_ENABLED`, `REFLEXIO_MISSING_VECTOR_BACKFILL_CAP`). | | `governance/` | Subject-reference and retention/barrier policy contracts: `config.py`, `service.py`, `subject_refs.py`. Storage and lineage code use these helpers to preserve governance constraints during writes, supersedes, and cleanup. | | `pre_retrieval/` | `QueryReformulator` (`_query_reformulator.py`) + `DocumentExpander` (`_document_expander.py`) - query rewrite and doc expansion for recall. Compact by design; see [README](pre_retrieval/README.md). | | `tagging/` | `TaggingService` (`service.py`) + deferred `tagging_scheduler.py` - post-generation profile/playbook tagging. Compact by design; see [README](tagging/README.md). |