Release: Datastore Loaders - 2025.10.29 - #646
Merged
Merged
Conversation
…gs, and Transcripts (#643) This PR introduces three new LangChain-compatible Document Loaders that pull structured content from the AnswerAgentAI Datastore (Supabase) to power ingestion, chunking, and downstream RAG/search use cases. Each loader exposes rich filters, safe metadata shaping, and an optional text output mode. What’s included - AAIDomains loader (packages/components/nodes/documentloaders/AAIDomains/AAIDomains.ts) - Filters: searchTerm (domain_name/meta_title/meta_description), includeTags/excludeTags, includeTagsLogic (OR/AND), isValid (all/true/false/null), hasAnalysis (all/analyzed/not_analyzed), limit - Pagination with page size up to 100; stops early when results exhausted - Reliability: 60s fetch timeout + exponential backoff (3 retries) + small inter-page delay - Performance: selects only essential fields; explicitly excludes heavy JSONB history fields - Tag handling: flattens domain_tags join to tags array; supports include/exclude logic - Content: builds markdown pageContent with AI analysis (optional) + meta info - Metadata: includes core fields, AI-derived fields, tag slugs/labels, ISO timestamps, and numeric timestamp variants; source markers (source=aai_datastore, source_type=domain) - Metadata shaping: merge additional metadata; omit via keys list or * to keep only custom metadata - Output: Document[] or concatenated text (with TextSplitter support) - AAITags loader (packages/components/nodes/documentloaders/AAITags/AAITags.ts) - Filters: searchTerm (slug/label/description), parentTagsOnly, childTagsOnly, limit - Optional inclusion of parent tag info via self-join alias - Content: simple text summary of tag, description, parent - Metadata: core fields, parent refs, hierarchy flags, timestamps, source markers (source=aai_datastore, source_type=tag) - Metadata shaping and text output mode consistent with Domains loader - AAITranscripts loader (packages/components/nodes/documentloaders/AAITranscripts/AAITranscripts.ts) - Filters (per visible portion of diff): searchTerm (transcript/summary), dateFrom/dateTo (ISO), includeTags/excludeTags with OR/AND logic, hasAnalysis, sentimentMin/sentimentMax, limit - Consistent metadata shaping and output behavior with the other loaders - Note: diff snippet was truncated; remaining implementation aligns with the same patterns (Supabase client, filters, Document construction) - Assets - answerai-square-black.png icon added for each loader Environment variables - AAI_DATASTORE_SUPABASE_URL - AAI_DATASTORE_SUPABASE_SERVICE_ROLE_KEY Notes: - These use the service role key; do not expose in client/browser contexts. - Required for all three loaders. Why - Provide first-class, configurable ingestion nodes for AAI data (domains, tags, transcripts), enabling targeted datasets via tags, validity/analysis state, and date/sentiment filters. - Improve resiliency and performance for large domain tables while avoiding heavy JSON fields. Testing/Verification - Set AAI_DATASTORE_SUPABASE_URL and AAI_DATASTORE_SUPABASE_SERVICE_ROLE_KEY in your environment - In the node builder: - AAIDomains: try limit=50, includeTags="saas,ecommerce", includeTagsLogic="AND", hasAnalysis="analyzed"; verify tags/metadata and AI analysis sections are present when enabled - AAITags: set parentTagsOnly=true then childTagsOnly=true; verify parent info appears when includeParentInfo=true - AAITranscripts: filter by dateFrom/dateTo and a couple of tag slugs; adjust sentimentMin/sentimentMax; verify results and metadata - Switch output to “text” to confirm concatenation and escaping behavior - Optionally test TextSplitter to ensure documents chunk as expected Performance/Security - Domains loader limits selected columns and paginates to reduce payload size - 60s request timeout, exponential backoff, and inter-page delay help protect the datastore - Uses service-role key; ensure server-side execution only Breaking changes - None Documentation - Add “AnswerAgentAI Datastore” section to internal docs covering: - Required env vars - Node inputs/filters and examples - Metadata keys, omit rules, and source markers - Known constraints (service-role usage, large-table pagination) Links - Diff: https://patch-diff.githubusercontent.com/raw/the-answerai/theanswer/pull/643.diff Sources: https://patch-diff.githubusercontent.com/raw/the-answerai/theanswer/pull/643.diff --------- Co-authored-by: Max Techera <maxi.techerag@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
maxtechera
temporarily deployed
to
staging - aai-unified2-flowise-moonstruck
October 29, 2025 17:11 — with 
Render
Inactive
Render
Inactive
maxtechera
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Testing