feat(snowflake): add read-only Snowflake datasource gem#292
Open
bexchauveto wants to merge 5 commits intomainfrom
Open
feat(snowflake): add read-only Snowflake datasource gem#292bexchauveto wants to merge 5 commits intomainfrom
bexchauveto wants to merge 5 commits intomainfrom
Conversation
9 new issues
|
![qltysh[bot]](https://avatars.githubusercontent.com/in/890766?s=60&v=4){kind=small}
| execute_to_hashes(sql, binds, projection.to_a) | ||
| end | ||
|
|
||
| def aggregate(_caller, filter, aggregation, limit = nil) |
| ) | ||
| @primary_keys << column_name if field.is_primary_key | ||
| add_field(column_name, field) | ||
| end |
| attr_reader :pool | ||
|
|
||
| def initialize(conn_str:, tables: nil, schema: nil, | ||
| pool_size: DEFAULT_POOL_SIZE, pool_timeout: DEFAULT_POOL_TIMEOUT, |
| source.add_field(relation_name, relation) | ||
| end | ||
| rescue ::ODBC::Error => e | ||
| warn "[forest_admin_datasource_snowflake] FK introspection skipped: #{e.message}" |
| result += equality + orderables if %w[Date Dateonly Time Number].include?(type) | ||
| result += equality + orderables + strings if type == 'String' | ||
|
|
||
| result |
| ConditionTreeBranch = ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Nodes::ConditionTreeBranch | ||
| ConditionTreeLeaf = ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Nodes::ConditionTreeLeaf | ||
|
|
||
| def initialize(collection, projection: nil, filter: nil, aggregation: nil, limit: nil) |
| sql << " OFFSET #{Integer(offset)}" if offset | ||
| end | ||
|
|
||
| [sql, @binds] |
| translate_leaf(node) | ||
| else | ||
| raise ForestAdminDatasourceSnowflake::Error, "Unsupported condition tree node: #{node.class}" | ||
| end |
bexchauveto
force-pushed
the
feat/snowflake-datasource
branch
from
1ab8be1 to
465613a
Compare
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Introduces forest_admin_datasource_snowflake — a native Forest Admin
datasource backed by Snowflake via ODBC, with no ActiveRecord involvement.
Read-only first pass: list/aggregate query translation, schema introspection,
connection pooling, retry-on-disconnect, optional FK auto-discovery, and
optional per-session statement timeout.
Type translation:
- ODBC SQL types → Forest types via Parser::Column
- Snowflake-native types (VARIANT/OBJECT/ARRAY/BINARY/VARBINARY) detected
via INFORMATION_SCHEMA.COLUMNS and override the ODBC mapping
- ODBC::Date / ODBC::TimeStamp / ODBC::Time coerced to native Ruby
Date/Time so JSON serialization works (otherwise they emit `{}`)
- JSON-typed columns parsed into Ruby objects
- Session-level TIMEZONE='UTC' so TIMESTAMP_NTZ/LTZ/TZ all serialize
consistently as UTC
CI integration: added the package to lint/test matrices in build.yml
with a unixODBC apt step. 77 specs cover the introspection, query
translator, identifier quoting, type mapping, and connection retry paths.
bexchauveto
force-pushed
the
feat/snowflake-datasource
branch
from
465613a to
e6253b4
Compare
Previously when Aggregation.field was nil, build_aggregation_expression emitted invalid SQL like SUM() because Identifier.quote(nil) returns an empty string. Raise an explicit ForestAdminDatasourceSnowflake::Error so the failure surfaces at translation time instead of as a Snowflake syntax error at execution. COUNT keeps its existing nil-as-* fallback.
| "#{op}(#{q(field)})" | ||
| else | ||
| raise ForestAdminDatasourceSnowflake::Error, "Unsupported aggregation operation: #{op}" | ||
| end |
visible_tables and Collection#fetch_fields called stmt.drop after stmt.fetch_all without an ensure block, so a fetch failure would leak the statement handle. Wrap the fetch in begin/ensure to match the pattern already used by fetch_snowflake_native_types and run_session_statement.
Member
|
The PR is missing changes to
Without this, the gem won't be published to RubyGems even after the PR is merged. |
…ipeline Adds the new package to the three .releaserc.js sections so semantic-release will: bump its VERSION constant on release (prepareCmd), build and push the gem to RubyGems (successCmd), and commit the version bump back to the repo (@semantic-release/git assets). Without this the gem stays at 0.0.x and never reaches RubyGems.
Snowflake's session token has a finite TTL; once it expires the driver returns "08001 (390114) Authentication token has expired." This message didn't match any of CONNECTION_LOST_PATTERNS, so the gem propagated the error to Forest instead of cycling the pool. Cycling forces a fresh drvconnect, which re-authenticates with the credentials in conn_str. Adds matchers for the exact Snowflake string and a generic "token expired" fallback so similar phrasings from other drivers also recover.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces
forest_admin_datasource_snowflake— a native Forest Admin datasource backed by Snowflake via ODBC, with no ActiveRecord involvement.Read-only first pass. Covers list/aggregate query translation, schema introspection, connection pooling, retry-on-disconnect, optional FK auto-discovery, and optional per-session statement timeout. Wired into
build.ymllint/test matrices with aunixodbc-devapt step.Highlights
Type translation
Parser::ColumnVARIANT/OBJECT/ARRAY/BINARY/VARBINARY) detected viaINFORMATION_SCHEMA.COLUMNSand override the generic ODBC mapping — without this, the Snowflake ODBC driver labels them allSQL_VARCHARODBC::Date/ODBC::TimeStamp/ODBC::Timecoerced to native RubyDate/Time(otherwise they JSON-serialize as{})TIMEZONE='UTC'soTIMESTAMP_NTZ/LTZ/TZall serialize consistentlyOperational
connection_pool-backed pool, sized viapool_size:(default 5)with_connectionretries the block once on connection-lost ODBC errors and cycles the pool to drop stale handlesstatement_timeout:(in seconds) issuesALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDSintrospect_relations: truediscovers Snowflake-internal FKs viaSHOW IMPORTED KEYS IN SCHEMA(Snowflake doesn't ship the full ANSIINFORMATION_SCHEMA, so the join-style query isn't usable)Read-only enforcement
is_read_only: true, so Forest's schema emitter computes collection-levelisReadOnly: trueautomatically and the UI hides create/edit/delete affordancescreate/update/deleteraiseForestExceptionwith an explicit read-only message as a defence-in-depth guardCoverage
77 specs across:
Parser::Column— ODBC and Snowflake-native type mapping (24)Utils::Identifier— case-preserving quoting (5)Utils::Query— condition tree translation, IN/LIKE/ILIKE, sort/page (16)Datasource— introspection, pool, session settings, retry, FK discovery, native-type fetch (17)Collection— schema introspection, list/aggregate, JSON parsing, write guards (15)Caveats / follow-ups
SNOWFLAKE_*env vars are present).Collectionare explicit so a later subclass can override safely.BILLING_USAGE.CUSTOMER_ID→ Postgrescustomers.id) cannot be auto-discovered — Snowflake only knows about its own metadata. These need to be wired manually withadd_many_to_one_relation/add_one_to_many_relationat the agent layer.Test plan
cd packages/forest_admin_datasource_snowflake && BUNDLE_GEMFILE=Gemfile-test bundle install && BUNDLE_GEMFILE=Gemfile-test bundle exec rspecpasses (77 examples, 0 failures)bundle exec rubocop packages/forest_admin_datasource_snowflakepasses (clean)bundle exec rubocoprepo-wide passes (still clean after.rubocop.ymlexclusion additions)Note
Add read-only Snowflake datasource gem backed by ODBC
forest_admin_datasource_snowflakegem that connects Forest Admin to Snowflake via ODBC using a pooled connection (connection_pool).Datasourcediscovers visible tables as read-only collections, applies UTC session settings, and optionally introspects foreign keys viaSHOW IMPORTED KEYSto addManyToOnerelations.Collectionintrospects column schemas at init, maps ODBC/Snowflake native types (includingVARIANT/OBJECT/ARRAY→Json) to Forest types, and coerces ODBC date/timestamp values on read. Write operations (create,update,delete) raise a read-only error.Utils::Querytranslates Forest condition trees into parameterized Snowflake SQL, supporting WHERE, ORDER BY, LIMIT/OFFSET, and aggregations.Macroscope summarized 6334231.