Proposal: ThoughtSpot converter (bidirectional, TML ↔ Ossie) #285

Activity

1. djwaldo

   added a commit that references this issue on Jul 30, 2026

   docs(ossie): post-ready touch-ups (calendars, aliases, relationship e…

   ...

   c6dc13e
2. djwaldo commented on Aug 31, 2026

   

   djwaldo

   on Aug 31, 2026

   ContributorAuthor

   More actions

   Cross-linking #269 — @utkarshparekh opened it on 26 July, three days before this issue, asking the same question. I did not find it before filing this one. Suggesting we consolidate here, with #269 closed as the earlier duplicate once @utkarshparekh and the maintainers are happy with that; my full reply to the points raised there is on #269 and I won't repeat it.

   Bringing the substance to dev@ossie.apache.org as CONTRIBUTING asks — thread: [DISCUSS] ThoughtSpot converter: scope, licensing, and where it should live, sent today. Three questions there: the TML object scope (Model + Table + SQL View; not Worksheets, which are deprecated and no longer importable), the thoughtspot_tml EULA boundary that #269 raised, and where a vendor converter should live given the 25 August discussion about per-vendor repositories.

   The construct mapping and the expression/function mapping will follow as a draft PR rather than comments here — each is around 90KB, past GitHub's comment size limit, and line-level review is what a 146-row mapping table actually needs.

   One correction to this issue's original text, since it will otherwise mislead a reader: the proposed module names tml_to_ossie / ossie_to_tml predate the OSI → Ossie rename and do not follow the vendor-named convention the other converters use. Read them as thoughtspot_to_ossie / ossie_to_thoughtspot.
3. jbonofre

   added a commit that references this issue on Sep 1, 2026

   Add THOUGHTSPOT to the Dialect enum and SKIP_SQL_VALIDATION (#351)

   ...

   211ce66
4. WilliamKelley

   added a commit that references this issue on Sep 3, 2026

   Add THOUGHTSPOT to the Dialect enum and SKIP_SQL_VALIDATION (apache#351)

   ...

   0dd290a
5. djwaldo commented on Sep 8, 2026

   

   djwaldo

   on Sep 8, 2026

   ContributorAuthor

   More actions

   The converter is up as #364.

   It follows the scope proposed here, with two changes worth calling out against what I
   originally described.

   Object scope narrowed as promised. Model TML plus Table and SQL View documents.
   Worksheets are out, for the reason given above — they are the deprecated predecessor of
   the Model and Worksheet import is blocked on current Cloud builds, so a converter
   targeting them would be dead on arrival.

   Expressions are not translated across dialects, and that is deliberate. The sketch
   above implied a function-mapping document would drive translation in both directions. It
   does drive Ossie -> TML — all 146 constructs the expression language defines are mapped
   to a ThoughtSpot rendering. But in the other direction a ThoughtSpot formula is carried
   verbatim under a THOUGHTSPOT dialect entry, with a portable ANSI_SQL sibling only
   where the whole expression is a bare column reference.

   I changed position on that after looking at what this repository already does rather than
   at what I had assumed: no converter here re-renders an expression from one dialect into
   another, the Databricks converter carries expressions verbatim under its own vendor
   dialect and reads that first, and the specification's own stated default is to pass
   unknown values through. The upside is that round-trip fidelity is exact rather than
   best-effort. The full reasoning is in the converter's README.

   custom_extensions[THOUGHTSPOT] carries what the sketch predicted — join types,
   aggregation defaults, formats, and the ThoughtSpot-only column properties — plus the
   physical column names and connection-specific type spellings that turned out to be needed
   for an exact round trip.

   804 tests, offline, Python 3.10-3.14, PyYAML the only runtime dependency. The TPC-DS
   fixtures mirror examples/tpcds_semantic_model.yaml so the output is comparable with the
   other converters.

   One implementation note that may interest anyone doing the same mapping. Ossie has no
   cardinality field on a relationship — from is documented as the many side and to as the one
   side, so direction carries it. ThoughtSpot joins carry an explicit cardinality including
   ONE_TO_MANY, which cannot be expressed in Ossie in the direction TML declares it, so the
   converter swaps the endpoints for that case and stashes the swap so the return trip reproduces
   the original join.

   Worth knowing: validation/validate.py does not catch an inverted relationship — its
   key-coverage check skips datasets that declare no keys. An earlier revision of the branch
   emitted the un-swapped form and validated clean.

   Happy to close this issue once the PR lands, or now if maintainers prefer to track it
   there.
6. djwaldo commented on Sep 25, 2026

   

   djwaldo

   on Sep 25, 2026

   ContributorAuthor

   More actions

   Closing: the converter merged as #364 on 2026-09-24 (d628f683). Thanks @jbonofre for the reviews and the merge.

   Against the scope proposed here:

   • Bidirectional — ossie-thoughtspot to-ossie / to-tml, both shipped
   • File-to-file Python, converters/databricks package shape — including the TPC-DS fixture pair, round-trip and property-based tests, and a per-converter CI workflow
   • custom_extensions with vendor_name: THOUGHTSPOT — shipped, and core-spec/spec.md carries the rows for both the dialect and the vendor attributes

   One thing I promised here and did not deliver in the form described. I said the construct- and function-mapping documents would be posted on this issue for community review before the code PR. They were not. What exists instead is generated from the code and ships with the converter — converters/thoughtspot/docs/expression-mapping.md (all 146 constructs) and docs/reverse-inventory.md — regenerated by tools/generate_reference_docs.py and gated byte-for-byte by a test, so the documentation cannot drift from the catalog it describes. I think that is a better artifact than a static comment on an issue would have been, but it is not what I said I would do, and the community review step I offered did not happen.

   Two follow-ups found since the merge, both by round-tripping real ThoughtSpot Models rather than by review or tests, are tracked separately in #459 and apache/ossie-converters#30 with a fix in #463.

   @utkarshparekh — #269 asked the same question three days before this issue and is still open. I suggested consolidating back in August; it may be worth closing too, but that is yours rather than mine.
7. djwaldo

   closed this as completedon Sep 25, 2026