#476 - Restore OmniStudio export compatibility

packages/vlocity-deploy/README.md

@@ -21,12 +21,12 @@ This library is built from scratch to provide a fast and reliable way to deploy
 
 ### Also...
 
+-   Vlocode supports exporting datapacks with configurable export definitions
 -   Vlocode does client-side OmniScript to LWC compilation and deployment
 -   Vlocode does client-side OmniScript Activation avoiding SOQL limit exceptions when activating big scripts
 
 ## What does it **not** do...
 
--   Vlocode vlocity-deploy library is meant for deploying datapacks **not** for exporting/extracting them from an org
 -   Activation of flex cards
 -   Provide you with a (fancy) UI
 
@@ -228,3 +228,29 @@ interface DatapackDeploymentSpec {
     afterDeployRecord?(event: readonly DatapackDeploymentRecord[]): Promise<any> | any;
 }
 ```
+
+## Export identity and output
+
+An export definition can specify `exportKey` independently of `matchingKeyFields`:
+
+```yaml
+OmniScript:
+  objectType: OmniProcess
+  name: [Type, SubType, Language]
+  exportKey: [Type, SubType, Language]
+  matchingKeyFields: [Type, SubType, Language, VersionNumber]
+```
+
+During expansion, this writes `OmniProcess/Type/SubType/Language` to `VlocityRecordSourceKey`. `exportKey` is an ordered field list: values are joined with slashes and prefixed with the object type. It only reads the exported datapack; it does not query Salesforce or affect record matching, lookup resolution, or the keys returned by `exportObject`.
+
+Expansion applies the referenced object's `exportKey` to both internal matching references and external lookup references, including nested lookup fields. This also works when the referenced datapack is exported separately: its key is computed from the fields on the reference. Full records retain their original identity inputs in reserved `VlocityRecordExportKeyValues` metadata; references use it for fields outside their matching key. This metadata survives consolidated JSON exports and is removed during expansion. Identity lookup fields are resolved even when omitted from output, so records and references produce the same portable key. These values do not become deployment lookup criteria. Matching fields remain unchanged; only serialized source-key values and parent keys are rewritten. References without an `exportKey` retain their original keys. Namespace-qualified field names are supported. Omitted or empty field lists retain the original source key. Different source versions may share an output key, preserving the existing overwrite behavior. Include `VersionNumber` in the list when deliberately writing version-specific keys. Fields used in the list remain available until `ignoreFields` is applied.
+
+Output settings apply to expanded files and consolidated exports:
+
+- `ignoreFields` removes fields after ordering and filename evaluation, so excluded fields can still control output order and names.
+- `sortParentField` names a parent lookup for depth-first traversal of an embedded array. `sortFields` orders siblings. Circular parent links fail the export.
+- `sortMode: vlocity` uses the configured sort fields with empty values last and canonical JSON as the final tie-breaker. OmniStudio mapping definitions use the six legacy mapping priorities.
+- `nullValues` defaults to `preserve`; `omit` excludes null and empty-string fields, and `emptyString` converts nulls to empty strings. These rules apply to SObject bodies, preserving external matching instructions.
+- `parseJson` controls JSON parsing of text fields and can be overridden per field. OmniStudio definitions preserve text unless a field explicitly contains structured JSON. Arrays inside JSON payloads retain their order unless `sortFields` is explicitly configured.
+
+The bundled native and managed OmniStudio definitions use versionless script keys, hierarchical elements, legacy mapping order and type-specific empty-field policies. Integration Procedures expand sample input into `_SampleInput.json`; OmniScript JavaScript and HTML fields retain their file extensions.

packages/vlocity-deploy/src/__tests__/__fixtures__/exportParity/README.md

@@ -0,0 +1,36 @@
+# Issue #476: Build Tools / direct export parity
+
+These six fixtures lock the expanded-output contract of **Vlocity Build Tools 1.14.18** for native and managed OmniStudio. All records, IDs, names and payloads are synthetic. No org data or credentials are included.
+
+Each fixture contains:
+
+- `input`: fake Salesforce records, field-reference metadata and matching-key fields.
+- `expectedFiles`: the actual files produced by Build Tools, including filenames and JSON text. These are generated independently of Vlocode's direct exporter.
+
+The `Build Tools 1.14.18 export parity (#476)` tests in `datapackExporter.test.ts` run the real direct exporter, definition store and expander with only Salesforce schema/data and matching-key retrieval mocked. They compare all file paths and contents against this frozen oracle. JSON whitespace is disregarded; property order, array order, field presence, value types, source keys and filenames are checked.
+
+Coverage of the four reported differences:
+
+1. Parent/child traversal with multiple parents and repeated sibling sequence numbers; ignored order/level fields must survive until sorting.
+2. Mapping priority order, numeric zero, empty fields, and formula mappings tied on every configured sort field, requiring the legacy canonical-JSON tiebreaker.
+3. Versionless root and child export identities and matching references, with version 64 still present in the input and deployment matching definition.
+4. DataRaptor empty strings, omitted empty script fields, JSON-as-text, recursively ordered JSON properties, positional JSON arrays, IP `_SampleInput.json`, OS `_JavaScript.js`, and XML filenames/content.
+
+## Explicit comparison boundaries
+
+Only two known optional metadata differences are normalized by the test:
+
+- Extra matching fields on **internal** `VlocityMatchingKeyObject` references. Their object type and source-key value are still compared. External lookup objects are not normalized.
+- A source key on an unreferenced managed `DRMapItem__c` record. Build Tools omits it; direct export can retain it. Mapping fields, record count and order are still compared.
+
+The fixtures do not represent byte-identical output in those two respects and do not test Salesforce's server-side DataPack export API or deployment. The tools transport adapter provides SObject/reference envelopes from synthetic records, using the empty-string transport convention and native DataRaptor IsActive exclusion observed in the live investigation. Build Tools handles key generation, sorting, pruning, parsing and file expansion itself.
+
+## Regeneration
+
+From the repository root:
+
+```sh
+node packages/vlocity-deploy/src/__tests__/__fixtures__/exportParity/generate.mjs
+```
+
+This runs offline using the extension's pinned Build Tools dependency and this repository's dependency patches. It does not call the direct exporter. It checks the Build Tools version, rejects attempted network requests and removes its temporary directory afterwards. Review oracle changes rather than regenerating fixtures merely to make a failing direct-export test pass.

packages/vlocity-deploy/src/__tests__/__fixtures__/exportParity/generate.mjs

@@ -0,0 +1,116 @@
+// Manual oracle generation: uses Build Tools only, never the direct exporter.
+import fs from 'node:fs';
+import path from 'node:path';
+import os from 'node:os';
+import process from 'node:process';
+import console from 'node:console';
+import { createRequire } from 'node:module';
+import { fileURLToPath } from 'node:url';
+const fixtureDir = path.dirname(fileURLToPath(import.meta.url));
+const extensionRequire = createRequire(path.resolve(fixtureDir, '../../../../../vscode-extension/package.json'));
+const yaml = extensionRequire('js-yaml');
+extensionRequire.extensions['.yaml'] = (module, file) => { module.exports = yaml.load(fs.readFileSync(file, 'utf8')); };
+const Vlocity = extensionRequire('vlocity/lib/vlocity');
+const version = extensionRequire('vlocity/package.json').version;
+if (version !== '1.14.18') throw new Error(`Review the oracle upgrade before regenerating: ${version}`);
+const temp = fs.mkdtempSync(path.join(os.tmpdir(), 'vlocode-export-parity-'));
+
+function fixture(runtime, datapackType) {
+    const managed = runtime === 'managed';
+    const field = (native, custom) => managed ? `%vlocity_namespace%__${custom}__c` : native;
+    const dr = datapackType === 'DataRaptor';
+    const rootType = field(dr ? 'OmniDataTransform' : 'OmniProcess', dr ? 'DRBundle' : 'OmniScript');
+    const childType = field(dr ? 'OmniDataTransformItem' : 'OmniProcessElement', dr ? 'DRMapItem' : 'Element');
+    const ownerField = field(dr ? 'OmniDataTransformationId' : 'OmniProcessId', dr ? 'DRBundleId' : 'OmniScriptId');
+    const parentField = field('ParentElementId', 'ParentElementId');
+    const rootId = 'a00000000000001AAA';
+    const root = { Id: rootId, __type: rootType, Name: dr ? 'MappingParity' : 'ScriptParity' };
+    const references = { [childType]: { [ownerField]: rootType, ...(!dr && { [parentField]: childType }) } };
+    const matchingKeys = { [rootType]: dr ? ['Name'] : [field('Type', 'Type'), field('SubType', 'SubType'), field('Language', 'Language'), field('VersionNumber', 'Version')],
+        [childType]: dr ? (managed ? [field('GlobalKey', 'MapId'), 'Name'] : []) : [ownerField, 'Name'] };
+    let children;
+    if (dr) {
+        Object.assign(root, { [field('GlobalKey', 'GlobalKey')]: 'fixture-global-key',
+            [field('ExpectedInputJson', 'InputJson')]: '{"z":[{"Name":"Z"},{"Name":"A"}],"a":1}',
+            [field('ExpectedOutputXml', 'TargetOutXml')]: '<result>fixture</result>', Description: null });
+        if (!managed) root.IsActive = true;
+        const priorities = ['InputObjectQuerySequence', 'FilterGroup', 'OutputCreationSequence', 'OutputObjectName', 'OutputFieldName', 'InputFieldName']
+            .map((name, index) => field(name, ['InterfaceObjectLookupOrder', 'FilterGroup', 'DomainObjectCreationOrder', 'DomainObjectAPIName', 'DomainObjectFieldAPIName', 'InterfaceFieldAPIName'][index]));
+        children = [
+            [1, 1, 1, 'Record', 'Z', 'inputZ'],
+            [null, null, null, null, null, null],
+            [1, 1, 1, 'Record', 'A', 'inputA'],
+            [null, null, null, null, null, null],
+            [0, 1, 1, 'Record', 'Zero', 'inputZero']
+        ].map((values, index) => ({ Id: `a0100000000000${index+1}AAA`, __type: childType, Name: root.Name,
+            [ownerField]: rootId, [field('GlobalKey', 'MapId')]: `mapping-${index+1}`,
+            ...Object.fromEntries(priorities.map((name, i) => [name, values[i]])),
+            [field('FormulaExpression', 'Formula')]: index === 1 ? 'Z_FORMULA' : index === 3 ? 'A_FORMULA' : null,
+            [field('FormulaConverted', 'FormulaConverted')]: null, Empty: null }));
+    } else {
+        Object.assign(root, { [field('Type', 'Type')]: 'Fixture', [field('SubType', 'SubType')]: datapackType,
+            [field('Language', 'Language')]: 'English', [field('VersionNumber', 'Version')]: 64,
+            [field('IsIntegrationProcedure', 'IsProcedure')]: datapackType === 'IntegrationProcedure',
+            [field('IsActive', 'IsActive')]: true,
+            [field('PropertySetConfig', 'PropertySet')]: '{"z":[{"Name":"Z"},{"Name":"A"}],"a":1}',
+            [field('CustomJavaScript', 'CustomJavaScript')]: datapackType === 'IntegrationProcedure' ? '{"z":2,"a":1}' : 'console.log("fixture");',
+            [field('ElementTypeComponentMapping', 'OmniScriptDefinitions')]: '{"z":2,"a":1}', Description: null });
+        const specs = [['First',1,0,null],['ChildB',1,1,5],['ChildA',1,1,4],['ParentA',2,0,null],['ParentB',3,0,null],['Last',4,0,null]];
+        children = specs.map(([Name, order, level, parent], index) => ({ Id: `a0100000000000${index+1}AAA`, __type: childType, Name,
+            [ownerField]: rootId, [parentField]: parent ? `a0100000000000${parent}AAA` : null,
+            [field('SequenceNumber', 'Order')]: order, [field('Level', 'Level')]: level,
+            [field('PropertySetConfig', 'PropertySet')]: '{"z":2,"a":1}', Empty: null }));
+    }
+    return { runtime, datapackType, rootId, rootType, childType, references, matchingKeys, records: [root, ...children] };
+}
+
+// Build Tools normally receives SObjects/references from the Salesforce DataPack service.
+// This adapter supplies that transport from the synthetic records; no Salesforce connection is used.
+function toolsTransport(f) {
+    const byId = new Map(f.records.map(record => [record.Id, record]));
+    const convert = (record, reference = false) => {
+        const output = { VlocityDataPackType: reference ? 'VlocityMatchingKeyObject' : 'SObject', VlocityRecordSObjectType: record.__type,
+            [reference ? 'VlocityMatchingRecordSourceKey' : 'VlocityRecordSourceKey']: `fixture/${record.Id}` };
+        if (reference) { output.Name = record.Name; return output; }
+        for (const [name, value] of Object.entries(record)) {
+            if (name === '__type' || (f.datapackType === 'DataRaptor' && f.runtime === 'standard' && name === 'IsActive')) continue;
+            output[name] = f.references[record.__type]?.[name] && value ? convert(byId.get(value), true) : value;
+            // The DataPack service transport represents empty fields as strings.
+            if (value === null) output[name] = '';
+        }
+        return output;
+    };
+    const root = convert(byId.get(f.rootId));
+    root[f.childType] = f.records.filter(record => record.Id !== f.rootId).map(record => convert(record));
+    return { dataPacks: [{ VlocityDataPackType: f.datapackType, VlocityDataPackKey: `${f.datapackType}/Fixture`,
+        VlocityDataPackData: { [f.rootType]: [root] } }] };
+}
+
+(async () => {
+    for (const runtime of ['standard', 'managed']) for (const type of ['DataRaptor', 'IntegrationProcedure', 'OmniScript']) {
+        const input = fixture(runtime, type);
+        const v = new Vlocity({ tempFolder: temp, quiet: true });
+        v.namespace = 'vlocity_cmt';
+        v.namespacePrefix = 'vlocity_cmt__';
+        v.datapacksutils.dataPacksExpandedDefinition = v.datapacksutils.updateExpandedDefinitionNamespace(v.datapacksutils.dataPacksExpandedDefinition);
+        v.datapacksutils.getAllReferenceFields = async () => [];
+        // Fail any accidental network usage by fixture generation.
+        v.jsForceConnection.request = () => { throw new Error('Parity fixture generation must remain offline'); };
+        const job = { projectPath: temp, maxDepth: 0 };
+        await v.datapacksjob.initializeJobInfo(job, 'ExpandFile');
+        const folder = path.join(temp, runtime, type);
+        await v.datapacksexpand.expand(folder, toolsTransport(input), job);
+        if (job.errors.length) throw new Error(job.errors.join('\n'));
+        const files = {};
+        function read(dir) {
+            for (const name of fs.readdirSync(dir)) {
+                const file = path.join(dir, name);
+                if (fs.statSync(file).isDirectory()) read(file);
+                else files[path.relative(folder, file)] = fs.readFileSync(file, 'utf8');
+            }
+        }
+        read(folder);
+        fs.writeFileSync(path.join(fixtureDir, `${runtime}-${type}.json`), JSON.stringify({ toolsVersion: version, input, expectedFiles: files }, null, 2)+'\n');
+        console.log(runtime, type, Object.keys(files).length);
+    }
+})().catch(error => { console.error(error); process.exitCode = 1; }).finally(() => fs.rmSync(temp, { recursive: true }));