Discuss workflow macro design and lifecycle

[carloea2]

Feature Summary

Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.

References:

• feat: add workflow macro UI support #5179: frontend-oriented workflow macro implementation
• Add workflow macros for importing, grouping, and collapsing reusable sub-workflows #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
• [Hackathon] feat: AI-augmented macro operators #5115: hackathon macro-operator prototype with a different UX
• Added operator grouping feature #754: previous operator grouping/collapse behavior

Proposed Solution or Design

Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:

1. Macro creation UX

   • Should users create a macro by selecting existing operators and choosing Create Macro from the context menu?
   • Should users create a macro by importing an existing workflow?
   • Should both flows be supported?

2. Macro representation and reuse

   • Should a macro look and behave like a normal logical operator?
   • Should macro input/output ports be derived from the internal sub-DAG?
   • Should reusable macros appear directly in the operator menu, for example under My Macros?
   • Should the long-term design also be able to represent operators such as Hash Join as macros?

3. Macro internal view and editing

   • Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
   • Are edits local to one workflow snapshot, or shared across all uses of the macro?
   • Should macro internals expose explicit input/output ports instead of dummy source/sink operators?

4. Runtime and statistics behavior

   • Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
   • Should the backend be unaware of macros?
   • When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?

5. Macro lifecycle and persistence

   • How should macro definitions and macro instances be stored?
   • How should users remove, rename, disable, search, share, or version macros?
   • What documentation, tutorials, and tests are needed before enabling the feature by default?

Impact / Priority

?

Affected Area

• Workflow UI
• Workflow Engine (Amber)
• Storage / Metadata
• Documentation / Tutorials

[carloea2]

@chenlica @Yicong-Huang @Xiao-zhen-Liu @aglinxinyuan

[Yicong-Huang]

Let me convert this to a Discussion post. sorry if that's not clear. for uncertain topics, we use disucussions. once a consensus is reached, we open an issue (or feel free to use umbrella issue + multiple sub-issues) to track decided todos. Discussions can have back and forth ideas. issues should be clear on what exactly to do.

[Yicong-Huang]

I shared my preferences in #5179 already. If you are having an offline discussion, please report back here.

[carloea2]

In that case, we can just wait for @chenlica and an internal meeting to decide and report the decision here.

[chenlica]

@carloea2 Let's do a meeting to discuss the details and report the results here.

[Xiao-zhen-Liu]

We discussed this feature offline. We agreed that the design in #5115 is more general in that it has explicit encapsulation of the input and output ports, and that the user chooses which ports of the macro workflow is exposed.

[chenlica]

Do we have any plan to finish this important feature?

[zuozhiw]

I saw the demo video and I would have absolute no problem and will be very happy if we can ship #5115 as-is

[chenlica]

Maybe @carloea2 can take the lead to finish it? Notice that @Xiao-zhen-Liu's availability in the summer is limited. It will be good to have a discussion before that, and report the plan here.

[carloea2]

I think we can wait for @Xiao-zhen-Liu to come back since he raised the PR first, and his design was chosen, I could take his code and raise the PRs but I feel it is not right

[aglinxinyuan]

I don't think we're on the same page. Let's have a group discussion next Monday. If we cannot finish, we can have one more discussion on Wednesday.

[chenlica]

We had an offline discussion about this topic. @carloea2 : please summarize the results.

[carloea2]

In summary:

• We will keep the first implementation simple.
• First version will not allow reusability of a macro or sharing.
• The main changes will be frontend related according to the experience Xiaozhen designed, by adding some properties to the workflow json.

[Yicong-Huang]

Hi @carloea2 could you please elaborate more? Thanks.

[carloea2]

There are no more details at the moment, we just agreed on the user experience.

Design Video for reference #5115

[Yicong-Huang]

Please involve the community in the discussion, instead of "make an offline decision and inform the community".

If you had an offline discussion, please post more details so we can continue the discussion here.

Cc @chenlica @aglinxinyuan @Xiao-zhen-Liu please share more details.

[carloea2]

I did not request an offline discussion.

[chenlica]

@carloea2 Please provide more details as suggested.

[carloea2]

Ok.

1. Frontend will do the expansion, If we have workflow 1 with macro A, and macro A has macro B, then the whole hierarchical expansion will be done on frontend, and the backend does not need to be aware of this.
2. We agreed to show the macro UI as in Xiaozhen demo and if I remember correctly and introduce buttons to add/remove inputs/outputs.
3. We have not defined details of what the frontend should show about statistics.

A general question: if the goal is to meaningfully involve the broader community, shouldn’t the whole design and decision-making discussions happen in a public, async, and searchable place like GitHub?

[chenlica]

@carloea2 Thanks for the summary. Not sure if I fully understand your question. I believe we are allowed to do offline discussions for efficiency reasons, but we need to publish the results and decisions here to maintain the transparency.

[yangzhang75]

Hi, I was suggested to fold my recent proposal (#6066, now closed) into this discussion.

I've been prototyping something adjacent to macros: a "Blocks" gallery for saving and reusing workflow pieces across workflows. Reading through this thread, my design is different in direction (drop-in reuse vs. in-place collapsing). But a few pieces from the prototype might be useful for the macros work:

• Built-in Sample macros (a small read-only starter set in the panel, for onboarding)
• Mini-map preview of each macro (SVG rendering with real operator icons)
• Snip-region save (box-select a sub-region of the canvas, save only that part as a macro instead of the whole workflow)
• Apply-to-canvas logic (fresh UUIDs, remap links, place on empty area)

Demo of the prototype: https://drive.google.com/file/d/12pPeBUvbLlsrO5t5pp0C9G2xeEordTD2/view?usp=sharing

Would any of these be useful to fold into the macros direction? Thank you!

[chenlica]

@carloea2 Please chime in. If needed, we can have a discussion and report the results here.

[carloea2]

@yangzhang75 do you want to have a zoom meeting so we can understand your proposal and design in comparison with the one presented here?

[yangzhang75]

Sure. What times work for you this week?

[chenlica]

I suggest we do not do this type "scheduling meeting" messages on GitHub.

@Yicong-Huang : what's a good practice for such messages?

[yangzhang75]

After meeting with @carloea2, here's a summary of what we discussed.

The base macro feature (the #5115 direction — explicit input/output port encapsulation and in-place collapse) will be built first. Once it's merged, I can open separate PRs adding these extra features on top:

• Live preview for a macro — a zoomable mini-map of a collapsed macro's internal sub-DAG (with real operator icons), so you can see what's inside before expanding it.
• More flexible selection when creating a macro — a box/region ("snip") select to turn part of the canvas into a macro, in addition to selecting operators one by one.
• Macro gallery for browsing and reusing saved macros — this fits later, alongside the reuse/sharing phase (v1 doesn't include reuse).