Skip to content

Implement EcoMap - #66

Open
pedrodesu wants to merge 3 commits into
typst:mainfrom
pedrodesu:map
Open

Implement EcoMap#66
pedrodesu wants to merge 3 commits into
typst:mainfrom
pedrodesu:map

Conversation

@pedrodesu

Copy link
Copy Markdown
Contributor

This implementation is probably missing minor things, but it should do very decently for a first draft. Let me know if there are things that should be replaced or bettered.

This implements EcoMap and tests for everything EcoMap. EcoMap can be inline, or spilled, where it's a simple CoW Rc (or Arc in case of sync flag added) wrapper of a hashmap (std's or hashbrown's in a no_std environment). This state is determined by a const generic N with a sensible default that the user may or may not provide.
We also integrate serde and rayon's parallel iterators in the case of sync.

@pedrodesu

Copy link
Copy Markdown
Contributor Author

Apparently hashbrown requests a Cargo version that supports edition 2024. Might make sense to update that.

@pedrodesu pedrodesu mentioned this pull request Jul 4, 2026
@laurmaedje

Copy link
Copy Markdown
Member

Thanks for taking a stab at this. However, I'm afraid that the design goal I would have in mind for EcoMap diverges quite a bit from this implementation. What this implementation does is:

  • Inline storage
  • Fall back to Arc<HashMap<..>> for heap storage (two allocations and double indirection)

What I would have in mind:

  • No inline storage
  • A single shared allocation for the reference count and the underlying data (and thus also no double indirection)

This would make EcoMap a direct parallel of EcoVec, since EcoVec doesn't have inline storage either. Of course, EcoString does have it, but it's a bit more special and not a generic collection container, so it's not the reference point I would take.

I'm not sure whether it's possible to do this with hashbrown, which is one of the reasons why I didn't even attempt to add this before. As, to keep performance parity with hashbrown, we might have to basically maintain a fork.

Another priority for me would be to have the code structured in a way that makes an EcoIndexVec feasible with as little duplication as possible (or maybe to even have insertion ordering by default if possible without excessive overhead) as that would be the most interesting collection in the context of Typst (to back the dictionary type).

@pedrodesu

pedrodesu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

My apologies, I had fundamentally misunderstood the layout constraints for EcoMap.

Indeed, if we want to mimic EcoVec and its single indirection, we can't use hashbrown because of its current constraints. HashTable completely isolates its internal pointers. hashbrown does not provide any public constructor to wrap around pre-populated, raw pointer sections inside a custom contiguous layout anymore.

We have two viable paths forward, either we drop hashbrown entirely and write ourselves a lightweight and optimized linear probing hash engine directly over our trailing contiguous allocation slice (in which we keep single-pointer indirection and a single heap allocation but we don't get hashbrown-level SIMD-optimized parallel probing and essentially need to reinvent the wheel a bit; performance shouldn't hurt too much with a decent implementation in comparison though), or we accept double-indirection where it's needed. I think that keeping a fork of hashbrown is a losing battle.

If the single-pointer indirection requirement is non-negotiable then the former seems like our only option.

Let me know which compromise aligns best with typst's philosophy and I can try to work on a solution.

@pedrodesu pedrodesu closed this Jul 10, 2026
@pedrodesu
pedrodesu deleted the map branch July 10, 2026 15:27
@pedrodesu
pedrodesu restored the map branch July 10, 2026 16:13
@pedrodesu pedrodesu reopened this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants