Empty store changes

[acurrieclark]

Hello @incipit0 @alexjg @mvr @olynch et al
I have been working on getting sync up and running with automerge repo and there are a few changes which I would be most grateful for your input on.

This PR is a largely LLM generated example of what I think is required:

Empty Stores
Firstly, it enables creation of empty stores, which can then be hydrated from the synced commit chunks. This is used to load coln stores of any schema, even if the schema is unknown. It also allows us to buffer out-of-order chunks until the root and dependencies arrive; Automerge Repo and Subduction do not guarantee the arrival order of entries in the hash graph.

Essentially this means that from typescript we can call StoreHandle.empty() to initialise a store within repo without needing to use the schema.

Calling StoreHandle.heads() in typescript will also return an empty array until the store has loaded, allowing us to determine store readiness.

Additional Types
These are some Realm typings which I found in one of the test helpers for coln-js-runtime and which I have found useful in the higher level typescript work. Included here for completeness.

Transaction recovery
A store should be rolled back if a transaction throws prior to commit. This was an LLM discovery, so I'll let it describe the issue better that I can:

On main, if the change callback throws before commit():

• The store remains owned by the active TransactionHandle.
• takeStore() cannot reclaim it because it only handles failed commit attempts.
• The staged changes have not mutated the store, but the store itself becomes inaccessible.
  Our change adds abort(): takeStore() now returns the original store and discards the transaction’s staged operations. So the store is both reclaimed and effectively rolled back.

This makes sense to me, but I will leave it to you to determine if this is the correct behaviour.

Please let me know your thoughts and questions when you can.

Many thanks, Alex

[incipit0]

So this is actually a wrapper around decode_store_chunks and it prepares the data into Vec<Chunk> which can then be fed into decode_store_chunks? This should probably be placed near store::graph_with_root_commit() method because they are all methods about creating a store, one takes a realm, another takes a bunch of commits.

By the way the table oid allocation logic has changed recently, so you need to rebase.

[acurrieclark]

Thanks for the heads up about changes on main. I'll rebase my own working tree and see if there is anything worth pushing back here.

[incipit0]

I have renamed the this function to Store::try_from_commit_bytes, should be the same functionality though.

[incipit0]

It is not sufficient to just return the store as abort. Although the store is not modified until we commit, we need to invalidate the handle when we abort. See txn::inner::TxnInner::invalidate_handles.

I suggest having an abort method in TxnInner, and then have both OwnedTxn and normal Txn's abort method call it.

Also add a test to check that indeed handles are invalidated after calling abort.

[acurrieclark]

The only thought here is to ensure that following an error within a transaction, the coln store is still available within the automerge-repo handle and another transaction could be attempted.

[incipit0]

Yes, with abort(), you should get back the store unchanged and still be able to use it.

[incipit0]

Can you document a bit why Option is not sufficient?

[acurrieclark]

I am afraid I don't consider myself qualified to answer this question.

[alexjg]

If we use an Option then we need to figure out what to do with changes received whilst waiting for the store to become ready. There's an argument to be made that automerge-repo/subduction should handle causal delivery but for the time being they don't. I think the pragmatic thing to do here is to handle the causal delivery within coln-js-runtime, which means we need to queue commits here.

That said, we will also need to queue commits in the event of non-causal delivery even if we're in the Ready state and so maybe the queue should live directly on Store and not in StoreHandleState.

[incipit0]

Add a bit explanation about the buffering behaviour here. Also add a TODO that this should be moved somewhere else.

[acurrieclark]

Very happy to add the TODO here. Where do you think the best location for this code would be?

The buffering behaviour shown here is designed to be more of an example and is likely far from optimal. From automerge-repo's perspective, however, the only real requirements are that we can start with an empty initial store and that entries to the hash graph can be applied even should they arrive out of order. From what I understand, this could be at any time, not just on the initial sync. @alexjg may be able to clarify further.

[acurrieclark]

@incipit0 Thanks for you review. My apologies, I should have been clearer at the top of the conversation about the intention of this PR. I am by no means a Rust dev and wouldn't claim to have enough understanding over the nuance of the language nor the subtleties of your work on Coln to suggest code which should actually be merged.

This code is merely the working model which provides the automerge-repo integration with the required API. I would offer no opinion on the actual design of how this is achieved, but it was felt that showing the mostly LLM produced code might serve as a better method of demonstrating the changes to those who actually know and understand the codebase.

I'll add this to the initial comment so that it's clear for anyone else reviewing.

On the review points where I feel I may be qualified to comment, I'll reply directly.

[incipit0]

See #131