ListView Handling normalization

[a10y]

Currently, Vortex uses ListView as its canonical List encoding data type. That means internally, lists are always treated as ListView.

We currently have a weird mismatch between how ListView's are written and how execute_arrow() works for them.

By default at write, we have two compressor pathways

• BtrBlocks which will always emit List to the file
• Compact which will always emit ListView to the file

Note that the file metadata for ListViewArray does not contain any information about if the ListView is compatible zero-copy with a List memory layout.

The ListViewArray::execute_arrow function if no Arrow DataType is provided, will emit ListArray. This means that users who write data with Compact compressor and try and read it out as Arrow (which is a very common codepath) actually get the worst of all worlds performance, because ListView will go through the naive_rebuild pathway.

There are two options that come to mind for how we fix performance of the write(ListView) + read(List) use-case:

1. Write a is_zero_copy_to_list metadata field in the flatbuffer for ListView so that we know to skip the path. This could potentially lead to malicious behavior if we e.g. build a ListArray::new_unchecked b/c we assume some correctness property about this flag that is not true
2. Force writing List in CompactCompressor

[connortsui20]

I believe there is a third option that we have brought up in the past where we do not store is_zero_copy_to_list on disk but we try to have some sort of custom layout strategy where we chunk this in a smart way.

To be honest, I am still in support of storing this flag in metadata. If this is somehow a terrible idea and there is actually a better way of converting a ListViewArray into a ListArray (I really do not think there is, but for the sake of argument), it's not like we can't have multiple "fast path" flags stored in metadata where we prioritize one fast path over the other.

CC @robert3005 @joseph-isaacs since I remember we talked about this several months ago.

[joseph-isaacs]

How did this come up?

I am not sure that we will always choose List, only of the ListView is list-like. @asubiotto uses compressed ListView where element are referenced more than one in a ListView.

Also for datafusion scanning there will also be a target data_type?

Compact should likely try to compress listview to lists too?

Regarding ListView metadata, shall we roll that in with the ListLayout work we will conduct soon?

[robert3005]

I think the fix here is to properly chunk our arrays and store listviews, then on read we can skip reading sizes if we can somehow plumb information that we are reading into list.

[asubiotto]

AFAIU #6322 removed the compact compressor so we always go through BtrBlocks now, right? I think that the heuristic to write list/listview in BtrBlocks should guard against these cases. If you have a zctl list and write that, List will always be emitted to the file.

If this is related a perf issue on 0.58.0, I think it might be worth to check if it still exists on latest develop. I also fixed some cases where we were losing zctl information: #6304

[robert3005]

There's a lot of logic now to improve ListView<->List roundtrips. The other part is to have listlayouts