Add Arrow buffer panels - #31
Conversation
|
Found some issues with nested types and some things that could be improved in general. Trying to guide Claude, hopefully it doesn't get confused =) Will mark the PR ready for review once I am done. |
raulcd
left a comment
raulcd
left a comment
There was a problem hiding this comment.
Thanks @AlenkaF that looks cool. I saw that the string view also shows the views buffer without showing prefix, variadic buffer index, size, offset. Could be interesting to show that if possible too (or as an improvement for the future).
|
To be fair, as an initial concept and as a PoC looks really cool! Thanks! |
|
Exactly! I have made some updates, the more complex types should be ok now and I have created an example table to mimic the Introduction diagrams. Will add a print screen after lunch, pushing linter fixes now. |
|
|
@raulcd see current visualization above. I still plan to add support for other types that are currently missing. But the structure and current state should be good for a review. |
|
Added support for missing types, see: 
|
raulcd
left a comment
raulcd
left a comment
There was a problem hiding this comment.
This is great @AlenkaF !
The only thing I find we could improve is for the views buffer to show the length when the string is less than 12, currently it only shows the value but not the size, even though the size is part of the views buffer:
We can follow up this as a future improvement! What do you think?
|
Good catch! I would like to correct it now. Am already working on it ;) |
|
Done: 
I have also updated the example for list view, so it is easier to compare to the diagrams from our docs: 
I might post the example code somewhere so it is easy to duplicate. |
raulcd
left a comment
raulcd
left a comment
There was a problem hiding this comment.
I am merging this and probably kicking off a new 0.3.1 release afterwards so we have this released for our following talk
2 participants
This PR adds a tab on the IPC interactive view where the buffers are visualised for the first 2 record batches and first 10 rows (slice of the buffer). The layout has been inspired by the diagrams in the Arrow Format Intro docs and it uses Python memoryview to get the content of each buffer. The validity buffer is manipulated to be in logical order matching the position in the array.
Note, this has been done by Claude and I delegated tasks and checked the changes. I also suggested the use of nanoarrow to get the buffers content but unfortunately it doesn't support all the types. Might be good idea in the future.