dart: Return dart:typed_data equivalent where possible#8289
Conversation
|
|
I'm not opposed to that, but I'd like to see some insight from one of the Dart maintainers to see if this aligns with their view for the Dart side. Ideally, we could expand the same to |
|
Ready for review. I tested using @tompark's example code, with the latest version of flatbuffers on pub.dev as well as this fork, and this fork prints the expected |
|
I squashed the changes and removed a change related to my other PR to prevent conflicts between the two. @vaind Hi Ivan, is there any way you might be able to spare some bandwidth in the relatively near future to help review and potentially land both of my open PRs? Cheers. |
I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test. |
I think it's OK to have different actual return type on big endian, the only contract is that it is compatible with List, everything else is "just" an optimization. It's OK for tests to assert the return value is the expected type based on endianness though |
Sounds good to me. Sorry for the delay. I'll get this updated this week, hopefully on the earlier half as opposed to the later half. |
vaind
left a comment
There was a problem hiding this comment.
LGTM.
Are you going to do the other lists (int32, ...) too or is this PR finished?
I'll extend this to other types! I'll expand the tests as well. |
|
@vaind I couldn't find out what is causing Float64List to fail, so I am going to revert the changes to the Float types and mark them with a todo to return their dart:typed_data equivalent so the PR can be restored to a working state and will be ready for review. |
vaind
left a comment
There was a problem hiding this comment.
LGTM but I cannot merge as I'm not a maintainer.
|
This is the final version of the PR, pending any code change requests from review. Hoping to see this actually go somewhere now, |
|
@aardappel Could you perhaps give this a look in spare time? |
|
@NotTsunami sorry, I have zero knowledge of Dart. I guess I am fine to merge if @vaind thinks it is ok. @dbaileychess who probably know Dart better than I do. Also, there is a failing test. |
Thanks. Should be fixed. Pending review on the latest version of the PR now, then. |
|
Please merge it soon. |
|
This pull request is stale because it has been open 6 months with no activity. Please comment or label |
|
@vaind would you be able to spare some time to help look over this (and maybe a couple other) Dart PRs? I am happy to help get them merged. I see you reviewed this one already, but @NotTsunami has refactored it since then. @NotTsunami there is also a new conflict, would you be up for pushing a fix? |
vaind
left a comment
There was a problem hiding this comment.
Most of this looks OK except for alignment changes. I'm a bit rusty on this but am pretty sure these would break.
Therefore I've asked CC to review all the _prepare() calls in relation to how alignment should work in FlatBuffers spec. FlatBuffers Dart: Alignment Bug Analysis & DemonstrationDate: 2025-12-01 Executive Summary
Impact: Runtime crashes ( Fix: Revert to match the Int64/Uint64 pattern. Table of Contents
Bug Demonstration with TestsNew Test:
|
| Type | Size | Required Alignment |
|---|---|---|
| uint8/int8/bool | 1 byte | 1 byte |
| uint16/int16 | 2 bytes | 2 bytes |
| uint32/int32/float32 | 4 bytes | 4 bytes |
| uint64/int64/float64 | 8 bytes | 8 bytes |
Vector Alignment Rules
From C++ flatbuffer_builder.h:
template<typename T>
void StartVector(size_t len, size_t elemsize, size_t alignment) {
NotNested();
nested = true;
// Align to the Length type (uint32 = 4 bytes)
PreAlign<uint32_t>(len * elemsize);
// Then align to the element's alignment requirement
PreAlign(len * elemsize, alignment);
}Critical rule: Vector data must be aligned to the element's alignment requirement, not just the length field's alignment.
Vector Memory Layout
[padding] [element_data (aligned to element_size)] [4-byte length header]
^--- Must be aligned to element's requirement
For Float64 vectors: data must start at 8-byte aligned address.
Pattern Analysis: All _prepare() Uses
✅ Pattern 1: Single Scalars (CORRECT)
Used by: addBool, addInt32, addFloat64, putUint64, etc.
_prepare(_sizeofFloat64, 1); // Align to 8 bytes, write 1 float64
_prepare(_sizeofInt32, 1); // Align to 4 bytes, write 1 int32
_prepare(_sizeofUint8, 1); // Align to 1 byte, write 1 uint8Why correct:
- Aligns each scalar to its natural size
- Sets
_maxAlignappropriately
✅ Pattern 2: Int64/Uint64 Vectors (CORRECT)
Location: Line 589, 604
// writeListInt64
_prepare(_sizeofInt64, values.length, additionalBytes: _sizeofUint32);
// writeListUint64
_prepare(_sizeofUint64, values.length, additionalBytes: _sizeofUint32);Calculation:
size = 8→ aligns to 8 bytescount = N→ space for N int64 elementsadditionalBytes = 4→ plus 4-byte length header- Total space:
8 × N + 4bytes - Sets
_maxAlign = 8✓
Memory layout after writing:
[int64_N] ... [int64_1] [4-byte length] <- All int64s are 8-byte aligned
✅ Pattern 3: Int32/Uint32 Vectors (CORRECT)
Location: Line 619, 634
// writeListInt32
_prepare(_sizeofUint32, 1 + values.length);
// writeListUint32
_prepare(_sizeofUint32, 1 + values.length);Calculation:
size = 4→ aligns to 4 bytescount = 1 + N→ length field + N elements- Total space:
4 × (1 + N)bytes - Sets
_maxAlign = 4✓
✅ Pattern 4: Int16/Uint16/Int8/Uint8 Vectors (CORRECT)
Location: Line 649, 664, 684, 699
// writeListInt16
_prepare(_sizeofUint32, 1, additionalBytes: 2 * values.length);
// writeListInt8
_prepare(_sizeofUint32, 1, additionalBytes: values.length);Why correct:
- Aligns to 4 bytes (over-aligned, but safe)
- 4-byte alignment satisfies 2-byte (int16) and 1-byte (int8) requirements
- Sets
_maxAlign = 4✓
❌ Pattern 5: Float64 Vectors (INCORRECT)
Location: Line 557-558
// writeListFloat64
_prepare(_sizeofUint32, 1, additionalBytes: values.length * _sizeofFloat64);Calculation:
size = 4→ aligns to only 4 bytes ❌count = 1→ space for 1 uint32 (the length)additionalBytes = N × 8→ plus float64 data- Total space:
4 + 8 × Nbytes ✓ - But alignment is only 4 bytes! ❌
- Does NOT set
_maxAlign = 8! ❌
Problems:
- Float64 data requires 8-byte alignment, not 4-byte
_maxAlignstays at previous value (e.g., 1 or 4), not updated to 8- Buffer reallocations won't maintain 8-byte alignment
- Will fail on architectures requiring strict alignment
- Dart VM enforces 8-byte alignment for Float64List.view() and throws RangeError
⚠️ Pattern 6: Float32 Vectors (QUESTIONABLE)
Location: Line 573-574
// writeListFloat32
_prepare(_sizeofUint32, 1, additionalBytes: values.length * _sizeofFloat32);Calculation:
size = 4→ aligns to 4 bytes ✓ (sufficient for float32)- But uses different pattern than Float64 (inconsistent)
Why questionable:
- Works correctly (4-byte alignment is sufficient)
- But inconsistent with Int32/Uint32 and Int64/Uint64 patterns
- Should match other vector implementations for maintainability
The Critical Bug Explained
Concrete Example: Bug Manifestation
Scenario: Write a Float64 list after writing an Int32
Builder builder = Builder();
builder.putInt32(42); // Step 1: _tail = 4
int offset = builder.writeListFloat64([1.0, 2.0]); // Step 2: BUG!Trace Through Current (Broken) Code
Step 1: After putInt32(42)
_prepare(_sizeofInt32, 1); // size=4, count=1
// _tail = 0 + 4 = 4Step 2: writeListFloat64([1.0, 2.0])
_prepare(_sizeofUint32, 1, additionalBytes: 16);
// size = 4
// count = 1
// additionalBytes = 2 × 8 = 16
// dataSize = 4 × 1 + 16 = 20
// alignDelta = (-(4 + 20)) & (4-1) = (-24) & 3 = 0
// bufSize = 0 + 20 = 20
// _tail = 4 + 20 = 24Step 3: Writing data (assuming buffer size = 1024 bytes)
_setUint32AtTail(24, 2); // Length at buffer[1024 - 24] = buffer[1000]
tail = 24 - 4 = 20;
_setFloat64AtTail(20, 1.0); // Float at buffer[1024 - 20] = buffer[1004]
tail = 20 - 8 = 12;
_setFloat64AtTail(12, 2.0); // Float at buffer[1024 - 12] = buffer[1012]Step 4: Alignment verification
Assuming buffer starts at an 8-byte aligned address:
buffer[1000]→ length field (OK, alignment not critical)buffer[1004] % 8 = 4❌ Float64 NOT 8-byte aligned!buffer[1012] % 8 = 4❌ Float64 NOT 8-byte aligned!
Step 5: Reading back (CRASH!)
BufferContext buf = BufferContext.fromBytes(byteList);
List<double> items = const Float64ListReader().read(buf, 0);
// Tries to create: bc._asFloat64List(listOffset + 4, length)
// Which calls: _buffer.buffer.asFloat64List(offset, length)
// Dart VM checks alignment and throws:
// RangeError: Offset (996) must be a multiple of BYTES_PER_ELEMENT (8)Consequences
- ✅ PROVEN: Runtime crashes with
RangeError(demonstrated by test) - Undefined behavior on ARM, SPARC, MIPS (strict alignment required)
- Performance penalty on x86/x64 (unaligned access is slower)
- Cannot use native typed arrays (Dart enforces alignment for
Float64List.view) - Violates FlatBuffers specification - buffers may be incompatible with other implementations
Why Original Tests Passed (False Negative)
Current Test Code
From dart/test/flat_buffers_test.dart:536-551:
test('write and read Float64List', () {
List<double> values = [1, 2, 3.14159, -5.12345, 6, 7];
List<int> byteList;
{
Builder builder = Builder(initialSize: 0); // ⚠️ Empty builder!
int offset = builder.writeListFloat64(values);
builder.finish(offset);
byteList = builder.buffer;
}
// verify alignment
ByteData byteData = ByteData.view(Uint8List.fromList(byteList).buffer);
int vectorOffset = byteData.getUint32(0, Endian.little);
expect(vectorOffset, 4);
int floatDataOffset = vectorOffset + 4;
expect(floatDataOffset % 8, 0); // Checks 8-byte alignment
// ...
}Why It Passes
- Starts with empty builder:
_tail = 0 - Trace through:
_prepare(_sizeofUint32, 1, additionalBytes: 48); // dataSize = 4 + 48 = 52 // alignDelta = (-(0 + 52)) & 3 = (-52) & 3 = 0 // _tail = 0 + 52 = 52
- Data positions (in 1024-byte buffer):
- Length at position 52 → buffer[972]
- First float at position 48 → buffer[976]
- Subsequent floats at 40, 32, 24, 16, 8
- Alignment check:
- buffer[976] % 8 = 0 ✓ (by coincidence!)
- buffer[968] % 8 = 0 ✓
- All positions happen to be 8-byte aligned by luck
The test passes by coincidence because the specific values chosen happen to result in 8-byte aligned positions when starting from an empty buffer.
Correct Implementation
Float64 (Fixed)
/// Write the given list of 64-bit float [values].
int writeListFloat64(List<double> values) {
assert(!_inVTable);
// ✅ Match Int64/Uint64 pattern: align to 8 bytes
_prepare(_sizeofFloat64, values.length, additionalBytes: _sizeofUint32);
final result = _tail;
var tail = _tail;
_setUint32AtTail(tail, values.length);
tail -= _sizeofUint32;
for (final value in values) {
_setFloat64AtTail(tail, value);
tail -= _sizeofFloat64;
}
return result;
}Why correct:
size = 8→ Aligns to 8-byte boundaries ✓count = N→ Space for N float64 elements ✓additionalBytes = 4→ Plus 4-byte length header ✓- Sets
_maxAlign = 8✓ - Total space:
8 × N + 4bytes ✓ - Dart VM will accept Float64List.view() without RangeError ✓
Float32 (Fixed for Consistency)
/// Write the given list of 32-bit float [values].
int writeListFloat32(List<double> values) {
assert(!_inVTable);
// ✅ Match Int32/Uint32 pattern: align to 4 bytes
_prepare(_sizeofFloat32, values.length, additionalBytes: _sizeofUint32);
final result = _tail;
var tail = _tail;
_setUint32AtTail(tail, values.length);
tail -= _sizeofUint32;
for (final value in values) {
_setFloat32AtTail(tail, value);
tail -= _sizeofFloat32;
}
return result;
}Why correct:
- Consistent with Int64/Uint64 pattern
- 4-byte alignment is sufficient for float32
- Easier to maintain and understand
Recommendations
🔴 Critical (Must Fix Immediately)
-
Apply the correct fix for Float64/Float32
- Change to match Int64/Uint64 pattern (shown above)
- This is a correctness bug, not just a style issue
- Prevents runtime crashes
-
Verify the new tests pass after fix
cd dart dart test test/flat_buffers_test.dart --name "withNonZeroOffset"
Both tests should now pass.
-
Test buffer reallocation
- Ensure
_maxAlignis preserved across reallocations - Test with large float64 arrays that trigger buffer growth
- Ensure
🟡 High Priority
-
Verify backward compatibility
- Will existing serialized buffers break?
- The alignment change may alter buffer layouts
- Consider versioning or migration strategy
-
Add alignment verification assertions (optional, for debugging)
void _setFloat64AtTail(int tail, double x) { assert(((_buf.lengthInBytes - tail) % 8) == 0, 'Float64 must be 8-byte aligned'); _buf.setFloat64(_buf.lengthInBytes - tail, x, Endian.little); }
-
Document the alignment requirements
- Add comments explaining why each pattern is used
- Document
_maxAlignand its importance
🟢 Nice to Have
-
Consistency check for all vector types
- Consider refactoring to a single
_writeVector<T>()helper - Eliminate pattern variations
- Consider refactoring to a single
-
Cross-reference with other implementations
- Verify Dart matches C++, Java, Python alignment behavior
- Add cross-language compatibility tests
Summary Table: All _prepare() Uses
| Function | Line | Call Pattern | Alignment | Status |
|---|---|---|---|---|
| addBool | 200 | (1, 1) |
1 byte | ✅ |
| addInt32 | 211 | (4, 1) |
4 bytes | ✅ |
| addFloat64 | 304 | (8, 1) |
8 bytes | ✅ |
| putFloat64 | 418 | (8, 1) |
8 bytes | ✅ |
| writeListInt64 | 589 | (8, N, +4) |
8 bytes | ✅ |
| writeListUint64 | 604 | (8, N, +4) |
8 bytes | ✅ |
| writeListFloat64 | 557 | (4, 1, +8N) | 4 bytes | ❌ BROKEN |
| writeListFloat32 | 573 | (4, 1, +4N) | 4 bytes | |
| writeListInt32 | 619 | (4, 1+N) |
4 bytes | ✅ |
| writeListUint32 | 634 | (4, 1+N) |
4 bytes | ✅ |
| writeListInt16 | 649 | (4, 1, +2N) |
4 bytes | ✅ |
| writeListUint16 | 664 | (4, 1, +2N) |
4 bytes | ✅ |
| writeListInt8 | 684 | (4, 1, +N) |
4 bytes | ✅ |
| writeListUint8 | 699 | (4, 1, +N) |
4 bytes | ✅ |
| writeString | 745, 766 | (4, 1, +len) |
4 bytes | ✅ |
Impact Assessment
Affected Code Paths
-
Any code that writes Float64 lists after other data
- Tables with multiple fields
- Nested structures
- Complex buffers
-
Buffer reallocation scenarios
- When
_maxAlignis not set to 8 - Large buffers that trigger growth
- When
-
Cross-language compatibility
- Buffers created by Dart may be unreadable by C++/Java/Python
- Buffers created by other languages may not use Dart's native
Float64List
Not Affected
- Float64 lists in empty builders (by coincidence)
- Float64 lists as the first data written (by coincidence)
- Non little-endian systems (uses fallback
_FbFloat64Listwrapper)
Conclusion
The recent Float32/Float64 "fix" (commit 9210c38b) introduced a critical alignment bug that has been proven with a failing test:
- ✅ Bug confirmed:
test_writeList_ofFloat64_withNonZeroOffset()fails withRangeError - ✅ Root cause identified: Using
_prepare(_sizeofUint32, 1, ...)instead of_prepare(_sizeofFloat64, N, ...) - ✅ Correct fix documented: Match the Int64/Uint64 pattern
- ⬜ Next step: Apply the fix and verify all tests pass
The test suite now definitively proves the bug exists and will verify when it's fixed.
References:
- FlatBuffers alignment specification: C++
flatbuffer_builder.h - Dart implementation:
dart/lib/flat_buffers.dart - Test file:
dart/test/flat_buffers_test.dart - Commit with bug:
9210c38b- "Return dart:typed_data equivalent for Float32List and Float64List" - Fix commit:
b0a37007- "Fix failing test"
Test commands:
# Reproduce the bug (should fail)
cd dart && dart test test/flat_buffers_test.dart --name "test_writeList_ofFloat64_withNonZeroOffset"
# After fix (should pass)
cd dart && dart test test/flat_buffers_test.dart --name "withNonZeroOffset"|
@vaind Should be resolved now, added the asserts and new test cases as well as rebased over master |
Stems from discussion in #8183. This should allow users to be able to access a Uint8List directly without copying over from a list, thus actually offering zero-copy access (for this class, at least) in both the lazy path and the non-lazy path. This is pretty important for signal processing and image processing where we want to avoid any unnecessary list copies.