Skip to content

dart: Return dart:typed_data equivalent where possible#8289

Open
NotTsunami wants to merge 1 commit into
google:masterfrom
NotTsunami:master
Open

dart: Return dart:typed_data equivalent where possible#8289
NotTsunami wants to merge 1 commit into
google:masterfrom
NotTsunami:master

Conversation

@NotTsunami
Copy link
Copy Markdown
Contributor

@NotTsunami NotTsunami commented Apr 22, 2024

Stems from discussion in #8183. This should allow users to be able to access a Uint8List directly without copying over from a list, thus actually offering zero-copy access (for this class, at least) in both the lazy path and the non-lazy path. This is pretty important for signal processing and image processing where we want to avoid any unnecessary list copies.

@github-actions github-actions Bot added the dart label Apr 22, 2024
@jamesderlin
Copy link
Copy Markdown

jamesderlin commented Apr 22, 2024

ByteBuffer has an asInt8List method along with asUint8List, so for symmetry perhaps a Int8List _asInt8List(int offset, int length) method could be added to BufferContext, and _FbInt8List could be removed?

@NotTsunami
Copy link
Copy Markdown
Contributor Author

NotTsunami commented Apr 22, 2024

ByteBuffer has an asInt8List method along with asUint8List, so for symmetry perhaps a Int8List _asInt8List(int offset, int length) method could be added Buffeto rContext, and _FbInt8List could be removed?

I'm not opposed to that, but I'd like to see some insight from one of the Dart maintainers to see if this aligns with their view for the Dart side. Ideally, we could expand the same to Uint16ListReader to use their more space-efficient dart:typed_data alternatives (assuming those two follow the same pattern of being subtypes of List<int>), but then the question arises should those classes also offer lazy/greedy paths?

@NotTsunami NotTsunami marked this pull request as ready for review April 23, 2024 14:14
@NotTsunami
Copy link
Copy Markdown
Contributor Author

NotTsunami commented Apr 23, 2024

Ready for review. I tested using @tompark's example code, with the latest version of flatbuffers on pub.dev as well as this fork, and this fork prints the expected bundle1.image1 is already a Uint8List, len=29149 while the latest version on pub.dev does not. The only difference in my test path is that I manually executed the steps from setup.sh instead of writing a script. All tests are passing as well.

@NotTsunami NotTsunami changed the title dart: Always return Uint8List for Uint8ListReader dart: Return dart:typed_data equivalent where possible Apr 23, 2024
@NotTsunami
Copy link
Copy Markdown
Contributor Author

I squashed the changes and removed a change related to my other PR to prevent conflicts between the two.

@vaind Hi Ivan, is there any way you might be able to spare some bandwidth in the relatively near future to help review and potentially land both of my open PRs? Cheers.

bdero added a commit to bdero/flutter_scene that referenced this pull request May 15, 2024
Copy link
Copy Markdown
Contributor

@vaind vaind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

Comment thread dart/lib/flat_buffers.dart Outdated
Comment thread dart/lib/flat_buffers.dart Outdated
Comment thread dart/lib/flat_buffers.dart Outdated
@NotTsunami
Copy link
Copy Markdown
Contributor Author

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

@vaind
Copy link
Copy Markdown
Contributor

vaind commented May 18, 2024

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

I think it's OK to have different actual return type on big endian, the only contract is that it is compatible with List, everything else is "just" an optimization. It's OK for tests to assert the return value is the expected type based on endianness though

@NotTsunami
Copy link
Copy Markdown
Contributor Author

Thanks a lot. It looks pretty good to me, although there need to be some changes to make the code correct on Big endian machines (I know these are basically non-existent, but since Dart supports them, we should too).

I think I'd rather add a separate path or built-in for big-endian, otherwise the newly added conditions to the tests will cause the tests to fail on big-endian, as the old classes do not return Uint8List, Int8List, Uint16List, etc on the lazy path. I think the only supported test environment (theoretically) would be ARM on Linux running in big-endian mode? I'll try and set up a qemu environment to test.

I think it's OK to have different actual return type on big endian, the only contract is that it is compatible with List, everything else is "just" an optimization. It's OK for tests to assert the return value is the expected type based on endianness though

Sounds good to me. Sorry for the delay. I'll get this updated this week, hopefully on the earlier half as opposed to the later half.

Comment thread dart/test/flat_buffers_test.dart Outdated
Copy link
Copy Markdown
Contributor

@vaind vaind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Are you going to do the other lists (int32, ...) too or is this PR finished?

Comment thread dart/test/flat_buffers_test.dart Outdated
@NotTsunami
Copy link
Copy Markdown
Contributor Author

LGTM.

Are you going to do the other lists (int32, ...) too or is this PR finished?

I'll extend this to other types! I'll expand the tests as well.

@NotTsunami
Copy link
Copy Markdown
Contributor Author

@vaind I couldn't find out what is causing Float64List to fail, so I am going to revert the changes to the Float types and mark them with a todo to return their dart:typed_data equivalent so the PR can be restored to a working state and will be ready for review.

Copy link
Copy Markdown
Contributor

@vaind vaind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I cannot merge as I'm not a maintainer.

@NotTsunami
Copy link
Copy Markdown
Contributor Author

NotTsunami commented Feb 17, 2025

I had some time to sit down and review this again, and saw that the old code was causing potential misalignment in writeListFloat32 and writeListFloat64. Before we were treating all of the data as part of the count in _prepare, which technically does calculate the correct number of bytes but can cause misalignment, now we treat the length field correctly as a uint32 rather than as part of the float data, and specify the data as additionalBytes. I also let GH rebase this.

This is the final version of the PR, pending any code change requests from review. Hoping to see this actually go somewhere now,

@NotTsunami NotTsunami requested a review from vaind February 17, 2025 21:29
@NotTsunami
Copy link
Copy Markdown
Contributor Author

@aardappel Could you perhaps give this a look in spare time?

@aardappel
Copy link
Copy Markdown
Collaborator

aardappel commented Feb 17, 2025

@NotTsunami sorry, I have zero knowledge of Dart. I guess I am fine to merge if @vaind thinks it is ok.

@dbaileychess who probably know Dart better than I do.

Also, there is a failing test.

@NotTsunami
Copy link
Copy Markdown
Contributor Author

@NotTsunami sorry, I have zero knowledge of Dart. I guess I am fine to merge if @vaind thinks it is ok.

@dbaileychess who probably know Dart better than I do.

Also, there is a failing test.

Thanks. Should be fixed. Pending review on the latest version of the PR now, then.

@sumitsharansatsangi
Copy link
Copy Markdown

Please merge it soon.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Nov 3, 2025

This pull request is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

@github-actions github-actions Bot added stale and removed stale labels Nov 3, 2025
@jtdavis777
Copy link
Copy Markdown
Collaborator

@vaind would you be able to spare some time to help look over this (and maybe a couple other) Dart PRs? I am happy to help get them merged. I see you reviewed this one already, but @NotTsunami has refactored it since then.

@NotTsunami there is also a new conflict, would you be up for pushing a fix?

Copy link
Copy Markdown
Contributor

@vaind vaind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this looks OK except for alignment changes. I'm a bit rusty on this but am pretty sure these would break.

Comment thread dart/lib/flat_buffers.dart Outdated
@vaind
Copy link
Copy Markdown
Contributor

vaind commented Dec 1, 2025

I'm a bit rusty on this but am pretty sure these would break.

Therefore I've asked CC to review all the _prepare() calls in relation to how alignment should work in FlatBuffers spec.


FlatBuffers Dart: Alignment Bug Analysis & Demonstration

Date: 2025-12-01
Reviewer: Claude Code
Scope: Dart implementation alignment correctness review
Status: 🔴 CRITICAL BUG CONFIRMED


Executive Summary

⚠️ CRITICAL BUG FOUND: The recent "fix" for Float32/Float64 list serialization (commit 9210c38b) incorrectly implements alignment and only appears to work due to inadequate test coverage.

Impact: Runtime crashes (RangeError) when Float64 lists are written after other data.

Fix: Revert to match the Int64/Uint64 pattern.


Table of Contents

  1. Bug Demonstration with Tests
  2. How _prepare() Works
  3. FlatBuffers Alignment Specification
  4. Pattern Analysis: All _prepare() Uses
  5. The Critical Bug Explained
  6. Why Original Tests Passed (False Negative)
  7. Correct Implementation
  8. Recommendations

Bug Demonstration with Tests

New Test: test_writeList_ofFloat64_withNonZeroOffset()

Location: dart/test/flat_buffers_test.dart:564-609

Purpose: Demonstrates the Float64 alignment bug by writing data before the Float64 list.

Result:FAILS

RangeError: Offset (996) must be a multiple of BYTES_PER_ELEMENT (8)
dart:typed_data                                                    _ByteBuffer.asFloat64List
package:flat_buffers/flat_buffers.dart 72:10                       BufferContext._asFloat64List
package:flat_buffers/flat_buffers.dart 926:17                      Float64ListReader.read
test/flat_buffers_test.dart 603:52                                 BuilderTest.test_writeList_ofFloat64_withNonZeroOffset

Test Code:

void test_writeList_ofFloat64_withNonZeroOffset() {
  // This test exposes the alignment bug by writing data before the Float64 list
  List<double> values = <double>[1.0, 2.0, 3.0];
  List<int> byteList;
  {
    Builder builder = Builder();

    // Write an int32 first to create non-zero _tail
    // This causes Float64 data to be misaligned with the current buggy implementation
    builder.putInt32(42);

    int offset = builder.writeListFloat64(values);
    builder.finish(offset);
    byteList = builder.buffer;
  }

  // ... alignment verification ...
  expect(floatDataOffset % 8, 0,
         reason: 'Float64 array data must be 8-byte aligned per FlatBuffers spec. '
                 'Current implementation only aligns to 4 bytes, which fails '
                 'when _tail is not already 8-byte aligned.');
}

Why it fails:

  1. Test writes putInt32(42) first, setting _tail = 4
  2. Then calls writeListFloat64([1.0, 2.0, 3.0])
  3. Current buggy implementation: _prepare(_sizeofUint32, 1, additionalBytes: 24)
  4. This only aligns to 4 bytes, not 8 bytes
  5. Float64 data ends up at offset 996, which is NOT 8-byte aligned (996 % 8 = 4)
  6. Dart VM enforces alignment when creating Float64List.view()
  7. RangeError is thrown

New Test: test_writeList_ofFloat32_withNonZeroOffset()

Location: dart/test/flat_buffers_test.dart:641-681

Purpose: Verifies Float32 alignment with non-zero initial offset.

Result:PASSES

Why it passes:

  • Float32 requires 4-byte alignment
  • Current implementation aligns to 4 bytes
  • Test writes putInt16(42) first (2 bytes)
  • _prepare(_sizeofUint32, 1, ...) correctly aligns to 4 bytes
  • Float32 data is properly 4-byte aligned

Original Test: test_writeList_ofFloat64()

Location: dart/test/flat_buffers_test.dart:534-562

Result:PASSES (false negative)

Why it passes (incorrectly):

  • Uses empty builder: Builder(initialSize: 0)
  • _tail starts at 0
  • By luck, data positions end up 8-byte aligned
  • Does not catch the bug

Why test is inadequate:

  • ❌ Doesn't test with non-zero initial _tail
  • ❌ Doesn't test mixed data types (e.g., int32 before float64)
  • ❌ Doesn't verify _maxAlign is correctly set
  • ❌ Doesn't test buffer reallocation scenarios
  • ❌ Doesn't test on big-endian systems (uses fallback code path)

Running the Tests

To reproduce the bug:

cd dart
dart test test/flat_buffers_test.dart --name "test_writeList_ofFloat64_withNonZeroOffset"

Expected output:

RangeError: Offset (996) must be a multiple of BYTES_PER_ELEMENT (8)

After applying the fix:

dart test test/flat_buffers_test.dart --name "withNonZeroOffset"

Both tests should pass.


How _prepare() Works

Function Signature

void _prepare(int size, int count, {int additionalBytes = 0})

Location: dart/lib/flat_buffers.dart:805-834

Parameters

  • size: Element size AND the alignment requirement (must be power of 2)
  • count: Number of elements of size size
  • additionalBytes: Extra bytes beyond size × count (e.g., vector length headers)

Algorithm

void _prepare(int size, int count, {int additionalBytes = 0}) {
  assert(!_finished);

  // 1. Update maximum alignment tracking (critical for buffer reallocation)
  if (_maxAlign < size) {
    _maxAlign = size;  // ⚠️ Only updated if size > current _maxAlign
  }

  // 2. Calculate total space needed
  final dataSize = size * count + additionalBytes;

  // 3. Calculate padding needed for alignment to 'size' bytes
  final alignDelta = (-(_tail + dataSize)) & (size - 1);

  // 4. Total buffer space to allocate
  final bufSize = alignDelta + dataSize;

  // 5. Ensure capacity, reallocate if needed
  // ... (buffer growth code)

  // 6. Zero out padding bytes
  for (var i = _tail + 1; i <= _tail + alignDelta; i++) {
    _setUint8AtTail(i, 0);
  }

  // 7. Update tail pointer
  _tail += bufSize;
}

Key Insight: Alignment is ALWAYS to size

The formula alignDelta = (-(_tail + dataSize)) & (size - 1) ensures that after writing dataSize bytes, the buffer position is aligned to size-byte boundaries.

The size parameter determines both:

  • How many bytes per element (size × count)
  • What alignment boundary to use (size bytes)

FlatBuffers Alignment Specification

Based on official FlatBuffers documentation and C++ reference implementation:

Scalar Alignment Rules

Type Size Required Alignment
uint8/int8/bool 1 byte 1 byte
uint16/int16 2 bytes 2 bytes
uint32/int32/float32 4 bytes 4 bytes
uint64/int64/float64 8 bytes 8 bytes

Vector Alignment Rules

From C++ flatbuffer_builder.h:

template<typename T>
void StartVector(size_t len, size_t elemsize, size_t alignment) {
  NotNested();
  nested = true;
  // Align to the Length type (uint32 = 4 bytes)
  PreAlign<uint32_t>(len * elemsize);
  // Then align to the element's alignment requirement
  PreAlign(len * elemsize, alignment);
}

Critical rule: Vector data must be aligned to the element's alignment requirement, not just the length field's alignment.

Vector Memory Layout

[padding] [element_data (aligned to element_size)] [4-byte length header]
          ^--- Must be aligned to element's requirement

For Float64 vectors: data must start at 8-byte aligned address.


Pattern Analysis: All _prepare() Uses

✅ Pattern 1: Single Scalars (CORRECT)

Used by: addBool, addInt32, addFloat64, putUint64, etc.

_prepare(_sizeofFloat64, 1);  // Align to 8 bytes, write 1 float64
_prepare(_sizeofInt32, 1);    // Align to 4 bytes, write 1 int32
_prepare(_sizeofUint8, 1);    // Align to 1 byte, write 1 uint8

Why correct:

  • Aligns each scalar to its natural size
  • Sets _maxAlign appropriately

✅ Pattern 2: Int64/Uint64 Vectors (CORRECT)

Location: Line 589, 604

// writeListInt64
_prepare(_sizeofInt64, values.length, additionalBytes: _sizeofUint32);

// writeListUint64
_prepare(_sizeofUint64, values.length, additionalBytes: _sizeofUint32);

Calculation:

  • size = 8 → aligns to 8 bytes
  • count = N → space for N int64 elements
  • additionalBytes = 4 → plus 4-byte length header
  • Total space: 8 × N + 4 bytes
  • Sets _maxAlign = 8

Memory layout after writing:

[int64_N] ... [int64_1] [4-byte length] <- All int64s are 8-byte aligned

✅ Pattern 3: Int32/Uint32 Vectors (CORRECT)

Location: Line 619, 634

// writeListInt32
_prepare(_sizeofUint32, 1 + values.length);

// writeListUint32
_prepare(_sizeofUint32, 1 + values.length);

Calculation:

  • size = 4 → aligns to 4 bytes
  • count = 1 + N → length field + N elements
  • Total space: 4 × (1 + N) bytes
  • Sets _maxAlign = 4

✅ Pattern 4: Int16/Uint16/Int8/Uint8 Vectors (CORRECT)

Location: Line 649, 664, 684, 699

// writeListInt16
_prepare(_sizeofUint32, 1, additionalBytes: 2 * values.length);

// writeListInt8
_prepare(_sizeofUint32, 1, additionalBytes: values.length);

Why correct:

  • Aligns to 4 bytes (over-aligned, but safe)
  • 4-byte alignment satisfies 2-byte (int16) and 1-byte (int8) requirements
  • Sets _maxAlign = 4

❌ Pattern 5: Float64 Vectors (INCORRECT)

Location: Line 557-558

// writeListFloat64
_prepare(_sizeofUint32, 1, additionalBytes: values.length * _sizeofFloat64);

Calculation:

  • size = 4aligns to only 4 bytes
  • count = 1 → space for 1 uint32 (the length)
  • additionalBytes = N × 8 → plus float64 data
  • Total space: 4 + 8 × N bytes ✓
  • But alignment is only 4 bytes!
  • Does NOT set _maxAlign = 8!

Problems:

  1. Float64 data requires 8-byte alignment, not 4-byte
  2. _maxAlign stays at previous value (e.g., 1 or 4), not updated to 8
  3. Buffer reallocations won't maintain 8-byte alignment
  4. Will fail on architectures requiring strict alignment
  5. Dart VM enforces 8-byte alignment for Float64List.view() and throws RangeError

⚠️ Pattern 6: Float32 Vectors (QUESTIONABLE)

Location: Line 573-574

// writeListFloat32
_prepare(_sizeofUint32, 1, additionalBytes: values.length * _sizeofFloat32);

Calculation:

  • size = 4 → aligns to 4 bytes ✓ (sufficient for float32)
  • But uses different pattern than Float64 (inconsistent)

Why questionable:

  • Works correctly (4-byte alignment is sufficient)
  • But inconsistent with Int32/Uint32 and Int64/Uint64 patterns
  • Should match other vector implementations for maintainability

The Critical Bug Explained

Concrete Example: Bug Manifestation

Scenario: Write a Float64 list after writing an Int32

Builder builder = Builder();
builder.putInt32(42);               // Step 1: _tail = 4
int offset = builder.writeListFloat64([1.0, 2.0]);  // Step 2: BUG!

Trace Through Current (Broken) Code

Step 1: After putInt32(42)

_prepare(_sizeofInt32, 1);  // size=4, count=1
// _tail = 0 + 4 = 4

Step 2: writeListFloat64([1.0, 2.0])

_prepare(_sizeofUint32, 1, additionalBytes: 16);
// size = 4
// count = 1
// additionalBytes = 2 × 8 = 16
// dataSize = 4 × 1 + 16 = 20
// alignDelta = (-(4 + 20)) & (4-1) = (-24) & 3 = 0
// bufSize = 0 + 20 = 20
// _tail = 4 + 20 = 24

Step 3: Writing data (assuming buffer size = 1024 bytes)

_setUint32AtTail(24, 2);        // Length at buffer[1024 - 24] = buffer[1000]
tail = 24 - 4 = 20;
_setFloat64AtTail(20, 1.0);     // Float at buffer[1024 - 20] = buffer[1004]
tail = 20 - 8 = 12;
_setFloat64AtTail(12, 2.0);     // Float at buffer[1024 - 12] = buffer[1012]

Step 4: Alignment verification

Assuming buffer starts at an 8-byte aligned address:

  • buffer[1000] → length field (OK, alignment not critical)
  • buffer[1004] % 8 = 4Float64 NOT 8-byte aligned!
  • buffer[1012] % 8 = 4Float64 NOT 8-byte aligned!

Step 5: Reading back (CRASH!)

BufferContext buf = BufferContext.fromBytes(byteList);
List<double> items = const Float64ListReader().read(buf, 0);
// Tries to create: bc._asFloat64List(listOffset + 4, length)
// Which calls: _buffer.buffer.asFloat64List(offset, length)
// Dart VM checks alignment and throws:
// RangeError: Offset (996) must be a multiple of BYTES_PER_ELEMENT (8)

Consequences

  1. PROVEN: Runtime crashes with RangeError (demonstrated by test)
  2. Undefined behavior on ARM, SPARC, MIPS (strict alignment required)
  3. Performance penalty on x86/x64 (unaligned access is slower)
  4. Cannot use native typed arrays (Dart enforces alignment for Float64List.view)
  5. Violates FlatBuffers specification - buffers may be incompatible with other implementations

Why Original Tests Passed (False Negative)

Current Test Code

From dart/test/flat_buffers_test.dart:536-551:

test('write and read Float64List', () {
  List<double> values = [1, 2, 3.14159, -5.12345, 6, 7];
  List<int> byteList;
  {
    Builder builder = Builder(initialSize: 0);  // ⚠️ Empty builder!
    int offset = builder.writeListFloat64(values);
    builder.finish(offset);
    byteList = builder.buffer;
  }

  // verify alignment
  ByteData byteData = ByteData.view(Uint8List.fromList(byteList).buffer);
  int vectorOffset = byteData.getUint32(0, Endian.little);
  expect(vectorOffset, 4);
  int floatDataOffset = vectorOffset + 4;
  expect(floatDataOffset % 8, 0);  // Checks 8-byte alignment
  // ...
}

Why It Passes

  1. Starts with empty builder: _tail = 0
  2. Trace through:
    _prepare(_sizeofUint32, 1, additionalBytes: 48);
    // dataSize = 4 + 48 = 52
    // alignDelta = (-(0 + 52)) & 3 = (-52) & 3 = 0
    // _tail = 0 + 52 = 52
  3. Data positions (in 1024-byte buffer):
    • Length at position 52 → buffer[972]
    • First float at position 48 → buffer[976]
    • Subsequent floats at 40, 32, 24, 16, 8
  4. Alignment check:
    • buffer[976] % 8 = 0 ✓ (by coincidence!)
    • buffer[968] % 8 = 0 ✓
    • All positions happen to be 8-byte aligned by luck

The test passes by coincidence because the specific values chosen happen to result in 8-byte aligned positions when starting from an empty buffer.


Correct Implementation

Float64 (Fixed)

/// Write the given list of 64-bit float [values].
int writeListFloat64(List<double> values) {
  assert(!_inVTable);
  // ✅ Match Int64/Uint64 pattern: align to 8 bytes
  _prepare(_sizeofFloat64, values.length, additionalBytes: _sizeofUint32);
  final result = _tail;
  var tail = _tail;
  _setUint32AtTail(tail, values.length);
  tail -= _sizeofUint32;
  for (final value in values) {
    _setFloat64AtTail(tail, value);
    tail -= _sizeofFloat64;
  }
  return result;
}

Why correct:

  • size = 8 → Aligns to 8-byte boundaries ✓
  • count = N → Space for N float64 elements ✓
  • additionalBytes = 4 → Plus 4-byte length header ✓
  • Sets _maxAlign = 8
  • Total space: 8 × N + 4 bytes ✓
  • Dart VM will accept Float64List.view() without RangeError

Float32 (Fixed for Consistency)

/// Write the given list of 32-bit float [values].
int writeListFloat32(List<double> values) {
  assert(!_inVTable);
  // ✅ Match Int32/Uint32 pattern: align to 4 bytes
  _prepare(_sizeofFloat32, values.length, additionalBytes: _sizeofUint32);
  final result = _tail;
  var tail = _tail;
  _setUint32AtTail(tail, values.length);
  tail -= _sizeofUint32;
  for (final value in values) {
    _setFloat32AtTail(tail, value);
    tail -= _sizeofFloat32;
  }
  return result;
}

Why correct:

  • Consistent with Int64/Uint64 pattern
  • 4-byte alignment is sufficient for float32
  • Easier to maintain and understand

Recommendations

🔴 Critical (Must Fix Immediately)

  1. Apply the correct fix for Float64/Float32

    • Change to match Int64/Uint64 pattern (shown above)
    • This is a correctness bug, not just a style issue
    • Prevents runtime crashes
  2. Verify the new tests pass after fix

    cd dart
    dart test test/flat_buffers_test.dart --name "withNonZeroOffset"

    Both tests should now pass.

  3. Test buffer reallocation

    • Ensure _maxAlign is preserved across reallocations
    • Test with large float64 arrays that trigger buffer growth

🟡 High Priority

  1. Verify backward compatibility

    • Will existing serialized buffers break?
    • The alignment change may alter buffer layouts
    • Consider versioning or migration strategy
  2. Add alignment verification assertions (optional, for debugging)

    void _setFloat64AtTail(int tail, double x) {
      assert(((_buf.lengthInBytes - tail) % 8) == 0,
             'Float64 must be 8-byte aligned');
      _buf.setFloat64(_buf.lengthInBytes - tail, x, Endian.little);
    }
  3. Document the alignment requirements

    • Add comments explaining why each pattern is used
    • Document _maxAlign and its importance

🟢 Nice to Have

  1. Consistency check for all vector types

    • Consider refactoring to a single _writeVector<T>() helper
    • Eliminate pattern variations
  2. Cross-reference with other implementations

    • Verify Dart matches C++, Java, Python alignment behavior
    • Add cross-language compatibility tests

Summary Table: All _prepare() Uses

Function Line Call Pattern Alignment Status
addBool 200 (1, 1) 1 byte
addInt32 211 (4, 1) 4 bytes
addFloat64 304 (8, 1) 8 bytes
putFloat64 418 (8, 1) 8 bytes
writeListInt64 589 (8, N, +4) 8 bytes
writeListUint64 604 (8, N, +4) 8 bytes
writeListFloat64 557 (4, 1, +8N) 4 bytes ❌ BROKEN
writeListFloat32 573 (4, 1, +4N) 4 bytes ⚠️ Works but wrong
writeListInt32 619 (4, 1+N) 4 bytes
writeListUint32 634 (4, 1+N) 4 bytes
writeListInt16 649 (4, 1, +2N) 4 bytes
writeListUint16 664 (4, 1, +2N) 4 bytes
writeListInt8 684 (4, 1, +N) 4 bytes
writeListUint8 699 (4, 1, +N) 4 bytes
writeString 745, 766 (4, 1, +len) 4 bytes

Impact Assessment

Affected Code Paths

  1. Any code that writes Float64 lists after other data

    • Tables with multiple fields
    • Nested structures
    • Complex buffers
  2. Buffer reallocation scenarios

    • When _maxAlign is not set to 8
    • Large buffers that trigger growth
  3. Cross-language compatibility

    • Buffers created by Dart may be unreadable by C++/Java/Python
    • Buffers created by other languages may not use Dart's native Float64List

Not Affected

  • Float64 lists in empty builders (by coincidence)
  • Float64 lists as the first data written (by coincidence)
  • Non little-endian systems (uses fallback _FbFloat64List wrapper)

Conclusion

The recent Float32/Float64 "fix" (commit 9210c38b) introduced a critical alignment bug that has been proven with a failing test:

  1. Bug confirmed: test_writeList_ofFloat64_withNonZeroOffset() fails with RangeError
  2. Root cause identified: Using _prepare(_sizeofUint32, 1, ...) instead of _prepare(_sizeofFloat64, N, ...)
  3. Correct fix documented: Match the Int64/Uint64 pattern
  4. Next step: Apply the fix and verify all tests pass

The test suite now definitively proves the bug exists and will verify when it's fixed.


References:

  • FlatBuffers alignment specification: C++ flatbuffer_builder.h
  • Dart implementation: dart/lib/flat_buffers.dart
  • Test file: dart/test/flat_buffers_test.dart
  • Commit with bug: 9210c38b - "Return dart:typed_data equivalent for Float32List and Float64List"
  • Fix commit: b0a37007 - "Fix failing test"

Test commands:

# Reproduce the bug (should fail)
cd dart && dart test test/flat_buffers_test.dart --name "test_writeList_ofFloat64_withNonZeroOffset"

# After fix (should pass)
cd dart && dart test test/flat_buffers_test.dart --name "withNonZeroOffset"

@NotTsunami
Copy link
Copy Markdown
Contributor Author

NotTsunami commented May 19, 2026

@vaind Should be resolved now, added the asserts and new test cases as well as rebased over master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants