BF16 Unified Function Accelerator on FPGA

English | 简体中文

A unified BF16 accelerator for high-frequency neural-network operators on the AMD Kria KV260. The design consolidates elementwise arithmetic, normalization, softmax, and activation functions into one reusable FPGA datapath, with a complete Vivado project and a PYNQ-based evaluation environment.

The accelerator targets the operators surrounding matrix multiplication in Transformer and modern neural-network workloads. By sharing arithmetic primitives and dispatching data through operation-specific pipelines, it provides a compact hardware platform for studying operator fusion, BF16 computing, and PS/PL collaboration.

Contents

• Highlights
• Supported Operations
• Architecture
• Data Organization
• RTL Design
• Control Interface
• Performance
• Repository Structure
• Requirements
• Vivado Workflow
• PYNQ Workflow
• Vector Generation and Verification

Highlights

• Seven operators in one accelerator — Elementwise Add, Elementwise Multiply, Softmax, LayerNorm, RMSNorm, SiLU, and GeLU.
• BF16 storage and interfaces — compact 16-bit tensor representation with internal FP32 conversion where additional arithmetic range is useful.
• Parallel data movement — eight 128-bit BRAM banks provide high-bandwidth access between the processing system and programmable logic.
• Reusable arithmetic building blocks — addition, multiplication, accumulation, maximum, exponentiation, reciprocal, and reciprocal-square-root units feed the operator pipelines.
• End-to-end KV260 stack — includes the Vivado 2025.1 project, exported XSA, PYNQ overlay files, Jupyter notebook, test-vector generator, and accuracy comparison tools.
• Simulation and board evaluation — the same 64 × 768 BF16 workload format is supported by the SystemVerilog testbench and the PYNQ host flow.

Supported Operations

Operator	Role in Neural Networks	Board Latency	Accuracy Score
Elementwise Add	Residual connections and tensor fusion	4,400 ns	1.000
Elementwise Multiply	Gating and pointwise scaling	4,921 ns	1.000
Softmax	Attention probability normalization	10,200 ns	1.000
LayerNorm	Feature normalization	25,530 ns	1.000
RMSNorm	Root-mean-square normalization	25,700 ns	1.000
SiLU	Smooth activation and gated MLPs	4,950 ns	1.000
GeLU	Transformer feed-forward activation	9,860 ns	1.000

The included comparison tool reports the relative L2 error, mean absolute error, maximum absolute error, differing-value count, and the normalized accuracy score (A_f). The table above summarizes the board-level measurements included with the project.

Architecture

┌──────────────────────────── Kria KV260 PS ────────────────────────────┐
│  PYNQ Notebook                                                       │
│  ├── loads bitstream and HWH                                         │
│  ├── packs 64 × 768 BF16 tensors                                     │
│  ├── accesses eight AXI BRAM controllers                             │
│  └── starts the accelerator and measures completion latency          │
└─────────────────────┬───────────────────────────┬─────────────────────┘
                      │ AXI4-Lite                 │ AXI BRAM
┌─────────────────────▼───────────────────────────▼─────────────────────┐
│                         Accelerator                                  │
│  ┌────────────────┐   ┌───────────────────────────────────────────┐  │
│  │ Control plane  │   │ Eight-bank tensor memory                  │  │
│  │ start/function │   │ 4 × input A | 4 × input B / output       │  │
│  └───────┬────────┘   └──────────────────┬────────────────────────┘  │
│          │                               │                           │
│  ┌───────▼───────────────────────────────▼────────────────────────┐  │
│  │ BF16/FP32 conversion and shared floating-point primitives      │  │
│  └───────┬────────────────────────────────────────────────────────┘  │
│          │                                                           │
│  ┌───────▼────────────────────────────────────────────────────────┐  │
│  │ Add | Mul | Softmax | LayerNorm | RMSNorm | SiLU | GeLU        │  │
│  └────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

The programmable-logic datapath is organized around three layers:

1. Memory scheduling — ram2fpu2ram_top selects an 8-, 16-, or 32-line transfer engine for the active operator.
2. Unified floating-point engine — line8_top_fpu dispatches converted input data through the selected operator pipeline.
3. Arithmetic primitives — dedicated modules implement add, multiply, accumulate, maximum, exponent, reciprocal, reciprocal square root, and BF16/FP32 conversion.

Data Organization

The reference workload uses two BF16 tensors:

Shape:            64 rows × 768 columns
Element width:    16 bits (BF16)
BRAM word width:  128 bits
Packing:          8 BF16 values per BRAM word
Memory banks:     8

Tensor rows are distributed across the BRAM banks:

BRAM	Input Mapping	Result Mapping
BRAM 0	Tensor A, rows 0–15	—
BRAM 1	Tensor A, rows 16–31	—
BRAM 2	Tensor A, rows 32–47	—
BRAM 3	Tensor A, rows 48–63	—
BRAM 4	Tensor B, rows 0–15	Output rows 0–15
BRAM 5	Tensor B, rows 16–31	Output rows 16–31
BRAM 6	Tensor B, rows 32–47	Output rows 32–47
BRAM 7	Tensor B, rows 48–63	Output rows 48–63

Within each bank, eight values from the same column are packed into one 128-bit word. The notebook provides matching pack and unpack routines, keeping software layout and RTL addressing aligned.

RTL Design

The main hand-written sources are located in PS_vivado/PS.srcs/sources_1/.

Module	Responsibility
Accelerator.v	Top-level AXI4-Lite control, BRAM integration, and compute datapath
axi_lite_reg_ctrl.v	Start, completion, operation selection, and reset registers
ram2fpu2ram_top.v	Selects the transfer engine for each operator
ram2fpu2ram_8.v	Eight-line memory-to-FPU scheduling
ram2fpu2ram_16.v	Sixteen-line memory-to-FPU scheduling
ram2fpu2ram_32.v	Thirty-two-line memory-to-FPU scheduling
fpu/line8_top_fpu.v	Unified operator dispatch and shared arithmetic routing
fpu/line8_*.v	Softmax, normalization, activation, and elementwise pipelines
fpu/alu_8line/*.v	Floating-point arithmetic and conversion primitives

The operator pipelines reuse a common set of ALU blocks:

• add.v and mul.v for arithmetic
• acu.v for accumulation
• max.v for reduction maximum
• exp.v for exponential evaluation
• rec.v and rsr.v for reciprocal and reciprocal square root
• bf16_2_fp32.v and fp32_2_bf16.v for format conversion
• fifo_8x16x1024.v and pipeline helpers for alignment and buffering

Control Interface

The accelerator is controlled through a compact AXI4-Lite register bank:

Offset	Register	Access	Description
0x00	Control	Read/Write	Write bit 0 to start; read bit 1 for completion
0x04	Function Select	Read/Write	Write the 3-bit function code; reads return the internal one-hot selection
0x30	Reset	Write	Write bit 0 to issue a global reset pulse

Function codes in the current RTL control block are:

Code	Operation
1	Softmax
2	LayerNorm
3	RMSNorm
4	SiLU
5	Elementwise Add
6	Elementwise Multiply
7	GeLU

The PYNQ notebook uses a small compiled helper, fast_poll.c, to start the core, poll the completion bit, enforce a timeout, and return nanosecond-resolution execution time.

Performance

At the 300 MHz PL clock configured by the evaluation notebook, all seven operators complete the 64 × 768 reference workload in 25.7 μs or less. Elementwise arithmetic and SiLU complete in approximately 5 μs, while the reduction-heavy normalization operators remain below 26 μs.

Category	Operations	Observed Range
Elementwise	Add, Multiply	4.40–4.92 μs
Activation	SiLU, GeLU	4.95–9.86 μs
Attention	Softmax	10.20 μs
Normalization	LayerNorm, RMSNorm	25.53–25.70 μs

Repository Structure

.
├── PS_vivado/
│   ├── PS.xpr                         # Vivado 2025.1 project
│   ├── design_1_wrapper.xsa           # Exported hardware platform
│   ├── PS.srcs/sources_1/             # Accelerator RTL and block design
│   ├── PS.srcs/sim_1/new/             # Testbench and comparison script
│   ├── PS.gen/                         # Generated block-design and IP output
│   └── PS.runs/                        # Synthesis results
├── project_PYNQ/Accelerator/
│   ├── HW/design_1.bit                # KV260 overlay bitstream
│   ├── HW/design_1.hwh                # PYNQ hardware metadata
│   ├── fpu32.ipynb                    # Board evaluation notebook
│   ├── fast_poll.c                    # Low-overhead latency measurement helper
│   ├── test_tensor_generator.py       # BF16 inputs and golden references
│   └── compare.py                     # Accuracy and error analysis
└── docs/
    └── README_zh-CN.md                # Chinese documentation

Requirements

Hardware

• AMD Kria KV260 Vision AI Starter Kit
• USB, Ethernet, or local Jupyter access to the board
• A PYNQ image compatible with the KV260

FPGA Tools

• AMD Vivado 2025.1
• KV260 board files and device support

Evaluation Environment

• Python 3
• PYNQ
• NumPy
• GCC, used by the notebook to build libfast_poll.so
• PyTorch, used by test_tensor_generator.py to produce reference tensors

Vivado Workflow

Open the Project

1. Start Vivado 2025.1.
2. Open PS_vivado/PS.xpr.
3. Confirm the target part xck26-sfvc784-2LV-c and board xilinx.com:kv260_som:part0:1.4.
4. Open the block design to inspect the Zynq UltraScale+ PS, accelerator IP, AXI interconnect, and eight BRAM controllers.

Run Post-Synthesis Simulation

The simulation environment is located in PS_vivado/PS.srcs/sim_1/new/.

1. In tb_accelerator.sv, set FILE_IN and FILE_IN2 to the two input-vector files.
2. Set the seven GOLDEN_REF paths to the corresponding reference outputs.
3. Update the compare.py invocation path used by the testbench.
4. In the simulation copy of compare.py, set the input-vector paths used for error diagnostics.
5. Launch post-synthesis functional simulation.

The testbench loads all eight BRAM banks, runs every supported function, writes each result to its simulation directory, and prints the accuracy comparison in the Tcl console.

PYNQ Workflow

Deploy the Overlay

Copy the complete accelerator directory to the KV260 Jupyter workspace:

project_PYNQ/Accelerator
    -> /root/jupyter_notebooks/Accelerator

The HW/ directory already pairs the overlay artifacts required by PYNQ:

• design_1.bit — programmable-logic bitstream
• design_1.hwh — IP and address metadata

Configure the Notebook

Open fpu32.ipynb and update:

1. The overlay path in the loading cell.
2. The input and output vector paths in the evaluation cell.
3. The generated-output and golden-reference paths in the comparison cell.

The notebook loads the overlay, sets fclk0 to 300 MHz, maps all eight BRAM controllers, transfers the BF16 tensors, launches each operator, reads back the results, and reports execution time.

Vector Generation and Verification

Generate Inputs and Golden References

test_tensor_generator.py creates deterministic 64 × 768 BF16 inputs and software reference outputs for all seven operations.

Set the seed and output directories near the beginning of the script, then run:

python test_tensor_generator.py

The generator covers representative random distributions, normalization patterns, activation ranges, and BF16 edge cases. Tensor values are serialized as four-digit hexadecimal BF16 tokens.

Compare Hardware Results

Use compare.py with a golden-reference file and a hardware-output file:

python compare.py reference.hex output.hex

The report includes:

• sample count
• mean absolute error
• maximum absolute error and its index
• relative L2 error
• normalized accuracy score (A_f)
• number of differing BF16 values
• finite and non-finite mismatch diagnostics