Reinforcement learning FPGA implementation related resources

Background

I research energy efficient reinforcement learning (RL). Here I compiled implementation-related resources, e.g., frameworks, libraries, readings.

Reading

• Two introductory exercises

  1. Learning value functions

  2. Neural network weight updates using gradient descent

  Solutions

• Chapter about RL from the book Arificial intelligence

  • free book from University of Columbia

  • if you want a compact and different view than the standard RL book by Sutton

• an object-oriented approach to linear neural networks

  • from an open source book about deep learning

  • does not implement neural networks from scratch, but uses many popular ML libraries

• Deep Learning from Scratch

  • implements neural networks from scratch in Python

  • not free

  • once prepared slides for a short lecture based on the book. Here is code used in the slides.

• Neural Networks from Scratch

  • similar to the last

  • not free

  • has many useful animations

Frameworks, Libraries, tools

• FINN

  • neural network inference on FPGAs

  • quantized neural networks

  • FINN examples

    • even the project is active, no newer boards like PYNQ-Z2 or Alveo U50 are tested.

• QONNX

  • introduces quantized operators for ONNX

• brevitas

  • quantized implementations of the most PyTorch layers, e.g., QuantConv1d

  • enables low-precision arithmetic (8bit, 4bit etc) which reduces DSP and memory footprint usage

• QKeras

  • quantized implementations of Keras layers

  • e.g., smooth_sigmoid(x), hard_sigmoid(x) …

  • includes an energy consumption estimator

• Netron

  • ONNX model visualization

• rule4ml

  • resource and latency estimation for ML on FPGA

• hls4ml

  • converts neural network models to FPGA firmware

• HLSFactory

  • framework for HLSing many configurations of a design and comparing the results

• Vitis Libraries

  • Vitis libraries for HLS

  • docs

• Vitis AI

  • for flexible AI inference compared to Brevitas & FINN

  • compiled code is run on an a micro-coded DPU (deep learning processing units)

  • docs

• Ramulator

  • cycle-accurate RAM simulator including HBM

Neural network implementations

• Neural network on an FPGA

  • SystemVerilog

  • datatypes are fixed, probably no quantization possible

• NeuralNetworkAccelerator

  • SystemVerilog

  • datatypes are fixed, probably no quantization possible

• AccDNN

  • AI model to Verilog

  • by IBM, but not maintained

• SpinalHDL CNN accelerator

  • common operators in CNN

  • configurable multiplier etc

    • no documentation

• spatten

  • sparse attention for LLMs, contains a hardware implementation

  • includes a dot product implementation

Math libraries

• Chainsaw

  • hardware design library based on SpinalHDL

  • includes a systolic array

• SpinalHDL math library

  • floating point with user programmable exponent and mantissa

• PiMAC

  • a pipelined multiplier in SpinalHDL

Reinforcement learning implementations

• HeteroRL

  • RL via DPC++ (SYCL) and Torch

• Q learning acceleration

  • three stage pipeline

• Proximal policy optimization (PPO) implementation

  • not documented

• A Toolkit for benchmarking FPGA-accelerated Reinforcement Learning

  • not maintained

• Experiences with Vitis AI for deep RL

  • pruning increases performance

  • tried only on GPUs, not on FPGAs

• Vitis HLS based reinforcement learning code

  • no documentation

Universal robot arm as .NET WPF application