REPOSITORY > RESULTS

Doctoral dissertation

Automatically generating fast hardware implementations of deeply quantized neural networks

Author(s): Jure Vreča (Author), Anton Biasizzo (Supervisor)

Thesis defense date: 14.11.2025

Organization: MPŠ - Mednarodna podiplomska šola Jožefa Stefana

PID: 20.500.12556/ReVIS-14993

Views: 0 | Downloads: 0

Abstract

This thesis studies the automatic design of hardware for ultra-low latency inference of
neural networks. Such hardware fills a niche in the application of neural networks where
results on the order of a few microseconds are required. An example use case for such
hardware is the trigger system at the CERN Large Hadron Collider, which has to react
in microseconds to prevent a data buffer overflow due to a data stream of terabytes per
second. This is unachievable for most of today’s hardware, as it is not optimized for ultralow
latency, but rather for high throughput. In addition, modern hardware accelerators
almost universally use various forms of DRAM memory to store the parameter data of
neural networks. Such external memories use busses that incur a high latency penalty.
The key to achieving ultra-low latency is to remove the external memory, store all
data on-chip, and process the neural network in a highly parallel manner. However, this
is problematic because neural networks typically have a large number of parameters and
the storage capacity of the on-chip memory is limited. To address this issue, two neural
network optimization techniques are investigated: quantization and pruning. The quantization
techniques use advanced training methods that can quantize neural networks to use
parameters with very small bitwidth. The second optimization technique is pruning. It
focuses on removing unnecessary connections from the neural network and thus eliminating
redundant parameters. By using these two methods together, non-trivial neural networks
can be implemented on modern FPGA circuits in a highly parallel manner.
In this work, we develop a methodology for implementing highly parallel hardware
implementations of quantized neural networks based on hardware generators that use
structural descriptions. One of the goals of this approach is to provide an alternative
to solutions based on high-level synthesis algorithms that have difficulty scaling to larger
neural networks, since such algorithms have polynomial time complexity. We develop an
open source tool based on the Chisel Hardware Construction Language that is a practical
implementation of our method and is capable of generating direct circuit implementations
of quantized and pruned neural networks. We compare our tool with similar tools based
on high-level synthesis technology.

Attachments

Cite this work