It is easy to describe an AI system by its model: the architecture, parameter count, accuracy, or number of operations. Those numbers matter, but they do not tell us whether the model can become a useful system on a constrained device.
Deployment changes the question. A model that looks efficient on paper may still miss its latency target, exceed the available memory, move too much data, or depend on numerical operations that are expensive on the target platform. At the edge, efficiency is not a property of the neural network alone. It emerges from the interaction between the model, its representation, the runtime, and the hardware.
The model is only the beginning
Consider a convolutional neural network intended for a microcontroller or FPGA. Before it can run, several practical questions have to be answered:
- Which numerical format should represent weights and activations?
- Where will intermediate feature maps be stored?
- How much data must move to and from external memory?
- How should work be divided into tiles that fit on chip?
- Can computation overlap with data movement?
- What happens when one layer has very different dimensions from the next?
These choices are connected. A smaller numerical format reduces storage and bandwidth, but it can also change accuracy and accumulator requirements. A larger tile may improve reuse, but it consumes more on-chip memory. More parallel compute can raise peak throughput, but it is useful only if the memory system can supply data at the required rate.
Optimizing one number in isolation can therefore move the bottleneck rather than remove it.
Why data movement matters
Multiply-accumulate operations are the visible part of a CNN accelerator, but moving operands and intermediate results often dominates the real cost. External-memory accesses consume time and energy, while on-chip memories are limited and have their own banking and bandwidth constraints.
This makes reuse central to accelerator design. If an input tile or a set of weights can remain on chip while multiple pieces of work consume it, external traffic falls. Achieving that reuse, however, requires the schedule and memory hierarchy to agree: buffers must be large enough, accesses must avoid conflicts, and downstream stages must keep pace with the producer.
The resulting performance is a property of the complete dataflow, not just the arithmetic unit.
The role of fixed-point representation
Quantization is another systems decision. Replacing floating-point values with carefully selected fixed-point representations can reduce memory use and make arithmetic substantially more hardware-friendly. But a successful deployment needs more than a quantized model file.
The software model, conversion tools, test data, and hardware implementation must share the same interpretation of scale, rounding, saturation, and accumulator width. Otherwise, a model that appears correct during training can behave differently after implementation.
My fixed-point FPGA workflow studies this bridge between numerical optimization and hardware realization. The goal is reproducibility across the entire path, rather than treating model quantization and accelerator implementation as separate exercises.
Flexibility has a cost—and a purpose
A highly specialized accelerator can be very efficient for one layer shape. Real networks, however, contain layers with different channel counts, spatial dimensions, kernels, and strides. Rebuilding the hardware for every change is rarely a practical deployment strategy.
My current TileCNN research explores a different balance: keep the compute hardware reusable and move layer-specific tiling and execution decisions into a software-generated schedule. This introduces the cost of generality, but it also creates a platform where scheduling, memory reuse, and bottlenecks can be studied across heterogeneous layers.
The important question is not whether flexibility is free—it is not. The useful question is how to make that flexibility predictable, bounded, and worthwhile.
Building from both directions
I approach efficient AI from both the model and hardware sides. TinyEmergencyNet began with the model-design question: how can an aerial-scene classifier become small enough for constrained deployment? FPGA work such as TileCNN begins closer to the execution question: how should data and computation move through a hardware accelerator?
The two directions meet in the same place. A hardware-aware model is most useful when it has a credible deployment path, and an accelerator is most useful when its assumptions match the models it needs to run.
That intersection—between algorithms and working systems—is the focus of my research and the main subject of this blog.