Multi-Camera Arrays Onboard: How FPGA/MPSoC Runs Real-Time AI Inference at the Edge
When four cameras on a line all feed back to a central server for detection, bandwidth cannot keep up and latency cannot be compressed. The better path is to do capture, preprocessing and inference on the board itself, sending back only detection results and a compressed preview. This article splits the edge onboard-AI architecture into PS/PL roles, with the bandwidth budget and model deployment path.
1. Why Push AI Inference Onto the Board
The legacy centralized approach on multi-camera lines: each camera feeds a switch, raw streams go back to a central server, and a server GPU runs detection on everything. With few cameras it works, but at more channels two problems appear: raw uncompressed bandwidth is huge, and centralized inference adds unavoidable network round-trip and queuing latency.
First the raw bandwidth math: a single 1080p30, YUV422 (2 bytes per pixel) raw stream = 1920×1080×2×30 ≈ 124 MB/s ≈ 995 Mb/s; four channels stack to about 4 Gb/s of uncompressed flow, before server-side decode, queueing and encoding overhead. At this scale the link cost and end-to-end latency of a centralized architecture are hard to bring down.
The onboard scheme reverses this: the board does preprocessing and inference locally, and only sends out structured results (bounding box, class, confidence, a few KB per frame) plus one compressed preview; raw pixels never leave the board.
2. Onboard AI Architecture: What PS and PL Each Handle
The Zynq UltraScale+ MPSoC places a quad-core A53 application processor (PS) and programmable logic (PL) in one chip, matching “scheduling plus system” with “parallel data paths.”
1. PL: Capture and Preprocessing Pipeline
Camera signals (MIPI CSI, USB or GigE) enter the PL, where hardware parallelism first runs denoising, scaling, color-space conversion and format normalization. These operations are naturally per-pixel and parallel; a PL pipeline does them far faster than a serial CPU and leaves the A53 free.
2. PL: Parallel Inference on Quantized Models
Convolution, pooling and activation run in the PL as a parallel array of DSP slices: weights sit in on-chip Block RAM/UltraRAM or PL-side DDR4, and input feature maps flow through the multiply-accumulate array. After INT8 quantization the model size and memory traffic drop sharply. The PL runs no OS; its deterministic pipeline holds a fixed beat per frame.
3. PS: Scheduling, Linux and Result Reporting
The PS quad-core A53 runs Linux for task scheduling, host communication and packaging results; the built-in VCU encoder simultaneously compresses one preview frame to H.264/H.265 and sends it over the network only when a person needs to view it.
3. Bandwidth Budget: Why Raw Multi-Camera Flow Cannot Go Back
| Link stage | Single 1080p30 | Four channels | Note |
|---|---|---|---|
| Raw uncompressed (YUV422) | ~995 Mb/s | ~4.0 Gb/s | Legacy full-return-to-server approach |
| Structured result after preprocess | a few KB/frame | a few KB/frame | box + class + confidence |
| Compressed preview (H.264) | 2–4 Mb/s | 8–16 Mb/s | Sent only when viewed |
Onboard processing cuts outbound bandwidth from Gb/s to Mb/s or even Kb/s; the central server receives results only and no longer reserves decode and inference per camera. This “raw pixels stay on board, only results leave” boundary is the core edge advantage over a centralized scheme.
4. Model Deployment and Board Selection
1. Quantize and Prune First
Quantize the model to INT8 on board, prune and cut weights to shrink volume; then split layer pipelines by on-chip storage and DSP capacity so the whole weight table need not enter the chip at once. A Vitis-AI style flow compiles, quantizes and deploys the model to PL acceleration units, while the PS Linux still hosts the application.
2. Pick a Board by Channel Count and Compute
Edge Vision Board Roles: Entry Prototype / Multi-Channel Compute / Pure-FPGA Parallel
- Entry edge prototype (DUUZ4EV, RFQ): XCZU4EV, 192K logic cells, 728 DSP, built-in VCU, PS 4GB + PL 4GB DDR4; for one-to-two camera edge AI prototypes and fast PYNQ validation.
- Multi-channel high compute (DUUZ5EV, RFQ): XCZU5EV, 256K logic cells, 1,248 DSP, 16 GTH pairs, FMC-LPC expansion; the platform for more channels and heavier models.
- Pure-FPGA parallel expansion (XCKU15P, ¥15,999): Kintex UltraScale+ core board for parallel multi-camera stream processing and high-speed aggregation where a PS system is not needed.
Edge AI inference is not shrinking the server GPU; it means slicing the whole “capture–preprocess–inference–report” chain on board: PL carries parallel pixel processing and quantized convolution, PS carries scheduling and communication, and the VCU compresses the preview along the way. Raw pixels stay on board and only results leave — both bandwidth and latency loosen up.
