Real-Time Multi-Camera Smart City Surveillance System

Real-Time Multi-Camera Smart City Surveillance System is a large urban development project in Bien Hoa, Dong Nai, Viet Nam. My company, part of Nova Group, developed a computer vision system to support city monitoring across multiple camera locations.

The system supported several practical computer vision applications, such as red-light violation detection, license plate recognition, crowd detection, and face recognition. The table below summarizes the core applications discussed in this case study and the initial vision approach used for each.

The full production system included additional applications, but these selected tasks are sufficient to explain the main architecture and engineering decisions.

Camera streams were pulled using GStreamer and sent to a server for AI processing. The processed results were then displayed to security operators. The system therefore had to connect video ingestion, multiple AI models, application logic, and real-time output within one stable pipeline.

The project was developed by a seven-member team. To preserve privacy, team members are represented by letters.

My main ownership: the HydraNet architecture and its integration, optimization, runtime behavior, and modular pipeline design.

The work was driven by two main requirements:

1. Build a computer vision system that could be deployed on both edge devices and servers.
2. Keep the system stable and accurate while making it practical to scale across a large number of cameras.

In practice, this created several engineering constraints at the same time: limited GPU memory, multiple AI tasks competing for compute, real-time latency requirements, different camera frame rates, and the need to keep the code maintainable as new applications were added.

A typical computer vision system streams video from a camera, sends frames to an AI model, and displays the result. When several AI applications are needed, the straightforward approach is to run separate models. For example, one model for object detection, another for face recognition, and another for classification.

From my analysis, these models could usually be viewed as two major parts: a backbone that extracts features and a task-specific neck/head that converts those features into the required output. The key issue was that the backbone represented a large share of the system resource cost, roughly 50–85% in my analysis. Running a separate backbone for every task therefore created a large amount of duplicated work.

To reduce this duplication, I proposed HydraNet: a design in which multiple tasks share the same backbone instead of loading a separate backbone for every model.
The backbone needed to balance accuracy and latency, so ResNet-50 was selected. The backbone was trained using the YOLO object-detection task. For the other tasks, the backbone was frozen and only the task-specific neck/head components were trained.

This design reduced the reported resource cost by more than 50% for the overall system because new tasks added their task-specific components without requiring another full backbone. The benefit became more visible as more AI tasks were integrated.

The trade-off was accuracy. Because the same backbone was reused across tasks, some models did not reach the accuracy reported by their original standalone implementations. In our practical evaluation, the applications remained above 90% accuracy, with face recognition above 95%, which was considered sufficient for the intended application.

The first HydraNet design still had three major limitations:

• Serial execution reduced overall FPS.
• Treating the model as one large block made debugging, updating, and resource control less flexible.
• The architecture was harder to scale and its outputs were harder to synchronize across parallel tasks.

To address these limitations, I split the system into multiple processes and threads. Each process handled a group of cameras, while individual model tasks were separated into threads.
The backbone ran in its own thread, and the task-specific neck/head components ran in separate threads. This allowed independent stages to overlap instead of forcing every task to execute serially. As more applications were added, additional application threads could be connected to the pipeline.

The number of threads depended mainly on the tasks inside the pipeline. The number of processes depended on the available GPU, CPU, RAM, and VRAM. If one HydraNet process did not use all available resources, another process could be added to handle more cameras.
The AI pipeline was configured so that both the effective input rate and the processed output rate were typically around 5–10 FPS per camera, which was sufficient for the tracking, AI analysis, and monitoring tasks in this project. Additional processes were introduced only when the system still had enough available GPU, VRAM, CPU, and RAM capacity. We also deliberately avoided using the hardware at 100% utilization, leaving some resource headroom to absorb short-term workload spikes without causing instability, excessive frame drops, or out-of-memory errors.

GPU Memory Optimization

At the beginning of optimization, I used a single process so that resource behavior was easier to isolate. On a 4 GB GPU, the unoptimized HydraNet pipeline repeatedly ran out of CUDA memory, so I profiled the system stage by stage.

I first disabled later threads and enabled components one at a time. For example, the input thread first, then the backbone, and then the remaining tasks. This made it possible to observe how much RAM and VRAM each block added. Because the backbone consumed the largest share, I focused on it first and then applied the same principles to the other stages.

• Disabled gradient tracking with torch.no_grad() during inference.
• Deleted tensors that were no longer needed and cleared cached GPU memory when appropriate.
• Moved AI outputs back to the CPU with detach().cpu() when GPU processing was no longer required.
• Used lightweight image-processing methods instead of AI models for simple tasks such as motion or color detection.
• Reduced inference precision from FP32 to FP16 before integrating the models into HydraNet.

After these changes, the system used approximately 3.2–3.8 GB of VRAM and ran stably within the 4 GB limit.

Reducing Computation Cost and Improving FPS

The next stage focused on throughput. In a pipeline, the overall FPS is constrained by the stage that takes the longest to process. In HydraNet, one important bottleneck was the backbone thread, so I focused on reducing its processing time and then applied similar methods to the other stages.

• Converted the models to TensorRT for NVIDIA GPU inference. This produced a significant speed improvement in our tests.
• Reduced the input resolution from HD to 640 × 640 to improve speed while maintaining acceptable accuracy.
• Used FP16 inference instead of FP32 to reduce computation and memory cost.
• Split expensive pipeline stages into separate threads so that AI inference could run at the same time as supporting operations such as tensor copying, data preparation, or post-processing. This improved throughput, but it introduced a trade-off: higher FPS could come with higher end-to-end latency.
• Tested larger batch sizes instead of processing only single frames. Batching could increase throughput, but it also increased VRAM usage.

These optimizations were applied across the broader pipeline. The purpose of this case study is to show the main techniques rather than every implementation detail used during development.

Motion-Triggered Inference

Running expensive AI inference continuously is wasteful when the scene does not change. To reduce unnecessary processing, I used motion-triggered inference. A lightweight image-processing stage compared the current image with the background and triggered AI processing when motion was detected. Additional filtering was used to reduce noise.

Dropping Frames When Buffers Are Full

Frames were passed between threads using Python queues. During temporary processing peaks, a downstream stage could fall behind and its buffer could fill. Instead of allowing latency to grow continuously, the pipeline dropped older frames and prioritized newer ones. For this real-time use case, keeping the displayed information current was more useful than processing every historical frame.

FPS Control

Camera streams could arrive at different frame rates depending on the camera and network conditions. I added an FPS-control block to prevent the AI pipeline from receiving frames faster than it could process them. Frames were sampled at a controlled rate so that the AI stages received a workload closer to their actual processing capacity, reducing random buffer overflow and unnecessary latency.

Output Synchronization

Parallel tasks do not finish at the same time, so combining their results into one output frame required synchronization. I reduced unnecessary copying by passing NumPy references so that multiple threads could work and draw results with the same underlying image object.
The remaining challenge was timing. For example, if one task finished in 0.1 ms and another in 0.3 ms, the first result could be ready before the second task had finished drawing its output.
The main implementation treated the object-detection application as the priority output thread because it was the longest-running task in the HydraNet pipeline. The final displayed reference was taken from that thread, so faster tasks had time to finish and draw their results onto the shared image before the object-detection output was released.
I also tested a mutex-based alternative in which completed tasks waited until the slowest task finished before results were merged. This approach worked in testing but was not integrated into the final pipeline.

As the number of threads increased, I organized the system as a sequence of pipeline blocks instead of allowing every component to connect directly to every other component.

I thought of the information moving through the pipeline as a shared packet. Each packet carried two main groups of data:

• Fundamental information, such as the image and its ID.
• AI information produced by the previous pipeline stages.

These values were stored in a dictionary using consistent field names across the different threads. Keeping the same structure made the data easier to track and reduced the amount of thread-specific handling code.

Threads were connected and synchronized using queue and condition objects. I wrapped this connection logic in separate functions instead of passing all synchronization objects directly through every thread constructor. This kept the pipeline structure clearer as the system grew.

In addition to designing and optimizing HydraNet, I also guided and supported other team members in developing applications around YOLO and related vision methods. The table below summarizes the main approaches described in this case study.

The work produced several practical outcomes reported in this draft:

• The shared-backbone design reduced the reported system resource cost by more than 50%.
• After GPU-memory optimization, HydraNet ran at approximately 3.2–3.8 GB VRAM on a 4 GB GPU.
• The shared-backbone approach maintained more than 90% accuracy across the applications in the practical evaluation, with face recognition above 95%.
• The final system ran on an NVIDIA A5000 with 24 cameras at the same time and achieved approximately 5–10 FPS per camera.

The final design was shaped by several practical trade-offs rather than by a single optimization target:

• Sharing one backbone reduced resource cost, but some task-specific models did not reach the accuracy of their original standalone implementations.
• Adding parallel stages could improve throughput, but higher throughput could also increase end-to-end latency.
• Larger batch sizes could increase processing throughput, but required more VRAM.
• Adding more processes increased camera capacity only while CPU, GPU, RAM, and VRAM headroom remained available.
• The main output-synchronization strategy relied on the object-detection path being the longest-running task; a mutex-based alternative was tested but not used in the final pipeline.