Deep Learning (DL) functions usually require processing video knowledge for duties comparable to object detection, classification, and segmentation. Nonetheless, standard video processing pipelines are usually inefficient for deep studying inference, resulting in efficiency bottlenecks. On this publish will leverage PyTorch and FFmpeg with NVIDIA {hardware} acceleration to realize this optimisation.
The inefficiency comes from how video frames are usually decoded and transferred between CPU and GPU. The usual workflow that we could discover within the majority of tutorials comply with this construction:
- Decode Frames on CPU: Video recordsdata are first decoded into uncooked frames utilizing CPU-based decoding instruments (e.g., OpenCV, FFmpeg with out GPU help).
- Switch to GPU: These frames are then transferred from CPU to GPU reminiscence to carry out deep studying inference utilizing frameworks like TensorFlow, Pytorch, ONNX, and so on.
- Inference on GPU: As soon as the frames are in GPU reminiscence, the mannequin performs inference.
- Switch Again to CPU (if wanted): Some post-processing steps could require knowledge to be moved again to the CPU.
This CPU-GPU switch course of introduces a major efficiency bottleneck, particularly when processing high-resolution movies at excessive body charges. The pointless reminiscence copies and context switches decelerate the general inference pace, limiting real-time processing capabilities.
For example, the next snippet has the everyday Video Processing pipeline that you simply got here throughout if you end up beginning to study deep studying:
The Answer: GPU-Primarily based Video Decoding and Inference
A extra environment friendly method is to preserve the whole pipeline on the GPU, from video decoding to inference, eliminating redundant CPU-GPU transfers. This may be achieved utilizing FFmpeg with NVIDIA GPU {hardware} acceleration.
Key Optimisations
- GPU-Accelerated Video Decoding: As a substitute of utilizing CPU-based decoding, we leverage FFmpeg with NVIDIA GPU acceleration (NVDEC) to decode video frames instantly on the GPU.
- Zero-Copy Body Processing: The decoded frames stay in GPU reminiscence, avoiding pointless reminiscence transfers.
- GPU-Optimized Inference: As soon as the frames are decoded, we carry out inference instantly utilizing any mannequin on the identical GPU, considerably decreasing latency.
Palms on!
Stipulations
To be able to obtain the aforementioned enhancements, we will likely be utilizing the next dependencies:
Set up
Please, to get a deep perception of how FFmpeg is put in with NVIDIA gpu acceleration, comply with these instructions.
Examined with:
- System: Ubuntu 22.04
- NVIDIA Driver Model: 550.120
- CUDA Model: 12.4
- Torch: 2.4.0
- Torchaudio: 2.4.0
- Torchvision: 0.19.0
1. Set up the NV-Codecs
2. Clone and configure FFmpeg
3. Validate whether or not the set up was profitable with torchaudio.utils
Time to code an optimised pipeline!
Benchmarking
To benchmark whether or not it’s making any distinction, we will likely be utilizing this video from Pexels by Pawel Perzanowski. Since most movies there are actually brief, I’ve stacked the identical video a number of instances to supply some outcomes with completely different video lengths. The unique video is 32 seconds lengthy which supplies us a complete of 960 frames. The brand new modified movies have 5520 and 9300 frames respectively.
Authentic video
- typical workflow: 28.51s
- optimised workflow: 24.2s
Okay… it doesn’t appear to be an actual enchancment, proper? Let’s take a look at it with longer movies.
Modified video v1 (5520 frames)
- typical workflow: 118.72s
- optimised workflow: 100.23s
Modified video v2 (9300 frames)
- typical workflow: 292.26s
- optimised workflow: 240.85s
Because the video period will increase, the advantages of the optimization turn into extra evident. Within the longest take a look at case, we obtain an 18% speedup, demonstrating a major discount in processing time. These efficiency positive aspects are significantly essential when dealing with massive video datasets and even in real-time video evaluation duties, the place small effectivity enhancements accumulate into substantial time financial savings.
Conclusion
In at this time’s publish, we’ve explored two video processing pipelines, the everyday one the place frames are copied from CPU to GPU, introducing noticeable bottlenecks, and an optimised pipeline, by which frames are decoded within the GPU and cross them on to inference, saving a significantly period of time as movies’ period improve.
