Make your parallel NumPy code fast: the secret sauce
When dealing with parallel processing of large NumPy arrays such as image or video data, you should be aware of this simple approach to speeding up your code.
Multiprocessing versus Concurrency in Python
First, a quick primer on some terminology.
In Python, if we want to take full advantage of the processing power of your CPU, you need need to use multiprocessing (typically achieved via the multiprocessing library). This library is therefore well suited for CPU intensive tasks. If we wish to efficiently do many things at once using a single processor, i.e. achieve concurrency, we can use Python’s libraries for asynchronous work — namely threading or asyncio. In both cases, a common programming practice for sharing information safely between the processes/threads is via the use of a Queue.
There are two types…
Make your video reading code faster
If you are working with video in Computer Vision, these libraries might help you make your code more efficient with only a few lines of code.
The typical way to read videos in Python
A typical pattern for reading videos in Python is using the OpenCV:
Are there any problems with this approach? It is entirely functional, but there are a couple of limitations:
- It is not optimised for speed
- OpenCV is not designed as a media library. This one may surprise you (it certainly did me!) but the developers only added the ability to read videos like this for “demo purposes” initially
[The] main purpose of OpenCV library is not highly correlated with [the] goals of [a] Media I/O library. Video I/O and Image I/O things added into OpenCV for demo purposes…
Benjamin Lowe
AI Engineer and Scientist
