Today I'll show you how to create an image mosaic generator using python, OpenCV and numpy. But, what is an image mosaic you may ask? Here's an example with the picture above.
In this picture, every “pixel” is an actual image. Isn't it cool?
This project will make use of the following principles :
- Image processing
If you want to follow along I'll detail every step, or you can grab the finished code here on my Github. The readme is pretty detailed and should give you everything to install the required packages and use the application.
Going through the image
The first thing we want to do is go through our target image to look at the colors.
First, we'll read the image with opencv and it will be stored as a 2D numpy array.
import cv2 as cv target_im = cv.imread("path/to/image.jpg")
Actually, it will have three arrays stacked together, one for each R,G,B channel. To see that, print the shape of the array:
print(target_im.shape) >>> (512,512,3)
As we can see, the image is 512x512 pixels by 3 channels. We can now just go through the image like we would with a normal 2D array.
There's two thing we can do here.
- The first way is to take each pixel, get the color, and find a image closest to that pixel. The resulting image will have the entire matched image as a replacement for this pixel. Obviously, we can resize the image when replacing the pixel, otherwise the resulting image would be gigantic.
- The second option is to set a “patch” size that will go through the image. It will select multiple pixels on which we can apply a clustering algorithm to get the most dominant colors. By doing that, we pixelate the resulting image, because we take multiple pixel to create only one.
In the final code, I actually mixed both techniques, that way, it's possible to set the output resolution, and the pixel density (the size of the patch).
What is clustering?
Before showing you the code for this part, let me brief you quickly on clustering. If you're not familiar with it, it's basically a technique to gather unlabelled data into clusters. There exist a lot of algorithm for that, but a well known and very simple to use is K-means. Its main advantage is the ability to set the number of clusters we desire. Why use clustering?
Why bother to use a complex algorithm to find the dominant color of an image? Can't we simply use the average? Well, the answer is in the question, we are talking here about DOMINANT color, and not average! This is because of the way we perceive colors.
Still not convinced? Let me show you an example:
Aadam on its blogpost showed us the effect of average and clustering on an image.
On the left image, we see the average of all colors. As you can see, it's not what we feel as the dominant color. On the right, we took the centroid (the average) of the largest clusters of colors.
Note that because we take the centroid (the average) of the biggest cluster, if you set the number of cluster to one, you actually get the same result as the average.
Alright, now we understand why clustering is useful, let's have some code!
for i in range(row): for j in range(col): color = getDominantColor(target_im[y1:y2, x1:x2,:]) # Find the image that fits best the curr dominant color match = findBestColorMatch(color, dominant_colors) # Get the image im_match = images[dominant_colors.index(match)]
This piece of code is pretty simple, please note I removed the x and y calculation because they depend on if you take every pixel or a patch as I described earlier.
With the two for loops, we go through the pixels of the image.
getDominantColor() uses K-means to get the, well, dominant color, you guessed it! We then try to find a match between the calculated color and the image we have. When we have the match (the matching color), we can retrieve the actual image. After that, we can resize or do what we want to the image, before putting it into our resulting image.
And basically that's it! You can play with the size of the images, and the size of the “scanning” patch to see the effect on the resulting image.
We worked with RGB images, but it's much easier and gives better results if the images are in grayscale. With OpenCV, simply use:
grayscale = cv.cvtColor(im_RGB, cv.COLOR_RGB2GRAY)
Also, do not hesitate to resize your images before applying K-means, it will significantly increase the speed.
This post would be a bit short, so I might as well talk quickly about how I used multiprocessing to speed up the process. Indeed, running K-means on each image to get the dominant color can be a bit long. Remember, you have to use it on each patch of the target image, and also on each image used to construct the final one.
Here I'm talking about a CPU-bound task, that's why I use processes. If you want to dig a bit more the difference between processes and threads, I recommend you this article.
# Create a pool of thread # (same as number of cores) pool = Pool() # Create mapping for each image dominant_colors = pool.map(getDominantColor, images) pool.close() pool.join()
This little piece of code above will spawn as many processes as we have cores, because we didn't specify any number when calling
Pool(). Then we simply use
pool.map() to map a list to a function. In this case, images is a list of numpy arrays representing the images. The function will do its job of assigning images to processes and applying the function on them.
With this piece of code, I'm extracting the dominant colors of each image that will be used to create the target image. This process is super slow if done with one process. In the same way, you can use multiprocessing to process each patch of pixels when you go through your image. I actually did exactly that in the final code.
Keep in mind that spawning processes is a bit long, so the benefits will not be as big if your task is relatively short. In this case, there's a significant speedup, especially if you're working with a high number of images.
Now you have all the basics to create an image-mosaic generator!. Again, you can check the code here.