How focus stacking software works: image alignment

This is the fourth of a series of posts about taking focus stacked images of insect specimens to illustrate identification guides. The first part was a general overview, the second a description of the setup I use to acquire the stack and the third concerned lighting the specimen. In this post I will start to consider the focus stacking software.

The job of the software is to find the image in the stack where any given point is best focussed and then to combine them into a single, composite image which is as focussed as possible throughout. Obviously, it is not magic - if some feature is not in focus in any of the images in the stack, then it will not be in focus in the final, composite image! The process has the following steps:

1. Align the images in the stack so that any given feature is in exactly the same position in each image,
2. Find the image in which any given point is best focussed,
3. Combine the best focussed parts into a single, composite image.

In this post, I will consider the first step: image alignment.

Why is image alignment necessary?

Since the final, composite image will be built from pixels selected from different images up and down the stack, if features are not in the same place in each of these images, the result will be a confused mess!

Result of combining an unaligned stack

 

If the camera was mounted on a focus rail when the stack was acquired, why aren't the images already aligned?

Well they probably are - reasonably, but inevitable not exactly.

Firstly, the magnification of the subject inevitably changes as the focal point is changed. This is true whether you move the camera towards or away from the specimen, or fix the camera and change the focus point of the lens. The amount the magnification changes is small, but will noticeably degrade the finished image if not corrected (for example, the multiple rims of the eye in the image above).

Secondly, it is very difficult to fix the camera on a focus rail so that its axis is exactly aligned with that of the rail. If their axes are not exactly aligned, as you move the camera backwards and forwards, the image will shift slightly across the field of view. Given the magnification involved, especially if you are working at large reproduction ratios, even the tiniest deviation from true will be magnified to a noticeable shift.

Finally, if you take the stack hand held in the field, for example by using the continuous shooting mode of your camera to take a series of images as you rock backwards and forwards, then you may also rotate the camera slightly in the process. Rotation should not be an issue if the camera is fixed on a tripod and/or focus rail.

How does alignment work?

There are a number of possible approaches, but the software I wrote tackled this task as follows:

1. Extract the luminance data from each image,
2. Extract edges from each image (for example, Laplacian convolution),
3. For each pair of images down the stack (A:B, B:C, C:D, etc.) calculate a cross-correlation between (a sample of) pixels luminances and vary the x and y shift, magnification and optionally, rotation, of one of the images to maximise this cross correlation (for example, using the simplex algorithm).
4. This results in a series of estimates of x,y shift, magnification and potentially rotation changes between successive pairs of images.
5. A new set of images can then be written to temporary storage consisting of a straight copy of the first image and then a re-interpolation of each subsequent image with the estimated x,y shift, magnification and rotation changes applied.

An original image (left), luminosity (centre) and edge extraction (right - colour inverted for clarity)

This can be very nicely visualised by making the images in the stack into the frames of an animation. This is a useful way of checking that the stack has been successfully aligned. If it has, the resulting animation, after alignment, should show no shifts in the image - just the focal point moving through it.

Here is a before and after animation of a rather extreme case. This stack shows a much large lateral shift between images than is typical because it was taken using a camera mounted on one eyepiece of a binocular microscope. Focusing up and down causes quite a bit of lateral shift in the image in this case, because of the bent image path inherent in the design of a binocular microscope:

Before alignment - this stack shows a rather extreme degree of lateral shift because the images were taken using a camera mounted on one eyepiece of a binocular microscope.

After alignment (using align_image_stack.exe).

In a previous post,  I noted that align_image_stack.exe from the Hugin package does a very good job in this respect. Most focus stacking software aligns the images as part of its processing.It may offer the option of turning correction for rotation on or off. If you know that rotation is not a factor (the camera was fixed on a tripod and/or focussing rail) then it is worth turning rotation correction off because that simplifies and speeds up the necessary calculations.

Image alignment can benefit greatly from parallel processing if you computer has multiple CPUs. As described above, the necessary changes are typically calculated for successive pairs of images through the stack and each of these calculations can be carried out by one of the computers processors quite independently. Consequently, two processors will take almost exactly half as long as one, and four processors almost exactly a quarter as long - it scales linearly. Several of the dedicated focus stacking packages take advantage of this to speed up processing if you have a computer with more than one CPU. (Hugin's align_image_stack.exe does not support parallel processing at the time of writing).

In the next post, we will look at finding the most focussed image.