Know your camera - the aperture of the iris

In the previous post I described how the sharpness of my videocamera (Panasonic hc-x929/x920) varies as a function of it's f-stop. The conclusion was that the sharpness was optimal at two different f-stop values (f2.8 and f5.6) with a local minimum at f4 where the sharpness is reduced.
This behaviour is odd for a lens as I would expect just one f number where the sharpness is optimal with decreasing sharpness towards the extreme ends of the range. This motivated me to do another test to figure out how my iris changed as I changed the f-stop setting on the camera.

Have you noticed e.g. in films how out-of-focus streetlights in the background of a scene are often rendered as hexagons, octagons, circles or elipses? This is because a lens projects each out-of-focus point as a surface in the shape of the aperture. As different lenses have different implementations of their iris, these "point sources" essentially show you the shape of the aperture as created by the iris in your lens. They only show up when the light source is essentially a point source because then you only see one projection of your aperture. If the light comes from e.g. an entire building in the background than each point of that building creates a projection of the aperture and all these projections are superimposed onto each other. This will then create a blurred image of the building without the shape of the aperture being recognizable.
So one way of seeing the shape of the aperture is to take an out-of-focus image of a point source of light against a dark background. The smaller your light source is, the clearer the image of the aperture will be.

In order to see the shape of the aperture of my camera I took a halogen light and placed a piece of thin cardboard in front of it (at a safe distance, halogen lights can generate a lot of heat and radiation!). In the piece of cardboard I had pierced a small pinhole. Then I pointed my camera at the pin-hole in the cardboard and aligned it on a straight line with the pinhole and the light so I could see the brightest part of the light through the pinhole. Then I set the smallest f-number I could set (by first setting a fast shutter speed, after all it is a Panasonic camcorder, so no aperture priority mode here), switched to manual focus and chose a combination of zoom-factor and focusing distance which would give me the largest image of my aperture. I also had to experiment a little with the distance between the camera and the cardboard to determine what would work best.
Then I took a series of pictures, each with a different f-stop setting (and shutter speed to get proper exposure). This is the series of shots I got:

f 1.7

f 2.0

f 2.4

f 2.8

f 3.6

f 4.0

f 4.8

f 5.6

f 6.8

f 8.0

f 9.6

f 11

f 14

f 16

What are we looking at:

• Between f1.5 and f2.8 you can see how an 8-blade iris creates an aperture in the shape of an octagon which gets smaller as the f number increases. (Also notice the diffraction rings at the outer edges of the aperture).
• Between f2.8 and f5.6 a piece of smoked glass is shifted in front of the aperture covering a greater part of the aperture with increasing f number. The iris, and therefore the aperture does not change. Notice how the edges of the sheet of glass line up with the edges of the blades of the iris.
• Between f5.6 and f11 the aperture is reduced further as the blades of the iris contract further
• From f14 the aperture has changed to a shape more closely resembling a diamond and reduces further in size.

The "piece of smoked glass" which gets shifted in front of the aperture must be a built-in ND filter. Why was it designed in this way?
As indicated in my previous post, each lens has a "sweet spot" in it's aperture range where the sharpness of the image is optimal. Wide apertures soften the image due to spherical aberration and small apertures lead to diffraction diffusion. But this is a small-sensor camera. This means that the surface area onto which the image is projected is small when compared to other camera's, and the individual pixels on the sensor capturing the light must be very small given the resolution of the image. This implies that although the lens might be capable of projecting a particular part of the scene on just one pixel, the aperture will at fairly low f-numbers already smear this light out over neighboring pixels due to diffraction diffusion, thereby reducing the sharpness of the image.
If changing the f-number would only lead to a variation in aperture size, the range over which you could vary the f-number without sacrificing the sharpness of the image would become very small. Only the shutter speed would remain as a variable to control exposure.
To increase the useful range of f-stop settings the movable ND-filter is used. It allows you to control the amount of light passing through the aperture without having to change it's size. The aperture now stays in it's sweet spot between f-stops 2.8 and 5.6, that's a range of two stops. This implies the filter has a strength of ND4, i.e. only one-fourth of the total amount of light passes through it.
How does this explain the two distinct maxima in sharpness as a function of f-number? From the point the ND filter is shifted in front of the aperture you can consider the aperture to consist of essentially two apertures superimposed upon each other.

The first aperture has the shape and size of the physical aperture as determined by the iris blades. Only one-fourth of the total amount of light passes through it, i.e. the amount of light when the ND filter would be fully in place. Let's call this aperture 1. The second aperture consists of the area of the aperture as determined by the aperture blades which is not covered by the ND filter. This you could also call the "virtual" aperture. Initially, at f2.8 or lower f-numbers the remaining three-quarters of the light passes through this aperture. Lets'call this aperture 2. The final image as projected on the sensor is the two images of these two apertures superimposed upon each other.
Between f2.8 and f5.6 the image from aperture 1 is constant in both brightness and sharpness. The image from aperture 2 will decrease in brightness as the area which is not covered by the ND filter is reduced. As aperture 2 is reduced in size, the image produced by it will increasingly suffer from diffraction diffusion. As the ND filter is gradually shifted in front of aperture 1, aperture 2 produces initially the brightest image and therefore the reduction in sharpness due to diffraction difusion is clearly visible. However, as the ND filter shifts further, the contribution of aperture 2 to the total image brightness is gradually reduced to zero. Therefore it's detrimental effect on the total image also becomes less visible and eventually vanishes completely. This effectively leads to an initial reduction in sharpness which is restored once the ND filter has shifted completely in place and the "virtual" aperture 2 has closed completely. So this effectively explains the dip in sharpness.
The extension of the sweet spot of the lens by two stops comes at the price of a dip in sharpness during that extension. Incresing the strength of the ND filter could extend the range of the sweet spot even further but the dip in sharpness would be even more pronounced as the detrimental effect of virtual aperture 2 would be visible at smaller sizes of the virtual aperture due to it's relatively greater brightness compared to that of physical aperture 1. Likewise, reducing the strength of the ND filter would reduce the depth of the dip in sharpness but at the cost of reducing the range of extension of the sweet spot. I think this is an excellent demonstration of how the designers of this camera had to strike a balance between various design parameters. When it comes to designing an optical system, there are no free lunches; You can alter a design parameter to enhance the performance in one area, but you will suffer in another.
As a thought experiment let's see what would happen if you would want to upgrade this camera to UHD (or 4K) resolution. If you would want to keep the size, weight and cost essentially identical, the sensor size and size of the lens elements would have to remain the same. The increase in resolution would require the number of effective pixels to be increased by a factor of four or put differently, the diameter of each effective pixel would have to be halved. First, this would obiously lead to a reduction of the signal-to-noise ratio per pixel, assuming the increase in efficiency of a newer generation of sensor is less then a factor of four (which is a pretty safe assumption to make).
Second, assuming the lens itself would actually resolve the amount of detail required for this new generation of camera, any diffraction diffusion effects would already become vissible at wider apertures due to the smaller size of the pixels. To maintain an identical perceived sharpness you can no longer use the lowest end of the f-scale as spherical aberration will be more clearly visible. The usability of the high-end of the f-scale is also reduced as diffraction diffusion by the physical aperture will become visible at lower f-numbers as before. Furthermore, the amount of ND-filtering will have to be reduced in order to reduce the amount of diffraction diffusion caused by the virtual aperture, thereby reducing the extension of the sweet spot of the lens. In the end, you may very well end up with a camera which only allows you to vary the shutterspeed if you want the sharpness of your images to match the resolution.
What ways are there out of this? Well you could ofcourse increase the sensor size. This would imply an increase in the lens unit aswell, and therefore a total increase in size, weight and ofcourse cost. let's try another, not so obvious resolution; An increase in frame rate. A high shutter speed is only a problem when used in combination with a low frame rate as it leads to a stroboscopic effect. Suppose we could vary the frame rate along with our shutter speed. A shutter speed of 1/250 at 120 frames per second would simply lead to buttery smooth, natural looking motion. And as long as displays can not show us 4K images at 120 fps or for those who prefer a more filmic look at 30 fps we might even consider artificially reducing the framerate after they have been captured and deducing motion blurr from the differences between frames. Now I have to admit, this option has some challenges in itself. Increasing the readout speed (and therefore the framerate) of a sensor is anything but trivial. Encoding, storing and processing all these frames requires a lot of computing power and even when the framerate is artificially reduced to ease the encoding, transportation and storage requirements, the gains will propably be largely negated by the cost of creating artificial motion blur. You would at least end up with a camera that would eat batteries like there is no tomorrow and a big exhaustion fan to get rid af all the excess heat. I suspect the engineers would go with scenario one.
Well, at least we now have an explanation for the fact that we have two different f-stop values at which the sharpness is at it's maximum value and along the way we have gained a bit of empathy for the engineers who make these machines and have to make the tough compromises.