Know your camera - whitebalance

© STEPHEN WILKES

In the previous post I made a series of shots at different f-stops but equal exposure to observe the effect of the ND filter on the bokeh. The ND filter had no noticable effect on the rendering of the bokeh but I did notice something else in the series of pictures. The colors weren't entirely identical accross the series while the lighting was. It was an overcast day. The white balance of the camera was fixed at the "cloudy" preset so something else must have altered the colors. By loading the images in a graphics program and using the histogram and color picker tools I could see there was a slight shift in color towards cyan as the f-number increased. This could be explained by the ND-filter introducing a slight color cast to the image. ND filters are rarely exactly color neutral. If the built-in filter is slightly cyan, a slight color shift can be expected as the ND-filter covers a larger or smaller portion of the aperture.

f2.8

f5.6

If the hue of the ND filter is constant enough across individual copies, the camera engineers could let the camera automatically correct for the hue of the ND filter by tweaking the white balance setting as a function of f-number. Just nudge the white balance slightly towards red as the f-stop increases from f2.8 to f5.6 and the user would never notice a change in hue while the f-number is changed. Apparantly the camera does not perform such a correction. Or alternatively, it does, but there is a small margin of error, due to the variation in hue of the individual ND filters, and it is this margin of error observed here. I suspect it simply performs no correction.
So if we want to set the right white-balance, how should it be done? The first step consists of selecting what light source to balance on. This step may not always be as trivial as it initially seems. Let me demonstrate this by the following example.

Imagine you are standing on a square surrounded by white buildings. It is early morning, so the sun is still fairly close to the horizon. You want to make some shots of one particular building of which the wall facing the square is still in the shadow. You first want to take a total shot of the building so you position yourself at the other side of the square which is still bathing in the sun. By the time you have set up your camera and you look at the building you notice that the walls still have a bluish hue as they are still in the shadows while the walls closest to you appear white. In an effort to depict the building realistically you set the white balance in such a way that the walls close to you appear white, just as they appear to you. Then you record just the face of the building you wanted to film.
Next, you move closer to the building to shoot some closeups of various details of the building. You cross the square again and put up your camera in the shadow. By the time you are ready to make your first shot you notice that the wall of the building you want to shoot now appear white while the walls of the buildings on the opposite side of the square have a slight warm-yellow tone. Again you adjust your white balance again but now ensure that the walls closest to you again appear white.
While taking all these shots you have ensured that the colors of all images appear to the viewer as they have appeared to you. However, when the viewer views the end-result he will see a total of the building followed by a series of close-ups where the hue of the building has suddenly changed from one shot to the next. The viewer is unaware of the surrounding of the building and might not know when the camera was in the shadow and when in the sun.
The point of the previous example is that the choice of what light source to balance on depends on the contextual information the viewer has about the subject. It is ultimately a subjective choice which the camera operator has to make, depending on what the viewer will know or more precisely; what the director wants the viewer to know.
Once we know what light source we want to balance on we should let the camera measure the spectrum of the light source. Through either the color filters of the sensor (on a single chip bayer sensor) or the prism of a three-chip camera (like the Panasonic x929 I use) this spectrum will be split in three parts. Each part corresponds to a primary color channel of the camera (whose spectral range may partially overlap). It will measure the amount of light captured in each part. Then, usually assuming a spectral distribution of a black body or similar full-spectrum light source, it can infer the spectral distribution of the light source. An amplification factor will be applied to each channel to ensure that each channel will give off a signal of identical strength when they captures the pure, un-altered light from the light source.
In this process the camera makes an assumption about the spectral distribution of the light source. This assumption is correct for sunlight and roughly also to incandescent lights. Certain lightsources like fluorescent, led-lights or streetlights have a spectral distribution which is very different. Therefore if one whitebalances on such light sources, some colors may be recorded correctly, but others will not. Fortunately there are e.g. led-light sources which have been tweaked in such a way that they still give reasonably correct results despite the incorrect assumption of the camera. Furthermore, a camera may have special white balance pre-sets which take into account the particular spectral distribution of certain light sources. E.g. the fluorescent preset. For these particular light sources best results may be obtained by using the presets.
For full-spectrum light sources with a spectral distribution resembling that of sunlight we have to allow the camera to measure the light. Ideally we would simply like to point the camera at the light source and let it measure the light but unfortunately the lens will project an image of the light source on the sensor while we would like the light to be evenly distributed across the entire sensor. Fortunately there are tools to circumvent that problem.
One such tool is the white translucent lenscap or an expodisc like tool. Hold these in front of the lens while pointing the camera at the lightsource. These diffuse the light before it enters the lens causing an even distribution of all light across the image sensor.
A second tool is the grey card. Point the card at the light source and ensure it gets illuminated by the light source. Then point the camera at the grey card and ensure it fills the entire frame. The light is now being measured indirectly, after having been reflected by the card.
With both tools it is important that they are spectrally-neutral. This means that they transmit (lenscap) or reflect (grey card) all visible frequencies to the same degree. Otherwise one would measure the spectrum of the lightsource after it has been modified by the lenscap or card. After having set the white balance in this way with a wrong card, the card may look grey, but other colors may be off again as when using a light source with an unusual spectral distribution.
The next step is to ensure that the even, featureless image which gets projected in this way doesn't get under- or overexposed. Under exposure will lead to a weak signal from which no accurate measurement can be made. Over exposure will cause all channels ro give their maximum signal regardless of the spectral distribution of the light. Even when the image isn't fully overexposed, at high exposures the different color channels might start to repond slightly differently to an increase in brightness, thereby throwing off your white balance. In practice you can safely white balance on exposures from say 30% to 70%.
In fact, the built-in ND filter in de x929 complicates things a little. As the ND filter is not entirely color neutral you should first set your exposure to whatever is required for the shot, and then white-balance. If you alter the exposure after the white-balance measurement and the position of the ND filter is changed, your colors change slightly.
The built-in ND filters of professional video camera's are designed differently and it now becomes clear why. Professional camera's have a filter wheel with multiple ND filters with different strengths. You can set the wheel in three or four positions to change the ND strength. However, each filter will always cover the aperture completely. This has two advantages, first there will be no reduction of sharpness as described in this post, and second there is no hue-change while changing the f-stop. The only thing one has to think of with a professional camera is that one should set the position of the filter wheel before setting the white balance. Apart from this, the filters might also be made of better materials or a built-in electronic correction might be applied wich would eliminate the need to set the filter wheel beforehand, this I do not know.
And then the measurement can be made by either pressing the white balance button or (as in the case of the x929) activating the manual white balance reading through the menu. Now your colors should be accurate (to the best capability of your camera) across the entire color spectrum.
It may be tempting at times to use other objects which appear white or grey, and some objects may give satisfying results (e.g. clouds at noon) but they may also throw off your colors significantly (e.g. a piece of printer paper).
First of all, if you look critically at the objects around you, you may find it quite hard to find an object which truly appears white or grey. White comes in many shades and true white is a rare color. Furthermore, certain colors are downright misleading. The washing powder you use to whash your white clothing contains optical whiteners. These are chemicals which absorb UV light and re-emit it as a slightly bluish light. The bluish light overshadows the somewhat yellowish, garish light which comes from the clothing fibers themselves, giving them a bright white appearance when viewed under natural light (which contains a substantial amount of uv light). White balance on a white t-shirt and your colors may take on an unexpected warm, reddish tone.
Many types of papers are whitened using a similar principle which also make them unsuitable for an accurate measurement. Also realize that translucent lenscaps and grey-cards are usually made of plastics or similar materials which may change color due to aging under the influence of uv-light. Such aged lens-caps or grey cards may throw off your colors.
If a manual white balance as described above is not appropriate because the light source is not a full-spectrum light source, you might achieve better results with any of the presets which have been optimized for a specific light source.

If the light source is appropriate but the procedure can not be executed, e.g. because the camera can not be exposed to the same light as the subject, then you should also use a preset to get at least as close as possible. You can correct the colors of your clips after you have recorded them on a computer. This is easy as you can apply the same correction on all shots made under the same lighting.

Note to self: stop using AWB.

In any case it is best not to use the auto white-balance feature. Your white balance may change during a shot which makes it much harder, if not impossible to correct afterwards. Trust me, I have learned this the hard way. By the way, this example was shot with a Panasonic NV-MX300, a predecessor of the X920.
In addition to this, manufacturers may have tuned the auto-white balance and the pre-sets in certain ways which gives the camera a certain "look". For some reason unknown to me, Panasonic camcorders often have a slightly greenish tone to them when using a whitebalance preset or auto-whitebalance. When a Sony camera is off, it usually is too bluish and from the little experience I have with Canon I get the impression they tend to favor magenta. With a manual white balance, all these errors disappear and they all can produce proper colors.

Given the differences in color rendition of the three camera's in the above clip I can already conclude that they must all have been operating in their auto-everything modes. The author confirms this in a comment to the video. It may be tempting to draw conclusions about the quality of colors of these camera's but you can only realy draw conclusions on the quality of their auto-everything modes. A camera with a poor auto-white balance may very well out-perform a camera with a more clever auto-white balance mode once both camera's are properly manually white-balanced.

Know your camera - stars and rainbows

In the previous post I disected the exposure system of my camera (the Panasocic hc-x929/x920); I showed the combination of iris blades and ND-filter used and showed how it affected the sharpness of the image. In order to get an optimal sharpness of my images I must select an f-stop near f 2.8 or f 5.6. There are however also some other image effects which get affected by the choice of f-stop. Although these will generally not be decisive in choosing a particular f-stop I still wanted to show them to you in this post.
The first, most obvious effect is how the bokeh is rendered.
I must add to this that we are talking here about a small sensor camera which by the nature of it's design has a very large depth-of-field (or DOF) in most situations. So out-of-focus elements will not be as common in it's footage as in the footage of e.g. a full frame dslr. Only at long focal lengths (i.e. zoomed in) or when focussing at very close distances and shooting at wide open apertures will this camera be capable of separating the subject from it's environment by bringing the environment out of focus while keeping the subject in focus. And as the DOF increases with increasing f number I'm pretty sure that beyond f11 the DOF will effectively run from the lens to infinity. It is the shape of the aperture between f1.5 and f2.8 which effectively determines it's rendering of bokeh.
Out-of-focus dots of light will be rendered as a projection of the aperture. Knowing what I know now about the aperture system of my camera I can conclude that the bokeh will be a regular octagon between f2.5 and f2.8 and also between f5.6 and f11. However, above f2.8 and below f5.6 I might expect an irregular bokeh which may give out-of-focus elements an unusual look. Beyond f11 the bokeh will have changed to a diamond shape.
Deducing is one thing, testing another. So I took a couple of pictures of the same out-of-focus subject at different apertures but with identical brightness (as far as this was possible) to see if I would notice anything unusual between f2.8 and f5.6. Obviously the image gets sharper as the f-number increases. Here are a few (click to enlarge):

f2.8

f4.0

f5.6

I can't see anything unusual about the bokeh, The f4.0 picture looks normal to me, as did the other pictures with other f numbers which I will not bother to show. So the "unusual" bokeh which I expected between f2.8 and f5.6 did not appear.
The second effect is how bright point sources of light are rendered. These are effectively rendered as stars where the points of the stars are oriented perpendicular to the orientation of the edges of the aperture blades. Let's show a few examples (click to enlarge):

f2.8 f3.1

f5.1 f5.6

The following can be obeserved:

• The star gets smaller with increasing f-number
• Between f3.1 and f5.2 an additional pair of vertical legs are superimposed onto the star which are slightly mis-aligned with the original short legs. (Not very easy to see).
• At f5.2 a subtle diffraction pattern can be seen in the vertical legs where the light is separated in a rainbow pattern.

This can be explained by the design of the combination of the iris and built-in ND filter. As soon as the ND filter is shifted in front of the aperture, additional horizontal edges are introduced which add an additional pair of vertical legs to the star. Apparently the edges of the ND filter are not perfectly aligned with the edges of the iris giving them a slightly different angle.
When the ND filter covers the aperture almost entirely, two small slits remain which due to their small diameter cause a diffraction pattern like you might have seen in a typical high-school physics experiment. As the location of the minima and maxima depend on the wavelength of the light, the light effectively gets separated into it's individual wavelengths, giving a series of overlapping rainbow patterns. Once the ND filter covers the aperture entirely,  the star is regular again and has diminished in size.
Now, as a little encore I would like to show you some patters which occur at very specific combinations of focal length and focus distance. The aperture was in all cases wide open.

The top image appears when focussing on a short distance with the light source completely out of focus. In the center image the light source is brought slightly more into focus and on the bottom image the light source is almost in focus. The upper two images show artifacts which I haven't encountered yet in normal footage I shot with the camera, the bottom shot shows a familiar artifact; colorful "globes" surrounding a bright light source.
I'm not entirely sure what is causing this, but I suspect it is related to the fact that this model is a three-ccd camera (or 3mos as Panasonic likes to call it). Maybe the colorful patches of red, green and blue are internal reflections in the prism block which bounce back from the lens element, closest to the prism block, back towards the sensor. The distance between this lens element and the prism block will then determine if the reflections are in focus by the time they reach the sensors or not.
Notice how the camera interprets a range of wavelengths as one solid color of either red, green or blue. Only where the patches overlap as in the horizontal line to the right, a mix color is shown, in this case yellow.
Another effect caused by the fact that this is a three-ccd camera are the colorful fringes of the bokeh which can be seen at it's widest apertures. When you look at the image of the wide open aperture at the start of this post, or in the previous post, you can see a blue edge on the upper right, a red edge on the upper left and a green edge at the bottom. Sometimes when filming street lights in the background, such subtle colored edges can be seen, as if the colors don't align there perfectly. This is the telltale sign of a three-ccd camera.

Know your camera - the aperture of the iris

In the previous post I described how the sharpness of my videocamera (Panasonic hc-x929/x920) varies as a function of it's f-stop. The conclusion was that the sharpness was optimal at two different f-stop values (f2.8 and f5.6) with a local minimum at f4 where the sharpness is reduced.
This behaviour is odd for a lens as I would expect just one f number where the sharpness is optimal with decreasing sharpness towards the extreme ends of the range. This motivated me to do another test to figure out how my iris changed as I changed the f-stop setting on the camera.

Have you noticed e.g. in films how out-of-focus streetlights in the background of a scene are often rendered as hexagons, octagons, circles or elipses? This is because a lens projects each out-of-focus point as a surface in the shape of the aperture. As different lenses have different implementations of their iris, these "point sources" essentially show you the shape of the aperture as created by the iris in your lens. They only show up when the light source is essentially a point source because then you only see one projection of your aperture. If the light comes from e.g. an entire building in the background than each point of that building creates a projection of the aperture and all these projections are superimposed onto each other. This will then create a blurred image of the building without the shape of the aperture being recognizable.
So one way of seeing the shape of the aperture is to take an out-of-focus image of a point source of light against a dark background. The smaller your light source is, the clearer the image of the aperture will be.

In order to see the shape of the aperture of my camera I took a halogen light and placed a piece of thin cardboard in front of it (at a safe distance, halogen lights can generate a lot of heat and radiation!). In the piece of cardboard I had pierced a small pinhole. Then I pointed my camera at the pin-hole in the cardboard and aligned it on a straight line with the pinhole and the light so I could see the brightest part of the light through the pinhole. Then I set the smallest f-number I could set (by first setting a fast shutter speed, after all it is a Panasonic camcorder, so no aperture priority mode here), switched to manual focus and chose a combination of zoom-factor and focusing distance which would give me the largest image of my aperture. I also had to experiment a little with the distance between the camera and the cardboard to determine what would work best.
Then I took a series of pictures, each with a different f-stop setting (and shutter speed to get proper exposure). This is the series of shots I got:

f 1.7

f 2.0

f 2.4

f 2.8

f 3.6

f 4.0

f 4.8

f 5.6

f 6.8

f 8.0

f 9.6

f 11

f 14

f 16

What are we looking at:

• Between f1.5 and f2.8 you can see how an 8-blade iris creates an aperture in the shape of an octagon which gets smaller as the f number increases. (Also notice the diffraction rings at the outer edges of the aperture).
• Between f2.8 and f5.6 a piece of smoked glass is shifted in front of the aperture covering a greater part of the aperture with increasing f number. The iris, and therefore the aperture does not change. Notice how the edges of the sheet of glass line up with the edges of the blades of the iris.
• Between f5.6 and f11 the aperture is reduced further as the blades of the iris contract further
• From f14 the aperture has changed to a shape more closely resembling a diamond and reduces further in size.

The "piece of smoked glass" which gets shifted in front of the aperture must be a built-in ND filter. Why was it designed in this way?
As indicated in my previous post, each lens has a "sweet spot" in it's aperture range where the sharpness of the image is optimal. Wide apertures soften the image due to spherical aberration and small apertures lead to diffraction diffusion. But this is a small-sensor camera. This means that the surface area onto which the image is projected is small when compared to other camera's, and the individual pixels on the sensor capturing the light must be very small given the resolution of the image. This implies that although the lens might be capable of projecting a particular part of the scene on just one pixel, the aperture will at fairly low f-numbers already smear this light out over neighboring pixels due to diffraction diffusion, thereby reducing the sharpness of the image.
If changing the f-number would only lead to a variation in aperture size, the range over which you could vary the f-number without sacrificing the sharpness of the image would become very small. Only the shutter speed would remain as a variable to control exposure.
To increase the useful range of f-stop settings the movable ND-filter is used. It allows you to control the amount of light passing through the aperture without having to change it's size. The aperture now stays in it's sweet spot between f-stops 2.8 and 5.6, that's a range of two stops. This implies the filter has a strength of ND4, i.e. only one-fourth of the total amount of light passes through it.
How does this explain the two distinct maxima in sharpness as a function of f-number? From the point the ND filter is shifted in front of the aperture you can consider the aperture to consist of essentially two apertures superimposed upon each other.

The first aperture has the shape and size of the physical aperture as determined by the iris blades. Only one-fourth of the total amount of light passes through it, i.e. the amount of light when the ND filter would be fully in place. Let's call this aperture 1. The second aperture consists of the area of the aperture as determined by the aperture blades which is not covered by the ND filter. This you could also call the "virtual" aperture. Initially, at f2.8 or lower f-numbers the remaining three-quarters of the light passes through this aperture. Lets'call this aperture 2. The final image as projected on the sensor is the two images of these two apertures superimposed upon each other.
Between f2.8 and f5.6 the image from aperture 1 is constant in both brightness and sharpness. The image from aperture 2 will decrease in brightness as the area which is not covered by the ND filter is reduced. As aperture 2 is reduced in size, the image produced by it will increasingly suffer from diffraction diffusion. As the ND filter is gradually shifted in front of aperture 1, aperture 2 produces initially the brightest image and therefore the reduction in sharpness due to diffraction difusion is clearly visible. However, as the ND filter shifts further, the contribution of aperture 2 to the total image brightness is gradually reduced to zero. Therefore it's detrimental effect on the total image also becomes less visible and eventually vanishes completely. This effectively leads to an initial reduction in sharpness which is restored once the ND filter has shifted completely in place and the "virtual" aperture 2 has closed completely. So this effectively explains the dip in sharpness.
The extension of the sweet spot of the lens by two stops comes at the price of a dip in sharpness during that extension. Incresing the strength of the ND filter could extend the range of the sweet spot even further but the dip in sharpness would be even more pronounced as the detrimental effect of virtual aperture 2 would be visible at smaller sizes of the virtual aperture due to it's relatively greater brightness compared to that of physical aperture 1. Likewise, reducing the strength of the ND filter would reduce the depth of the dip in sharpness but at the cost of reducing the range of extension of the sweet spot. I think this is an excellent demonstration of how the designers of this camera had to strike a balance between various design parameters. When it comes to designing an optical system, there are no free lunches; You can alter a design parameter to enhance the performance in one area, but you will suffer in another.
As a thought experiment let's see what would happen if you would want to upgrade this camera to UHD (or 4K) resolution. If you would want to keep the size, weight and cost essentially identical, the sensor size and size of the lens elements would have to remain the same. The increase in resolution would require the number of effective pixels to be increased by a factor of four or put differently, the diameter of each effective pixel would have to be halved. First, this would obiously lead to a reduction of the signal-to-noise ratio per pixel, assuming the increase in efficiency of a newer generation of sensor is less then a factor of four (which is a pretty safe assumption to make).
Second, assuming the lens itself would actually resolve the amount of detail required for this new generation of camera, any diffraction diffusion effects would already become vissible at wider apertures due to the smaller size of the pixels. To maintain an identical perceived sharpness you can no longer use the lowest end of the f-scale as spherical aberration will be more clearly visible. The usability of the high-end of the f-scale is also reduced as diffraction diffusion by the physical aperture will become visible at lower f-numbers as before. Furthermore, the amount of ND-filtering will have to be reduced in order to reduce the amount of diffraction diffusion caused by the virtual aperture, thereby reducing the extension of the sweet spot of the lens. In the end, you may very well end up with a camera which only allows you to vary the shutterspeed if you want the sharpness of your images to match the resolution.
What ways are there out of this? Well you could ofcourse increase the sensor size. This would imply an increase in the lens unit aswell, and therefore a total increase in size, weight and ofcourse cost. let's try another, not so obvious resolution; An increase in frame rate. A high shutter speed is only a problem when used in combination with a low frame rate as it leads to a stroboscopic effect. Suppose we could vary the frame rate along with our shutter speed. A shutter speed of 1/250 at 120 frames per second would simply lead to buttery smooth, natural looking motion. And as long as displays can not show us 4K images at 120 fps or for those who prefer a more filmic look at 30 fps we might even consider artificially reducing the framerate after they have been captured and deducing motion blurr from the differences between frames. Now I have to admit, this option has some challenges in itself. Increasing the readout speed (and therefore the framerate) of a sensor is anything but trivial. Encoding, storing and processing all these frames requires a lot of computing power and even when the framerate is artificially reduced to ease the encoding, transportation and storage requirements, the gains will propably be largely negated by the cost of creating artificial motion blur. You would at least end up with a camera that would eat batteries like there is no tomorrow and a big exhaustion fan to get rid af all the excess heat. I suspect the engineers would go with scenario one.
Well, at least we now have an explanation for the fact that we have two different f-stop values at which the sharpness is at it's maximum value and along the way we have gained a bit of empathy for the engineers who make these machines and have to make the tough compromises.