Most of digital cameras have a Bayer color
filter array in front of the photosensitive sensor. The Bayer filter array is a
repeating pattern of four colored filters. Half of the filters are green, one
quarter is red and one quarter is blue. The reason for more green elements in
Bayer color filter array is greater impact of green light on images, because
human eye is more sensitive to green than to red or blue light. The Bayer
filter allows only red, green or blue part of spectrum to reach photosensitive
sensor, distinguishing colors on a camera’s sensors that can’t recognize
colors.
Every pixel (the smallest element of an image)
of color images contains three intensities of light for three elementary
colors: red, green and blue. Sensor reading for every pixel behind a Bayer
filter contains only one third of the necessary data, intensity of only one of
the elementary colors. Data captured by camera’s sensor is like a mosaic;
pixels with only red, green or blue color. Steps required to produce an image
are in the sample bellow. All images are saved with scale 1:1 in lossless PNG
file, no loss in quality as for JPG images. The best way to see real details in
an image on screen is to look at image in its original size. For finer details
you could go with double size (200%) without image enhancements, or without
“smooth image when zoomed-in” option. Smoothing an image could blur differences
when comparing two similar images.
The picture bellow contains five images. The
first four images are showing the path from the camera’s sensor readings to the
final color photograph. The fifth image is a conversion from color to black and
white image. The first image marked with ‘R’ is a raw camera output, light
intensities captured by camera’s sensor behind Bayer filter. Upper part of an
image is black and white, with shades of gray as seen by sensor. The lower part
of image is color-coded with Bayer filter colors (mosaic). The missing two
color’s intensities are set to middle value of the whole image. This is more
accurate than to set them to zero. Also setting missing colors to zero gives
very dark image. The second image marked with ‘W’ is a raw camera output with
white balance multipliers. Upper and lower parts of the ‘W’ are shown in the
same way as for the ‘R’. The second image has accurate colors. It looks green
because of the double number of green mosaic parts. The mesh like pattern over
an image, inevitable by Bayer filter design, is obvious on both ‘R’ and ‘W’
images. To avoid this pattern and obtain a full color image, missing color data
are reconstructed with various demosaicing algorithms. It is a guessing of
missing data, interpolation of surrounding pixels of the corresponding colors,
based on an assumption that the color of an area in the image is relatively
constant. Result of interpolation is visible on the third image marked with
‘I’. This image looks dark because brightness perception of human eye is a
power function, while sensor has a linear response to light.
The next modification of an image is to convert
linear pixel values to the power function of the human eye. It is called gamma
function. Proper full color image is marked with ‘G’. The last image marked
with ‘D’ is the color image converted to black and white (desaturated).
All steps in the sample above are exact
calculations except interpolation. The interpolation always adds some noise
(errors) to an image. It could be very small, but always exist. The only way to
eliminate this noise is not to use interpolation, which is the case with a
camera without Bayer filter. A camera with Foveon sensor captures intensities
of all three elementary colors for each pixel, works similar to a camera with
color film. For black and white photography, cameras without Bayer filter
exist. They work similar to cameras with black and white film. Aside of avoid
interpolation, an additional advantage of those cameras is that they need one
EV step less to make photographs, because there is no Bayer filter’s light
loss.
Deep infrared images above 830 nanometers are
monochromatic (gray scale). They look exactly the same, regardless of which
part of the Bayer filter the light passes through. This allows us to skip
interpolation and have better images!
The method described here will work with deep
infrared filters equivalent to RG830 or even better RG850, but will not work
well with standard infrared filter as RG715. Differences in spectral response
(transmittance) of red, green and blue part of Bayer filter bellow 830
nanometers, will leave mesh like pattern on images with standard infrared or
red filters.
Bellow is the same scene with 830 nanometers
infrared pass filter, following the same steps as for color image. A custom
white balance for infrared images was used. Bayer filter mesh pattern visible
on raw image, vanishes on the image with custom white balance. Smaller
differences in spectral response of red, green and blue part of Bayer filter
for infrared part of spectrum, compared to visible part of spectrum causes mesh
pattern on raw images less visible on infrared than on color images. Images
marked with ‘I’ and ‘G’ are full color images. They look black and white as
expected for correct white balance. The final desaturated image is a real black
and white image. This last step is not necessary, but a real black and white
image is smaller in size and easier to manipulate with. Small white balance
imperfection adds a subtle orange cast to the image marked ‘G’, while
desaturated image marked with ‘D’ is completely gray. Increasing contrast of an
image, could add unwanted color cast on a color image, but not on a black and
white image.
A
simpler way to produce a black and white infrared image is shown bellow.
Starting with the same raw image, staying with shades of gray as seen by
sensor, apply only white balance multipliers and gamma function. A simpler
solution is usually a better solution. Bellow is a way to make a black and
white infrared image in three instead of five steps. One of skipped steps is
interpolation. Without interpolation noise, the final image should be better.
The ‘R’ and ‘W’ images are shown, as before, with upper part black and white,
and lower part as color-coded with Bayer filter colors. Interpolation is not
necessary because deep infrared light pass through red, green and blue part of
Bayer filter with the same light intensity loss. Image is monochromatic. No
need to reconstruct missing colors. Image is a real gray scale image, no need
for conversion to black and white too.
As expected, image marked with ‘G’ bellow, made
without interpolation is a slightly better than image above marked with ‘D’ made
with interpolation and desaturation.
To have a better look on image differences,
bellow are differences calculated by subtracting pixel values of images made
with those two methods. Brightness of a pixel is proportional to difference
between pixels on two images; a black pixel means no difference. The first
image on the left shows relative difference between image made with common five
steps method with interpolation, and simple three steps method image without
interpolation. Image is a real color image, not a color-coded image. Every
pixel on this image has one color channel correct, and two interpolated
channels, that could have noise (interpolation errors). This noise is always
with one of color channels intensity equal to zero. All noise pixels have a
saturated color, combination of two elementary colors. Another way to see
differences is to convert image made with interpolation to gray scale image,
and subtract it from the image made without interpolation. Resulting relative
differences are in the middle. The last image on the right shows only
interpolation relative errors, without color manipulations errors. White
balance in five steps method was changed to “average color channels of the
whole image”, the same as in three steps method. Type of conversion to gray was
also changed to be the same as in three steps method. Default change to sRGB
color space is skipped. This is a channel mixer, with nine coefficients,
specific for each camera. Colors on images in sRGB color space will look the
same regardless of type of camera after this conversion. For a monochrome
infrared image this just adds noise. These three changes leave only
interpolation errors when subtracting this image from three steps image.
You can recognize the shape of a wall on
interpolation noise images. Interpolation noise is higher on edges. Errors from
interpolation alone, compared to complete five steps method errors, are smaller
(darker right image than the middle one) and less random. Bright pixels, errors
of the interpolation on the image on the right side, are mostly where the edges
are – the wall and branches.
Average
absolute error was 4.3% of white pixel for common five steps method, and 4% for
interpolation. Adaptive Homogeneity-Directed interpolation was used.
The best way to isolate interpolation errors
completely accurate is to take an infrared image with a true monochromatic
camera without Bayer filter, remove sensor’s data not captured by Bayer filter
camera, and let interpolation algorithm to reconstruct missing data. It would
be interesting to see these interpolation errors, but not having infrared
camera without Bayer filter prevent me doing this.
Bellow are two images made from the same raw
data. The first image on the top was made with, the second image on the bottom
without interpolation, following five and three steps methods. Both images are
scaled to fit on the screen. Images look the same. Differences exist, but it is
hard to see them on images with reduced size.
Bellow are the same two images converted to
lossy JPG format with quality 40. Reducing image quality makes differences
visible. After conversion, imperfections pop out on image with interpolation,
but remain moderate on image without interpolation.
Now we are going to compare a part of this
image marked with green rectangle with 1:1 scale. Four different ways were used
to make black and white images from a raw file. Images are sorted from the
worst to the best looking image. The first image was made with interpolation
with daylight (auto) white balance multipliers. Image bellow shows impact of
wrong white balance when converting raw file to a gray scale image.
The next image is a little better, the same
five steps method with interpolation, described earlier, but with a correct
custom white balance.
The third image bellow was also converted to
black and white (gray scale) with interpolation, but desaturation step was
changed. On the third image all three color channels contribute to intensity of
gray with an equal amount. The second image used definition of luma for
conversion to black and white, default for the most of photo software.
Brightness of a pixel with luma depends mostly on green channel. Luma is
usually a better option for black and white conversion of color photographs,
but one third of each color channel type of conversion gives better infrared
images as you can see.
The best image bellow is made with simple three
steps conversion without interpolation.
Another three samples shows differences in
images with and without interpolation. On the left side is part of an image
made with interpolation, and on the right side is the same part of image made
without interpolation. Next image is part of the scene seen before, inside the
yellow square. Picture on the right side, made without interpolation is clearer
(not grainy), details are more distinct.
Another example that skipping interpolation
produces a better image is bellow.
The last example is a part of image, but this
time enlarged, with scale 2:1. Enlargement helps to see fine differences.
Letter “a” on the right image, made without interpolation looks better. Also
light pole arches are looking more natural on the same image.
Photographers who changed a Bayer filter camera
with a camera with Foveon sensor or with a dedicated black
and white camera without Bayer filter, would very likely agree, that images
made on a camera without Bayer filter (without interpolation) are better
looking images. Differences in images are small, but clearly visible.
For truly monochromatic, deep infrared images, Bayer
filter is useless. Interpolation adds noise to an infrared image. Using method
described here, gives almost the same result as if you have removed Bayer
filter from your deep infrared camera. Only difference is that you can’t shoot
one EV step faster.
The simplest way to make a deep infrared image
without interpolation is with dcraw. Dcraw is an excellent free multiplatform
raw file decoder made by David Coffin. It is used as backend software in lots
of image converters and viewers. Source code is available on author’s web page.
Dcraw is included in most Linux distributions. You can find and download
dcraw.exe for Windows easily. Another option is to compile the source code. The
whole installation of dcraw is a simple copy of executable file to a folder
that is in the operating system path.
After have dcraw saved on computer, follow
steps:
·
Open terminal (Command prompt)
·
Go to folder with raw deep infrared image(s)
·
Type: dcraw
-d -a name.raw
Options -d -a are case sensitive, no
space between dash (minus sign) and option letter. Raw image file with
extension is name.raw, for example DSC_1004.NEF.
Black and white image name.pgm will be
created in the same folder. You can upload file name.pgm in your photo
editing software for enhancement of image or file type conversion.
Option -d is for document mode, light
intensity of pixels with gamma function, without interpolation.
Option -a is for averaging the whole
image for white balance. This allows a correct black and white image,
regardless on white balance set in camera.
This is enough for practical purpose, to make
images without interpolation noise. If you want to try more option some
variations are:
Use -w instead of -a for white
balance set in camera. Use -6 option for 16 bit samples instead of
default 8 bits. Use -T for tiff instead of pgm type of output file.
You can add -j -t 0 to ensure no
rotation or stretching of raw sensors readings. This makes difference only with
cameras with non-square sensor’s elements or with cameras with diagonal
orientation of sensor’s elements as Fujifilm S2 Pro camera.
ICC profile file contains description of
specific color space. It is unique for each camera type. You can find it in
installed camera manufacturer’s software for raw file manipulation, or on
software CD for your camera. It has an icm extension. To use ICC profile with
dcraw add option: -p iccprofile.icm. Here iccprofile.icm is ICC profile
file name. Usually output image will look better with profile, although it can
cause mesh like pattern to appear with some files.
One of unique use of dcraw is to get unmodified
(completely raw) sensor’s reading of a digital camera with: dcraw -D -4 -j
-t 0 name.raw . This file is equivalent to upper part of image marked with
‘R’ in three steps method of getting infrared image from raw file described
earlier. If you are interested, you could finish the next two steps by
yourself: First repeat white balance multiplication on every 2 by 2 square of
pixels (makes image W). Second step, apply gamma function on each pixel (makes
image G). For the first step you can get Bayer filter pattern, along with other
information with dcraw -i -v
name.raw . If you want to average all three colors, what is the best
solution for deep infrared image, you don’t need to know Bayer filter pattern.
The last version of dcraw is from year 2018.
Dcraw could fail to open raw images on some of digital cameras less than three
years old (this was written in 2021).
If you prefer graphic software to command line
dcraw, the only graphic software I know than can be used is Darktable.
Photoshop don’t have any demosaicing controls, and can’t be used for this
purpose. Settings in Darktable to have similar result to dcraw use are:
·
Demosaic = passthrough (monochrome)
·
White balance = camera
You can adjust picture in Darktable, or save it
and adjust in other software. Disadvantage of using Darktable instead of dcraw
is that Darktable don’t have Average white balance option. If you look at image
with magnification and see mesh like pattern, you should manually correct white
balance till mesh like pattern disappears. This requires fine tuning of two of
white balance multipliers. Not having a proper white balance will void gains of
not using interpolation.
If you like programming you can try LibRaw. It is a free and open-source library that could help you to
start writing your own raw image software.
Final thoughts. If you only publish small
infrared images on web, method you are going to use will not make a visible
difference. If you print full size images, small but clearly visible
improvement in infrared images is worth of using method described here.