A baseball stadium photographed with a fisheye lens: the whole stadium fits in a circle and straight lines bend — Photo: Arturo Pardavila III, CC BY 2.0, Wikimedia Commons

Computer Vision, lecture 2

Image formation

How a camera turns light into a grid of numbers

Last time's question

A straight road drawn by a pinhole camera: the two edges and the centre line meet at one point on the horizon. Poles that are equally far apart look closer together further away

Why do the two sides of a straight road meet at one point?

The path of a ray of light

Diagram: light from a lamp hits a surface, bounces off, goes through the camera lens and lands on the sensor — Szeliski (2022), Fig. 2.14

Geometry: where does a point land in the picture?
Lenses: how does a lens collect light?
Light and colour: how bright and what colour?
Numbers: how does the camera turn light into numbers?

Part 1: Geometry

Where does a point in the world land in the picture?

The pinhole camera

An old engraving of a camera obscura: a dark room with a small hole, and a picture of the garden outside on the wall — Camera obscura, 1755. James Ayscough, public domain, via Wikimedia Commons

Two similar triangles: $\frac{x}{f} = \frac{X}{Z}$ , so $x = f \frac{X}{Z}$ (and $y = f \frac{Y}{Z}$ ).

Draw this on the board yourself, step by step. It is the most important drawing today.

Draw a wall with a tiny hole. Behind it, draw a screen at distance $f$ . This is the picture plane.
Draw an object (an arrow) with height $X$ , at distance $Z$ in front of the hole.
Draw one straight line from the top of the arrow, through the hole, to the screen.
Show the two triangles: a big one outside (sides $X$ and $Z$ ) and a small one inside (sides $x$ and $f$ ). They have the same angles, so their sides have the same ratio.

Say these results out loud:

We divide by $Z$ . Twice as far away means half as big in the picture. This is perspective.
The picture is upside down, because the light line crosses through the hole. Cameras and our brain turn it back. In maths we often put the screen in front of the hole to avoid the minus sign.
$f$ is the focal length. A bigger $f$ gives a bigger picture. This is zooming in.

The engraving on the right is a camera obscura ("dark room" in Latin): a dark room with a small hole in one wall. The garden outside appears upside down on the opposite wall. People have used this for hundreds of years. You can make one with a shoebox, or with a dark room and a small hole in the curtain.

Why parallel lines meet

A point on a line going in direction $d$ : $P (t) = P_{0} + t d$

x (t) = f \frac{X _{0} + t d _{X}}{Z _{0} + t d _{Z}} t \to \infty f \frac{d _{X}}{d _{Z}}

The end point depends only on the direction $d$ , not on the start $P_{0}$ .
So all lines with the same direction meet at one point: the vanishing point.
Lines parallel to the picture ( $d_{Z} = 0$ ) stay parallel.

This answers the road question with three lines of maths. Go slowly.

Imagine walking along the road. $t$ gets bigger, and the point gets further away.
When $t$ is very large, the start point $P_{0}$ does not matter any more. Only the direction matters.
The left and right sides of the road start at different places, but they have the same direction. So they go to the same point in the picture.
For a road straight ahead, the direction is $(0, 0, 1)$ . So the vanishing point is at $x = 0$ : the centre of the picture, where the camera is looking.

Check: "Where is the vanishing point of the vertical poles?" Their direction is $(0, 1, 0)$ , so $d_{Z} = 0$ . They have no vanishing point: vertical lines stay parallel in the picture, as long as the camera is not tilted. If you tilt the camera up at a tall building, the vertical lines start to meet. That is why buildings look like they are falling backwards in photos.

Other camera models

Six drawings of the same 3D object seen with different camera models, including orthographic and perspective — Szeliski (2022), Fig. 2.7

Orthographic: $x = X, y = Y$ . We simply drop the distance. This works well when the depth of the object is small compared with how far away it is.

From metres to pixels: the matrix K

A picture uses pixels, and pixel (0, 0) is at the top left:

u = f_{x} \frac{X}{Z} + c_{x} v = f_{y} \frac{Y}{Z} + c_{y}

With homogeneous coordinates this is one matrix multiplication:

λ u v 1 = K (intrinsics) f_{x} 00 0 f_{y} 0 c_{x} c_{y} 1 X Y Z

Two changes from the pinhole formula:

Units. $f$ was in millimetres, but a picture has pixels. So we write $f$ in pixels: $f_{x} = f / (width of one pixel)$ . For example, a 4 mm lens with 0.0014 mm pixels gives $f_{x} \approx 2900$ pixels.
Starting point. Pixel numbers start at the top-left corner. But the centre of the lens points at the middle of the sensor, at pixel $(c_{x}, c_{y})$ . This is the principal point. So we add $c_{x}$ and $c_{y}$ .

Homogeneous coordinates, in simple words: we add a 1 at the end of the point. We agree that $(a, b, c)$ and $(2 a, 2 b, 2 c)$ mean the same point. To get the pixel back, divide by the last number. Here the last number is $Z$ , so the division by $Z$ happens at the end. The benefit: the whole projection becomes one matrix multiplication. We will use this again in lecture 8.

$K$ is called the intrinsic matrix ("intrinsic" means "belonging to the thing itself"). It belongs to the camera and lens, not to where the camera stands.

Where the camera is: R and t

A world point first moves into the camera's view, then it is projected:

λ u v 1 = K [R ∣ t] X_{w} Y_{w} Z_{w} 1

$R$ (rotation) and $t$ (translation, a shift): the extrinsics, 6 numbers
$K$ : the intrinsics, 4 numbers ( $f_{x}, f_{y}, c_{x}, c_{y}$ )
$P = K [R ∣ t]$ : the 3×4 camera matrix

Try it (pairs, 4 minutes)

A camera has $f_{x} = f_{y} = 800$ pixels and principal point $(c_{x}, c_{y}) = (640, 360)$ . The camera is at $(0, 0, 0)$ and looks along $+ Z$ .

Where does the point $(1, 0.5, 4)$ (in metres) land?
Where does $(2, 1, 8)$ land? Why?
How far away is a 1.8 m tall person who looks 180 pixels tall?

Field of view

A pinhole triangle: half the angle of view, the focal length f, and half the sensor width W/2 — Szeliski (2022), Fig. 2.10

Part 2: Lenses

A pinhole makes a perfect picture, but lets in almost no light.

Why we need lenses

Four versions of the same cat photo: a tiny hole gives a sharp but very dark and noisy picture; a small hole gives a dim, slightly blurry picture; a large hole gives a bright but very blurry picture; a lens gives a bright and sharp picture — Cat photo: Caleb Woods, CC0, via Wikimedia Commons

Focus and depth of field

A thin lens with focal length 100 mm makes a sharp picture of an object 5 m away on a screen 102 mm behind the lens. Points at other distances make a small blurry circle c — Szeliski (2022), Fig. 2.19

Thin lens rule: $\frac{1}{z _{o}} + \frac{1}{z _{i}} = \frac{1}{f}$

Only one distance $z_{o}$ is perfectly sharp. A smaller opening makes the blur circle $c$ smaller, so more distances look sharp. This range is the depth of field.

Explain with the drawing:

An object at distance $z_{o}$ is perfectly sharp on a sensor at distance $z_{i}$ behind the lens. For $f = 100$ mm and $z_{o} = 5$ m, $z_{i} = 102$ mm.
Points closer or further away are sharp in front of or behind the sensor. On the sensor they become a small disc, the blur circle $c$ .
If $c$ is smaller than one pixel, the point still looks sharp. The range of distances where this is true is the depth of field.

The opening of the lens is the aperture $d$ . Photographers use the f-number $N = f / d$ (f/2, f/4, f/8…). A bigger f-number means a smaller opening: less light, but more depth of field.

Examples students know: portrait mode on phones blurs the background (small depth of field). Phones fake this: they estimate distance with a vision method, and then blur the far parts. Landscape photos are sharp everywhere (small opening).

Link to vision: the amount of blur tells us something about distance. This is called depth from defocus.

Lenses are not perfect

Three photos showing lens distortion: a straight view with barrel distortion, the same view with pincushion distortion, and a fisheye view of a room that covers almost 180 degrees — Szeliski (2022), Fig. 2.13

Radial distortion: straight lines bend outwards (barrel) or inwards (pincushion). We describe it with $r^{'} = r (1 + k_{1} r^{2} + k_{2} r^{4})$ and remove it after calibration.
Vignetting: the corners are darker. Chromatic aberration: colours focus at slightly different places.

Part 3: Light and colour

How bright, and what colour?

Surfaces reflect light in different ways

A white marble statue of a woman's head: smooth shading on the cheek and neck, small bright shiny spots, and dark folds — Szeliski (2022), Fig. 2.16

Matte (diffuse): light spreads evenly. Brightness depends on the angle between the light and the surface.
Shiny (specular): a bright spot that moves when you move.
Shadows and folds: little light gets in.

One pixel number mixes light, material, shape and viewpoint.

Colour is three numbers

Two drawings: mixing red, green and blue light makes cyan, magenta, yellow and white; mixing cyan, magenta and yellow inks makes red, green, blue and black — Szeliski (2022), Fig. 2.28

Light has many wavelengths (colours of the rainbow), from about 400 to 700 nm.
Our eyes have three types of colour cells, so we see only three numbers' worth.
Cameras copy this with three filters: R, G, B. Screens add light (left). Printers remove light with ink (right).

Inside the camera

Diagram of the steps inside a camera: light goes through the lens, aperture and shutter; the sensor, gain and converter make RAW data; then demosaicing, noise removal and sharpening, white balance, a brightness curve and compression make a JPEG file — Szeliski (2022), Fig. 2.23

Follow the arrows on the slide:

Lens, aperture, shutter: how much light comes in, and for how long.
Sensor (almost always CMOS today): each pixel counts light particles (photons) and turns them into electric charge. Gain (ISO) makes the signal stronger, but also the noise. The converter (ADC) turns the voltage into a number, usually with 10 to 14 bits. This is the RAW file.
The image processor inside the camera: demosaicing (next slides), noise removal, sharpening, white balance (making white things look white under yellow indoor light), a brightness curve (gamma), and JPEG compression.

What this means for vision: a JPEG photo is heavily processed. The numbers are not simply proportional to the light. Phones also add strong "AI" processing. For measuring tasks, engineers often use RAW pictures.

Noise sources to name: photon noise (light arrives in random packets, so dark pictures are noisier), electronic noise, and rounding noise (numbers are rounded to whole numbers).

One colour per pixel: the Bayer filter

Left: a 4 by 4 grid of red, green and blue filters, with two greens for each red and blue. Right: the same grid, with the two missing colours at each pixel written in lower case as guesses — Szeliski (2022), Fig. 2.31

Each pixel measures only one colour. There are two greens for each red and blue.
The two missing colours (lower case, right) are guessed from the neighbours.

Rebuilding colour: demosaicing

Three zoomed views of the edge of a red umbrella: each pixel coloured by its one filter, the single number stored for each pixel in grey, and the full-colour picture rebuilt from it — Umbrella photo: keith ellwood, CC BY 2.0, via Wikimedia Commons

Gamma: the numbers are not the light

Curves of output against input: the camera curve goes up quickly and then flattens; the screen curve does the opposite; a dashed straight line shows "no change"

Stored number ≈ light $^{1/2.2}$ . More numbers are used for dark tones, where our eyes notice small differences.

Part 4: From signal to numbers

Sampling: how many pixels?

The same cat photo at 960 by 640, 120 by 80, 40 by 26 and 20 by 13 pixels, all shown at the same size; the small ones look like big blocks — Cat photo: Caleb Woods, CC0, via Wikimedia Commons

Quantization: how many levels?

The same grey cat photo with 256, 16, 4 and 2 brightness levels. With 4 and 2 levels, smooth areas become flat patches with false edges — Cat photo: Caleb Woods, CC0, via Wikimedia Commons

Colour spaces

A person with a red umbrella shown as colour, as separate R, G and B channels, as hue, saturation and value channels, and as a mask that keeps only the red umbrella — Umbrella photo: keith ellwood, CC BY 2.0, via Wikimedia Commons

HSV separates hue (which colour) from saturation (how strong) and value (how bright).

Look at the R, G and B channels. The umbrella is bright in R and dark in G and B. But some light grey areas (the sky, the wet ground) are also bright in R. With RGB it is hard to say "red" without also catching bright grey.

In HSV, the hue says which colour, and does not change much with brightness. So one hue rule plus one saturation rule (strong colour only) gives the mask at the bottom left: almost only the umbrella. A few small red things are also caught.

Look at the hue picture: in grey areas the hue looks like random noise. Grey has no clear colour, so its hue is unstable. That is why we also need the saturation rule.

Other colour spaces to mention:

Grey: $0.299 R + 0.587 G + 0.114 B$ , weighted by how sensitive our eyes are.
YCbCr: brightness plus two colour channels. JPEG and video use it.
L*a*b*: made so that equal distances look equally different to people. Used to measure colour.

Live: finding a colour

import cv2
img = cv2.imread("red-umbrella.jpg")        # BGR, uint8
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)  # H is 0..179 (!)
h, s, v = cv2.split(hsv)

red = ((h <= 8) | (h >= 168)) & (s > 120) & (v > 50)
print(red.mean())                           # 0.028 -> about 3% of the photo

cv2.imwrite("mask.png", red.astype("uint8") * 255)

Compression changes the numbers

Three crops of the same flower photo saved as JPEG with lower and lower quality: the lowest quality shows square blocks and smeared colour — Szeliski (2022), Fig. 2.34

JPEG throws away detail we probably will not notice: blocks, ringing near edges, smeared colour.

Summary

Geometry: $x = f X / Z$ . Far things look small. Parallel lines meet at vanishing points. Distance is lost. $P = K [R ∣ t]$ .
Lenses: they collect light, but bring focus, depth of field and distortion.
Light and colour: one pixel mixes light, material, shape and viewpoint. Colour is three filtered numbers.
Inside the camera: Bayer filter → demosaicing → white balance → gamma → JPEG. The file is not raw light.
Numbers: sampling (how many pixels), quantization (how many levels), colour space (how to describe colour).

Exit ticket

A point is 10 m away and appears 50 pixels from the centre of the picture. Where would it appear if it were 20 m away, in the same direction?
Why do phone cameras have two green pixels for every red and blue one?
Name two reasons why a pixel number is not a direct measure of the light from that point.

Next lecture: pictures as signals. Histograms, frequency, and why fine stripes can turn into big false patterns.