Week 1 · Day 5 Lab — Engineered vs learned features¶

AIMS Cameroon · Introduction to Computer Vision

Learning goals

  1. Build a complete "features + classifier" recogniser and compare raw pixels with HOG.
  2. Measure robustness to small shifts and connect it to the design of HOG.
  3. Learn a convolution kernel from input/output examples by gradient descent.
  4. Count parameters of dense vs convolutional layers.
In [ ]:
# If you are on Google Colab or Kaggle, everything below is pre-installed.
# Locally:  pip install numpy matplotlib scipy scikit-image scikit-learn opencv-python
import numpy as np
import matplotlib.pyplot as plt
import cv2, skimage, scipy
from skimage import data, color, util, exposure, filters, feature, measure, transform
from scipy import ndimage as ndi

plt.rcParams["figure.dpi"] = 100
rng = np.random.default_rng(0)

def show(*images, titles=None, cmap="gray", size=4, **kw):
    """Display images side by side."""
    n = len(images)
    fig, axes = plt.subplots(1, n, figsize=(size * n, size))
    axes = np.atleast_1d(axes)
    for i, (ax, im) in enumerate(zip(axes, images)):
        ax.imshow(im, cmap=cmap if np.ndim(im) == 2 else None, **kw)
        ax.axis("off")
        if titles:
            ax.set_title(titles[i])
    plt.tight_layout(); plt.show()

print("numpy", np.__version__, "| opencv", cv2.__version__, "| scikit-image", skimage.__version__)
In [ ]:
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

digits = load_digits()
X = np.stack([cv2.resize(im, (24, 24), interpolation=cv2.INTER_LINEAR) for im in digits.images])
y = digits.target
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
print(X.shape, "train", len(Xtr), "test", len(Xte))
show(*X[:6], titles=[str(t) for t in y[:6]], size=1.8)

1. Pixels vs HOG¶

✏️ Exercise 1.1¶

  1. hog_features(images, ppc=6, orientations=9) returns one HOG vector per image (cells_per_block=(2, 2)).
  2. Train two pipelines StandardScaler → LogisticRegression(max_iter=3000): one on flattened pixels, one on HOG.
  3. Report test accuracy for both.
In [ ]:
def hog_features(images, ppc=6, orientations=9):
    # TODO
    raise NotImplementedError

def make_clf():
    return make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000))

raw_clf = None  # TODO
hog_clf = None  # TODO

✏️ Exercise 1.2 — robustness to shifts¶

Shift every test image by s pixels right and down (ndi.shift(im, (s, s), order=0)) for s = 0, 1, 2, 3 and plot the accuracy of both models. Which step of HOG explains the difference?

In [ ]:
# TODO

✏️ Exercise 1.3 — tune the representation¶

Try ppc ∈ {4, 6, 8, 12} and orientations ∈ {4, 9, 18} on the unshifted and the 2-px-shifted test set. Report a small table. What is the trade-off controlled by the cell size?

(In a real project you would tune on a validation split, not on the test set. Why?)

In [ ]:
# TODO

2. Learning a filter by gradient descent¶

Given an image $I$ and a target $T=I\star k^\*$ for an unknown $3\times3$ kernel $k^\*$, learn $w$ minimising $$ \mathcal L(w)=\frac1N\sum_p\big(\langle w,P_p\rangle - T[p]\big)^2,\qquad \nabla_w\mathcal L=\frac2N\sum_p e_p P_p. $$

Stack all patches as rows of a matrix $\mathbf P\in\mathbb R^{N\times 9}$. Then $\mathcal L(w)=w^\top A w-2w^\top b+\text{const}$ with $A=\mathbf P^\top\mathbf P/N$, $b=\mathbf P^\top T/N$, and the gradient is simply $2(Aw-b)$, so we can precompute $A$ and $b$ once.

✏️ Exercise 2.1¶

Implement learn_kernel(img, target, steps=1500, lr=0.1, seed=0) returning the learned $3\times3$ kernel and the loss history. Get the patches with sliding_window_view(np.pad(img, 1, mode="symmetric"), (3, 3)) (NumPy's symmetric padding equals SciPy's reflect).

In [ ]:
def learn_kernel(img, target, steps=1500, lr=0.1, seed=0):
    # TODO: build P, A, b; initialise w randomly; repeat: record loss, w -= lr * gradient
    raise NotImplementedError
In [ ]:
img = data.camera().astype(float)[::2, ::2]
img = (img - img.mean()) / img.std()      # standardise: try removing this line and see what happens!
SOBEL_X = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], float)
LAPLACE = np.array([[0, 1, 0], [1, -4, 1], [0, 1, 0]], float)
GAUSS = np.outer([1, 2, 1], [1, 2, 1]) / 16
fig, ax = plt.subplots(1, 3, figsize=(12, 3))
for a, (name, k_true) in zip(ax, [("Sobel x", SOBEL_X), ("Laplacian", LAPLACE), ("Gaussian", GAUSS)]):
    target = ndi.correlate(img, k_true, mode="reflect")
    w, losses = learn_kernel(img, target)
    print(f"{name:10s} max |w - k*| = {np.abs(w - k_true).max():.3f}\n{np.round(w, 2)}")
    a.semilogy(np.maximum(losses, 1e-12)); a.set_title(f"{name}: loss")
plt.show()

✏️ Exercise 2.2 — why is learning slow on natural images?¶

Compute the eigenvalues of $2A$ for (a) the standardised camera image and (b) an image of white Gaussian noise of the same size. The step size must satisfy $\eta<2/\lambda_{\max}$, and the number of steps needed grows with the condition number $\lambda_{\max}/\lambda_{\min}$. Learn Sobel from the noise image: how many steps does it need? Why are natural images so badly conditioned? (Hint: neighbouring pixels are highly correlated.)

In [ ]:
# TODO

✏️ Exercise 2.3 — noisy supervision¶

Real labels are noisy. Add Gaussian noise with standard deviation 0.1 * target.std(), 0.5 * ..., 1.0 * ... to the Sobel target and learn again. How close is the kernel? Why does averaging over ~65 000 pixels make learning so robust here? (Think: how many unknowns, how many equations?)

In [ ]:
# TODO

3. Counting parameters¶

✏️ Exercise 3.1¶

Write dense_params(n_in, n_out) and conv_params(K, c_in, c_out) (both including biases) and use them to fill in:

layer parameters
dense: 224×224×3 → 1000 ?
conv 3×3, 3 → 64 ?
conv 5×5, 32 → 64 ?
two conv 3×3: 32 → 64 → 64 ?
In [ ]:
def dense_params(n_in, n_out):
    # TODO
    raise NotImplementedError

def conv_params(K, c_in, c_out):
    # TODO
    raise NotImplementedError

4. Stretch — random convolutional features¶

Before Week 2, a teaser: how good are random (untrained) filters?

  1. Draw 16 random 5×5 filters, convolve each digit with each filter, apply ReLU, then 4×4 average pooling. Flatten to a feature vector.
  2. Train the same logistic regression and compare with raw pixels and HOG, unshifted and at a 2-px shift.

What does this tell you about the value of the architecture (convolution + non-linearity + pooling) even before any filter is learned?

In [ ]:
# TODO

5. Reflection¶

  1. In two sentences: what does a CNN keep from classical vision, and what does it change?
  2. Your HOG features were designed by Dalal & Triggs for pedestrians, yet they work on digits. Why do hand-designed gradient features transfer across tasks? Where would you expect them to fail?
  3. In Exercise 2 the target was produced by a known kernel. In Week 2 the "target" will be a class label. What new difficulty does that introduce?