CNN

D

With corr2d_multi_in_out, our model is able to find a bunch of clues in an image.
Now we give it the ability to make decisions using these clues.
The decision making is through the linear function.

@op
def linear[o, i](
    x: Tensor[float][[i]],
    A: Tensor[float][[o, i]],
    b: Tensor[float][[o]]
) -> Tensor[float][[o]]:
    return (x * A).sum(1) + b

D

We do it using three layers of linear. Here's the complete predict of LeNet.

class LeNet(Model):
    def predict(
        x: Tensor[float][[1, 28, 28]],
        params: Tuple[
            Tensor[float][[6, 1, 5, 5]],
            Tensor[float][[6]],
            Tensor[float][[16, 6, 5, 5]],
            Tensor[float][[16]],
            Tensor[float][[120, 16 * 5 * 5]],
            Tensor[float][[120]],
            Tensor[float][[84, 120]],
            Tensor[float][[84]],
            Tensor[float][[10, 84]],
            Tensor[float][[10]]
        ]
    ) -> Tensor[float][[10]]:
        (p1, b1, p2, b2, lA1, lb1, lA2, lb2, lA3, lb3) = params
        layer1 = larger(corr2d_multi_in_out(x, p1, b1, 1, 1, 2, 2), 0)
        layer1 = pool2d_avg(layer1, 2, 2, 2, 2, 0, 0)
        layer2 = larger(corr2d_multi_in_out(layer1, p2, b2, 1, 1, 0, 0), 0)
        layer2 = pool2d_avg(layer2, 2, 2, 2, 2, 0, 0)
        flat = layer2.reshape([16 * 5 * 5])
        layer3 = larger(linear(flat, lA1, lb1), 0)
        layer4 = larger(linear(layer3, lA2, lb2), 0)
        layer5 = linear(layer4, lA3, lb3)
        return layer5

After layer4, we use reshape to flatten the rank-3 tensor to rank-1.

D

loss starts with the output of predict. What are the desired properties of this Tensor[float][[10]]?

D

Good. We need a list of probabilities that add up to 1, and each should be non-negative.
Here's our first function.

@op
def softmax[n](x: Tensor[float][[n]]) -> Tensor[float][[n]]:
    x_exp = x.exp()
    return x_exp / x_exp.sum(0)

We raise Euler's number to the power of each score, ensuring positivity,
and then divide these positive numbers by their sum.

D

Any positive base works, but Euler's number is usually more convenient.
The gradient of x.exp() is itself. So learning can be made more efficient.
Let's run an example.

D

With softmax, we may define our loss function: cross_entropy.
There are n possible items and m inputs. For each input, ys_pred stores predicted scores for all items. ys contains the indices of the actual items, such as 0 for T-shirt, 1 for dress, and 2 for sneaker.

@op
def cross_entropy[m, n](
    ys_pred: Tensor[float][[m, n]],
    ys: Tensor[int][[m]]
) -> float:
    target_probs = [
        probs[idx]
        for (probs, idx) in zip(softmax(ys_pred), ys)
    ]
    return -target_probs.log().avg(0)

D

It amplifies the loss when our model is confidently wrong.
If a picture is a dress but our probability is 0.00001, then the logged loss is 9.21, much larger than a linear penalty.

D

Our LeNet is complete!

learning_rate = 0.1

class LeNet(Model):
    def predict(
        x: Tensor[float][[1, 28, 28]],
        params: Tuple[
            Tensor[float][[6, 1, 5, 5]],
            Tensor[float][[6]],
            Tensor[float][[16, 6, 5, 5]],
            Tensor[float][[16]],
            Tensor[float][[120, 16 * 5 * 5]],
            Tensor[float][[120]],
            Tensor[float][[84, 120]],
            Tensor[float][[84]],
            Tensor[float][[10, 84]],
            Tensor[float][[10]]
        ]
    ) -> Tensor[float][[10]]:
        (p1, b1, p2, b2, lA1, lb1, lA2, lb2, lA3, lb3) = params
        layer1 = larger(corr2d_multi_in_out(x, p1, b1, 1, 1, 2, 2), 0)
        layer1 = pool2d_avg(layer1, 2, 2, 2, 2, 0, 0)
        layer2 = larger(corr2d_multi_in_out(layer1, p2, b2, 1, 1, 0, 0), 0)
        layer2 = pool2d_avg(layer2, 2, 2, 2, 2, 0, 0)
        flat = layer2.reshape([16 * 5 * 5])
        layer3 = larger(linear(flat, lA1, lb1), 0)
        layer4 = larger(linear(layer3, lA2, lb2), 0)
        layer5 = linear(layer4, lA3, lb3)
        return layer5

    def loss[m](ys_pred: Tensor[float][[m, 10]], ys: Tensor[int][[m]]) -> float:
        return cross_entropy(ys_pred, ys)

    def update(p: float, g: float) -> float:
        return p - learning_rate * g

learning_rate is a hyperparameter. We may tune it for better learning results.

D

Our upcoming model has even more parameters. So yes, we need a better way to organize the parameters.
Let's start with revamping linear.

class LinearParams[o, i]:
    A: Tensor[float][[o, i]]
    b: Tensor[float][[o]]

@op
def linear[i, o](
    x: Tensor[float][[i]],
    p: LinearParams[float, o, i]
) -> Tensor[float][[o]]:
    return (x * p.A).sum(1) + p.b

Now it's your turn. Revamp corr2d_multi_in_out.

D

Next we create a class for LeNet parameters. How many corr2d layers and linear layers are there?

D

So, here are the parameters for LeNet.

class LeNetParams:
    corr1: Corr2dParams[6, 1, 5, 5]
    corr2: Corr2dParams[16, 6, 5, 5]
    linear1: LinearParams[float, 120, 400]
    linear2: LinearParams[float, 84, 120]
    linear3: LinearParams[float, 10, 84]

We pack two parameters for two corr2d layers, and three parameters for three linear layers. We also instantiate the os and is with concrete numbers.

D

Yes, here's our final LeNet, with parameters properly organized.

class LeNet(Model):
    def predict(
        x: Tensor[float][[1, 28, 28]],
        p: LeNetParams
    ) -> Tensor[float][[10]]:
        layer1 = larger(corr2d_multi_in_out(x, p.corr1, 1, 1, 2, 2), 0)
        layer1 = pool2d(layer1, 2, 2, 2, 2, 0, 0)
        layer2 = larger(corr2d_multi_in_out(layer1, p.corr2, 1, 1, 0, 0), 0)
        layer2 = pool2d(layer2, 2, 2, 2, 2, 0, 0)
        flat = layer2.reshape([16 * 5 * 5])
        layer3 = larger(linear(flat, p.linear1), 0)
        layer4 = larger(linear(layer3, p.linear2), 0)
        layer5 = linear(layer4, p.linear3)
        return layer5

    def loss[n](ys_pred: Tensor[float][[n, 10]], ys: Tensor[int][[n]]) -> float:
        return cross_entropy(ys_pred, ys)

    def update(p: float, g: float) -> float:
        return p - lr * g

Now it's time to take a break. In the next chapter, we will build a transformer.