The Complete Line

D

We are ready for our first Model. Let's start with a refreshed Line.

from pypie import Model, Tensor
from typing import Tuple

class Line(Model):
    def predict(x: float, params: Tuple[float, float]) -> float:
        w = params[0]
        b = params[1]
        y = w * x + b
        return y

    def loss[n](ys_pred: Tensor[float][[n]], ys: Tensor[float][[n]]) -> float:
        return ((ys_pred - ys) ** 2).sum(0)

We always omit @ops for the definitions in a Model.

D

A Model learns by repeating three steps: predict with the current params, compute a loss, and then update the params.
Line is not complete, since it doesn't know how to update yet.

D

Here's how we update.

    def update(p: float, g: float) -> float:
        return p - 0.01 * g

During each repetition, the Model finds a gradient g for each scalar p in params. The gradient tells how loss changes with respect to p: if g is positive, increasing p raises loss; if g is negative, increasing p lowers loss.
To reduce loss, update adjusts p in the opposite direction of g with a small factor. The adjusted value becomes part of the new params for the next repetition.

D

Yes, it is now ready to learn. The learn function expects four inputs: xs, ys, params, and revs for the number of repetitions.
Try 50 revs, with the xs, ys, and params from the last chapter.

D

Take this challenge!

from pypie import rand

xs = rand([1000], -10.0, 10.0)
real_params = (1.0, 0.5)
ys = Line.predict(xs, real_params) + rand([1000], -1.0, 1.0)

rand takes a shape, a lower bound, and an upper bound. It generates a Tensor of that shape using random numbers within the bounds.

D

nan means not a number.
Machines store numbers with limited precision and range. Repeated squaring and summing can make gradients too large, so updates can blow up params; this is called exploding gradients.
Gradients may also get very small. Then updates become tiny and learning nearly stops; this is called vanishing gradients.

D

Yes, it starts with a new function, smooth.

@op
def smooth(decay: float, avg: float, g: float) -> float:
    return decay * avg + (1.0 - decay) * g

Try an example. For decay, 0.9 is usually a safe default.

D

Now we give update extra information. Here are two friends of update, called inflate and deflate.

    def inflate(p: float) -> Tuple[float, float]:
        return (p, 0)

    def deflate(P: Tuple[float, float]) -> float:
        return P[0]

When learning starts, it inflates each scalar in params with an additional float.
When learning ends, it deflates params by removing the additional floats.

D

Good observation. Here is the updated update. The added float stores the smoothed g, using its average of previous repetitions.

    def update(P: Tuple[float, float], g: float) -> Tuple[float, float]:
        avg = smooth(0.9, P[1], g ** 2)
        alpha = 0.01 / (1e-8 + sqrt(avg))
        return (P[0] - alpha * g, avg)

Instead of the fixed 0.01, we adjust p with a more adaptive alpha.
This approach is called RMSProp, for propagating the root-mean-square.

D

Maybe magical, but not arbitrary. In practice, people place variables in those positions, called hyperparameters, and then adjust their values based on science, engineering, and sometimes alchemy.
Here's our new Model using RMSProp.

class LineRMS(Model):
    def predict(x: float, params: Tuple[float, float]) -> float:
        w = params[0]
        b = params[1]
        y = w * x + b
        return y

    def loss[n](ys_pred: Tensor[float][[n]], ys: Tensor[float][[n]]) -> float:
        return ((ys_pred - ys) ** 2).sum(0)

    def update(P: Tuple[float, float], g: float) -> Tuple[float, float]:
        avg = smooth(0.9, P[1], g ** 2)
        alpha = 0.01 / (1e-8 + sqrt(avg))
        return (P[0] - alpha * g, avg)

    def inflate(p: float) -> Tuple[float, float]:
        return (p, 0)

    def deflate(P: Tuple[float, float]) -> float:
        return P[0]

Let's try the challenging example with 200 revs.