Tutorial

A Forward Line

Ready to define a function?

At last!

First: line.

from pypie import op, Tensor
from typing import Tuple

@op
def line(x: float, params: Tuple[float, float]) -> float:
    w = params[0]
    b = params[1]
    y = w * x + b
    return y

These imports are new.

pypie gives us tensors and learning primitives.
typing gives us other type annotations like Tuple.
Tuple[float, float] means params contains exactly two floats.

What about op?

@op marks the next def as a pypie operator.
That lets pypie validate its correctness and parallelize its execution.

In line, we unpack params into w and b, compute y = w * x + b, then return y.
Is that the whole story?

Yes.
What is the type of y?

I want to sanity-check.
Earlier, + showed up with int tensors.
Here w, x, and b are floats.
Why is w * x + b still valid?

Good check. Let's generalize the type of +:
[T] (x: Tensor[T][[]], y: Tensor[T][[]]) -> Tensor[T][[]].
Here T is an implicit variable. It makes both inputs share the same numeric type. At call time, pypie figures out the concrete T from the actual inputs.
BTW, * has the same type as +.

So here T becomes float, and w * x + b has type float.

Exactly. pypie compares the return expression type against the definition.
If they differ, it raises a type error.

Then if I annotate the result as int...

@op
def bad_line(x: float, params: Tuple[float, float]) -> int:
    w = params[0]
    b = params[1]
    y = w * x + b
    return y

Right. The type checker reports:
int != float.

Because line is rank polymorphic, the same definition applies to tensors.
Let's try params = (1.0, 0.5) and a Tensor of xs.

Like this?

params = (1.0, 0.5)
xs = Tensor([-4.2, 1.0, 1.5, 3.0])
ys = line(xs, params)
print(ys)

Result: Tensor([-3.7, 1.5, 2.0, 3.5]).

Those are the four points on the line.

Visualization of four points on the line

Clean line! No surprises.

Next step: learn the line.

Learn what?

It means finding params from xs and ys.
So far, we chose params, made xs, and computed ys.
In practice, we first have some real data for xs and ys and then estimate params.
Let's pretend:
xs and ys are real data;
true params are unknown.
Then we write a program that estimates params.

Okay. How do we begin?

Start with an initial guess for params.
Any guess works; say (0, 0).
Run line(xs, params) and call the output ys_pred.

You mean this?

params = (0, 0)
ys_pred = line(xs, params)
print(ys_pred)

We get Tensor([0, 0, 0, 0]). ys_pred is far from the real ys.

Expected.
Now we need to measure how far off we are. This is called the loss.

Do we create a new function to measure the loss?

It's called loss.

@op
def loss[n](ys_pred: Tensor[float][[n]], ys: Tensor[float][[n]]) -> float:
    return ((ys_pred - ys) ** 2).sum(0)

Does [n] introduce a new implicit variable?

Yes, n describes a dimension of the input tensors. It tells PyPie that ys_pred and ys share the same shape. So that ys_pred - ys is valid.

Why square with ** 2 and then call sum(0)?

Almost always, we use a scalar as the loss.
We first square each difference, so negatives and positives cannot cancel out during the following sum.
This gives us a Tensor[float][[n]].
We then collapse the first dimension of this tensor to one scalar, using sum(0).
Let's run loss.

Like this?

print(loss(ys_pred, ys))

It prints 32.19, the loss at (0, 0)?

Exactly.
Then we update params, recompute loss, and repeat.

Do we keep going until loss is 0?

Usually not.
For real data, exact zero is uncommon and often undesirable.
In practice, the training data usually contain some noise and errors. Reaching zero often means overfitting: params have also learned from the noise and errors, and may perform worse on new, unseen data.
For now, we simply update and repeat for a fixed number of iterations.

Will we define more functions to update and repeat?

We will, in the next chapter.

Brain break earned!