Inspect a Neuron's Learning Step

In this chapter

• Follow the chain rule through a nonlinear prediction.
• Compare an analytic derivative with a finite-difference estimate.
• Recognize what a toy training loop does not establish.

Make learning a visible calculation

A neural model combines parameterized operations with a loss, then updates its parameters to reduce that loss. The machinery becomes easier to reason about when one step can be checked by hand. Our toy neuron predicts tanh(weight times input). Tanh restricts the output to between minus one and one, so the synthetic targets use that same scale. This is a calculus demonstration, not a direct model for unscaled repair hours.

For squared error, the derivative with respect to the prediction is twice the prediction error. The derivative of tanh with respect to its input is one minus tanh squared. Multiplying by the input gives the derivative with respect to the weight. Average these contributions over the examples to obtain the loss gradient. A gradient-descent step subtracts a positive learning rate times that gradient; it does not simply move the weight toward the label.

Verify the derivative independently

The finite-difference estimate evaluates the loss on either side of the current weight. It approximates the local slope without reusing the analytic derivative. The two calculations should agree within the stated tolerance, and a modest update should reduce this toy loss. Save the snippet as neuron_step.py and run it locally.

from math import tanh, isclose
from statistics import mean

examples = [(-1.0, -0.6), (0.0, 0.0), (1.0, 0.6)]

def loss(weight):
    return mean((tanh(weight * x) - target) ** 2 for x, target in examples)

def gradient(weight):
    terms = []
    for x, target in examples:
        prediction = tanh(weight * x)
        terms.append(2 * (prediction - target) * (1 - prediction ** 2) * x)
    return mean(terms)

weight = 0.2
epsilon = 1e-6
numeric = (loss(weight + epsilon) - loss(weight - epsilon)) / (2 * epsilon)
assert isclose(gradient(weight), numeric, rel_tol=1e-5, abs_tol=1e-7)
updated = weight - 0.2 * gradient(weight)
assert loss(updated) < loss(weight)
print({"analytic_gradient": gradient(weight), "numeric_gradient": numeric,
       "loss_before": loss(weight), "loss_after": loss(updated)})

Recognize optimization failures

The failure mode is trusting a decreasing training loss as proof of generalization. The toy uses the same three examples to calculate its gradient and inspect the update. It establishes a derivative and an optimization step, not future predictive quality. A large learning rate can overshoot; saturated activations can produce small gradients; poorly scaled inputs can distort optimization. Finite differences also become unreliable if the step is so tiny that floating-point rounding overwhelms the change.

In a documented PyTorch extension, represent parameters as tensors, compute the loss through supported operations and use automatic differentiation. Compare one tiny autograd result with this independent calculation before expanding the network. Keep the time-aware validation split and baseline comparison from earlier lessons. A larger network does not repair a leaked target, and automatic differentiation does not select the right target, evaluation protocol or business decision for you.

Run a controlled gradient experiment

• Repeat the gradient comparison at weights 0.1, 0.5 and 1.0.
• Try several learning rates and record loss before and after a single step.
• Explain why tanh output cannot directly represent an unrestricted positive turnaround duration without changing the model or target representation.

Expected checks

• Analytic and numeric gradients agree within an explicit tolerance.
• Learning-rate experiments are labeled training demonstrations.
• Any PyTorch extension retains a held-out evaluation and transparent baseline.

Check your understanding

What does a successful finite-difference check establish?

• That the model will generalize to future repairs.
• That the analytic gradient agrees with a local numerical approximation for the tested case.
• That a deeper network is always preferable.

Official tools & further reading

• PyTorch automatic differentiation tutorial