Timeseries Engineering

Prices around 100 fed straight into a tanh unit with small random weights: 77% of the activations are saturated above 0.99, where the gradient is almost zero. Standardise the same prices first and 0% are. Neural forecasters need scaled inputs — usually per series — before anything else matters.

• Scaling with statistics from the whole series leaks the future into training; fit the scaler on the training window only.
• RNNs need many related series to shine; on a single short series classical models often win (Hewamalage et al. 2021).
• Recursive multi-step forecasting compounds errors; direct or sequence-to-sequence outputs are usually more stable.

rng = np.random.default_rng(10)
prices = 100 + rng.normal(0, 5, 1_000)            # raw prices around 100
w = rng.normal(0, 0.1, 1_000)                      # small random input weights
saturated_raw = np.mean(np.abs(np.tanh(w * prices)) > 0.99)
z = (prices - prices.mean()) / prices.std()        # standardized
saturated_std = np.mean(np.abs(np.tanh(w * z)) > 0.99)

Assumes import numpy as np and import pandas as pd. Each snippet simulates or makes up its own series, as the comments say.

# PyTorch — basic RNN for forecasting
import torch.nn as nn

class TSRNN(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super().__init__()
        self.rnn = nn.RNN(input_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, 1)
    def forward(self, x):
        out, _ = self.rnn(x)
        return self.fc(out[:, -1, :])