TypeScript ML examples often stop at calling an existing model. I wanted to check the other path: define a decoder, train it on text, evaluate it, and save its state, all from a TypeScript application.
I used Affon's decoder app. It defines embeddings, causal attention, and six decoder layers. The bundled WikiText-2 text becomes 256-token windows. The training workflow evaluates batches, checks validation loss, and writes a checkpoint.
Train with a smaller tokenizer
With Affon 0.4.2 installed on macOS arm64, this runs on Metal from a source checkout:
git clone https://github.com/neucine/affon.git
cd affon
mkdir -p apps/decoder-lm/artifacts/wikitext-8k
RUNTIME_PACKAGE_PATH="$PWD/packages" \
AFFON_TRAIN_CONFIG=apps/decoder-lm/configs/train-decoder-lm-wikitext-8k.config.json \
affon run apps/decoder-lm/train.ts
This configuration uses an 8,192-token tokenizer trained on the WikiText-2 training split. It runs a six-layer decoder for 256 training batches, then evaluates up to 32 validation batches. In the saved run, validation loss moved from 9.070 to 6.579, final training loss was 7.093, and the app saved first-run-epoch-1. The generated sample still repeated words and symbols. This capped run shows learning on those batches; it does not establish useful generated text. Loss from a different tokenizer is not directly comparable.
Three full epochs on CUDA
A longer run used the same 8,192-token vocabulary, six decoder layers, and 256-token windows on an RTX 3090. Each epoch covered 4,995 batches of four windows, followed by the full validation split.
| Completed epoch | Training loss | Validation loss | Validation perplexity |
|---|---|---|---|
| 1 | 6.1247 | 5.4641 | 236.1 |
| 2 | 5.2572 | 5.1339 | 169.7 |
| 3 | 5.0241 | 5.0668 | 158.7 |
Perplexity expresses next-token prediction error; lower is better for the same tokenizer and evaluation data. Training loss is the average while the weights change during each epoch. Validation uses the saved end-of-epoch weights on held-out text.
The model is learning, but its text is still repetitive and incoherent. With the same prompt, The history of, epoch three began:
The history of the Church of the Church of the United War , the Bestory of the Church of the Church of the Dangis . He was built in the English , which was established in the Church of the American and the early period of the Mlego-Parian of the English " . The king was the earliest known as the American life of the Church
Epoch one used the published Affon 0.4.2 binary. Epoch two restored its model and Adam optimizer checkpoint using a local runtime with a CUDA reduction fix. The first resumed loss agreed within 0.000001. That fix is not part of the published 0.4.2 binary. These are results after 14,985 training steps, not a text-quality benchmark or a speed comparison with other frameworks.
Why the model definition matters
Affon calls the computation a Program. The decoder Program names its input shape and parameters, including the token embedding table and the layers that use it. A training Program is derived from the model, a cross-entropy loss, and an optimizer. The app compiles those Programs for a Session and keeps model state explicit.
That structure gives me something to inspect before execution: which operations and tensor shapes the model asks for, and where parameters belong in the nested decoder. It also lets the same model definition serve training and inference. The model source and workflow show the full path.
The complete walkthrough covers installation, output files, and the training configuration. If you work on TypeScript ML, I would value feedback on the Program boundary: what would you need to inspect before trusting a larger training run?
Top comments (0)