Small model · from scratch

From a folder of text to a model I can talk to

The whole experiment is one loop: prepare the right data, build a small prediction machine, watch it learn, then teach it the shape of a conversation.

STATUS · EXPERIMENTING STARTING POINT · RANDOM WEIGHTS RESULTS · NONE CLAIMED YET

The map

Six stops. Each one leaves a useful thing for the next.

  1. 01 · DEFINE

    Pick one job

    Write the test before the training.

  2. 02 · PREPARE

    Curate licensed text

    Clean, deduplicate, split, record rights.

  3. 03 · BUILD

    Tokenizer + transformer

    Choose the vocabulary and model shape.

  4. 04 · PRE-TRAIN

    Predict the next token

    Guess, score, adjust—many times.

  5. 05 · SHAPE

    Fine-tune for chat

    Imitate reviewed conversations.

  6. 06 · CHECK

    Evaluate and repeat

    Keep the best checkpoint; fix the weakest stage.

Two datasets teach two different lessons

Broad corpus lots of clean text
Base model learns how text continues
Chat JSONL smaller, reviewed examples
user assistant user assistant
Chat behavior learns how to respond

License gate: permission to read something is not automatically permission to train on it or redistribute the result.

What I am actually building

A tokenizer turns text into pieces. A randomly initialized transformer repeatedly mixes those pieces and predicts what comes next.

Contextsize of the desk
Widthroom per token
Blocksprocessing depth
Headsparallel views
Parameterslearned numbers

BPE tokenizer: starts with tiny pieces, then joins common neighbors—such as train + ing. The model sees token IDs, not raw words.

One training step, drawn out

01 · BATCH Load sequences small rows of token IDs
02 · GUESS Predict next tokens one probability list per position
03 · SCORE Calculate loss how wrong were the guesses?
04 · ADJUST Optimizer moves weights learning rate sets step size
05 · SAVE Checkpoint weights + training state

8 sequences × 256 tokens = 2,048 training tokens per step · 10,000 steps ≈ 20.5M token positions

WARMUPgentle early updates
STEADY TRAININGrepeat steps across shuffled data
ONE EPOCHabout one corpus pass
SAMPLEcontinue a plain prompt

The loss chart tells me when to keep going—and when to stop

Illustrative training and validation loss curves Both curves fall during useful learning. Later, training loss keeps falling while validation loss rises, which signals overfitting. training loss validation loss best checkpoint
BOTH FALLuseful learning
BOTH STAY HIGHundertrained or misconfigured
TRAIN ↓ · VALIDATION ↑overfitting; stop or revise

ILLUSTRATION, NOT A RESULT · COMPARE VALIDATION LOSS ONLY WITH THE SAME DATA + TOKENIZER SETUP

Base model → useful chat model

Sample the base

Continue unfinished prose; do not expect chat yet.

Supervised fine-tune

Imitate role-labeled conversations in JSONL.

Optional preference pass

Learn from chosen vs. rejected answers.

Evaluate + iterate

Fixed prompts, held-out text, failure review.

  1. 01
    Goal + fixed testsdefine success first
  2. 02
    Rights + provenanceknow every source
  3. 03
    Frozen validation splitnever train on the exam
  4. 04
    Tiny pipeline testprove the loop works
  5. 05
    Record each rundata, settings, checkpoint
  6. 06
    Scale one thingdata, model, context, or time
Freshness limit: the model only knows what was in its training data. New facts require newer licensed data and more training, or a separate retrieval system that supplies current information at answer time.