Day 1 ·
Starting ExeaLM
An open 0.1B language model built by high schoolers, with retrieval and agent messaging inside its layers.
- Date
- 2026-11-10
- Read
- 4 min read
- Topics
- Plan, Architecture
- By
- The Exea team
parameters planned
training tokens planned
circuit types a TONN layer can hold
ExeaLM is an open language model built by high schoolers at Exea Labs. It will have 0.1 billion parameters. It will train on about 100 billion tokens of English, Japanese and code, then be post-trained into an assistant. The code, the data recipe, every result and these notes are public.
01The bet
A model of 0.1B parameters is small. On its own it can hold little knowledge and work on short problems only. ExeaLM tests whether two abilities, built into the architecture, change that:
- Its own retrieval. The model learns its own embeddings. It decides what to search for, then reads text from a read-only library. Its knowledge does not all have to fit in 100 million weights.
- Talking to copies of itself. Several copies of the model, a , share a message board. A large problem can be split across the copies. They pass what they find over the board.
Both are trained from the first step of pretraining, end to end, with ordinary gradients. The headline test comes later: a swarm of 0.1B agents with retrieval against one larger open model, on the same tasks, at the same inference compute. This is a hypothesis. These notes will say so plainly if it fails.
02Why 0.1B parameters and 100B tokens
100B tokens on 0.1B parameters is 1,000 tokens per parameter, about 50 times the compute-optimal ratio of about 20. Small models are trained past that ratio on purpose, because they are cheap to run afterwards. A swarm runs many copies, so that matters even more here.
The returns fall quickly. By the loss fit of Hoffmann et al. (2022), predicted loss drops by about 0.19 going from 20B to 100B tokens. From 100B to 200B it drops by about 0.06; from 200B to 400B, by about 0.05. At about 300 TFLOPs per GPU, 100B tokens on 0.1B parameters is about 56 GPU-hours.
03The architecture: TONN
ExeaLM is built on TONN (pronounced "tone"), the architecture from our paper. A layer replaces attention with a row of small :
- One input projection reads the residual stream (640 channels) into 20 slots. Each slot gets three , A, B and C, of 32 channels each.
- Each slot runs one circuit type: a delay, a short convolution, a running sum, a state space, a small attention head, and others.
- Each slot's output is divided by a running estimate of its size, so every slot speaks at the same volume.
- One output projection writes the 20 results back to the residual stream. A gated MLP follows, as in a transformer.
Which circuit sits in which slot is not fixed by hand. During training, TONN's switches off one group of circuits at a time. It measures how much the loss rises, then redraws the weakest slots. The network picks part of its own architecture from the data.
Copies stream A from a fixed number of steps back (the lag).
Adds A now, B one gap back and C two gaps back, then a smooth gate (silu). Good at short word pieces.
Keeps a sum of A that fades by a learned rate per channel.
Like the running sum, but B decides at every step how much of the past to keep.
Multiplies a lagged A by B and keeps a fading sum of the result.
A running sum of products, read out through C. It behaves like an induction head: see a pair, recall it later.
A weighted average of B over the whole past, with weights exp(A).
A Mamba-like state: B sets the step size at each position, so the circuit chooses what to remember.
A state that rotates as it fades (S4D-like), so it can track rhythms and periodic patterns.
silu(A) times B at the same position. A small per-position MLP.
A from a few steps back times B now.
One small attention head inside a slot: A asks, B answers, C is what gets passed on.
Searches a read-only library with the model's own embeddings and reads the chunks it finds.
Posts a message after every chunk and reads the messages other copies of the model posted.
04Two new circuits
ExeaLM adds two circuit types to TONN's menu. Every TONN layer is guaranteed one of each. The search cannot remove them.
- RG, retrieval. Once per of 64 tokens, the circuit turns the chunk it just read into a query, searches the library with the model's own chunk encoder, takes the top 4 chunks and reads them.
- AG, the agent board. After each chunk, every agent posts a message: a value, a key and a strength. In later chunks, every agent reads the earlier posts on the board with attention, its teammates' and its own. An empty entry stands for "nothing useful".
In training, consecutive rows of a batch form swarms of 4 that share one board. Gradients flow through the board, so an agent learns what to post from how much its teammates' losses fall.
05What went up on day 1
- The repository, the Apache 2.0 license, the contributing guide and the code of conduct.
- The skeleton (commit 7a63ac9): the TONN model with RG and AG, a datastore, a board, data download and packing, a pretraining loop, and SFT, DPO and GRPO losses, with 16 tests.
- The plan in DATA.md: the token budget and two candidate mixes.
- A to-do list in the README. Most items run on a laptop or a free Colab GPU.
06What day 1 does not show
Nothing in this post is tested yet. The skeleton runs and its 16 tests pass. No model has been trained on real data yet. The bet in the first section is a hypothesis. The next post tests both new circuits on tasks built so that they can only be solved through them.
Who did the work
