Agents learn to talk. The model builds its own RAG.

Day 1 put two new circuits into TONN: a message board shared by copies of the model (AG) and retrieval from a library (RG). Day 2 asked whether either one learns anything. Each test below is built so that only the new circuit can solve it. Each has a control that removes the circuit.

All runs are on one Mac Studio GPU (Apple M3 Ultra). The models are small, from 0.8M to 13M parameters, so these results test mechanisms, not the final model.

01The relay: agents must talk to score

Task. The information is split across a swarm. Fact holders read chunks like [FACT, key, value]. The asker reads questions like [ASK, key, ?] about those facts. Only the asker is scored. Values are drawn at random for every example, so the asker can beat chance (1 in 64) only if the fact holders pass the facts over the board.

The fact holders get no loss of their own. Everything they learn to post comes from the asker's errors, flowing back through the board.

Result. Trained as a swarm, the asker answers 99.8% of questions correctly, on all three seeds. Cut the board at test time and it falls to 1.5%, which is chance. The same model trained alone stays at chance (1.4% to 2.2%).

The board also scales a little. With 4 agents, 3 fact holders with 4 facts each and one asker asking about all 12, accuracy is 99.8%. With 16 facts per agent it is 99.9%.

What this does not show is that a board helps on ordinary text. That needs a different test.

02Swarms on real text

Setup. FineWeb-Edu as raw bytes. In a split swarm, 4 agents read 4 consecutive windows of one stretch of text, so each agent could use what the agents before it read. The control is the same batches with the board cut during training. Every model is scored on the same held-out swarms, in bits per byte.

On pure TONN (5.2M parameters), the board gains 0.006 bits per byte. On the hybrid that became the default later on day 2, the gain more than doubles:

• Swarms beat the matched control by 0.014 bits per byte on average (about 0.8%), on both seeds.
• Every position in the swarm gains.
• The swarm-trained model is better than the control even with its board cut, by 0.010. Training as a swarm improves each agent on its own, beyond what it reads.

Split batches have a cost. A batch of 32 rows read in swarms of 4 holds 8 independent stretches of text instead of 32. On pure TONN the control trained on split batches is 0.021 worse than ordinary training. Pretraining will need more rows per batch, or swarms that read related documents instead of one stretch.

03The model builds its own RAG

Task. A library of facts [FACT, k1, k2, value], with two-token keys and values drawn fresh at every training step, so nothing can be memorized in the weights. The model reads a question [ASK, k1, k2] and must give the answer [ANS, value] in the next chunk. Chance is 1 in 64.

To beat chance, the model has to learn every part of retrieval by itself: its own embeddings for the library, what to search for, which result to trust, and how to read the answer out of it.

Result. With the library attached, the model answers 100% of questions over 1,024 facts it has never seen. With the library removed it falls to 1.5%. The same model trained without a library reaches 2.3%.

04Three fixes before retrieval learned anything

The first design never learned. For 4,000 steps, accuracy sat at chance. Small diagnostic runs found three separate blockers. Each one alone was enough to stop learning.

1. The query and the stored keys lived in unrelated spaces. The query came from the circuit's own stream; the keys came from a separate encoder. The search was no better than random. Fix: the query for chunk c is the same encoder applied to chunk c-1, plus a learned share of the circuit's own stream that starts at zero.
2. An output gate scrambled the retrieved value. The read was multiplied by a gate computed from the reading position. A linear probe recovered the value from the read 68% of the time before the gate and 2.5% after it. Fix: no gate.
3. A zero-initialized matrix drowned the retrieved tokens' identity. It added the encoder's in-chunk states to each retrieved token. Adam moves every entry of a zero matrix by about the learning rate per step, whatever the gradient, so within 100 steps the added noise swamped the token identity the match depends on. Fix: retrieved tokens are plain token embeddings, read through the layer's own input projection.

Two smaller choices helped. The keys look one token back, so "find the key, take the token after it" is one attention step. And the weighting of the 4 retrieved chunks starts soft, so all four get gradient early on.

Controls along the way confirmed where the problem was. The same model learned the lookup with the facts in its own context (95% in 1,000 steps), so the task was learnable; the failure was in retrieval. With two fixed libraries, the broken design could still fit answers that differ only in the library, so gradients were flowing.

05Also decided: a 32k tokenizer

Byte-level BPE tokenizers with 32k, 49k and 64k tokens, trained on 200 MB of English, 160 MB of Japanese and 80 MB of code. Bigger vocabularies pack more text into each token. But at a fixed number of training tokens, their extra embeddings are seen less often. They also cost parameters.

At 0.1B, 32k instead of 49k frees 10.5M parameters, about 10% of the model. The tokenizer is on Hugging Face.

06What this does not show

The relay and the lookup prove the circuits can learn. They do not show that retrieval helps on real text. That test, with a result we had to correct, is in Retrieval on real text needs a better search.