Exea LabsBlog
日本語Japanese version

Inside ExeaLM

The 0.1B model, from the layer stack down to one circuit.

Fig. 01ExeaLM 0.1B
00 TONN010203 TONN040506 TONN070809 TONN101112 TONN13RESIDUAL STREAM14 LAYERS640 CHANNELS
14 layers. About 90M parameters.

Plates with circuits are TONN layers. The rest are attention. Point at a layer.

  • Memory
  • Short range
  • Running state
  • Gates and products
  • Attention
01

One TONN layer

One projection reads the residual stream into 20 slots. Each slot runs one small circuit on three streams. One projection writes the results back.

Fig. 02
DDCCCIIIEPEPSMSMOSOSOSOSRPRPRGAGto the next layerResidual stream, 640 channelsInput projection20 slots × streams A, B, C (32 channels each)One circuit per slotUnit scale per slotMask (the search switches slots off to measure them)Output projectionAdd back to the residual streamGated MLP
Point at a circuit.
One TONN layer, top to bottom. Point at a circuit to see what it computes.
02

The circuit menu

Fourteen circuit types can fill a slot. During training, a search keeps the ones that lower the loss.

Fig. 03
Delay
Short range
D

Copies stream A from a fixed number of steps back (the lag).

Short convolution
Short range
C

Adds A now, B one gap back and C two gaps back, then a smooth gate (silu). Good at short word pieces.

Leaky running sum
Running state
I

Keeps a sum of A that fades by a learned rate per channel.

Gated running sum
Running state
GI

Like the running sum, but B decides at every step how much of the past to keep.

Running sum of a product
Running state
MI

Multiplies a lagged A by B and keeps a fading sum of the result.

Gated product memory
Running state
MIM

A running sum of products, read out through C. It behaves like an induction head: see a pair, recall it later.

Softmax pooling
Running state
EP

A weighted average of B over the whole past, with weights exp(A).

Selective state space
Running state
SM

A Mamba-like state: B sets the step size at each position, so the circuit chooses what to remember.

Oscillating memory
Running state
OS

A state that rotates as it fades (S4D-like), so it can track rhythms and periodic patterns.

Gated unit
Gates and products
G

silu(A) times B at the same position. A small per-position MLP.

Product
Gates and products
M

A from a few steps back times B now.

Rotary attention
Attention
RP

One small attention head inside a slot: A asks, B answers, C is what gets passed on.

Retrieval
Memory
RG

Searches a read-only library with the model's own embeddings and reads the chunks it finds.

Agent board
Memory
AG

Posts a message after every chunk and reads the messages other copies of the model posted.

Every circuit type on the same input. Faint bars are the input; the line is the output.
03

Retrieval

Once per 64 tokens, the model searches a read-only library with its own embeddings and reads the top 4 chunks.

Fig. 04
TheThamesflowseastthroughcentralLondonto… 64

Retrieval runs once per 64-token chunk.

The retrieval circuit, one stage at a time.
04

The board

Copies of the model post a message after every chunk and read each other's posts. Gradients flow through the board.

Fig. 05

Every agent reads its first chunk.

BoardnothingAgent 1FACT k7 42FACT k2 61Agent 2FACT k3 19FACT k9 30Agent 3FACT k5 8FACT k1 5AskerASK k7 ?ANS …chunk 1chunk 2
The relay task: three agents hold facts, the asker needs one. Cut the board to see the asker guess.
05

The numbers

Width
640
Layers
14, TONN at 0, 3, 6, 9, 12
Attention
10 heads of 64 channels
TONN slots
20 circuits of 32 channels
Memory circuits
1 retrieval and 1 board per TONN layer
Chunk
64 tokens, top 4 retrieved
Swarm
4 agents per board
Tokenizer
byte-level BPE, 32,768 tokens
Parameters
about 90M
Training
about 100B tokens, planned