herambve

Chasing Copy Heads in a Residual Streams Avenue

Sun, 16 Nov 2025 00:00:00 GMT

draft: true

WIP

I’ve been looking into attention mechanics recently, and this is my list of questions I want to answer in relation to copy heads to document my understanding:

1. Definitions

copy heads, the OV circuit, frobenius distance, identity matrix, singular value decomposition, etc.

2. Background of all this

how did I reach to this point of thinking about copy heads?
the essence of W_OV: why we need it, what it does, what it signifies (identity matrix relevance, wwhether there is transformation or whether the information from the incoming residual stream is merely getting put into the blender, when W_OV is essential: is it calculated at any point during training or inference?)

3. Importance of copy heads across different layers

how do I know when a head is a copy head? more formalized and comprehensive than the definition
is a copy head in, say, layer 1, as important as that in, say, layer 12 of a transformer?
is a copy head even essential at all? why don’t we simply focus on transform heads?

4. Critical questions

Do we not pass all the new features that are learned in specific heads (in turn specific layers) to the residual stream?
Does this not mean the features are always there in the residual stream for, say, head 3 in layer 13 if it wants to look up the a specific feature information from, say, head 6 in layer 1? why specifically require a copy head here? (mostly to do with row specifications ⇒ residual stream is like a spreadsheet of rows and columns ⇒ what if we want the info on row 3 of head 1.6 when we are in row 42 of head 13.3? can’t simply utilize the residual stream ⇒ attention is how we copy info from a row to the other, no?)

All answers to be expanded with concrete examples (python, torch, visual diagrams).

Got nerd sniped by copy heads; I couldn’t stop thinking of this meme in the process & thought this applied to them lol

I found copy heads pretty interesting as I was going through some mechanical interpretability basics; I had jotted down some questions on a piece of paper for testing myself at the end of my learning to get a better intuition behind this concept, and I write this log answering all those questions as a reinforcement of my understanding.

Definitions

Before diving into the mechanics behind copy heads across different transformer layers and their importance, some definitions are necessary to establish the foundations. Some key definitions coming up.

Copy heads

Copy heads are attention heads that are identity matrices or are close to being identity matrices. Inside a multi-head attention phase, we split the embedded dimensions into specific number of heads and do a parallel computation: in each head, we calculate the attention weights, do a matmul between attention weights and the V (here V = x @ W_V) to calculate attention output and then when we do another matmul between the attention output matrix and a projection matrix W_O, we get what we call an output projection.

Sometimes, an attention head barely transforms any information that it receives from the residual stream; what it outputs could simply be a copy (or close to a copy) of what it receives from the residual stream; in other words, a weighted average of the input and nothing more, nothing less. This sort of an attention head is called a copy head.

In order to get a clear picture behind a copy head, I think it’s a good idea to get a quick refresher behind how we reach the point of an attention head.

Refresher: How did we reach this rabbit hole?

An attention head, in a transformer, exists inside a layer (or a block), and is a factor of the transformer’s embedding dimension (d_model in the original transformer paper) that stores rich representations & semantic features during the training.

I need to add an illustration of a transformer block here
and also an illustration of attention heads

Let’s assume that I have an input_text called: “I like hot”.

This is an input word, at the very start of the process, before any embeddings, before any forward passes.

Parameters that I’d like to assume for the sake of this example:

batch = 1 (since we have but a single sentence)
seqlen = 3 (sequence length the size of input text)
d_model = 32 (this will be the embedding dimension for every word, i.e., input token id)
n_layers = 8 (total number of transformer blocks)
n_heads = 4 (number of heads in each layer, factor of d_model, will expand more soon) vocab_size = 100

In a transformer, the input_text gets tokenized, which is simply a method to assign unique numeric representation to words or characters. Tokenization can be done via various methods, and not all of them break down a sequence of words (i.e., the input_text here) 1:1 into tokens but for the sake of this worklog, let me assume that each word from the input_text above corresponds to 1 token, kind of like:

tokenizer = {
    "I": 1,
    "like": 2,
    "hot": 3,
} # a simple lookup table, basically a hash map/dict

This is still, to some extent, an oversimplification, but let’s think of tokenization as a step where we create a lookup table (like a dictionary) of words (i.e., tokens) and their unique IDs.

At this point, if we think of this input sequence of unique token IDs as a tensor, it would be:

tokenized_input = torch.tensor([1, 2, 3])
tokenized_input.shape # torch.Size([3])

Transformers do computation parallelly in batches, so generally, inputs are tokenized in batches as well. Basically, a single ‘batch’ represents a collection of tokens, and in a normal LLM, during training, there are countless number of batches, and each batch holds an enormous size of text (tons of sentences). In our case, since we have but just input_text, we are assuming that we have 1 batch, and that means our sequence length will be the length of our tokens, i.e., 3.

So I unsqueeze the tokenized_input to better represent it as [batch, seqlen]:

tokenized_input = tokens.unsequeeze(0)
tokenized_input # torch.tensor([[1, 2, 3]])
tokenized_input.shape # torch.Size([1, 3])

So this new tokenized_input represents the tokenized text that has 1 batch, and the batch’s sequence length is 3.

Moving on, we initialize random embedding layers/dimensions to each token. In the original tranformer paper, there were 512 embedding dimentions (d_model) assigned to each token, but for demonstration, let me simply start off with 32 dimensions per token.

In PyTorch, this is usually done via the torch.nn.Embedding module, where it could be something like:

embedding = torch.nn.Embedding(num_embeddings=37000, embedding_dim=512)
embedding.weight.shape # torch.Size([37000, 512])

token_embedding = embedding(tokenized_input)
token_embedding.shape # torch.Size([1, 3, 512])

Here, 37000 was the vocab size in the original transformer, and each vocab token had a dimension of 512. torch.nn.Embedding creates a lookup table (similar to the tokenizer above) of a certain vocabulary size (that we determine based on the kind of tokenizer we have created) where each token ID has its own unique d_model-dimensional vector. Nothing fancy going on here.

In order to assign random embedding weights for my example, let me think of a vocabulary number. From what I’ve read, it seems that vocab number is generally determined first, and then the tokenizer creates those amounts of tokens and their token ID representations. For a model, any token ID that does not fall inside the tokenizer’s vocab hash map will lead to the model not recognizing it. For me, if I look at the tokenizer has map from above, my vocab_size will be just 3, but let me extend it a little more and create a vocab_size of 100:

tokenizer = {
    "nice": 0,
    "I": 1,
    "like": 2,
    "hot": 3,
    ...
    "?": 98,
    "coffee": 99
}

Right, so this means I can simply create my token embedding for the tokenized_input as follows:

embedding = torch.nn.Embedding(100, 32)
token_embedding = embedding(tokenized_input)
token_embedding.shape # [1, 3, 32]

So each token now also has its vector weight representation of 32 dimensions.

We also add positional encoding from here, which I won’t get into details, but the shape of token_embedding remains the same ([1, 3, 32]) as a result, and finally, what we have is ready to be fed as the first residual stream to the first multihead attention of the first layer of the transformer.

These calculations (token ID calculation, tokenization + vocab prep, embedding, positional encoding) are not done more than once. After the resulting embedded + positionally encoded input IDs get fed into the first sublayer, it is, as the paper[1] I’m reading claims, a residual stream that flows sequentially across each layer of the transformer (layer = transformer block).

Making some progress here. Now, this token_embedding goes into the first MHA of the first layer, and something like the following takes place:

import torch.nn as nn


class MultiHeadAttention(nn.Module):
    def __init__(self, dtype, n_heads: int = 4, d_model: int = 32) -> None:
        super().__init__()
        assert d_model % n_heads == 0, "n_heads should be a factor of d_model"
        self.dtype = dtype
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_heads = d_model // n_heads

        self.W_q = nn.Linear(d_model, d_model, dtype=self.dtype)
        self.W_k = nn.Linear(d_model, d_model, dtype=self.dtype)
        self.W_v = nn.Linear(d_model, d_model, dtype=self.dtype)
        self.W_o = nn.Linear(d_model, d_model, dtype=self.dtype)

    def forward(self, x): # x == token_embedding
        if x.dtype != self.dtype:
            x.dtype = x.to(self.dtype)
        batch, seqlen, d_model = x.shape
        q = self.W_q(x)
        k = self.W_k(x)
        v = self.W_v(x)

        q = q.view(batch, seqlen, self.n_heads, self.d_heads).transpose(1, 2)
        k = k.view(batch, seqlen, self.n_heads, self.d_heads).transpose(1, 2)
        v = v.view(batch, seqlen, self.n_heads, self.d_heads).transpose(1, 2)

        attn_scores = q @ k.transpose(-1, -2)
        attn_weights = torch.softmax(attn_scores / self.d_heads**0.5, dim=-1)

        attn_output = attn_weights @ v
        attn_output_concat = (
            attn_output.transpose(1, 2).contiguous().view(batch, seqlen, d_model)
        )

        proj_output = self.W_o(attn_output_concat)
        return (
            attn_weights,
            proj_output,
        )

This is a very simple multi-head attention (MHA) mechanism, and this sub-layer exists in each transformer block. Each MHA mechanism involves splitting the provided x (token_embedding) into n_heads number of heads to do parallel attention computation (attn_output in the code block above) using d_heads that are of size d_model // n_heads (followed immediately by concatenating all these outputs to rejoin all the split heads — basically, in a way, doing n_heads * d_heads to get back the original d_model — and do a matmul with the projection weight W_O to get the final proj_output). So, in my demo, d_model is 32, and if I choose n_heads as 4, my d_heads (the dimension that each head gets) becomes 8.

We’re now with 4 heads per transformer layer, and each head is doing 8 computations in parallel. But is each head that has been computed and concatenated with the other heads doing meaningful transformation with new learned features and semantics, or could it just be calculating a weighted average of the provided input x and nothing more?

contd…

[1]A Mathematical Framework for Transformer Circuits, Anthropic, 2021. https://transformer-circuits.pub/2021/framework/index.html

Engineering Systems from First Principles

Mon, 31 Aug 2026 00:00:00 GMT

Building from the Ground Up

In an ecosystem where high-level abstractions dominate, there is immense value in understanding what happens underneath the hood:

Systems & Memory: How data structures, concurrency, and memory allocation operate in low-level environments like C/C++.
Distributed Backends: How asynchronous message queues (Redis & Celery) decouple heavy compute workloads from real-time API latency in FastAPI microservices.
Applied AI: How fine-tuned models (e.g. RoBERTa for emotion classification, Whisper for speech-to-text) can be optimized to run efficiently under constrained resource environments.

This blog is where I document my technical experiments, architectural decisions, and worklogs as I build systems and explore new technologies.

Stay tuned for upcoming deep dives!

Inside a Transformer: A Worklog

Thu, 30 Oct 2025 00:00:00 GMT

draft: true

This is currently purely a worklog in progress. Things are not formatted yet.

On residual stream as a communication channel

Residual stream?

image source

Lots about transformer layers (or blocks) & how information flows through them like a ‘stream’, as if different students in a class were given a topic to write an essay about on a long whiteboard & each student writes something new + reads what was written before

Matters mainly for ‘reading’ and ‘writing’: reading the previous layers’ outputs, and writing one’s output (which also happens to be the output from that nth layer added to the previous layers’ outputs)

block 0 ⇒ output ⇒ fed to block 1 ⇒ output (enriched with its own + block 0’s info) ⇒ fed to block 3…

this is a residual stream

reading = mainly Q @ K^T

writing = x = x + attn_output

Inside the attention sub-layer: the mechanics

Note on sub_layer: I believe this is the term the original Transformer uses to call its said block’s primary sections. For instance, the encoder has 2 sub-layers: the multihead self-attention (MHA) layer and the the FFN (or MLP) layer.

I used to understand the steps involved in the attention sub-layer, but never really got around to understanding the “how” and “why” of it.

My knowledge of the MHA mechanism, until I went deep into this, was mostly the following steps:

# assumptions
batch = 1
seqlen = 3
d_model = 32
d_head = 8
assert d_model % d_head == 0
heads = d_model // d_head

1. Initialize the weights for the incoming input x (input = also understood as the residual stream up till that point)

# assuming this already has the embeddings + positional encodings
x = torch.randn(batch, seqlen, d_model)
print(x.shape) # [1, 3, 32]

W_Q = torch.randn(d_model, d_model) # [32, 32]
W_K = torch.randn(d_model, d_model) # [32, 32]
W_V = torch.randn(d_model, d_model) # [32, 32]
W_O = torch.randn(d_model, d_model) # [32, 32]

2. Compute the query, key, and value vectors and split them into heads.

q = (x.__matmul__(W_Q)).view(batch, seqlen, heads, d_head).transpose(1, 2)
k = (x.__matmul__(W_K)).view(batch, seqlen, heads, d_head).transpose(1, 2)
v = (x.__matmul__(W_V)).view(batch, seqlen, heads, d_head).transpose(1, 2)
q.shape, k.shape, v.shape # [1, 4, 3, 8], [1, 4, 3, 8], [1, 4, 3, 8]

3. Compute the attention scores

attn_scores = q.__matmul__(torch.transpose(k, -2, -1))
attn_scores.shape # [1, 4, 3, 3]

mask should also be applied in case of an attention inside the decoder, omitting for now (sample masking is done later here)

4. Compute the attention weights

attn_weights = torch.softmax(attn_scores / d_head ** 0.5, dim=-1)
attn_weights.shape # [1, 4, 3, 3]

5. Compute the attended values

attended_vals = attn_weights.__matmul__(v)
attended_vals.shape # [1, 4, 3, 8]

6. Combine the split heads back into the original dimension

output_heads_combined = attended_vals.transpose(1, 2).contiguous().view(batch, seqlen, d_model)
output_heads_combined.shape # [1, 3, 32]

7. Compute the output projection

output_proj = output_heads_combined.__matmul__(W_O)
output_proj.shape # [1, 3, 32]

But what is the intuition behind these steps? What is the best way of understanding the “how” and “why” of these steps? A few specific questions:

What is the attention weight trying to tell us?
There are batches of tokens that are being processed in parallel to get the output (input + output_proj, i.e., the updated “residual stream”) from the MHA sub-layer of the transformer. How do I exactly visualize this? What happens sequentially? What happens in parallel?
Why do we need the output projection? What is the intuition behind it?

Trying to gain some deeper intuition

The Anthropic paper[1] suggests that we break down the MHA sub-layer into 2 “circuits”: the QK circuit and the OV circuit.

The following is a set of calculations that I did (once again), step-by-step, to gain some deeper intuition.

1. Initializing some initial initializables

For this, I will assume the input is: “The cat sat”. I will also assume the d_model = 8, and d_q[-1] = 4.

torch.manual_seed(42)
seqlen = 3 # because the input is "The cat sat"
attn_scores = torch.randn(seqlen, seqlen) # randomly initializing for the sake of quick testing
d_hidden = 8 # aka d_model
q_last_layer = 4 # aka d_q[-1]

2. Masking, softmaxing, and getting the attention weights (not going to do head splitting here for the sake of simplicity)

upper_triangular = torch.triu(torch.ones_like(attn_scores), 1)
mask_bool = upper_triangular.bool()[:, :]
attn_scores_w_causal_mask = torch.masked_fill(attn_scores, mask_bool, value=torch.tensor(float("-inf")))
attn_w = torch.softmax(attn_scores_w_causal_mask / q_last_layer ** 0.5, -1)
attn_scores_w_causal_mask, attn_scores, attn_w

This gives us:

# attn_scores_w_causal_mask
tensor([[ 0.3367,    -inf,    -inf],
         [ 0.2303, -1.1229,    -inf],
         [ 2.2082, -0.6380,  0.4617]]),

 # attn_scores
 tensor([[ 0.3367,  0.1288,  0.2345],
         [ 0.2303, -1.1229, -0.1863],
         [ 2.2082, -0.6380,  0.4617]]),

 # attn_w
 tensor([[1.0000, 0.0000, 0.0000],
         [0.6630, 0.3370, 0.0000],
         [0.6029, 0.1453, 0.2518]])

The following are assumed to be the attention scores, attention scores (with masking), and attention weights for the input “The cat sat”.

(tensor([[ 0.3367,    -inf,    -inf],
         [ 0.2303, -1.1229,    -inf],
         [ 2.2082, -0.6380,  0.4617]]),

 tensor([[ 0.3367,  0.1288,  0.2345],
         [ 0.2303, -1.1229, -0.1863],
         [ 2.2082, -0.6380,  0.4617]]),

 tensor([[1.0000, 0.0000, 0.0000],
         [0.6630, 0.3370, 0.0000],
         [0.6029, 0.1453, 0.2518]]))

Looking at the 3x3 attention weights matrix, I can picture this as:

\begin{array}{c|ccc} & \text{The} & \text{cat} & \text{sat} \\[0.5em] \hline \\[0.05em] \text{The} & 1.0000 & 0.0000 & 0.0000 \\[0.5em] \text{cat} & 0.6630 & 0.3370 & 0.0000 \\[0.5em] \text{sat} & 0.6029 & 0.1453 & 0.2518 \end{array}

Here, I can think of it like this:

to understand “cat”, how much will I have to look at the token “the”? —> exactly 1.000, so strong relevance for this computation.
to understand “sat”, how much will I have to look at the token “the”? —> 0.6029, so a pretty solid relevance for this computation.
to understand “sat”, how much will I have to look at the token “cat”? —> 0.1453, so an okay-ish relevance, not as strong as the one just before.

3. Computing the attended values

# assuming:
X_IN = torch.randn(seqlen, d_hidden) # [3, 8]
W_V_ = torch.randn(d_hidden, d_hidden)
V = X_IN.__matmul__(W_V_)
W_O_ = torch.randn(d_hidden, d_hidden)
ATTENDED = attn_w.__matmul__(V)
print(f"""W_V: {W_V_.shape} # [8, 8]
{W_V_}

V: {V.shape} # [3, 8]
{V}

attended weights: {ATTENDED.shape} # [3, 8]
{ATTENDED}
""")

# which gives us something like:
W_V: torch.Size([8, 8])
tensor([[ 0.0349,  0.3211,  1.5736, -0.8455,  1.3123,  0.6872, -1.0892, -0.3553],
        [-1.4181,  0.8963,  0.0499,  2.2667,  1.1790, -0.4345, -1.3864, -1.2862],
        [-0.8371, -0.9224,  1.8113,  0.1606,  0.3672,  0.1754,  1.3852, -0.4459],
        [-1.2024,  0.7078, -1.0759,  0.5357,  1.1754,  0.5612, -0.4527, -0.7718],
        [ 0.1453,  0.2311,  0.0087, -0.1423,  0.1971, -1.1441,  0.3383,  1.6992],
        [ 2.8140,  0.3598, -0.0898,  0.4584, -0.5644,  1.0563, -1.4692,  1.4332],
        [ 0.7281, -0.7106, -0.6021,  0.9604,  0.4048, -1.3543, -0.4976,  0.4747],
        [-0.1976,  1.2683,  1.2243,  0.0981,  1.7423, -1.3527,  0.2191,  0.5526]])

V: torch.Size([3, 8])
tensor([[-3.3538, -2.6614,  5.1813, -0.1024,  0.1555, -0.1970, -0.2466, -2.8980],
        [-0.4393,  1.2727,  0.4840, -3.8536, -1.8261,  5.3907,  0.3668, -2.1857],
        [-0.3070, -0.7861,  2.5214, -1.5471, -1.1485,  1.0364,  2.6574, -0.5662]])

attended weights: torch.Size([3, 8])
tensor([[-3.3538, -2.6614,  5.1813, -0.1024,  0.1555, -0.1970, -0.2466, -2.8980],
        [-2.3715, -1.3356,  3.5982, -1.3666, -0.5123,  1.6862, -0.0399, -2.6579],
        [-2.1632, -1.6177,  3.8292, -1.0111, -0.4607,  0.9254,  0.5737, -2.2074]])

4. Computing the output projection

OUTPUT_PROJECTION = ATTENDED.__matmul__(W_O_)
print(f"""{OUTPUT_PROJECTION.shape}
{OUTPUT_PROJECTION}""")

# which gives us something like:
torch.Size([3, 8])
tensor([[ 10.8347,  -4.9027,   7.4389,  -2.5553,   9.3776,  -2.1711, -14.9518,
          -5.0850],
        [  8.7450,  -3.9095,   3.6260,  -2.4063,   5.4063,  -1.0348, -11.8651,
          -2.1552],
        [  9.2331,  -4.1688,   2.8769,  -2.0400,   4.9977,  -1.7839, -11.5206,
          -2.4551]])

5. Combining the attended values and the output projection for the updated residual stream

residual_stream = X_IN + OUTPUT_PROJECTION
# this is bascially:
#   x_in + attended_val @ W_O {here, x_in = (enbeddings + pos_encodings), and attended_val @ W_O = output_projection}
# = x_in + (attention_weights @ V) @ W_O
# = x_in + ((Q @ K.T) @ (x_in @ W_V) @ W_O) {simplifying the attn weights a little here, softmax and being divide by d_k ** 0.5 are omitted}
residual_stream.shape, residual_stream

# which gives us something like:
(torch.Size([3, 8]),
 tensor([[ 12.7616,  -3.4154,   8.3396,  -4.6608,  10.0561,  -3.4056, -14.9948,
           -6.6897],
         [  9.1009,  -4.5961,   3.1327,  -2.1649,   4.2954,  -0.9433, -14.1820,
           -2.3720],
         [  8.9234,  -4.5645,   3.6803,  -2.6616,   4.4057,  -1.8470, -12.3491,
           -2.1242]]))

and that’s the output from the MHA sub-layer of the transformer.

A little more intuition on what this means

For, say, the word “sat” (3rd word in the sequence):

W_sat = residual_stream[2]

This is:

residual["sat"] aka residual[2]
= x_in["sat"] + attended_val["sat"] @ W_O

Let’s assume our attention weight matrix was:

\begin{array}{c|ccc} & \text{The} & \text{cat} & \text{sat} \\[0.5em] \hline \\[0.05em] \text{The} & 1.00 & 0.00 & 0.00 \\[0.5em] \text{cat} & 0.74 & 0.26 & 0.00 \\[0.5em] \text{sat} & 0.20 & 0.50 & 0.30 \end{array}

This means:

attention_weights["sat"] = [0,20, 0.50, 0.30]

So, residual["sat"] is:

=> x_in["sat"] + (attention_weights["sat"] @ V) @ W_O
=> x_in["sat"] + ([0.2, 0.5, 0.3] @ [V_The, V_cat, V_sat]) @ W_O
=> x_in["sat"] + (0.2 * V_The + 0.5 * V_cat + 0.3 * V_sat) @ W_O
=> x_in["sat"] + (0.2 * x_in["the"] + 0.5 * x_in["cat"] + 0.3 * x_in["sat"]) @ W_O

Hence W_sat is:

x_in["sat"] + (0.2 * x_in["the"] + 0.5 * x_in["cat"] + 0.3 * x_in["sat"]) @ W_O

So basically…

We can divide this process into 2 parts:

the QK circuit
the OV circuit

# the QK circuit:
Q @ K.T
=> (x @ W_Q) @ (x @ W_K).T

# the OV circuit:
(attention_weight @ V) @ W_O
=> (attention_weight @ (x @ W_V) @ W_O)
=> (attention_weight @ x) @ (W_V @ W_O)

Why talk about W_OV = W_V @ W_O? => For analysis and interpretability

Instead of thinking “first we transform by W_V, then later by W_O,” we can think: “This attention head applies a single combined transformation W_OV to move information.”

Update on the questions I asked myself earlier

What is the attention weight trying to tell us?
=> I know now how to think about it.
There are batches of tokens that are being processed in parallel to get the output (input + output_proj, i.e., the updated “residual stream”) from the MHA sub-layer of the transformer. How do I exactly visualize this? What happens sequentially? What happens in parallel?
=> Need to get my head around this. I’m sure this is much simpler than I’m making it out to be. Will update this soon.
Why do we need the output projection? What is the intuition behind it?
=> This is basically how much to move the information forward based on the attention weights. Will update more details on this to come.

Still lots of questions to answer. Will update this as I go along.

[1]A Mathematical Framework for Transformer Circuits, Anthropic, 2021. https://transformer-circuits.pub/2021/framework/index.html

Self-Hosting Git with Gitea + Tailscale + Cloudflare Tunnel

Thu, 09 Jul 2026 00:00:00 GMT

draft: true

(wip)

I’ve kinda always wanted to keep my own self-hosted backup of my GitHub, so I finally went ahead and tried it on my mac mini. I used Gitea for the Git server, GitHub pull mirrors for existing repos, Tailscale for private SSH access, Cloudflare Tunnel for the public web UI, and a few daily jobs for backup and cleanup.

I am writing this as I try the setup myself. So this is not meant to be a perfectly detached checklist copied from docs. It is more like me writing down what I did, where I changed my mind, and what I would probably keep if I had to set it up again.

At first, I thought this might be about replacing GitHub. It is not, at least not for me. GitHub is still good (or at least is still the industry standard) at public repo hosting, pull requests, comments, issues, stars, releases, and general discovery. What I wanted was another place where my repos live. A machine I control, reachable privately, visible publicly for read-only browsing, backed up daily, and restorable if something goes wrong.

The version I ended up with looks roughly like this.

GitHub
  source of truth for GitHub-first repos
  pull requests, issues, stars, comments

Gitea on the always-on machine
  local Git server
  native GitHub pull mirrors
  private repos
  public web UI
  daily backups

Tailscale
  private SSH Git access

Cloudflare Tunnel
  public HTTPS access to the Gitea web UI

launchd
  keeps Gitea, cloudflared, backups, metadata refresh, and cleanup running

I am using placeholders throughout the post.

~/gitea-stack for the local stack directory
<github-user> for the GitHub account
<gitea-user> for the Gitea account
<tailscale-ip> for the private Tailscale address
git.example.com for the public hostname

I am intentionally not putting real local usernames, private IPs, tunnel IDs, email addresses, token paths, or secrets in a public post.

draft: true

1. What I Ended Up Building

I ended up thinking about the setup as two kinds of repos.

The first kind is a GitHub-first repo.

laptop -> push to GitHub -> Gitea pulls from GitHub

For repos like this, I let GitHub stay the source of truth. Gitea is a native pull mirror. It keeps a local copy of branches and tags, and it shows the mirror icon in the UI.

The second kind is a Gitea-first repo.

laptop -> push to Gitea
optional later step -> publish to GitHub

For repos like this, I start on the local Git server. They stay local unless I explicitly decide to publish them to GitHub.

This split mattered more than I expected.

When a repo already lives on GitHub and people interact with it there, I prefer to let GitHub stay in charge. Pull requests, comments, issues, and stars are not Git objects. Mirroring commits alone does not move that whole social layer.

When a repo starts locally, Gitea can be the source of truth. I can still publish it to GitHub later if that becomes useful.

I kept this shape because it lets GitHub do what GitHub is good at, while still giving me a local Git home that I can control and back up.

draft: true

2. Why Gitea

I picked Gitea because it is small and boring in a good way.

These were the pieces I needed.

Git over SSH
public and private repos
a web UI
native repository mirrors
SQLite support
a single binary install path
easy file-based backup

This was the mental model I liked.

launchd -> gitea binary -> SQLite + bare Git repos

There is no container layer in my version. There is nothing wrong with Docker, but I chose the direct binary install because it is easy to understand. If something fails, I only have a few places I usually check.

the Gitea process
app.ini
the SQLite database
the bare repositories
the launchd service
the logs

That felt like a manageable list.

draft: true

3. Directory Layout

I kept the stack under one directory.

~/gitea-stack/
  bin/
  custom/conf/
  data/
  repositories/
  log/
  run/
  scripts/
  secrets/
  metadata/
  lists/

Each directory had a clear job for me.

bin/ for the Gitea binary
custom/conf/ for Gitea config
data/ for the SQLite database and Gitea state
repositories/ for bare Git repos
log/ for service and job logs
run/ for lock files
scripts/ for helper commands
secrets/ for tokens and generated secrets
metadata/ for GitHub metadata snapshots
lists/ for generated repo lists

This layout made backup simpler. Most state lives in one directory. Logs and temporary files can be excluded. I add anything outside the stack, such as SSH key material used by the service, to the backup explicitly.

I created the base directories with this command.

mkdir -p ~/gitea-stack/{bin,custom/conf,data,repositories,log,run,scripts,secrets,metadata,lists}
chmod 700 ~/gitea-stack/secrets

draft: true

4. Installing Gitea

I installed Gitea from the official binary.

On Apple Silicon, I used the darwin-arm64 build. On Intel Macs, the equivalent would be darwin-amd64.

cd ~/gitea-stack/bin

curl -L -o gitea "https://dl.gitea.com/gitea/<version>/gitea-<version>-darwin-arm64"
curl -L -o gitea.asc "https://dl.gitea.com/gitea/<version>/gitea-<version>-darwin-arm64.asc"
chmod +x gitea

I verified the signature.

gpg --keyserver hkps://keys.openpgp.org --recv 7C9E68152594688862D62AF62D9AE806EC1592E2
gpg --verify gitea.asc gitea

I also checked that Git existed.

git --version

Gitea shells out to Git, so the machine needs Git installed too.

draft: true

5. First Run And Web Install

I ran Gitea manually first.

GITEA_WORK_DIR=~/gitea-stack \
  ~/gitea-stack/bin/gitea web \
  --config ~/gitea-stack/custom/conf/app.ini \
  --work-path ~/gitea-stack

At this point I opened the installer only on a private address. I did not want the first install screen exposed publicly.

These were the basic settings I used.

database set to SQLite
repository root set to ~/gitea-stack/repositories
app data path set to ~/gitea-stack/data
web port set to 3000
SSH port set to 2222 or another non-default port
registration disabled after the first admin user is created

After the first install, I locked down the config.

chmod 600 ~/gitea-stack/custom/conf/app.ini

Then I restarted Gitea and checked the health endpoint.

curl http://<private-address>:3000/api/healthz

At this stage, I still kept the server private.

draft: true

6. Moving Secrets Out Of app.ini

Gitea creates several secret values during setup. I did not want those values pasted directly into app.ini, so I moved them into files under secrets/ and pointed app.ini to those files.

The config supports file URIs.

SECRET_KEY_URI = file:/path/to/secrets/secret-key
INTERNAL_TOKEN_URI = file:/path/to/secrets/internal-token
JWT_SECRET_URI = file:/path/to/secrets/oauth2-jwt-secret
LFS_JWT_SECRET_URI = file:/path/to/secrets/lfs-jwt-secret

The real paths are absolute local paths on the machine. I am not putting those exact paths in a public post.

I locked the files down.

chmod 700 ~/gitea-stack/secrets
chmod 600 ~/gitea-stack/secrets/*

I wanted app.ini to describe where secrets live, not become the place where every secret value is pasted forever.

draft: true

7. Running Gitea With launchd

On macOS, I used launchd to run Gitea after reboot.

The service runs this command.

~/gitea-stack/bin/gitea web \
  --config ~/gitea-stack/custom/conf/app.ini \
  --work-path ~/gitea-stack

The LaunchDaemon has these settings.

RunAtLoad
KeepAlive
working directory set to ~/gitea-stack
stdout path set to ~/gitea-stack/log/launchd.out.log
stderr path set to ~/gitea-stack/log/launchd.err.log
user set to a normal local user, not root

After installing the plist, I checked it.

sudo launchctl print system/<service-name>

Then I checked Gitea.

curl http://<private-address>:3000/api/healthz

I also rebooted the machine and checked again. A server that does not survive reboot does not feel finished to me.

draft: true

8. Private SSH Access With Tailscale

I used Tailscale before doing anything public.

This was the first networking choice I made because it gave me a private path before anything was on the public internet.

This was the private path.

laptop -> Tailscale -> Mac mini -> Gitea

This let another laptop clone and push without SSH-ing into the Mac mini first.

On the laptop, I created a dedicated SSH key.

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_gitea -C "gitea"

Then I added the public key to Gitea.

cat ~/.ssh/id_ed25519_gitea.pub

My SSH config entry looks like this.

Host gitea
  HostName <tailscale-ip>
  Port 2222
  User git
  IdentityFile ~/.ssh/id_ed25519_gitea
  IdentitiesOnly yes

I tested it with this command.

ssh -T gitea

Then I set a repo remote.

git remote set-url origin gitea:<gitea-user>/<repo>.git

At this point, private Git over SSH worked through Tailscale.

draft: true

9. Choosing The Mirror Direction

There are two possible mirror directions.

Gitea -> GitHub
GitHub -> Gitea

For GitHub-first repos, I chose GitHub ⇒ Gitea.

GitHub is more than a Git remote. It also holds pull requests, comments, issues, stars, releases, and public repo pages. When the repo’s public activity already happens on GitHub, I think GitHub is the better source of truth.

So Gitea pulls from GitHub.

push to GitHub -> Gitea pulls from GitHub

I liked this direction because it gave me the things I cared about.

native mirror icon in Gitea
GitHub PRs staying on GitHub
GitHub comments staying on GitHub
public repo discovery staying on GitHub
no custom push hooks for GitHub-source repos
no write deploy keys for every GitHub-source repo
a local mirror and backup on the Mac mini

For Gitea-first repos, Gitea ⇒ GitHub can still make sense. The important part for me is that publishing to GitHub stays explicit.

I started with the idea of pushing from Gitea to GitHub for everything. I moved away from that because it made GitHub’s pull request and issue layer feel secondary, even though most public repo activity already happens there.

draft: true

10. Creating Native Pull Mirrors

For an existing GitHub repo, I created a native Gitea pull mirror.

GitHub repo -> Gitea migration -> native pull mirror

For public repos, the GitHub URL was enough.

For private repos, Gitea needs credentials that can read the repo. I used a fine-grained GitHub token with read-only contents access. I did not give it write access.

I stored the token in a protected file.

~/gitea-stack/secrets/github-pull-mirror-token

Then I locked it down.

chmod 600 ~/gitea-stack/secrets/github-pull-mirror-token

I did not put this token in .zshrc, dotfiles, shell exports, or public scripts.

For automatic imports, I wanted this behavior.

list GitHub repos
skip forks
preserve public/private visibility
create native pull mirrors for missing repos
use the read-only token for private repos
avoid write deploy keys unless the repo is Gitea-first

I set the mirror interval to something short.

15 minutes

A webhook would make syncing more instant, but it also adds another public integration to secure. A short pull interval felt good enough.

draft: true

11. Handling Gitea-First Repos

For a new repo that starts on Gitea, this is my flow.

git init repo-name
cd repo-name
git remote add origin gitea:<gitea-user>/repo-name.git
git push -u origin main

I set push-created repos to default to private.

When I want the repo to appear on GitHub later, I publish it deliberately.

gitea-push-to-github --private repo-name

gitea-push-to-github --public repo-name

A Gitea-to-GitHub push hook fits this case. The repo started on Gitea, so pushing it to GitHub is an intentional step instead of the default behavior for everything.

draft: true

12. GitHub Metadata

Git stores commits, branches, tags, and files.

It does not store GitHub descriptions, topics, stars, homepage URLs, archived status, or visibility in a way Gitea automatically reuses.

So I kept a metadata refresh step.

GitHub API -> metadata JSON -> selected fields applied to Gitea

These are the fields I cared about.

description
homepage URL
topics
star count
archived state
visibility

I did not fake GitHub stars as Gitea stars. I stored the GitHub star count as metadata instead.

I run this daily, but only write to Gitea when something changed. Otherwise every repo starts looking recently updated even when no code changed.

draft: true

13. Repository Updated Time

Gitea’s repository list has an Updated ... ago value.

For a mirror-heavy setup, this can become noisy. Mirror syncs and metadata refreshes can bump the repository timestamp even if the latest commit did not change.

I wanted the repo list to mean this.

repo list updated time = newest branch commit time

So I added a small normalizer.

every 60 seconds:
  for each repo:
    find newest branch commit time
    set repository updated time to that

I left the native mirror sync timestamp alone. Gitea can still show when the mirror last synced on the repo page.

So there are two separate signals.

repo list for last commit activity
mirror banner for last mirror sync

draft: true

14. Daily Jobs

The daily automation ended up as this schedule.

every 60 seconds:
  normalize repository updated times

03:05:
  discover new non-fork GitHub repos and import missing ones

03:20:
  refresh GitHub metadata

03:45:
  create backup and run maintenance

The discovery job does not sync existing repos. Native mirrors already do that on their own interval.

The discovery job only answers one question.

is there a new non-fork GitHub repo missing from Gitea?

When there is one, the job imports it as a native pull mirror.

I care about this because backups only include what exists in Gitea. If a new GitHub repo never gets imported, the Gitea backup cannot include it.

draft: true

15. Backups

My backup includes these files.

Gitea config
SQLite database
repositories
metadata
secrets
required SSH key material

It excludes these files.

logs
temp files
cache dirs

The backup job does a few things.

stop Gitea briefly
create a .tar.gz
start Gitea again through launchd
run maintenance
keep the newest 30 backups

I also run a restore check. Creating archives is not enough.

The restore check does this.

extract the newest archive into a temp dir
check that config, database, repos, metadata, and secrets exist
run SQLite quick_check
sample bare repos with git fsck
delete the temp extraction

That gave me more confidence than just seeing archive files pile up.

draft: true

16. Maintenance

The large files I found were not scripts.

They were old temp clones, migration bundles, and logs.

My maintenance job does this.

keep the newest 30 backups
remove stale metadata for deleted repos
remove old sync/import/migration temp dirs
remove old restore-check temp dirs
remove old import logs
trim large launchd stdout/stderr logs

It keeps the small server small.

draft: true

17. Public Access With Cloudflare Tunnel

Tailscale handles private access. Cloudflare Tunnel handles public web access.

I added Cloudflare only after the private Tailscale setup worked. That made the steps easier to debug. First I made Git work privately, then I made the web UI public.

I was partly inspired by this blog post, which explains the model nicely.

The high-level path became this.

browser -> Cloudflare -> cloudflared on the Git server -> local Gitea

The Git server does not need an inbound router port. cloudflared opens an outbound connection to Cloudflare, and Cloudflare routes the public hostname through that connection.

This was the basic flow I used.

cloudflared tunnel login
cloudflared tunnel create gitea
cloudflared tunnel route dns gitea git.example.com
cloudflared tunnel run gitea

My config looked like this.

tunnel: <tunnel-uuid>
credentials-file: ~/.cloudflared/<tunnel-uuid>.json

ingress:
  - hostname: git.example.com
    service: http://127.0.0.1:3000
  - service: http_status:404

For my actual setup, I stored the credentials file in a protected secrets directory and ran cloudflared with launchd.

draft: true

18. Public Web, Private SSH

I kept web and SSH separate.

public web
  https://git.example.com
  Cloudflare Tunnel

private Git SSH
  gitea:<gitea-user>/<repo>.git
  Tailscale

The public internet can view public repos.

Git pushes still go through Tailscale.

This leaves the Gitea SSH server private. I do not have a reason to expose it publicly right now.

Gitea still enforces permissions.

registration disabled
private repos hidden from anonymous users
HTTP Git disabled
basic auth disabled
admin 2FA enabled
reverse-proxy auth disabled
webhooks disabled unless needed
config and secret files locked down

Cloudflare gets traffic to the Git server. Gitea decides what each visitor can see.

draft: true

19. Security Checklist

Before I called the setup done, I checked these things.

production mode is on
registration is disabled
push-created repos default to private
public repos are visible to anonymous users
private repos are hidden from anonymous users
HTTP Git is disabled
SSH is not public
browser login works
admin 2FA is enabled
generated secrets live in files
secret files are mode 600
secret directory is mode 700
Gitea config is mode 600
Cloudflare tunnel credentials are mode 600
private GitHub mirror token is read-only
GitHub-source repos have no active push hooks
backups are mode 600
restore check passes
audit passes

Self-hosting does not help much if tokens leak into shell files, public notes, logs, or old backup archives.

draft: true

20. Landing Page And Short Links

Gitea repo paths look like this.

https://git.example.com/<gitea-user>/<repo>

The default path is fine, but I wanted the public root page to feel more like a profile page. Gitea can show a profile README through a .profile repo.

For short repo links, I added a small Cloudflare Worker.

https://git.example.com/repo-name

It redirects to this.

https://git.example.com/<gitea-user>/repo-name

I made sure the Worker does not redirect real Gitea paths.

/user
/api
/assets
/explore
/<gitea-user>/...

draft: true

21. Normal Workflow

For a GitHub-first repo, I push to GitHub.

git push origin main

Gitea pulls the change on the mirror interval.

For a new GitHub repo, I let the import job bring it into Gitea.

gitea-import-new-github-repos --dry-run
gitea-import-new-github-repos

For a Gitea-first repo, I push to Gitea.

git remote add origin gitea:<gitea-user>/repo-name.git
git push -u origin main

To publish a Gitea-first repo to GitHub, I run this command.

gitea-push-to-github --private repo-name

For checks, I use these commands.

gitea-audit
gitea-security-check
gitea-restore-check

For a manual backup, I use this command.

gitea-backup

The setup has landed here for me. GitHub stays for now for public collaboration (@Cursor yall completing your GitHub killer anytime soon?), Gitea for a local mirror and Git home, Tailscale for private SSH, Cloudflare Tunnel for public web, and daily jobs for the boring parts.

Attention-Only vs. All-Layer LoRA: A Worklog

Fri, 15 May 2026 00:00:00 GMT

draft: true

This is the worklog version of the LoRA and Friends experiment.

The final public artifacts are:

model checkpoints: sumitdotml/lora-and-friends
dataset: sumitdotml/lora-and-friends-dataset

1. Before The Run

2026-04-19: the first plan had too many moving parts

I started with too many possible experiments in the air: full fine-tuning, MoE, RL, transfer evaluation, Tinker defaults, LoRA target modules, dataset choice, benchmark choice, and budget. That was not a runnable project but a pile of directions.

I removed most of it. Phase one became:

one model: Qwen3-8B
one task family: math reasoning
one final benchmark anchor: GSM8K
one supervised fine-tuning comparison: attention-only LoRA vs all-layer LoRA

I kept Qwen3-8B as the working model. Qwen3-8B-Base stayed in reserve for a different post-training story, and a smaller future model would have added a model-choice variable that I did not want in the first comparison.

The target-module comparison was the part I wanted to isolate:

condition	intended adapter scope
`attention_only`	attention projections
`all_layer`	attention and MLP projections

That meant everything else had to become boring on purpose: same base model, same data, same rank, same training schedule, same evaluation prompt, same answer extraction, same benchmark, and same checkpoint-selection rule.

Why GSM8K became the benchmark, not the main training set

GSM8K was tempting as a training source because it is clean and familiar. But the train split has only 7,473 rows, and the test split has 1,319 rows. I wanted the final score to be held out and easy to explain, so I kept GSM8K as the benchmark anchor.

The training source moved to nvidia/OpenMathInstruct-2, especially the train_1M split. The schema was simple enough to render into chat format:

{
	"problem": "Solve for $y$:\n\n$$\\frac{y^2 - 3y + 2}{y - 2} = y + 1$$",
	"generated_solution": "... \\[ y = \\boxed{2} \\]",
	"expected_answer": "2",
	"problem_source": "augmented_math"
}

I also looked at unsloth/OpenMathReasoning-mini. It had more character and much longer traces, but the token profile made it harder to fit into the budget. OpenMathInstruct-2 looked less exciting, but it had the right shape for a controlled SFT comparison.

The first sizing pass made the budget concrete. A 500-row tokenizer sample from OpenMathInstruct-2 averaged 456.9 rendered tokens per example, with p90 = 813 and p95 = 963. At Tinker’s then-current Qwen3-8B training price of $0.40 / M tokens, a 30k-example run started to look feasible.

The first target was a mildly balanced 30k dataset:

30k working raw dataset
- 21,000  augmented_math
-  7,000  augmented_gsm8k
-  1,000  math
-  1,000  gsm8k

The first plan did not survive.

The first run sheet was useful even though it was wrong

The first run sheet had a training split of 27,000 rows and a validation split of 3,000 rows. With the early weighted mean estimate, that came out to about 11.70M train tokens per epoch and about $4.68 per epoch.

The pilot shape was also already there:

pilot sweep
- train rows: 5,000
- val rows:     500
- 2 conditions x 3 LR values x 1 seed x 1 epoch

And the main comparison shape was visible:

thesis comparison
- 2 conditions x 3 seeds x 2 epochs
- total train cost: about $56.17

Those numbers were later superseded, but they were still useful. They forced me to think in runs, seeds, rows, tokens, and dollars instead of treating “fine-tune a model” as one vague action.

The benchmark side looked cheap in comparison. One full GSM8K eval was estimated around cents, not dollars of training cost. So from the beginning, training data and training schedule were the real budget levers.

2. Before Training

2026-04-20: the first dataset looked usable until I read it

The initial openmath_30k artifacts matched the recipe exactly: 27,000 train rows and 3,000 validation rows. Structurally it looked clean. Every inspected training row had a final \boxed{...} answer. The first rendered dataset under qwen3_disable_thinking also looked reasonable.

Then the manual audit started doing its job.

One sampled augmented_gsm8k row reasoned itself into clipping a percentage answer to 100 because a fictional budget was too small. A tetrahedron row noticed a fractional tetrahedron count was impossible and still pushed through to a final ratio. A pattern search found disclaimer-like repair language:

cannot spend more than she has
doesn't align with the logical outcome
we need to set the value to 100

At first I treated this as a light cleanup problem. I removed a couple of flagged rows, built filtered artifacts, and added manual review packs. The mismatch count fell, the obvious bad rows got smaller, and the dataset felt closer.

But each pass exposed another class of issue.

The augmented branch kept asking for another pass

The suspicious rows were concentrated in the augmented sources. The failures were not all the same, but they had a family resemblance: impossible integer counts, prompt-generation residue, unsupported case-bashing, and answers that looked right only because the generated reasoning had quietly changed the problem.

The review packs were small enough to inspect but large enough to change my mind. A 53-row suspicious train pack for curated_v2 produced 29 removals. A later 61-row suspicious train pack produced 30 hard removals, all from augmented_math. These were not tiny formatting mismatches but rows with broken reasoning, contaminated prompts, or invalid proofs.

The curation sequence became longer than I wanted:

curated_v2: after the first train and validation suspicious-row removals
curated_v3: after another suspicious train review
curated_v4: after validation cleanup
curated_v5: after backfilling weak augmented_math rows
curated_v6: after removing prompt-generation debris like A new problem: and The new problem is:

One excerpt from the log captures the real issue:

The problem was no longer "find one more regex,"
it was "stop trusting augmented_math as the backbone of the dataset."

At that point I stopped trying to rescue the 30k augmented-heavy recipe. If the dataset takes repeated repair passes and still produces obvious bad rows in random review, then it stops being a clean experimental input and becomes the experiment itself.

I did not want the final write-up to be “attention-only LoRA versus all-layer LoRA, but also maybe the augmented data had hidden garbage.” That would have made every downstream result harder to trust.

2026-04-20 to 2026-04-21: I rebuilt around original rows only

OpenMathInstruct-2 train_1M had 29,468 original-source rows across gsm8k and math. After the strict gate, 28,166 survived cleanly enough to build a new candidate.

The accepted source counts were:

{
	"gsm8k": 14618,
	"math": 13548
}

The original-only dataset was smaller than the 30k target, but the quality profile was much better. It cleared the automatic checks that the augmented branch kept failing:

boxed-match rate, train: 1.0
boxed-match rate, validation: 1.0
suspicious rows, train: 0
suspicious rows, validation: 0
prompt-wrapper contamination hits, train: 0
prompt-wrapper contamination hits, validation: 0

I froze this path as openmath_original_clean.

After a later split repair, the final retained manifest became:

split	`gsm8k` source rows	`math` source rows	total
train	13,145	12,203	25,348
validation	1,473	1,345	2,818

The old openmath_30k* lineage was retired after that. I wanted one clear dataset path in the repo, not a dozen stale artifact branches that future-me or another agent might accidentally treat as active.

2026-04-21: I kept the system prompt

The render sanity check compared the frozen dataset with and without the system prompt under the actual Qwen/Qwen3-8B chat template. The prompt added a constant 27 tokens per example.

{
	"full_mean_with_system": 340.25,
	"full_mean_without_system": 313.25,
	"full_mean_delta": 27.0
}

That overhead was small enough to keep. More importantly, the prompt made the output contract explicit:

You are a careful math solver. Solve the problem step by step. Put the final answer in \boxed{}.

I kept it because the raw problems themselves do not always carry the boxed-answer instruction. I also wanted training and evaluation to mirror each other as closely as possible.

3. Contracts And Sanity Checks

2026-04-22: I stopped relying on memory

Once the dataset path was stable, the next risk was format drift. It is easy to train with one prompt, evaluate with another, parse answers a third way, and then accidentally report a prompt experiment as a model experiment.

So I wrote the contracts down before the final result existed.

I also had to clean up the planning docs. Some TODO labels were too compressed. A label like “contamination check” made sense to me while I was in the middle of the work, but it hid the actual operation: compare training-side gsm8k problem text against held-out GSM8K test questions after canonical normalization, then block training if overlap is greater than zero.

So I added an execution-clarity rule for the project: open tasks needed to say what the action was, why it mattered, what counted as done, and what happened if it failed. It sounds procedural, but it kept the later runbook from turning into shorthand only I could decode.

The evaluation contract froze:

the same system prompt as training
greedy decoding with temperature = 0
enable_thinking=False
max_new_tokens = 512
no custom stop tokens
boxed-answer extraction only
exact match after normalization
the contamination report path

The contamination check was a gate, not a note. If training-side gsm8k rows overlapped the GSM8K test questions, the benchmark had to change or the dataset had to be rebuilt.

The result schema was frozen around retained JSONL output under artifacts/results/<run_id>/. It also kept token_count and cost in the canonical schema even when those values were null, because I did not want the schema to change later just because telemetry became available.

I also standardized on the word condition for the comparison label, which sounds minor but made later artifacts easier to read:

{
	"condition": "attention_only",
	"seed": 0,
	"step": 3169
}

2026-05-03: the dataset split still had a hidden problem

When I ran the dataset-integrity gate, the benchmark side passed. The training-side gsm8k rows had 0 overlap with the openai/gsm8k test split.

But the local train/validation split failed a different check. OpenMath can include multiple accepted solutions for the same underlying problem. The original builder had split rows independently, so variants of the same problem could land on both sides.

That would make validation loss too friendly. It would also make checkpoint or LR decisions look cleaner than they were.

The fix was to group by canonical problem text before splitting. The retained report after the rebuild says:

{
	"status": "pass",
	"overlap_count": 0,
	"train_val_overlap": {
		"row_id_overlap_count": 0,
		"problem_text_overlap_count": 0,
		"overlapped_val_row_count": 0
	}
}

The final contamination report compared 13,145 training rows with source == "gsm8k" against 1,319 GSM8K test questions. It found overlap_count = 0.

The raw files were not enough on their own, and the split rule mattered.

2026-05-04: the dataset went to Hugging Face

After the repaired split, the frozen dataset payloads were published to sumitdotml/lora-and-friends-dataset.

The local repo kept scripts, manifests, checksums, and audit evidence. The Hub repo carried the payloads:

payload	rows	bytes
raw train	25,348	25,528,287
raw validation	2,818	2,787,397
rendered train	25,348	29,989,535
rendered validation	2,818	3,283,365

At this point I had a real dataset, a render path, an evaluation contract, and a contamination gate. I still did not know whether the training path would work.

4. First Tinker Check

2026-05-04: the first backend check caught a renderer trap

The first Tinker smoke pass was supposed to be boring. It was one of the smallest checks in the whole project: can Tinker accept Qwen/Qwen3-8B, rank 8, and the two LoRA condition switches?

The Tinker cookbook qwen3_disable_thinking renderer was correct for generation prompts, but its supervised-training path did not match the frozen SFT render contract. The project contract was the Hugging Face chat template with enable_thinking=False, which renders an empty thinking block before the assistant answer:

<think>

</think>

I changed the smoke runner so Tinker datums came from AutoTokenizer.apply_chat_template(..., enable_thinking=False), then masked loss only after the rendered prompt prefix.

The retained smoke pass then gave me enough to lock the adapter defaults:

condition	train rows in smoke	optimizer steps	validation NLL
`attention_only`	1	1	1.5048651695251465
`all_layer`	8	1	1.4733978509902954

Tinker accepted:

{
	"attention_only": {
		"train_attn": true,
		"train_mlp": false,
		"train_unembed": false
	},
	"all_layer": {
		"train_attn": true,
		"train_mlp": true,
		"train_unembed": false
	}
}

The public Tinker API did not expose local lora_alpha or lora_dropout fields through this training path. I recorded those as backend-owned instead of pretending the runner controlled them.

There was one more small backend detail in the log: the first smoke attempt printed that the Tinker SDK version was outdated. I upgraded to tinker==0.18.2, reran the smoke pass, and the warning did not reappear. Backend version drift becomes hard to reconstruct later, so I kept the note.

The smoke pass also created checkpoint paths with a seven-day TTL. That did not matter for the final result yet, but it foreshadowed the export urgency after the main sweep.

2026-05-05: I ran the untouched baseline before touching LoRA

The baseline was Qwen/Qwen3-8B on openai/gsm8k, config main, split test, with enable_thinking=false, temperature=0, and max_new_tokens=512.

The retained baseline run was:

artifacts/results/baseline-qwen3-8b-gsm8k-001/

It scored:

examples: 1,319
correct: 1,115
accuracy: 0.8453373768006065
extraction failures: 31
total eval tokens: 505,694

I wanted this number before LR selection and before the main comparison. Otherwise, the final LoRA scores would float without a reference.

5. Learning Rate And Batch Shape

2026-05-06: the LR-selection plan was too slow

The first LR-selection protocol used 5,000 train rows and 500 validation rows. On paper it was reasonable. In practice it projected to roughly 20 hours for the six-run sweep on Tinker.

I rescaled it to a deterministic slice:

first 512 train rows
first 128 validation rows
seed 7
LR grid: 1e-4, 3e-4, 1e-3
two conditions
one epoch
effective batch size 8

The narrow purpose was to choose a peak LR per condition, not to run a final benchmark or a condition comparison.

The selected LR was 3e-4 for both:

condition	LR	best validation NLL
`attention_only`	`1e-4`	0.3786645046540731
`attention_only`	`3e-4`	0.3632619345728878
`attention_only`	`1e-3`	0.3644437038722405
`all_layer`	`1e-4`	0.3648794147648033
`all_layer`	`3e-4`	0.3559855057286731
`all_layer`	`1e-3`	0.37397296784836664

I kept seed 7 reserved for selection work so the main comparison seeds could be 0, 1, and 2.

The LR-selection runner also caught a plain implementation bug before the expensive runs. I had assumed Tinker cookbook datum weights were raw torch tensors. They were TensorData objects. The live one-step probe failed, the runner got fixed, and that mistake went into AGENT_MISTAKES.md. The final result depends on the boring part where a runner can survive a one-step live probe before it is trusted for a sweep.

I also split the training helper code into smaller modules:

training/common.py
training/sft.py
training/lora.py

The runnable scripts stayed separate: one for the smoke pass, one for LR selection, and later one for the main training sweep. This kept the orchestration scripts readable enough that I could still inspect what was being frozen.

2026-05-07: speed was tempting, but batch size was not free

The initial runner shape used eight one-datum train requests before each optimizer step. A throughput probe showed that request shape mattered a lot at the same effective batch size:

request shape	seconds per optimizer step
`single_datum_calls`	20.22187466151081
`batched_datums`	5.201307859155349
`batched_datums_pipelined`	2.4104866901249693

The fast version, batched_datums_pipelined, submits one batched forward_backward_async(...) request and queues the optimizer request before waiting. That kept the nominal effective batch size at 8; it only changed how the same batch was sent to Tinker.

Then larger batches looked tempting. Batch 512 and 1024 were much faster in throughput probes. But batch size is a real hyperparameter for LoRA, so I made the larger-batch path earn its way in by validation NLL, and it did not.

condition	selected batch/LR	selected NLL	best larger-batch candidate	candidate NLL
`attention_only`	batch 8, `3e-4`	0.3632619345728878	batch 512, `1e-3`	0.37616809419132946
`all_layer`	batch 8, `3e-4`	0.3559855057286731	batch 512, `1e-3`	0.3569802998485914

So the main run kept effective batch size 8, used the pipelined request shape, and kept the final partial batch of 4 rows rather than wrapping around and duplicating examples. The fast path was real, but it was only allowed to change operations, not silently change the scientific comparison.

6. The Six Runs

2026-05-08: `main-001` started

The main run expanded to six sequential condition/seed runs:

main-001-attention_only-seed-0
main-001-attention_only-seed-1
main-001-attention_only-seed-2
main-001-all_layer-seed-0
main-001-all_layer-seed-1
main-001-all_layer-seed-2

The launch command was:

uv run training/run_main_training.py --run-prefix main-001

Each run used:

train rows: 25,348
validation rows: 2,818
epochs: 2
effective batch size: 8
optimizer steps per epoch: 3,169
total optimizer steps: 6,338
warmup steps: 190
peak LR: 3e-4
min LR: 3e-5

The validation/checkpoint cadence was 1000, 2000, 3169, 4000, 5000, 6000, and 6338.

I monitored this like a long-running job, because it was one. The raw log has many repeated entries that look boring at first glance:

validation step=1000 nll=...
validation step=2000 nll=...
validation step=3169 nll=...
...

Those entries showed that the run was still alive, that validation/checkpoint blocks resumed correctly, and that every retained checkpoint path was read from the active run rather than guessed from a sibling run.

I had already made that mistake once while logging a checkpoint URL by analogy from a sibling run. The fix was simple: read the exact checkpoint string from the active run’s metrics.jsonl or summary before writing it down. After that, I treated checkpoint paths as evidence, not as strings I could reconstruct from memory.

Step 3169 kept showing up

The first attention-only seed hit:

main-001-attention_only-seed-0: validation step=3169 nll=0.336164

Its final summary selected step 3169, not the final step 6338:

selected best validation step: 3169
selected validation_mean_nll: 0.33616363178874076

Then seed 1 selected step 3169, then seed 2, then all-layer seed 0, then all-layer seed 1, then all-layer seed 2.

At the end, every run selected the one-epoch boundary:

run	selected step	validation NLL
`main-001-attention_only-seed-0`	3169	0.33616363178874076
`main-001-attention_only-seed-1`	3169	0.33621202263413363
`main-001-attention_only-seed-2`	3169	0.33651007850933906
`main-001-all_layer-seed-0`	3169	0.33611938013674597
`main-001-all_layer-seed-1`	3169	0.33655583715915666
`main-001-all_layer-seed-2`	3169	0.33634131648081267

The selected NLL means were close:

attention_only: 0.3362952443107378
all_layer: 0.3363388445922384

But the post-3169 behavior was different. Attention-only stayed near the selected NLL. All-layer jumped more sharply after the one-epoch boundary; at step 4000, all-layer mean NLL was 0.3457733619534809.

I did not treat that as the final result though; it was a training diagnostic, and the actual comparison still had to come from GSM8K.

For all-layer seed 1, the NLL moved from 0.33655583715915666 at step 3169 to 0.34549824556309505 at step 4000. For all-layer seed 2, it moved from 0.33634131648081267 to 0.3461580484910246.

The lowest-NLL checkpoint rule prevented later checkpoints from becoming attractive simply because they were later or because the run had continued longer.

The sweep finished with six pass summaries

The final all-layer seed completed on 2026-05-09. The log entry was satisfying because the sweep had finally become a finite object:

completion artifacts now exist in artifacts/results/main-001-all_layer-seed-2/:
manifest.json, metrics.jsonl, sample_render.txt, summary.json

Each of the six selected checkpoints had a seven-day Tinker TTL, so the next step was urgent enough: export the selected checkpoint states into something that could be evaluated and published.

7. Checkpoint Export

2026-05-09: training weights were not sampler weights

The selected training checkpoints used weights/... paths. I initially wanted to use those directly for publication/evaluation, but the sampling client rejected that shape.

The actual server response was:

tinker.BadRequestError: Error code: 400 - {'detail': 'model_path must point to a sampler_weights checkpoint, got weights'}

So the selected training state checkpoints had to be converted to sampler-format weights before any evaluation could run.

The working export path was:

load_state_async(training_checkpoint_path)
save_weights_for_sampler_async(export_name)

The six sampler checkpoint URIs then looked like:

tinker://.../sampler_weights/export-main-001-attention_only-seed-0-step-3169
tinker://.../sampler_weights/export-main-001-all_layer-seed-2-step-3169

After that, the converted adapters were uploaded to the Hugging Face model repo. The published layout is:

checkpoints/best-checkpoints/attention_only/seed-{0,1,2}/step-3169/
checkpoints/best-checkpoints/all_layer/seed-{0,1,2}/step-3169/

The sampler weights also made the adapter-size difference concrete. The attention-only sampler exports were about 30.8M bytes each in the Tinker checkpoint listing, while the all-layer sampler exports were about 87.5M bytes each. That was expected from the larger adapter scope, but it was useful to see it in the retained checkpoint metadata rather than only as intuition.

8. GSM8K Evaluation

2026-05-10: the evaluation runner needed checkpoint awareness

The baseline path already existed, but checkpoint evaluation needed more safeguards. I added support for:

--checkpoint-path
explicit --seed
explicit --condition
a run-id slug that would not collide when checkpoint paths ended similarly
step parsing only from literal step-<digits> tokens

I also made the argument validation fail fast if a checkpoint path was passed without condition or seed. I did not want a LoRA checkpoint quietly defaulting to base.

Before launching the full sweep, I ran a one-example probe with attention_only seed 0. It returned 1/1 correct and confirmed the full path:

sampler URI -> tokenizer -> Tinker sample -> boxed-answer extraction -> scoring -> retained artifacts

The six eval runs

The full sweep ran at --concurrency 16, no limit, against the six selected sampler checkpoints.

The baseline stayed:

accuracy: 0.8453373768006065
correct: 1115/1319
extraction failures: 31

The six LoRA eval runs were:

condition	seed	accuracy	correct / total	extraction failures
`attention_only`	0	0.9044730856709629	1193 / 1319	6
`attention_only`	1	0.9067475360121304	1196 / 1319	2
`attention_only`	2	0.9052312357846853	1194 / 1319	5
`all_layer`	0	0.8991660348749052	1186 / 1319	5
`all_layer`	1	0.9021986353297953	1190 / 1319	4
`all_layer`	2	0.9014404852160728	1189 / 1319	8

Aggregated by condition:

condition	mean accuracy	min	max	range
`attention_only`	0.9054839524892596	0.9044730856709629	0.9067475360121304	0.0022744503411676
`all_layer`	0.9009350518069245	0.8991660348749052	0.9021986353297953	0.0030326004548901

The mean gap was 0.0045489006823352, or about 0.455 percentage points, in favor of attention-only. Under the frozen rule, that is below the 0.01 threshold for a winner claim. So the honest reading is local: attention-only ended higher in this run set and used a smaller adapter, but the result stays inside the pre-declared inconclusive band.

The disagreement view made the result easier to read

The net accuracy chart showed the final result, but the disagreement chart explained its shape better.

For paired seeds:

seed	attention-only only	all-layer only	both correct	both wrong	delta
0	54	47	1139	79	+7
1	49	43	1147	80	+6
2	46	41	1148	84	+5

The two conditions mostly agree, with the difference coming from a narrow band of examples where attention-only is correct slightly more often than the reverse.

9. Figures And Cleanup

2026-05-10: the figure pass changed how I told the result

The figure pipeline produced eight retained artifacts. This worklog uses four of them:

fig_03: GSM8K accuracy by condition
fig_04: per-seed prediction disagreement
fig_05: validation NLL by training step
fig_08: throughput by request shape

The figure pass forced me to separate claims that were getting blurred together:

validation NLL explains checkpoint selection
GSM8K accuracy explains benchmark performance
prediction disagreement explains the narrow accuracy gap
throughput explains an operational runner choice

The fig_04 design took the most iteration. The original idea was a slope/delta view showing net correct-count deltas of +7, +6, and +5. It kept reading like a progression over time or a tiny effect exaggerated by the axis.

The raw log records several rejected forms: vertical slope, leader labels, corner text boxes, horizontal dumbbell, vertical paired columns, and delta-only bars. The recurring problem was that the visual form was louder than the claim. A 0.4 to 0.5 percentage-point gap should not be drawn like a dramatic swing.

The final chart became per-seed disagreement small multiples. That matched the actual question better: where do the two conditions disagree?

The log records the final contingency numbers before implementation:

seed 0: attention_only_only=54, all_layer_only=47, both_correct=1139, both_wrong=79
seed 1: attention_only_only=49, all_layer_only=43, both_correct=1147, both_wrong=80
seed 2: attention_only_only=46, all_layer_only=41, both_correct=1148, both_wrong=84

I kept the filename as fig_04_paired_seed_slope so existing artifact paths did not move, but the chart itself became the disagreement view.

The cleanup rule

The result can say:

both LoRA conditions improved over the untouched baseline on GSM8K
attention-only had the higher three-seed mean in this retained run set
attention-only had a small paired-seed disagreement edge
the gap is below the frozen 0.01 threshold for a winner claim

The result cannot say:

attention-only LoRA is generally better
all-layer LoRA is bad
the same result should hold across other ranks, models, datasets, or benchmarks
validation NLL explains the benchmark gap as a mechanism

10. What This Leaves Me With

The final comparison was the easy sentence at the end of a lot of setup.

I started too broad and had to cut the project down to one comparison.
I thought the first dataset recipe was close, then the audits kept proving otherwise.
I abandoned the augmented-heavy branch because it made the dataset itself the unstable variable.
I kept the system prompt because the output contract needed to be explicit.
I froze the eval contract before the result existed.
I had to repair train/validation leakage by grouping canonical problem text.
The smoke pass caught a render mismatch before paid runs.
LR selection picked 3e-4 for both conditions.
Throughput work mattered, but only after batch size stayed controlled.
All six main runs selected step 3169.
Training checkpoints had to be converted to sampler weights before evaluation.
The disagreement chart was more honest than only talking about a mean gap.

The result is small, but the path to making it trustworthy was not.

If I were doing the next version, I would keep the same discipline: narrow the question first, freeze contracts before seeing benchmark numbers, treat dataset audits as part of the experiment, and make every convenience change prove that it does not change the comparison. The follow-up question I would carry forward is when a narrower adapter is enough, and when broader adaptation is worth the extra size.

herambve

Chasing Copy Heads in a Residual Streams Avenue

1. Definitions

2. Background of all this

3. Importance of copy heads across different layers

4. Critical questions

Definitions

Copy heads

Refresher: How did we reach this rabbit hole?

Engineering Systems from First Principles

Building from the Ground Up

Inside a Transformer: A Worklog

On residual stream as a communication channel

Inside the attention sub-layer: the mechanics

Trying to gain some deeper intuition

A little more intuition on what this means

So basically…

Update on the questions I asked myself earlier

Self-Hosting Git with Gitea + Tailscale + Cloudflare Tunnel

1. What I Ended Up Building

2. Why Gitea

3. Directory Layout

4. Installing Gitea

5. First Run And Web Install

6. Moving Secrets Out Of app.ini

7. Running Gitea With launchd

8. Private SSH Access With Tailscale

9. Choosing The Mirror Direction

10. Creating Native Pull Mirrors

11. Handling Gitea-First Repos

12. GitHub Metadata

13. Repository Updated Time

14. Daily Jobs

15. Backups

16. Maintenance

17. Public Access With Cloudflare Tunnel

18. Public Web, Private SSH

19. Security Checklist

20. Landing Page And Short Links

21. Normal Workflow

Attention-Only vs. All-Layer LoRA: A Worklog

1. Before The Run

2026-04-19: the first plan had too many moving parts

Why GSM8K became the benchmark, not the main training set

The first run sheet was useful even though it was wrong

2. Before Training

2026-04-20: the first dataset looked usable until I read it

The augmented branch kept asking for another pass

2026-04-20 to 2026-04-21: I rebuilt around original rows only

2026-04-21: I kept the system prompt

3. Contracts And Sanity Checks

2026-04-22: I stopped relying on memory

2026-05-03: the dataset split still had a hidden problem

2026-05-04: the dataset went to Hugging Face

4. First Tinker Check

2026-05-04: the first backend check caught a renderer trap

2026-05-05: I ran the untouched baseline before touching LoRA

5. Learning Rate And Batch Shape

2026-05-06: the LR-selection plan was too slow

2026-05-07: speed was tempting, but batch size was not free

6. The Six Runs

2026-05-08: main-001 started

Step 3169 kept showing up

The sweep finished with six pass summaries

7. Checkpoint Export

2026-05-09: training weights were not sampler weights

8. GSM8K Evaluation

2026-05-10: the evaluation runner needed checkpoint awareness

The six eval runs

The disagreement view made the result easier to read

9. Figures And Cleanup

2026-05-10: the figure pass changed how I told the result

The cleanup rule

10. What This Leaves Me With

2026-05-08: `main-001` started