log 2026 · 09 · 28 约 6,600 字~6,700 words 阅读 26 分钟30 min read

复现 grokking 论文踩的坑 Reproducing a grokking paper, and the pits I fell into

没有 weight decay 也 grok 了；然后把每个轴敲回坏值，看它死不死。 It grokked without weight decay. Then we knocked each axis back to a bad value, one at a time, to see what dies.

华罗庚讲读书，说要先把书读厚，再把书读薄。读厚是把每一句都拆开、每个例子都补上，读薄是最后只剩一条线。我第一次听见 grokking 这个词的时候想到的就是这句话——网络先把训练集一题一题背下来（读厚），背完了再在某一刻突然把它压成一条规则（读薄）。中间那段什么都没发生的空白，华罗庚没讲，论文也讲不清楚。

Hua Luogeng, the mathematician, said that to read a book you first read it thick, then read it thin. Reading it thick means taking every sentence apart and filling in every example; reading it thin means that in the end only one thread is left. The first time I heard the word grokking, that's what I thought of — the network first memorizes the training set one problem at a time (reading thick), and then, at some moment, suddenly compresses it into a single rule (reading thin). The blank stretch in between where nothing seems to happen — Hua Luogeng didn't talk about it, and the papers can't really explain it either.

这篇讲的是我花了三个月研究"网络什么时候是真懂、什么时候只是死记"，然后发现这段时间里最生动的样本是我自己：实验一批批跑，结论一条条写进账本，过两个星期回头看，虽然实验数据没有memorize，但grok了一些对神经网络行为的直觉。

This post is about the three months I spent studying "when does a network actually understand, and when is it just memorizing", only to find that the most vivid specimen of the whole period was me: experiments ran batch after batch, conclusions went into the ledger one by one, and looking back two weeks later, I hadn't memorized the experimental data, but I had grokked some intuitions about how neural networks behave.

所以这篇的主线不是发现，是复现前人实验和验证自己一些小猜想过程。

So the through-line of this post isn't a discovery. It's the process of reproducing other people's experiments and checking a few small hunches of my own.

这篇里的实验是我和 AI 搭档 Lux 一起做的：我定方向、做判断，他写代码、跑 run。下文的"我们"指我们俩。

The experiments in this post were done together with my AI partner, Lux: I set the direction and made the calls; he wrote the code and ran the runs. Every "we" below means the two of us.

§ 1背景：grokking 和 Prieto 那篇background: grokking and the Prieto paper

grokking 是延迟泛化。train acc 早早到 100%，test acc 长期贴着随机线，然后在远远超过 overfit 的某一步突然跳上去。Power 2022 最早把这件事命名出来。

Grokking is delayed generalization. Train acc hits 100% early, test acc hugs the chance line for a long time, and then, at some step far past the point of overfitting, it suddenly jumps up. Power 2022 was the first to give this a name.

这篇里所有实验的任务都是模算术：给定一个素数 p，把 (a, b) → (a ∘ b) mod p 这张乘法表随机切出一部分当训练集，剩下的当测试集。下文出现的 p113 / frac0.4 就是"模 113、训练集占整张表的 40%"，p59 / frac0.3 同理。这种任务的好处是干净——规律确实存在、确实可以被完整学到，所以"没泛化"只能是学习器的方式不够好。

Every experiment in this post uses modular arithmetic as the task: pick a prime p, randomly cut out part of the operation table (a, b) → (a ∘ b) mod p as the training set, and use the rest as the test set. The p113 / frac0.4 you'll see below means "mod 113, training set is 40% of the whole table"; p59 / frac0.3 works the same way. The nice thing about this kind of task is that it's clean — the rule really exists and really can be learned in full, so "didn't generalize" can only mean the learner's way of learning wasn't good enough.

过去流行的解释是 weight decay：wd 给参数范数施加"效率压力"，记忆电路和泛化电路同时存在的时候，泛化电路用更小的范数达到同样的 logit，于是在长期竞争里胜出。Varma 2023 的 circuit efficiency 是这条线最完整的版本。

The popular explanation used to be weight decay: wd puts "efficiency pressure" on the parameter norm; when a memorizing circuit and a generalizing circuit coexist, the generalizing one reaches the same logits with a smaller norm, so it wins the long-run competition. Varma 2023's circuit efficiency is the most complete version of this line.

Prieto 2025 挑的就是这条共识。他们的主张是：没有 wd 时 grokking 难以发生，真正的障碍是浮点数值崩溃。因果链两段——overfit 之后梯度退化成 NLM（Naïve Loss Minimization，等比放大权重就能降 loss，但不改变任何预测），NLM 把 logit 推到浮点精度的极限，触发 Softmax Collapse，正确类的梯度被吸收误差压成精确的 0，训练在规则电路成型之前就死机了。他们的解药是 StableMax，把 softmax 的饱和门槛从 logit gap≈16 推到 ≈8×10⁶；以及 ⊥Grad，把梯度里"只放大 logits"的平行分量剔掉，只留下正交分量去学规律。

Prieto 2025 goes straight at this consensus. Their claim: without wd, grokking has a hard time happening, and the real obstacle is floating-point numerical collapse. The causal chain has two links — after overfitting, the gradient degenerates into NLM (Naïve Loss Minimization: you can lower the loss just by scaling the weights up proportionally, without changing a single prediction); NLM pushes the logits to the limit of floating-point precision and triggers Softmax Collapse, where the gradient for the correct class gets crushed by absorption error to exactly 0, and training dies before the rule circuit has formed. Their fixes are StableMax, which pushes softmax's saturation threshold from a logit gap of ≈16 out to ≈8×10⁶; and ⊥Grad, which strips out the component of the gradient parallel to "just scale up the logits" and keeps only the orthogonal component to learn the rule.

我要复现它的原因很简单：

My reason for reproducing it is simple:

如果规律真的在数据里，那么一个足够耐心的学习器应该能把它捞出来，跟有没有人在旁边施加范数压力没关系。wd 可以是加速器，不该是许可证。Prieto 说的正是这件事，所以我想亲手看一眼。

If the rule is really in the data, a patient enough learner should be able to fish it out, whether or not someone is standing next to it applying norm pressure. wd can be an accelerator; it shouldn't be a license. That's exactly what Prieto is saying, so I wanted to see it with my own eyes.

StableMax 管住了 softmax 的饱和，可前面还有两道更早的闸口：fp32 的精度约 1.2e-7，Adam 分母的默认 eps 是 1e-8。比这更细的信号会被吃掉，后面几节都会撞上它们。

StableMax keeps softmax from saturating, but two gates come earlier: fp32's precision is about 1.2e-7, and the default eps in Adam's denominator is 1e-8. Anything finer than that gets swallowed, and several later sections run into them.

fig 1 · standard grok step (image not loaded)

§ 2复现：没有 weight decay，六个 seed 全 grok 了reproduction: no weight decay, all six seeds grokked

我们跑了两组配置：

We ran two configurations:

两组独立配置、两种架构、六个 seed，全部在没有 weight decay 的情况下 grok 了。而且 test acc 不是慢慢爬上去的，是那种典型的台阶：先在低位横着走几万步，然后在一万步左右的时间里冲到 0.99。

Two independent configurations, two architectures, six seeds — every one of them grokked without weight decay. And test acc didn't creep up slowly; it was the classic staircase: first it moves sideways at a low level for tens of thousands of steps, then it charges up to 0.99 in roughly ten thousand steps.

节 1 那句题眼是规律的传递按理说没有 wd 也能走。现在它有了两组独立的证据——一组 transformer、一组 MLP，架构、步数、优化器超参都不一样，唯一共同点是 wd=0 并且数值是按论文调的。所以 wd 不是 grokking 的许可证。它是加速器，而且是个非常强的加速器（强到什么程度，节 4b 有数）。

The key line in §1 is that getting the rule across should, in principle, work without wd. Now that line has two independent pieces of evidence — one transformer, one MLP, with architecture, step count and optimizer hyperparameters all different; the only things they share are wd=0 and numerics tuned the way the paper says. So wd is not grokking's license. It's an accelerator, and a very strong one (how strong — §4b has numbers).

复现之外，有三个细节值得盯着：

Beyond the reproduction itself, three details are worth keeping an eye on:

grok 时间的方差很大。 同一个配置三个 seed，54k / 80k / 116k，最快和最慢差 2 倍。对照后面节 4a 那个有 wd 的 lr sweep——lr 1e-2 的三个 seed 全部落在 1–2k。没有 wd 的 grok 像在等一个随机事件：StableMax 让网络活着，但没有一股稳定的力推它去泛化，靠优化器自己晃过去，所以几点晃到看运气。这个方差不是统计噪声那种可以取平均抹掉的东西，它直接决定了"跑到几步才能判不 grok"：说一个 run 不 grok 之前，得先写清楚看到了第几步。

The variance in grok time is large. Same config, three seeds: 54k / 80k / 116k — the fastest and slowest differ by 2×. Compare that with the lr sweep with wd in §4a, where all three seeds at lr 1e-2 land in 1–2k. Grokking without wd is like waiting for a random event: StableMax keeps the network alive, but there's no steady force pushing it toward generalization; it's left to the optimizer to wobble its way over, so when it gets there is down to luck. This variance isn't the statistical-noise kind you can average away; it directly decides "how many steps you have to run before you can call it no-grok": before you say a run doesn't grok, you have to write down how many steps you actually watched.

解更脏。 嵌入的 Fourier top-5 占比约 0.5，有 wd 的 grok 解是约 0.7。没有 wd 也能长出规则电路，但旁边留着一堆没清干净的记忆残渣。wd 的第二个作用是清场——除了防饱和之外，它还在 grok 之后继续把没用的部分削掉。

The solution is dirtier. The Fourier top-5 share of the embedding is about 0.5, versus about 0.7 for a grokked solution with wd. The rule circuit can grow without wd, but there's a pile of uncleared memorization residue left lying next to it. wd's second job is cleanup — besides preventing saturation, it keeps trimming away the useless parts after grokking.

范数一路爆炸照样 grok。 MLP 严格复现那组，参数范数从 22 涨到 44000，grok 了。按节 1 那套"效率压力"的解释，范数涨上天应该意味着解越来越不经济、越来越靠记忆，可它偏偏在这中间泛化了。这条伏笔留到节 5 收。

The norm blows up all the way, and it groks anyway. In the strict MLP reproduction, the parameter norm went from 22 to 44000, and it grokked. By the "efficiency pressure" account from §1, a norm shooting through the roof should mean the solution is getting less and less economical and leaning more and more on memorization — yet that's exactly where it chose to generalize. I'll pay this one off in §5.

fig 2 · seed variance, no wd (image not loaded)

fig 3 · norm runaway, still groks (image not loaded)

这一节没复现的What this section didn't reproduce

• 论文最强的那条线是 ⊥Grad + StableMax 合用（又快又紧凑），我们只单用了 StableMax，合用没跑。
• 论文 Fig4 用的是 SGD，我们用的是 Adam / AdamW(λ=0)。Adam 的归一化让 NLM 跑得更猛，范数的绝对数字不可以直接和论文对。

• The paper's strongest line is ⊥Grad + StableMax together (fast and compact); we only used StableMax on its own and never ran the combination.
• The paper's Fig4 uses SGD; we used Adam / AdamW(λ=0). Adam's normalization makes NLM run harder, so our absolute norm numbers can't be compared directly with the paper's.

§ 3消融：从能 grok 的配置出发，每次敲回一个轴knockouts: start from a config that groks, knock back one axis at a time

节 2 那两组配置能 grok，是因为好几个轴都按论文调过了：任务、步数、eps、精度、架构。但"都调过了才活"不等于"缺一个都不行"，也许其中一两个轴就是全部。要说缺一不可，得把每个轴单独敲回坏值看它死不死。

The two configs in §2 grok because several axes were tuned per the paper: task, steps, eps, precision, architecture. But "it only lives when all of them are tuned" isn't the same as "take away any one and it dies"; maybe one or two of those axes are the whole story. To claim every one is necessary, you have to knock each axis back to its bad value on its own and see whether it dies.

基线取节 2 那份 MLP 严格复现的配置（one-hot MLP [200,200] / lr 0.01 / β₂ 0.999 / eps 1e-25 / fp64 / p113 frac0.4），拉到 300k 步，六个 arm 各把一个轴敲回坏值，3 seed，18 个 run。18/18 完成、0 failure、下载的结果文件与服务器副本逐个 SHA-256 核对 36/36 一致。

The baseline is the strict MLP reproduction config from §2 (one-hot MLP [200,200] / lr 0.01 / β₂ 0.999 / eps 1e-25 / fp64 / p113 frac0.4), stretched to 300k steps; six arms each knock one axis back to a bad value, 3 seeds, 18 runs. 18/18 finished, 0 failures, and the downloaded result files were checked one by one against the server copies by SHA-256: 36/36 match.

为什么 eps 和精度能决定生死，节 1 已经讲完了：StableMax 的全部意义是把饱和门槛推远，让梯度别被吸收误差压成精确的 0。但 fp32 的 ε≈1.2e-7 和 Adam 分母的 eps=1e-8 这两道闸口比 StableMax 的门槛先到。换了窗户，门还堵着。

Why eps and precision decide life or death was already covered in §1: the whole point of StableMax is to push the saturation threshold further out, so the gradient doesn't get crushed to exactly 0 by absorption error. But fp32's ε≈1.2e-7 and the eps=1e-8 in Adam's denominator are two gates that come before StableMax's threshold. You swapped out the window, and the door is still blocked.

ko_xf 那两条崩的不是泛化回落，是事故。seed 0 在 10k 触到 test 0.987，15k 掉到 0.064；seed 2 在 25k 触到 0.930，30k 掉到 0.011；两次 train acc 同步塌到 0.04 以下。如果只有 test 掉、train 还在 1.0，那是泛化解又被记忆解挤掉；train 一起塌，是优化本身出了事。seed 1 从未触线（峰值 0.656）。

The two ko_xf runs that collapsed aren't generalization receding — they're accidents. Seed 0 touched test 0.987 at 10k and was down to 0.064 by 15k; seed 2 touched 0.930 at 25k and was down to 0.011 by 30k; both times train acc caved in below 0.04 at the same moment. If only test dropped while train stayed at 1.0, that would be the generalizing solution getting squeezed out by the memorizing one again; train caving in along with it means the optimization itself broke. Seed 1 never touched the line (peak 0.656).

后来把这两条按 500 步一格重跑，每 10 步记一次 train loss，看到的顺序是：先有一个 loss 尖峰，比崩前 5k 步的中位数高 10⁶ 到 10⁸ 倍，两三百步之后 train acc 才塌。

Later we reran those two with a checkpoint every 500 steps and train loss logged every 10 steps, and the order we saw was: first a loss spike, 10⁶ to 10⁸ times the median of the 5k steps before the crash, and only two or three hundred steps later does train acc cave in.

要往下拆，得先说 Adam 怎么走一步：步长 ≈ lr × m ／ (√v + eps)。m 是梯度的滑动平均，管往哪走；v 是梯度平方的滑动平均，管这一步迈多大；β₁、β₂ 是这两个平均的记性（β₁ 0.9 大约记 10 步，β₂ 0.999 大约记 1000 步），eps 是分母底下垫的地板。ko_xf 这组是 lr 0.01、β₂ 0.999、eps 1e-25。

To take this further apart, first how Adam takes a step: step ≈ lr × m / (√v + eps). m is a moving average of the gradient and decides which way to go; v is a moving average of the squared gradient and decides how big the step is; β₁ and β₂ are the memories of those two averages (β₁ 0.9 remembers roughly 10 steps, β₂ 0.999 roughly 1000), and eps is the floor propped under the denominator. ko_xf runs at lr 0.01, β₂ 0.999, eps 1e-25.

第一刀先动 β₂。只把它改成 0.98、别的一个字不动，三个 seed 在 1–2k 触线，到 150k 末 test 1／.998／.997，但 seed 0 在 3k–7k 之间还短崩过三次，又自己爬了回来。反过来，在 lr 0.001 的 transformer 组（节 2 那组）把 β₂ 从 0.98 改成 0.999，只是变慢、变抖，没有猝死。β₂ 调快能救大部分，救不干净。

First cut: β₂. Change only β₂ to 0.98, not one other character, and all three seeds touch the line at 1–2k and finish at 150k with test 1 / .998 / .997 — but seed 0 still crashes briefly three times between 3k and 7k and climbs back on its own. Conversely, in the lr 0.001 transformer group (the one from §2), changing β₂ from 0.98 to 0.999 only makes it slower and jumpier — no sudden death. A faster β₂ saves most of it, not all of it.

第二刀把 β₂ 和 eps 正交着拧：lr 钉在 0.01、wd 0，β₂ 取 0.98／0.999，eps 取 1e-25／1e-8／1e-4，每格 3 seed，跑满 150k 不早停，checkpoint 250 步一格，18 个 run 全部跑完、0 failure。这里"崩"的定义是 train acc 曾到过 0.5，之后又跌破 0.5。

Second cut: turn β₂ and eps orthogonally. lr pinned at 0.01, wd 0, β₂ in 0.98 / 0.999, eps in 1e-25 / 1e-8 / 1e-4, 3 seeds per cell, the full 150k with no early stopping, a checkpoint every 250 steps; all 18 runs finished, 0 failures. "Crash" here means train acc had reached 0.5 and later fell back below 0.5.

eps=1e-25 那一行，β₂ 0.999 三条全崩，两条到 150k 都没爬回来，第三条前后崩了二十次，最后停在 test 0.96。eps 一抬到 1e-8，两种 β₂ 的六个 seed 一次都没崩，也照样 grok，末 test 全在 0.95 以上。所以 eps 大并没有把模型压成学不会，它只是把那一脚踩飞的步子拦住了。

In the eps=1e-25 rows, all three β₂ 0.999 runs crash, two of them never climb back by 150k, and the third crashes twenty times in all and ends at test 0.96. Raise eps to 1e-8 and none of the six seeds, under either β₂, crashes even once — and they still grok, all finishing with test above 0.95. So a bigger eps doesn't press the model into being unable to learn; it just stops the step that goes flying.

我的读法：train 背完之后梯度很小，v 也缩得很小；eps 几乎是 0 的时候，分母就只剩 √v。梯度突然冒个头，记性短的 m 十来步就跟上了，记性长的 v 还以为梯度很小，一个正常的 m 除以一个过时的极小分母，一步就迈飞了。β₂ 调短是让 v 跟得快，eps 抬高是给分母一个底，两刀各挡一截。Adam 原论文 §2.1 其实写过这件事的上界：当 (1−β₁) > √(1−β₂)，单个参数一步最多走 lr × (1−β₁)/√(1−β₂)，否则最多走 lr。β₁ 0.9、β₂ 0.999 代进去是约 3.16 × lr，β₂ 0.98 就回到 1 × lr。β₁ 和 β₂ 的记性差得越远，最坏那一步越猛；可 1 × lr 乘上几万个参数一起顶格，照样能掀翻模型，所以 β₂=0.98 那格还是崩过一条。

My reading: once train is memorized the gradient is small, and v shrinks to small too; with eps practically 0, the denominator is nothing but √v. When the gradient suddenly pokes its head up, the short-memory m catches up within ten-odd steps, while the long-memory v still thinks the gradient is tiny — a normal-sized m divided by a stale, tiny denominator, and one step goes flying. Shortening β₂ lets v catch up faster; raising eps gives the denominator a floor; each cut blocks part of it. The original Adam paper actually writes down the bound for this in §2.1: when (1−β₁) > √(1−β₂), a single parameter moves at most lr × (1−β₁)/√(1−β₂) per step, and at most lr otherwise. Plug in β₁ 0.9, β₂ 0.999 and you get about 3.16 × lr; with β₂ 0.98 it's back to 1 × lr. The further apart the memories of β₁ and β₂, the harder the worst step hits. But 1 × lr across tens of thousands of parameters all maxed out at once can still knock the model over, which is why one run in the β₂=0.98 cell still crashed.

能写到哪：在这套配方里，极小的 eps 是触线后崩塌的参与因素，β₂ 0.999 把风险放大，但它不单独决定崩不崩。Adam 内部那个比值我们只在 250 步一格的 checkpoint 上记了一个代理量，没逐步记，所以不能说每一个崩点都按上面这条链走，也不往别的 lr、架构、任务外推。

How far this goes: in this recipe, a tiny eps is a participating cause of the crash after touching the line, and β₂ 0.999 amplifies the risk, but β₂ alone doesn't decide whether it crashes. We only logged a proxy for that ratio inside Adam, at 250-step checkpoints, not step by step, so we can't say every crash follows the chain above, and none of this extrapolates to other lrs, architectures or tasks.

fig 4 · knockout accuracy trajectories (image not loaded)

这张表答不了的事What this table can't answer

• ko_p59 同时动了模数和数据量（p113/0.4→p59/0.3），所以单看这一行，死因归不到模数头上。后来拆开补跑：只改模数（p59 / frac0.4）、只改数据量（p113 / frac0.3）各 3 seed，300k 步内全没 grok（末 test 分别 0.02–0.04 和 0.002–0.007）。两个轴单拎出来都够死。
• eps 和 fp32 是同一个死因的两个剂量，这是我的判断，不是测出来的。 两条死法看着不一样（末范数 2.2k 对 4.5k，末 test 0.05 对 0.18），但我信它们死于同一件事：带着泛化信息的那部分梯度本来就小，小到掉进了数值精度下面。fp32 是整体精度太粗，这部分信息直接丢了；eps=1e-8 是把 Adam 的分母垫高，把本来就小的梯度再压一截，压到不起作用。都是被同一道闸口拦下，只是闸口高低不同。ko_lr 不在这一类：lr 低不是死，是步子小、走得慢，它照样走到了泛化（70k–100k grok）。我们没有直接量那部分梯度有多大，所以这条只能写成判断。
  这里要补一个反方向的例子：上面 ko_xf 那组里，同一个 eps=1e-8 是救命的，高 lr 的 transformer 靠它不崩；在这张表的 MLP 上，它让三条全死。eps 太小是没有刹车，太大是刹车一直踩着。同一个 1e-8，为什么在 transformer 上不压人，我们没测，留作问题。

• ko_p59 changes both the modulus and the amount of data (p113/0.4→p59/0.3), so from this row alone you can't pin the death on the modulus. We later split it apart and ran more: modulus only (p59 / frac0.4) and data only (p113 / frac0.3), 3 seeds each — none grokked within 300k steps (final test 0.02–0.04 and 0.002–0.007 respectively). Either axis pulled out on its own is enough to kill it.
• eps and fp32 are two doses of the same cause of death — that's my judgment, not something we measured. The two deaths look different (final norm 2.2k vs 4.5k, final test 0.05 vs 0.18), but I believe they die of the same thing: the part of the gradient that carries the generalization information is small to begin with, small enough to fall below numerical precision. fp32 is precision that's too coarse across the board, so that information is simply lost; eps=1e-8 props up Adam's denominator and presses the already-small gradient down another notch, until it stops doing anything. Both are stopped by the same gate, just a gate set at different heights. ko_lr doesn't belong in this category: low lr isn't death, it's small steps and slow walking, and it still made it to generalization (grokked at 70k–100k). We never directly measured how big that part of the gradient is, so this one can only be written as a judgment.
  One example pointing the other way belongs here: in the ko_xf group above, the very same eps=1e-8 is the lifesaver — it's what keeps the high-lr transformer from crashing; on this table's MLP, it kills all three. Too small an eps is having no brakes; too big is riding the brakes the whole way. Why the same 1e-8 doesn't crush the transformer, we didn't test; we're leaving it as an open question.

§ 4alr 在有 wd 的世界里lr in a world with wd

这一节换 regime 了，主战场是 softmax + wd=1 的 transformer（1 层，w128，p59 / frac0.3，60k 步），和节 2、节 3 的 StableMax 无 wd 完全不是一回事。

This section switches regime. The main battlefield is a softmax + wd=1 transformer (1 layer, w128, p59 / frac0.3, 60k steps) — a completely different animal from the StableMax, no-wd setups in §2 and §3.

Power §3.3说 grokking 对 lr 挑剔，窗口窄。我们扫出来的是：

Power §3.3 says grokking is picky about lr, with a narrow window. What our sweep turned up:

lr 越大，grok 越快。 B 组 3 seed：lr 1e-2 在 1–2k，3e-3 在 3–6k，1e-3 在 13–22k，3e-4 在 60k 内没过 0.9。单 seed 的 lr_sweep_out 同向：2k / 3k / 21k。快慢差 10 倍。

The bigger the lr, the faster the grok. Group B, 3 seeds: lr 1e-2 at 1–2k, 3e-3 at 3–6k, 1e-3 at 13–22k, and 3e-4 didn't cross 0.9 within 60k. The single-seed lr_sweep_out points the same way: 2k / 3k / 21k. A 10× gap between fast and slow.

一个小定量：lr ≥ 1e-3 时，onset ≈ (13–19)/lr。 把每一档的 grok step 乘上 lr，三档全落在 13–19 这条窄带里：1e-2 × 1.5k ≈ 15，3e-3 × 4.5k ≈ 13.5，1e-3 × 17.5k ≈ 17.5。像是网络要走一段固定的"距离"，lr 只是步幅。

A small quantitative note: for lr ≥ 1e-3, onset ≈ (13–19)/lr. Multiply each setting's grok step by its lr and all three land in a narrow 13–19 band: 1e-2 × 1.5k ≈ 15, 3e-3 × 4.5k ≈ 13.5, 1e-3 × 17.5k ≈ 17.5. It's as if the network has to cover a fixed "distance", and lr is just the stride.

这个乘积能落在一条这么窄的带里，本身比"越大越快"更值得注意。乘积近似守恒说的是一件更强的事：在这个 regime 里，从开始到 grok，参数要走的路程长度跟 lr 几乎无关，lr 只决定这段路你一步迈多大。当然三档 × 3 seed的样本量小，没有外部复现，所以我只把它写成观察。

That this product lands in such a narrow band is, by itself, more interesting than "bigger is faster". An approximately conserved product says something stronger: in this regime, the length of the path the parameters travel from the start to grokking is almost independent of lr; lr only decides how big a step you take along it. Of course, three settings × 3 seeds is a small sample with no outside replication, so I'm only writing it down as an observation.

lr过大：优化不稳定。 lr 3e-2 / 1e-1 / 3e-1 三档全是 memorize only——但看曲线就知道不是"窗口关了"，是 train 本身就没稳住。这个区别很重要：如果是"窗口关了"，说明 grokking 这个现象对 lr 有一个内在的上限；如果只是训练发散，那它和 grokking 无关，换个 warmup、换个 clip 说不定就过去了。所以"窄窗"的上沿在这个 setup 里应该改判成：lr 大到训不动为止，能训动就 grok 得更快。

lr too large: unstable optimization. lr 3e-2 / 1e-1 / 3e-1 are all memorize-only — but one look at the curves tells you it isn't "the window closed", it's that train itself never settled. The distinction matters: if "the window closed", grokking as a phenomenon would have an intrinsic upper limit on lr; if it's just training diverging, that has nothing to do with grokking, and a warmup or a clip might well get you past it. So in this setup the upper edge of the "narrow window" should be re-ruled as: lr can go up until training stops working, and as long as it still trains, it groks faster.

lr 过小：慢，而且比 1/lr 更慢。 3e-4 那档在 60k 内没过 0.9。按 13–19 那个乘积反推，它该在 43k–63k 起飞，正好贴着预算的边，所以这一行最容易写错：它"没 grok"和它"该 grok 的时刻"落在同一个位置，按"lr 太小则窗口关闭"写下去，就是用预算长度伪造一个现象。

lr too small: slow, and slower than 1/lr. The 3e-4 setting didn't cross 0.9 within 60k. Working backward from the 13–19 product, it should take off at 43k–63k, right up against the edge of the budget, which makes this the easiest row to get wrong: where it "didn't grok" and where it "should grok" sit in the same spot, and if you write it up as "lr too small closes the window", you're forging a phenomenon out of the budget length.

后来把这档延到 120k 跑了三个 seed：两条过了，78k 和 106k；一条到 120k 还没过（test 0.284）。所以下沿不是死刑。但 78k × 3e-4 ≈ 23、106k × 3e-4 ≈ 32，出了 13–19 那条带：到这一档，等待涨得比 1/lr 快。上面那个乘积因此只在 lr ≥ 1e-3 成立；3e-4 单独记一行，乘积 23–32，外加一条 120k 内截尾。

Later we extended this setting to 120k with three seeds: two crossed, at 78k and 106k; one still hadn't crossed at 120k (test 0.284). So the lower edge isn't a death sentence. But 78k × 3e-4 ≈ 23 and 106k × 3e-4 ≈ 32, outside the 13–19 band: at this setting the wait grows faster than 1/lr. So the product above only holds for lr ≥ 1e-3; 3e-4 gets a row of its own, product 23–32, plus one run censored within 120k.

fig 5 · lr window, three settings (image not loaded)

fig 6 · good / slow / wall (image not loaded)

这一组的 β₂ 是 0.98，checkpoint 每 1000 步一格；3e-4 延长那批也是 1000 步一格。

This group's β₂ is 0.98, with a checkpoint every 1000 steps; the extended 3e-4 batch is also one every 1000 steps.

§ 4b换个 regime，同一个旋钮换了角色switch regimes, and the same knob plays a different part

节 3 敲除表里的 ko_lr（StableMax、无 wd、MLP，只动 lr 这一个变量）把 lr 从 0.01 敲到 0.001，grok 从 60–70k 挪到 70–100k——只慢了约 1.3 倍，不是 10 倍。乘积 step×lr 是 650 对 85，散了 8 倍。上一节 lr 缩 10 倍等 10 倍，这里 lr 缩 10 倍只等 1.3 倍。

In the §3 knockout table, ko_lr (StableMax, no wd, MLP, changing only the one variable lr) knocks lr from 0.01 down to 0.001, and grokking moves from 60–70k to 70–100k — only about 1.3× slower, not 10×. The product step×lr is 650 vs 85, scattered by 8×. In the previous section, shrinking lr 10× meant waiting 10× longer; here shrinking lr 10× means waiting only 1.3× longer.

这就不只是常数变了，是上一节的∝1/lr规律不再适用了。所以问题升级：lr 是不是只在 wd 在场的时候才是油门。我们打算有 wd／无 wd × 两个任务，每格扫 lr。

That's not just the constant changing; the ∝1/lr law from the previous section no longer applies. So the question escalates: is lr only the gas pedal when wd is in the room? Our plan: with wd / without wd × two tasks, sweeping lr in every cell.

网格结果Grid results

共同 setup：p113 / frac0.4 / AdamW / StableMax / fp64 / eps 1e-25，lr 四档 0.01 / 3e-3 / 1e-3 / 3e-4，wd 两臂 0 和 1，预算 400k 步，checkpoint 每 5000 步一格，3 seed。

Shared setup: p113 / frac0.4 / AdamW / StableMax / fp64 / eps 1e-25, four lr settings 0.01 / 3e-3 / 1e-3 / 3e-4, two wd arms, 0 and 1, a 400k-step budget, a checkpoint every 5000 steps, 3 seeds.

① MLP × mod_add（主网格，24 run）——每格是 grok step 中位数，括号里是 step×lr：

① MLP × mod_add (main grid, 24 runs) — each cell is the median grok step, with step×lr in parentheses:

② MLP × mod_mul（换任务，24 run）：

② MLP × mod_mul (different task, 24 runs):

wd=0 那一列：高 lr 段是平的，低 lr 段才按 1/lr 走The wd=0 column: flat at high lr, 1/lr only at low lr

从 0.01 降到 3e-3，mod_add 上还快了一点（65k→50k，三个 seed 是 60/65/70k 对 50/50/55k，不重叠），mod_mul 上持平（60k→65k）。再往下才开始变慢，而且比 1/lr 更陡：1e-3→3e-4，mod_add 从 85k 到 337k，3.3 倍的 lr 换来 4 倍的等待；mod_mul 直接 >3.6 倍。

Going from 0.01 down to 3e-3, mod_add actually got a little faster (65k→50k; the three seeds are 60/65/70k vs 50/50/55k, no overlap), and mod_mul stayed level (60k→65k). Only further down does it start to slow, and more steeply than 1/lr: 1e-3→3e-4, mod_add goes from 85k to 337k, a 3.3× cut in lr buying a 4× longer wait; mod_mul goes straight past 3.6×.

所以这四个点不能用一条直线拟合：高 lr 那两档几乎不随 lr 动，低 lr 那两档才按 1/lr 走，是两段不同的规律接在一起。高 lr 段另有东西在限速（乘积 650 对 85 就是它留下的痕迹），低 lr 段 lr 才重新变回油门。

So these four points can't be fit with one straight line: the two high-lr settings barely move with lr, only the two low-lr settings follow 1/lr — two different laws spliced together. In the high-lr range something else is setting the speed limit (the 650 vs 85 product is the trace it leaves); only in the low-lr range does lr turn back into the gas pedal.

在同一个网格里、同一个 lr 旋钮上，wd=1 时它一路都是油门，wd=0 时它在高 lr 段几乎不起作用。 这是这篇的主 claim，范围是 one-hot MLP 上的两个模算术任务；换任务（mod_add → mod_mul）形状不变。

In the same grid, on the same lr knob: at wd=1 it's a gas pedal all the way along; at wd=0 it does almost nothing in the high-lr range. That's the main claim of this post, scoped to two modular-arithmetic tasks on a one-hot MLP; switching tasks (mod_add → mod_mul) doesn't change the shape.

要说清楚的是它和文献的关系。lr、wd 各自怎么决定 grok 时刻，近一年已经有人系统测过：arXiv:2603.13331 报告了 lr 与 wd 的逆标度，arXiv:2606.30388 推了 lr、batch、L2 的标度关系，arXiv:2609.10657 在 384 个两层 MLP 配置上拟出 T_grok ∝ η^−0.50 λ^−0.64，用的 lr 网格和我们一模一样。所以"wd 在场时 lr 是油门"不是我们的发现。这篇能加的是那三篇网格里都没有的一格：wd = 0。他们的 wd 从 0.1 起，我们把 0 和 1 并排放，看到的是同一个旋钮在两边定律形状不同。从 0 到 0.1 之间形状在哪里翻过去，下一小节补上。

What needs spelling out is how this relates to the literature. How lr and wd each set the grok time has been measured systematically in the past year: arXiv:2603.13331 reports inverse scaling in lr and wd, arXiv:2606.30388 derives scaling relations among lr, batch size and L2, and arXiv:2609.10657 fits T_grok ∝ η^−0.50 λ^−0.64 over 384 two-layer MLP configs, using exactly the same lr grid we did. So "with wd present, lr is the gas pedal" is not our finding. What this post can add is the one cell none of those three grids has: wd = 0. Their wd starts at 0.1; we put 0 and 1 side by side and see the same knob obeying differently shaped laws on the two sides. Where between 0 and 0.1 the shape flips is filled in in the next subsection.

wd 从 0 加到 2：没有开关，是一路往下wd from 0 to 2: no switch, just downhill all the way

补的那批是 MLP × mod_add，wd 取 0 / 0.03 / 0.1 / 0.3 / 0.5 / 1 / 2 七档，lr 还是那四档，3 seed，72 个 run，checkpoint 从 5000 步加密到 250 步，0 failure。

The follow-up batch is MLP × mod_add, wd at seven settings 0 / 0.03 / 0.1 / 0.3 / 0.5 / 1 / 2, the same four lrs, 3 seeds, 72 runs, checkpoints densified from every 5000 steps to every 250, 0 failures.

每一档 lr 下，wd 越大 grok 越早，测到的范围里没有 U 形，也就是没有"wd 加过头反而变慢"的那一段。lr 1e-3 那一行最直观（每格是 grok step 中位数）：

At every lr, the bigger the wd the earlier the grok, and within the range we measured there's no U-shape — no stretch where too much wd starts slowing things down again. The lr 1e-3 row shows it most plainly (each cell is the median grok step):

wd 只加到 0.03，等待就少了一大半。lr 的斜率也跟着 wd 变。写成 grok step ∝ lr^−α，α 越大，lr 越像油门；低 lr 段用 3e-4 到 1e-3，高 lr 段用 3e-3 到 1e-2，分开算：

Adding just 0.03 of wd cuts the wait by more than half. The lr slope moves with wd too. Write it as grok step ∝ lr^−α: the bigger α, the more lr acts like a gas pedal. The low-lr range runs from 3e-4 to 1e-3; the high-lr range runs from 3e-3 to 1e-2. We fit them separately:

wd=0 的高 lr 段是平的，甚至略微反向（就是上面 65k 对 50k 那点差）；wd 只要有 0.03，高 lr 段就有了斜率。所以上面那个问题的答案是：形状没有在哪个点"翻过去"。wd 一离开 0，lr 就重新接上油门，接上多少随 wd 变。wd=1 的低 lr 段 α≈0.93，和节 4a 那个 softmax + wd=1 transformer 的 ∝1/lr 对得上。

At wd=0 the high-lr range is flat, even slightly reversed (that's the 65k vs 50k gap above); give it even 0.03 of wd and the high-lr range has a slope. So the answer to the question above is: the shape doesn't "flip" at any one point. As soon as wd leaves 0, lr gets reconnected to the gas, and how much depends on wd. At wd=1 the low-lr α≈0.93, which lines up with the ∝1/lr of the softmax + wd=1 transformer in §4a.

限制有三条。wd=0、lr 3e-4 那格只有 2/3 在 400k 内过线，算不出干净的 α，表里空着。最快的几格（wd 1、2 在 lr 0.01）只有 500–750 步，离 250 步的新地板只差两三格，高 lr 段那几个 α 只当描述看。一条直线拟全段会把这种转折抹平，所以分两段报；可分两段本身也是我们挑的，换个切点数字会变。

Three limits. In the wd=0, lr 3e-4 cell only 2 of 3 crossed within 400k, so there's no clean α and the table leaves it blank. The fastest cells (wd 1 and 2 at lr 0.01) take only 500–750 steps, just two or three checkpoints above the new 250-step floor, so the high-lr αs are descriptive only. One straight line over the whole range would smooth this bend away, so we report two segments; but the split is itself our choice, and moving the cut point changes the numbers.

顺带：3e-4 那格的 step×lr＝4.5，和节 4a 那个 softmax+wd1 transformer 的 13–19 不是同一个常数。这个常数也不许搬家。

While we're here: the 3e-4 cell's step×lr = 4.5, which is not the same constant as the 13–19 of the softmax+wd1 transformer in §4a. This constant doesn't get to move house either.

transformer 那一列：β₂ 换稳了，还是不进拟合The transformer column: stable β₂ now, still out of the fit

第三个网格是 transformer × mod_add。第一次跑用的是 β₂ 0.999，test 曲线在弹：触线之后掉下来，有的再弹上去，"grok step"抓到的是尖峰不是台阶，原因就是节 3 拆的那件事。按 β₂ 0.98 重跑了 24 个 run（150k、250 步一格、不早停）：

The third grid is transformer × mod_add. The first run used β₂ 0.999 and the test curves bounced: they touched the line, fell back down, some bounced back up; the "grok step" was catching a spike, not a staircase, for exactly the reason taken apart in §3. We reran it at β₂ 0.98, 24 runs (150k, a checkpoint every 250 steps, no early stopping):

• wd=1：12 条全过线，中位 grok step 在 lr 0.01 / 3e-3 / 1e-3 / 3e-4 上是 250 / 750 / 2250 / 3250。单调，但最快那档贴在 250 步的地板上。
• wd=0：lr 0.01 三条在 750–1750 步过线，3e-3 中位 10.75k，1e-3 中位 79.5k，3e-4 反而是 23.25k（只有 2/3 过线，截尾那条不算）。低 lr 端不单调，seed 之间差得很远。

• wd=1: all 12 cross the line; median grok steps at lr 0.01 / 3e-3 / 1e-3 / 3e-4 are 250 / 750 / 2250 / 3250. Monotone, but the fastest setting sits on the 250-step floor.
• wd=0: the three lr 0.01 runs cross at 750–1750 steps, 3e-3 has a median of 10.75k, 1e-3 a median of 79.5k, and 3e-4 is back down at 23.25k (only 2 of 3 crossed; the censored one isn't counted). Non-monotone at the low-lr end, with seeds far apart.

所以这一列还是不画线、不算 α。有一件事值得记：transformer 在 wd=0 的高 lr 段不是平的，0.01 比 3e-3 快了约 10 倍，和 MLP 那条"高 lr 段 lr 不管用"的形状对不上。本篇的主 claim 因此只落在 MLP 上。

So this column still gets no line and no α. One thing worth noting: the transformer's high-lr range at wd=0 isn't flat — 0.01 is about 10× faster than 3e-3 — which doesn't match the MLP's "lr stops mattering at high lr" shape. That's why this post's main claim stays on the MLP.

一条让我安心不少的复现核对A reproducibility check that put my mind at ease

lw 网格里 lr 0.01 和 1e-3 的 wd=0 三个 seed，与节 3 那批 ko_base / ko_lr 逐 run 完全相同（中位 65k / 85k，ratio 1.000）——同 seed 全确定性。敲除表的那两个点站得住，这次只是往两头各加了一档。

In the lw grid, the three wd=0 seeds at lr 0.01 and 1e-3 are identical, run for run, to the ko_base / ko_lr batch from §3 (medians 65k / 85k, ratio 1.000) — same seed, fully deterministic. Those two points in the knockout table hold up; this time we just added one more setting at each end.

另外我问过一句"是不是谁抖一下就 grok"，答案是：MLP 那 24 条曲线全都光滑，不抖。

I also asked at one point, "does something just jitter its way into grokking?" The answer: all 24 MLP curves are smooth. No jitter.

fig 7 · grok step vs lr, wd=0 / wd=1 (image not loaded)

fig 8 · all 24 MLP wd=0 curves (image not loaded)

fig 9 · censored at 400k, still climbing (image not loaded)

我对这条曲线的读法：lr 不是唯一的油门，到了高 lr 段它的作用就不明显了。65k 和 50k 之间那点差，我不把它当成"高 lr 反而更慢"的证据：mod_add 上快了一点，mod_mul 上慢了一点，方向不一致，差距和 seed 之间的散布（约 10k）是一个量级。高 lr 段更像是平的，平是因为别的东西在主导，我猜是两样：更新本身的噪声（步子大、抖得凶，把 lr 带来的加速吃掉），加上 seed 之间的随机差。这是猜测，我们没量噪声。

My reading of this curve: lr isn't the only gas pedal, and once you're in the high-lr range its effect fades. I don't take the small gap between 65k and 50k as evidence that "a higher lr is actually slower": mod_add got a bit faster, mod_mul a bit slower, the directions disagree, and the gap is the same order of magnitude as the spread between seeds (about 10k). The high-lr range looks more like it's flat, and it's flat because something else is in charge. My guess is two things: the noise in the updates themselves (big steps, hard shaking, eating up the speedup lr brings), plus random seed-to-seed differences. That's a guess; we didn't measure the noise.

§ 5范数的三种命运three fates of the norm

这一节是我的招牌观点的主场，所以先说清楚台子是怎么搭的：三条命运来自三个不同的 regime，数字别混着比。

This section is home turf for my signature take, so let me first be clear about how the stage is built: the three fates come from three different regimes, so don't compare the numbers across them.

命运① 准点崩（Softmax Collapse）。 softmax、没有 wd 的 transformer：train acc 拉满之后，某一步 train acc 自己从 1.0 砸到 0.02–0.08，参数范数同时从约 135 跳到 190–310，test 全程贴地没起来过。它和节 3 的 ko_xf 一样，都是训练集跟着一起塌；是不是同一个机制，我们没判。这是 Prieto 讲的 SC 机制的复现——logits 等比放大把 softmax 推进饱和区，梯度被吸收误差压成精确的 0。网络不是没学会，是心脏骤停。范数在这里是凶器。

Fate ①: collapse right on cue (Softmax Collapse). A softmax transformer without wd: after train acc maxes out, at some step train acc smashes itself down from 1.0 to 0.02–0.08, while the parameter norm jumps from about 135 to 190–310; test stays on the floor the whole time and never gets up. Like ko_xf in §3, the training set caves in along with it; whether it's the same mechanism, we didn't rule. This is a reproduction of the SC mechanism Prieto describes — scaling the logits up proportionally pushes softmax into saturation, and the gradient gets crushed by absorption error to exactly 0. The network didn't fail to learn; its heart stopped. Here the norm is the murder weapon.

命运② 锯齿（slingshot），降级处理。 ortho 系的曲线上，范数跳台阶和 loss 尖峰对齐，一放一收像弹弓，对应 Thilak 2022 的 slingshot。但我们的采样是 2000 步一格，太粗，尖峰细节全靠两三个点撑着。所以这条在本文里只写成疑似，引 Thilak，不当发现，也不配图。

Fate ②: sawtooth (slingshot), downgraded. On the ortho-family curves, the norm's stair-steps line up with loss spikes, letting go and pulling back like a slingshot, which matches Thilak 2022's slingshot. But our sampling was one point every 2000 steps, far too coarse, and the spike details are held up by just two or three points. So in this post it's written only as suspected, citing Thilak — not a finding, and no figure.

命运③ 范数爆上天，grok 迟迟不来。 L_mlp（s0）的参数范数一路涨到 9.5 万，600k 步内 test 没过 0.9，而且最后 50k 步 test 还在以每 50k 约 0.005 的速度爬：不是平台，是极慢地爬。（这组是 lr 0.001 / β₂ 0.98 的 MLP，它为什么这么慢不在这篇的范围里。）所以这个例子说的不是"范数大就不 grok"，是范数在涨和 grok 未到可以共存 60 万步以上；截尾，不是死刑。

Fate ③: the norm blasts off and grokking keeps not coming. The parameter norm of L_mlp (s0) climbs all the way to 95k, test doesn't cross 0.9 within 600k steps, and over the last 50k steps test is still creeping up at about 0.005 per 50k: not a plateau, an extremely slow climb. (This group is an MLP at lr 0.001 / β₂ 0.98; why it's this slow is outside the scope of this post.) So what this example says isn't "a big norm means no grok"; it's that a rising norm and a grok that hasn't arrived yet can coexist for more than 600k steps. Censored, not a death sentence.

四象限：三格有人住，第四格我们没有例子Four quadrants: three are occupied, and for the fourth we have no example

原来的讲义里写的是"四象限全有人住"，那是我在填这张 2×2 空表时当场发现的漏洞——只点得出三格。第四格的候选是节 4a 那个 softmax + wd=1 的 transformer（它在 1–2k 就 grok，范数大概率没飞），但我们没量过。

The original lecture notes said "all four quadrants are occupied"; that's a hole I caught on the spot while filling in this empty 2×2 table — I could only point to three cells. The candidate for the fourth is the softmax + wd=1 transformer from §4a (it groks at 1–2k, and its norm very likely didn't take off), but we never measured it.

一个追问，逼出了这节最硬的一段材料One follow-up question squeezed out the hardest material in this section

我当时问了一句：连 NLM 都跑不起来，难道 train loss 也高？

At the time I asked: if even NLM can't get going, is the train loss high too?

不高。ko_eps 和 ko_fp32 的三个 seed final train acc 全是 1.0，train 集全背完了，而且三条曲线 20k 步前基本重合。分道发生在"loss 还能往下掉多少"：ko_base 的 loss 掉到 1e-19 量级，末范数 93k；ko_eps 卡在 1e-7 附近，步长被 1e-8 的分母顶住，末范数只有 2.2k；ko_fp32 冻在 1e-10，梯度精确为 0，末范数 4.4–4.7k。（梯度量级是从 loss 量级估的，没有直接量过，这里必须写成估算。）

It isn't. For ko_eps and ko_fp32, all three seeds have final train acc 1.0 — the training set is fully memorized — and the three curves basically overlap before 20k steps. Where they part ways is "how much further the loss can still fall": ko_base's loss drops to the 1e-19 range, final norm 93k; ko_eps gets stuck around 1e-7, its step size pinned by the 1e-8 denominator, final norm only 2.2k; ko_fp32 freezes at 1e-10, gradient exactly 0, final norm 4.4–4.7k. (The gradient magnitudes are estimated from the loss magnitudes, not measured directly, so this has to be written as an estimate.)

所以"范数只有几千"和"没 grok"是同一个死因的两个症状，不是范数小导致没 grok。这也回答了我后来的另一个问题——几千步不 grok 怎么约束范数：没有人约束它，是梯度被吸收误差压成了 0，它自己涨不动。

So "the norm is only a few thousand" and "didn't grok" are two symptoms of the same cause of death, not a small norm causing the failure to grok. It also answers another question I asked later — what holds the norm down at a few thousand when there's no grok: nothing holds it down; the gradient got crushed to 0 by absorption error, and the norm can't grow on its own.

收到脊梁上Back to the spine

按效率压力那套解释，泛化＝范数降下来、解变经济。我们这边的证据是范数对 grok 非充分（爆到 9.5 万也没来）非必要（爆到 9.4 万照样来）。另外两条线同向：大 init 的范数在 5k 步内断崖砸平，test 照样慢爬；MNIST 上 dropout 把逃逸步从约 15400 压到 200，泛化提速的时候范数不按剧本走（这条是博客③的料，这里只做旁证）。

Under the efficiency-pressure account, generalization = the norm comes down and the solution gets economical. Our evidence says the norm is not sufficient for grokking (exploded to 95k and it still didn't come) and not necessary (exploded to 94k and it came anyway). Two other lines point the same way: with a large init, the norm drops off a cliff and flattens out within 5k steps, and test still creeps up slowly; on MNIST, dropout pulls the escape step from about 15400 down to 200, and while generalization speeds up, the norm doesn't follow the script (that one's material for blog ③; here it's only corroboration).

范数至多是仪表盘上的一根针：能判 NLM 在不在跑、优化还活不活着。不能判离 grok 还有多远——ko_base 在 test 还只有 0.5 的时候范数才 6.6k，而 L_mlp 涨到 9.5 万都没到。

At most, the norm is a needle on the dashboard: it can tell you whether NLM is running and whether the optimization is still alive. It can't tell you how far you are from grokking — ko_base's norm was only 6.6k while test was still at 0.5, and L_mlp climbed to 95k without getting there.

剩下一件事我们确实没拆开：wd 在 lr×wd 网格里同时把 grok 从几万步压进 5k 以内、又把范数摁住了。同一个旋钮动了两件事，这批数据分不开哪件是因。

One thing we genuinely didn't pull apart: in the lr×wd grid, wd simultaneously squeezed grokking from tens of thousands of steps down to under 5k and pinned the norm down. One knob moved two things, and this batch of data can't tell which one is the cause.

fig 10 · softmax collapse (image not loaded)

fig 11 · the norm does not vote (image not loaded)

我的猜测是这样一幅图：泛化的方向一直都在，只是它的梯度小、步子小。NLM（靠把 logits 整体放大来降 loss）是另一条步子大得多的捷径，范数就是沿着它涨上去的。只要 NLM 没有大到把泛化那点小梯度压到精度以下、压成 0，泛化就还有机会走到。

My guess is a picture like this: the direction of generalization is there all along, it's just that its gradient is small and its steps are small. NLM (lowering the loss by scaling up the logits as a whole) is another shortcut with much bigger steps, and the norm is what climbs along it. As long as NLM doesn't get big enough to press that small generalization gradient below precision, down to 0, generalization still has a chance to get there.

这一条能把前面几节串起来：ko_base 没有 wd，范数涨到 9.3 万，但 fp64 加 1e-25 的 eps 让那点小梯度活了下来，所以照样 grok；ko_eps 和 ko_fp32 是闸口先到，小梯度被吸收成 0，所以死（节 3 那条"同一个死因的两个剂量"就是它）；wd 则是反过来把 NLM 那条捷径摁住，泛化方向相对变大，所以快。wd 同时"压了范数"又"加快了 grok"，在这幅图里是同一个动作的两个读数。

This one idea strings the earlier sections together: ko_base has no wd and its norm climbs to 93k, but fp64 plus an eps of 1e-25 keeps that small gradient alive, so it groks anyway; for ko_eps and ko_fp32 the gate arrives first and the small gradient gets absorbed to 0, so they die (that's the "two doses of the same cause of death" from §3); wd works the other way round, pinning down the NLM shortcut so the generalization direction becomes relatively bigger, so it's fast. wd "pressing the norm down" and "speeding up grokking" at the same time are, in this picture, two readings of one and the same action.

这是猜测，这批数据没把它拆开。要验它，得单独削弱 NLM、不动别的，或者直接量泛化那部分梯度占总梯度多少，看它在各个 arm 里是不是正好在精度线上下分开。

This is a guess; this batch of data didn't pull it apart. To test it you'd have to weaken NLM on its own without touching anything else, or directly measure what share of the total gradient the generalization part is, and see whether, across the arms, it splits exactly above and below the precision line.

§ 6一个被我们自己敲掉一半的假说a hypothesis we knocked half-down ourselves

这一节是 process 叙事的收口：一个我们自己立的解释，一次写在跑之前的预测，和一个部分否定的结果。

This section is where the process story closes: an explanation we put up ourselves, a prediction written down before the runs, and a partially negative result.

假说怎么来的Where the hypothesis came from

看 MLP wd=0 那一臂的曲线，注意到一件怪事：lr 从 0.01 到 3e-4 差了 33 倍，但 test acc "离地"的时刻（首个 test ≥0.1 的 checkpoint）全挤在 25–30k。

Looking at the curves of the MLP wd=0 arm, I noticed something odd: lr differs by 33× from 0.01 to 3e-4, yet the moment test acc "leaves the ground" (the first checkpoint with test ≥0.1) is all crammed into 25–30k.

我当时正好在复习 Adam，自己算了一下：如果梯度方向来回翻，一阶动量 m 的稳态会被压到 1/19 量级。方向乱的时候 Adam 会自己收步子——于是假说是：Adam 的 m/√v 把 lr 抵消掉了，所以离地时刻看不出 lr。

I happened to be reviewing Adam at the time, and worked it out myself: if the gradient direction keeps flipping back and forth, the steady state of the first moment m gets pushed down to the order of 1/19. When the directions are a mess, Adam reins its own steps in — so the hypothesis was: Adam's m/√v cancels out lr, which is why you can't see lr in the departure moment.

然后我说：敲了吧，让 opus 写代码。

Then I said: knock it out, let opus write the code.

预测写在跑之前The prediction goes in before the run

这一次我们做了一件前面几节都没做的事：把预测写进工单，再发射。写下来的好处不是为了显得严谨，是为了事后不能改口——一个能被自己反悔的假说，跑多少 run 都是在收集支持证据。

This time we did something none of the earlier sections did: write the prediction into the work order, then launch. The point of writing it down isn't to look rigorous; it's so you can't change your story afterward — a hypothesis you're allowed to take back is just collecting supporting evidence, however many runs you throw at it.

工单原话：β₁=0（敲掉动量）之后，范数涨速应该随 lr 走，÷lr 之后四档应该接近常数。如果 β₁=0 下四档离地仍然齐刷刷在 25–30k，假说就死，转去查"离地由数据决定"。

The work order, verbatim: with β₁=0 (momentum knocked out), the norm growth rate should follow lr, and after ÷lr the four settings should be close to constant. If under β₁=0 the four settings still leave the ground in lockstep at 25–30k, the hypothesis dies, and we switch to checking "departure is decided by the data".

16 个 run，0 failure，10 grokked / 6 memorized_only。β₁=0.9 的基线直接用 lw_out 那 12 个 wd=0 run，不重跑。checkpoint 2500 步一格。

16 runs, 0 failures, 10 grokked / 6 memorized_only. The β₁=0.9 baseline reuses the 12 wd=0 runs in lw_out directly, no rerun. A checkpoint every 2500 steps.

往下看表之前，你可以先猜一下：动量整个敲掉之后，四档 lr 的离地时刻是会拉开到接近 33 倍，还是照样挤在一起？

Before you look at the tables below, take a guess: with momentum knocked out entirely, will the departure moments of the four lr settings spread out to nearly 33×, or stay bunched together like before?

离地时刻（首个 test ≥0.1 的 checkpoint，中位数）

Departure moment (first checkpoint with test ≥0.1, median)

20–30k 段参数范数涨速（原始／÷lr，中位数）

Parameter-norm growth rate over 20–30k (raw / ÷lr, median)

结果Result

lr 跨了 33 倍，离地时刻在 β₁=0 下只拉开 1.78 倍。不是 33 倍，但也不再是基线那种 1.2 倍的"齐刷刷"。÷lr 之后的范数涨速，三档 β₁ 全都跨 14–17 倍，预测里那个"接近常数"没出现。

lr spans 33×, and under β₁=0 the departure moments only spread out by 1.78×. Not 33×, but no longer the baseline's 1.2× "lockstep" either. The ÷lr norm growth rates span 14–17× for all three β₁ settings; the "close to constant" from the prediction never showed up.

所以结论不是"假说死了"，是：动量不是主因。m/√v 这个归一化里，把 lr 吃掉的主要是 √v 那一半，而 √v 我们没单独敲过。 敲掉 β₁ 只动了分子，分母的自适应缩放原封不动，剩下 14–17 倍的 ÷lr 跨度正是它留下的。

So the conclusion isn't "the hypothesis is dead"; it's: momentum isn't the main cause. In the m/√v normalization, what eats lr is mainly the √v half, and we never knocked out √v on its own. Knocking out β₁ only touched the numerator; the denominator's adaptive scaling was left untouched, and the remaining 14–17× ÷lr span is exactly what it leaves behind.

假说立的时候，我心里想的是"方向乱 → m 被平均掉 → 步子自己收小"，全部注意力在分子上。实测把这半边切掉之后，效应只掉了一点点，说明分母那半边一直在干更重的活。这不是推翻，是把同一个机制的责任重新分配了一次——原来的问句"是不是 Adam 把 lr 抵消了"答案仍然偏"是"，但抵消它的是谁，换了人。√v 怎么吃掉 lr，我们的猜测是它的记性太长：β₂=0.999 的 v 大概记着最近一千步的梯度平方。训练里 loss 一路降，梯度 g 跟着缩，可 √v 还停在过去那些大梯度的水平，于是 m/√v 越来越小，步子自己收了。lr 越大，loss 降得越快，g 缩得越快，和 √v 拉开的差距越大，步子收得越狠，大 lr 多走的那部分被吃回去。动量 m 也有记性（β₁=0.9 约十步），g 往下掉时 m 慢半拍、给分子撑了一点腰；β₁=0 把这点腰撤掉，缩得更狠，这正好对上下面那个"方向相反"的数。这些都还是猜测，没量过每一步的 m/√v。

When I put the hypothesis up, what I had in mind was "messy directions → m gets averaged out → steps shrink on their own", with all my attention on the numerator. When the measurement cut that half away, the effect only dropped a tiny bit, which says the denominator half had been doing the heavier lifting all along. This isn't an overturn; it's a reassignment of blame within the same mechanism — the answer to the original question, "is Adam cancelling out lr?", still leans "yes", but who's doing the cancelling has changed hands. How √v eats lr: our guess is that its memory is too long. The v of β₂=0.999 roughly remembers the squared gradients of the last thousand steps. During training the loss keeps falling and the gradient g shrinks with it, but √v is still sitting at the level of those big past gradients, so m/√v gets smaller and smaller and the step reins itself in. The bigger the lr, the faster the loss falls, the faster g shrinks, the wider the gap from √v, the harder the step reins in, and the extra ground a big lr covers gets eaten back. Momentum m has a memory too (about ten steps at β₁=0.9): when g drops, m lags half a beat and props the numerator up a little; β₁=0 takes that prop away, so it shrinks even harder, which lines up nicely with the "opposite direction" number coming up below. All of this is still guesswork; we never measured m/√v step by step.

下一发该敲 √v。Adam 里单独拿掉 √v 就是带动量的 SGD，所以最干净的一刀是直接换优化器。预测先写死：如果 √v 是吃掉 lr 的那个，换成 SGD 以后，四档 lr 的离地时刻应该跟着 lr 拉开、接近 33 倍；要是换了还挤在一起，这个解释就死了，得另找凶手。工单还没发。

The next shot should knock out √v. Take √v alone out of Adam and what you have is SGD with momentum, so the cleanest cut is to swap the optimizer outright. Prediction locked in first: if √v is the thing eating lr, then after switching to SGD the departure moments of the four lr settings should spread out with lr, close to 33×; if they're still bunched together after the switch, this explanation is dead and we have to find another culprit. The work order hasn't gone out yet.

还有一个细节值得单独写出来，因为它和预测方向相反：原始范数涨速对 lr 的依赖，β₁=0.9 时跨 2.34 倍，β₁=0 时跨 1.94 倍——敲掉动量之后，涨速对 lr 的依赖反而变弱了，也就是抵消变强了。如果动量是抵消 lr 的那个东西，这个数应该往另一个方向走。（这两个数都在上表里，比值是我自己算的。）

One more detail deserves its own mention, because it goes in the opposite direction from the prediction: the raw norm growth rate's dependence on lr spans 2.34× at β₁=0.9 and 1.94× at β₁=0 — after knocking out momentum, the growth rate's dependence on lr actually got weaker, meaning the cancellation got stronger. If momentum were the thing cancelling lr, this number should have moved the other way. (Both numbers are in the table above; the ratio is my own arithmetic.)

fig 12 · departure timing vs lr, per β₁ (image not loaded)

这一节的坑Pitfalls in this section

• 每一格只有 2 个 seed（基线格 3 个）。β₁=0.99 的 lr 0.01 那格，两个 seed 过线时刻差了 3 倍（262.5k 对 82.5k）。
• checkpoint 2500 步一格，"离地时刻"的分辨率就是 2500 步。
• 工单里有一项"可选但很值"的直接测量——每个 checkpoint 记 |m/(√v+eps)| 的均值和梯度余弦。没有它，范数涨速只是 m/√v 的代理指标。

• Each cell has only 2 seeds (3 in the baseline cells). In the β₁=0.99, lr 0.01 cell, the two seeds' crossing times differ by 3× (262.5k vs 82.5k).
• A checkpoint every 2500 steps means the resolution of the "departure moment" is 2500 steps.
• The work order had one "optional but very worth it" direct measurement — logging the mean of |m/(√v+eps)| and the gradient cosine at every checkpoint. Without it, the norm growth rate is only a proxy for m/√v.

§ 7收尾wrapping up

写到这里，这篇能留给读者的其实就两样东西。

Having got this far, there are really just two things this post can leave the reader with.

一是节 5 那幅图：泛化的方向一直都在，只是步子小；grok 来不来，看的是它有没有被 NLM 和数值精度一起压到 0 以下。eps、fp32、wd 这三个旋钮，在这幅图里拧的是同一个东西。

One is the picture from §5: the direction of generalization is there all along, its steps are just small; whether grokking comes depends on whether NLM and numerical precision, together, press it down to zero. eps, fp32 and wd — in this picture, those three knobs are all turning the same thing.

二是节 6 写死的那个赌局：把 Adam 换成 SGD，四档 lr 的离地时刻应该拉开到接近 33 倍。拉开了，"√v 吃掉 lr"这个猜测就站住了一半；不拉开，它就死了，得另找凶手。结果出来我会回来补在这里，不管是哪一种。

The other is the bet locked in in §6: swap Adam for SGD, and the departure moments of the four lr settings should spread out to nearly 33×. If they spread, the guess that "√v eats lr" is half vindicated; if they don't, it's dead and we find another culprit. When the result comes in I'll come back and add it here, whichever way it goes.

预告Coming next

下一篇想写的是题眼那条线：一个不为任务定制的通用学习器，哪些数据生成结构它能从样本里发现、哪些发现不了，边界由什么决定。 材料是 a·b+a² 这一族耦合函数的 arrested 天花板——数据轴上的临界比例、步数轴上 500k 不破的天花板，以及为什么"不能塞进 ansatz"和"没有泛化解"是两件事。那篇的 related work 会重得多（Furuta 2402.16726 点名测过同一家族并报 fail，Varma 2309.02390 的 semi-grokking 已经占了"arrested 是真稳态"这个说法），能不能写成一篇而不是一段 related work，还要看一个命门实验做不做得出来。

What I want to write next is the key thread: for a general-purpose learner that isn't tailored to the task, which data-generating structures can it discover from samples, which can't it, and what decides the boundary. The material is the arrested ceiling of the a·b+a² family of coupled functions — the critical ratio along the data axis, the ceiling along the step axis that still hasn't broken at 500k, and why "can't be fit into an ansatz" and "there's no generalizing solution" are two different things. The related work for that one will be a lot heavier (Furuta 2402.16726 explicitly tested the same family and reported fail; Varma 2309.02390's semi-grokking has already claimed "arrested is a genuine steady state"), and whether it can become a post of its own rather than a paragraph of related work depends on whether one make-or-break experiment can be pulled off.

参考文献references

• Power, A. et al. (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177. ICLR 2022 MLSI Workshop. —— grokking 的命名与最初现象；本文节 4a 引用的 lr 窗口在 §3.3，配置在 Appendix A.1.2。
• Prieto, L. et al. (2025). Grokking at the Edge of Numerical Stability. arXiv:2501.04697. ICLR 2025. —— Softmax Collapse / NLM / StableMax / ⊥Grad，本文节 1、2、3、5 复现的对象。本文卖的是过程和那张敲除表，机制全部是他们的。
• Nanda, N. et al. (2023). Progress Measures for Grokking via Mechanistic Interpretability. arXiv:2301.05217. ICLR 2023. —— Fourier 电路与 progress measures；本文节 2 提到的"Fourier top-5 占比"沿用这条线的工具。
• Thilak, V. et al. (2022). The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon. —— 本文节 5 命运②只标注为"疑似 slingshot"，证据不够，引此文备查。
• Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. arXiv:1412.6980. ICLR 2015. —— 本文节 3 引用的单步步长上界在其 §2.1。

• Power, A. et al. (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177. ICLR 2022 MLSI Workshop. — Named grokking and the original phenomenon; the lr window cited in §4a of this post is in §3.3, the config in Appendix A.1.2.
• Prieto, L. et al. (2025). Grokking at the Edge of Numerical Stability. arXiv:2501.04697. ICLR 2025. — Softmax Collapse / NLM / StableMax / ⊥Grad; the thing §1, 2, 3 and 5 of this post reproduce. What this post is selling is the process and that knockout table; the mechanisms are all theirs.
• Nanda, N. et al. (2023). Progress Measures for Grokking via Mechanistic Interpretability. arXiv:2301.05217. ICLR 2023. — Fourier circuits and progress measures; the "Fourier top-5 share" mentioned in §2 of this post uses this line's tools.
• Thilak, V. et al. (2022). The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon. — Fate ② in §5 of this post is only labeled "suspected slingshot"; the evidence isn't enough, so this is cited for reference.
• Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. arXiv:1412.6980. ICLR 2015. — The per-step size bound cited in §3 of this post is in its §2.1.

博客②预告涉及：Furuta, H. et al. (2024), arXiv:2402.16726；Varma, V. et al. (2023), Explaining Grokking through Circuit Efficiency, arXiv:2309.02390, ICLR 2024。

Referenced in the blog ② preview: Furuta, H. et al. (2024), arXiv:2402.16726; Varma, V. et al. (2023), Explaining Grokking through Circuit Efficiency, arXiv:2309.02390, ICLR 2024.