The question is whether a small amount of reinforcement learning on a strong model necessarily produces only a small behavioral effect. Dwarkesh Patel describes relatively few parameters changing, while an unresolved speaker emphasizes small model changes; Beren Millidge likewise describes reinforcement learning after mid-training as mainly adjusting an existing policy. Millidge nevertheless argues that even a small amount of learning can dramatically alter behavior when the starting model is strong.
One segment per claim
“can still be like super dramatic you know even if it's like even like one bit can change like your function space a lot and it can like rule out like half the hypothesis space which is huge so like i don't think it's necessarily the case it's like small amounts of bits small amounts of rl once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily”
“That's why actually very few parameters are changed. during an RL training step.”
“Like the reason RL is so nice is because like, yeah, it changes a very, very small amount about the model and there's like a lot of evidence for why this is the case.”
“often and then what rl does on top of that is it does like a lot of you know essentially tweaking to the policy and so this is one of the reasons why it doesn't need like as many bits as you would naively think it doesn't have to learn all of these behaviors from scratch it needs just like a few bits from these episodes which you do get and then the other thing that i really point out in my blog is that these bits are actually extremely high signal compared to like regular like pre-training which is why you need rl at all versus just like sfting on like successful reasoning traces”