Same disclaimer as last time: everything I learned comes from HuggingFace’s LLM Course and DeepLearning.AI’s Fine-tuning & RL for LLMs: Intro to Post-training, and the best way I know to check whether I understand something is to try to explain it!

What RL is, and where it sits

Reinforcement Learning is a type of training you apply to LLMs, primarily to make them more helpful, more aligned to human preferences, and increasingly to make them great at reasoning and problem solving — especially in the specific environments of your own software or systems.

It is different from SFT. In SFT we never let the model generate anything. Instead, we give it an expected answer and iteratively adapt its weights to make that exact answer more likely. In RL we do the opposite: we let the model generate, we score what it produced with a reward, and according to that reward we adapt its weights towards the behaviours that scored well, and away from those that didn’t.

RL is extremely powerful if the data is well prepared. It is the primary way of making models feel personal, helpful, safe and smart. It is also more expensive than SFT, less predictable, and far more experiment-driven. No free lunch.

It helps to see where all of this sits. Training a modern model runs roughly in this order: pretraining, then SFT, then — if you want human preferences — training a reward model, and finally RL. The previous post covered the second box; this one covers the last two.

Reasoning is the easiest category to understand. RL is powerful because you let the model explore plenty of potential solutions (because you ask it to generate tokens), and reinforce those that lead to successful answers. You don’t care that much how it arrived at the solution. This also lets LLMs find solution strategies that nobody wrote down in the training data, sometimes ones that surprise the people running the training.

Which architecture of RL?

There are plenty of different algorithms and types of reinforcement learning. We won’t dive into them too much, because they share the same fundamental principles and differ in the details. I will focus on a particular subset — GRPO, trained against verifier feedback and against rewards learned from human feedback.

As I was learning about the other techniques — PPO and DPO — I felt that GRPO gives the right foundation, and based on a few signals about how software companies are doing RL for their own needs, it also feels the most common. It comes from the DeepSeekMath paper and it is the algorithm behind DeepSeek-R1.

What the data looks like

The data for SFT is pairs of inputs and outputs. The data for RL is quite different. To train the model with RL, you need a set of inputs (prompts), and a way to reward the model for what it generates from them. The key, unique insight is that you don’t need to know the solution — you only need to know what to ask, and how to score whatever comes back.

In this type of dataset, notice that while we do have the final answer in the row, we don’t have the actual solution. We just know the answer is 5, or 10, or 72. Let’s see how to design a reward for a dataset like that.

Reward with verifiers

The most basic example of reward modelling for mathematical prompts is just checking the final answer of the model and giving it a point — or points. The absolute value doesn’t really matter as long as it is consistent across the entire training run.

See one challenge with this scoring mechanism - the answer is either correct, or not. It’s only binary. A model can reason in the right direction, or perform actions in the right direction, make a mistake at the last step and it’s still a zero score. This is called sparse reward: it’s all-or-nothing.

You can make a sparse reward denser, by handing out partial credit for getting parts of it right. This is called reward shaping. These can be more narrow checks like following the right format you expect or calling tools that you would expect to be called.

During the exercises in the HuggingFace and DeepLearning.AI courses you also experiment with rewarding content indications that the model is going in the right direction. Say 0.1 points if the answer uses reasoning words like “therefore” or “hence”, 0.1 points if it contains actual mathematical operations. There is an inherent risk to that, called reward hacking, that we will dive into later in the post — and you can already see it in rollout R4 above. But it’s a good example to visualise the technique, and it did give positive results during the course exercises.

Before we go further, let’s align on some nomenclature that will come up from now on.

Training the model with RL

RL from human preferences is something that end-user-facing labs will do. My bet is that most of the RL training inside software companies will happen around verifiers instead, because there is far more value in agents performing actions and reasoning reliably in your environment than in adjusting their helpfulness or character. But it’s still interesting to know how the frontier and open source models get to where they are.

We look into that later, after walking through the training loop, because the loop is exactly the same for rewards from verifiers and rewards from human feedback. Since we just talked through how to design rewards from verifiers, let’s dive into the training.

What you need for RL training

(1) A reward signal. That’s either a verifier, which we just covered, or a Reward Model trained on human preferences, which we’ll build later in this post.

(2) A training dataset. You need input prompts that represent the behaviour you want to teach your model. If it’s about reasoning, these prompts need to be reasoning tasks — maths or coding — with answers that you can verify. If it’s agentic behaviour navigating your product’s tools, the prompts need to represent your users’ needs and expect those tools to be used — and here too you don’t need the solution, you need an end state you can check: the right record got created, the right tool got called with the right arguments, the tests pass. If it’s helpfulness, the prompts need to represent when you expect your model to be more helpful — but in this case, because you will use a Reward Model trained from human feedback, these prompts don’t come with expected answers at all.

RL input prompts are the most important thing to get right. You are always trying to teach your model something specific, and you want to make sure the prompts are specific enough.

(3) A training environment. If it’s just maths reasoning, you only need inference of the model itself. But if you want to teach your model to use tools or interact with a sandbox through the filesystem, you will need to provide a fresh, throwaway environment for every single rollout. As you’ll see in a moment, GRPO generates 8 rollouts from each prompt, which means 8 independent environments, and none of them can leak state into another. A lot of software engineering is hidden in creating reliable environments.

The training loop

The training loop is basically about going through your input prompts, and each time giving the model a fresh environment to operate in.

(1) Take a prompt from your training dataset and ask the model to generate N rollouts. This might be a one-shot output, or multi-step tool use navigating your environment with seeded tools, files, or data.

As I said earlier, I am writing about GRPO. The core concept is that you generate N rollouts from the same prompt. Typically that’s somewhere between 4 and 16 — it’s 8 by default in TRL, and 8 is what this post assumes throughout. I will explain in a second why it matters so much, but it already highlights that RL training is computationally expensive.

(2) Apply the reward to each rollout. If you are using a verifier, parse the output and score it — checking the final answer to a maths problem, running the code to see whether the tests pass, counting which tools have been called, or checking that the output has the structure you asked for. If you are using a Reward Model, run the output through it.

(3) Average the rewards across the group. This is your baseline, and it describes what this model roughly scores on this prompt, given the shape of its weights right now.

(4) For each rollout, subtract the baseline from its reward. This number is the advantage, and it answers a very specific question: compared to how this model usually does on this prompt, was this particular attempt better or worse? Positive advantage means this rollout was above the model’s own average. Negative means below it. We also divide by the standard deviation of the group, so that a prompt where all 8 rollouts scored between 0.42 and 0.48 produces advantages on a similar scale to a prompt where they scored between 0 and 1.

advantage = (reward - group_mean) / group_std

Why bother with a baseline at all? Because without it, we would only ever be pushing the model towards what it just did. If every reward is positive, every rollout gets reinforced — the good ones a bit more, the bad ones a bit less, but everything moves in the same direction. That still points roughly the right way on average, but it is an incredibly noisy way to get there. Subtracting the average is what lets us say “this one, more” and “that one, less” in the same update.

Notice that the absolute values of the reward has now disappeared. The only thing that stays consistently after this operation are gaps (differences) between these rewards. That’s why I said earlier that absolute rewards don’t matter as long as they’re consistent within the run.

This is finally the answer to why we generate N rollouts instead of one. We need something to compare against. PPO trains a whole separate model, a “critic”, whose only job is to predict what reward to expect for a given prompt — which means another model to train, tune and keep in memory. GRPO throws that away and does something much simpler: generate a group, use the group’s own average as the baseline. That is literally what “Group Relative” in Group Relative Policy Optimization means. You pay for it with 8× the generation, but you don’t need another model at all, which means less complexity and, most of all, less memory.

There is also a significant consequence here for what kind of prompts provide best value for buck. If all 8 rollouts get exactly the same reward — all of them solved the problem, or all of them failed — then the group average equals every individual reward, every advantage is zero, and this prompt teaches your model absolutely nothing. You only lost money on compute. That’s why a lot of researcher instinct is in what kind of prompts to include in RL training set. Ideally those that your model sometimes gets right, sometimes wrong, and you want to make sure that it gets those right ones more often.

This is also the strong reason to apply reward shaping. Notice that the reward is still one number per finished rollout — shaping doesn’t feed the model anything mid-generation, it just stops the score being all-or-nothing. What that buys you is spread inside the group. On a hard prompt where all 8 rollouts fail, a binary reward gives you eight zeros and zero gradient. A shaped reward separates them — this one at least got the format right, that one got three steps in — and suddenly the group has something to rank, so the prompt still teaches the model something instead of being thrown away.

(5) Now calculate the loss. In SFT we handed the model an expected answer and nudged its weights to make that answer more likely. We never let it generate. Here we do the opposite — we let the model generate, and then we nudge its weights to make what it generated more likely, or less likely, depending on the sign of the advantage for that specific rollout.

So we take each rollout, run it again through the model in a forward pass (exactly like SFT — the tokens are already fixed from the rollout, we just want to know how likely the model was to produce them), and get the log probability of each token it generated. Then we multiply by that rollout’s advantage:

loss_for_rollout_k = -advantage_k * (sum of log probs of its tokens) / number_of_tokens_k

That’s it. If the advantage is positive, minimising this loss pushes the probability of those exact tokens up — the model becomes more likely to produce that kind of output next time. If the advantage is negative, the sign flips and the same machinery pushes those tokens down. If the advantage is zero, the whole term is zero and nothing moves.

We divide by the number of tokens so that a long, rambling output doesn’t get a bigger weight update just for being long.

This is somewhat simplified form. Real GRPO implementations wrap it in something called clipping, which exists because in practice you reuse the same batch of rollouts for a few gradient steps, and the model doing the updating slowly drifts away from the model that generated them. I am skipping this here as this is not that important to build instinct on how RL works.

That formula is for a single rollout, and we generated 8 of them. Each one has its own advantage, and each one produces its own loss. To get the loss for this prompt, we average across the group:

         1    N
loss  =  ─ ·  Σ   loss_for_rollout_k
         N   k=1

where N = 8, the number of rollouts in the group

The group is not 8 training examples — it is one training example made of 8 attempts.

Which gives us the shortest possible summary of what RL actually is: it’s SFT on the model’s own output, where each sample is weighted by how much better or worse it was than the model’s average attempt.

(6) There’s one problem left. If you just run steps 1–5 over and over, the model will happily optimise for higher rewards while quietly losing everything it learned during SFT or previous RL runs — it drifts towards whatever scores well and away from being a decent model in general. This is the same catastrophic forgetting from the SFT post, arriving by a different route.

The standard fix is the KL penalty. You keep a frozen copy of the model you started from — the reference model — and add a term to the loss that charges the model for moving too far from what the reference would have said. A parameter called beta sets the price. Too tight and your model can’t learn anything new; too loose and it wanders off somewhere that scores beautifully and reads terribly. The nice property is that it’s cheap: no extra generation is involved, just a second forward pass over tokens you already have. Keep in mind that forward pass is different than generating rollout, and generally is pretty cheap compared with generating a rollout. The details of how that distance actually gets measured are more maths than this post needs, so I’m leaving them for you to dig into.

Worth flagging that this is one of the places where practice has moved: TRL now ships with beta=0.0 by default, on the grounds that for verifiable rewards over short runs the KL term costs memory and buys little. That is a defensible default for maths and code, and a much less defensible one when your reward is a model trained on human preferences.

Also worth noting that at this point you are holding the model you are training, a frozen reference copy of it, and possibly a reward model, all in memory at once — while also running generation. This is the real reason RL is so much more expensive than SFT.

Reward hacking — the thing that will actually go wrong

Let’s get back to our dense reward example from before, where we gave partial credit for using reasoning language in mathematical prompts. The inherent risk is that the model learns to repeat those reasoning words and collect the reward, instead of actually reasoning. That was rollout R4 in Figure 4, and it is not a contrived example — it is the first thing that happens.

This is reward hacking: the model optimises the thing you measured instead of the thing you meant, and it is very good at finding the gap between the two. It’s basically Goodhart’s law of Machine Learning.

Rewards from verifiers are the safest ground here, because “did the test suite pass” is hard to fake. There are far more variables in rewards from human feedback, and it can lead to models becoming sycophantic (remember GPT-4o?), repetitive, or too constrained.

The KL penalty from step (6) is a partial defence, in that it limits how far the model can go chasing the reward. The real defences are boring: look at the outputs, and evaluate with something other than the thing you trained on. Which brings us to the next section.

How do you know it worked?

In SFT you can mostly watch the loss go down. In RL, we track the mean reward over time. However, mean reward over time is an indication of the model getting better at getting our rewards, not necessarily of the model getting better for the users. Those are the same thing only if your reward function is perfect, and it isn’t.

So the eval setup needs:

• A held-out set of prompts, never trained on, scored the same way. It catches ordinary overfitting similarly to verfication set in SFT.
• An eval that doesn’t use your reward function at all. If you trained against a verifier, evaluate with a benchmark, an LLM judge, or humans reading outputs. This is the one that catches reward hacking.
• A general capability eval with nothing to do with your task, to catch the model getting worse at everything else while it gets better at yours.
• Actually reading the outputs. Most reward hacks are instantly visible to a human and completely invisible in a metric.

Reward from human feedback — training a reward model

Let’s say your goal is to make the model more helpful, or more enjoyable to use.

How do we score rollouts like these? There’s no answer to regex out, no test suite to run. And if you employed humans to score them live during training, that would be prohibitively expensive, slow and unreliable — a human would definitely have a different bar in the first 30 minutes of work than after 6 hours of reading AI outputs (um, what was score 4 again? what was 5?). All of this because, contrary to SFT, the reward has to be applied during training, on output the model has only just generated.

It turns out that LLMs make great reward models, and one LLM can read the generated answer from another and give it a score. The bet we’re making is that judging which of two answers is better is an easier task than writing the better one — so we train that judge first, and then use it to RL the model we actually care about. The previous section on reward hacking is largely about the ways this bet goes wrong.

Training a reward model is still based on actual human preferences, but this time those preferences can be collected asynchronously, ahead of time, which makes it far more tractable than trying to put a human in the training loop. This is the shape Christiano et al. proposed in 2017 and InstructGPT turned into the standard recipe.

Data preparation for a reward model

To train a reward model, you need:

(1) An LLM — often your SFT checkpoint, the same model you’re about to RL — with a scalar reward head instead of the usual language modelling head, so that instead of generating the next token, it returns a single number. However, I would expect that many companies figure out how to use smaller base models for reward models (like InstructGPT later mentioned).

What does that really mean? Again, I recommend the SFT post for some basics. I explained there that an LLM produces, for each token position, a vector of logits that can be softmaxed into a probability distribution over the entire vocabulary. So when we ask the model to generate text, it does that, we sample a token from that distribution, and repeat.

If you go one step earlier, it turns out these models are performing multiple layered matrix operations to produce what’s called a hidden state for each token, which is later transformed by the language modelling head into the logits I mentioned. The reward head does the exact same job with a different output size: it takes that same hidden state and turns it into one number instead of one per token in the vocabulary. Same body, different head. Everything the model already learned about reading text stays exactly where it was, and we only swap the thing sitting on top of it.

(2) A set of training examples. This is a different training dataset than the one you will use for RL. These consist of an input prompt, a set of different outputs, and a human-picked ranking of those outputs from best to worst. In the early days of ChatGPT you would have seen it generate two outputs and ask which one you preferred. Or you may have used something like LMArena, which shows you outputs from different models and asks you to pick a winner. All of this data can be used to train a reward model.

Before we look at the training loop, let’s talk about preparing the dataset. Say we have an input prompt and 4 outputs ranked by humans. From this data, we create all possible pairs — with 4 outputs, that’s 6 pairs. Each pair is just “this one was preferred, this one was not”.

Notice that we did not assign any score to any output. Nobody ever said that the best answer is worth 9 points and the worst one is worth 2. Our humans only gave us an ordering, so all we can extract from it is “A beats B”.

The question we are teaching the model to answer is: given these two answers to the same prompt, which one did the human prefer? That is a yes/no question — first or second — and that’s what makes this a classification problem. In essence, we will be training our base LLM to become a classifier of human preferences.

Training the reward model

The actual loop is fairly simple and similar to SFT, with one important twist. In SFT we always had a ground truth to compare against — the expected token. Here we don’t. There is no “correct reward” anywhere in our data, only the information that one output beat another. So the loop is built around pairs:

(1) Take one pair from your dataset — the preferred output and the dispreferred one, both answering the same prompt.

(2) Run both of them through the model in a forward pass. The model reads the whole prompt-plus-answer sequence, and the reward head takes the hidden state at the very last token — the point where the model has seen everything — and turns it into a single number. Two forward passes, two scalars. Call them r_preferred and r_dispreferred.

(3) Subtract them: r_preferred - r_dispreferred. This single number is what the entire training is about. If it’s positive, our model agrees with the human. If it’s negative, our model disagrees. If it’s zero, our model has no opinion at all.

(4) Now, some maths. How do we take that scalar and turn it into a loss? We apply a sigmoid to it. The sigmoid returns 0.5 at 0, and asymptotically approaches 0 and 1 at −∞ and +∞. So a difference of zero gives 0.5 — a coin flip, the model telling us it has no idea which one the human liked. Anything positive gets closer to 1; anything negative gets closer to 0.

What the sigmoid gives us is exactly this: the probability that our model would agree with the human on this pair.

(5) And now we calculate the loss the same way we did in SFT. We have a probability, and we want it to be as close to 1 as possible, so we take the negative log of it:

loss = -log( sigmoid(r_preferred - r_dispreferred) )

If the model agrees strongly with the human, the sigmoid is close to 1, the log is close to 0, and the loss is tiny — nothing to fix. If the model got the pair backwards, the sigmoid is close to 0, the negative log explodes, and the loss is huge. Exactly the same shape of feedback as in SFT, just measured on a comparison instead of on an expected token. This is called the Bradley-Terry loss.

(6) We go through backpropagation and gradient calculation, and (7) we update the weights of the model to minimise the loss using an optimizer. Both exactly as in SFT.

How much data do you actually need?

For the reward model

The published numbers span a wide range, depending on how much you care about the result.

The useful way to read that spread: a reward model for a narrow, well-defined preference is a much smaller project than a reward model for general helpfulness. If you only need “does this answer follow our support tone”, you are closer to the low end than the high end.

InstructGPT’s reward model was 6B parameters against a 175B policy (policy means the actual LLM being trained). And Llama 2 collected its annotations in weekly batches and retrained its reward models as the data arrived, across five successive RLHF rounds — because each new policy shifted the distribution of outputs away from what the previous reward model had been trained on. Reward models go stale, and they go stale because your training is working.

For the RL run itself

The number that matters isn’t prompts, it’s total generations, because that’s what you’re paying for.

Two open reference points to anchor against. AI2’s Tülu 3 — the most completely documented open post-training recipe there is — runs its RL-with-verifiable-rewards stage on 29,946 prompts: 7,473 from GSM8K, 7,500 from MATH, and 14,973 instruction-following prompts with programmatically checkable constraints bolted on. At the other end, DeepSeekMath, the paper GRPO comes from, ran GRPO over roughly 144,000 chain-of-thought maths questions, sampling 64 outputs per question.

That is a factor of five in prompts and a factor of forty in total generations between two runs that are both, in the scheme of things, narrow and verifiable. Which is the honest summary of this section: there is no rule of thumb here comparable to the SFT ladder. Thirty thousand well-chosen, correctly-difficult prompts beat three hundred thousand that the model either always gets right or never does.

And it is worth remembering that none of these stages happen in isolation. DeepSeek-R1’s pipeline alternates them — a small cold-start SFT, then RL, then ~600k reasoning samples collected by rejection-sampling from that RL checkpoint, mixed with ~200k non-reasoning samples, SFT on all 800k, and then RL again. The two posts describe two halves of one loop, and in a real recipe you go round it more than once.

Where to go next

If you want to actually run this, TRL’s GRPOTrainer is about as few lines as SFTTrainer was — you supply a dataset of prompts and a Python function that returns a number, and the group generation, the baseline, the advantages and the loss all happen inside trainer.train().

If you need the reward model half too, that is RewardTrainer. You hand it a preference dataset of chosen/rejected pairs, it swaps in the scalar reward head for you, and it applies exactly the Bradley-Terry loss from the section above.

Which, exactly as with SFT, is why it is worth knowing what they are. The code will not tell you that a group of eight identical rewards taught your model nothing, that your reward function is being gamed, or that your prompts are the wrong difficulty. Those are the failures that actually happen, and every one of them is invisible in the loss curve.

Everything I learned comes from two courses, and I would recommend both without reservation: HuggingFace’s LLM Course, which is free, genuinely excellent, and the best explanation of transformers and tokenizers I have found anywhere; and DeepLearning.AI’s Fine-tuning & RL for LLMs: Intro to Post-training, which expanded specifically on RL and more math and theory behind it. If any part of this post leaves you wanting more depth or try these in practice, those are where to go.

It is written for people with a solid technical background / math understanding but no particular machine learning one. There is some maths, but I have tried to keep it to the parts you can see on the peripherals of the models, rather than actual internals, like how the attention mechanism works, etc.

1. What a language model actually does

I am not going to dive into transformer internals. There is a lot of nuance in the architectures, and HuggingFace’s material covers it far better than I would. (Although I will probably write another post about that at some point!)

The short version is that a language model does one thing: it understands, predicts, and ultimately picks the next token.

Predicting the next token means producing a score for every single token in the model’s vocabulary - this is called distribution. The vocabulary is the fixed list of text chunks a model can read and emit — typically 50,000 to 250,000 of them, and they are fragments rather than whole words, so a real one holds entries like " the", " Berlin", " fine", "tun", "ing" and "!".

For computational reasons those scores are not probabilities — they are unbounded floats called logits. What the model emits at each position is a logit vector: one number per vocabulary entry, so a vector 150,000-odd numbers long. In code you will meet it with a batch and a sequence wrapped around it, as a tensor of shape [batch, sequence, vocab_size], but at any single position it is just that one long vector.

To actually generate text, you need to sample from that, and to sample you first need real probabilities. Softmax is the transformation that gets you there: it exponentiates every logit and then divides by the total. After softmax, the whole vocabulary sums to 1.

What you then do with that distribution is a separate choice. You can take the most probable token every time — greedy decoding — or you can genuinely draw from it, which is what temperature and top-p are shaping. Those are the knobs you meet on day one of any inference API.

Taking the most probable token has a name worth knowing, because it turns up in the figures below and in every piece of ML writing you will read: the argmax. Literally “the argument that maximises” — not the largest probability itself, but which token it belongs to. When a figure marks a row as argmax, it just means “this is the token the model would have picked, if you had asked it to generate here”

Here is the thing to carry forward: sampling is an inference-time activity. It plays no part in SFT training. We never need the model to pick anything. We only ever need the distribution — and in a moment you will see why that is enough.

2. What supervised fine-tuning is for

The goal of SFT is to teach the model format, conversational ability, and an awareness of who is saying what. Base models do not have this. They do not know that “user” means you and “assistant” means them. Frontier labs fix that by showing the model an enormous number of example conversations, and out the other side comes something that can hold a coherent chat.

The base model has read millions of dialogues; it already knows what a helpful answer looks like. What it lacks is the knowledge that it should be producing one right now, rather than continuing your text with three more plausible user questions. SFT helps with exactly this.

Keep this in mind when you get to the dataset sizes below. A thousand samples cannot teach a model to converse from scratch. It can absolutely tell it which of its existing behaviours to bring to the front.

Essentially every model you actually chat with has been through this. It is the standard path, and in practice it gets repeated many times, with different datasets, at different stages of a model’s life.

3. What an SFT dataset looks like

Conceptually a dataset is a list of input–output pairs. In practice you will almost never see it stored as raw concatenated strings. It is structured messages — a list of {"role": ..., "content": ...} objects — and the flattening into a single string happens at tokenization time, via the model’s chat template.

Chat template is the model-specific recipe for which special tokens wrap each role, and every model family has its own.

So where the data comes from? Broadly, two sources: (1) humans writing the answers — expensive, high quality, and that’s what initial GPT-3 was trained on; (2) distillation - where you generate answers with a stronger model and train your smaller one on them. Most open datasets you will find today are the second kind.

Using the right techniques, even small numbers of samples can have an observable impact. The rule of thumb I learned — though I have not personally gone above a few hundred — looks like this:

4. What actually happens during training

It is called supervised because we are directly teaching the model to increase the probability of specific tokens given specific inputs. “Here is the answer; be more likely to say it.”

Let us do this with an example small enough to hold in your head. Imagine a language model with a five-token vocabulary that is quite good at repeating exactly what the user said. We want to teach it something new: answer the user with their tokens in reverse order.

• Vocabulary: <user>, <asst>, tic, tac, <eos> — that last one means end of sequence, and tells generation to stop
• Input: <user> tic tac <asst>
• Expected output: tac tic <eos>

First we need to know how the model is doing on this sample. And here is the part that surprises people: we never let the model generate anything. We hand it the expected output and ask how likely it was to have produced it. How surprised it was is the loss.

Models predict one token at a time, but our training samples are multiple tokens long. So the loss is calculated for each token separately and then aggregated over the sample.

Because we do not need the model to generate, we can give it the entire concatenated sequence at once:

That is it. The whole thing, prompt and answer together, in one go. We then ask the model what it would have predicted at each position, based on tokens before it (differently said, conditioned only on the tokens before it). What comes back, for every position in the sequence, is a distribution over the vocabulary.

The distribution at position n is a prediction of the token at position n+1. So the distribution that ought to be putting its mass on our first output token tac does not live at tac — it lives one slot to the left, at <asst>. Every position predicts its successor.

On the left you see the logits, the actual output of the model. On the right, those logits after softmax. Note again that plain normalisation would also make things sum to 1; it is the exponentiation that makes the gaps meaningful.

The loss for a single token is -log(probability of the expected token). Higher probability, smaller loss. Lower probability, larger loss. That is the whole idea. It is called cross-entropy loss, and it is the name you will see in every training codebase.

If you ever want to translate a loss number into something intuitive, exponentiate it: perplexity = exp(mean loss). A loss of 2.3 is a perplexity of about 10, which reads as “the model was about as unsure as if it were picking uniformly between 10 options”. It is the same number your tooling reports, and a friendlier scale to eyeball.

Teacher forcing

We then move to the next token — but we do not let the model’s own pick from the previous step influence anything. We ask it to predict token n+1 at every position n, always assuming the previous tokens are the ground-truth ones from our sequence. This is called teacher forcing. The model cannot cheat by peeking ahead, because LLM has a causal mask baked into it: each position sees only itself and its predecessors (well, this is true only about decoder-type models, but that’s for different post).

The main reason for doing it this way is that it makes the whole thing parallel. All of this happens in a single forward pass. We do not run the model once per token; every position gets its prediction simultaneously, because every position’s input is already known up front. That is what makes training feasible on today’s compute. There is a secondary benefit too: if we fed the model its own picks, early-training mistakes would compound and drag the rest of the sequence somewhere unrelated to what we are trying to teach.

Masking

Not every position counts. We want to teach the model the answer, not to re-teach it the user’s question, so the losses on prompt tokens are masked out and thrown away. In code you will see them labelled -100, which is the value PyTorch’s cross-entropy ignores by default (assuming that you use HuggingFace Transformers library).

One thing that is worth calling out here is <eos>. It is a real token, it is the last one in our target, and it is trained exactly like the rest: the model learns to predict “stop here” from the tokens before it. If you don’t teach your models to generate end of sequence tokens, you end up with a model that answers your question correctly and then keeps going forever, inventing a new user turn and answering that too.

Adding it up

The loss for the whole sequence is the sum of all the surviving tokens’ losses.

So for each input–output sequence, we have a loss. Now we need to work out how each model weight — and there are billions of them, recently trillions — influenced the output, and therefore this loss. That is genuinely complex differential calculus, but let us take it on trust that it works. Using gradient backpropagation, we can evaluate whether any particular weight contributed positively or negatively to the loss. We want to minimise the loss, so that the model has the highest chance of producing the output we gave it, and backprop tells us whether to increase or decrease each weight.

Then the optimizer applies that adjustment. And we repeat.

5. What about reasoning?

One of the big discoveries of 2022 was that you can dramatically improve a model’s reasoning if you prompt it to think step by step before answering. That is really the core of what “reasoning models” are: models trained to think step by step so that you do not have to ask.

Training reasoning is a multi-step process, a mixture of SFT and reinforcement learning — more on RL another time. SFT is usually where it starts. You teach the model to wrap its thinking in <think>...</think> using samples like this:

6. Knobs worth turning

There are a handful of things you can tune, and they interact more than you would like.

• Batch size — how many samples you push through before touching the weights at all. You run the forward pass on all of them, aggregate their losses into a single number, and backpropagate once. That one gradient is therefore an average over many examples rather than a reaction to a single one, so an odd or badly written sample cannot yank the weights on its own — that averaging is the whole of why batching stabilises training. The cost is fewer updates per epoch: at batch size 8, a 1,000-sample dataset gives you 125 optimizer steps instead of 1,000. On a small dataset that can leave you without enough updates to teach the model anything at all.
• Loss normalisation — which is the question the batch immediately raises: aggregate how? You sum the per-token losses, but what you divide that sum by is a choice — the number of tokens in the batch, or the number of samples. Divide per token and a long answer contributes more gradient than a short one, so length quietly becomes weight. Most libraries default to per-token. It only bites if your dataset mixes one-line answers with long ones — but then it decides which of them your model is really learning from.
• Learning rate — how much we adjust the weights each step. Higher can mean faster learning, or oscillation. For SFT it is typically 1e-5 to 2e-5, roughly an order of magnitude below pretraining, because we are nudging an already-capable model rather than building one.
• Epochs — how many passes over the same data. Usually 1 to 3. Beyond that you start memorising.

It is also worth saying how you know it worked. Loss going down on a held-out set is necessary but not sufficient — a model can get better at reproducing your dataset while getting worse at everything else. You have to actually look at the outputs and ideally have a set of evals that proves that your model continues to work well in other domains that it previously worked well in.

That failure mode has a name: catastrophic forgetting, sometimes called the alignment tax. Train hard on a narrow dataset and the model gets great at your task while quietly losing capability everywhere else. It was never told to preserve anything, only to lower loss on what you showed it.

The standard mitigation is boring and effective: mix general-purpose data into your narrow dataset, so the training signal keeps pulling in both directions. The other half is evaluation. Keep a small set of prompts that have nothing to do with your task, and check them after every run.

7. Can you do this yourself?

Yes. The default path is just expensive.

To adjust every parameter, you need to hold three things in memory at once: the weights, the gradients for those weights, and the optimizer state.

That last one deserves a sentence, because it is where most of the memory goes. The optimizer is the piece from the end of section 4 — the thing that takes backprop’s verdict on each weight and decides how far to actually move it. In practice that optimizer is almost always Adam, and Adam does not just apply the raw gradient. For every single weight it keeps a running average of that weight’s recent gradients, and a running average of their squared magnitudes. It uses the two together to give each parameter its own effective step size, so weights with small or noisy gradients still move sensibly instead of being drowned out. It works very well. It also means two extra numbers stored per parameter, forever, plus a full-precision master copy of the weights themselves.

Add it all up and the rule of thumb is around 16 bytes per parameter.

The cheaper way is LoRA. You freeze the original weights entirely and train small adapter matrices alongside them, which ends up being well under 1% of the parameters. Nothing frozen needs a gradient or an optimizer slot, so the entire optimizer tower — the biggest block in that bar — simply disappears, and you are left holding little more than the weights themselves.

And honestly, this is not much code:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:2000]")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B",
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
    args=SFTConfig(
        output_dir="qwen-sft",
        num_train_epochs=2,
        learning_rate=1e-4,           # LoRA wants ~10x a full fine-tune
        per_device_train_batch_size=4,
        max_length=1024,
        completion_only_loss=True,    # mask the prompt, score only the answer
        bf16=True,
    ),
)

trainer.train()

Every concept in this post is somewhere in those twenty lines. SFTConfig picks the learning rate and the number of epochs. completion_only_loss is the mask. LoraConfig is the reason it fits on one GPU. The chat template comes along with the tokenizer, and the shift-by-one, the softmax, the negative log and the sum all happen inside trainer.train() without ever asking your permission.

Which is exactly why it is worth knowing what they are.

We had been through this once already. It’s painful to watch your plants die like that, and it’s also a waste of plants, money, and time.

So I decided to fix it properly. I finished a masters in control engineering, and over a decade ago I spent a lot of time with microcontrollers and electronics. I hadn’t had a good excuse to get back into it since, and this felt like the perfect one. What if I could build a small, controlled watering system that adjusts to the moisture of the soil and the weather outside?

The scope, and where I was wrong

My initial scope was ambitious: a wifi-enabled controller so I could adjust things remotely, the ability to water a few plants on the balcony, and moisture sensors driving electronic valves to regulate the flow of water.

The moisture sensors turned out to be very low quality and never gave reliable readings, so I dropped the idea, along with the solenoid valves, and replaced them with micro-drippers from a Gardena garden watering system.

A classic lesson to start from the simplest thing and add scope only if you prove initial value :)

No taps on Berlin balconies

The biggest constraint had nothing to do with electronics. In Berlin apartments you don’t get running water on the balcony, so the whole system has to carry its own water and generate its own pressure.

I solved it with a carboy — the kind used for home wine production, bought at a local construction market — and a 12V pump that pushes water from the carboy through the drip lines. Refillable, self-contained, and it sits happily in the corner of the balcony.

The components

The core build stayed small:

• MOSFET to switch the water pump from the ESP32 — eBay
• Schottky diode for the pump’s flyback spike — eBay
• Boost converter stepping 5V up to 12V to power the pump — eBay
• 12V water pump — eBay
• ESP32 starter kit as the controller — Amazon

The extended scope I ended up shelving:

• Moisture sensors — eBay
• Solenoid valves — eBay

And the Gardena-compatible micro-drip parts that replaced the valves:

• Pipe (4/7mm) — Amazon
• Drippers and holders — Amazon
• T-connectors — Amazon
• Carboy for home wine production, from the local construction market

The electrical diagram

The wiring is quite simple and I entirely built it on a prototyping board. A single USB-C 5V input feeds everything, and one MOSFET does all the switching. I used Claude to remind me all the nuances of how to keep the MOSFET switching safe.

USB-C 5V powers both the ESP32 and a boost converter that steps 5V up to an adjustable 12V. The ESP32’s GPIO drives the gate of an N-MOSFET with PWM.

The MOSFET low-side switches the 12V pump: pump positive to +12V, pump negative to the drain, source to ground. A 10kΩ resistor pulls the gate to ground, holding the FET off during ESP32 boot and reset.

A Schottky flyback diode clamps the pump’s inductive turn-off spike, cathode to +12V and anode to the drain. Three rails run through it all: +5V from the USB/boost input, +12V from the boost output to the pump, and a common ground tying the USB-C, ESP32, boost, MOSFET source, and diode return together.

The code

The firmware lives on GitHub and was generated in 2025, mainly with the help of GitHub Copilot back then: github.com/jniechcial/hydration-station.

The control plane

I didn’t want to depend on the cloud for something this small, so the ESP32 serves a simple website on my local network. Everything I need to run the system — trigger a watering, set a schedule, check the state — is a page on my LAN, no app and no external service.

Lessons, and what I’d do next

It’s a very reliable basic watering system, but it still needs a bit of manual oversight for the best outcome, especially during extremely hot spells. That said, it did keep the flowers alive during 2 weeks we were away on holiday, with a neighbour just topping up the water tank twice in that time.

The main limitation is that the watering is uniform, but the plants are not. You can tune each dripper, but what you really want is per-plant logic: water this one once a day, that one twice. Leave the system with a lot of active time to keep the thirsty pots happy, and the smaller pots quietly get over-watered.

I’d love to get back to the valves-and-sensors idea. If I could find moisture sensors I actually trust, there’s still the visual problem of routing meters and meters of individual power and control cables around the balcony without it looking like a bomb.

Moving everything off the prototype board onto a proper custom PCB, with a 3D-printed enclosure, would be the satisfying finish. Maybe at some point, if I find the time!

What did I do at HubSpot?

I led engineering in the Flywheel Product Line, and our mission was to reinvent our own GTM motions with AI. The opportunity is obviously huge with over 4000 people in GTM roles, and SMB-centric, 300k-strong customer base.

This mission only solidified around mid-to-late 2023, after the ChatGPT moment, and the opportunity and shape of the org shifted from a more traditional business-systems team to a product-led innovation team within HubSpot. It was a truly creative and innovative time where we tried tens of ideas, rebuilt products many times as models got stronger, and learned a ton about both GTM and AI more broadly.

We were early with our ideas. We prototyped and productionized internally plenty of things you now see live in HubSpot, Salesforce, Gong or many other GTM startups. Things like call summaries with structured data extraction against sales methodology, smart deal progression, deal coaching and best-next-actions, demo environment generation, customer agents, and agentic enrichment systems. Many, many more.

We worked hand-in-hand with our RevOps folks, business strategists, and business analysts. They bring exceptionally deep domain expertise, and when paired with a strong, high-agency product team, you can deploy AI effectively.

But it wasn’t an easy ride from the start. When I started in 2024, the org was in need of a strong turnaround. The mission was too big for the shape of the talent, processes, and principles the team had at the time. I had to turn that org around, elevate our best folks, and help us push through the high-density learning ahead of us.

Turning around the org of 20+ teams

The first challenge was the sheer size of the org. We were over 110 engineers, over 20 teams, and 10+ Director+ stakeholders on the GTM side.

The first step is correctly identifying problems, based on a triangulated view from teams on the ground. I believe that in a turnaround scenario, when you join, the classic listening tour only works at a small scale of a few teams. With this much diversity in engineers and levels, and a lack of trust on both sides, you need to get close to the work and be willing to step down a few levels. You just shouldn’t blindly trust anything, stay sceptical and validate and understand very closely.

I spent over 3 months working most of my day as an engineer, directly with 4 teams (and some adjacent teams from time to time), in different parts of the org. The embed gave me a rare opportunity to build trust and deeply triangulate the problems and opportunities. I could hear them, probe them with other folks in the team or stakeholders, and validate them myself as someone who is part of the day-to-day team. If I ever join an engineering org for turnaround purposes, I would definitely repeat that.

After that, my direct team of senior managers and my product partners built our first pass at a new mission, vision, and strategy. This turnaround was pretty big (from order-taker to innovation hub), and landing a change like this is all about messaging. This takes quarters to land, and one of my team members gave me a rule of thumb I still use: if you feel like you’re communicating just right and it’s not uncomfortable for you, you’re not communicating enough.

During my embed, I spent a lot of time identifying high-performers, validating that with my management team, and later elevating them and strengthening their ownership and agency. You need folks who will champion the new world at every level, and I think it’s critical to connect a few levels down in your org to make sure the message lands exactly right, without anything lost in translation. Folks look up to other successful folks, and peer pressure, peer coaching and peer role modeling are one of the highest leverage options.

Last, speed needs to be uncomfortable, even for you and your most trusted people. I learned that after Eoghan McCabe came back to Intercom in 2022 as CEO and really shook us deeply in how fast you can move. It stayed with me as a beacon of how fast you need to swing the pendulum to see the change. In HubSpot, everyone told me I was moving too fast. Many tenured people told me that rolling out change this way wasn’t the HubSpot way. This might have been true in the past, but in turnaround you need to swing hard.

Reinventing with AI means flip-flopping between a blank check and scrutiny of impact

If you’re following X right now (July 2026), you’ve definitely seen the growing concern about token cost, ROI, and budgeting. None of this existed just four months ago. We’ve seen this cycle repeat multiple times internally since I started (not about costs though, but about impact). Generally, I’ve seen broad conviction on AI across execs and senior leadership, but at some point expectations shift and you move abruptly from “that’s awesome, can you also do X” to “okay, but where’s the impact.”

There is so much cool stuff you can build with AI. It’s very easy to spend weeks to months building exactly the wrong thing. We spent a lot of time breaking down the customer journey — what are reps’ actual jobs-to-be-done, what are they doing day to day, and which of those tasks are both time-consuming and feasible for AI, at its current state, to deliver with great quality.

From there, we found ways to measure performance on these jobs — leading metrics, often output metrics. Where possible, we showed the connection to outcome metrics, and in a few places even built causal models that proved causality.

One interesting observation: incentives and the bar need to move up as you build conviction that your product is making people more efficient. That’s what actually forces a behavior shift and lets you capitalize on the time you’ve earned back. Without that, your beta testers will usually show outsized impact, and as you roll out from a population of 150 people to 1,500, you’ll see lower impact numbers because some percentage of people just take it easier.

Introducing AI to your GTM team has to compound over time

Throwing in tools without a clear vision for how they compound isn’t helpful. It makes your data murky and makes it hard to compound value. We rolled out three huge tools at a similar time, in an uncoordinated way - Lovable, Glean, and Claude. Within months we lost trust in our data, saw big shifts in rep behavior, and watched people drift away from proven tools toward more exploratory builder tools. That was a mistake I wish we avoided.

Instead, choose a small set of tools that you’re going to manage, enable, and iterate on. They need to be highly extensible, continuously invested in, and built to enable experimentation. Claude Code/Cowork is a good example, and I can totally see why they are winning the market share that fast. They are just an awesome compounding platform.

On the other hand, this also means that there is often a potential compounding effect on your impact metrics. Many of our bets quickly showed good indicative metrics, but still took almost a year to play out with visible, double-digit impact on the entire rep population.

Lessons from building products translate to building internal AI for your GTM

My product partner and I came from Twilio and Intercom respectively, where we spent our time building core products. We brought some of that approach to internal GTM.

We anchored on user research and deep understanding of problems and user jobs. We always started with a small alpha group that had a mandate to be early adopters and partner with us. We sometimes had to give them a small quota relief to ensure they had time.

Later, once we had feedback that the solution was proving valuable and covered most edge cases, we’d expand to a larger beta group (no mandate, no relief). This gave us high-confidence numbers and let us compare beta-group performance to non-beta-group performance to prove impact.

After having that data, we would roll out, enable, and sometimes even force usage after alignment with rep leadership.

The most important lesson here: if you move too fast through these stages, you lose the opportunity to prove impact, and you’ll most likely ship an underwhelming product to a broad audience, losing their interest and burning the bridge early. While distribution is easier for internal products than for commercial ones, you still can’t cheat real value or user perception.

AI adoption is led by leadership and managers

I think every company has already realized that to push AI adoption, you need to push from the top, keep people accountable, and role-model the behavior.

We saw great acceleration when our sales leadership set a quota for AI adoption across their teams. This forces the org to be more open-minded and active in product development - providing feedback and giving you more quantitative insight.

To make that work, though, your managers need to see the value in AI first. They need to use it. That’s what gets them talking about it, role-modeling it, and getting their team to follow. One of our products have seen very fast adoption among sales reps, mostly because we optimized it and enabled for first for managers. They loved it, and used it on their pipeline reviews with their teams and created great word of mouth marketing.

RevOps-as-code

We haven’t gotten there yet fully, but I believe Operations will go through a shift similar to DevOps and infra-as-code.

Everything will become defined in code, instead of visual workflows and a patchwork of tools stitched together. Code is testable, observable, reversible, and Claude can navigate it and reason through second- and third-order consequences in ways that visual workflows, lists, and recipes just can’t.

We’ve seen this play out many times: workflows stitched together through tools become safer to change, more reliable, and better tested and understood once they move to code. And non-engineers can easily suggest changes by working with Claude Code, opening PRs, and — after review by engineers — getting them safely into production.

I don’t think there exists a good platform yet to actually do that at scale, and with x-company transferable knowledge (what terraform was). If this prediction comes true, I would bet we will see inspiration in tools like Apache Airflow, and I wouldn’t be surprised if Workato goes this direction, or AWS comes up with a more generic solution.

The ML, Data Science, and Engineering roles are getting closer together

Especially if you are building AI products, it will be hard to say when traditional software engineering ends and machine learning starts, and vice-versa. You just can’t build successful products without evaluations, creating benchmark datasets, experimentation. Not even going further into fine tuning or your own RL.

You still definitely need experts to set direction and make a handful of significant, multi-quarter decisions. But everything in between seems to boil down to great engineering talent that has good fundamentals, can stretch across domains and has access to frontier model with sizable budget.

There’s nuance in speed, though. A product engineer building a classification engine with good recall and precision is totally feasible, but it’ll probably take a few weeks and some oversight from an expert. An expert would probably do it in days. But as the muscle gets stronger, it gets faster.

I think the solution will more and more frequently be one team - MLEs and data scientists working hand in hand with product engineers, with frequent tours of duty with software engineers building up their MLE chops by leading the work under supervision.

Everyone needs platform-wide AI primitives that accelerate the entire company

Our DevEx team at HubSpot did an exceptional job building critical AI primitives that power core HubSpot features and are highly extensible. We were lucky that HubSpot’s business also relies on those so they received a big amount of investment that we didn’t need to pay for, but there are great open-source options out there too.

First, our principle was to expose everything as MCP endpoints. All internal systems should be MCP-first. That lets us integrate with Claude, HubSpot Breeze, or whatever other tool pops up in the coming months.

Second, you need an agent harness that’s highly extensible. We built our custom, but Agno was also used in a few places at HubSpot. I didn’t play around with Claude Managed Agents, but I would be vary of locking yourself down to just Anthropic models.

You also need a knowledge base that’s accessible programmatically via RAG, text search, and with metadata filters. We had to build something specifically for our use-case, but there are plenty of good products out there.

I strongly believe that you also need storage for the AI insights and signals your data scientists and AI engineers generate. Discoverability is key and we quickly realised that there is huge amount of insight that we generate, don’t scale, and duplicate. That’s why we built our own platform, that mapped entities 1:1 with our HubSpot instance. It acted as an extension of a CRM, but for unstructured data and agent-first discoverability.

And last but not least, you need an evals system. We had some custom built solutions but really struggled to take it off at scale of many teams. Later on, we migrated to Braintrust and its ease of use let us dramatically ramp up eval coverage and get both engineers and PMs close to agent traces. </content>

The reality is that writing software is becoming cheaper and faster at an incredible rate, but the systems we’ve built around that production are lagging. Our expectations of who does what - between the engineer, tech lead, PM, or designer - remain rooted in an older era. Our product development cycles, release processes, customer access protocols, and even interview loops were all battle-tested for a world where building a feature took a month. They are not built for a world where that same feature takes a couple of days.

The Innovator’s Dilemma in engineering

We must embrace this change through the lens of the Innovator’s Dilemma. We see AI startups and frontier labs moving with exceptional speed. If you are an incumbent, you need to disrupt your own delivery model. This is about far more than just giving your team Claude Code; it’s about rethinking the entire system: tools, processes, performance expectations, and roles. Engineers are smart and they don’t stay idle. Currently, we see more code being produced everywhere, but too often that code is channeled into areas most comfortable for software engineers: tech debt, low-priority to-dos, minor refactoring or unambiguous small features. While valuable, that isn’t where the real business impact lies.

It is possible to have incredibly capable models, 6 OpenCode windows open all day, 12 ralph loops running 24/7, make 600 commits and close 400 linear issues a day, and yet ship absolutely nothing of value to anyone. pic.twitter.com/oRG4N1L6i2

— Nate Berkopec (@nateberkopec) January 10, 2026

A useful mental model of you can expand your software as a company gives you three dimensions:

• Horizontal expansion: Moving into new product lines. However, doing that successfully in a way that contributes to the business growth is constrained by your GTM engine and distribution. It’s pretty ambiguous both from product and GTM perspective and no “pure” software engineer could drive anything like this.
• Vertical deepening: Making existing features more powerful. However, this dimension is constrained by your user ability to absorb change and adopt new software. It’s pretty ambiguous from a product perspective and usually only very senior software engineers would autonomously drive changes like this.
• Surface quality improvements: Small fixes, product improvements, performance, COGS and refactors. This area is usually pretty safe, predictable and unambiguous. That’s what most engineers usually focus on.

The first two buckets are today constrained by PM and UX bottleneck. Therefore, most AI-driven acceleration in writing software ends up in the third bucket. The real challenge for leaders is shifting those gains into the first and second ones and it’s very hard to do this without evolving the entire system of product delivery.

The new talent bar

To address this, I am explicitly operationalizing three dimensions of talent within my organization at HubSpot through performance expectations and growth plans. To remain competitive, software engineers need to embrace these three pillars:

1. You need to be a Product Engineer

AI is an extension of your agency in delivering business value. Historically, there has been a “fat-head” distribution of impact, where certain, usually very senior, engineers had outsized influence because they could identify and disambiguate high-value problems.

I think that in the coming year we are going to see massive compression of expected behaviours among software engineering levels. What was once a Staff-level behavior - taking ownership of the “surround system” of PMs and designers - is becoming the baseline for mid-level engineers. What will still be obviously different is the expected level of impact - you would expect significantly more differentiation and value from staff than from mid-level engineer. But if an engineer can ship 50-70% more code today, but the PM and design “surround system” isn’t ready for that velocity, they become the bottleneck.

Software is no longer the bottleneck.

— martin_casado (@martin_casado) January 4, 2026

To bridge this, engineers must become subject matter experts in the products they build. You must understand the customer and the business model. You need to be opinionated, research and develop some bets that you want to explore proactively, be able to ask the right questions and find answers yourself, and even think about and act on things like activation, enablement or support.

Being a Product Engineer is about taking ownership of the surrounding system of the past. You need to become a subject matter expert in products you are developing, understand your customers and how you ultimately make money as a business, and be able to identify, and take ownership of actual business outcomes. Performance will increasingly be measured by business impact rather than output metrics like PR counts.

If you would like to learn more about that, some great posts about this:

• https://blog.pragmaticengineer.com/the-product-minded-engineer/
• https://leerob.com/product-engineers
• https://x.com/levie/status/2010055953157357622

2. You need to be fullstack

While specialization will always exist, for example in infrastructure, my bet is that the total number of “siloed” engineers will decline. If your value is tied to your ability to deliver business outcomes, you cannot be limited by the stack; business value rarely stops at the boundary of a frontend or backend change. With AI-assisted coding, there is no longer an excuse to specialize in a single language. You must be an engineer who understands the entire system - from browser behavior to distributed system design. You should still cultivate “T-shaped” depth in one or two areas, but you must feel comfortable navigating the entire stack with AI as your navigator.

A timely read on product engineering: and why the frontend/backend split just doesn’t make much sense any more (at startups, the very least, and at nimble teams and companies) - by @leerob https://t.co/PLWzrei4rJ pic.twitter.com/4014jBUmCI

— Gergely Orosz (@GergelyOrosz) December 31, 2025

3. AI is a key part of the stack

AI used to be the exclusive domain of MLEs. Today, it is being democratized. However, the best products are more than just thin wrappers around an LLM; they are complex systems involving RAG, online/offline evaluations, experimentation pipelines, and context management.

These principles are becoming as fundamental as distributed systems design. Just as a backend expert needs to feel comfortable with the frontend, they must now feel comfortable operating within AI systems. AI is no longer a “plugin”; it is a core component of the modern stack.

It’s not your decision. It’s a market decision.

If you don’t lean in now, you risk waking up one day to find you are no longer competitive. This shift toward high-agency, commercially-minded, fullstack engineers is already the “bread and butter” of startups.

With future advances in AI, engineers with this talent stack will be unstoppable

-strong eng skills
-good (top 10%) product sense
-decent (top 25%) commercial sense
-high agency
-clear communication

They’re valuable today, but with AI they won’t need large teams nor any PM help

— Shreyas Doshi (@shreyas) January 16, 2024

For incumbents, the clock is ticking. You cannot catch up to the speed of a hungry startup just by buying tools. Their advantage comes from the entire system of delivery and the shape of the talent they attract. As an engineer, you must grow into these categories. As a leader, you must operationalize them. The tools are available to everyone — the system is what will set you apart.

There is way more that came to my mind that I decided to not put online at this stage. After 6 years with intercom, I am not sure how much I take for granted at this stage vs. what is a genuine good lesson to share. I am curious to hear how these things resonate with you and especially if you have counter lessons!

Leadership

1. Always start from the problem and ensure alignment with your stakeholders that this is the right framing, scope and priority of the problem.

Hands down, starting from the problem is the most powerful, life-changing lesson from Intercom. It applies to everything and provides an almost magical amount of focus in everything you do. What is the problem really? How do you know it? How will you know it’s solved? Does this sound worth spending time on? Applies to product, processes, org shape, home decoration, car repair, anything. It provides clarity like nothing else.

2. Mission, vision, ownership and strategy are the most impactful tools to achieve high performing orgs.

I wrote more about it here in a slightly different context, but with a similar conclusion - work with orgs on mission, vision, ensure they have and feel real ownership to create their own strategy and then keep them accountable for progress. Autonomy and sense of identity drive exceptional engagement and motivation which is critical whenever times get hard (and they will!). It trumps tactical goals setting or even worse, zooming in as a leader to micromanage, despite these tools providing faster but limited return.

3. When choosing long-term strategy - conviction, resiliency and communication are critical.

We created and executed a 2-year long product and technical strategy that shipped a new outbound product for Intercom and migrated all existing customers into it from legacy systems. We doubted ourselves plenty of times. It’s critical to maintain conviction, resiliency and ways of ensuring you are on the right track, according to schedule. Underpromise, overdeliver, keep shipping, stay focused. At the same time, you need to keep your stakeholders and supporters informed and close enough that they don’t get misaligned on timelines, progress or complexity of the work you are going through.

4. Quick and dirty process accelerators are great to test locally and scale globally when proven to work.

When I was EM in one of my teams, we had over 200 open issues. I couldn’t wrap my head around that and either understand how our state changes or what’s most important to work on. I came up with a script that scored each issue based on how many customers were affected . This quick and dirty prioritisation system got huge traction at Intercom and became our default and org-wide tool with its own Tableau dashboards. Similarly, when I was tasked to create an escalation process for our sales team, we ran a local experimental version with one product group and simple Coda doc before making it a global default with its own tooling. Test processes locally and make sure they solve a problem.

5. You can achieve spectacular outcomes when focusing on one thing, but with real trade-offs.

The last year at Intercom (2023) was a year of impatience and moving fast under the new CEO. Moving fast was the most important thing across the company. We did spectacular things, we made aggressive and ambitious decisions, we moved the fastest I saw Intercom move, despite being 3x bigger than when I first joined. This was the right thing to do in the wake of AI even though it came with real trade-offs on our product health that we had to pay down year later.

People & management

1. Fair, objective and transparent are the most important principles in compensation and performance evaluation.

I deeply believe in these three principles when evaluating performance or deciding on compensation incentives or raises. There are plenty of processes that help you achieve them - transparent and written expectations across levels, calibration and promotion meetings, obligatory peer feedback, compensation bands or compensation policies.

2. Find the right-shaped opportunities to help people thrive and achieve impact.

Sometimes folks don’t perform and it’s important to ask yourself if there is a better place in the organisation that would benefit from their skillset, instead of managing them out from their current position or coaching them to grow where they are. I found a few folks who performed amazingly after we found the right-shaped opportunities for them. But don’t let under-performance slip when there are no high-confidence opportunities in sight. You can’t manage underperformance by moving people around.

3. There is always a reason why someone is underperforming, learn that before you act.

Empathy and trust will let you understand why someone is underperforming. As a manager, my job is to get the best out of my team and it’s critical to understand the problem - why someone might be underperforming? Stress at home, financial problems, personal conflict or health problems can happen behind the scenes without your reports notifying them about them upfront. Dig and understand what’s the reason to find the right shaped way forward before you act.

4. Building managerial support networks in your team.

Management is a lonely job, your managers should rely on each other. I wrote more about it here.

5. Routines can help you avoid burn-out and keep you on track.

Planning, reflections, operation data reviews - it all creates a personal operating system of routines that provide time and activities that help you get perspective. It’s self-management practice that helps me to not burn out and stay on top of what’s going on. It helps me to identify and know when to take a break. Dropping my routines is usually the first sign that I am going sideways and need to take care of myself.

Industry and customers trends

1. Support is a slow moving industry. It takes a lot of energy and work to see the change.

Support orgs on average have high inertia. Change management and switching costs are expensive, and if it’s not showing clear ROI, it’s very tough to drive any change in tools or processes. It works, but it’s tougher than it sounds. Adoption is a big challenge.

2. Support has a chance to generate money, not only be a cost centre.

There is a huge opportunity ahead of Customer Support orgs to generate revenue and not only be a cost centre. AI is the tool to enable it. Remove the friction by engaging with AI bots and create exceptional co-pilot experiences that help agents drive value, not just manage questions.

3. Sales and marketing are moving very fast. Experiment and expect immediate results.

Sales and marketing have easier adoption curves, but way harder to retain. Showing ROI and before/after data is critical. If you can’t deliver, the tool or the process is out. I ran a rollout of an embedded iPaaS solution for our high-spend customers and failed. With reflection, we have set the revenue bar too high to show ROI quickly and never earned sales team trust that this can work. Go for quick, visible ROI.

4. Omnichannel is great, but email is still considered the primary channel for many.

During my time in Outbound product at Intercom, it was clear that customers wanted email. They loved our omnichannel. Especially in-apps with their impressive engagement rates. But new things do not always replace old, and often rather expand the landscape. Marketers need reliable, powerful email tools to be your primary tool.

5. In-app is the best channel for engagement, support and x-/re-/up-selling.

I loved to see some exceptionally well designed proactive support, consisting of product tours, banners, tooltips and timely send in-app messages. It makes the experience of onboarding magical. I loved that we could message tens of our customers and ask them for feedback, and they would reply in less than 24h, with additional thoughts. It’s magical both ways.

6. Partners need well designed incentives and reporting to drive action, and it’s harder than sales because it’s based on trust and relationship, not a work agreement.

Intercom had to deprioritise our investment in App Partnerships and thus we weren’t able to build valuable long-term relationships with our key developers. When we ultimately had some ideas on how they could help us and our common customers, it was tough to get traction. Partnerships are amazing but need to be nurtured, based on the right incentives and give insight into the right reporting to drive behaviours.

7. Data management is evolving, painful, and it’s tough to find a silver bullet.

We owned a relatively flexible, broadly used data platform. It was the area of product with the biggest amount of feedback. But making an impact in this space was very tough. The market is evolving fast, with new approaches like ETL, reverse-ETLs, point-integrations, CDPs. Your customer base will be spread over multiple different “best ways of managing your data”. And because of that, there was never a single problem. It’s always been long lists of problems, similar in some shapes but different. To see meaningful change and improvement in data management, it’s all about strategy and consistency. Small investment shots here and there rarely work.

8. Data integrations are getting more complex the more you dive into them.

I learned to never underestimate x-system integrations. From the distance, they always sound relatively easy - if this then that, sync objects in system A with the same objects in system B. But the closer you get there, the more differences you discover. Small nuances, API rate limits, race conditions, retries and lack of idempotence, and many more angles. The devil is always in the details, and the jobs your customers are trying to achieve with these integrations are often similar but fundamentally unique. Integrations are super complex the deeper you get and don’t underestimate it.

9. Building easy to adopt integrations is removing friction, but reduces the TAM.

The most easy to use integrations are layers of abstractions on top of core capabilities (like APIs or iPaaS blocks). They remove a lot of friction, usually working out-of-the-box. What I realised over the years is that building broadly adoptable integrations is very tough as every company has unique needs, setup and their own internal IT mess to navigate. Building these abstractions makes it easier to adopt, but significantly reduces the TAM, which can end up with a lower number of customers using them versus a more complex but powerful version.

10. AI will fundamentally change how we build integrations.

However, I believe it will play out differently for deterministic, high-volume integrations and non-predictable, dynamic integrations. Co-pilot experiences will dramatically accelerate users in iPaaS tools like Zapier or Workato while creating repeatable, high-volume, predictable process automation. This will accelerate them, while maintaining today’s reliability and ultimately deterministic behaviour of these integrations.

AI agents that can reason about which tools to use and adapt to dynamic and unpredictable input will remove a need for building any integrations (outside of API capabilities) in human-triggered activities like reporting, data exploration or asking for help.

Product & Technology

1. Betting only on the future is like shorting the present and it carries opportunity cost.

Intercom is an extremely innovative company and our senior leadership is excellent in having great insight into what trends will make waves in the market. But betting only on that, building only for the future, is like shorting the present and thus carries the opportunity cost. I think we lived that both with emails and phone, being confident for years that these channels were dying just to realise year by year that they were not shrinking at all. I find it a useful mental model when you think about the future trends - would you short present and be willing to pay the premium if you continue to be wrong?

2. Every developer needs to have AI in their toolbox.

When we worked on Fin, our AI team was fundamental to making it work, we wouldn’t be able to do it as a product team. However, as time goes by, more and more excellent features have been built into that system directly by our product teams. The initial push might need specialisation, but it’s important to expose AI to your org as soon as possible. Another engineer shared with me this anecdote - 2008 companies had whole teams that did “mobile design”. Only a few years later it was part of everyone’s job.

3. Monoliths can scale very well, give solid deployment safety and high leverage for developer experience and observability.

I continue to be amazed how well Intercom Rails monolith scaled over the years. Exceptional engineering decisions, keeping it simple, huge leverage that our only developer experience team has, and sticking to proven cloud technologies helped us navigate growth really well. Monolith is also something that keeps our ability to ship very fast, very often (by sweating things like fast rollbacks or quick CI). We found a few services we owned painful to work with in comparison - teams had to be slowed down by maintaining their own dependencies, deployment pipelines or infrastructure updates. I never worked in professional, high-scale microservices architecture but can’t wait to learn about the trade-offs there.

4. How often you ship is critical. It’s your heartbeat.

You might not ship it to customers, but there is always a way to ship to production in a safe way. If you can’t find it, keep looking. It builds muscles for accelerating even more when necessary, and keeps you from losing impatience culturally. Show off the progress, ideally on a regular company-wide demo.

5. Create plenty of system models, mental models, apply different layers of abstractions. Looking at the same problem from different angles really expands your horizons and builds alignment.

At Intercom I learned that the best meeting starts when someone picks up the marker. System models, made up on the spot, adapted, evolved as you talk - all of this really help sharpen your understanding of the problem, its complexities and dependencies. Write them down, show them in your next conversation, ask for someone else’s mental model. Doing this work together with your partners and stakeholders accelerates how fast you collaborate and removes misalignment. Shared mental models are very useful for moving fast.

6. Own and know your data, without excuses.

I had plenty of excuses in the past for my understanding of product data without the help of data analyst to craft queries and analysis techniques. I have no mercy for myself since ChatGPT. The quality and confidence in which I can navigate my product space without a dedicated analyst has increased dramatically and I expect that everyone at least slightly technical can do it now.

I also want to say big thank you to everyone I worked with and that helped me form these lessons over the years. Intercom is a truly special company!

Last but not least, thank you to my two long-term partners in crime that provide me with great review and feedback on that blog post, Stephen Forbes and Aidan Lynch!

Addressing such complex issues without being hands-on or without deep comprehension of the problems can feel overwhelming.

Here is the strategy my leadership team and I adopted to improve our situation. This approach incorporated cross-functional collaboration among product, design, and engineering teams.

Commit and understand constraints

First, it’s crucial to commit to the turnaround alongside your leadership team. This is especially significant for those who have been part of the setup for an extended period, who may feel inertia or hopelessness. Set a time-bound, feasible goal to instigate a positive shift—something like, “We will establish a positive trajectory for the organization with a long-term improvement plan within the next quarter.”

Second, align with your leadership team on what are your constraints. What cannot be changed? There are three main categories that come to mind:

• Headcount: Is it possible to bring in new team members to alleviate pressure or inject a positive spirit, or is your headcount fixed?
• Priorities: What commitments must your group deliver? Ensure that you stay focused here—nothing demoralizes a team more than missing an important commitment.
• Performance: How are the members of your leadership team performing? Know who you can depend on and who needs closer management, or potentially, who needs to transition out.

Understand your problems

Listening to diverse voices within your teams and assessing problems before committing to priorities is vital. Don’t get swayed by the most vocal issues; they might not necessarily be the most crucial to solve. To understand and triangulate problems, we used multiple sources:

• Work with your management team - they will most probably have great understanding of problems, but it’s important to validate and triangulate further.
• Engagement Survey - for the purpose of moving fast, we used ChatGPT to quickly come up with a good first pass of questions.
• Skip-levels - as director of engineering, I met with all 20 engineers in the group.

Conduct your first round of prioritization with your management team. It’s impractical to involve literally every engineer in such a working sessions.

Share back to the teams

Share everything you’ve learned with your teams. Create space for them to offer additional context or challenge your framing of the problems. For us, this involved one and a half hour sessions of unstructured conversation with each team. We also shared a prioritized list of problems, inviting further feedback in a less intimidating setup.

Get alignment with your senior leadership.

Ensure that you have the support of senior leadership and their understanding of the situation’s gravity.

This is especially important if one of your problems is working under too much pressure. To alleviate it and create slack you need to be ruthless with priorities and only focus on what’s the most important, which is very tough without senior leadership alignment.

Move fast on what’s obvious

After problem discovery exercise, you might see some high-priority ones that are obvious to solve, just no-one before pulled the trigger on the solution. I suggest to fix such problems immediately and communicate that you will learn and adapt, but right now optimising for building momentum.

For us, some things that we did immediately were better balancing the time spent on net new product vs. issues, planning for a bit of slack for escalations from our customers that were consistently happening and kicking off work on product and technical strategy, as we knew it will take time to develop those.

Recharge everyone’s batteries

To drive change, everyone’s energy and optimism are essential. I used my tried-and-tested strategy to manage team burnout following an incident.

We sent everyone on a brief break, not counted against their holiday allocation. This forced recharge helped maintain a positive outlook.

Create a vision of an org that everyone wants to be part of

Creating a visual description of your goals is important. A vision that balances your teams’ challenges and the broader organization’s needs can inspire optimism and aspiration, as long as it’s grounded in reality.

Our vision was rooted in the problems identified earlier. We focused on five pillars—product, customers, people, structure, technology—as we had experienced issues in each. For example, one of the pillars was framed as:

Product - Building what is aligned to broader Intercom strategy, carrying our own confidence in it and delivering impact

• We follow our product strategy to ensure that we contribute to long-term success of Intercom. We are adaptive and embrace change coming from outside but we always think through and are transparent about trade-offs.
• We build a support channel first and foremost, and thus we keep end user experience top of mind.
• We work aligned to our R&D principles. We move fast, we start from the problem, we ship often to learn and we deliver outcomes.

As you see, I described each pillar using three bullet points to set the direction, leaving room for ambiguity where we had to do deeper and more detailed work to align on.

We then collaborated with EMs, PMs, and designers to build alignment through a short workshop where we walked through each pillar and together brainstormed examples of behaviours that support these pillars and those that are anti-patterns.

Lastly, we shared the vision with teams for feedback and alignment.

Assess your teams against the vision to start tracking progress and trajectory of qualitative change.

We conducted a survey for an assessment of how teams felt about aligning with the aspirational vision. This gave us a benchmark and reality check of the gap between our current state and aspirations.

We re-prioritized problems based on effort and impact. We then shared this back to the teams and engaged volunteers to initiate low-effort, high-impact changes, creating immediate momentum.

Rinse, repeat.

This is a long journey, and your vision serves as a beacon for everyone on how to operate. While I plan to repeat the assessment and problem prioritization every quarter, smaller organizations may need to do this monthly. Remember, nothing is set in stone; stay open to refining the vision over time.

Summary

Managing turnarounds is strenuous. It’s essential to have a clear plan to avoid burning out. I hope this framework equips you with a practical strategy to navigate towards stability and sustainability.

I believe though that speed here is extremely important. There will be a lot of inertia to fight and a lot of engagement to build. Both of these things take a lot of time. The faster you move, the better chance that your team will give you a chance and believe in the change in the trajectory before they start scouting for new jobs. Speed is life here.

One of the most impactful factors contributing to my growth has been the peers I had the fortune of meeting throughout my career and establishing coaching relationships with them. These relationships provided me with multiple perspectives for problem-solving and project development. I hate working alone; my instinct is always to find partners in crime or coaching partners to bounce off ideas.

Peer coaching has not only helped me navigate tough times, but it has also alleviated anxiety by teaching me that I’m not alone in experiencing emotions and struggles – it’s normal.

Seeking peers to talk to was one of the most important goals for my first year at Intercom a few years ago. Many of these relationships remain my most trusted partnerships, though some have moved on, and we don’t communicate as frequently. Most of these connections were established during the earlier days of the company when it was smaller than Intercom is today. I’ve observed that managers who joined in the past two years often lack this natural coaching circle.

While it’s common for managers to suggest coaching partners for specific projects requiring growth or when someone is underperforming, I believe as managers of managers, we are missing an opportunity. We should ensure that coaching partners are always available for our teams, not just when there’s specific guidance on what to focus on, but also as a means for our directs to determine what is most valuable for them.

Coaching assignments

Approximately a year ago, I introduced the concept of rotating peer coaching to my team of four managers who report to me. We decided that every six weeks, they would form two pairs and hold weekly coaching meetings, regardless of their current projects.

We defined goals for these meetings as follows:

1. Build relationships through collaboration
2. Enhance coaching skills in areas with limited context
3. Increase accountability by having someone invested in personal growth

Building relationships through working together

Of these three goals, I consider this the most crucial. In the age of remote work, establishing long-term relationships can be incredibly challenging, especially outside of one’s primary team. These managers excel at fostering relationships within their teams, whether between engineers or product managers. However, what about their own relationships as managers?

Having colleagues at the same level, without a power imbalance, whom you can trust, is a tremendous asset that can help navigate difficult times. If this peer also reports to the same manager as you, it adds further value. Shared experiences among managers, including their mistakes and subsequent learnings, create a sense of optimism that despite any errors made, there is a secondary support system. Additionally, more tenured reports can offer guidance on how to work effectively with you, often pointing out unconscious patterns.

Enhancing Coaching Skills in Areas with Limited Context

This goal is also significant. It provides a safe space for improving coaching skills. Coaching can feel awkward, especially when starting out and attempting to “talk less, question more.” While building relationships remains a priority, trying out different coaching techniques within this safe space offers excellent practice.

Increasing Accountability through Invested Support

While this is the last priority, it is still a valuable goal. Essentially, it brings healthy peer pressure. Participants discuss their work and what they aim to achieve, creating an unspoken expectation of self-accountability during subsequent coaching sessions a week or two later.

Closing thoughts

So far we had great positive feedback about these sessions. Folks in my team really enjoy them, they built stronger relationships with each other, they are less dependent on me and continuously growing their exposure. For a few quarters, this was consistently the highlight of the culture in our team. As much as it feels awkward at the start, it’s definitely worth doing.

If you are managing managers, consider starting it. If you are an engineering manager, suggest that to your team!

Finding the right balance between moving fast and keeping high-quality documentation is very hard for scaling companies. In practically all cases, it’s always better to make trade-offs on documentation and move faster building product.

Another common problem of scaling engineering organisations is how to scale your best practices, cultivate culture and maintain knowledge. The most critical moments that shape engineering culture at the organisation are when you have to make trade-offs and rationalise decisions. Even well done, architecture documentation misses the rationale, representing only the result. And usually, it’s easy to see the decision being made. For example, we decided to model this system in that way, which is generally understandable from the codebase. But what’s missing is why did we make this decision?

If you feel that these problems are present in your organisation, experiment with Architecture Design Records.

What are Architecture Design Records?

Architecture Design Records are documents that describe:

• the engineering problem that the team is facing at a time,
• requirements that we need to meet to solve this problem,
• a variety of solutions that we brainstormed before choosing a solution with their trade-offs
• and the detailed description of the solution we chose

The exact template that the organisation uses will vary depending on the evolution of the process as you adopt it, your organisation’s current need, and your specific product.

At Intercom, we call them Tech Design documents and use Google Docs to write and store them. I know about other companies that write them in markdown and store in Github, in respective locations to code files that they touch.

I am not opinionated on the shape and format of this document. I think organisations should experiment and evolve their processes continuously, so it’s more important to start and iterate than come up with a perfect template.

However, I firmly believe that the past rationale and trade-offs analysis is one of the most valuable lessons that your organisation can take. Environment changes, constraints come and go, but the way someone distilled a problem, compared the options and made a decision is a prize on its own. That’s your engineering culture manifesting itself in real life. If your values and principles show in these documents, your culture is strong. But if they don’t, they are aspirational - and you need to work harder as a leader to ingrain them.

Who is a stakeholder of such documents? The team that wrote the document is a primary stakeholder. It increases the team’s engineering excellence and will be an artefact that the team itself will mostly use in the future. Additionally, in the case, you are a software consulting business, ADR stakeholder is also your customer. You can keep them up to date and aligned with your work.

Who is an approver? By default, especially in smaller or growing companies, it’s the team itself. More mature companies or those with higher costs of making mistakes, tend to have Architecture Review Boards that approve these docs.

How you create them is as important as the documents itself.

Writing Application Design Records is a process in itself. And the way your organisation does it is equally important as the output document!

First of all, creating ADRs can easily enforce collaboration and mentoring. A well-rounded collaboration group can have a technical leader overseeing the solution, more junior members learning through practice and someone detached from the team to bring unbiased opinion.

This process also scales your engineering excellence. By writing ADRs for significant problems, you allow your most senior leaders to scale their influence. If your Principal Engineer collaborates with a team of 15-20 engineers, it’s challenging for her to track all the critical decisions organically. ADRs ensure that teams will solicit other senior leaders’ inputs in a scalable and organised way.

You also create opportunities for peer review of the work even before the first line of code is written. As the saying goes, an hour of planning can save a week of development.

And last but not least, you share knowledge between teams that don’t work together organically. Especially in the remote environment!

How to trial and adopt the process?

Writing documents can be a gruelling and disengaging task for engineers if they don’t understand the reasoning, can’t immediately see the value or leaders leading by example.

Try the following process to introduce ADRs as an experiment.

1. Lead by your most senior engineers.

The first ADS should be created under the leadership of one of your most tenured and experienced engineers. Lead by example and show the value of knowledge sharing and engineering excellence to your organisation from your best folks.

2. Adopt only for a few significant design decisions.

Don’t say that it’s obligatory to write ADRs for almost everything from now on. You might get there if your organisation becomes hard fans of the process, but don’t start there. Writing one or two every quarter to solidify the most important developments is already a great start. On the other hand, at Intercom, each team plan in 6-week cycles, and we would usually expect 5-6 tech design documents per team per cycle.

3. Share widely and continuously look for feedback.

Share your ADRs and ask for feedback as often as possible. We created a separate channel on Slack to share all our ADRs. It’s a big stream of artefacts, but every engineer can pick the ones they are most interested in. You work in customer-facing team day-to-day but would like to grow your skills and learn a bit about infrastructure? Pick and read ADRs from our infrastructure teams!

Did you try some form of Architecture Design Records in your organisation? Do you have lessons or observations you would like to share?

I could say that I recommend this book to every manager at every stage of their career. In people management and leadership, the situations where you will benefit from reading this book happen all the time. However, the reality is that these situations happen way more often in our private life - with our partners, parents or kids. This framework will help you be a better person for everyone around you.

About the Nonviolent Communication framework

The book describes how we can both talk and listen in a way that is not violent and capable of building connection and deepening the relationships by embracing any emotion, good or bad. Used in a sentence, it looks like this:

“When I (see, hear, smell, etc.) …, I feel …, because I need …. . Would you be willing to …”

As an example:

When I see that you are late for the meeting, I feel angry, because I need to know that I can rely on you at situations like this one. Would you be willing to be on time tomorrow?

In other words:

1. We describe the concrete actions that are happening that affect us.
2. We talk about how we feel in relation to what we observe.
3. We explain our needs and desires that created our feelings.
4. And we ask for concrete actions that will make the situation better for us.

When listening to someone, you can paraphrase what they talk about using this framework. It will help you identify their emotions and drive the conversation to talk about feelings and needs openly.

If you reflect on a few of your last emotional conversations, you will quickly realise how surprisingly often we don’t try to understand each other. You would say that if I didn’t try to understand my partner in conversation, at least I was trying to explain what I wanted. With reflection, I am not even sure if I understood myself what I wanted!

Obviously, no one speaks like that. It feels artificial. But being aware of it and trying to cover all four points of the framework (observation, emotion, need, request) will make your conversations more productive. As soon as you try to think about your emotions and what you need and request, you will connect with the other human being. After that, you are set to find the needs and requests of your partners.

Realisation of unexpected weakness

I realised how poor in emotions my vocabulary is day to day. I don’t reflect or express how I feel. While I often have to diagnose emotions at work, I realised that with a broader palette of words, I could connect more to my partners. This helps in building emotional intelligence and ability to go through and lead difficult situations.

Five lessons to take away

Analyses of others are actually expressions of our own needs and values.

So simple yet powerful statement. Especially when being a manager of someone who is analysing or judging the performance of their peers. You need to keep it in mind and help them understand what that need is.

Observations specific to time and context are a foundation of nonviolent communication.

Evaluations - observations that are generalised - bring negative emotions and can only be read as unfair criticism. There is a big difference between “You are late today” and “You are always late”. The first is an observation; the second is an evaluation.

Use positive language when making requests.

People are often confused when you use negative requests. Even more, they tend to be more resistant. When you say “I don’t want you to act like that”, you don’t mean that you will be happy about any other behaviour. You look for a specific behaviour, tell it.

Ask before offering advice or reassurance.

It is often frustrating for someone needing empathy to have us assume that they want reassurance or “fix-it” advice.

Reflect back messages that are emotionally charged.

Empathy is a respectful understanding of what others are experiencing. It’s not about feeling the same emotions as the other person or agreeing with these emotions. It’s about understanding them and making another person feel understood.