DEV Community

Cover image for How to Handle Incidents in a Small Team
Mansur Fattakhov
Mansur Fattakhov

Posted on Originally published at mind.mansur.expert

How to Handle Incidents in a Small Team

As long as the service is running, it's easy for a small team to assume it doesn't need an incident process. Everyone knows everyone else, you can get hold of anyone, and if something breaks, you'll figure it out in chat. That agreement gets its first real test when one person is rolling back a release, another is changing the configuration, and someone in chat is asking when things will work again. Everyone is doing something useful. Nobody has the whole picture.

My article on alerts was about noticing a problem. This is what comes next: the alert arrives, somebody opens a laptop, and then what? In a small team, one or two people should be able to put a process in place without a dedicated department or a form that takes half an hour to fill in. You need a few agreements that help you restore service and leave you with something to show for the outage besides exhaustion.

A shared channel: who's taking what?

My starting point is a shared incident channel. When a report comes in, that's where we say who's picking up which piece. Everyone needs to be able to see it: one person is checking application errors, another is looking at infrastructure, a third is finding out which users are affected. A read receipt doesn't tell you who's actually working on the problem.

For a major incident, we open a dedicated channel, for example inc-2026-10-06-http-outage in Slack. The date and short description tell you which outage it is and make the discussion easy to find later. We leave a link in the shared channel and move the conversation to the incident channel. That's where we collect checks, results, and decisions.

Get on a quick call to divide the work and assess the risks

When an incident is serious, it helps to get on a call straight away. In those first few minutes, we need a common picture of what's broken, how many users are affected, and what could happen if the problem continues. Current impact and potential risk can be very different. Errors might still be rare, but the incident is hardly minor if data could be lost or the failure could spread.

On the same call, we agree on who does what and who keeps track of the overall situation. People need to know what the others are doing. Otherwise, someone may change a setting just as someone else rolls back a release, leaving us to guess which action helped and what else changed in the meantime. You don't need an elaborate set of roles. Divide the work explicitly and talk through the next few steps.

Everyone records their own actions and results

It's worth posting a short account of the call in the incident channel: what we know about the impact, what each person is working on, what we've decided to try, and when we'll post an update. Anyone joining later can catch up without asking for a recap. Once the service is back, there's already material for the postmortem.

As the work progresses, it helps if each person notes the time and what they did: when they took the task, what they started investigating or changing, and what happened. Short personal notes are fine; they can go into a shared timeline after the incident. This matters especially when several people work in parallel. Even after a brief outage, it's surprisingly hard to reconstruct the order of events from memory.

If people are already talking in the incident channel, they can post progress there. Each message already has a name and a timestamp. An engineer might write: “I'll take restoring database access. First I'll check whether the database is reachable from the internal network.” Then they come back with the result: “I can reach the database from the internal network, but connections from the application nodes aren't getting through. I'm checking network access from those nodes.” This is an illustrative entry. What matters is the link between what the engineer set out to check and what they found, whatever the eventual cause.

These messages serve two purposes. The team can see where things stand, what's been checked, and who's working on what. The postmortem gets a sequence of events with names and times. Even if someone keeps private notes, the key findings should still go back into the channel. After recovery, we can bring the notes and messages together in a single timeline. Keep the checks that led nowhere, too: they explain what we knew at the time and why the next step made sense.

In the working record, it helps to distinguish observations from assumptions: when we noticed the problem, what users couldn't do, what we tried, and what happened. “The errors stopped after the rollback” is an observation. “The latest release definitely caused the errors” is still a hypothesis. Write both with equal certainty and the postmortem will start with people defending the first guess that paid off.

Restore service first

During an outage it's easy to get absorbed in the explanation: why did this happen, which line is responsible, who changed the setting? What users need right now is a working service. Sometimes rolling back, disabling the failing path, or temporarily limiting a feature is enough to give them that. We'll still need to find the cause. Recovery and investigation solve different problems and can move at different speeds.

Metrics tell an engineer how the system responded to a fix: whether errors dropped, latency returned to normal, and requests are being processed again. That's valuable evidence. It's even better if we can repeat the user journey that actually broke. If the team has testers, they can run through it and check the outcome. If the error only affects a particular user and we can't reproduce it ourselves, we can get in touch and ask them to try again.

This is where the project manager and the rest of the team can help. They can run checks, contact the user, and bring the answer back to the channel. Communication and verification can happen while the engineer works on recovery. The shared record then shows both what the metrics say and what the user experienced. If the graphs are back to normal and the user hasn't replied, say exactly that: we see technical recovery and we're still waiting for confirmation on this particular case.

When choosing the next step, I'd look first at how easy it would be to undo. Disabling a new processing path may be safer than reworking shared configuration on the fly. A rollback isn't harmless by default, either: if the release has already changed the data, the old version may no longer be able to handle it. Before acting, take a moment to say what you expect to see and what you'll do if things get worse. That gives everyone the same way to judge the result of the recovery step you're about to try.

Time permitting, save the evidence that the change will erase: a section of the logs, when the errors started, the running version, settings that changed, or the state of a queue. Don't delay recovery for the sake of a perfect archive. Someone who isn't applying the fix can gather what's available and leave links in the channel. Sensitive data stays in the storage intended for it rather than spreading through screenshots in the shared chat. After a restart, a process may look perfectly healthy and tell you nothing about what was wrong ten minutes ago.

A blameless postmortem with clear ownership of the fixes

Choosing not to look for someone to blame doesn't let everyone off the hook once the meeting is over. It's more useful to ask why an action seemed reasonable with the information available, which safeguard was missing, and what will now change. You can't verify “be more careful.” You can verify a task with an owner, a deadline, and a clear definition of done.

By the time we write the postmortem, our action items are already links to issues in the tracker, each with an assignee and a due date. “Add a database connectivity check” on its own doesn't tell you who will do it or when. The issue should make the intended change and its acceptance criteria clear. From the postmortem, anyone can jump straight to the discussion and current status.

Several actions may be necessary: fix the cause, add a missing check, clarify the recovery procedure. Each gets its own issue, owner, and due date. That's how a postmortem becomes part of the regular rhythm I described in the processes track. The team's commitments make it into its work plan, where they stay visible long after the incident stops feeling urgent.

Keeping the outcome alive a week later

The next day, the team is back to its usual work. The urgent messages scroll out of view, the incident channel goes quiet, and the promised fix has to compete with the next release. Even a good postmortem can come to nothing here: everyone understood what happened, but a month later nobody remembers what became of “we should do something about that.”

I'd keep a short entry for each incident in a shared register: the date, the impact users saw, links to the channel, timeline, and postmortem, then the follow-up issues and their current status. A spreadsheet or a page of links is enough. There's no need to copy the whole postmortem; the point is to find a particular incident and spot unfinished work.

Revisit those issues during normal team planning. If a deadline has passed, it's more useful to understand why and agree on what to do next than to move the date. Sometimes the original fix turns out to be too big; sometimes another change makes it unnecessary. Leave that decision on the issue, too: why it was replaced or closed and what risk remains. A green incident status means the service is back. It doesn't mean every promised improvement is in place.

Before the next postmortem, look for similar incidents in the register. You might find that you've seen this outage before, the follow-up issue is still open, and the team has just spent the first twenty minutes running the same check again. At that point the useful question practically asks itself: what's stopping us from finishing the fix?

This simple setup has limits. If several incidents run at once, multiple teams get involved, or ownership regularly gets lost in handoffs, agreements in a single chat won't be enough. You'll need explicit arrangements for coordination, on-call coverage, and handoffs. Add that complexity when you've seen the problem it will solve. To start with, it's enough that everyone can tell who's restoring service, who's talking to users, and where to find the shared record.

What should be left after an incident?

Before everyone goes back to their day, I'd check five things:

  • It's clear who picked up the initial report and how the work was divided.
  • Actions and results are in one place, with names and timestamps.
  • Metrics and a check of the affected user journey confirm recovery; any remaining uncertainty is recorded.
  • The postmortem links to issues with assignees, deadlines, and acceptance criteria.
  • The issues have a place in the team's work plan, where their status will come up again.

You can tell an incident process is working from a small moment: somebody joins the conversation and quickly understands what's going on and how they can help. They don't have to piece the story together from three private chats or interrupt the engineer who's restoring database access. And a week later, the team can open the record and see what changed because of the outage. That's worth agreeing on in advance.

Top comments (0)