Low Batteries and Actual Fires: The Importance of Knowing When and How to Escalate a Project Issue

It’s Thursday night. One more day of work before I’m on vacation for a week. I’m awake later than I plan to be, but, that’s how it goes right before vacation, right? Swirling thoughts about handing off projects, did I make all the proper plans for the house/kids/pets while I’ll be away, did I pack too many clothes? Then, finally, I doze off to sleep.

At 2 a.m., I’m rocketed awake by the smoke detector VIOLENTLY going off in the house. It wasn’t just beeping, it had an ear-shattering beep and a voice that screamed “FIRE!” over and over.

This led to the following:

• I punched the side table as I reached for the light causing a nice purple bruise
• I flew out of bed and was just able to get shorts on in time to not scar my children for life with my bare butt
• I ran through the house, heart racing, room-to-room, trying to find the source of the fire and the noise
• I determined there was no fire. Instead, the battery in the smoke detector needed to be changed
• I felt incredibly lucky we had a 9v battery in the house and changed the battery in the smoke detector and went back to bed.
• 15 minutes later, it went off again. (Turns out, we changed the battery in the WRONG smoke detector.)

I then spent the next 2 hours staring at the ceiling waiting for it to happen again. And the whole time, all I could think about was the importance of appropriate situation-specific user notifications.

Does a dead battery warrant the same notification level as a house fire?

I can imagine the conversation that took place at the smoke detector company that led to a low battery warning waking my entire family at 2 in the morning. It probably went something like this:

Doug: “A smoke detector has one job; inform you when there is a possible fire. So, when there’s a fire, the smoke detector should scream as loud as it can.”

Sue: “But if the battery is dead, the smoke detector can’t inform anybody about anything. So, if the battery is low, it should scream as loud as it can.”

Doug: “You are so right, Sue. Turn the volume up to 11! Who’s ready to go to lunch?”

While this conversation and resolution may be fine at the fictitious company I just made up, the jump from ‘dead battery’ to ‘house fire’ notifications doesn’t work in the real world. At Revelry, we know that understanding when and how to escalate a project issue is critical.

As a team, how do we escalate issues and requests for help?

At Revelry, we track our work in GitHub Issues. Those Issues are considered the Source of Truth™. So, when we really have a question that needs answering that is blocking our work; or there is a newly-discovered dependency to an in-progress Issue; or a bug causes system errors after-hours and we rush to our firehoses to put out the immediate danger … we communicate all of the above in Issues via comments.

We tag people, we assign people, and we should not be afraid to reach for the blocked label to make it very clear the state of an Issue.

Slack pings are helpful to help draw attention to an immediate problem. But I see that we actually have a pretty nice scaffold in place to be able to escalate alerting:

Does your Issue require the help of a specific person? (if yes, maybe we should talk about more pairing, more documented code, more helpful test suites, but I digress…)

• GitHub comment including an @ mention + assignment + blocked label
• @username mention in Slack, to get a specific person’s attention
• Absolutely need an immediate response and the person didn’t respond on Slack? Great! That’s why we have the team’s phone numbers within an easy Slack or Wiki search

Are you unsure who can help with your Issue?

• it’s still really, really helpful to start within your immediate team and choose a person to help you. @username still applies here. as a general rule, you’re more likely to grab someone’s attention if you choose a someone to ask. if you’ve ever tried to recruit volunteer help for an organization, you know what I mean.
• maybe, maybe, if the person you chose cannot help you (unavailable, doesn’t have the answer), then you can choose a higher level of alerting the team. We very intentionally try not to reach for @here and then even higher for @channel. It’s pretty rare that a single Issue needs to cause everyone attached to a channel to stop and react.
• we use a #firedrill to raise an alarm about possible fire (note: not intended to make everyone run around as if their heads are on fire); this helps raise the visibility of a need for help beyond the immediate project team AND also serves as a nice alternative to an @here or @channel. Many team members have eyes on #firedrill and you’re likely to get some help from coaches, directors, etc.

Not all messages are created equal

Issues and errors come in all shapes and sizes. In the case of the smoke alarm, it has two:

1. There is a fire
2. The battery needs to be replaced

Both of these are clearly important messages to convey, but they certainly aren’t on the same level. If there was, in fact, an actual fire, an ear-splitting screech and a voice yelling FIRE is 100% appropriate. Everybody needs that information RIGHT THIS MINUTE. But, couldn’t the “replace the battery soon” message been delivered in a better way? Maybe a quick beep and flashing red light, for example?

---

Agile software development communication productivity project management team communication