Here’s How We Survived a Two Week Fire Drill

The Revelry Lean Agile process is a thing to behold. When the process runs smoothly, any team member can get to work on any project and understand exactly their place in the sprint, and communication flows beautifully. And unlike some software development teams, we don’t run from the term “fire drill”. Among our many team communication channels in Slack, #firedrill is a place where any person at any time can speak up and say “I smell smoke” about any project.

Speaking up about potential challenges is a normal and important part of our process. When a teammate raises a fire drill, available or relevant teammates participate in the topic thread. Sometimes, we resolve the issue in a few minutes. And other times, we identify a major challenge and take the conversation to the product owner to discuss next steps.

Life and requirements is full of surprises, and our process is not immune to getting turned on its head and landing us in front of a real, raging, big fat fire.

This is not a drill. I repeat, this is not a drill.

Recently, we were tasked with needing to help a product owner get a “win”, but it called for changing requirements and meeting an aggressive deadline. This turned into a two week fire drill. How the hell did we survive a two week fire drill?

We survived – and we thrived – because we practice fire drills every day. We practice speaking up, smelling smoke, asking for help, and calling our shots. And here’s exactly what we did when this particular inferno rolled up on our door. (“Check that door for heat!”)

Here’s how we did it:

1. Call in reinforcements.
2. Set expectations.
3. Communicate.
4. Follow a specific implementation process.
5. Ask for help (again).
6. Keep learning.

When facing a major fire drill, call for reinforcements.

You see this all of the time in real emergencies. The next fire station over needs to come help out. When your team already has a process in place, they’re equipped to facilitate the supplemental team. And that’s what we did.

Our team was fully booked, but there was a lot of work to do. We were able to call up some software contractors who were familiar with the integration we were adding. Now, we had a small army: 7 full-time software engineers including contractors, an engineer on the client side, three product stakeholders, and me, in the role of producer. That’s a lot of cooks in the kitchen, and for an agile project, this can be tough to manage.

Prepare the team and set expectations.

The demands from the client were high, but they were ready to make themselves readily available at all times for questions. This is important to set in advance: Our team might be writing code at 10 PM, and if we’ve got a question, it can’t block us until the next day. Make sure that your client understands that during these (hopefully rare) intense and demanding project moments, they need to be as involved as the developers.

All pieces have fall in place as one true team, or your project is set up for failure. Everyone has skin in the game, and everyone should be equally invested in success.

Establish and adhere to a solid team communication protocol.

At Revelry, we take team communication pretty seriously in the first place. And when you’re preparing to battle a major fire by working with a large group of developers and stakeholders, communication is everything.

We use Slack and GitHub Issues for our communication, but whichever forms of messaging you use – use them often. Be loud and deliberate about what you’re doing and when you need help. Just like you would if you were part of a team battling a four alarm fire. Don’t be ashamed to ask for help, and don’t wait. If you think for a second that you need help, ask for help.

With a small window and a large team to manage, we certainly faced a lot of obstacles. Imagine three separate groups of developers stepping on the same code and files concurrently, while continuing to face changing requirements. We knew we had to talk loud and talk often. Here’s how we deployed our communication protocol:

We track issues via GitHub tickets.

Everyone on the team self assigned anything they picked up, and they’d call out their plan with a comment. And when they took a break, they updated the ticket with a comment that documented progress and status. With all the moving pieces, these constant updates even served to help others solve problems that they were working on.

Conversation happens in Slack.

Even though not everyone we pulled in for the project was familiar with Slack, we still believed in our process so much that we were able to rub off on the rest of the team. At Revelry, we’re as open about our development process as possible. We ask questions, out loud, in public channels, in front of our clients. We’re not concerned about hiding what we don’t know. We make sure everyone knows that we are always actively learning.

I repeat, conversation happens out in the open in Slack.

It’s tempting to send a private message for fear of “interrupting” the channel. We had to be really diligent about directing the contractors and the client stakeholders to keep all chatter, comments, and decisions in the public project channel. (We still have to be.) Keeping these conversations out in the open helps people do their best work. It helps us keep from duplicating tasks, and it helps avoid mistakes and confusion.

And then, decisions get copied back over onto the GitHub tickets.

As a producer, I was sure to capture and document every decision point, no matter how minor it seemed. So, if we hashed something out in Slack, I made sure to copy that resolution over into the ticket for context. If I had to get on a call with a team member or stakeholder, I’d take notes and be sure to transfer this knowledge to the team.

Taking these steps to hold conversations out in the open and document decisions helps everyone move faster. We pretty much eliminate any re-working of completed features because of some seemingly arbitrary decision that ended up affecting huge portions of our implementation.

Be deliberate about your implementation approach.

As I mentioned, a lot of the same files get touched and all of the issues can easily become dependent on one another. This adds a giant margin for error and conflicts on the implementation side. It’s easy to pick up a ticket, crawl into the weeds, and get tunnel vision. So, obviously, we’ve got the communication requirement, but I really want to get into the specifics on deployment.

All of our team members had to be very deliberate about discussing and calling out their approaches on implementation, so they didn’t break each others’ code. Making incremental ticket updates and dropping a mention to anyone who the update might affect is – again – very important.

We had to take extra diligence on Pull Requests as well. One way we accomplished this was bringing in eyes from outside of the project to review the code. Not having a been as close to the application prompted many good questions on implementation strategies that forced us to justify or re-think the way features were being implemented. In the end, this really helped the code quality stay high while moving at a breakneck pace.

Speaking of code quality, tests are a big part of Revelry’s implementation. But because we were moving at great speeds and facing down deadlines, we made some concessions in this respect. We were writing fewer tests, for sure, but not skipping them altogether. On smaller tickets and obvious workflows, we punted on tests in favor of time. We implemented testing on the larger, stickier issues so they did not break the application. For example, if you’re facing a crunch like this, you might skip the test for logging into the app but write a few for that giant feature with tons of calculations.

Ask for help, ask for help, and… ask for help.

There was no time to spend floundering on a problem. We made it clear: if you’re stuck for more than 30 minutes on a problem, ask for – no, demand – help. Yes, of course everyone is busy. Don’t ever let that deter you from dropping a message in Slack, a comment on a GitHub ticket. or even a phone call to your PM if the ball stops moving.

Chances are, with a large team, someone has come across a similar situation. It’s possible that your five-hour problem has already been solved by another team member, so – voila! – it’s now a half-hour problem. Your team is there to support you; it would be silly to leave that resource untapped. But so often, people do just that.

So if I haven’t made it clear, ask for help when you need it. You can even be proactive and announce that you are pretty sure you will need help shortly. A teammate may be reaching a stopping point, and they can get ready to set up a quick pairing session with you.

This was something we reinforced over and over again during this fire drill. Yell, and yell loud. There were more than a few examples where calling out for help early and often paid off. I’ll give you an example.

We rely on a platform of internal tools here at Revelry, and our latest is a React component toolkit called Harmonium. When one of our contractors expressed some uneasiness about a feature and prepared to build a React component from scratch, we were able to point him to Harmonium. Had he not said anything, he could have spent half a day on something that was already done!

Just keep learning.

There is much wisdom to gain from analyzing mistakes. I won’t go into all the mistakes we made on this project. What I can tell you is that we learn from them and improve with each passing week. Retros are helpful – we definitely practice these at Revelry – but you don’t need a meeting to learn from a mistake and share it with others so they don’t meet a similar fate.

Learn all the time, and share all the time. When you learn something useful, drop it in Slack or in a ticket comment. Share that wealth of knowledge.

Fire drills are good. Run them well.

No, a two-week fire drill should not be typical in agile software development. But you can survive it if you routinely and calmly practice the process. If you happen to face a fast-approaching deadline with a ton of moving parts, you can do it. I believe in you. There is no secret sauce here, it just comes down to hard work and communication.

• Deputize a team to help you. Don’t go it alone.
• Take a step back, and have a look at the 10,000 foot view. Follow your process, and don’t get hasty or overreact.
• Communicate openly and loudly, about your intentions and your failures. Knowledge is power, and you can’t stop an informed team.
• Keep your work quality high. You may have to take some shortcuts here and there for speed sake but don’t throw your standards out the window.
• Ask for help early and often, even try to anticipate the need for help if you can.
• Learn from your mistakes, every day. And share them!

---

Agile Sprints Team Communication