Putting our brands first: How Bold prepares for peak season traffic in 2022

As the Chief Technology Officer (CTO) and one of the four founders of Bold Commerce, Eric Boisjoli has both the real-life experience of making Black Friday a success, and a wealth of knowledge surrounding the technical side of running an industry-leading software development company.

There's a quote I like to say. Well, I don’t actually say it, but it’s mounted on my wall:

“Merchants first, then tech.”

This quote keeps something very important in the forefront of our minds: although we are always trying to build amazing, innovative tech, our mission is to make sure merchants are able to make a living while doing what they love and are passionate about.

It gives us a proper perspective when deciding whether or not something should be done, because we can ask ourselves, “does it ultimately benefit the merchant?”

The last thing we want merchants to see on Black Friday or Cyber Monday (BFCM) when they try to run one of our apps on their store is this: 

We've learned a lot in our years of business, and these experiences — both good and bad — have drastically changed how we prepare for the holiday season. We are constantly making improvements to the way we operate, whether it's upgrading the quality and cross-functionality of our apps, improving the way we load test, or making the operational and technical changes required to achieve the highest degree of PCI compliance.

Hopefully I’ll be able to share some of what we learned so that you can stay motivated, avoid burning out, and help support the biggest ecommerce event of the year — even if it looks a little different than usual. 

Peak season starts early at Bold 

It's extremely important that we remain agile and plan early, so we typically start our Black Friday planning around June. 

In the years since Covid-19, BFCM has become more of an all-November event instead of being focused on one weekend, with a much heavier emphasis on online sales. This makes it extra important that we are prepared for both sustained increase in volume as well as extremes for the merchants who rely on our apps and solutions. 

We typically start our preparations by reviewing the previous year’s Black Friday performance and determine what we did well, what exceeded our expectations, and what didn’t go so well. We don’t ignore those things, though — we improve and learn from our mistakes. 

By August, we’ve made our predictions about the volume of sales, traffic, and support we’re expecting based on last year’s numbers, and trends we’ve seen so far this year.

With this information in mind, we put together a single, department-wide calendar spanning the next three months, which includes all initiatives that are geared towards the BFCM weekend. We put everything on one calendar because this isn’t a single team initiative; it’s an event that requires collaboration from all departments in our company, and we need everyone working together to make this event successful.

Measure and optimize everything

Every year we introduce a lot of new systems and processes, and we do our best to track the relevant stats and key performance indicators (KPIs) for those systems.

Leading up to Black Friday, we go over everything to ensure we have all the needed KPIs, and to ensure that there are systems in place to monitor and alert us to any looming problems.

In addition to giving us insights, I like to challenge myself and the team to see how much we can improve our KPIs in order to position ourselves better for BFCM. Although the majority of what I’m personally looking at is ‘requests per minute’, ‘response time’, ‘server load’, ‘cache hit %’, and ‘uptime’, this extends to the support and marketing teams as well.

Testing for scale

When I go camping, regardless of the forecast I set up a rain shelter — I’d rather be prepared than have to set it up during a storm. Bold uses this same logic during its Black Friday preparations.

It takes roughly the whole year to reach the amount of traffic that we experienced over the previous BFCM weekend. Each subsequent holiday we’ll see double or triple the amount of traffic, and it’s hitting us all at once.

We have continued to push boundaries of load testing on our top applications to ensure we are ready for increased volume. Where in past years each team has load tested independently then come together to test as a group, this year we are testing as a full team. We are also looking to implement weekly automated load testing, while continuing our manual checks. 

We have to take on the very difficult task of putting measures in place to prepare us for traffic we’ve never experienced before. Plus, it’s hard to mimic real traffic because we can’t predict which apps will be used most, and we don’t know which of our newer releases will get the most attention from our customers.

Despite these unknowns, we leverage tools such as ‘Hey’, which allows us to stress test our servers, sending tremendous amounts of fabricated traffic to our apps. We have also integrated a new software called Locust, a load-testing tool that behaves more like a customer going through the buyers journey, providing useful insights. 

By monitoring these tools, we get insight into how our environment will handle the volume and scale it to meet the needs of the upcoming holiday, as well as determine any bottlenecks in our systems.

Our intention is to slowly (and safely) increase the amount of traffic weekly, so once it hits Black Friday, we will be confident and ready for the real deal.

When we set up processes, we try to put redundancies or backup mechanisms in place with the intent that if something were to break, we have the confidence in knowing there’s going to be a backup system that should be able to pick up the slack.

That’s all well and good, but as everyone should know, a backup is only valid if it’s tested. Because of this, we try to have planned service outages every few months to ensure that the backups kick in and perform as expected.

Be ready for anything

Even though you tested for everything you can think of, you still don’t know what you don’t know. It’s likely impossible to know every possible scenario that can potentially occur, so it’s important to have a backup plan. 

In a time of chaos, it’s nice to have a simple cheat sheet of priorities. Deciding which limb to cut off in order to save the rest of the body is never a fun conversation, but it’s better to discuss it when things are going well, rather than while you’re stressed and panicking. 

At Bold we have a few such priority lists:

1. Shoppers should be able to check out on a store.
2. Shoppers should be able to leverage the functionality of the app.
3. Merchants should be able to manage the apps.
4. Bold’s support team should be able to help merchants.
5. Background tasks should complete in a timely manner.

Within that, we have a prioritized list of applications, sorted by the magnitude of the impact that it would have on the merchants, should the app not function properly.

We want to be as prepared as possible, and even when we can’t be, we’ve got a guide to help us respond in the correct order.

Keep your finger on the pulse

Although we work on the initial plan together, the collaboration doesn’t end there.

We have a BFCM committee that meets regularly. As we get closer to the date, the stand-ups we do become more frequent (in fact, daily). We do this because it’s critical to ensure that everyone is aware of how each team and department is doing so that we can make smarter decisions regarding our plan.

It’s important to know that we have the support capacity before sending out a big marketing campaign. As well, if a specific Black Friday app feature is going to be delayed, the other teams should be made aware sooner rather than later, so they can change their strategies accordingly.

Implement a soft code freeze

Bold is a very agile company and we release code very frequently. We’ve put in place a lot of measures to ensure that application changes can be deployed very easily, and we believe that releasing smaller, less complicated changes will have less unexpected side effects.

However, during the holiday season we get a lot more particular and we begin to scrutinize what kind of releases we’re putting out. We ask ourselves: “Is this going to help the merchant?”; “Is this needed for Black Friday?”; “Does this add potential additional risk that we don’t need to take on?”

We need to make sure that the things we’re putting out around this time are going to provide enough value to the merchant, or are very much Black Friday-related. That way, we can prioritize the important pieces for Black Friday and hold off on other releases, so as not to add incidental bugs.

A new normal

We typically take an all-hands-on-deck approach for Black Friday, everyone in the office working together, sharing food, and supporting one another. 

It will obviously look a little different this year, but we are still hoping to have a small, socially-distanced group together in the office, particularly for the over-night shifts. We're still talking about how this will work as we want to make sure our employees are safe.

Our goal is to make it as normal of a Black Friday weekend as possible, despite the changes in circumstances. 

We will still take the approach of having a BFCM “Command Center," each team working around the clock, with others on-call. We will be actively answering tickets, watching our KPIs, investigating any little blips, and communicating with all of our teams. That way, we have a continuous pulse on how things are going. 

We'll use the previous year’s traffic to anticipate and plan for all the spikes to help us determine the staffing requirements. When do we need the most support, and when can I sleep? 

If there’s anything to take away from this, it's that Black Friday should be treated like any other project.

• Have a detailed plan.
• Find out what your risks are and what you can do to eliminate or mitigate them.
• Test and improve your systems.
• Have contingencies.
• Over-communicate.
• Get an office hammock.
  

It's showtime!

Even though I’m the CTO and my responsibility is to make sure our apps and solutions are built with high quality, are running properly, and we don’t go down (especially around the holidays), I feel the role extends way beyond just that.

Because communication is so important not only year-round, but at this time of year, my role extends into making sure there is good communication between all departments, because Black Friday isn’t just about us, our code, or our servers — it’s about the merchants being successful. 

The thing I love most about Black Friday is that it brings the entire company together with the unified mission of helping our merchants do their best. 

No matter what happens, we’re going to be ready to act. 

For more tips, resources, and ways to make the most of the biggest online sales event of the year, check out our Peak Season Starts Here home page for holiday content, useful tips, and exciting offers.