{"id":2028,"date":"2019-12-11T11:33:35","date_gmt":"2019-12-11T10:33:35","guid":{"rendered":"https:\/\/www.codemotion.com\/magazine\/uncategorized\/how-to-scale-your-devops-from-100-servers-to-1000\/"},"modified":"2021-12-23T14:12:41","modified_gmt":"2021-12-23T13:12:41","slug":"how-to-scale-your-devops-from-100-servers-to-1000","status":"publish","type":"post","link":"https:\/\/www.codemotion.com\/magazine\/devops\/how-to-scale-your-devops-from-100-servers-to-1000\/","title":{"rendered":"How to scale your DevOps from 100 servers to 1,000+"},"content":{"rendered":"

Codemotion Amsterdam 2019<\/strong> had a huge variety of talks across more than a dozen tracks. One of the ones that stood out for me was this talk on DevOps<\/span><\/strong>, given by Pat Hermens<\/strong> of Coolblue. Coolblue<\/strong> is one of the biggest online retailers in the Netherlands<\/span>. They generated revenues of \u20ac1.3bn in 2018. Since they were founded 20 years ago, they have seen exponential growth. This is reflected both in their revenues but also in their development team<\/span> which doubles in size every 18 months and currently employs 240 developers<\/span>.<\/p>\n

The challenge of scale<\/h2>\n

Many people might naively wonder: what is it with scale<\/span><\/strong>? Why are 1,000 servers<\/span> so much harder to handle than 10 servers<\/span>? Pat shared a very pertinent quote from Edsger Dijkstra<\/span><\/strong>:<\/p>\n

\n

\u201cApparently, we are too trained to disregard differences in scale<\/span>, to treat them as \u2018gradual differences that are not essential.\u2019 We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true!\u201d<\/p>\n<\/blockquote>\n

Some years back, Pat and a colleague gave a talk at a predecessor to Codemotion<\/span>. This talk looked at the prerequisites in order to be able to scale<\/span> your development successfully. They called this the \u201cfaster to master\u201d checklist<\/strong>.<\/p>\n

\"\"<\/picture><\/figure>\n

But simply checking all these boxes is not enough to ensure you can scale<\/span>. You need to do a few other things.<\/p>\n

Four Stories of DevOps scaling<\/h2>\n

Pat shared four stories with us to illustrate the other requirements<\/span> to ensure successful scale<\/span>-up: responsibility<\/strong>, autonomy<\/strong>, ownership <\/strong>and failure<\/span><\/strong>.<\/p>\n

Responsibility<\/h3>\n

In the past, Coolblue used a hub and spoke model for deploying<\/span> code. The Hosting<\/span> and Deployment<\/span> team<\/span> (effectively DevOps<\/span>) sat in the centre with each development team<\/span> going through them for any decisions\/knowledge about deployment. This model began to be a blocker since all requests had to go through the one team<\/span>. As a result, informal knowledge sharing began to happen.<\/p>\n

\"\"<\/picture><\/figure>\n

Andy in Team A might ask Eve in Team E how to deploy<\/span> a new DB. The problem is that this informal knowledge sharing doesn\u2019t scale<\/span> at all<\/strong>. The solution is to turn the Hosting and Deployment team<\/span> into a Centre of Knowledge. Then the deployment<\/span> team<\/span> becomes just another development team<\/span>. If you couple that with an automated process for deploying a new infrastructure, you end up giving responsibility to the developers<\/span> and empowering them.<\/p>\n

Autonomy<\/h3>\n

Within bounds, autonomy is essential for scaling<\/span>. All of Coolblue\u2019s systems are called Vanessa-X. Their modern systems such as Vanessa-de-Prix and Vanessa-Longstocking, are web applications<\/span> using Serilog, Splunk and DataDog to enable real-time data<\/span> logging, dashboards and audit trails in an easy-to-integrate fashion. However, the company<\/span> is still heavily reliant on Vanessa-Optimus-Prime. This is the original system and is a monolithic desktop<\/span> application<\/span> based on Delphi (which shows how old it is!). The system runs on thousands of machines across the company<\/span> and is still central to how the rest of the system works.<\/p>\n

An autonomous team<\/span> was given the challenge of working out how to add Vanessa-Optimus-Prime to the new data<\/span> logging, dashboard and audit system. \u201cEasy\u201d they thought. Each application<\/span> has a collector class<\/span>. We\u2019ll just use that to collect all the data<\/span> and send it to the logging system. Unfortunately, that kills performance so it doesn\u2019t scale<\/span>. Their next thought was to use UDP<\/span> to send the logs to an agent on the network. This would act as an aggregator. But then security<\/span><\/strong> gently explained to them why this was not a good idea! (Basically, UDP<\/span> traffic can kill a network and becomes a security<\/span> nightmare). Aha! they thought. Why not switch to using TCP? That way you have controlled traffic in the network. Sadly, they hadn\u2019t allowed for the sheer number of concurrent connections needed. The collector ran out of sockets and hung. In turn, this caused all the clients to hang as they waited for connections.<\/p>\n

Finally, the team<\/span> stopped trying to come up with quick fixes and looked at what they could do from an infrastructure viewpoint. They came up with a solution based on the AWS<\/span> API<\/span> gateway<\/strong>. This was then connected to a Lambda<\/span> Function<\/strong> to pre-process the data<\/span> which then was sent to Amazon CloudWatch<\/span><\/strong>. This solution worked well so they finally submitted it to DevOps<\/span> for approval. The only change DevOps<\/span> made was to insert Kinesis webstreams. This makes sure the system can\u2019t become too expensive. Now the system is capable of handling about 5m log events per week.<\/p>\n

\"\"<\/picture><\/figure>\n

Without the autonomy to try out multiple approaches<\/strong>, this problem would have taken much longer to solve.<\/p>\n

Ownership<\/h3>\n

Ownership is sometimes scary. People feel exposed if they have to take ownership of important decisions<\/strong>. At Coolblue, the build environment is based on Team City. But who actually owns the environment? Well, actually teams<\/span> own their own unique build environment. The first thing that happens before any build is the build.ps<\/em> script<\/span> is called. Each team<\/span> can configure this script<\/span> as they choose. As a result, pretty much any build configuration<\/span> is feasible. And no one else even need know what you are trying!<\/p>\n

Failure<\/h3>\n

One of Pat\u2019s favourite books is \u201cFailing Forward\u201d, by John C. Maxwell. Core to Maxwell\u2019s view is that what matters is how failure<\/span> is accepted and what changes it triggers<\/strong>. Coolblue owns a fleet of delivery vehicles. Recently they added electric bicycles to the fleet. During the trial phase for the bicycles, everyone in the team<\/span> suddenly received a Slack notification late one afternoon. One of the key aspects of Coolblue\u2019s infrastructure is the dashboard that monitors their services. If anything goes wrong, it sends out a Slack notification. On this occasion, the on-call team<\/span> was able to quickly spot that two processed were hanging. They terminated and restarted these and within minutes all was happy again.<\/p>\n

\"\"<\/picture><\/figure>\n

So, what has that to do with the delivery bikes, you might ask. Well, during this trial phase, the maximum size of package allowed was set to 50cc. One of the most popular lines Coolblue deliver is a particular coffee machine. That machine comes in a 49cc box. However, this week, the manufacturer had chosen to add a milk frother as a free gift. Suddenly the boxes were too big. As a result, when the bike courier came to load the packages for their delivery round, they wouldn\u2019t fit. In turn, that would put all bikes offline and the whole system would grind to a halt. Knowing this would be a problem, the team<\/span> had reduced the maximum allowed package slightly, thus triggering the system to reassign all the loads properly. However, it turned out that not every instance had been updated properly. Thus, later in the day, these instances hung and triggered the outage.<\/p>\n

After this event, the team<\/span> was required<\/span> to send an RFO (reason for outage). These RFOs are reviewed each month and any important lessons are learned and shared across all other teams<\/span>. If there is a major failure<\/span>, then a War Room is called, where everyone in the company<\/span> is able to help review the issue. Fortunately, Pat said, there has been no need for any War Rooms for a long time.<\/p>\n

Conclusions<\/h2>\n

Coolblue has been able to scale up pretty effectively. In part, this is down to embracing the \u201cfaster to master\u201d checklist. But it is also down to how their company<\/span> culture embraces the four key concepts of responsibility, autonomy, ownership and failure<\/span>. Get these right, and scaling becomes much easier<\/strong>.<\/p>\n