Grokking the Solutions Architect Interview
Vote

0% completed

​

Choosing Compute: Servers, Containers, and Functions

Three Ways to Run Code

How Capacity Grows

One Day of Traffic

Choosing by Workload Shape

The Online Store Decision

Using This in an Interview

Key Takeaways

Practice Questions

Title image: three ways to run the online store, virtual machines, containers, and functions, each shown as a card with its typical start time.
Title image: three ways to run the online store, virtual machines, containers, and functions, each shown as a card with its typical start time.

At 4 a.m., the retailer's online store receives about 30 orders an hour. At 8 p.m., its busiest normal hour, it receives about 400. On the first morning of the holiday sale, it receives about 2,000 orders in the hour after the sale opens at 9 a.m. and 2,400 by 10 a.m.

In the old data center, three fixed servers carried this traffic. Each server handles about 200 orders an hour within the target response time, so the three together served 600 orders an hour. That was enough for any normal day, but it was only a quarter of the sale peak.

This lesson compares the three main ways to run code in the cloud against that traffic pattern. For each one, it shows the billing, the speed of growth, and the work left to the team.

Three Ways to Run Code

A cloud platform runs code on virtual machines, in containers, or as functions, and each option moves a different part of the work to the provider.

A virtual machine (VM) is a server that the provider creates from shared hardware in a few minutes. You choose its size and operating system. You also install and patch the software, and you pay for each second it runs. On AWS, VMs are EC2 instances.

A container runs an application from a packaged file (an image) that holds the application and its libraries, so that it starts the same way on any server, usually in seconds. A managed orchestrator is a service that places containers on servers, restarts failed ones, and adds more with traffic. On AWS, ECS and EKS are orchestrators. With Fargate, an option for both, the provider also runs the servers underneath, so you pay for the processor and memory each container reserves.

A serverless function is code that the provider runs only when an event arrives, such as a web request or a new file. You pay for each request and for each millisecond of run time at the memory size you choose, and you pay nothing when no event arrives. On AWS, functions run on Lambda, and one call can run for at most 15 minutes. The first request to a new copy of a function waits while the provider starts it, which is called a cold start.

Azure and Google Cloud offer equivalents of all three options under different names.

OptionBilled byTypical start timeMain limitTeam effort
Virtual machineSecond of run time3 to 5 minutesYou size and patch itHighest
Container on FargateReserved processor and memory per secondUnder a minuteApplication must be packaged as an imageMedium
Function on LambdaRequest and millisecond of run timeUnder a second, longer on a cold start15 minutes per callLowest

The start times are typical values for a small application. A large image or a slow application start makes every option slower.

How Capacity Grows

An auto scaling group is a set of VMs that the platform keeps between a minimum and a maximum count, adding or removing VMs by a rule. The most common rule is target tracking: the group keeps a measured value near a target, such as average processor use at 50 percent.

New capacity does not arrive at once. With one-minute monitoring turned on, the rule usually waits for a few readings, about 2 minutes in all. The VM then takes 3 to 5 minutes to start the store application. Finally, the load balancer, the service that spreads requests across the VMs, checks that it answers. In total, a new VM usually serves its first order 5 to 7 minutes after traffic rises. A container waits for the same readings but starts in under a minute, so it serves in about 3 minutes.

How long new capacity takes to serve traffic: about 6 minutes for a virtual machine, about 3 minutes for a container, and about 1 second for a function.
How long new capacity takes to serve traffic: about 6 minutes for a virtual machine, about 3 minutes for a container, and about 1 second for a function.

On a normal evening, traffic rises from 200 to 400 orders an hour over several hours, so a delay of 6 minutes does not matter. At the sale, traffic rises from 600 to 2,000 orders an hour within minutes. A group that reacts only to processor use would run short of capacity during the busiest minutes of the year.

The fix is scheduled scaling, a rule that changes the group's size at a set time. The retailer's group raises its minimum to 12 VMs at 8 a.m. on each sale day, because 2,400 divided by 200 is 12. Target tracking still runs above that minimum and adds VMs if traffic exceeds the plan.

One Day of Traffic

The diagram compares the capacity of each approach on the first sale day.

Orders an hour on the first sale day against capacity: three fixed servers stop at 600, while the auto scaling group rises to 12 virtual machines at 8 a.m. and stays above traffic.
Orders an hour on the first sale day against capacity: three fixed servers stop at 600, while the auto scaling group rises to 12 virtual machines at 8 a.m. and stays above traffic.

The fixed servers lack capacity from 9 a.m. to about 11 p.m. At the last sale, this shortage made the store unavailable for 3 hours on the first morning. The auto scaling group is ready one hour before the sale opens.

Choosing by Workload Shape

The shape of the load over time decides the compute option more than the application's language or size.

Workload shapeRetailer exampleFirst choiceReason
SteadyERP system, similar load all dayVMs, rehostedNo scaling needed; a discount for committing to a year or more of use lowers the price
SpikyOnline store, 6 times normal in sale weekAuto scaling VMs or containersCapacity follows traffic, with scheduled scaling for known peaks
Event-drivenResizing each new product imageFunctionsRuns only when a file arrives, nothing to pay between uploads
Long-running batchNightly sales report, about 40 minutesContainers or VMsOne function call stops at 15 minutes

Functions become expensive when traffic is high and steady, because per-request billing then costs more than a VM that is busy all day.

The Online Store Decision

The SA compared two ways to provide sale capacity. Fixed VMs sized for the sale would run 12 VMs all year: 12 times 8,760 hours is 105,120 VM-hours. The auto scaling group runs about 2.5 VMs on an average normal day and 12 during the sale week.

Normal days: 358 days x 24 hours x 2.5 VMs = 21,480 VM-hours Sale week: 7 days x 24 hours x 12 VMs = 2,016 VM-hours Total: 23,496 VM-hours 23,496 / 105,120 = 0.22, about 22 percent of the fixed option

VM-hours are not cost, because prices and discounts differ, but the gap explains most of the saving in the retailer's estimate.

The SA chose VMs in an auto scaling group and rejected the other two options. Containers would scale faster, but the store was not packaged as images. The 12 engineers would also have had to learn an orchestrator before the sale in month 9. Functions would have required rewriting the store, and cold starts at 9 a.m. would affect the first customers of the sale. The proposal recorded containers as a later step, after the third wave.

In the sale in month 9, the busiest hour reached about 2,600 orders. Target tracking added a 13th VM above the scheduled 12, because 2,600 divided by 200 is 13.

Using This in an Interview

When you choose compute, name the workload shape first, then the option, then the option you rejected and why.

"The store runs 400 orders an hour normally and 2,400 in sale week, and each server handles about 200. I would use an auto scaling group with a minimum of 2 VMs and a scheduled rise to 12 before the sale opens, because metric-based scaling takes about 6 minutes. I would not use functions, because the store would need a rewrite and the team has a fixed deadline."

A mid-level answer chooses a sensible option. A senior answer also states the capacity arithmetic, the scaling delay, and the team's ability to run the choice.

Expect two follow-up questions:

  • "Why not containers?" They start faster, but the team must first package the application and learn an orchestrator. Plan them for after the migration.
  • "What if the sale is twice as big as planned?" Target tracking adds VMs above the scheduled minimum, so set the maximum well above 12, such as 20, and test it.

Key Takeaways

  • VMs, containers, and functions differ in billing, start time, limits, and the work left to the team.
  • New VMs take about 5 to 7 minutes to serve traffic, so known peaks need scheduled scaling.
  • Choose by workload shape: steady, spiky, event-driven, or long-running batch.
  • The retailer's store uses an auto scaling group because the team could run it before the sale.

The next lesson, Choosing Storage and Databases, matches each kind of retailer data to the store that suits it.

Practice Questions

1. Why does target tracking alone fail at the start of the sale?

<details><summary>Show answer</summary>

Because new VMs take about 5 to 7 minutes to serve traffic. At 9 a.m., traffic rises from 600 to about 2,000 orders an hour within minutes. Scheduled scaling starts the 12 VMs an hour before the sale opens.

</details>

2. The nightly sales report takes about 40 minutes. Can it run as one Lambda function call?

<details><summary>Show answer</summary>

No, because one call can run for at most 15 minutes. Run it as a container task or on a VM, or split it into parts that each finish within the limit.

</details>

3. The sale grows to 3,600 orders an hour. How many VMs does the store need?

<details><summary>Show answer</summary>

About 18, because 3,600 divided by 200 is 18. The scheduled minimum should rise to 18, and the maximum must be above that number. A new load test should confirm that the database also handles the higher rate.

</details>

Reading Progress

0%


Vote for new content

On This Page

Three Ways to Run Code

How Capacity Grows

One Day of Traffic

Choosing by Workload Shape

The Online Store Decision

Using This in an Interview

Key Takeaways

Practice Questions