An ultra-low-latency exchange on AWS

Which latency

Speed matters here for two separate reasons, and the second one is easy to overlook.

Low latency means less time passes between seeing a price and acting on it, so participants compete on equal footing.

Predictable latency means a trader knows how long an order will take and whether there is still time to cancel. An exchange that is fast but erratic is harder to use than one that is slower and steady. This is why the numbers in the talk are p99 figures rather than averages.

Fairness is the third requirement. No participant may be structurally faster than another, or trust erodes and regulators take an interest.

Compute

Matching is largely single-threaded work, so what matters is how fast a single core runs rather than how many cores are available. On AWS a lowercase z in an instance name marks the high clock speed family.

Coinbase chose z1d, which gives fast cores, reasonable memory bandwidth and up to 1.8 TB of local NVMe storage. It is not the newest family available. It won on availability, because a newer instance might not exist in the region they needed, or might not be obtainable in the quantity they needed at the time they needed it.

The disk matters more than it first appears. Their system is a Raft cluster, which keeps a leader and several followers in agreement about the order of events. Raft's safety rule is that a node must write a message to disk before acting on it, so that a crash and restart cannot lose something the node has already committed to. That places a disk write in the path of every order rather than in the background, so a slow disk would undo everything else.

The environments explosion

The simple picture is two environments, one for untested code and one for tested code. Coinbase ended up with five kinds, and closer to twenty actual instances once multiplied across accounts and regions.

Every developer gets a personal environment. This came out of going fully remote, when coordinating shared environments between people stopped working.

An integration environment runs nightly and checks that everybody's features work together. Two things are tested there specifically: forward and backward compatibility between versions, and the rolling upgrade mechanism itself. There is only one production system, and getting an upgrade wrong there costs people money.

A sandbox is publicly accessible, carries the newest features and trades paper money. Market makers and client applications need to exercise new features heavily before routing real money through them, so the exchange is not ready when Coinbase's code works, it is ready when the participants' code works.

Shadow production runs a newer version quietly in the background, fed the same mirrored input as the real system, so its behaviour can be watched against genuine market traffic before anything is switched over.

Production itself gets the best hardware and configuration and is operated carefully.

Three layers

Their answer to that multiplication is to divide the software into three layers, each permitted to know different things.

The alternative is worth picturing. Somewhere in your code is a line that connects to 10.0.3.47. That works until the same line needs a different address in sandbox, another in Tokyo, and another in an account created last week. The identity of every environment ends up spread through application code, and each new environment means editing it.

The infrastructure layer provisions real AWS resources from a single template applied across accounts, regions and hardware. Expensive features live in that template as switches, enabled only where they matter, because a production grade setup cannot be afforded in every developer's environment. Yucong stresses that fast deprovisioning matters as much as fast provisioning.

The orchestration layer holds the mapping from pool names to actual machines. A developer names a pool and the orchestrator decides which machine and which network. Facing downward, infrastructure presents a uniform interface, which means service discovery rather than addresses written into configuration.

The application layer is composable and testable without the full infrastructure standing up. If code asks for the orderbook database by name rather than by address, that name can be pointed at a local file, a stub, or the real cluster, and the business logic is unaffected.

Yucong names both directions of the benefit in his closing remarks. Application developers stop having to think about infrastructure, and infrastructure engineers stop being the limiting factor on what application developers can build. The second point is the less obvious one, because in the tightly coupled version every feature that needs a new resource waits behind an infrastructure engineer.

The network

This section assumes some background, so it is worth setting out the pieces first.

A VPC is your own private network inside AWS. You choose an address range, machines launched into it receive addresses from that range, and they can reach each other. A fresh VPC reaches nothing else at all, including the internet, other VPCs, and AWS's own services. Connectivity is something you add deliberately, usually by attaching an internet gateway.

A load balancer is a single fixed address in front of several identical machines, passing each connection to one that is healthy. A public one is reachable from the internet and an internal one is not.

A cluster placement group is a request that AWS pack your instances physically close together inside a single availability zone, so that packets between them travel the shortest possible path.

Coinbase never added the internet route to production. That is what the slide means by a non-routable production VPC, and the reasoning is that without a path out, an attacker who gets in still cannot move data anywhere. Removing that route creates three problems, and each of the remaining boxes solves one of them.

The firewall VPC covers the things that genuinely need the internet. Production's route table sends outbound traffic there to be inspected and logged, and they built alerting on top of it that fires when something starts pushing unusual volume outward.

The AWS service endpoint covers S3, CloudWatch and similar services. AWS places an interface for one specific service inside your VPC at a private address, and DNS inside the VPC resolves to it, so application code is unchanged and the traffic never leaves AWS's network.

PrivateLink covers services living in another VPC. It exposes exactly one endpoint into production at a private address, leaving the two networks otherwise separate. VPC peering would also work and would be slightly faster, but it joins the networks so that anything on either side can reach anything on the other.

Trading traffic takes none of these detours. It arrives from the internet, passes through the public load balancer, and goes straight into the placement group. The firewall introduces latency, so the hot path does not touch it.

Many services need production data and production trust without needing microseconds, including balances, reporting, monitoring and administrative tooling. Those run inside the production VPC but outside the placement group, which is reserved for the ten or so hops involved in an order round trip.

Hardware and OS tuning

Renting a fast instance achieves little on its own. Yucong says the tuning improved their p99 by an order of magnitude or more.

NUMA

A large server contains more than one CPU chip, and each chip has its own RAM attached to it. Reading memory attached to your own chip is fast, while reading memory attached to the other chip crosses a link and takes longer, and nothing in the code indicates which of the two you received. The usual remedy is to pin threads and memory to matching chips. The z1d sizes Coinbase uses sit within a single memory domain, so the problem does not arise.

Hyperthreading

Hyperthreading presents each physical core as two logical cores, running a second thread during the moments the first is stalled waiting on memory. It generally increases throughput, which is why it is enabled by default almost everywhere.

The two logical cores share the physical core's L1 and L2 cache, which is the small pool of very fast memory next to the core. A matching engine's working set lives there, where a read costs a few nanoseconds rather than around a hundred. If the operating system schedules an unrelated thread onto the sibling core, that thread's data evicts yours, and your next read becomes far slower at a moment you cannot predict. The average barely moves while the tail gets much worse, which is the wrong trade for a system judged on p99.

Disabling hyperthreading gives the thread the whole core and the whole cache. Isolating the sibling logical CPU achieves the same result more precisely. Either approach exchanges some throughput for consistency.

Local NVMe

Instance store is a physical SSD inside the host machine, reached directly rather than over the network, which puts it in microseconds rather than milliseconds. It is also ephemeral, so if the instance stops, the disk is gone. Coinbase accepts that because Raft already replicates across nodes, so durability comes from the cluster rather than from any single disk.

The slide leaves one question open, which is whether to call fsync. When a program writes to a file, Linux holds the data in memory and flushes it to the device later. Calling fsync forces the write to reach the device before returning. Without it, writes are nearly free but a power failure takes the data with it. With it, the write is genuinely durable and every message on the hot path has become slower.

Raft's correctness argument assumes fsync is used. The pragmatic counterargument is that with several replicas each holding the log, losing power on all of them simultaneously is unlikely enough to accept. It is a judgement about correlated failure rather than a question with a correct answer, which is presumably why the slide leaves it as a question.

Reliability

Yucong's framing throughout is that giving something up in one dimension means winning it back in another. Placing the whole trading system in a single availability zone is the concession, and this section is what pays for it.

Real-time monitoring

Conventional monitoring samples metrics every fifteen or sixty seconds and forwards them to a service. A great deal happens in a single second on an exchange, so by the time such a system reports a problem, the opportunity to react has passed.

Stale data also rules out a control loop, meaning automation that responds on its own rather than paging a person. Bringing up a new version, confirming it is healthy and cutting traffic over is a control loop, and it requires signal you can trust in the moment. Without that you end up guessing, adding a wait here and a pause there, until a deployment takes four hours.

Their services push metrics directly to a central location inside their own stack instead of exposing anything to a third party agent over the internet. It is faster, and it also fits a production network with no route out.

Operations as code

The ordinary way to change something in production is for a person to log in and type. Nothing is reviewed, nothing is recorded, and engineers need standing access to production.

Instead the action is written as code and submitted for review, and a machine applies it. Their example is a job that generates mock trading activity in the dev environment, where a developer wants to add an environment variable, and a reviewer can see exactly what will change beforehand.

The second effect is that standard procedure gets enforced by construction rather than documented in a wiki. Cluster upgrades always proceed one node at a time and blue-green deployments always follow the same sequence, with no patching, hotfixing or manual step available.

Yucong adds that the important part is making this painless, because otherwise developers will find a way around it. A control that is irritating enough gets bypassed, and then you have lost both the control and any accurate picture of your own process.

Deployment needs spare machines

Blue-green deployment means standing up a complete second copy, checking it, moving traffic across, and then retiring the old one. The switch is instant and rollback is easy, but capacity requirements double for the duration. Rolling upgrades replace one node at a time, which is cheaper but leaves old and new versions running side by side, which is why compatibility is tested so carefully in the integration environment.

Either approach needs spare machines at the moment of deployment, and specifically z1d instances, in the correct availability zone, inside the same placement group. Yucong describes the combined constraint as close to an NP-hard problem.

Capacity reservation is what makes it workable, which means paying to hold that capacity in advance so it is waiting when needed. He notes that most people do not use it, and that for Coinbase it was the difference between deploying regularly and having a production environment that would take years to upgrade.

Replicating the input rather than the output

The usual approach is to replicate a system's outputs by copying the database and shipping the logs. Coinbase replicates the input stream instead, because the trading system is deterministic and the same inputs always produce the same outputs. Failing over means replaying the input stream through a fresh process until it reaches the identical state.

They run two availability zones within a region with replication between them, plus an offset backup on a different schedule, and rely on Aurora for the database side.

The residual risk is stated plainly. Remaining in one availability zone means an availability zone failure could lose data, and that is a business risk accepted deliberately in exchange for the latency. What they do not accept is a single point of failure inside that zone, which is a much more tractable problem.

Security

The stated goal is that production should be secure by default rather than dependent on the software being correct, so that the network still holds when code has a bug.

• AWS Shield handles denial of service, covering traffic originating inside the region as well as from the internet.
• Traffic observability sits on the firewall VPC, with automatic alerts when outbound volume looks unusual.
• IAM policies separate privileges. For money movement, one component can request a transfer and a different component executes it, so no single component can do both and there is no all-powerful system.
• mTLS and service authentication protect any call crossing a VPC boundary.

What comes next

Clients currently connect over the public internet to a network load balancer. For clients already running inside AWS, PrivateLink lets that traffic use the AWS backbone instead, which reduces jitter and makes latency more predictable.

Shared cluster placement groups now work across accounts, so a trading firm can place its own instances in the same placement group as the exchange gateway and peer its VPC across. That reduces physical distance, which is the only change that genuinely lowers latency.

Further out they mention kernel bypass, which the Aeron messaging library supports natively, and enhanced network adapters such as ENA Express.

Summary

• Predictable latency matters as much as low latency, which is why everything is measured at p99.
• Twenty environments is a normal outcome, and it is why none of them can be built by hand.
• The layers exist so that only one place knows real addresses, which turns adding an environment into a mapping rather than a code change.
• Removing production's internet route forces a deliberate, narrow door for every legitimate need.
• Fast hardware without operating system tuning achieves little, because cache behaviour and scheduling determine the tail.
• Every gain is paid for somewhere. The placement group costs capacity planning, the firewall costs latency, and a single availability zone costs disaster tolerance.
• Determinism allows inputs to be replicated instead of state.