When I last wrote about our DNS propagation pipeline in 2022, the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events, such as zone setting changes. However, the number of events was never the real problem. The problem was the kind of work some of those events needed, and how that work grows with the size of a zone. Some zones now hold more than 20 million records, and we expect them to keep growing toward hundreds of millions.

Each event starts a clock that measures how long it takes us to turn that change into data our authoritative DNS servers can use. We call that work a build. A build reads the current database state, converts it into the format used at the edge, and writes any changes to Quicksilver.

In 2022, per-record builds were 150 times faster than full builds on average. On a test zone with one million records, per-record builds were 4,250 times faster. We ended that post with the obvious next step: do the same thing for zone settings.

Why doesn't the Zone Builder just rebuild the setting that changed?

It should, and that is where we ended up. However, proving when it was safe took most of the work. A record event identifies what changed. A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Choosing too much work wastes capacity. Choosing too little can leave stale or incorrect DNS data at the edge.

We built a targeted build planner that chooses the smallest build we can prove is safe, and a new scheduler that combines work for the same zone. Zone setting changes no longer rebuild zones with tens of millions of records. Together, they cut peak memory by 85%, and the extra headroom let us safely shorten waits the pipeline has always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.

How DNS changes reach the edge

A DNS zone is the collection of records and settings for a domain. Cloudflare uses a multi-stage pipeline to turn a change to that zone into data available across our global network.

  1. A customer changes a DNS record through the DNS Records API or dashboard, or another Cloudflare service changes a zone setting.
  2. The change is stored in a database.
  3. The database change or another system produces a Kafka event.
  4. The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder.
  5. The Zone Builder reads the required zone data and converts it into keys understood by our authoritative DNS servers.
  6. The Zone Builder compares those keys with Quicksilver, our distributed key-value store. It writes only what changed, and Quicksilver propagates those changes across Cloudflare's network.

This is a simplified view. The Zone Builder does not copy database rows directly to the edge. It combines records with zone settings, applies DNS serving rules, and organizes the result so our authoritative DNS servers can read it quickly.

Why full builds stopped scaling

The original Zone Builder had one useful property: it did not need to know why a zone changed. It rebuilt the desired state from the current database state.

  1. Read the current zone and its settings.
  2. Read every DNS record in the zone.
  3. Convert those records into Quicksilver keys.
  4. Compare the new keys with those already stored.
  5. Remove stale keys, write changed keys, and update zone metadata.

The final write contains only changed keys. However, a full build still reads, encodes, and compares every record, and holds the whole zone in memory while it does so. That is cheap for a small zone. For our largest zones, with tens of millions of records, the zone does not fit in memory at all. I explain how we handle that below.

Zone settings made targeting harder. Universal SSL is one example. Enabling or disabling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain, such as example.com. CAA records tell certificate authorities whether they may issue a certificate for the domain.

A full build only needed to know that the zone changed. A targeted build must answer three questions:

  1. Which DNS names can change?
  2. Can shared metadata or the list of published zones change?
  3. What previous state must be removed from Quicksilver?

Answering those questions safely meant solving five problems, which the rest of this post covers in order:

  1. Events had to say what changed.
  2. Each change had to map to the smallest build that is still correct.
  3. Builds had to account for data shared by every zone with the same name.
  4. The database could change between the event and the build.
  5. Combining work could not lose a Kafka message.

Making events say what changed

We changed the event producer to include changed zone fields and settings, with previous and new values when available. We then traced each setting through the code that generates DNS output and classified what it could affect.

Not every event was that specific. Some database functions only update a timestamp to say, in effect, "something about this zone changed." Sometimes that is harmless bookkeeping. Sometimes it means data changed elsewhere and must be read again.

Those cases look identical in the event. Until we can identify the caller and its purpose, a timestamp-only update still causes a full build. The same rule applies to missing details, unknown events, or any change that may affect records anywhere in the zone. The planner never turns uncertainty into less work.

Choosing the smallest safe build

Quicksilver output has two parts that matter here. Record keys contain the DNS answers for one zone and name. A smaller shared metadata key tells our authoritative DNS servers how to use that zone, including where to find special records such as nameservers and wildcards. Some events change only record keys; others must update both.

With that distinction, the planner can choose one of five build types:

  1. Resources only: Rebuild the records at specific DNS names, which we call resources, without changing zone metadata. Ordinary A, AAAA, TXT, and CNAME changes usually take this path.
  2. Resources and metadata: Rebuild specific names and shared metadata. Nameserver and wildcard changes use this path.
  3. Apex and metadata: Rebuild the zone apex and metadata. Settings such as Universal SSL can use this path.
  4. Purge: Remove a zone that should no longer be published.
  5. Full zone: Rebuild every record when smaller work cannot be proven safe.

The rule is simple: every key a targeted build writes must be byte-for-byte identical to what a full build would write. If we cannot prove that, we do a full build.

Removing old state works as it did for per-record builds in 2022. Record events include deleted records and the previous version of updated ones. The Zone Builder then scans Quicksilver for the existing keys under each name it rebuilds, removing any that the new build did not write.

Full builds remain necessary when a change can affect records anywhere in the zone or when a zone may need to be published for the first time.

There is one deliberate exception. A full build holds every record from the database and every existing Quicksilver key for the zone in memory at once, and our largest zones do not fit. Above a fixed record-count threshold, the Zone Builder never escalates to a full build. It rebuilds the apex, the metadata, and every name identified by the queued events instead. We accept this tradeoff because a full build of those zones cannot finish, and, while it retries, it holds back every other change on its Kafka partition. Many of the cases that would otherwise need a full build involve several zones sharing one name, and zone holds reduce that risk by stopping the same name from being added to another account. For the rare cases that remain, such as enabling a zone on another network, we can manually rebuild individual names in the zone.

When one name means more than one zone

Quicksilver stores each zone's record keys separately. It also stores one metadata key shared by every Cloudflare zone with that domain name. This shared key acts like a directory: it tells our authoritative DNS servers which internal zone IDs they may use when answering a query.

Why can one name have several zone IDs? During customer onboarding, offboarding, and other lifecycle changes, multiple Cloudflare zone objects can briefly exist for the same public domain. We use a priority policy to choose a small, bounded set to publish.

Consider four internal zones for example.com: Zone 101, Zone 202, Zone 303, and Zone 404. The metadata key currently lists Zones 101, 202, and 303. Zone 404 exists, but it is not selected. The numbers identify separate Cloudflare zone objects; customers still see one domain name.

A TXT record change at www.example.com in Zone 202 affects only Zone 202's record keys, so the metadata key does not change. Nameserver and wildcard changes are different because the metadata key records information about them. Those builds must update both the zone's records and the metadata key.

At execution time, the Zone Builder reads the current list again. If it writes the metadata key, it rebuilds the whole key from that fresh view rather than patching an older copy. It writes the triggering zone's records before the metadata that refers to them.

Planning from the event, verifying before writing

An event describes what happened in the past. A build may wait while more events arrive and the database changes, so planning happens twice. The event first creates a conservative work list. Immediately before execution, the Zone Builder reads a fresh publication snapshot: the current zone settings, whether this Zone Builder should publish the zone to its network, and which same-name zone IDs belong in the metadata key. Here, a network is a separate, isolated copy of Quicksilver for one environment. Each Zone Builder deployment writes to one network, and a zone can be enabled on some networks but not others. For example, a zone can be served from Cloudflare's global network and also from the China Network.

Fresh state can change the result in four ways:

  1. A targeted build can become a full build if the zone now has a feature that requires complete-zone work.
  2. A build can become a purge if the zone should no longer be published to this network.
  3. A build can be skipped if the triggering zone no longer belongs in the metadata key.
  4. A metadata build can reconstruct the shared key from the current candidate set.

The fresh read comes from read-only copies of the database, called replicas, so builds do not compete with customer writes. Kafka delivery and database replication move independently: an event can arrive before its database change is visible on a replica. The DNS Kafka Aggregator delays events long enough to cover normal lag, and a final database check rejects the read if that replica is still behind.

Combining changes in one scheduler

Record work and full-zone work previously used different schedulers. Each could combine its own work, but not work from the other scheduler. We replaced them with one unified scheduler. The planner decides what must be built, and the scheduler decides when it can run.

When several events for one zone arrive before a build starts, the scheduler combines their work:

  1. An A record changes at www.example.com, requiring that name.
  2. A wildcard changes at *.example.com, requiring that name and shared metadata.
  3. The scheduler combines both names and the metadata update into one build.

Merging only adds work. If any event requires the complete zone, the combined build becomes a full build. A purge overrides every rebuild because its desired result is that the zone is no longer published.