A power outage knocked NIST's internet time servers out of perfect sync

Alfonso Maruccia

Posts: 2,507   +934
Staff
What the Clock: Internet time servers are a critical part of the infrastructure used by companies and organizations that rely on atomic clocks to keep accurate time. A US agency responsible for measurement and standards is warning that several of its time servers may no longer be able to deliver precise time signals without relying on external sources.

The National Institute of Standards and Technology recently warned that an atomic clock device installed at its Boulder campus had failed due to a prolonged power outage. As a result, the Internet Time Services hosted at the campus are currently unable to provide an accurate time reference to external organizations via the Network Time Protocol NTP.

Time servers and NTP are among the oldest networking technologies still in widespread use, providing a practical way to synchronize clocks over the internet using data packets. According to NIST's notice, the following NTP servers were affected by an unexpectedly long power outage:

  • time-a-b.nist.gov
  • time-b-b.nist.gov
  • time-c-b.nist.gov
  • time-d-b.nist.gov
  • time-e-b.nist.gov
  • ntp-b.nist.gov

The US agency said that the Boulder team quickly moved to secure alternate power sources, so that the "hydrogen maser clocks" would survive after the battery backups had been depleted. NIST also provided some additional details about the outage, which forced the campus to shut down many parts of the building's air-cooling system.

The initial power loss did not have an immediate impact on NIST's atomic clock services, as they rely on additional standby power generators. However, during a subsequent inspection, the team discovered that at least one of these critical generators had failed. As a result, the primary signal distribution chain of the atomic clock system was disrupted and temporarily lost synchronization with Coordinated Universal Time.

Jeff Sherman, team leader for NIST's atomic clock systems, later confirmed that utility power was restored a few days after the incident. While the Boulder Internet Time Services themselves did not lose power thanks to backup and battery generators, the NTP signal lagged behind universal time by approximately five millionths of a second.

Sherman noted that the NIST time system is typically about 5,000 times more precise than a deviation of a few microseconds. Timekeeping accuracy at the nanosecond scale is essential for telecommunications, integrity monitoring systems, positioning technologies such as GPS, and other critical scientific and industrial applications.

The NIST-F4 atomic fountain clock hosted at the Boulder campus came online just a few months ago, following years of assembly and testing. The agency has described the clock as one of the world's "elite" internet timekeepers and a cornerstone of more stable and secure global time measurement. Yet a series of unusually strong wind gusts – and the resulting damage to utility power lines – has now exposed vulnerabilities that threaten the project's reliability.

Permalink to story:

 
Meanwhile a power outage in San Francisco this weekend caused Waymo vehicles to stop functioning at some intersections and create gridlocks… Waymo took the service offline and their vehicles had to be towed away. Waymo said their cars were stuck because they were waiting to “confirm the state of the affected intersections”: https://www.nytimes.com/2025/12/21/...cisco-after-problems-during-power-outage.html

 
If you know the clock is out of sync, and more importantly by how much, there is obviously a baseline that you can compare it to. Either re-sync the clock, or include an adjustment value in NTP server messages.
 
With the growing demand for power—especially from data centers—it’s time to levy impact fees on these facilities to fund burying our entire electrical grid, extending fiber to every property, and building clean, modern power plants. Yes, the price tag sounds enormous to individuals, but for data centers it’s simply a cost of doing business. We can turn a growing problem into a long‑term solution. Residential power costs are already rising, and some grids in the Midwest and Eastern U.S. are now dipping into the 20% buffer zone. It’s a clear signal that the current model isn’t sustainable.
 
First thing I thought was it was Trump's fault for cutting personnel. Still not entirely sure he's not.
 
Back