How one malfunctioning router in Arkansas grounded every flight in America last week

Skye Jacobs

Posts: 2,300   +64
Staff
What we know so far: The nationwide ground stop that froze US air travel on Friday didn't start at a major hub or a busy control center. It began at a small FAA radio site in El Dorado, Arkansas, where a router installed during routine upgrade work began flooding the agency's private network with connection requests that the older equipment it connected to couldn't handle. Soon after, the links between pilots and controllers and the radar feeds that track aircraft started failing.

The result was more than 3,500 flights held for about five hours, leaving tens of thousands of travelers stranded.

The volume of traffic was far beyond anything the network normally handles. "We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds," Rebecca Guy, the FAA's technology chief, said on Saturday at a press conference.

El Dorado was one of those two places. The other was a radar facility in Amarillo, Texas, which had been upgraded several weeks earlier and showed a second problem that officials have not described. Engineers took both sites offline, but by then each had been hitting the network hundreds of times per second. That was enough to knock out other parts of the air traffic communications system, and the delays and cancellations carried into the weekend.

The Arkansas work was part of a long-running FAA effort to move its telecommunications from old copper wiring to fiber-optic cable. L3Harris, which owns and operates the network, has worked with the FAA on modernization for more than two decades and says it has finished more than 70% of the planned conversion. Crews chose to work on the El Dorado station because it seemed remote enough that a problem there would cause little harm, an assumption that turned out to be wrong.

The hardware involved is not the kind of router found in a home office. The FAA runs tens of thousands of these networking devices, including routers and switches, and they are larger, industrial-grade units built for harsh environments, Michael McCormick, a professor at Embry-Riddle Aeronautical University in Daytona Beach, Florida, told The New York Times.

They sit on a closed network meant to keep out cyberattacks. They also have to read and process radar data, flight plans and audio, then send each piece to the right place. The agency's equipment comes from Cisco Systems and Juniper Networks, which Hewlett Packard Enterprise bought last year, and the FAA has been slowly swapping older Cisco routers for newer Juniper models.

Aviation experts think the failure came from a speed mismatch. A new device that pushes out data faster than older equipment can answer can stall the older gear, and the trouble spreads from there. R. John Hansman, a professor of aeronautics and astronautics at the Massachusetts Institute of Technology, said the new equipment pushed older parts of the network beyond their usual pace. "Now you have multiple parts of the old system that are running much faster than they ever ran," he said. "You never had this problem, because you never had all of the elements running at that max speed."

When the network began to falter, McCormick said, it became "unpredictable and unreliable," and stopping departures was the right decision. "It was a smart decision to actually decide to ground-stop all air departures," he said.

Investigators are now trying to determine whether the router was defective or whether something went wrong during installation. Transportation Secretary Sean Duffy confirmed the problem on Sunday during an appearance on Face the Nation. "We actually have both routers with L3Harris, who operates and owns our network. FAA is on site as well. FBI is offering support. We're looking at that right now, and when we know what the root cause of this issue was, was it the install from L3Harris? Was it the router?" he said. "I'm going to tell you and the American people what we find."

The FAA says it has found no evidence of a cyberattack. L3Harris said it is helping with the investigation.

Not everyone accepts the official explanation. Robert W. Mann Jr., who founded an airline analysis and consulting firm, doubted that one device could cause this much damage. "The idea that you would have a single router take down a national network is just not plausible," he said. He saw the outage as evidence of a long-known weakness. "What we really have here is a demonstration that what we knew a decade ago was problematic done at small scale is really, really problematic done at nationwide scale."

The episode raises questions about redundancy in a system where trouble at one small facility could stop flights nationwide. Duffy and other officials blamed the aging infrastructure and said Congress needs to provide more money for upgrades. Experts warned that speeding up the work brings its own risks, since the legacy network must stay in service while new equipment is installed and fully tested. Some suggested doing hardware and software changes during night shifts, when far fewer planes are flying, rather than during busy daytime hours.

Permalink to story:

 
Another excuse to blame the previous admin?

I can't say I trust the current admin to own up to anything.

Since this would amount to a denial of service attack, although it came from the internal network, it seems that the network is not really hardened against such attacks. I get it, you don't expect such an attack from within, but you should be ready to handle such an attack IF it ever were to happen.

That is, IMO.
 
"One malfunctioning router in Arkansas grounded every flight in America last week"

Or so they say....the very next day Microsoft also had a world wide meltdown.

mmm....
Correlation does not equal causation. AI conspiracy theories are the next step in AI psychosis.
Another excuse to blame the previous admin?

I can't say I trust the current admin to own up to anything.

Since this would amount to a denial of service attack, although it came from the internal network, it seems that the network is not really hardened against such attacks. I get it, you don't expect such an attack from within, but you should be ready to handle such an attack IF it ever were to happen.

That is, IMO.
The issue with this idea is that the systems here are obsolete. They predate Microsoft. As in, the company, not just windows. They needed replaced decades ago.

Which was progressing but trump shut down in 2018. So......
 
The issue with this idea is that the systems here are obsolete. They predate Microsoft. As in, the company, not just windows. They needed replaced decades ago.

Which was progressing but trump shut down in 2018. So......
Yep. I was talking with my wife yesterday, and it seems like Trump wants to make everything cheaper for the federal government. The problem with that is that the costs will be pushed elsewhere, as I see it, and whatever "solution" he comes up with, the average citizen of the US will pay the same, if not more.

Kudos to you for pointing out the truth. He promised to upgrade infrastructure in the US, and never came through with that, either.

As I see it, everything this administration touches turns to crap.
 
The WSJ reports that there are about 60 of these Juniper routers on the network. So if #59 and #60 caused this problem by broadcasting large amounts of unintended messages, either the router was defective (which Juniper naturally disputes) or the router had a bad configuration.

It's been a minute since I've done this type of work but the latter sounds all too likely to me.

Then again I also did once long ago receive two routers from the same manufacturer which arrived with the same hardware network address (mac address) on ports on both. This should never happen and caused unusual and hard to diagnose symptoms as soon as they were both online and connected.

Eventually someone will point the finger at whoever made the last mistake with these particular routers, but in my mind the larger and remaining issue will be the possibility of huge consequences from a foreseeable mistake. The test, rollout, and mitigation processes all need improvement.
 
Yep. I was talking with my wife yesterday, and it seems like Trump wants to make everything cheaper for the federal government. The problem with that is that the costs will be pushed elsewhere, as I see it, and whatever "solution" he comes up with, the average citizen of the US will pay the same, if not more.

Kudos to you for pointing out the truth. He promised to upgrade infrastructure in the US, and never came through with that, either.

As I see it, everything this administration touches turns to crap.
Midas in reverse?
 
Syn flood or incorrect routing protocol configured on a new router, and the idjits are out in full force blaming Trump...
 
Why are they trying to merge the two network systems so soon? If I were doing this I would run the networks separate but monitored by a human in tandem, for a really long time, before trying to merge them? Make sure the new systems are solid before merging the data because you're flying people around in the air and that's pretty damn critical.
 
Back