№ 009 · Release
What a nationwide ground stop shows about redundancy: staged updates and flood limits in Mru Field
On November 13, data flooded the FAA's network from two remote sites, one where a new router was going in, and departures across the US were held for more than four hours. One change can stop a whole system in two ways: a bad update that reaches every copy, or a part with right answers that floods what it shares. Mru Field handles both.
What happened on November 13
At about 3:30 p.m. ET on Friday, November 13, the FAA had a technical problem, and it stopped departures at airports across the United States. It lifted the stop at 8:15 p.m. ET. Instability began at 17 of its 22 Air Route Traffic Control Centers and spread into several approach control facilities. Operators canceled about 12,462 flights that day. The FAA says not all of them were caused by the ground stop (FAA).
The FAA's account, November 15. The FAA's older equipment became unstable when data flooded its telecommunications network at 1,000 connection attempts every 4 to 5 seconds. Its initial investigation points to two sites: a radar site in Amarillo, Texas, and a radio site in El Dorado, Arkansas, where its network contractor, L3Harris, had been doing work that included a new router. When L3Harris blocked the two sites, the network stabilized. The FAA turned off a broadcasting function on about 60 similar routers, and it says there is no indication of a cyberattack (FAA). Transportation Secretary Sean Duffy said the router was part of a switch from copper wire to fiber. Whether the installation or the router itself was at fault is still under investigation (CNBC).
The first accounts, November 13. Before the FAA's account, the reports differed:
- Transportation Secretary Sean Duffy said telecommunications routers were overloaded with data (CNBC).
- Homeland Security Secretary Markwayne Mullin said new security patches in the older system appeared to overwhelm its routers. He said it did not appear to be a cyberattack (CBS News).
- CBS News, citing sources, reported that an update to the network that connects air traffic control facilities made the system reboot and crash.
Stopping was the right call. An aircraft on the ground can wait safely. With the links between control centers in doubt, holding departures is the safe state, and the FAA went to it.
Redundancy that fails together
A backup protects against a part that fails on its own. It does not protect against a cause that reaches every copy at once: the same update, the same overload, the same bad file. This is a common-mode fault.
A change was behind an earlier outage. On January 11, 2023, the FAA's NOTAM system failed, and the FAA paused domestic departures nationwide that morning. Its preliminary review found that contract personnel deleted files while they worked to "correct synchronization between the live primary database and a backup database" (NPR; FAA statement). The backup was part of the same change.
Friday also began during a change: work at one remote site that included a new router. The flood that followed reached 17 of 22 centers through the network they share. A change at one place can reach every copy that depends on it.
A change can reach past its own part in two ways, and redundancy fails differently in each:
| A bad change that | What happens to three copies | In Mru Field |
|---|---|---|
| Reaches every copy at once | All three stop together, or agree on the same wrong answer | Staged updates, 0.2.0 |
| Gives right answers but floods what it shares | All three agree, so the vote passes, and the traffic goes out | Flood limits, 0.3.0 |
The second row is why the second release exists: a vote and a staged update both compare answers, and right answers pass both.
What three replicas catch, and what they share
Mru Field runs an operator's task as three replicas on a system's own processor. Every replica gets the same input, and the supervisor delivers only outputs it has checked. When one replica is wrong, hung or dead, the other two outvote it or carry on without it.
Three copies of the same program share their bugs, so a fault that reaches all three at once needs a different check. We measure that limit and publish it: on the last replica, a fault that hits both computations alike gets through (at a common-mode share of 0.05, 23 of 400 bit flips). A new build is the change that reaches every copy. Brought into all three slots at once, a bad build takes all three replicas down together, or makes all three agree on the same wrong answer. Version 0.2.0 checks a new build before it reaches every copy. Version 0.3.0 adds limits that do not depend on the answers at all.
Staged updates
Version 0.2.0 brings a new build in one slot at a time, and the old build checks every delivered output until the last switch:
- First trial. Slot 2 runs the new build against two slots on the old build, for 1,000 clean cycles by default (
candidate_cycles). - Second trial. Slot 1 takes the new build for as many cycles again. Two slots now run the new build. An output is delivered only if the slot left on the old build gave the same bytes, so the new build cannot outvote the old one.
- The last switch. Slot 0 takes the new build, and the update is done.
Until the last switch, one fault of a slot on the new build rejects it: a wrong answer, a hang, a crash, a failed known-answer test, or a disagreement with the old build. A rejection takes the new build out of every slot that has it and starts the old build there again. If a slot on the old build fails during a trial, the update pauses, because nothing may be delivered on the strength of a build under test. A power cut during a trial resumes the trial from the checkpoint.
The bench shaped the rule in step 2. In an early build, two slots on the new build could outvote the old one on a bug that showed in the second trial. Step 2 closes that: the old build has the final say until the last switch.
Results on the bench
We gave the demo task a switch that makes a bad build: one that crashes, hangs, gives wrong answers from a set cycle, gives wrong answers on 1 input in 97, or fails its known-answer test. Each bad build ran for 5,000 cycles twice: in all three slots at once, and by staged update with trials of 1,000 cycles.
| New build | All at once | Staged update |
|---|---|---|
| Crashes on cycle 500 | All three lost; the run halts after 499 outputs | Rejected on cycle 500 · 0 wrong · 3 alive |
| Hangs on cycle 500 | All three lost; the run halts after 499 outputs | Rejected on cycle 500 · 0 wrong · 3 alive |
| Wrong from cycle 500 | 4,501 wrong outputs, none flagged | Rejected on cycle 500 · 0 wrong |
| Wrong on 1 input in 97 | 51 wrong outputs, none flagged | Rejected on cycle 97 · 0 wrong |
| Fails its known-answer test | No replica starts | Rejected at start · 0 wrong |
| Crashes on cycle 1,500 | All three lost; the run halts after 1,499 outputs | Rejected on cycle 1,500 · 1 withheld · 0 wrong |
| Wrong from cycle 1,500 | 3,501 wrong outputs, none flagged | Rejected on cycle 1,500 · 1 withheld · 0 wrong |
| Wrong from cycle 2,500 | 2,501 wrong | 2,501 wrong: after the last switch |
A hang on cycle 1,500 gave the same counts as the crash on 1,500. A good build went to all three slots on cycle 2,000, with 5,000 outputs delivered and 0 wrong. Every line is a run in staged.csv.
All at once, a crash or a hang took down all three replicas on the same cycle, and the run halted. A build with wrong answers was worse: all three replicas agreed, the vote passed, and 4,501 wrong outputs went out with nothing to flag them.
By staged update, every bad build was rejected on the cycle its bug first showed. Every slot went back to the old build, three replicas stayed alive, and no wrong output was delivered. A rejection in the second trial withheld one cycle.
Right answers, flooded network
A new part can do its own job and still harm what it shares a link with, through its traffic, not through its answers. Field checks answers, so we tested that case first. A build of our demo task that answers correctly and sends 100 network packets per request passed a staged update on the bench, and 400,100 packets went out. Version 0.3.0 adds three limits that hold whatever build a node runs:
The task network lock. With task_network = "deny", the default, each task process gets a seccomp filter before it starts (kernel documentation). It can open local AF_UNIX sockets and nothing else: every other socket call fails. The filter stays with every process the task starts. However wrong a build is, and however right its answers are, it cannot send to a network. Only the supervisor talks to the outside, at most one output per cycle and one heartbeat line per interval, and it never listens for connections. The lock is enforced on Linux on x86_64, ARM64 and 32-bit ARM.
The input backlog. A source that sends faster than the cycle builds a backlog. By default, every record waits its turn, so during a flood the input grows older for as long as the flood lasts. With input_backlog = "newest", the supervisor reads what is already waiting, keeps the newest complete record and skips the rest. It logs the backlog once when it begins and once when it clears, so a long flood does not also flood the log.
Writes with a deadline. In 0.2.0, a TCP peer that stopped reading held the supervisor without end: no cycles, no heartbeat, and its own run caps never ran. Every write to a TCP peer or a serial port now ends at 5 seconds. An output write that runs out of time stops the run with an error and keeps the checkpoint, so the service manager can start it again. The bench found one more fix: a serial source that never sent a newline could grow the supervisor's memory without limit. A long line is now cut as it arrives.
Flood limits on the bench
| Run | Without the limit | Field 0.3.0 |
|---|---|---|
| A build with right answers that sends 100 packets per request from cycle 500, all at once | 450,300 packets out; 2,000 delivered, 0 wrong | 0 packets out; 2,000 delivered, 0 wrong |
| The same build, by staged update | Updated on cycle 1,000; 400,100 packets out | Updated on cycle 1,000; 0 packets out |
| A source sending 10,000 records a second into 100 cycles a second | Input 12.0 s old by the last cycle | Input 0.8 ms old at the median; 122,489 skipped, one log line |
| The same source at 1,000 records a second | Input 11.1 s old by the last cycle | Input 0.6 ms old at the median; 11,175 skipped |
| An output peer that stops reading | Field 0.2.0 hung; its 30 s run cap never ran; killed at 60 s | Ended at 7.7 s with an error, checkpoint kept |
Every line is a row in lock.csv, backlog.csv or stall.csv. "Without the limit" is the lock off, input_backlog = "queue", or Field 0.2.0. Packets are the UDP datagrams a local listener read during the run. The supervisor's peak memory was about 13 MB in every backlog run.
The lock does not change what the vote sees. With the lock off and with it on, the build gave the same right answers, and the staged update took it in both times. The lock is what made its traffic zero.
Where stopping is not an option
At an airport, the safe state is on the ground. A seafloor observatory, a polar station or a spacecraft has no gate to wait at. A visit to a seafloor node can be a year away, and a spacecraft waits for its next contact. These systems still need updates: fixes, new compilers, new libraries. A staged update lets the system judge a new build against the one it already trusts, on its own, and keep working while it does.
These systems also share their links. A buoy shares a modem with its instruments, a substation shares a network with every relay on it, and a spacecraft shares a bus and a downlink with every other payload. On each, one part that sends too much can crowd out the rest, and its answers say nothing about it. The flood limits hold whatever build a node runs, including one that has not been checked yet.
Staged updates are in Mru Field 0.2.0, and flood limits in 0.3.0.
Sources
- Federal Aviation Administration, "Nationwide Ground Stop: Questions and Answers," November 15, 2026. faa.gov/newsroom/nationwide-ground-stop-questions-and-answers
- L. Josephs, "DOT Sec. Duffy ties router in southern Arkansas to issues that led to nationwide ground stop," CNBC, November 14, 2026. cnbc.com/2026/11/14/faa-ground-stop-flights-resume-delays.html
- L. Josephs, "FAA lifts ground stop after more than four hours, but flight cancellations will stretch into the weekend," CNBC, November 13, 2026. cnbc.com/2026/11/13/faa-ground-stop.html
- "FAA lifts nationwide ground stop that disrupted thousands of flights," CBS News, November 13, 2026. cbsnews.com/news/faa-issues-nationwide-ground-stop/
- "Update to network connecting ATC system led to outage before ground stop, sources say," CBS News (video), November 13, 2026. cbsnews.com/video/update-to-network-connecting-air-traffic-control-system-led-to-outage-before-ground-stop-sources-ay/
- Federal Aviation Administration, "FAA NOTAM Statement," January 19, 2023. faa.gov/newsroom/faa-notam-statement
- "FAA contractors deleted files and inadvertently grounded thousands of flights," NPR, January 19, 2023. ctpublic.org/2023-01-19/faa-contractors-deleted-files-and-inadvertently-grounded-thousands-of-flights
- Mru Field specification, version 0.4, section 10, "Staged updates," and section 11, "Floods." github.com/mruspace/field/blob/main/docs/spec.md
- Linux kernel documentation, "Seccomp BPF (SECure COMPuting with filters)." docs.kernel.org/userspace-api/seccomp_filter.html
