№ 010 · Release
Mru Flight 0.9.0: resets, sensors, cFS and bit flips
Runs 1 to 3 injected each fault at its symptom. Mru Flight 0.8.0 and 0.9.0 add what comes after a fault and one step closer to its cause: a unit reset and taken back, sensors that disagree, the same core under NASA's core Flight System, and bit flips in memory that nothing tells the manager about. The OPS-SAT team approved run 4 on November 16, 2026.
What is in run 4
On November 16, 2026, the OPS-SAT team approved a fourth run of experiment exp4051 on ESA's OPS-SAT PRETTY. The package, exp4051-mru-flight-v0.9.0.tar.gz, has the same shape as run 3: one entry script in the foreground, user space only, no network, 128 MB per process, everything written to toGround/. Six segments run one after the other:
- Automatic resets. Replica 1 stops answering at cycles 60 and 300. Replica 2's process is killed with SIGKILL at cycle 150, a real dead unit. The manager resets a silent replica, up to three times.
- The same faults with resets off.
- The same faults under fixed triple modular redundancy (TMR).
- Three simulated sensors: a spike, a drift and a frozen sensor.
- The same decision core as an application of NASA's core Flight System (cFS), with the faults of run 1.
- Real bit flips in each replica process's own memory, made by software.
Segments 1 to 5 simulate each fault at its symptom, as runs 1 to 3 did: a replica's answer is changed, withheld or stopped.
In ESA's OPS-SAT PRETTY environment image (32-bit ARM, emulated on a laptop), the whole package ran three times at full length: every exit 0, 0 wrong results in every segment.
A unit reset, and taken back only on the right answer
When a replica is retired for silence, the F´ RedundancyManager sends a reset on its resetOut port and starts an admission at the next cycle. The unit comes back only if it gives the right known answer. A unit still silent gets the next reset. After ResetLimit resets the manager emits ResetsExhausted and keeps the slot retired, so a dead unit cannot cause a reset loop. A replica retired for a wrong answer is never reset.
In a full-length native run of segments 1 to 3 in the build container (not the flight processor), the same faults gave three different ends:
| Segment | End of the segment | Wrong results |
|---|---|---|
| 1 · Shrinking quorum, resets on | 2 units live: replica 1 back twice, replica 2 kept out after 3 resets | 0 |
| 2 · Shrinking quorum, resets off | Replica 0 alone, in self-check | 0 |
| 3 · Fixed TMR | Halted at cycle 153, one unit left after the kill | 0 |
Source: the 0.8.0 entry of the changelog.
One value from readings that differ slightly
Three sensors that read the same quantity never agree to the bit, so a vote on equal values does not work for them. The core now chooses one value within a tolerance: the median of three if another reading is within the tolerance of it, the midpoint of two if they agree, a single reading marked unchecked. With two readings that disagree, it delivers nothing, since nothing says which one is right. SensorVoter excludes a sensor only after a set number of outlier cycles in a row (5 by default), so a short spike never excludes it.
Kani proves five properties for every set of three readings and every tolerance: an agreed value is agreed and in range, an outlier is never chosen, nothing is unchecked while two or more readings are in, the median is a median, ties included, and a sensor is excluded exactly after its limit. A unit test found a tie bug in the first version of the median.
Segment 4 runs a spike on sensor 1 at cycle 60, a drift on sensor 2 from cycle 120, and sensor 0 frozen from cycle 260, with a tolerance of 2.00 and noise up to 0.50. In a full-length native run, 399 cycles: 279 correct values, 0 wrong, and 120 cycles with no value after the two readings left disagreed from cycle 279. The spike was masked, and sensor 2 was excluded at cycle 137. A plain average of the same readings was wrong in 261 of the 399 cycles.
The same core in NASA's core Flight System
The decision core now also runs as a cFS application: three replicas as cFE child tasks, every decision through the core's C interface, a known-answer test of every replica before cycle 1, linked statically into one binary for the flight processor. No sample or lab application is built, so nothing opens a network socket. The app has the core's decisions and the F´ manager's rules around them. It does not have the automatic resets or the sensor voter; those are F´ only.
Segment 5 runs it with the faults of run 1: a bit flip at cycle 60, a stuck replica at 120, a hung one at 240. In ESA's environment image: 400 cycles, 318 correct, 3 disagreements caught, 0 wrong. The stuck replica 1 was retired on its known answer at cycle 122, the hung replica 0 was lost at cycle 242, and replica 2 went on alone in self-check. No socket was open, and nothing was left behind. Natively, under an enforced 128 MB limit, the process used 9.0 MB of address space.
Closer to the cause
Segment 6 injects at the cause, not the symptom. A thread in each replica process flips one bit at a time in that process's own memory: code, read-only data, data, heap and stacks, in proportion to their size, at random times, 2 flips per second per replica. Each flip is logged before it is made. The package's files are never changed. The manager runs the shrinking quorum with a known-answer test on the last replica every cycle, and it is told nothing about the flips.
Where a flip lands cannot be scheduled. In ESA's environment image and in 160 native runs at up to 200 flips per second, most flips changed nothing. Some crashed or hung a replica, and it was lost. Some made a replica answer wrong; the disagreement was caught and the replica retired. When the last replica went, Mru halted and delivered nothing. No run delivered a wrong result.
The flips found one real flaw before flight. A corrupted replica sent a reply too short to decode, and F´'s generated port code in the manager asserted on it and ended the manager process: 1 of 64 native runs at 200 flips per second from start-up. Faults injected at the symptom never sent the manager a malformed message, so they could not find it. In 0.9.0 the manager checks every message from a replica on its link before F´'s hub sees it. A message that is not exactly one well-formed reply, or one more than four in a cycle, is dropped and counted (links: dropped= in the summary). A replica that sends only garbage is then silent to the manager, and is lost. In 160 runs after the fix, the manager never crashed.
A replica that a flip sets spinning stops at 2 seconds of CPU time, one that writes without end stops at 256 KB, and no process of the run writes a core file.
What we said it would do
In segment 1, replica 1 is back at cycles 63 and 303, and replica 2 is kept out at cycle 158. Fixed TMR halts at cycle 153. Sensor 2 is excluded at cycle 137. Where the flips of segment 6 land cannot be predicted, so for them we predict only the outcome that matters: zero wrong results, in every segment.
Next
TU Graz runs the package on the FlatSat, then in orbit. We will publish the results against the prediction above. After run 4, the next step is independent processors, on boards on the bench.
