Third run in orbit: the F⁠´ components

What ran

On October 28 at 09:13 UTC, Mru Flight v0.7.0 ran on ESA's OPS-SAT PRETTY satellite, the third run of experiment exp4051 of the ESA OPS-SAT Space Lab. The package went to ESA with the run 2 results on October 26, ESA confirmed a third run on October 27, and TU Graz ran it on the FlatSat on the morning of October 28.

Runs 1 and 2 flew the fault-injection demo. Run 3 flies the same decision core as F⁠´ components, in the form a mission would integrate it. F⁠´ is NASA JPL's open-source flight software framework. The manager process runs RedundancyManager and HealthRegistry with F⁠´'s own command dispatcher, events, telemetry, health pings and command sequencer, on a timer of 100 ms (one cycle). Each of the three replicas is an F⁠´ process of its own, linked to the manager through F⁠´'s GenericHub over local sockets, with no network. Replica 0 is pinned to one of the processor's two cores, replicas 1 and 2 to the other.

The faults are the ones of run 1, commanded by F⁠´ command sequences, as an operator would command them. Three segments, 128 MB per process:

1. The shrinking quorum: a bit flip in replica 2 at cycle 60, replica 1 stuck at cycle 120, and at the start of cycle 240 the manager kills replica 0's process (SIGKILL). Cap 45 seconds.
2. Fixed triple redundancy (TMR) on the same faults. Cap 35 seconds.
3. The shrinking quorum with its state saved to a file, replica 1 stuck at cycle 200. After 35 seconds our script kills the manager without warning and starts it again for 30 seconds, with no command sequence.

The kills are our own software ending our own processes, in user space. Nothing on the platform was reset or commanded.

Three hosts, the same counts

The same package ran in ESA's image on our laptop on October 22, the run the prediction came from, on TU Graz's FlatSat on the flight processor on the morning of October 28, and in orbit the same day. Every segment gave the same counts on every host. No segment on any host gave a wrong result.

The segments run to a time cap at ten cycles a second, so the cycle counts barely change with the host. On the FlatSat the kill at cycle 240 came while replica 0's link errors were still in the manager's queue, so the manager withheld that cycle instead of blaming replica 0 for a missing answer (the late-cycle rule, added in v0.6.0), and retired replica 0 one cycle later, at 243. The counts did not change; in segment 2 the withheld cycle took the place of one caught disagreement. In orbit no cycle was withheld, as on the laptop. Total is the three segments plus start-up, from the run-info times.

In orbit

Segment 1. The bit flip at cycle 60 was outvoted. At cycle 120 replica 1 got stuck: it was outvoted, a known-answer test at cycle 122 named it, and the quorum retired it. Replicas 0 and 2 went on in compare mode. At the start of cycle 240 the manager killed replica 0's process. Its answers stopped, the quorum caught a disagreement on cycles 240 to 242, and at 242 retired replica 0 as lost. From cycle 243 replica 2 ran alone in self-check, computing each result twice, so it delivers a result every other cycle. 342 correct results and 0 wrong over 449 cycles.

Segment 2. Fixed TMR, on the same faults, delivered 239 correct results. After the kill it had one good replica and one stuck one, could not tell which was wrong, and caught a disagreement on every cycle to 348 without delivering a result.

Segment 3. Replica 1 got stuck at cycle 200 and was retired at 202. At 35 seconds the script killed the manager. Its state file, 21 bytes, held the policy, the three slots and cycle 303. Started again, the manager restored that state, logged one restore with replica 1 still retired, and went on with replicas 0 and 2 to cycle 601: 299 correct results and 0 wrong.

Each replica process ran on the core it was pinned to, in every segment. The manager used 1.7 MB resident and 11.8 MB of address space. All four processes together used 0.79% to 0.96% of wall time, measured on the SEPP in steps of 10 ms, under the registered 5% share.

What we said it would do

The prediction had four parts. Zero wrong results in every segment: zero. About 340 correct results in segment 1, with the quorum down to its last replica: 342, on replica 2 alone. 239 for fixed TMR, then a caught disagreement on every cycle: 239, then 109. One restore in segment 3, with the retired replica still retired: one, at cycle 303. We also wrote that the run would stop by itself after about two and a half minutes: 148 seconds.

What it shows, and what it does not

It shows the F⁠´ components doing in orbit what the demo did in runs 1 and 2: the shrinking quorum went down to its last replica with no wrong result, and fixed TMR, on the same faults, stopped delivering at cycle 240. It shows the parts a mission integrates: the components inside F⁠´, commanded by a sequence, reporting through F⁠´ events, a replica lost to a real process kill and found by its silence, and a manager restart that brings a retired replica back as retired. The F⁠´ components had only run on the ground before this run.

It does not show more than that. The three replicas were processes on one Linux processor with two cores, not three processors. The stuck value and the bit flip came from the test-only fault injector, and both kills came from our own software. The package had no memory sensor, so it says nothing about real upsets. Qualification evidence needs independent processors, real upsets and a long run.

Next

At most one more short run on PRETTY is planned: a failure case that ESA's operations team sees in practice, if ESA agrees. After that, the next step is independent processors, on boards on the bench.