Last post covered technical definitions of fault, error, and failure. In this post we will apply these definitions in a system.
Our context is a long-running service or server. It handles requests from many different consumers. Consumers may be human users, as in the case of a web site, or they may be other programs.
Engineering literature has many definitions of "availability." For our purpose we will use observed availability. That is the probability that the system survives between the time a request is submitted and the time it is retired. Mathematically, this can be expressed as the probability that the system does not fail between time T_0 and T_1, where the difference T_1 - T_0 is the request latency.
(There is a subtle issue with this definition of observed availability, but we can skirt it for the time being. It intrinsically assumes there is some other channel by which we can detect failures in the system. In a pure message-passing network such as TCP/IP, there is no way to distinguish between "failed" and "really, really slow." From the consumer's perspective, "too slow" is failed.)
The previous post established that faults will occur. To maintain availability, we must prevent faults from turning into failures. At the component level, we may apply fault-tolerance or fault-intolerance. Either way, we must assume that components will lose availability.
Stability, then, is the architectural characteristic that allows a system to maintain availability in the face of faults, errors, and partial failures.
At the system level, we can create stability by applying the principles of recovery-oriented computing.
- Severability. When a component is malfunctioning, we must be able to cut it off from the rest of the system. This must be done dynamically at runtime. That is, it must not require changes to configuration or rebooting of the system as a whole.
- Tolerance. Components must be able to absorb "shocks" without transmitting or amplifying them. When a component depends on a another component which is failing or severed, it must not exhibit higher latency or generate errors.
- Recoverability. Failing components must be restarted without restarting the entire system.
- Resilience. A component may have higher latency or error rate when under stress from partial failures or internal faults. However, when the external or internal condition is resolved, the component must return to its previous latency and error rate. That is, it must display no lasting damage from the period of high stress.
Of these characteristics, recoverability may be the easiest to achieve in today's architectures. Instance-level restarts of processes, virtual machines, or containers all achieve recoverability of a damaged components.
The remaining characteristics can be embedded in the code of a component via Circuit Breakers, Bulkheades, and Timeouts.