CytoSMART’s redundant storage stores multiple copies of your data synchronously within a region for durability. To ensure durability, we replicate the transaction synchronously across three different storage nodes across different fault domains and upgrade domains. A fault domain (FD) is a group of nodes that represent a physical unit of failure and can be considered as nodes belonging to the same physical rack. An upgrade domain (UD) is a group of nodes that will be upgraded together during the process of service upgrade (rollout). The three replicas are spread across UDs and FDs to ensure that data is available even if hardware failure impacts a single rack and when nodes are upgraded during a rollout.
In addition to returning success only when all three replicas are persisted, we store CRCs of the data to ensure correctness and periodically read and validate the CRCs to detect bit rot (random errors occurring on the disk media over a period of time).