We can all recall a few large-scale failures that have affected us. They clearly demonstrated how fragile modern life can be. How inconvenient it becomes when the energy system is knocked out by shelling: this has probably impacted absolutely everyone — cold, lack of internet, elevators, and in some places — water. But there were also smaller disasters that we noticed and overcame in just a few days.
For example, we all remember how on December 12, 2023, Kyivstar stopped working. It was impossible to make calls, use mobile internet, and along with the operator's network, some ATMs, POS terminals, and other services that depended on it temporarily stopped working. Recovery took several days. As a mother who takes her child to kindergarten and stays in touch in case she gets sick or some other emergency occurs, I remember how uncomfortable it was to run around the city without coverage.
But there were also disasters that did not happen.
For instance, in April 2022, Russian hackers attempted to shut down the Ukrainian energy system. The attack was stopped literally at the last moment. Most Ukrainians didn't even find out about it.
The modern IT industry fights against failures before they occur. And the best result of this work is when we don't even realize it's happening because the systems are so well duplicated that repairs go unnoticed.
This is related to the fact that engineers usually plan for failure, not success. If a data center fails — the website continues to work; the video doesn't stop when we cut the cable; files don't disappear with a broken server disk.
This logic applies to a regular website as well. When choosing a hosting or virtual server, it's important to consider not only speed and memory capacity but also what will happen to the project in case of a failure. What backups are available, where they are stored, who and how will restore operations — these questions are better asked before the first failure.
For us at Cityhost, such preparation is part of our daily work. The company has passed certification according to DSTU ISO/IEC 27001 and DSTU ISO/IEC 27701, which relate to information security management and personal data protection. This involves specific procedures: risk assessment, defining responsible persons, and action plans for responding to incidents and restoring operations.
A Bit of Disaster History
- The Cost of Errors (1950-60s)
- Turning Point: Tandem Computers
- RAID: Redundant Array of Independent Disks
- Y2K Disaster: The Most Expensive Apocalypse That Did Not Happen
Those who are younger may not remember this, but those who are older should recall the legend of the zero year disaster — the new millennium would arrive, timers would reset, technology would go haywire, and cities would plunge into apocalypse. Patience, we will return to this because there is much to tell, but for now…
The Cost of Errors (1950-60s)
The first computers simply broke down, and along with them, the work being done on them stopped. This was inconvenient, but there was nothing to be done: something burned out — we wait until it gets fixed. However, the prevalence of computers was lower, so downtime was uncomfortable but not as critical as it is now. Continuous availability of anything was not even discussed for a long time: one could endure.
Undoubtedly, engineers worked to make breakdowns happen less frequently. But the focus was on creating a computer that doesn't break. This direction dominated.
As always, everything came down to finances.
The first to seriously consider reliability were not operating system developers but people for whom a few minutes of downtime cost enormous amounts of money.
Initially — airlines.
By the mid-1960s, ticket booking systems were built from several computers that could substitute for each other. Simply because if the system stopped selling tickets, the business literally came to a halt.
Then banks followed suit. Later — stock exchanges. Even later — telecom.
As far as I understand, the essence was this: the more areas were computerized, thus speeding up and simplifying work, the more costly the errors became. Those who wanted to work continuously simply installed more computers.
Turning Point: Tandem Computers
In 1974, HP engineer James Treybing proposed the idea: not to try to make a computer that never breaks.
Instead — to make a computer that can continue to operate after a failure.
HP was not interested in this idea.
So he founded his own company — Tandem Computers. By 1976, it released the NonStop system, which had almost no single point of failure. If a processor failed, it was instantly replaced by another. If a disk failed — its mirrored copy continued to work. If a controller broke — there was an alternative path to access the data.
James Treybing
The first such computer was shipped to Citibank in May 1976. Subsequently, they equipped stock and commodity exchanges — including the Chicago Mercantile Exchange, London Stock Exchange, NASDAQ, etc.
RAID: Redundant Array of Independent Disks
By the late 1980s, it became clear: hard drives are one of the most vulnerable parts of a computer. They failed regularly, and along with them, all data could be lost.
So engineers posed an unexpected question.
What if we stopped relying on a single disk?
In 1988, researchers at the University of California, Berkeley published a paper that popularized the concept of RAID (Redundant Array of Inexpensive Disks) — a redundant array of independent disks. The essence was to use several smaller drives that work together instead of one large storage device.
Depending on the configuration, the array could solve various tasks:
- RAID 1 writes identical data simultaneously to two disks. If one fails, the other continues to operate without data loss.
- RAID 5 distributes data and parity information across three or more disks. If one disk fails, the system can automatically recover lost data after its replacement.
- RAID 10 combines mirroring and data distribution, providing both high speed and fault tolerance.
Not all RAID levels equally protect information. For example, RAID 0 is designed solely for speed enhancement and has no redundancy: a failure of one disk means loss of all data. That is why when talking about fault tolerance, RAID 1, RAID 5, RAID 6, or RAID 10 are usually meant.
And although today there are SSDs, distributed file systems, and cloud storage, the very idea of RAID has not disappeared. Most servers and NAS still use it in one form or another.
But here is what is important for us. All the time, technical thought has moved from seeking something fail-proof to making failures less critical. We accept that they will happen. So: there is no need to create a disk that never breaks. It is necessary to ensure that the system does not break down after its failure either.
Read also: How data centers operate "without power" and under constant shelling of Ukraine's energy system
Y2K Disaster: The Most Expensive Apocalypse That Did Not Happen
Let's return to the zero year disaster or Y2K. This is precisely the case where the disaster was averted so well that many decided it didn't exist. Meanwhile, it is one of the best-documented stories of disaster management in IT.
So, what was the concern: engineers in the 1990s realized that millions of programs used a two-digit year representation. And this was not a problem as long as the millennium lasted. People said it themselves: "In eighty-one, I bought a house." But at midnight of the new two-thousand year, the timer was supposed to reset.
It is by no means certain that all planes would necessarily crash or that power grids would definitely shut down. The problem was that no one knew which systems would fail and what consequences that would have.
Then engineers around the world began to work as if a disaster was about to happen or had already occurred.
“The task of repairing computer systems by the year 2000 caused an unprecedented mobilization of people, money, and managerial attention, which has few parallels in history,” writes “The Washington Post” in 1998.
By the way, this same correspondent mentions the sums spent on this work. The Federal Reserve, says journalist Rajiv Chandrasekaran, estimates that businesses will spend at least $50 billion on this in the U.S. alone. There are even more gigantic figures. Estimates of the costs to overcome the year 2000 problem varied, and the final figure has not been calculated to this day. However, it exceeds $300 billion worldwide, with about half of that amount falling on the U.S.
This was a massive audit of all software. Companies took various approaches:
- finding all places in the code where the date was important. Engineers reviewed millions of lines of code looking for places where the year was recorded in two digits. Often this was COBOL code written back in the 1960s-1970s. Documentation often no longer existed, and the authors had long since retired or left the company. That is why many retired programmers were temporarily brought back to work.
- changing programs and date usage in ATMs, medical equipment, power grid devices, aviation systems, etc. If changing the date record was impossible, the equipment was replaced;
- making backups of everything they had;
- conducting stress tests by artificially rolling the date in the system back to the zero year and observing what failed.
Finally, on the night of the new zero year, many did not celebrate but stood watch at work.
And when on January 1, 2000, a serious disaster did not occur, many decided that the danger was fabricated.
But in reality, it was one of the most successful operations to prevent a technological disaster in human history.
Humanity spent an amount comparable to the annual GDP of a large state, not to create something new, but to ensure that on January 1, 2000, everything continued to work as it had the day before.
Y2K was not the last "date problem." Similar failures recurred in 2022 (though we were not focused on that) when issues arose in Microsoft Exchange and some GPS navigators due to numeric format overflow. This was related to the fact that some programs recorded the date in the format YYMMDDHHMM and stored it as a 32-bit integer.
The next known milestone is the year 2038, and the industry has long been preparing for it. More about the upcoming date-related failures can be read no further than on Wikipedia.
However, this story became a huge push forward: look, all the things we know from our lives were used here — both backups and testing the system's resilience against various events.
In any case, the main conclusion that everyone ultimately made: the question is not whether something will break, but when. And how you prepare for it.
Three Main Principles of Modern Infrastructure
- Backup — if data is lost
- Redundancy — if something breaks
- Failover — if a disaster has already occurred
It gradually became clear that different disasters require different responses. Thus, three principles of modern infrastructure protection were formed.
Backup — if data is lost
The cause can be anything: disk failure, administrator error, ransomware, fire, or physical destruction of the data center.
That is why modern backup is not just a copy on a neighboring disk. The best practice is the 3-2-1 rule: three copies of data, two different media, and at least one copy — in another location. The war with Russia has only confirmed the importance of this approach: for many companies, a backup in another country has become a necessity.
Redundancy — if something breaks
How to ensure that the system does not stop working due to the failure of one component?
Modern services duplicate everything critical: servers, disks, network routes, power sources, and even entire data centers. The idea is simple: if one element fails, there is another that can take over its work.
Failover — if a disaster has already occurred
But a backup server alone does not solve anything. How to switch to it quickly enough that the user doesn't even notice?
For this, it is necessary to automatically switch to the backup system, check that it is functioning correctly, and continue operations without a long interruption. This switching is called failover. Thanks to it, even after the failure of an entire data center, cloud services, banks, streaming services, etc., can continue to operate.
These three principles do not replace each other.
- Backup restores lost data.
- Redundancy prevents the system from failing due to a single breakdown.
- Failover ensures a smooth transition to backup resources.
That is why modern IT relies less on the reliability of individual computers and more on the architecture of the entire system. Engineers no longer try to build a server that never breaks. They build infrastructure for which failure is not an exception but a scenario anticipated at the design stage.
Stories of (Non) Success
Even large tech companies with money, experienced engineers, and modern equipment are not immune to serious failures. Fortunately, some of them openly share their mistakes and ways to fix them. Thanks to this, the entire industry can learn from such incidents.
How GitLab Almost Lost Everything (2017)
In 2017, a GitLab administrator accidentally deleted the working database.
It seemed like nothing serious — there are backups. It turned out that one copy hadn't been updated in a long time, another was corrupted, and the third wouldn't even start; replication was also malfunctioning.
Ultimately, a completely different technology saved the situation — a snapshot, that is, an instant snapshot of the file system, which was stored on a test server. It was made about six hours before the incident. It was from this that the service was restored over 18 hours.
Six hours of changes in the database were lost irretrievably, but the database itself returned to operation.
“We also streamed the recovery process on YouTube, at the peak of the broadcast, about 5,000 viewers were watching it,” the company writes (a full description of the situation and recovery actions can be found here).
This story illustrates well why backup in modern IT is seen as a process, not just a file. It is not enough to create a backup once — it is necessary to regularly check that it can be restored and that it is up to date.
AWS and the Birth of Chaos Monkey
In 2008, Netflix experienced a very painful failure of its Oracle database. Because of this, the company could not ship DVDs to customers for three days. It was then that management decided: it was necessary to abandon a single large database and move to a cloud-based, distributed architecture. This resulted in a move to AWS (Amazon Web Services), a massive cloud platform that allows running services on Amazon's servers.
However, this move created a certain problem: where the system has thousands of servers, the cloud itself is protected from failure: other servers pick up the load of the one that broke. But is that enough for the operation of your specific service to be uninterrupted? After all, the cloud does not know how your work is structured — that responsibility lies with Netflix specialists. If the service architecture depends on one critical component, a failure will still lead to a breakdown.
To address this question, Netflix created the Chaos Monkey program in 2010, which randomly shuts down individual servers running specifically for that company as part of testing. If something breaks or fails after one or more servers are turned off — the functions are not duplicated sufficiently.
In 2012, the company open-sourced this program, and many other businesses began to implement something similar, rewritten for themselves.
Chaos Monkey did not become a program that everyone installed. Instead, it popularized the very idea of chaos engineering. Today, large companies regularly model failures, but often use their own tools or new platforms for failure testing.
If we take a step back, Chaos Monkey merely automated what engineers were doing manually during the preparation for Y2K: creating controlled failures to check whether the system would survive a real one.
We see that none of these approaches emerged suddenly; they all developed over time from practice. Reality is imperfect, and therefore we must constantly insure and duplicate everything important because problems will inevitably arise. It seems to me that this philosophy perfectly harmonizes with our far from perfect life during the war.
