On Sunday, January 22nd, 2023, at approximately 00:09:00 UTC, an incident caused 50% to 60% of Cardano nodes to drop off from the network. The issue impacted relay nodes, responsible for transmitting transaction data to peers, and block-producing nodes, which create blocks. Some nodes disconnected from peers, while others threw an exception and restarted. However, they were able to reconnect in a few minutes.
It’s the first time in 5 years that this kind of event has happened in Cardano.
Node Participation Drop off:
The network did not go down and continued to produce blocks during the node outage. However, there was a temporary degradation of block production for ~7 mins, where the block time increased to 1.5-2 minutes for three blocks. This incident is similar to the China mining ban event for BTC, where the network hash rate dropped ~50%, but blocks kept chugging along.
The incident was barely noticeable for most users. The service degradation was for ~7 mins, where some users' transactions were delayed.
Investigations are still ongoing, and a cause still hasn’t been founded. Most nodes went down at the same block height, which could mean that a rare sequence of transactions or a single transaction might have caused it. There is no reason to believe it is malicious or an attack, but there is a possibility we might see a replay of this unless a patch is released.
Source: Cardano Node GitHub
While this caused a brief period of degradation in the network, it is important to note that such issues were considered in the design of the Cardano node and consensus. Cardano's Ouroboros is a longest-chain protocol and remains operational under such outages. The systems behaved as expected, and most impacted nodes gracefully recovered without needing a network restart. This contrasts BFT chains like Solana, where the network shuts down and needs a reboot.
In conclusion, the incident that occurred on January 22nd, 2023, on the Cardano blockchain network was a transient issue that was handled as expected by the network's design. The network has fully recovered and is now operating normally. IOG and SPOs are working to identify the incident's root cause to prevent future similar issues.
The incident is a testament to the robustness and resilience of Cardano nodes, and it's design which can handle such incidents gracefully without causing major disruptions.
Thank you for the information. Great to know that the Cardano Blockchain remained on track and intact despite a slight delay with no reboot required.
Glad to read this. We will waiting for the post-mortem analysis.
thank you for your feedback
All of our nodes went down and quickly restarted and re-synced with no intervention.