Solana Outage: Full List Of SOL Network Blockchain Mainnet Failures

Written by

Eddie Mitchell

Cryptocurrency Writer

This week, the Solana Network suffered another major network outage following a technical malfunction that halted block production for almost five hours on 6 February 2024.

What's more, Solana's technical woes have now become a running joke in the crypto community, albeit, a very costly one. 

Here we take a look at where it has gone wrong for Solana in the past...

Solana mainnet down

According to the official Solana Status X account, the “mainnet-beta” experienced its first outage in just under a year, though it is one of many outages since 2021.

SOL tweet

Though Solana is yet to conclude its investigation as to the root causes, Matthew Sigel, head of digital assets research at VanEck, posited in a tweet that the outage was due to a mechanism that governs certain program functionalities failing.

Matthew Sigel tweet

At the time of writing, Solana has resumed block production following a successful upgrade and “a restart of the cluster by validator operators".

Solana’s track record

Solana has become one of the biggest contenders in the Web3 space, boasting solid technical capabilities that have made it the blockchain of choice for hundreds of thousands of users, including major institutional collaborators such as Visa.

However a string of significant network outages, averaging around once every two months since 2021, has caused a stir amongst observers, with many now questioning the integrity of the network going forward, how this is going to impact future developments and mainstream adoption, as well as whether or not Solana has a centralization problem.

Partial outages and network instability are par for the course when it comes to blockchain, and Solana as well as every other major network are no strangers to these minor events, but Solana does have a particularly bad track record when it comes to mainnet outages.

Here’s a list of every partial and major mainnet incident/outage:
 

Outage/incident date Major/partial Downtime/duration Cause
02 September 2021 Partial 35 minutes Unknown
14/15 September 2021 Major 17 hours, 21 minutes Memory overflow caused by Grape Protocol IDO bot transactions on Raydium.
6/12 January 2022 Partial 58 hours, 23 minutes High compute transactions reducing network capacity.
21/22 January 2022 Partial 29 hours, 22 minutes High network congestion as a result of excessive duplicate transactions.
30 April/1 May 2022 Major 13 hours, 44 minutes NFT minting bots overflowed the network.
1 June 2022 2 x major 4 hours, 10 minutes Mainnet outage due to consensus failure. Clock drift: on-chain time running behind wall clocks.
30 September/1 October 2022 Major 7 hours, 18 minutes Validator “hot-spare node” malfunction.
25 February 2023 Major 18 hours, 50 minutes Malfunctioning validator congested “Turbine” protocol.
6 February 2023 Major 4 hours, 46 minutes Yet to be determined.

Causes for the outages

14/15 September 2021: Memory Overflow

The first major incident to be recorded by Solana came as a result of Grape Protocol’s initial decentralized exchange (DEX) offering (IDO), hosted by the Solana-based decentralized finance (DeFi) platform, Raydium. 

Bots attempting to farm the token offering flooded and overwhelmed the network’s nodes, causing it to stop producing blocks when it could no longer reach network consensus.

6/12 January 2022: High Compute Transactions

For a week, Solana experienced significantly degraded performance and partial outages, which it states was due to “high compute transactions” that caused a significant reduction in network capacity, dropping it to several thousand transactions per second (TPS) instead of the stated 50,000 TPS.

21/22 January 2022: Excessive Duplicate Transactions

Another outage hit Solana in January when network congestion as the result of bots creating excessive duplicate transactions caused mainnet degradation. This issue would go on to be swiftly resolved but was a major blow to the network's reputation.

30 April/1 May 2022: Consensus Failure

Another bot-related issue struck when non-fungible token (NFT) minting bots flooded the network with a record-breaking four million TPS, congesting the network and knocking it out of consensus.

1 June 2022: Consensus Failure and Clock Drift

A buggy “durable transaction nonce” mechanism caused nodes to generate different outputs which lead to consensus failure, halting block production once again.

In addition, Solana’s blockchain clock had also drifted 30 minutes behind real-world time.

30 September/October 2022: Malfunctioning Validator Node

Due to a validator operator’s “hot-spare node” malfunctioning, duplicate blocks were produced at the same block height, and a bug in the fork selection logic stopped block producers from being able to build on top of the previous, resulting in consensus failure again.

25 February 2023: Malfunctioning Validator Node

One of Solana’s large single outages was the result of a malfunctioning validator broadcasting an excessively large block, flooding the network’s primary block-propagation protocol, “Turbine”, and resulting in a major performance decline and outages.

06 February 2024: Cause unknown

Though the team at Solana has yet to confirm the root cause of the incident, Matthew Sigel of VanEck has speculated that the outage was triggered by a bug within a mechanism that deploys, upgrades, and executes programs on Solana, forcing all network activities to stop.