Solana Outage: Full List Of SOL Network Blockchain Mainnet Failures

Updater face
Last updated Jun 06, 2024 | 03:22 PM UTC

As most crypto enthusiasts are already aware – the Solana blockchain is notorious for its regular network outages.

On 6 February 2024, the network suffered a major network breakdown following a technical malfunction that halted block production for almost five hours.

Over the years, Solana's technical woes have become a running joke in the crypto community, albeit, a very costly one. 

However, this has not deterred some of the industry's biggest names from launching their projects on the Solana Network.

Just last week on 29 May, it was revealed that the PayPal USD (PYUSD) stablecoin is now available on Solana. Commenting on the partnership, Solana Foundation's GM of payments Sheraz Shere said: “The Solana network’s speed and scalability make it the ideal blockchain for global financial institutions, like PayPal, to create new payment solutions that are accessible, cost-effective, and instantaneous.”

Indeed last year, Visa became one of the biggest financial giants to begin using Solana network for settlements, while in April of this year, Stripe announced that it will accept USDC stablecoin payments on Solana. 

What's more, there is talk circulating that the Ethereum-based Pendle, Ethena, and Aave are next in line to also adopt the Solana blockchain ecosystem. 

Such additions would further bolster Solana and create exciting opportunities for growth and development.

But will the network be able to handle all the pressure? Or will it only lead to further technical havoc in the future?

Here we take a look at where it has gone wrong for Solana in the past...

Solana mainnet down

Following February's meltdown, according to the official Solana Status X account, the “mainnet-beta” experienced its first outage in just under a year, though it is one of many outages since 2021.

SOL tweet

Matthew Sigel, head of digital assets research at VanEck, posited in a tweet that the outage was due to a mechanism that governs certain program functionalities failing.

Matthew Sigel tweet

Following the outage, Solana resumed block production following a successful upgrade and “a restart of the cluster by validator operators".

Solana’s track record

Solana has become one of the biggest contenders in the Web3 space, boasting solid technical capabilities that have made it the blockchain of choice for hundreds of thousands of users, including major institutional collaborators such as Visa.

However a string of significant network outages, averaging around once every two months since 2021, has caused a stir amongst observers, with many now questioning the integrity of the network going forward, how this is going to impact future developments and mainstream adoption, as well as whether or not Solana has a centralization problem.

Partial outages and network instability are par for the course when it comes to blockchain, and Solana as well as every other major network are no strangers to these minor events, but Solana does have a particularly bad track record when it comes to mainnet outages.

Here’s a list of every partial and major mainnet incident/outage:
 

Outage/incident date Major/partial Downtime/duration Cause
02 September 2021 Partial 35 minutes Unknown
14/15 September 2021 Major 17 hours, 21 minutes Memory overflow caused by Grape Protocol IDO bot transactions on Raydium.
6/12 January 2022 Partial 58 hours, 23 minutes High compute transactions reducing network capacity.
21/22 January 2022 Partial 29 hours, 22 minutes High network congestion as a result of excessive duplicate transactions.
30 April/1 May 2022 Major 13 hours, 44 minutes NFT minting bots overflowed the network.
1 June 2022 2 x major 4 hours, 10 minutes Mainnet outage due to consensus failure. Clock drift: on-chain time running behind wall clocks.
30 September/1 October 2022 Major 7 hours, 18 minutes Validator “hot-spare node” malfunction.
25 February 2023 Major 18 hours, 50 minutes Malfunctioning validator congested “Turbine” protocol.
6 February 2023 Major 4 hours, 46 minutes Yet to be determined.

Causes for the outages

14/15 September 2021: Memory Overflow

The first major incident to be recorded by Solana came as a result of Grape Protocol’s initial decentralized exchange (DEX) offering (IDO), hosted by the Solana-based decentralized finance (DeFi) platform, Raydium. 

Bots attempting to farm the token offering flooded and overwhelmed the network’s nodes, causing it to stop producing blocks when it could no longer reach network consensus.

6/12 January 2022: High Compute Transactions

For a week, Solana experienced significantly degraded performance and partial outages, which it states was due to “high compute transactions” that caused a significant reduction in network capacity, dropping it to several thousand transactions per second (TPS) instead of the stated 50,000 TPS.

21/22 January 2022: Excessive Duplicate Transactions

Another outage hit Solana in January when network congestion as the result of bots creating excessive duplicate transactions caused mainnet degradation. This issue would go on to be swiftly resolved but was a major blow to the network's reputation.

30 April/1 May 2022: Consensus Failure

Another bot-related issue struck when non-fungible token (NFT) minting bots flooded the network with a record-breaking four million TPS, congesting the network and knocking it out of consensus.

1 June 2022: Consensus Failure and Clock Drift

A buggy “durable transaction nonce” mechanism caused nodes to generate different outputs which lead to consensus failure, halting block production once again.

In addition, Solana’s blockchain clock had also drifted 30 minutes behind real-world time.

30 September/October 2022: Malfunctioning Validator Node

Due to a validator operator’s “hot-spare node” malfunctioning, duplicate blocks were produced at the same block height, and a bug in the fork selection logic stopped block producers from being able to build on top of the previous, resulting in consensus failure again.

25 February 2023: Malfunctioning Validator Node

One of Solana’s large single outages was the result of a malfunctioning validator broadcasting an excessively large block, flooding the network’s primary block-propagation protocol, “Turbine”, and resulting in a major performance decline and outages.

06 February 2024: Cause unknown

At the time, Matthew Sigel of VanEck speculated that the outage was triggered by a bug within a mechanism that deploys, upgrades, and executes programs on Solana, forcing all network activities to stop.

Written by

Eddie is a seasoned crypto writer and Bitcoin maximalist.