Oh, Maria(DB), Maria(DB)
Hey, Mariadb fans!
It’s time to talk about the elephant in the room.
No not that one.
AWS recently forced us to upgrade a few hundred RDS-MariaDB instances to 10.6.12. Suddenly one fine day out of nowhere, a bug woke up and wrecked havoc in production. So, what’s the deal with this bug? Well, it turns out that if you’re using certain queries with the ORDER BY and GROUP BY clauses, you could trigger a race condition. This causes the table to be inaccessible due to index corruption. You can’t access the table and operate on corrupt index. Oops! It’s like the software equivalent of a slippery banana peel — you don’t see it coming until it’s too late.
The cause of table stalls and index corruption is a bug in MariaDB 10.6 major version. A high concurrency condition and insert on duplicate key updates trigger this bug.
Related Tickets:
- [MDEV-29835] Partial server freeze
- [MDEV-30665] MariaDB 10.6.12 hang 2 days after upgrade
- [MDEV-30481] Hard lock up with queries in “Opening tables” state
- [MDEV-30638] Deadlock due to updating InnoDB statistics
The MariaDB team has acknowledged and addressed the issue in next version. However, that doesn’t mean it’s not a major problem for many AWS users since the next version where we hope that bug is fixed is not yet available. We know how frustrating it can be to upgrade to the latest database version, only to encounter unexpected issues that bring everything to a screeching halt. Like when you try to upgrade your wardrobe with some cool new clothes, but they don’t fit right, you’re back to wearing your old clothes.
The only way to fix it? You guessed it — restart the database. Yes, it may feel a little like we’re back in the days of Windows 95, but hey, at least we’re not dealing with dial-up internet anymore, right? However, no one from development teams has asked for a restart button yet, despite joking around that a “restart a day keeps the DBA away”.
Following meme was seen passed around after MariaDB upgrade disaster.
In the end, the MariaDB table/index corruption bug in version 10.6.12 reminds us that safety should not be compromised for cost benefits. We are in 2023 and this would not happen in PostgreSQL with a larger, more active open source community and higher development standards.
Edit : Next version (10.6.13) also has major server freeze related bugs so we had to wait for 10.6.14.
Personally, I would not recommend MariaDB to anyone, especially in the financial sector. There is just nothing production-ready about MariaDB.
