PLEASE always test your code in production

So I was testing my server's remote access from my ThinkPad.

Now, I just wanted to check for Tailscale access and speeds, but today I had the idea of jumping on 'btop' to see the stats because it's fun.

And normally I wouldn't have noticed anything because my disks were all pretty busy running IO related tasks. But today the I/O stats on my main disk caught my eye, it was a glaring yellow bar at nearly 100%. And I was asking myself: "What the hell is taking up all that I/O?".

So I let my AI agents investigate this issue, and it found something.

Turns out my lyrics server had performed countless expensive SQL queries against my 244GiB lyrics database, racking up 19TiB of reads over 33 hours and absolutely HAMMERING my disk I/O 💀, all because of a publisher service that ordered those expensive queries I never needed.

Funny thing is, even though AI fixed this issue in under 5 minutes, AI also wrote the same piece of code that caused 19TiB of reads and almost brought my disk I/O to a stall... So yeah, this incident is still serious (but not catastrophic), I would not have noticed this issue if I did not open btop today.

Moral of the story? Always test your code, whether it's written entirely by you or written by AI. It is always important to test it in dev and production, because you never know what might go wrong.

Happy Coding! 😄