• 1 post
  • 43 comments
Joined 3 years ago
Cake day: June 7th, 2023
  • I had a weird issue with a server SSD.

    6 months ahead of scheduled swap, it didn’t die, it just started reading and writing really sluggishly, making the whole server behave really weird. Disk smart statistics looked healthy and disk self tests passed with flying colors. Anyway, had to swap it early and do a re-install of the OS.

    The rest of my cluster temporarily took over running some pods and only saw downtime for a few pods that were dependent on some disks in the failing server.

    I guess the incident has restarted my interest in distributed storage.

  • Man, I know that feeling. One thing that helped me better deal with issues like this, was to have a changelog. Basically I write down what a setting was, what I changed it to and a reason. If something goes wrong, I can at least undo what changes I’ve made and see if it helps. It’s not perfect, but it might shave some hours off a RCA.

  • I run a pretty barebone Archlinux with several distroboxes. My main motivation for this setup is that I work on a lot of different projects that all have very different setups. Running them in distroboxes make sure I can just drop the box, once the project is finished, and all code and data is just wiped, without having any impact on my main setup.

  • At work we have the following quote on the fridge

    “A ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot — albeit a perfect one — to get an “A”. Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes — the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.”

    We are a software development company and my reply to this was basically that pot making hasn’t changed in a long time, it’s basically shaping and firing clay. Software development is comparatively new and has a vastly more dynamic landscape.

    Also, the comparison is stupid because we don’t write code, realize it was shit and write a new one. If we did business like that, we wouldn’t be in business.