For me, I only back up data I can’t replace, which is a small subset of the capacity of my NAS. Personal data like photos, password manager databases, personal documents, etc. get locally encrypted, then synced to a cloud storage provider. I have my encryption keys stored in a location that’s automatically synced to various personal devices and one off-site location maintained by a trusted party. I have the backups and encryption key sync configured to keep n old versions of the files (where the value of n depends on how critical the file is).

Incremental synchronization really keeps the bandwidth and storage costs down and the amount of data I am backing up makes file level backup a very reasonable option.

If I wanted to back up everything, I would set up a second system off-site and run backups over a secure tunnel.

If it’s happening every 15 minutes, it’s probably a systemd timer trying to kick off a unit on a schedule. Check for .timer files in your system and user systemd configuration and see if there are any configured to run every 15 minutes.

Whatever process is trying to start is probably exceeding the open files ulimit. ulimits can be set system-wide, per user, and per cgroup.

The ulimit may be too low, there may be some process leaking file handles (opening files periodically but never closing them), or the unit might be configured to run under the wrong user or cgroup.

If a reboot gets rid of the problem temporarily, it’s most likely a file handle leak. Remember that objects like network sockets also count as files for the purposes of the open files ulimit.

A tool like lsof can help you track down processes with a lot of open file handles.

As someone whose first “real” programming language was C, I recommend against writing anything new in C.

There are plenty of C projects you could consider contributing to as a way to learn the language (and if you want a long and prosperous career, knowing C can only help you) but language design has come a long way since the 70s and something like Golang or Zig or Rust would get you many of the advantages of C with many fewer pitfalls.

The OG Nazis were actually bozos too, they just had very good propaganda. So good that you’re still seeing the effects today.

I’ve worked in bash. I’ve written tools in bash that ended up having a significant lifetime.

Personally, you lost me at

reading from the database

Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don’t exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that’s table stakes, security-wise.

Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there’s an existing library that wraps the API you want to use in native language constructs.

I worked with dedupe products at a previous job. Media files generally deduplicate poorly.

I haven’t read the article but I work with Bloom filters at work sometimes.

Bloom filters basically tell you “this thing might be present” or “this thing is definitely not present”.

If you’re looking for a piece of data in a set of large files, being able to say “this data is definitely not in this file” saves you a bunch of time because you can skip over the file instead of searching through the whole thing just to figure out what you’re looking for isn’t there.

Most shells ignore setuid on scripts for security reasons.