Brendan's Ramblings

<< < Index > >>

My tale of Nix

2022-06-23

Background

I first learn of nix(os) the same way I learnt about rust, from two server evangelists (K900 and Greaka).
I had originally dismissed it as something that seemed too complex for my use cases and put it out of my head.

Now as you know I was a first year compsci student.
Before starting uni I spent time cleaning up almost all my projects so they could run without too many issues.
This was to ensure my attention would not be divided for college. And it seemed to be working.

Then came along Match 2022 and I get an email from Hetzner.
The emails stated that something on my server was scanning other servers on teh local network.
That is a big no-no.
It took me a little over 3 weeks to find the root of teh problem, my vanilla, up to date wordpress docker container

It is time

While the above incident was inconvenient it was a relatively easy problem to solve.
What it hilighted though was the fact that although I do have some backups in case of data loss I didnt have teh same for config settings.
Nor did I actually know/remember what was installed on teh server.

It was at this point I remembered Nix(OS), asked briefly about it in discord and shelved it until I finished my exams.

K900 and Greaka were delighted I expressed interest.

Prepwork

When my exams finished I took some time to take a look at the ecosystem I decided that usign flakes would be my best bet for moving forward.
I am grateful to the many folks who have done blog posts about flakes.
What drew me to flakes was how the specific dependencies were handled, via a lock file like node.js or rust.
Non flake nix seemed to be more hands on and that was not to my liking as I am quite prone to errors.

My goal was to create a parallel system that I could bring up to par with my server.
This took a fair chunk of work as I had to nix-ify all my existing programs/projects.

For some it was easy enough, others I had to spend a fair chunk of time getting it right.
The easiest ones were teh standalone compiled binaries that did nto requite an external database.
The MongoDB database was a pita as well since the one at the current time of writing did not compile, but that was sidestepped by pinning an older version.

My JS stuff was mostly fine on teh backend/cli stuff, but frontend was a bloody nightmare, in part due to teh number of dependencies.

I had some docker stuff, some I converted to native programs like wireguard. Others I preferred to keep it as a docker container like Valhiem and Satisfactory, I am using arion for the two of them.

I learnt a lot about Systemd in a short while, which I was already beginning to move twords with my recent rust projects.
There were a few bumps with using SystemD timers instead of cronjobs, but they bring far better logging and flexibility (also created my first nix function to make them easier to set up). The bumps were mostly that I now had to cleanly exit the program/script, which combined with teh better logging lead me to fixing many long standing bugs.
The level of logging I now have natively is feckin fantastic.

I am using agenix to store secrets, and while it initially felt strange committing secrets, now it feels pretty good.
Still need to figure out a way to easily commit on windows, maybe the docker image?

Installing

Moving day was hectic.

I had to:

Of these the installation was the most difficult.
My first attempt got nixos "installed" but after a reboot it had vanished.
Second attempt was more successful (actually read teh instructions about the partitions), however the server I have has a raid 1 config, and only one disk was being used.
The third attempt was successful, I reinstalled the default debian OS that Hetzner provide and used NixOS-Infect on it.

I had several false starts but I am really thankful to the technician who left teh KVM attacked for longer than I had booked for (it being a bank holiday monday probably helped).

Pulling my config in and getting it to work was not that difficult because I had been using teh parallel system and had worked out most of the kinks/pitfalls.

CI/CD

The next step after all of that was to setup my CI/CD pipelines.

While I can be a lazy fucker I do like to set things up well (I strive to be the top graph here).
In this case I like to be able to commit and push to git(lab) and it gets automatically deployed.

Previously all my programs were independent of each other so I could commit and deploy freely.
Under nixos the flake.lock had to be updated.

It took a little while to sort out but thanks to Gitlab's Multi project pipelines it ended up being quite nice. My NixOS repo is teh core of the process, every single time a child package is updated the pipeline updates teh lockfile and deploys it to my server.

This lends itself a couple of advantages:

To Do

There are a few things I do have to work on.

Backups

Greaka has a really nice backup system which I intend to yoink.
There are a few financial things thats holding me back (my bank is closing down) but it is in progress.

I also need to figure out how to properly backup mongodb on an ongoing basis.

Logging

I have plenty of space on the server but I may need to tone down the logging a bit, logrotate and all that.

Email

I plan on looking into migrating my email from mailcow to somethign nixos based, to help avail of the backup listed above

Side effects

Interesting projects are the ones where i feel like they have impacted the way I perceive and do things outside them.
Typescript made me fall in love with the safety of a good type system.
Rust taught and continues to teach me better ways of writing code.

Nix has taught me the value in locking in dependencies.
Lately I was doing some minor work on my homelab and I accidentally broke my gitlab instance.
I had done a pull to update all the containers and I updated them, turns out this did a major update to gitlab.

This got me thinking that virtually every docker/compose example uses latest for teh image.
This can make sense in documentation because then you dont have to update it every patch but it can catch new users off guard.
I now specifically have teh tag and version set on it, but latest now irks me and that is a good thing.