A couple of months ago I made the ill-fated decision to upgrade my pi-hole from version 5 to 6. Since then, my local DNS resolution has stopped working on devices other than the Pi.

Searching online didn't show up much - the odd forum post where someone changes a seemingly arbitrary setting and things magically work for them. Not so satisfying. So, I figure why not take a deeper dive into how the pi-hole works.

First, the problem. If I run dig blah.home @<pi address> on my laptop I get a timeout error on pi-address#53, where blah.home is in my list of custom DNS entries in the pihole admin page. If I run it on my Pi then it resolves the IPV4 address just fine. Looking at pihole's logs, in both cases the custom DNS file is consulted and the address is found for the A record. On the Pi a AAAA record is also checked. Now, you may say...firewall! Which would be reasonable except external addresses are resolving just fine on that port, as well as CNAME entries.

So what is going on? Well pi-hole serves DNS queries using its FTL server, which is a fork of dnsmasq. So first stop is figuring out how that works. You can enable packet dumping but my pcap game is not strong and I figure looking at packets won't be enough to tell me what exactly is going on.

I already had dnsmasq installed on the Pi and launched it with:

dnsmasq -h -d -q -p 5353 -A /domain.name/192.168.0.10

Here -A is a custom name-address mapping and -h d -q enable 'no-hosts mode', 'no-daemon mode' and 'log query mode' respectively.

On the client side if I then run:

dig domain.name @192.168.0.10 -p 5353

I get on the server:

dnsmasq: query[A] grafana.home from 192.168.0.74
dnsmasq: config grafana.home is 192.168.0.10

and on the client:

;; QUESTION SECTION:
;grafana.home.			IN	A

;; ANSWER SECTION:
grafana.home.		0	IN	A	192.168.0.10

;; Query time: 7 msec
;; SERVER: 192.168.0.10#5353(192.168.0.10) (UDP)

All as expected. If I try the same with my Pi I get:

;; communications error to 192.168.0.10#53: timed out
;; communications error to 192.168.0.10#53: timed out
;; communications error to 192.168.0.10#53: timed out

; <<>> DiG 9.19.24 <<>> navidrome.home @192.168.0.10 -p53
;; global options: +cmd
;; no servers could be reached

What gives here? I should at least see some sort of resolution error somewhere - rather than a refusal to respond at all.

Since there isn't much to go on in the logs let's see if we can build FTL and poke around - following: https://docs.pi-hole.net/ftldns/compile/. I had some trouble building mbdtls due to a seemingly defective archive download link - but otherwise this went fine.

Now we can run directly against the FTL development version:

sudo service pihole-FTL stop
sudo pihole-FTL debug

Some extra logging revealed that the wrapped dnsmasq version was indeed trying to send a message via its sndmsg call. So - time to take a look at some packets after all. Using tcpdump on the pi and my laptop, e.g.:

sudo tcpdump -i wlp2s0 host 192.168.0.10 and port 53

I could see the Pi sending a response back to the A record request - but that there was no response coming on the laptop.

15:37:04.236194 IP rex.local.42138 > raspberrypi.domain: 33845+ [1au] A? navidrome.home. (55)
15:37:04.319626 IP rex.local.34406 > raspberrypi.domain: 64344+ PTR? 10.0.168.192.in-addr.arpa. (43)
15:37:04.327971 IP raspberrypi.domain > rex.local.34406: 64344* 1/0/0 PTR raspberrypi. (68)
15:37:09.238439 IP rex.local.41264 > raspberrypi.domain: 33845+ [1au] A? navidrome.home. (55)
15:37:14.244129 IP rex.local.34274 > raspberrypi.domain: 33845+ [1au] A? navidrome.home. (55)
15:37:19.328530 IP rex.local.46479 > raspberrypi.domain: 61373+ PTR? 10.0.168.192.in-addr.arpa. (43)
15:37:19.351417 IP raspberrypi.domain > rex.local.46479: 61373* 1/0/0 PTR raspberrypi. (68)

15:39:54.619470 IP raspberrypi.domain > rex.59592: 51011* 1/0/1 A 192.168.0.10 (59)
15:39:59.622195 IP rex.36985 > raspberrypi.domain: 51011+ [1au] A? navidrome.home. (55)
15:39:59.622504 IP raspberrypi.domain > rex.36985: 51011* 1/0/1 A 192.168.0.10 (59)
15:40:04.627099 IP rex.51835 > raspberrypi.domain: 51011+ [1au] A? navidrome.home. (55)

This is the second year of my 'end of year recaps', this time for 2025.

As a high-level summary of the year, I didn't make the progress I'd hoped for in my 2025 goals, owing to a combination of increased professional workload and some health issues.

The increased professional workload was 'ok', in the sense that a lot of it involved building interesting things and learning new technologies. In particular:

• I was involved in setting up a new 'platform engineering' team - which has involved a lot of mentoring, process development and design and build of some foundational tooling.
• My main work project involves building a collaboration portal including elements for project proposal review and data sharing. We didn't have a template for building something like this at my employer so a lot of things have had to be set up from scratch
• I tried to continue tipping along with several smaller work projects, including in quantum computing, environmental science and machine learning

Of course, I had some new goals and toys to add to my existing 2024 ones, in particular Linux mobile, I picked up a OnePlus 6 and am running PostmarketOs on it.

My goals for 2025 still carry over many of my unfinished ones from 2024, but there is a gradual convergence toward a coherent set of projects I want to take on. Namely building my own tooling for:

• blogging (like every developer on the planet)
• music and video cataloging and playing - Navidrome is nice but doesn't do everything I want as a single service
• photo cataloging - again, Immich is great but the rate of change and feature addition is getting a bit overwhelming for me. I like to have control!
• family tree (building the tech and collecting the data) - this is a new project and one I'd really like to make a dent in
• Contact and calendar management - including learning CalDav - Nextcloud is just too complex and enterprise focused for what I want
• File management - including WebDav. Again, Nextcloud is great for something enterprisey but I'd like to really understand and control my stack.

So - this is a lot of new projects, some of which also need clients if I want to consume them on my Linux phone. One silver lining is that I've been building up a 'reusable' set of components on my work projects - which are open-source. So my plan is to focus on getting core functionality for these projects together using Django and Angular. Really I'd like something lighter and more exotic than these frameworks in the long term - but to make progress I need to reuse the stack I've invested a lot of time in in the last year.

The next big question is around infrastructure. Next year I'm going to need to upskill in Kubernetes and Ansible at work. I could, and probably should, use them for my side projects. However I still really want to play with more niche and maybe forward-looking tooling in my side-projects - particularly Guix and the LISP ecosystem. So I'll probably try to define and run the infrastructure and services I'm using in Guix at some point during the year, maybe not exactly at the start. This has the added bonus of having a clear impetus to learn Emacs LISP and really get my IDE set up the way I want, which isn't something I've achieved so far.

On the non-technical side I hope to make more progress in the veggie-patch. I grew some veggies in 2025 but my mobility limited what could be done. I'm hoping to add a couple of new raised boxes in 2026 and grow a bit more.

So here's my technical stack and roadmap plans for 2026:

• Blogging platform - Django and Angular
• Switch from Navidrome to a custom built multi-media tool
• Switch from Immich to a custom photo management tool
• Switch from Nextcloud to custom contact, calendar and file management tools
• Build a family tree tool - Django and Angular
• Auth central auth with Keycloak
• Add GotoSocial, or maybe learn enough Activity Pub to do something myself and move my Mastodon account there
• Keep chugging along with Forgejo
• Set up a proper monitoring stack
• Set up wireguard properly - so I can host from my Pi through a light VPS
• Move toward a coherent infrastructure management approach - hopefully with Guix
• Write necessary client apps for my phone - likely with Rust and GTK
• Sort out Podcast app - either contribute to GNOME Podcast or write something, there are a good few features I'd like

This is a post about switching IDE to emacs, what's gone well and what's been a struggle so far.

I started using emacs approximately 18 months ago, switching over from VS Code. My main reason for switching was getting nervy about the dominance of VS Code (which I still think is an excellent IDE) and general fatigue around tooling enshittifaction, privacy invasion and AI integration.

What's gone well?:

• Much less mouse - I learned vim keybindings and have set up a tiling window manager (Amethyst/Sway), which have both been a productivity and comfort revelation
• Treemacs and Projectile - I'm using emacs as an IDE and have a lot of projects - this has been a comforable way to manage them, even more so than in VS Code.
• No sore pinky - I remapped my Caps Key and with Evil mode my pinky is in good shape
• Packages and LSP - There have been packages and modes for most languages, frameworks and tools that I've needed (Angular, Containers, Python, Markdown) and at a basic level they've worked ok

What's been a struggle:

• Keybindings - Evil mode is relatively comfortable, but I still end up fat-fingering often and doing weird and unrecoverable (in an 'undo' rather than 'document loss' sense) things to the buffer. There can be an impedance mismatch with evil vs. native bindings and documentation which can be a pain for learning and tutorials.
• Lisp - Despite several starts I haven't made progress learning it yet, maybe because it's so different from what I'm used to (Python, C++). This makes it hard to customize and fully control my environment.
• Web Development - I have trouble with LSP freezes and crashes with Angular. The Angular LSP devs seem only interested in supporting the 'official' VS Code use-case which makes getting fixes difficult.

Some blasphemy:

• I haven't picked up magit - I have a very simple git flow and am pretty used to it - using magit for it feels like slowing down
• I haven't picked up Org Mode - I want most of my writing to be in Markdown, often required for collaboration with colleagues. I guess Markdown Mode covers a reasonable number of Org features. I'm too disorganised to have todo lists - maybe some day.
• Interactions are a bit slow - I haven't done any optimization but plan to eventually. That said, VS Code is snappy without needing to tinker and as far as I understand I'm limited to a single thread, which makes me think there is only so much optimization that can be done.
• I'm using Vim more - now that I'm more comfortable with Vim bindings I'm often using it for quick document edits or server work, rather than launching emacs.

What's next?

• Get better at navigation and using tooling - I'm still not using search, refactoring and code-base navigation tooling and bindings very well.
• Target evil to certain modes - In some modes (e.g. Info) the evil bindings are getting in the way, I should configure them to be active for a limited set of modes
• Learn some Emacs debugging, profiling and Lisp

To wrap up - my emacs journey has gone well so far and there are no regrets with the switch, especially since VS Code appears to be gradually enshittifying as expected. It's been a good bit of work getting to a basic comfort level and I feel like I still have a long way to go. Maybe I'll do another update at the 5 year mark.

Wifi

• bbs.archlinux.org/viewtopic.php?id=273440
• https://bugzilla.kernel.org/show_bug.cgi?id=215496

Touchpad

• https://www.dell.com/support/kbdoc/en-ie/000150104/precision-xps-ubuntu-general-touchpad-mouse-issue-fix

Raspberry Pi

• https://github.com/raspberrypi/linux/issues/6049

Linux Phone

My Linux phone, a OnePlus6 running PostmarketOS has developed a few issues in the past months. So I'm going to try to troubleshoot them.

No Headphone Audio

First, headphones - wired or Bluetooth no longer played audio - even though they are shown as the default device and GNOME's sound test plays on them. I had many happy months of listening to music and podcasts with then phone - which one day just came to a stop. I don't think it was related to an update - so what gives?

Let's connect to the phone. On the phone:

sudo service sshd start

# Find the IP address
ip addr

On my laptop:

ssh <phone_user>@<phone_ip>

Now, we have some fun dealing with the Linux audio zoo. At the lowest level there is ALSA. We can use the aplay command to list devices and play sound. To list devices:

aplay -l

there I couldn't find my headphones. Under aplay -L the 'PulseAudio Sound Server' is showing the default - even though I think pipewire is running too.

We can use pactl for PulseAudio. First, we check the available cards:

pactl list cards

This lists the built-in ALSA card as 'Card 0' with Earpiece, Speaker, and Headphone ports under a single HIFI profile - with the latter having lowest priority.

'Card 1' is the bluez card and driver - i.e. the bluetooth headset. The Bluetooth headset is showing an 'ad2p sink' and 'handsfree head unit' as its profiles and headphone-output as relevant output port.

Listed clients include the pactl, Gnome settings, Gnome podcasts app.

Now, the interesting bit - the Podcast app under sink-inputs is playing on 'Sink 1' - which is the Built-In Speaker. Why is this? I set it to play on the Bluetooth sink using pactl move-sink and hey presto - now the full sink management logic is working again - apps are playing on the default sink, even after plugging things in and out or restarting the phone. Why? Sorry, I have no idea, it seems like some sort of messed up PulseAudio caching..

I recently (yesterday) installed PostmarketOS with Phosh on a refurbished OnePlus 6. Since this is my first time using a Linux phone I thought it might be useful to track the issues I hit - since you tend to lose the 'noob-insights' as you get more used to a new platform (for better or worse).

Problems

• Can't launch new software in Software app

One of the first things I wanted to do with the Linux phone was show off on social media that I was using a Linux phone, how basic of me. I tried to install a Mastodon client through the Software App and one called 'Pinafore' showed up. After some awkward UI wrangling it marked as Installed. However it doesn't show up anywhere on the phone UI, including the Software app. If I try to launch it from the 'Installed' splash then the Install splash for the Software app itself opens. In the end I did my showing off through the web browser for now.

I didn't look for an issue report for this one yet - but of course maybe most people go straight to the console and use a list like this: https://wiki.postmarketos.org/wiki/Applications_by_function (In the end I installed tuba for this.

• Can't set a pin-like password

The OS comes with a preset pin-like (short, numerical) password. Naturally you will want to change it pretty early on.

Gnome Settings seems like an intuitive place to do it, it has a User profile and a UI to change the password. However it only accepts relatively long alphanumeric passwords, as you might expect on desktop. This doesn't work on mobile - you want to pick up your phone and unlock it pretty quick.

The workaround is to open a console and use passwd.

I found a defect report here: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/issues/2902 but it is marked as an upstream bug - I don't understand enough yet to know which upstream. I haven't found other reports yet (but am still getting up to speed on issue trackers).

• Occasionally can't select file in web browser - clipped UI

While continuing my quest to show off on social media I tried to share a screengrab on Mastodon via Firefox. This worked fine via the File picker on first attempt. Since I messed up the post I tried a second time but now the File picker was sized larger than the screen and I was stuck.

I didn't look for an issue report yet and am not fully sure how reproducible this is.

• Battery level not working

The battery is shown as always charging - I'm pretty sure I saw this in an FAQ/common issues - so will need to dig it out.

Nuisances

• Multiple settings apps

I tried to turn off haptic feedback for the keyboard. I got there in the end but it was awkward finding the setting in the multiple setting apps.

• Camera not working

This is probably a 'Problem' for others but I don't tend to use it much. However one of the reasons for going for the One Plus 6 was the camera support. I'm probably missing something from an FAQ/common issues for this - but 'out of the box', no camera.

Writing a Linux Mobile App

I recently got my hands on a Linux mobile running Phosh and PostmarketOS and am pretty happy with it. However there are a bunch of apps and features that I'd like, particularly integrations with Immich for photos and Navidrome for music streaming.

The most common Linux mobile apps seem to be based in the GNOME ecosystem, using gtk as a base UI toolkit and libadwaita for combined desktop and mobile UI support. I've spent plenty of time working with Qt before - but this is a first dip into the world of GTK.

Apparently new native applications should consider Rust over C and C++ (which I'm pretty familiar with). So, this seems like a good time to try to pick up some Rust as well. This (old) intro for C++ people seems useful: https://fasterthanli.me/articles/a-half-hour-to-learn-rust

Setting up the Development Environment

First, I want to set up a development environment. I'm using Guix, which will make this reproducible - although I hear doesn't play so nicely with Rust.

Let's get a Rust environment going in Guix shell. First I use guix search to see if I can find something claiming to be a rust toolchain:

guix search rust

It looks like a package called rust will do the job:

guix shell rust

I'm aware of Rust bindings for GTK (there is even a book). It seems like rust-gtk4 will do the job. As I build up shell packages it is time to start a manifest:

guix shell rust rust-gtk4 --export-manifest > manifest.scm

and load the shell with:

guix shell -m manifest.scm

I may later regret this, but I'm going to try to bundle the project with meson so will add meson and pkg-config to the environment.

Following the gtk-rs book I start with a hello-world:

use gtk::prelude::*;
use gtk::{glib, Application};

const APP_ID: &str = "org.gtk_rs.HelloWorld1";

fn main() -> glib::ExitCode {
    // Create a new application
    let app = Application::builder().application_id(APP_ID).build();

    // Run the application
    app.run()
}

Now we hit some issues. First, I need some Rust support in my Emacs and second, unsurprisingly, this doesn't compile with rustc since it doesn't know the path to the gtk crate. The emacs rustic package with rust-analyzer seem to be the way to go.

So, how do I combine gtk, rust and meson - well it seems like some horrible mix of cargo and meson boilerplat: captured here https://gitlab.gnome.org/World/Rust/gtk-rust-template/-/tree/master with more background here: https://coaxion.net/blog/2023/04/building-a-gstreamer-plugin-in-rust-with-meson-instead-of-cargo/

Let's go for it.

Guix

Guix is a package manager and derived Operating System based on the ideas of reproducible builds ands free software and the Guile Scheme language.

Using Guix an Operating System and user profiles can be defined declaratively in Scheme. This includes system services, using the GNU Shepherd init system and config file environment through the GNU Home module.

Thanks to the use of reproducible builds Guix is useful for setting up clean development environments with or without full container isolation.

Setting up a minimal Sway environment

Guix allows the development of a minimal computing environment by allowing easy rollbacks to previous generations, allowing experimentation and through tooling like guix graph which can show where dependencies are coming from.

Sway is a reasonably minimal tiling window manager based on Wayland, with good support in Guix.

While a Sway session can be launched through the login shell, it can be nicer to use a login system. greetd is often used for this purpose, with a Sway based 'greeter'. To set this up it is neccessary to set up greetd as a system service in the operating system definition.

Guix Shell

Guix Shell is a tool used to manage, amongst other things, development environments. You can invoke it with:

guix shell

to launch the environment.

Typically you will want some packages in the environment, for example for a gcc toolchain you can add:

guix shell gcc-toolchain

You can manage the packages in the environment with a manifest file. To create an initial one do:

guix shell --export-manifest

You can then append packages to the manifest list as needed.

Further Reading

• https://guix.gnu.org/en/blog/2023/dissecting-guix-part-1-derivations/
• Raspberry pi example: https://codeberg.org/guix/guix/src/commit/757aefeaa4df338e0564044ba344f1daa81c5f10/gnu/system/examples/raspberry-pi-64.tmpl
• Sway and Guix: gofranz.com/blog/all-in-on-sway/
• Sample user setup: https://github.com/MorganJamesSmith/dotfilehttps://bugzilla.kernel.org/show_bug.cgi?id=215496s/blob/master/.config/guix/config.scm
• akirakyle.com/config/guix
• Sway example: https://gitlab.com/endocrin/guix-sway/-/blob/master/guix-sway-config.scm?ref_type=heads
• Sway example: https://gitlab.com/mbakke/guix-sway-example

I like to keep track of places I've travelled to through the medium of bench photos. Here are some of the said benchs.

This article has some notes on packaging a graphical Linux mobile app. Particularly with phosh and postmarketos.

Here are some links:

PostmarketOs Packaging:

• https://wiki.postmarketos.org/wiki/Developing_with_postmarketOS
• https://wiki.postmarketos.org/wiki/Packaging

Alpine packaging:

• https://wiki.alpinelinux.org/wiki/APKBUILD_Reference
• https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package

Librem/Purism Docs:

• https://developer.puri.sm/Librem5/Apps/Packaging_Apps/index.html
• https://developer.puri.sm/Librem5/Apps/Publishing/index.html

Other Apps:

• https://forums.ankiweb.net/t/packaging-anki-desktop-for-aarch64-alpine-linux-postmarket-os/6152
• https://nemomobile.net/pages/Packaging_Glacier_for_postmarketOS/

PostmarketOS Setup

• Experience blog: https://connolly.tech/posts/2023_12-10-im-daily-driving-postmarketos-pt1/
• Qualcom Sdm845: https://gitlab.com/sdm845-mainline
• https://gist.github.com/stuzenz/63a7cac6b396a740b6f1fd470bf30edd

This article is a work in progress review of 'Forge Federation'. I'm writing it to learn about the technologies and groups involved. Eventually I hope to be able to contribute to some of this work - I'm using the article to get up to speed on the area.

The term 'Forge' is used to describe web based code repositories with additional features to support collaborative software development, for example 'pull requests' or issue tracking.

At the moment Github is almost ubiquitously used as a Forge. Ubiquity of any platform, regardless of its current values, is already a problem in my books. It is healthy to have viable alternatives for many reasons. However ubiquity of a closed-source commercial platform is to me very problematic, particularly when it is used to host critical open source software projects.

There are alternative forges, (fully) open source ones are naturally of interest. Codeberg is a nice example. The problem is that this forge is centralized - someone has to pay to keep the lights on and it is a single point of weakness for attack on software projects. The Codeberg service runs on the Frogejo software platform, which is a fork of the open-core (partially open source but with proprietary paid extras) Gitea platform.

Federation of Web Activities is an approach to build decentralized services and platforms. It is showing lots of promise in the area of Social Media, e.g. with Mastodon, Pixelfed, Peertube and a healthy collection of others. Many are built on the 'Activity Pub' procotol.

Forgejo has a current initiative to support federation based on Activity Pub. Its most recent feature is 'federated stars', with federated unstar next. There is a long roadmap (including tag forgejo/federation) toward having a fully federated forge. I think this work to enable Forge federation is foundationally important to allow open source projects to escape the Github trap and the Forgejo project is well placed to deliver it. So, I want to get involved.

To do so in a reasonable way I want to:

1. review the following:

• How Go works
• How Forgejo works
• How Activity Pub works
• Forgefed and Vervis
• Forgefriends https://f3.forgefriends.org/
• How federation in Forgejo works
  • UI https://gitea.com/xy/gitea/issues/8

in this article.

2. set up a Forgejo development environment so I can play around with things, I haven't thought of the best way to do that yet. My own 'production' Forgejo setup is still very basic.

3. see if I can find some good first issues in the Forgejo federation backlog, including helping with tests, documentation or setting up some infrastructure.

Logging and Monitoring

• rsyslog
• syslog: RFC5424

/var/log/syslog /var/log

• eBPF

• Zabbix

• Influx

• Grafana

• Prometheus (+ Grafana + Loki as stack)

• timescaleDB

• AlertManager

• Loki

• Graphite

• Spiceworks

• Crowdsec

• Netdata

• NodeExtractor/NodeExporter

• ELK - Elasticsearch, Kibana, Logstash

https://grafana.com/blog/2016/01/05/logs-and-metrics-and-graphs-oh-my/

Setting up Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/

Setting up Prometheus: https://github.com/prometheus/prometheus

Some things to measure:

• apt status (for security/critical updates that haven't been run yet)
• reboot needed (presence of /var/run/reboot-required)
• fail2ban jail status (how many are in each of our defined jails)
• CPU usage
• MySQL active, long-running processes, number of queries
• iostat numbers
• disk space
• SSL cert expiration date
• domain expiration date
• reachability (ping, domain resolution, specific string in an HTTP request)
• Application-specific checks (WordPress, Drupal, CRM, etc)
• postfix queue size

• apt/yum/fwupd/... pending updates
• mailqueue length, root's mailbox size: this is an indicator for stuff going wrong silently
• pending reboot after kernel update
• certain kinds of log entries (block device read error, OOMkills, core dumps).
• network checksum errors, dropped packets, martians
• presence or non-presence of USB devices: desktops should have keyboard and mouse. servers usually shouldn't. usb storage is sometimes forbidden.

Further Reading

• https://www.redhat.com/en/blog/log-aggregation-rsyslog

Auto updates

This article is my attempt to explain Linux internals to myself and act as a structured way of keeping notes.

Linux is an operating system kernel - which is distinct from the user-space where user applications will run. The kernel directly interfaces with hardware via its firmware and associated drivers. It also manages resources for sharing by user applications. Other open-source kernels that may be of interest are those based on BSD and the GNU Hurd microkernel.

The Linux kernel can be interacted with or controlled from user space by means of a File based API or a C API which aim to be POSIX compatible.

The types of hardware interfaced in the kernel include:

• graphics
• audio
• memory
• cpu
• gpu
• network
• keyboard and mouse
• storage and i/o peripherals

with the kernel handling control messages including power messages for associated devices.

As part of its resource management the kernel includes:

• filesystems and virtual file management
• memory management functionality
• process scheduling and interprocess communication
• networking
• security
• media support

In the user-space of operating systems that use the Linux kernel (will controversially refer to as Linux distros for brevity) some foundational elements and applications are:

• an init system - often systemd and previously collections of 'sysvinit scripts' or niche alternatives like shepherd.
• system daemons for administrative services
• a graphics system - oten wayland now replacing x org
• an audo or multimedia system - pipewire is becoming a common replacement for pulseaudio and others
• a terminal emulator and shell
• network interface management, such as ethernet or wifi
• a package manager
• a bootloader

APIs

https://en.wikipedia.org/wiki/Linux_kernel_interfaces#Linux_API

The kernel tries to follow the Portable Operating System Interface (POSIX) and Single Unix Specification where applicable.

File Based

Device drivers are interacted with in directories:

• /dev https://en.wikipedia.org/wiki/Device_file#DEVFS
• /sys

Processes are interacted with in:

• /proc
• /proc/sys

System calls and similar

https://en.wikipedia.org/wiki/System_call

ioctl (input/output control) is a system call for device specific io operations.

• sysctl (system control)
• ioctl (io control)
• fcntl (file control)

Other communication mechanisms include netlink sockets which allow IPC between both kernel and userspace programs. It is designed to be a more flexible successor to ioctl.

User Space

Init Systems

Systemd

Important utilities:

• systemctl
• journalctl
• notify
• loginctl
• systemd-boot

Important daemons:

• systemd
• journald
• resolved
• networkd
• logind
• user-session
• udevd

Important libraries:

• libnotify
• libudev

Devices

Devices are managed in user space with the udev utility - which has the following parts:

• libudev which can be used as a library for device info
• udevd daemon for managing the /dev virtual file hierarchy
• the udevadm command line utility for admin and diagnostics

Window management and device input events

Wayland and X are the two most common display servers on Linux systems. They both follow a client-server approach, where the latter in Wayland can be part of the compositor.

Display servers can react to device input events via the libinput library, which in turn uses libevdev to handle evdev ioctls from the kernel.

Network

iptables allows configuration of IP packet filter rules in the Linux kernel firewall, which are implemented as netfilter modules. nftables is a newer userspace utility that replaces iptables and similar.

netfilter operates by providing hooks to be called at different stages of packet processing, allowing decisions to be made on the fate of the packet. The user-space side of iptables has the concept of 'tables' - which define some function to perform on a packet and 'chains' which correspond to when the functions will be applied, mapping to the netfiler hooks. Chains has a priority controlling their order of application on the netfilter hooks.

The following are the netfilter hooks and corresponding built-in iptable chains:

• NF_IP_PRE_ROUTING / PREROUTING
• NF_IP_LOCAL_IN / INPUT
• NF_IP_FORWARD / FORWARD
• NF_IP_LOCAL_OUT / OUTPUT
• NF_IP_POST_ROUTING / POSTROUTING

The following are the available iptable tables:

• nat: network address translation
• filter : Where a packet can proceed or not - essentially firewalling
• mangle: IP header modification
• raw: allows marking packets as part of a connection
• security: used in SELinux and similar applications

iptable rules are placed in a specific chain of a specific table. Rules have a 'matching' component and a 'target' component, which gives an action. The actions can be terminating which stops chain evaluation or non-terminating.

avahi is a zero-configuration networking implementation (allows network service use by freshly networked computers or peripherals) including multicast DNS and DNS service discovery. Apples' Bonjour and Systemd's systemd-resolved are other implementations.

The Desktop-Bus (DBus) is a user space middleware allowing communication between multiple processes (e.g. IPC).

• https://www.digitalocean.com/community/tutorials/how-the-iptables-firewall-works

• https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture

• https://www.digitalocean.com/community/tutorials/how-to-choose-an-effective-firewall-policy-to-secure-your-servers

• https://www.chiark.greenend.org.uk/~peterb/network/drop-vs-reject

Audio

ALSA (Advanced Linux Sounds Architecture) - has both kernel and user-space elements, the later with alsa-lib.

Pipewire is a new low-level multimedia framework.

Power Managemenr

UPower (previously DeviceKit-power) is a power manement middleware - it spawns upowerd.

Systemd has some session and power management through its logind. The elogind is a standalone fork of this logind.

The systemctl suspend analogue for elogind is loginctl suspend.

Other

• avahi
• dbus
• udisks
• cgroups
• autofs
• kdbus
• Plokit (Policy Kit) https://en.wikipedia.org/wiki/Polkit
• Pluggable Authentication Module (PAM) https://en.wikipedia.org/wiki/Pluggable_Authentication_Module
• Name Service Switch (NSS) https://en.wikipedia.org/wiki/Name_Service_Switch
• procfs https://en.wikipedia.org/wiki/Procfs
• sysfs https://en.wikipedia.org/wiki/Sysfs

Kernel Space

Input events

• evdev
• /dev/input

Filesystems

• ext4

• btrfs

• xfs

• jfs

• fat32

• FUSE

Storage

• SCSI
• libATA

Virtualization

• KVM
• Xen

Process Management

• clone(2) and clone3(2)
• futex(7) and futex(2)
• Completely Fair Scheduler
• https://en.wikipedia.org/wiki/Earliest_eligible_virtual_deadline_first_scheduling
• Native Posix Thread Library (NPTL) is kernel side of pthreads

Security

• Linux Security Modules
• SELinux
• AppArmor
• POSIX ACLs

Memory

• DMA buffers

Audio

• Advanced Linux Sound Architecture (ALSA)

Graphics

• Direct Rendering Manager (DRM)
• Kernel Mode Setting (KMS)

Network

• New API
• mac80211
• Netfiler

Power and Control

• ACPI https://en.wikipedia.org/wiki/ACPI

PAM and Login

• https://kl.wtf/posts/2022/03/12/login-managers-an-introduction.html#pluggable-authentication-modules

This is a work in progress article I'm using to build up my understanding of computer networking.

Standards

OSI model

The Open Systems Interconnection model is a basis for standards development in systems interconnection.

Physical Layer

Converts digital bits into electrical, radio or optical signals

• NIC: Network Interface Controller
• Ethernet hub
• Network switch
• Physical transmission media

Standards at this layer include Bluetooth, Ethernet, USB

Data link layer

• Medium Access Control Address (MAC) https://en.wikipedia.org/wiki/MAC_address
• Logical link control (LLC)

Others

• network
• transport
• session
• presentation
• application

IEEE 802

Family of standards for local area networks (LANs) and other types.

Terms

• Packet filtering
• Network address translation (NAT)
  • masquerading
  • port forwarding
  • redirection
• Port translation

Internet Protocol Suite

• RFC 1122
• RFC 1123

Link Layer

• Address Resolution Procotol (ARP) https://en.wikipedia.org/wiki/Address_Resolution_Protocol

Transport Layer

• TCP
• UDP
• QUIC

Shell

• tmux

Backup

• Restic

Music

• Sublime Music

Self Hosting

This article is intended to be an ever-evolving guide to setting up a self-hosting solution, put together as I try out and learn various steps myself.

Location

I want to host with my own hardware where feasible, mostly so I have control over costs but general control and independence reasons are appealing also.

I don't have a static IP and don't really want the fuss of getting one, nor making my home IP publically visible. For this reason I've opted to run a small VPS on an external hosting provider to act as a public gateway to locally hosted content. I also want to run a VPN on this VPS, so I can access services only visible to my home network while away from it. For the external provider I went with Hetzner for reasons of price, reputation and that they are based in Europe. I wasn't too happy with the extent of personal data they collect during registration but understand why they do it.

Hardware

I have a hodge-podge of old hardware laying around and where possible am trying to use that over buying anything new. That said, my Raspberry Pi was ancient and I don't think it was up to the task of running my planned services, so I grabbed a Raspberry Pi 5 with 8 GB RAM. I had a couple of old SSDs - one 256 GB and one 2 TB, so hooked them up via PCIe to USB connectors. The Pi is running on WiFi at the moment, although I could attach it to Ethernet over the house wiring if there is a performance issue.

Public Host

I'm running a Debian image on a small Hetzner VPS. This hosts public facing services, such as my personal website and Forgejo code repository.

Home Network

My home network just consists of a Raspberry Pi based server for now - with access from various laptops, mobile clients and a Smart Tv. I don't have any short term plans to add to that. It isn't exposed to the public internet but is planned to be accessible by VPN.

Operating System

For now, I've gone with the default Debian image for the Raspberry Pi. I wasn't too careful setting it up so have ended up with a full blown desktop installation. Hopefully in future I can eventually get Guix running on there, which I'm currently happily using on my laptop.

Core Services

These are the core services that support access, monitoring and security on my system.

DNS

I'm using pi-hole for ad blocking and am planning to also use it for internal DHCP/DNS management on my home network.

For starters I set it up to act as a DHCP server. Before starting I needed to configure the pi to get a static IP, this is through /etc/network/interfaces.

Then I followed option 'Alternative 1' on the pi-hole getting started docs:

git clone --depth 1 https://github.com/pi-hole/pi-hole.git Pi-hole
cd "Pi-hole/automated install/"
sudo bash basic-install.sh

Finally I forced my router's DHCP server to only assign one IP, a fixed one to the pi and then enabled DHCP serving on the Pi.

VPN: Wireguard

I want to use Wireguard to allow access to services on my home network that are not intended to be public. It will also allow serving of services that are meant to be public via the VPS and reverse proxy.

https://www.digitalocean.com/community/tutorials/how-to-set-up-wireguard-on-ubuntu-20-04

Backup

Considering rsync or BorgBackup

Updates

Manual or watchtower

Monitoring

Undecided

Services

Credentials

Vaultwarden

Email

Custom self host for receive with, custom and gmail for sending

Photos

Immich

Music

Navidrome

Documents

Nextcloud

Calendar, Contacts, Notes

Nextcloud

Code

Forgejo

Personal Site

Proxied through VPN and reverse proxy

Social

Self host Mastodon or GoToSocial

This is a recap of some technical projects I've picked up, started or finished in 2024 and things I'd like to pick up in 2025.

Relevant Background

I'm a software developer with an interest in free software, open science, minimalism and yak shaving. I've been using Linux for about 15 years and finally rid myself of all things Windows some time last year.

I had mostly used Ubuntu for personal computing and a little CentOS/RedHat at work, but started looking around for other distros when they introduced Snaps and killed CentOS 8 respectively, cementing my feelings that corporate distros should only be used for corporate things. My distro search included brief spells with:

• Debian - found packages too out of date
• Fedora - found dnf slow on my clunky old laptop
• Gentoo - never got it fully set up correctly

before settling on Arch some time in 2023.

I've primarily worked with GUI tools including IDEs and heavy mouse use, dropping to the terminal only when needed.

2024 Recap

Operating Systems

I had been running Arch for much of 2023 using Wayland and suckless tools as much as possible, e.g. dwl. The system was stable but I struggled with the keyboard focused flow and sometimes needing a tool in a hurry - e.g. joining a wifi network on the road and trying to remember the terminal commands to do so.

I switched to Debian Sid early in the year - with the lofty goal of being able to contribute useful defect reports and maybe being able to get involved in some package maintenance. I switched back to GNOME and mousing about at the same time, which was a bit of a relief although floating windows did start to grate a bit.

I learned about Guix and along with being interested in learning Emacs decided to give the Guix System a shot, around March some time. I found the initial setup a bit tricky - since I needed some non-free kernel bits, but (presumably with some thanks to previous Gentoo experience) eventually pulled a system together. It has been rock solid since and I think my distro hopping days are over for now.

The big appeals of Guix to me are:

• having control over the assembly of the system and being able to build something minimal
• having a clean language to configure the system (Guile Scheme)
• its free software and community focus
• use of an alternative to SystemD - I think competition and having choice are healthy and I'm not running an enterprise on my laptop
• maybe some day using GNU Hurd - did I mention the yak shaving?

With the switch to Guix I switched to Sway as a window-manager, rather than dwl. Interestingly it is more minimal than I had thought feature-wise, properly setting up sway-lock and sway-idle are still on my todo list. This allowed me to remove a lot of GNOME and GDM things from my Guix setup (with David Wilson's awesome blog, videos and repo doing most of the heavy lifting). There is still a long way to go in terms of my understanding the lower level Linux userspace before chopping more off my Guix config, but that is what 2025 is for.

IDEs

Switching IDE was definitely my biggest and most time-consuming technical project of 2024. I haven't been happy with my IDE choices - basically ever. The Eclipse IDE was one of the first I used for Python and C++. It is free software and I'm grateful to the developers for building it, but I found every single aspect of it painful to use, from crashes to lockups and runaway resource use while indexing to a painful plugin management system, and did I mention the Java?

After many years of using Eclipse I used Visual Studio while working on a Windows-centric product for several years. It was adequate but I don't like the Microsoft C++ development environment for several reasons. Once I no longer had to write code for Windows I switched to VS Code on all platforms. VS Code is an amazing IDE, performance is great and tooling and plugins 'just work'. In my current role I write tutorials and best practice documentation for other developers and the general public. I'm really not a fan of Microsoft and although there is 'free' VS Codium with some of the nasties stripped out I'm wary of the now dominance of VS Code and the possibility of a web-browser like co-option of standards by a small number of large corporations. so, it didn't feel good to me to make documentation with this IDE as the prominent option.

This left me with the choice of Emacs and the Vi ecosystem. Since I was trying out Guix at the same time and was interested in the idea of LISP I decided to give Emacs a try. The experience is worthy of a dedicated post - but needless to say that transition was hugely time consuming and challenging - but I'm out the other side now and things are looking good. Once again I'm finally satisfied I've found a platform I can stick with for a long time.

Self Hosting and De-Googling

2024 has been a good year for first steps at self-hosting and degoogling. Some of the things I set up have been:

• Moving from Github to Codeberg and then primarily a self-hosted Forgejo instance
• Moving my personal site from Github pages and Python pelican generator to a custom generator and self-hosted server
• Making a self-hosted dynamic website for my wedding with CSS and Vanilla JS
• Moving all of my credentials to Bitwarden
• Moving my mail from Google to Migadu
• Dropping my Spotify and using Bandcamp, with self-hosted Navidrome for listening
• Archiving my photos via Immich self-hosting
• Setting up a Raspberry Pi for local self-hosting
• Running pi-hole for DNS blackholing and DHCP

Languages

2024 hasn't been very active for picking up new languages or language features. I mostly got back up to speed with recent developments in Typescript (and Angular) and Python. For Python the main things I picked up were the packaging system changes (e.g. Pyproject.toml), use of linters and the typing system (which I'm enjoying).

I did learn a bit about RPM packaging at the start of the year, which is possibly forgotten by now.

One change I did make was trying out more functional approaches to programming. I learned programming via OOP and have been using that everywhere. For some Python projects I've tried a more functional approach with immutable data classes and functions without side-effects and it has been pretty satisfying - I'm going to go further down this road in 2025 with language choices.

Side Projects and Open Source Contributions

2024 has been a mixed to disappointing year on the side projects front. For several (at least 5) years now I've been trying to build a library of 'things from scratch', e.g. a :

• compiler
• build system (e.g. make)
• pdf, png, midi, wav, svg, xml, json, yaml readers and writers
• 3d renderer
• GUI system
• static site generator (including markdown, html parsers)
• web server and client

I had originally written them in modern C++, but it didn't feel 'from scratch' enough. So I wrote replacements for STL containers and memory management - basically seeing if I could run something without the C++ standard library. If you are shouting 'why?!' right now - know that I am too.

The original goal of this 'from scratch' library was to build tooling for myself to use - since I mostly end up hitting rough edges or things I don't like about other tools. That and of course the 'for fun' or yak shaving element. However with Emacs and increased comfort in keyboard-driven workflows I need this tooling less now. I hope to use pieces of this library in 2025, while learning some of the other languages, particularly C, Rust and Guile Scheme.

The side project I did get completed was a website for my wedding. This was a 'from scratch' HTML/CSS/Vanilla JS frontend with a Cherrypy backend. It came out looking fairly ok, although building it at the same time as planning a wedding was probably unwise.

I am still yet to contribute to a (non-work) collaborative open source project - not good. Hopefully that will change in 2025.

Analysis and Theory

I haven't spent much time brushing up on analytical or theoretical skills this year - I've had a brief interaction with computational mechanics again, namely crystal plasticity theory but I can see more theoretical research on the horizon in 2025.

Sustainability

I grew my own vegetables in my small back garden this year including setting up a chunky box-planter. This was very successful and should hopefully be back next year.

I cycle to work most days but it has been a tough year for bike maintenance. I have a weird racer bike and seemingly need to do most maintenance myself - anything that can break on the bike has this year. Learning how to fix each issue has been slow.

2025 Plans

And so, the lofty plans for 2025...

Operating Systems

• Write up how low-level Linux ecosystem services work and fit together (e.g. PAM, elogind, DBus, Wayland seats) in the context of Guix, so I learn about them
• Handle power management on my laptop - including Sway Idle, at the moment I don't know how to suspend or check if it is in low power mode
  • Add a little icon to sway bar for charging
• Handle sound management on laptop - currently I have no keyboard control over sound levels
• Learn about and implement Guix Home
• Further customize and simplify my Guix config (yes, nice and vague)
• Learn about Nix and Nix Home for my work Mac

IDEs

• Try to not depend on treemacs so much - try dired only or similar
• Add email handling to Emacs config
• Learn more Emacs and Vim keyboard shortcuts - particularly document navigation and search
• Learn basic regex
• Learn and document how Emacs packaging works
• Learn magit properly
• Find a comfortable way to work with Emacs shell
• Find a comfortable way to browse web with Emacs
• Learn to use a LISP REPL in Emacs
• Learn Python and C debugging and profiling in Emacs
• Come up with a comfortable Emacs and Vim config for remote terminal sessions.
• Troubleshoot slow initial file search in Emacs on Mac
• Link Emacs forge support with forgejo

Self Hosting

• Set up a backup solution
• Expose locally hosted services via Wireguard and cloud instance
• Set up Nextcloud for documents, calendar and contacts
• Set up own mail server for receive
• Move forgejo to local hosting and set up a Foregejo runner
• Set up service monitoring
• Set up local network monitoring and altering
• Network my USB printer via an RPI

Languages

This is the area where I have most FOMO in terms of what to spend my time on (FWIW I have zero FOMO on not learning AI APIs or prompt engineering). There is such a rich environment of modern languages and development ecosystems to choose from and it seems there is plenty of scope for exciting developments.

At a meta level - I would like to learn how compilers work and programming language theory and design, but I'm not sure 2025 will allow me to go too deep there.

C

Of course, I 'known' C - I've written snippets of code and tutorials in it and have lots of C++ experience, but I don't 'know know' C. I've never written enough of it to have to deal with complexities, edge cases or to have thought about code style. I'd like to learn more of it since it is still foundational to low-level Linux utilities - which I want to understand, modify and maybe contribute to. This year I'd like to write a full side-project or two in C.

Rust

I like the idea of Rust and I'm tired of C++ and its package management disaster zone. Still, I'm wary of Rust's own approach to package management - seemingly following the 'pip' and 'npm' roads. I'm also wary of static linking and the complex build and compile ecosystem. While I do think it is possible in theory to write moderately safe C++ with suitable use of its modern pointers, I have sadly been exposed to large C++ projects in the real world.

So overall, the future is looking bright for Rust and all impressions I've read are that the language itself is pleasant to use. So, I'm on board. Hopefully I can get a project or two completed in Rust this year.

Guile Scheme

I'm quite sure I want to learn some Lisp - I've read too much on how different, powerful and fun it is not to and would like to build up my functional programming skills. Since it is now powering my operating system and IDE, it would seem important to know a bit too.

From reading around Scheme seems a well regarded Lisp dialect to pick up, and given my 'free software' leanings and use of Guix then Guile Scheme seems a good implementation. So, hopefully some Guile Scheme projects are on the horizon too.

I'm curious about more 'powerful' dialects I've heard about too, like Racket and Clojure although they may be for another year - along with Haskell on the functional programming odyssey.

Side Projects and Open Source Contributions

Personal Site

I'd like to get my personal site into better shape as a blog, including:

• supporting blog posts
• supporting RSS
• looking a bit nicer

Currently it is being built using Python - I'm thinking about using Scheme instead, possibly with Rust for Markdown parsing - which is related to another project of interest.

For blog posts, some I'd like to cover include:

• A review of the Supercomputing conference SC24
• A retrospective on moving from Vs Code to Emacs
• A review of my vegetable patch experience

I'd also like to start making some videos - so some OBs practice is on the cards.

Open Source

I'd like to contribute to Forgejo and particularly efforts around federation. I think that is a really promising feature for Open Science (which I think has no business depending on Github) where labs or institutes host their own forges with rich linked data features that Github will never build.

I'd also like to contribute to Guix - I'm not sure how yet but maybe I can do something useful around infrastructure or testing.

Creative

I'd like to get back making 3d renders and some electronic music - I'm not sure what form they'll take yet - maybe I can involve Scheme somehow.

Analysis and Theory

Some areas I'd like to cover more:

• Continuum Mechanics, Crystal Plasticity Theory and Finite Element solvers - getting back up to speed
• Implementation of digitial quantum simulators - e.g. how to solve quantum circuits on classical computers
• How transformers work in Machine Learning - my ML knowledge pretty much stops in 2017

And Beyond...

What is the big plan or picture? Basically:

• having a minimal, contrarian computing setup that I understand well and have a good degree of control over. Something involving Guix, Hurd, Emacs etc.
• having a comfortable setup for creative activities: writing, 3D rendering, video creation and music creation
• having a low footprint hosting setup - both financial cost and resource use-wise
• keeping up-to-date with modern technology and being able to tell hype from genuinely interesting progress

This is a collection of some blogs I've found interesting. Sometimes I move specific links into more dedicated articles.

• MÁIRÍN DUFFY: Open design, Senior principal interaction designer at Red Hat.
• Austin Henley: Developer tooling in academia and industry.
• Terence Eden: General tech ranting
• Neil Brown: Open source stuff and self-hosting - good level of technical detail
• Stephen Brennan: Low level C stuff
• Chris Wellons: C and C++ programming
• Ashton Wiersdorf: Programming language design - lispy stuff
• Leon Plickat: Linux systems tooling, programming
• Futurile: Guix stuff

This is an unstructured list of web-sites I find interesting. I keep them here rather than as a collection of browser bookmarks as I find this format easier to structure. Some get pulled into more structured articles as references or 'further reading' and eventually removed from this list. So in a way, this is sort of a 'to-do list' or staging area.

Tech compilations

• Digital Superpowers
• Arch overview open source
• Build your own x
• My Software Bookshelf

Compilers

• LLVM Tutorial
• Resource compiler
• Bison
• ELF Format

Assembly

• Asm hello world
• Linux syscall list

Languages

• Python module of the week
• Lesser known parts of the Python standard library
• OCaml Book
• C Coding Style
• Language power with Racket

File Formats

• Zlib spec
• zlib overview
• PNG spec
• Deflate spec
• png tutorial

File Systems

• Lustre

Data Structures

• Implementing a vector

Shell

• Introduction to bash scripting
• Command Line Interface Guidelines
• Terminal text
• TTY demystified
• Write a shell in C
• Mac CLI Tips
• Command line tools

Graphics

• Polygons and meshes
• xcb
• xcb
• opengl
• opengl
• cool animations
• Ocean waves
• Book of Shaders
• Demoscene
• More demoscene
• Demoscene basics

Audio

• Midi format
• midi format
• Midi format
• alsa

Text Editors

• Sysadmin guide to vim
• Sample emacs config
• Email in emacs

Networking

• sockets
• http made easy
• Wireguard for private network
• Sysadmin tools

Database

• Writing a kv store

Containers

• Linux containers

Web

• Drawing divs
• Dockerizing Django

Database

• Database storage structures

Publishing

• A Few Notes on Book Design
• MathMl fonts
• Text rendering
• The Visual Display of Quantitative Information

Architecture and Design

• Clean Coder Blog

Security

• Fuzzing GStreamer
• Joshua Rogers

Maths

• The Matrix Cookbook
• The Tensor Cookbook

Machine Learning

• MLE
• Gradient Descent Overview
• Generative Models Overview

Linked Data

• DOI URI Scheme

Games

• Flappy Bird C Clone

Operating Systems

• Unix History

Guix

• Soft dev environment blog

Live and Minimalist Computing

• Emacs like editor on steroids
• Direnv and .envrc files

Hardware

• Making an analogue synth

Gardening

• Planning a vegetable garden for beginners