What's up with test.thing (asg2025)

`test.thing` is a VM runner which targets guests using an API defined by systemd. It started after a conversation at devconf about turning `mkosi qemu` into a library. A quick intro. ~~composefs is an approach to image-mode systems without the disk images. Files are stored in a de-duplicated content-addressed storage with integrity guaranteed through fs-verity. The last year has seen an acceleration of development on composefs-rs, a pure Rust implementation of the ideas behind composefs. Our goal is unification of the storage of bootable system images (via bootc), application Flatpaks,...

Privilege delegation for rootless containers, what choices do we have? (asg2025)

Going for minimal containers with restricted system calls and unprivileged users is the usual Kubernetes approach these days, and it works great for most web apps. However, the development of more complex infrastructure extensions frequently hinders application functionality. While looking for a solution to deploy virtiofsd in an unprivileged container for KubeVirt, we stumbled on seccomp notifiers. Seccomp notifiers are a kernel feature which monitors syscalls and get notifications to a userspace application when a syscall is executed. Alternative options involved either the use of a custom protocol using UNIX sockets or the deployment of...

container-snap: Atomic Updates from OCI Images using Podman’s Btrfs Driver (asg2025)

Traditional package updates using tools like RPM or Zypper can introduce risks, such as incomplete updates or accidentally breaking the running system. To overcome these challenges, we developed **container-snap**, a prototype plugin designed to deliver atomic OS updates—updates that are fully applied or rolled back without compromising the system's state. container-snap leverages OCI images as the source for updates and integrates seamlessly with openSUSE’s [tukit](https://github.com/openSUSE/transactional-update) to enable transactional OS updates. By utilizing Podman’s btrfs storage driver, it creates btrfs subvolumes directly from OCI images, allowing systems to boot from the OCI image. This approach empowers...

OS as a Service at Meta Platforms (asg2025)

I overview how OS management is done at Meta. We run millions of Linux servers and we have to make sure that OS gets updated on all of them in a given period of time. To do that we developed several products: MetalOS (Image based version of CentOS), Antlir (image builder) and Rolling OS Update (a service that keeps a set of DNF repos in sync with upstream repos and uses them to update OS ) Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/VNCDRL/
Video:asg2025-332-eng-OS_as_a_Service_at_Meta_Platforms_hd.mp4

Unprivileged Containers, with Transient User Namespaces and ID Mapping, but Without SETUID Binaries (asg2025)

Many traditional container engines make use of the "subuid" concept and the "newuidmap" tool to implement a concept of "unprivileged" user-namespace containers on Linux. This approach has many shortcomings in my PoV, from both a security and scalability standpoint. Recent systemd versions provide a more powerful, more secure, mor scalable alternative, via systemd-nsresourced, systemd-mountfsd and other components. In this talk I want to shed some light on the problems with the "old ways", and in particular focus on what the "new ways" bring to the table, and how to make use of them in container runtimes. Licensed...

Details of a Scam

Longtime Crypto-Gram readers know that I collect personal experiences of people being scammed. Here’s an almost:

Then he added, “Here at Chase, we’ll never ask for your personal information or passwords.” On the contrary, he gave me more information—two “cancellation codes” and a long case number with four letters and 10 digits.

That’s when he offered to transfer me to his supervisor. That simple phrase, familiar from countless customer-service calls, draped a cloak of corporate competence over this unfolding drama. His supervisor. I mean, would a scammer have a supervisor?

The line went mute for a few seconds, and a second man greeted...

Systing: tracing for the lazy (asg2025)

Systing helps you solve problems in minutes rather than days. Out of the box it gives you everything you could possibly need, combined with perfetto’s visualization you will never be confused again. This talk will introduce systing, a tracer that is built on modern BPF tooling, purpose built to debug large applications with complicated interactions. This will be little talk and mostly demo. Two decades of experience debugging kernel problems has been poured into this tool to make it as straightforward as possible. I will walk through the basic usage, and show a case study investigation...

systemd: round table (asg2025)

Let's have an open discussion with systemd developers who are at ASG and users in the audience. We will open with the developers saying what they plan to work on in the near future, and then allow questions / comments from the audience. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/PXZGEL/
Video:asg2025-338-eng-systemd_round_table_hd.mp4

Why you should contribute to systemd! (asg2025)

I'll use these 20 minutes to explain why and why contributing to systemd is a great experience. We'll avoid beating dead horses by not discussing git forges and email, but instead focus on the development experience, from building systemd distribution packages from git main, running integration tests against those distribution packages, debugging failures, writing new tests, and installing the distribution packages on real hardware to debug issues. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/B8LJKD/
Video:asg2025-341-eng-Why_you_should_contribute_to_systemd_hd.mp4

Sandboxing services with Landlock (asg2025)

Landlock is an unprivileged kernel feature that enables all Linux users to sandbox their processes. Complementary to seccomp, developers can leverage Landlock to restrict their programs in a fine-grained way. While Landlock can be used by end users through sandboxer tools, there is currently no well-integrated solution to define security policies tailored to system services. Although AppArmor and seccomp security policies can already be tied to a system unit, we aim to provide a more dynamic, standalone, and unprivileged option with Landlock. In this talk, we'll briefly explain what Landlock is and highlight its differences...