Linux 6.18: An update to resolve deadlocks when reading many files by Systemd units

The imminent release of Linux kernel 6.18 provides a major technical fix for an issue that has affected a wide range of users and system administrators using Systemd in their GNU/Linux environments. This update comes after several reports of severe crashes when Systemd units read large numbers of files on systems mounted with the lazytime option. In response to these crashes, kernel developers and engineers have focused their efforts on optimizing dirty inode handling and drastically reducing the CPU overhead caused by these situations. This improvement is also essential for leading distributions such as Red Hat, Ubuntu, Fedora, Debian, SUSE, Arch Linux, and OpenSUSE, which rely heavily on Systemd for their system orchestration. Systemd-Related Deadlocks During Massive File Reads: Causes and Mechanisms in Detail

The deadlocks identified in Linux 6.18 stem from the way Systemd, the default service manager in many distributions, handles drives that read countless files on volumes mounted with the lazytime option enabled.

Lazytime is a file system mount option designed to optimize performance by limiting write operations to file timestamps (access, modification, and change times). Rather than immediately writing these timestamps to disk, lazytime only updates the in-memory version (inode in-memory). Physical writes are deferred until an important event such as an fsync or an in-memory eviction.

The problem arises when Systemd manages tasks or units that require the simultaneous reading of hundreds of thousands, or even millions, of files. At the end of such a unit (particularly in cron scripts or large-scale tasks), the internal logic for switching these dirty inodes from a cgroup (control group) to its parent cgroup proves extremely inefficient. This inefficiency stems from the fact that the function

inode_do_switch_wbs()

, responsible for switching these inodes to the appropriate list of dirty inodes, has quadratic algorithmic complexity. This means that if a cgroup has to process N number of dirty inodes, the time required increases proportionally to N², a fatal bottleneck in the case of large volumes. In practice, this results in prolonged system lockups, which can last several hours, with a 100% saturated CPU. These states negatively impact responsiveness and stability, sometimes causing service interruptions or requiring a manual restart. Key explanations for the major causes:Lazytime option to alleviate metadata writes

Systemd executing units with large file reads

• inode_do_switch_wbs() function with quadratic complexity
• Accumulation of up to millions of dirty inodes causing CPU saturation
• This issue was particularly noticeable on enterprise distributions and servers where the I/O load is constant, for example in Red Hat Enterprise Linux or SUSE Linux Enterprise environments, but also for user distributions like Ubuntu, Fedora, Debian, or Arch Linux. These ecosystems rely heavily on Systemd to automate and control critical periodic tasks.
• Discover the new features and improvements brought by the Linux 6.18 update. Performance, security, and new features: everything you need to know about this version of the Linux kernel.
• Real Impact on Administrators and Use Cases on Major Distributions

For a system administrator, a blocking situation like the one described can lead to:

Interruption or delay in the execution of other dependent Systemd units

100% CPU usage for several hours, blocking other essential processes

• Multiple manual interventions to restore stability
• For example, a backup script or a cleanup process launched via Systemd on a Fedora 42 machine, which relies heavily on reading various files, could see its performance drastically reduced by this problem, as observed in some community feedback.
• This phenomenon is also critical in server environments based on Debian or Ubuntu, which constitute a large part of the world’s open-source infrastructures. Also supported by Canonical, the issue impacted the fluidity of managed services, leading to a strong requirement for a kernel-level fix.
• https://www.youtube.com/watch?v=ktyX5mfpTFk

Christian Brauner’s proposed patches and deep optimization of the VFS file system

Systems engineer Christian Brauner, who works for Microsoft but is extremely active in the Linux community, has submitted a set of 12 patches affecting the VFS (Virtual File System) layer of the Linux 6.18 kernel. These patches specifically target the writeback mechanism involved in managing dirty inodes related to Systemd.

Purpose of the patches:

Reduce the complexity of the inode_do_switch_wbs() function by optimizing the algorithm for switching dirty inodes from child cgroups to the parent cgroup. The idea is to move from a naive approach with quadratic complexity to a more scalable approach, avoiding chronic CPU saturation.

The methodology includes:

Fine analysis of dirty inode behavior with lazytime enabled when exiting a cgroup Redesign of the b_dirty list used to sort and store write-back inodes

Performance improvements by reducing redundant iterations

• Implementation of test and demo scripts allowing developers to reproduce the problem on earlier kernel versions
• The expected results of these optimizations are spectacular, especially on distributions where Systemd controls all system services, such as Fedora, OpenSUSE, Arch Linux, or SUSE. These changes fit naturally into the Linux Foundation’s overall approach to ensuring the reliability and scalability of modern Linux environments.
• These patches, after review and validation, could be considered a turning point in the fine-grained management of system metadata on Linux 6.18, avoiding the blockages that caused numerous tickets on Fedora, Debian, and even Canonical trackers.
• Discover the new features, security improvements, and performance improvements brought by the Linux 6.18 update. Everything you need to know about this latest version of the Linux kernel.

Impacts for major distributions and service management under Systemd

Red Hat Enterprise Linux and Fedora:

benefit from improved stability in their server, cloud, and embedded environments, with less risk of CPU saturation during critical operations.

Debian and Ubuntu:

• These distributions can better support administration scripts via Systemd, improving reliability in cloud, virtual host, or user workstation infrastructures. OpenSUSE and SUSE Linux Enterprise:
• benefit from optimized I/O management during intensive read and write routines, particularly on server or workstation configurations. Arch Linux:
• Users reportedly notice increased fluidity when performing complex tasks, guaranteed by a more agile Linux kernel optimized in its VFS subsystem. These benefits affect all users who rely on Systemd, regardless of their preferred distribution. The performance gain translates into improved energy efficiency, a reduced risk of system disruption due to CPU overload, and greater overall stability. By extension, this kernel fix is ​​a reminder of the importance of collaboration between the Linux community, companies like Canonical and Microsoft (through its developers), and coordination by the Linux Foundation. It is this synergy that allows Linux to evolve to meet the current challenges of modern systems, whether embedded, server, or desktop.
• To delve deeper into these updates, enthusiasts can consult specialized articles covering recent kernel versions or practical implications for system tools, available, for example, on platforms like Linux in Caja

or

as part of Linux support for Fairphone 6

. Advanced techniques for diagnosing and managing dirty inode issues under Systemd For system administrators and Linux engineers, a deep understanding of the mechanisms behind this update is essential to anticipate and resolve similar anomalies. Here are some practical methods and recommended tools: Monitoring dirty inodes:Use tools like

iotop

or

• pidstat to identify processes that are consuming high CPU power and generating large writebacks. Examining mount options: Check for the presence of the lazytime option via mount or
• cat /proc/mounts to correlate heavy read blocks with the file system configuration. Consulting Systemd logs: Using journalctl , particularly to detect units affected by large reads followed by deadlocks.
• Experimenting in a test environment: Reproduce conditions using dedicated scripts like those provided in the Linux 6.18 patch, to validate behavior before deploying to production. Using specific performance profiles: via tools likeperf
• to precisely track bottlenecks in inode and writeback management. Mastering these techniques provides an undeniable advantage to those looking to optimize the performance of their Linux systems, particularly in contexts where Systemd is omnipresent. Many distributions, such as the recently released Fedora 42, offer better integration of diagnostic tools to facilitate this task.
• These best practices complement the Linux community’s ongoing efforts to ensure system robustness while providing backward compatibility with the vast ecosystem of applications and services. Discover all the new features and improvements brought by the Linux 6.18 update: improved performance, bug fixes, and new features for an optimized experience. Towards a more stable Linux future: implications and perspectives around the Linux 6.18 kernel and Systemd The correction made by Linux 6.18 is part of a logic of continuous improvement of performance and reliability, meeting the requirements of cloud infrastructures, professional workstations and the Internet of Things.

By reducing crashes associated with mass reading of files under Systemd, this release also sets the stage for more complex high-load scenarios common in the era of containers, Kubernetes orchestrations, and highly dynamic server solutions.

Possible development prospects:

Strengthening writeback mechanisms for asynchronous operations

Deeper integration of Systemd with core layers for better harmonization

Expanded support for recent hardware architectures, including Apple Silicon, in correlation with advances such as support for the Apple A11 in Linux 6.18

The developers’ initiatives are in synergy with leading distributions such as Canonical, Red Hat, SUSE and Arch Linux, which will fully benefit from the increased stability in their upcoming releases. In addition, this work supported by the Linux Foundation demonstrates the maturity reached by open-source projects in the management of critical systems.

• Enthusiasts, administrators and developers now have a healthier basis for deploying their Linux services. The path towards an ever more efficient and stable Linux is thus punctuated by targeted fixes and in-depth mechanical optimizations, which are the strength of this ecosystem.