An urgent fix in Linux 6.15 to address a 3x performance decline affecting Nginx and other software
The release of Linux 6.15 was marked by the implementation of a crucial fix to address a significant performance decline. Heralded as a key milestone in open source development, this update comes after the detection of an issue that significantly impacted the performance of web servers, particularly those using Nginx. Nginx, one of the most widely deployed software for managing high-load websites, saw its performance plummet by up to three times in some cases, forcing the community to react quickly.
Initial feedback indicated that this decline affected not only Nginx, but also other critical applications such as PostgreSQL, Memcached, and some high-performance computing software like OpenFOAM. The speed with which the issue was identified and fixed is a testament to the vitality of the Linux community and the importance placed on the continuous optimization of kernel components. Understanding its causes requires analyzing the development and testing process that identified this bug, as well as the impact on the web server ecosystem in 2025.
The Origin of the Performance Decline in Linux 6.15 and Its Immediate Effects
The issue emerged during intensive testing phases prior to the official release of Linux 6.15. Performance, which had been steadily improving until then, suddenly dropped in several scenarios. The root cause was identified as a change in memory management, specifically in the operation of the kernel’s memory allocator, which was altered during the merge of a patch intended to improve security. This change, while initially promising, caused an overload in handling HTTPS requests on Nginx.
The effects were quickly observed in benchmarks: the ability to handle a high volume of simultaneous HTTPS connections plummeted. Some performance tests, previously completed in a few milliseconds, now required several times longer, which significantly deteriorated the quality of service. The following figure illustrates the degradation, showing a drop of more than 70% in throughput in some real-life cases. Application
| Performance before | Performance after Linux 6.15 | Percentage of Degradation | Nginx HTTPS (bench 1) |
|---|---|---|---|
| 1000 requests/sec | 350 requests/sec | -65% | PostgreSQL |
| 2000 operations/sec | 1600 operations/sec | -20% | Memcached |
| 50,000 TPS | 25,000 TPS | -50% | OpenFOAM |
| 2200 simulations | 1400 simulations | -36% | This table summarizes the direct impact of the regression on several key software programs for modern servers. The performance drop, also critical for Nginx, resulted in lost productivity and a significant increase in processing times, forcing companies and hosting providers to act quickly. |
The Linux community responded in an exemplary manner to this performance crisis. Upon detection, a group of specialized engineers mobilized to analyze the responsible commits. Discussion, particularly on the official kernel mailing list (LKML), was swift and focused on code splitting. Collaboration between developers, including those from SUSE, Red Hat, and other major stakeholders, made it possible to contain the problem in less than 24 hours.
The work focused primarily on the patch signed by Vlastimil Babka, known for his expertise in memory management and kernel development. This patch aimed to correct the overload issue caused by the new memory management by restoring balance to the allocator. The testing phase confirmed that the regression had been contained, leading Linus Torvalds himself to quickly integrate the fix into the main branch. Initial analysis on the mailing listIdentification of the problematic commitDevelopment of the corrective patch
Validation via benchmarks
- Release to production in the Linux 6.15 stable branch
- Examples of impacted workloads and their post-patch restoration
- TCP bandwidth testing with Ethr on an AMD EPYC server
- Running CFD simulations with OpenFOAM
- PostgreSQL database service
Memcached distributed cache
- Each step was documented to ensure responsiveness and transparency, illustrating how a critical performance issue can be resolved in record time thanks to the commitment of the open source community.
- Impacts of the patch on overall performance and software development
- Impacts of the patch on overall performance and software development
- The success of this intervention demonstrates the resilience of the Linux ecosystem in the face of software optimization challenges. The fix implemented in version 6.15 restored lost stability while providing insights to avoid similar regressions in the future.
The proactive management of this crisis highlighted several areas for improvement:
Strengthening automated performance testing in the development cycle
Integrating more sophisticated benchmarks during review phases
Adopting rapid rollback mechanisms in the event of a critical regression
- By analyzing the specific cases related to Linux 6.15, it becomes clear that collaboration and agility in software development are essential to maintaining the competitiveness and security of the open source ecosystem. The rapid correction of this regression proves that a mature open source project can effectively address complex issues, even at a time when performance is crucial to the success of modern web services.
- https://www.youtube.com/watch?v=Bt8zZibFLgU