Observe a file with inotifywait to confirm the writing of a service

A service appears to be running, but the expected file remains empty. Before changing its configuration or digging through hundreds of lines in journalctl, check directly the directory where it should write.

inotifywait displays the creations, modifications, and movements seen by the kernel. You can confirm a limited write, then know when this tool is no longer sufficient.

Identify the path before starting the monitoring

Start with the absolute path. A systemd service might write to /var/log/service-name, in an application directory, or only in the system journal. On Debian and Ubuntu, first check the unit and its logs:

sudo systemctl status my-service --no-pager
sudo journalctl -u my-service -n 50 --no-pager

If you already have a specific file, check its owner, permissions, and available space. Monitoring does not fix a missing directory or a full disk.

sudo ls -ld /var/log/my-service
sudo df -h /var/log/my-service

Then install inotify-tools if the command is missing. The package goes by this name on Debian and Ubuntu.

sudo apt update
sudo apt install inotify-tools

On a RHEL, Fedora, or Rocky machine, the package is also generally called inotify-tools. Check its name in the configured repository before installation.

Track writes in a limited directory

Open a dedicated session and monitor a single directory. This command stays in the foreground until Ctrl+C.

sudo inotifywait -m -e create,modify,move,delete /var/log/my-service

The -m option keeps the monitoring open. The requested events avoid displaying simple file accesses. Then trigger the action that should cause the service to write, for example, a test request or a pre-scheduled restart.

A result such as CREATE application.log or MODIFY application.log confirms that the kernel has seen the operation in this folder. It does not yet indicate if the content is correct. Read the last lines of the file:

sudo tail -n 30 /var/log/my-service/application.log

The guide using tail to follow a log serves precisely to re-read the content after the event. If the service only writes to systemd, prefer journalctl -u my-service -f.

Test without touching the service

Before observing a production application, take a lab folder. This way, you can verify the syntax and permissions without mixing your events with those of an active service.

mkdir -p /tmp/inotify-test
inotifywait -m -e create,modify,move,delete /tmp/inotify-test

In a second session, create and then modify a file:

printf 'first linen' > /tmp/inotify-test/probe.log
printf 'second linen' >> /tmp/inotify-test/probe.log

The first session should display at least one creation and one modification. Stop it with Ctrl+C, then delete the test folder if you no longer need it.

I would choose this check before a configuration change, as it answers a simple question: does the service really reach the expected directory? It does not allow concluding on the quality of the written content.

The limits of inotifywait in production

inotify observes local events transmitted by the kernel. On a very active directory, the output quickly becomes difficult to exploit, and the event queue can overflow. You then need to return to application logs, metrics, or a suitable audit.

Do not use it as the sole proof for a security investigation. Also avoid directories containing secrets or thousands of temporary files. To identify a process holding a file or a port, lsof answers a different question.

When the event and the file content correspond to the test action, you have a clean basis to continue the diagnosis. If no event appears, recheck the path, the systemd unit, and the journal type before changing the service.

For the exact behavior of options, consult the inotifywait manual page.