Skip to content
System

Delete lines with grep without modifying the source file

Avatar illustré de Julien Macé 8 min read
Administrateur Linux filtrant des lignes de journal sur un poste de travail
Supprimer des lignes avec grep sans modifier le fichier source

Remove lines with grep on Linux without touching the original

You are preparing a list from a text export and want to remove the debug lines without modifying the received file? With grep, the -v option inverses the selection: the lines that do not match the pattern are sent to standard output. The source remains read-only. To begin, the grep search options are detailed in the general guide; here, we focus on controlled exclusion and the potential creation of a new copy.

The process is deliberately cautious: check the file, display what will remain, write to a temporary file placed near the destination, examine the diff, and then confirm the creation of a new file. Do not use this procedure to clean an active log, a system file, or elements of incident evidence.

Grep excludes lines from the output, not from the source file

Prepare a readable test file

First, test in a private directory. The following command creates a temporary folder and places an example in it. It does not write to a file in your current directory. If a temporary file already exists, mktemp -d will choose another.

atelier=$(mktemp -d) || exit 1
cd -- "$atelier" || exit 1
printf '%sn' 'INFO start' 'DEBUG cache' 'IP 192.0.2.1' 'IP 192x0x2x1' '' '   ' "$(printf 't')" '# comment' 'ERROR failure' 'DEBUG' 'DEBUG cache' 'End' > source.txt

The fixture contains an empty line, a line with three spaces and a line with a tabulation. The line DEBUG cache appears twice intentionally: the filter acts on each matching line, not just the first occurrence.

Display retained lines with -v

Before creating a copy, see what would be retained:

grep -Fv -e 'DEBUG' -- source.txt

-v selects the lines that do not match. -F treats DEBUG as literal text, and -e introduces the pattern. The separator -- marks the end of GNU options; the path that follows is therefore treated as a filename. The result contains neither DEBUG nor DEBUG cache. The other lines retain their order, including the empty line and those that contain spaces. Nothing has been removed from source.txt.

The standard output is the stream where grep writes its result, often displayed in the terminal. A filtering command does not edit the read file. This distinction avoids a serious error: never redirect the command to its own source, for example grep -Fv -e 'DEBUG' source.txt > source.txt. The shell empties the destination before grep reads its contents. A destination that points to the same file via a link can present the same risk.

Choose between exact text and regular expression

A dot in an address: start with -F

A dot is a special character in a regular expression. To exclude only the exact documentation address from the example, use a fixed string:

grep -Fv -e '192.0.2.1' -- source.txt

This command removes IP 192.0.2.1 but retains IP 192x0x2x1. Without -F, the dot in 192.0.2.1 can represent any character:

grep -v -e '192.0.2.1' -- source.txt

On the fixture, this variant also excludes the line with the letters x. For an identifier, a name, or an address to compare literally, I would choose -F: the pattern then retains its visible meaning and does not depend on regular expression rules.

Exclude an entire line with -x

To only remove the line exactly equal to DEBUG, add -x:

grep -Fxv -e 'DEBUG' -- source.txt

The line DEBUG disappears, while the two lines DEBUG cache remain. Spaces at the beginning or end also change the compared content. -x means “the whole line,” not “the word DEBUG anywhere in the line.”

Multiple patterns or an anchored regex

With multiple fixed patterns, repeat -e. A line containing either will be excluded:

grep -Fv -e 'DEBUG' -e 'ERROR' -- source.txt

If the requirement is for lines that start with a log level, an extended regular expression may be more suitable:

grep -Ev -e '^(DEBUG|ERROR)([[:space:]]|$)' -- source.txt

Both commands here give the same lines, but their rules are not interchangeable across all files. The first searches for the two strings anywhere. The second excludes a DEBUG or ERROR prefix followed by a space or the end of the line. For a pattern that starts with a dash, -e prevents it from being interpreted as an option, for example grep -Fv -e '-DEBUG' -- source.txt.

Remove empty lines without confusing spaces

Empty line: ^$

To exclude a line whose content is strictly empty, use a regular expression:

grep -v -e '^$' -- source.txt

The empty line disappears. The lines with three spaces and a tabulation remain, as they contain characters. Do not add -F to this command: with this option, ^$ would be searched as an ordinary character string, not as the start and end of a line.

Blank line: POSIX class and invisible characters

To exclude empty lines or those made up of whitespace recognized by the locale, use the POSIX class [[:space:]]:

grep -v -e '^[[:space:]]*$' -- source.txt

On the example, the empty line, the one with three spaces, and the one with a tabulation disappear. A line that contains spaces in the middle of useful text remains. The class is more portable than presenting s as a universal syntax. If the result is surprising, inspect the characters without modifying the source:

od -c source.txt

This reading allows one to spot a tabulation or a carriage return that does not show up in normal display.

Read the return code before validating the result

The return code indicates if grep selected any lines. With -v, it refers to the lines retained after inversion. For GNU grep, the code 0 means that at least one line was selected, 1 means none were selected, and normally, 2 means an error occurred. POSIX describes errors with a code greater than 1.

  • grep -Fv -e 'ABSENT' -- source.txt returns code 0 and retains the whole fixture: no line contained this pattern.
  • grep -v -e '' -- source.txt returns code 1 and writes no lines: an empty pattern matches every line.
  • grep -v -e '[' -- source.txt produces a regex error with GNU grep and normally returns code 2.
  • An absent source file is an error, not a valid empty result.

In a script, capture the status without losing its value:

rc=0
if grep -Fv -e 'DEBUG' -- source.txt; then
  rc=0
else
  rc=$?
fi
printf 'Grep code: %sn' "$rc"

Avoid if ! grep ...; then rc=$?: in this branch, $? corresponds to the logical inversion, not the original code from grep. Also, do not use -q to validate a file you are writing to: in the presence of a match, GNU grep might return 0 even if an error occurred. Keep error messages visible.

Create a filtered file only after verification

Prepare a temporary file, compare, then confirm

If you want a new output, work in a trusted personal directory with a stable source and a distinct destination. The script below is a Bash example for Linux using GNU grep and related GNU tools. It is not part of GNU grep. It rejects an empty pattern and an already present destination, writes to a temporary file in the output folder, checks the status of grep, shows a diff, and then requests explicit confirmation.

#!/usr/bin/env bash
set -u
source_file=${1:-./source.txt}
output_file=${2:-./result.txt}
pattern=${3-DEBUG}
if [ -z "$pattern" ]; then
  printf '%sn' 'Empty pattern rejected.' >&2
  exit 2
fi
if [ -e "$output_file" ] || [ -L "$output_file" ]; then
  printf '%sn' 'The destination already exists: choose another name.' >&2
  exit 2
fi
output_dir=$(dirname -- "$output_file")
tmp=$(mktemp "$output_dir/.grep-output.XXXXXX") || exit 2
trap 'rm -f -- "$tmp"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
rc=0
if grep -Fv -e "$pattern" -- "$source_file" > "$tmp"; then
  rc=0
else
  rc=$?
fi
case "$rc" in
  0) printf '%sn' 'Lines have been retained.' ;;
  1) printf '%sn' 'No lines retained: the output is empty, with no grep error.' ;;
  *) printf 'Grep error (%s): output abandoned.n' "$rc" >&2; exit "$rc" ;;
esac
diff_rc=0
diff -u -- "$source_file" "$tmp" || diff_rc=$?
if [ "$diff_rc" -gt 1 ]; then
  printf '%sn' 'Comparison impossible: output abandoned.' >&2
  exit 2
fi
printf '%sn' 'Read the diff: a difference (code 1) is normal.'
IFS= read -r -p 'Create the controlled output, even if empty? Type yes: ' confirmation || exit 1
[ "$confirmation" = yes ] || exit 1
mv -T -i -- "$tmp" "$output_file" || exit 2
if [ -e "$tmp" ]; then
  printf '%sn' 'Move refused: no output validation.' >&2
  exit 2
fi
printf 'Output created: %s; source not replaced.n' "$output_file"

In a working file named safe_filter.sh, save the script and then run:

bash safe_filter.sh ./source.txt ./result.txt DEBUG

Examine the diff before typing yes. The code 1 from diff means that the files differ, which is expected here; a code greater than 1 signals a comparison issue and causes the process to be abandoned. If no lines are retained, the script still presents the diff and requires the same confirmation before creating an empty output. On a grep error, an abort, or an impossible comparison, the temporary file is deleted, and the destination is not validated.

The temporary file is located in the output directory so that the final move remains on the same filesystem in the current case. mktemp gives it a non-predictable name and usual private permissions. The script does not copy the metadata, ACLs, or permissions from the source: the result is a new file. It reduces the risks of common overwrites but does not absolutely protect against concurrent races in a shared and malicious directory. Do not use this script while another process writes to the source.

Check the limits before using the copy

This method targets text files where a line can be treated independently of others. Binary data, structured JSON, or a CSV with line breaks in fields cannot be reliably filtered by simply deleting lines. GNU grep can report that a file contains binary data instead of producing the expected copy.

With a CRLF file, the carriage return may cause a line that appears empty not to match ^$. The class [[:space:]] can cover this character; check first with od -c and do not normalize silently. GNU grep also adds a final newline to an input that did not have one. The source remains intact, but one should not promise that the bytes of each retained line are always reproduced exactly. Classes and certain matches depend on the locale and may vary between GNU grep, BusyBox, and BSD.

For incident search, keep the original log: you can instead find an error in the logs with its context. If your need is to replace a text fragment in each line, it is no longer a line filter; see how to replace text with sed, while keeping a working copy and appropriate verification.

The operating rule remains simple: preview with grep -Fv, check the return code, compare the output, and validate a new file only after confirmation. The original remains your fallback point as long as you do not replace it.

Share Facebook X WhatsApp LinkedIn Pinterest Reddit Email