Some notes on the use of "sed" command for text processing automation

2018/1/7 2:30 AM posted in  Hacker's comments

sed, short for "stream editor", allows you to filter and transform text. A stream editor is used to perform basic text transformations on an input stream (a file, or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed's ability to filter text in a pipeline which particularly distinguishes it from other types of editors.1

"In-place" mode

Options Description
-n,--quiet, --silent Suppress automatic printing of pattern space.
-e script, --expression=script Add the script script to the commands to be executed.
-i[SUFFIX], --in-place[=SUFFIX] Edit files in place (this makes a backup with file extension SUFFIX, if SUFFIX is supplied).

The "-i" option is used to make modifications to files immediately. You can also make backups if you offer "SUFFIX" specifications as well. The commonly used "-e" option doesn't modify the files.

Use inside bash scripts

Using sed inside some bash script is an elegant way to exploit its true power. But here are some pitfalls you might come across along the way. If you like to echo the commands you want to excute before the actual excution, you might get the error " :sed: -e expression #1, char 1: unknown command: ' " ' ".
Below is a typical case:

inFile="$WORK/${SimName}.in"
replaceCmd="sed 's/sim.dat/${SimName}.dat/g' $inFile"
echo $replaceCmd
$replaceCmd

It seems that the single quotes inside the string "replaceCmd" weren't recongized by the sed command.

One way to get across this problem is to use eval command.
Instead of using

$replaceCmd

, you can use

eval "$replaceCmd"

This will make the sed command work properly.
But using eval is in general not recommended due to security reasons.2

The recommended method is to wrap the sed command inside a function:

replaceCmd(){
    sed "s/sim.dat/$1.dat/g" "$2"
}
inFile="$work/$SimName.in"
replaceCmd "$SimName" "$inFile"

This method is found in LinuxQustions.org forum.3

Oh, hells no! Never, and I mean NEVER, use eval for something as trivial as this. Indeed, you should only very rarely need it at all. It can be quite a security risk, and is almost never really necessary in general scripting.

In addition, variables are for storing data, not code. The spaces, quotes, and other special characters cease to be special when in a variable (unless the expansion is unquoted, then some of them still have an effect, but likely not the one you want; see my next point), and in any case leads to unclear, obfuscated code flow. If you need repeated use of the same command, don't put it inside a variable, use a function.

A second problem here is that if the sed command is surrounded by single quotes, the variable inside them does not get expanded. No quotes is even worse, as the shell attempts to perform word-splitting and glob expansion after the substitution.