Skip to main content

Merging text files and adding separator [Resolved]

I want to add a separator like this "==============" and a blank new line

I tried to do this, but failed and causes high CPU usage. i mean the cpu become rotate ery fast and noisy when i run the script

This needs to be for around 100000 text files.

this is the code that I use

#!/bin/bash
for F in *.txt ; do
    type "$F"
    echo .
    echo ========
    echo . 
done >> Combined.txt;

please advice


Question Credit: Shervan
Question Reference
Asked October 10, 2018
Posted Under: Unix Linux
32 views
4 Answers

I would simplify your commands as follows:

  #!/bin/bash
  for file in *.txt; do
  cat $file >> Combined.txt
  printf '\n\n=========\n\n' >> Combined.txt
  done

credit: Goro
Answered October 10, 2018

Using FNR and NR in awk

#!/bin/bash

outfile="$( mktemp combined.txt.XXXXXX )"

echo "Output file: ${outfile}"

awk 'FNR==1 && NR>1 { printf("\n%s\n\n","========") } 1' *.txt > "${outfile}"

echo "Finished."

A line-by-line description:

outfile="$( mktemp combined.txt.XXXXXX )"

Use mktemp to create an empty new file with a unique name (eg, combined.txt.HDpgMn). You can use more X characters for a longer random suffix. Enclose the command in "$(...)" to store the new file's name in the variable outfile.

echo "Saving to file: ${outfile}"

Print the name of the output file. (When the script has finished, you may wish to rename the output file to remove the string of random characters following the .txt.)

awk 'FNR==1 && NR>1 { printf("\n%s\n\n","========") } 1' *.txt > "${outfile}"

Print...

  • a blank line,
  • a short line of "=" characters,
  • and another blank line

...at the start of each input file, except for the first input file. FNR counts the input file line numbers, resetting at the start of each file. NR counts the line numbers and does not reset.

In the awk statement, the 1 just before the closing single quotation mark evaluates to TRUE for every line, and performs the default action of printing that line. (In other words, awk '1' works like cat.)

echo "Finished."

Inform the user when the script is done. (Not strictly necessary, since you'll see the command prompt anyway, but it doesn't hurt.)


credit: Gaultheria
Answered October 10, 2018

Why not simply

printf "\n\n=====\n\n" > XTMP
cat $(printf "%s XTMP " *.txt) > combined.tmp

Put the separator into a temp file, and make use of printf's feature to repeat the formatting string for every argument it finds, so the cat command will look like

cat 1.txt XTMP 2.txt XTMP ... n.txt XTMP

You may run into system limits (e.g. LINE_MAX), though...


credit: RudiC
Answered October 10, 2018
Your Answer
D:\Adnan\Candoerz\CandoProject\vQA