Bash Tips: Iterate through Lines in a File

Another fast and flexible automation pattern

March 10, 2023

Another Short Introduction to Bash

As a software engineer, I consider Bash to be an often overlooked tool for solving problems. Bash and the terminal can be a powerful tool for quickly completing simple tasks. For example, one pattern I frequently use is iterating through the lines of a file and performing some operation with each line.

The Basic Pattern

Here are two equivalent examples that simply print out each line in a file.

while read line; do
  echo $line
done; < file.txt
      

cat file.txt | while read line; do
  echo $line
done;
      

The while loop iterates through the lines in file.txt, assinging the value of the line to the line variable, and printing it out. In these examples, the output is trivial but shows the basic syntax for iterating through the lines in a file. To execute more significant tasks, simply extend the basic pattern.

Examples

Download a list of files from an S3 bucket.

while read file; do
  aws cp S3://bucket/$file .
done; < files.txt
      

Upload a list of files to an S3 bucket.

while read file; do
  aws cp $file S3://bucket/$file
done; < files.txt
      

Create a folder and an empty report card for each student in a list.

while read name; do
  mkdir $name;
  touch $name/report-card.txt;
done; < names.txt