Bash Tips: Find and Iterate through Files in a Folder

A fast and flexible automation pattern

March 1, 2023

Another Short Introduction to Bash

The standard way of interacting with a computer is through the graphical user interface – GUI for short. Another method that predates the GUI is the command line, where the user enters text prompts which tell the computer what to do. Common operations include navigating the file system, reading and writing files, and running programs. These basic operations are easily accomplished in the GUI but the command line is still useful, especially for automation.

A common pattern in useful automations is finding and operating on a specific set of files within a folder. For example, say you have an inbox folder full of documents and you want to move each PDF to different folder, while keeping the rest of the files unchanged. Such a task quickly grows tedious to do in the GUI but remains simple when using the command line, regardless of there being 10 or 10,000 PDFs.

Step 1: Find Files

Bash has a 'find' command which takes as input 2 parameters: a directory and a pattern, then ouputs every matching file contained in the directry (also searching through subdirectories). The following snippet will output the list of PDFs that exist in the current directory.

find . -name "*.pdf";
      

Step 2: Iterate through files

Like most languages, Bash makes it easy to iterate through a list. Here’s a snippet that takes in the results of the find command and iterates through it, in this case printing out each filename.

for file in `find . -name "*.pdf"`; do
  echo $file;
done
      

Replacing the echo line, we can use the filename in other ways. Let’s move each file to a directory named pdfs, completing the example automation.

for file in `find . -name "*.pdf"`; do
  mv $file ../pdfs/;
done
      

More Examples

Delete all .DS_Store files

for file in `find . -name ".DS_Store"`; do
  rm $file;
done
      

Put all logs into a single log file

echo > logs.txt # create or empty the log file

for file in `find . -name "*log.txt"`; do
  echo $file >> logs.txt; # append the filename
  cat $file >> logs.txt;  # log contents
  echo >> logs.txt;       # new line
done
      

Create a darkened version of each png

for file in `find . -name "*.png"`; do
  # file: daisy.png -> out: darkened_daisy.png
  out=`dirname $file`"/darkened_"`basename $file`;
  convert -brightness-contrast -30x0 $file $out;
done
      

Create a jpg of each png

for file in `find . -name "*.png"`; do
  convert $file ${file/png/jpg};
done
      

Upload all json files to an S3 bucket

for file in `find . -name "*.json"`; do
  aws s3 cp $file s3://json/`basename $file`;
done