I Love Doing Backups

For years, I have been doing regular backups of my desktop computer, using the old-yet-still-usable software Simple Backup. This program still does a good job of backing up the files I want, and restoring them when necessary. For example, in the past I have accidentally deleted character models from my games, deleted configuration files when I didn’t want to, and completely messed up important documents. It’s always important to have a backup on hand for just such an occasion. Also, I like to see how much compression was applied to my backup files when they’re put into a tarball. To this end, I have written a tiny script for some large files, or files which require a different backup solution.

I record and edit videos on a regular basis. If you hop on over to my YouTube channel, you’d see that I have various videos of different lengths. These can’t be easily backed up because I usually can’t afford large, external hard drives. So I need to compress them down. Luckily, I know of a way to compress the files down significantly.

There is a program named pixz which fully utilizes all of the cores of a CPU. I take that program, have it compress a huge video file, and throw it on to one of my external hard drives. Usually, though, I don’t apply this to just a file; I’ll apply this to a directory. That’s why I use this script.


# This is a script for rolling up a directory into a tarball. This uses the "pixz" program, which, in turn, utilizes many cores.

print_help ()
cat <<EOF

  Usage: archive-directory.sh DIRECTORY FILE_NAME [-p NUMBER-OF-CORES]

  When you use this script, it will use the "pv" and "pixz" programs to quickly roll the directory DIRECTORY into an "xz" tarball.  Note that "pixz" utilizes all of your CPU so as to quickly and efficiently archive the directory.  Also note that "pixz" uses level 8 compression, so be prepared for your computer to sound like a jet engine.

  Copyright (C) 2017 Jason Anderson


# Check for two arguments
if [ "$#" -lt 2 ]; then
  exit 6

# First check to see whether the pixz program is installed.
if [ ! "$(which pixz)" ]; then
  echo "Please install \"pixz\" (try using apt to install it)."
  exit 1

# See whether they passed anything for the first argument.
if [ -z "$1" ]; then
  echo "Please pass a directory as the first argument."
  exit 2

# See whether they passed a directory as an argument.
if [ ! -d "$1" ]; then
  echo "Please pass a directory as the argument."
  exit 3

# Check for an output file as the second argument
if [ -z "$2" ]; then
  echo "Please pass the output file name as the second argument."
  exit 4

# Check for the "pv" program
if [ ! "$(which pv)" ]; then
  echo "Please install the \"pv\" program (try using apt to install it)."
  exit 5

# Check for the number of cores they passed, if any.
if [ "$3" == "-p" ] && [ -n "$4" ] && [ "$4" -lt "$(nproc)"  ]; then

a=$(du -sb "$1" | awk '{print $1}')
# a=$(stat --printf="%s" "$1")
tar -cf - "$1" | pv -s "$a" | pixz -8 -p "$number_of_cores" > "$2"

exit 0

With this script, I can see how long it’ll take to compress the directory, along with the video files included. The pv program is nice for seeing how long it’ll take to actually compress the directory. The advantage is that, with a good CPU and a decent amount of RAM, it doesn’t take a long time to perform the backups. The downside is that your computer will sound like a jet engine, so if you can, get a water-cooled system.

I love compressing the hell out of files which are multiple gigabytes in size due to the satisfaction of utilizing the LZMA algorithm. It’s great fun watching the computer crunch the numbers and seeing how good of a compression it gets. Best of all, I save space. What’s not to like?

Jason Anderson

Jason Anderson has been hacking up computers for nearly 20 years and has been using Linux for over 15 years. Among that, he has a BBA in Accounting. Look him up on Twitter at @FakeJasonA and on Mastodon on @ertain@mast.linuxgamecast.com

Leave a Reply

Your email address will not be published. Required fields are marked *