For years, I have been doing regular backups of my desktop computer, using the old-yet-still-usable software Simple Backup. This program still does a good job of backing up the files I want, and restoring them when necessary. For example, in the past I have accidentally deleted character models from my games, deleted configuration files when I didn’t want to, and completely messed up important documents. It’s always important to have a backup on hand for just such an occasion. Also, I like to see how much compression was applied to my backup files when they’re put into a tarball. To this end, I have written a tiny script for some large files, or files which require a different backup solution.
I record and edit videos on a regular basis. If you hop on over to my YouTube channel, you’d see that I have various videos of different lengths. These can’t be easily backed up because I usually can’t afford large, external hard drives. So I need to compress them down. Luckily, I know of a way to compress the files down significantly.
There is a program named pixz which fully utilizes all of the cores of a CPU. I take that program, have it compress a huge video file, and throw it on to one of my external hard drives. Usually, though, I don’t apply this to just a file; I’ll apply this to a directory. That’s why I use this script.
#!/bin/bash
# This is a script for rolling up a directory into a tarball. This uses the "pixz" program, which, in turn, utilizes many cores.
print_help ()
{
cat <<EOF
Usage: archive-directory.sh DIRECTORY FILE_NAME [-p NUMBER-OF-CORES]
When you use this script, it will use the "pv" and "pixz" programs to quickly roll the directory DIRECTORY into an "xz" tarball. Note that "pixz" utilizes all of your CPU so as to quickly and efficiently archive the directory. Also note that "pixz" uses level 8 compression, so be prepared for your computer to sound like a jet engine.
Copyright (C) 2017 Jason Anderson
EOF
}
# Check for two arguments
if [ "$#" -lt 2 ]; then
print_help
exit 6
fi
# First check to see whether the pixz program is installed.
if [ ! "$(which pixz)" ]; then
echo "Please install \"pixz\" (try using apt to install it)."
exit 1
fi
# See whether they passed anything for the first argument.
if [ -z "$1" ]; then
echo "Please pass a directory as the first argument."
exit 2
fi
# See whether they passed a directory as an argument.
if [ ! -d "$1" ]; then
echo "Please pass a directory as the argument."
exit 3
fi
# Check for an output file as the second argument
if [ -z "$2" ]; then
echo "Please pass the output file name as the second argument."
exit 4
fi
# Check for the "pv" program
if [ ! "$(which pv)" ]; then
echo "Please install the \"pv\" program (try using apt to install it)."
exit 5
fi
# Check for the number of cores they passed, if any.
if [ "$3" == "-p" ] && [ -n "$4" ] && [ "$4" -lt "$(nproc)" ]; then
number_of_cores="$4"
else
number_of_cores=$(nproc)
fi
a=$(du -sb "$1" | awk '{print $1}')
# a=$(stat --printf="%s" "$1")
tar -cf - "$1" | pv -s "$a" | pixz -8 -p "$number_of_cores" > "$2"
exit 0
With this script, I can see how long it’ll take to compress the directory, along with the video files included. The pv program is nice for seeing how long it’ll take to actually compress the directory. The advantage is that, with a good CPU and a decent amount of RAM, it doesn’t take a long time to perform the backups. The downside is that your computer will sound like a jet engine, so if you can, get a water-cooled system.
I love compressing the hell out of files which are multiple gigabytes in size due to the satisfaction of utilizing the LZMA algorithm. It’s great fun watching the computer crunch the numbers and seeing how good of a compression it gets. Best of all, I save space. What’s not to like?
