Checking Integrity of A Debian/Ubuntu System

Sometimes, a Linux filesystem becomes corrupted, system files are damaged, or some crucial files get lost. This often happens, regardless of which filesystem (ext2, ext3, ext4, jfs, reiserfs, reiser4, or xfs) is used. There are many possible reasons, such as:

• Unstable hardware, for example, memory or hard drive problem
  
• Overheat, power surge, quake or another environmental disaster
  
• Buggy software, such as a bug in the kernel or the filesystem driver
  
• Compromised security, for example, network intrusion or attack
  
• Worm or virus infection

Files in Linux systems can be categorized into the following three:

1. Verifiable System Files
   
   In Linux systems that are managed by packages (such as Debian and Ubuntu), these files are installed by packages and make up the bulk of the filesystem. These files reside in such directories as /bin, /lib, /sbin and /usr. They are usually static, which means they don't normally change except when the system is updated, or locally compiled binaries are installed.
   
2. Changeable System Files
   
   These files are auxiliary system files for system configuration, initialization or customization, and system data (such as logs and cache). They reside in /boot, /etc, /opt, /srv and /var.
   
3. User Data
   
   These files are created and used by superuser (a.k.a root) and normal users, or software-generated during casual user activities. Typically, they are in /home, /media, /mnt and /root.
   

This post focuses on verifiable system files (installed by packages). When the filesystem becomes corrupted (but not completely unreadable), it is possible to verify and restore the system integrity by using package checksums. Before you continue, make sure to fsck the filesystem.

e2fsck -r -v /dev/sda7

In this example, /dev/sda7 points to an ext2 partition we're going to check. Be aware that you cannot fsck a mounted filesystem. Therefore, boot with a Debian Live CD (or a Ubuntu CD) and run fsck. After you've performed fsck, there may be some files created in the /lost+found directory. We'll deal with them later. First, mount the filesystem.

mount -t ext2 /dev/sda7 /mnt

Go to /var/lib/dpkg/info. Then, concatenate all the md5sums files. Most, if not all, Debian and Ubuntu packages come with a md5sum file that we can use to check the integrity of the package and the files installed by the package.

cd /var/lib/dpkg/info
cat *.md5sums | sort > /dev/shm/all.md5

all.md5 has md5 checksums of all the files installed on the system. Now, check the files on the Debian/Ubuntu system against the concatenated md5sums file.

cd /
md5sum -c /dev/shm/all.md5 > /dev/shm/check.txt 2>&1

/dev/shm/check.txt now contains the results of the integrity check. It looks like this:

bin/bash: OK
bin/bunzip2: OK
bin/bzcat: FAILED

In this example, /bin/bzcat is damaged. To find all the missing or damaged files, use a command like this one:

grep -v ': OK$' /dev/shm/check.txt

Let's reinstall this file. First, find out which package this file belongs to.

dpkg -S /bin/bzcat

We'll see the following result.

bzip2: /bin/bzcat

Now we know that we need to reinstall bzip2. Let's download the package.

dpkg -p bzip2 | grep 'Filename: '

This command will let us know the name of the package to download. Use wget to download it.

wget ftp://ftp.us.debian.org/debian/pool/main/b/bzip2/bzip2_1.0.5-4_i386.deb

You can just reinstall the package.

dpkg -i bzip2_1.0.5-4_i386.deb

Or, you can just extract one file:

dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xf - ./bin/bzcat

Alternatively,

dpkg --fsys-tarfile bzip2_1.0.5-4_i386.deb | tar xOf - ./bin/bzcat > /mnt/bin/bzcat

To restore a file from the /lost+found directory, you can also use the MD5SUMS file. First, run md5sum on files in /lost+found.

cd /lost+found
md5sum *

You may get an output like this.

9aaa2176d20c1b1203e3abbac55a2513  #124531

To find out what #124531 file is originally, find its md5 checksum from the all.md5 file above.

grep 9aaa /dev/shm/all.md5

You'll get a result like this.

9aaa2176d20c1b1203e3abbac55a2513  bin/bzip2

Now you can just move it to its place.

mv \#124531 /mnt/bin/bzip2

After you restore all damaged files and restore files from /lost+found, you can find missing files in the system. Go to /var/lib/dpkg/info again and concatenate all the list files.

cd /var/lib/dpkg/info
cat *.list | sort | uniq > /dev/shm/all.txt

The .list files in the /var/lib/dpkg/info directore show the list of files installed by packages. Let's find what's missing from the system.

cd /
for f in $(cat /dev/shm/all.txt ); do test -e "$f" || echo "$f" >> /dev/shm/nonexist.txt ; done

The file /dev/shm/nonexist.txt will show which files are missing from the system. You can then replace the missing files as done previously.