Finding Bad With Package Managers

Most Linux distributions include a package manager. Package managers are a huge step up from the past. You used to have to manually download software’s source code, build it, and deal with library and third-party tool dependencies on your own.

Most mainstream distributions use RPM or dpkg-based package managers, but there are also a handful of other package formats.

One often-overlooked feature that most package managers include is a summary of the checksums for each of the files included in a particular package. This can be used to verify the integrity of your systems.

These scenarios are suspicious and should be investigated:

  • If a file in a directory typically in $PATH (/bin, /sbin, /usr/bin, /usr/sbin, …) has a different hash than what was provided by its package.
  • If a library’s hash differs from what was provided by a package.
  • If configuration files differ from what was provided by the package and the changes can’t be accounted for.
  • If a binary or library isn’t provided by a package. Example: /bin/fsck.reiserfs exists but wasn’t put there by a package.

dpkg(deb)-Based Systems

Debian, Ubuntu, Mint, Kali, and several others use dpkg to manage packages.

Verifying a package

Scenario: you suspect a package has been tampered with. For demonstration purposes, let’s say that you believe an attacker has modified /bin/netstat to hide network connections.

You can test your suspicions if you know the package name that provides /bin/netstat. To look this value up, use the -S option. In this case, the package which provides /bin/netstat is net-tools. Verify net-tools with the -V flag:

% dpkg -S /bin/netstat
net-tools: /bin/netstat
% dpkg -V net-tools

There was no output, indicating that netstat was not tampered with. Here, we will demonstrate that this actually works by tampering with netstat and trying again:

% cp /bin/netstat .
% sudo -s
# echo garbage >>/bin/netstat
# dpkg -V net-tools
??5??????   /bin/netstat
# cp ./netstat /bin/netstat
# chown root.root /bin/netstat
# ls -l /bin/netstat
-rwxr-xr-x 1 root root 119624 Feb 24 16:44 /bin/netstat*
# dpkg -V net-tools

After netstat was modified, dpkg -V showed that its checksum differed (the ??5?????? column. See ). Replacing netstat with its original copy yielded no output. An explaination of the fields in this column is found in dpkg’s man page:

       --verify-format format-name
              Sets the output format for the --verify command
              (since dpkg 1.17.2).

              The  only currently supported output format is rpm, which
              consists of a line for every path that failed any check.  The
              lines start with 9 characters to report each specific check
              result, a ‘?’  implies  the check  could not be done (lack of
              support, file permissions, etc), ‘.’ implies the check passed,
              and an alphanumeric character implies a specific check failed;
              the md5sum verification failure (the file  contents  have
              changed) is denoted with a ‘5’ on the third character.  The
              line is followed by a space and an attribute character
              (currently ‘c’ for conffiles), another space and the pathname.

All packages can be verified with a 1-liner:

# for pkg in $(dpkg -l | grep ^ii | awk {'print $2'}); do dpkg -V $pkg; done

debsums

debsums is a tool that you can install which allows you to validate all packages with one easy command. To install it:

# apt install debsums

Example usage:

# debsums -c
/usr/x86_64-linux-gnu/security/pam_unix.so
/bin/ps
/bin/netstat

You should read the man page for debsums for more examples.

Files not provided by a package

A decent Indicator of Compromise (IoC) is when files in directories such as /bin, /sbin, /lib, … contain files NOT provided by a package. This occurs when someone places files in these directories manually.

# find /bin/ -exec dpkg -S {} \; 2>&1 |grep "no path found matching"
dpkg-query: no path found matching pattern /bin/backdoor

Where the checksums are stored

Checksums for these files are located in /var/lib/dpkg/info/*.md5sums

The format is self-explanatory if you look at any of these files:

# cat /var/lib/dpkg/info/at.md5sums
507610d547edc3e7b98777cd1d5ad084  lib/systemd/system/atd.service
2c0733b064a62cb0c4cbbbd531465120  usr/bin/at
03265a0c11d5b695529f629b1e860eb1  usr/bin/batch
50fd51201dda267285da1d0eadd50736  usr/sbin/atd
38c61cea8340b1d2191535464235fd6a  usr/share/doc/at/Problems
fb98b5905a0ec916dd6277268f86dd10  usr/share/doc/at/README
...snip...

This is useful for writing scripts that go above and beyond what is offered by dpkg and debsums. Be advised that attackers can modify these files. For example, in the last section, /bin/backdoor wasn’t provided by a package. An entry can be added to one of these .md5sums files and its accompanying .lists file to make it appear like a legitimate element of a package:

# md5sum /bin/backdoor
d41d8cd98f00b204e9800998ecf8427e  /bin/backdoor
# vi /var/lib/dpkg/info/at.list
# vi /var/lib/dpkg/info/at.md5sums
# dpkg -S /bin/backdoor
at: /bin/backdoor
# dpkg -V at
# grep backdoor /var/lib/dpkg/info/at.*
/var/lib/dpkg/info/at.list:/bin/backdoor
/var/lib/dpkg/info/at.md5sums:d41d8cd98f00b204e9800998ecf8427e  bin/backdoor

The moral of this story is to take outputs of tools with a grain of salt. Bring your own verified good tools and hash sets to forensic/incident response engagements.

If you suspect the md5sums files on disk were tampered with

As noted in the previous section, an attacker can add bogus entries to dpkg’s database, masking their activity. You should curate your own lists of hashes for systems that you maintain. If possible, you should keep installed packages handy on read-only media to serve as a source of truth. You may have to search the internet for package mirror sites that provide the specific version of the package you are investigating.

Be prepared!

RPM-Based Systems

RHEL, CentOS, SuSE, and others are RPM-based. These techniques should work on these distributions.

Verifying a package

Scenario: you think that someone has tampered with /bin/date. You can figure out which package provides date and verify it:

$ rpm -q --whatprovides /bin/date
coreutils-8.22-23.el7.x86_64
$ rpm -V coreutils

There was no output because the hashes in the RPM database match what is on disk. Let’s mess up /bin/date to trigger this check:

# cp /bin/date .
# echo garbage >>/bin/date
# rpm -V coreutils
S.5....T.    /usr/bin/date
# cp ./date /bin/date
cp: overwrite ‘/bin/date’? y
# rpm -V coreutils
.......T.    /usr/bin/date

dpkg mimics rpm’s output format for verification. In the first run of rpm -V, it indicates that the Size, MD5, and Timestamp are different. After replacing the date with its original copy, only the timestamp differs.

Verifying all packages is simple:

$ rpm -Va

Files not provided by a package

Files not provided by a package in directories such as /bin, /sbin, etc are suspicious. This one-liner will search for these files:

$ find /bin/ -type f -exec rpm -q --whatprovides {} \; |grep "is not owned by any package"

Where the checksums are stored

/var/lib/rpm/ contains a set of Berkeley DB files that are full of package metadata. I was unsuccessful in trying to hex edit these files to tamper with file hashes, but I’m sure its possible.

Extracting RPMs without installing them

You may wish to examine the contents of an RPM without installing it. To do this, you need the rpm2cpio utility:

$ mkdir scratch
$ cp whatever.rpm scratch
$ cd scratch
$ rpm2cpio whatever.rpm | cpio -idmv

Honorable Mentions

Here are some other package types that aren’t as common as dpkg or rpms and some brief observations.

Slackware

Slackware packages are gzipped tarballs. From what I can tell, they do not include checksum information.

Arch

Arch packages are xzipped tar files. These .txz files contain a gzipped .MTREE file which includes sha256 hashes of files provided by each package. paccheck appears to be a tool able to verify these packages.

FreeBSD pkg

FreeBSD packages are xzipped tar files. The +MANIFEST file included within each package contains a JSON object containing sha256 hashes of each file included in the package. It appears that you can verify packages with pkg check -s|–checksums.

FreeBSD includes the mtree utility, which creates a database of hashes. These databases can be copied to another system or read-only media and compared to what is actually on disk at a later date.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s