Most Linux distributions include a package manager. Package managers are a huge step up from the past. You used to have to manually download software’s source code, build it, and deal with library and third-party tool dependencies on your own.
Most mainstream distributions use RPM or dpkg-based package managers, but there are also a handful of other package formats.
One often-overlooked feature that most package managers include is a summary of the checksums for each of the files included in a particular package. This can be used to verify the integrity of your systems.
These scenarios are suspicious and should be investigated:
- If a file in a directory typically in $PATH (/bin, /sbin, /usr/bin, /usr/sbin, …) has a different hash than what was provided by its package.
- If a library’s hash differs from what was provided by a package.
- If configuration files differ from what was provided by the package and the changes can’t be accounted for.
- If a binary or library isn’t provided by a package. Example: /bin/fsck.reiserfs exists but wasn’t put there by a package.
dpkg(deb)-Based Systems
Debian, Ubuntu, Mint, Kali, and several others use dpkg to manage packages.
Verifying a package
Scenario: you suspect a package has been tampered with. For demonstration purposes, let’s say that you believe an attacker has modified /bin/netstat to hide network connections.
You can test your suspicions if you know the package name that provides /bin/netstat. To look this value up, use the -S option. In this case, the package which provides /bin/netstat is net-tools. Verify net-tools with the -V flag:
% dpkg -S /bin/netstat net-tools: /bin/netstat % dpkg -V net-tools
There was no output, indicating that netstat was not tampered with. Here, we will demonstrate that this actually works by tampering with netstat and trying again:
% cp /bin/netstat . % sudo -s # echo garbage >>/bin/netstat # dpkg -V net-tools ??5?????? /bin/netstat # cp ./netstat /bin/netstat # chown root.root /bin/netstat # ls -l /bin/netstat -rwxr-xr-x 1 root root 119624 Feb 24 16:44 /bin/netstat* # dpkg -V net-tools
After netstat was modified, dpkg -V showed that its checksum differed (the ??5?????? column. See ). Replacing netstat with its original copy yielded no output. An explaination of the fields in this column is found in dpkg’s man page:
--verify-format format-name
Sets the output format for the --verify command
(since dpkg 1.17.2).
The only currently supported output format is rpm, which
consists of a line for every path that failed any check. The
lines start with 9 characters to report each specific check
result, a ‘?’ implies the check could not be done (lack of
support, file permissions, etc), ‘.’ implies the check passed,
and an alphanumeric character implies a specific check failed;
the md5sum verification failure (the file contents have
changed) is denoted with a ‘5’ on the third character. The
line is followed by a space and an attribute character
(currently ‘c’ for conffiles), another space and the pathname.
All packages can be verified with a 1-liner:
# for pkg in $(dpkg -l | grep ^ii | awk {'print $2'}); do dpkg -V $pkg; done
debsums
debsums is a tool that you can install which allows you to validate all packages with one easy command. To install it:
# apt install debsums
Example usage:
# debsums -c /usr/x86_64-linux-gnu/security/pam_unix.so /bin/ps /bin/netstat
You should read the man page for debsums for more examples.
Files not provided by a package
A decent Indicator of Compromise (IoC) is when files in directories such as /bin, /sbin, /lib, … contain files NOT provided by a package. This occurs when someone places files in these directories manually.
# find /bin/ -exec dpkg -S {} \; 2>&1 |grep "no path found matching"
dpkg-query: no path found matching pattern /bin/backdoor
Where the checksums are stored
Checksums for these files are located in /var/lib/dpkg/info/*.md5sums
The format is self-explanatory if you look at any of these files:
# cat /var/lib/dpkg/info/at.md5sums 507610d547edc3e7b98777cd1d5ad084 lib/systemd/system/atd.service 2c0733b064a62cb0c4cbbbd531465120 usr/bin/at 03265a0c11d5b695529f629b1e860eb1 usr/bin/batch 50fd51201dda267285da1d0eadd50736 usr/sbin/atd 38c61cea8340b1d2191535464235fd6a usr/share/doc/at/Problems fb98b5905a0ec916dd6277268f86dd10 usr/share/doc/at/README ...snip...
This is useful for writing scripts that go above and beyond what is offered by dpkg and debsums. Be advised that attackers can modify these files. For example, in the last section, /bin/backdoor wasn’t provided by a package. An entry can be added to one of these .md5sums files and its accompanying .lists file to make it appear like a legitimate element of a package:
# md5sum /bin/backdoor d41d8cd98f00b204e9800998ecf8427e /bin/backdoor # vi /var/lib/dpkg/info/at.list # vi /var/lib/dpkg/info/at.md5sums # dpkg -S /bin/backdoor at: /bin/backdoor # dpkg -V at # grep backdoor /var/lib/dpkg/info/at.* /var/lib/dpkg/info/at.list:/bin/backdoor /var/lib/dpkg/info/at.md5sums:d41d8cd98f00b204e9800998ecf8427e bin/backdoor
The moral of this story is to take outputs of tools with a grain of salt. Bring your own verified good tools and hash sets to forensic/incident response engagements.
If you suspect the md5sums files on disk were tampered with
As noted in the previous section, an attacker can add bogus entries to dpkg’s database, masking their activity. You should curate your own lists of hashes for systems that you maintain. If possible, you should keep installed packages handy on read-only media to serve as a source of truth. You may have to search the internet for package mirror sites that provide the specific version of the package you are investigating.
Be prepared!
RPM-Based Systems
RHEL, CentOS, SuSE, and others are RPM-based. These techniques should work on these distributions.
Verifying a package
Scenario: you think that someone has tampered with /bin/date. You can figure out which package provides date and verify it:
$ rpm -q --whatprovides /bin/date coreutils-8.22-23.el7.x86_64 $ rpm -V coreutils
There was no output because the hashes in the RPM database match what is on disk. Let’s mess up /bin/date to trigger this check:
# cp /bin/date . # echo garbage >>/bin/date # rpm -V coreutils S.5....T. /usr/bin/date # cp ./date /bin/date cp: overwrite ‘/bin/date’? y # rpm -V coreutils .......T. /usr/bin/date
dpkg mimics rpm’s output format for verification. In the first run of rpm -V, it indicates that the Size, MD5, and Timestamp are different. After replacing the date with its original copy, only the timestamp differs.
Verifying all packages is simple:
$ rpm -Va
Files not provided by a package
Files not provided by a package in directories such as /bin, /sbin, etc are suspicious. This one-liner will search for these files:
$ find /bin/ -type f -exec rpm -q --whatprovides {} \; |grep "is not owned by any package"
Where the checksums are stored
/var/lib/rpm/ contains a set of Berkeley DB files that are full of package metadata. I was unsuccessful in trying to hex edit these files to tamper with file hashes, but I’m sure its possible.
Extracting RPMs without installing them
You may wish to examine the contents of an RPM without installing it. To do this, you need the rpm2cpio utility:
$ mkdir scratch $ cp whatever.rpm scratch $ cd scratch $ rpm2cpio whatever.rpm | cpio -idmv
Honorable Mentions
Here are some other package types that aren’t as common as dpkg or rpms and some brief observations.
Slackware
Slackware packages are gzipped tarballs. From what I can tell, they do not include checksum information.
Arch
Arch packages are xzipped tar files. These .txz files contain a gzipped .MTREE file which includes sha256 hashes of files provided by each package. paccheck appears to be a tool able to verify these packages.
FreeBSD pkg
FreeBSD packages are xzipped tar files. The +MANIFEST file included within each package contains a JSON object containing sha256 hashes of each file included in the package. It appears that you can verify packages with pkg check -s|–checksums.
FreeBSD includes the mtree utility, which creates a database of hashes. These databases can be copied to another system or read-only media and compared to what is actually on disk at a later date.
DMFR SECURITY 