This post demonstrates a method to find masquerading processes on Linux systems. This technique may work on other operating systems which have implemented procfs, such as FreeBSD.

What is Process Masquerading?

Process masquerading is a technique used by malware which makes a malicious process appear benign with the intent to trick systems administrators or incident responders.

Malware can overwrite its own argument vector, making it appear to be a httpd process or a kernel thread rather than ./awesome_malware.py.

When an administrator views the process list with ps, top, or similar tools, the malware is easily overlooked as something that is supposed to be running.

Real-World Process Masquerading Example

In the following code snippet, the argument vector is overwritten with a random member of a list of innocent-looking processes:

my @rps = ("/usr/local/apache/bin/httpd -DSSL",
           "/usr/sbin/httpd -k start -DSSL",
           "/usr/sbin/httpd",
           "/usr/sbin/sshd -i",
           "/usr/sbin/sshd",
           "/usr/sbin/sshd -D",
           "/usr/sbin/apache2 -k start",
           "/sbin/syslogd",
           "/sbin/klogd -c 1 -x -x",
           "/usr/sbin/acpid",
           "/usr/sbin/cron");
my $process = $rps[rand scalar @rps];

The original source for this malware can be viewed here: https://gist.github.com/tlongren/afe81698cafaafcbe386#file-bot-pl-L35

If this malware is running, it will show up as if it were httpd, apache, sshd, syslogd, klogd, acpid, or cron in a process list rather than something like /dev/shm/ddos-perlbot.pl.

Since these bogus process names are so common on Linux hosts, it is very easy to overlook.

Methodology

One method to detect these processes is by comparing each running process’ /proc/PID/cmdline to the Name value in /proc/PID/status. If these don’t match, it is worth investigating the process.

In my experience, GNU screen and systemd are common false positives.

This is a simple shell script which reveals these processes:

#!/bin/sh

for process in $(find /proc/[0-9]*/ -maxdepth 0 -type d); do
    pid=$(echo $process | cut -d / -f 3)

    if [ ! -e ${process}cmdline ]; then
	continue
    fi

    cmdline=$(tr '\0' '|' < ${process}cmdline)
    if [ "$cmdline" = "" ]; then
	continue
    fi

    cmd=$(echo $cmdline |cut -d '|' -f 1)
    status=$(grep -E "^Name:" ${process}status)

    echo $cmdline | grep $(echo $status |awk '{print $2}') >/dev/null
    if [ $? = 1 ]; then
	echo "PID:             $pid"
        echo "command:         $cmd"
	echo "/proc/X/cmdline: $cmdline"
	echo "/proc/X/status:  $status"
	echo
    fi
done

Example

This C program will overwrite its process name with “fake”, which will make it show up as “fake” in the output of ps. It sleeps for 5 minutes, giving us plenty of time to observe its behavior.

#include <unistd.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[]) {
	/* Overwrite argv[0] with "fake" */
	memset(argv[0], 0, strlen(argv[0]));
	strcpy(argv[0], "fake");

	sleep(300); /* Sleep for 5 minutes */

	return EXIT_SUCCESS;
}

Next, we will compile and run this program, and check what this process shows up as with ps.

daniel@wildcat ~ % gcc -o masquerade masquerade.c 
daniel@wildcat ~ % ./masquerade &
[1] 461828
daniel@wildcat ~ % ps ax |grep 461828
 461828 pts/2    SN     0:00 fake
 461911 pts/2    S+     0:00 grep --color 461828

Running the script from the previous section reveals this process. Notice the systemd false positive:

daniel@wildcat ~ % ./masq.sh 
PID:             1
command:         /sbin/init
/proc/X/cmdline: /sbin/init|splash|
/proc/X/status:  Name:	systemd

PID:             461828
command:         fake
/proc/X/cmdline: fake|||||||||
/proc/X/status:  Name:	masquerade