Using procfs For Forensics and Incident Response

procfs is useful when responding to incidents. It is usually mounted at /proc on Linux, FreeBSD, and Solaris. This post details some procfs files I have found useful in the context of cybersecurity. The procfs manual page can be found here.

procfs is a virtual file system which provides access to kernel structures. procfs may be accessed with common shell utilities such as cat or grep, or basic file system I/O facilities in any programming language. procfs provides information about processes, networking, devices, processors, memory, and other system statistics.

This is not an exhaustive list of interesting things provided by procfs, and may be updated in the future.

Basic procfs Usage

Here is a real-world example from the procps-ng project which demonstrates using procfs to get a system’s uptime.

First, let’s see what is in /proc/uptime. RedHat provides fantastic documentation for procfs which explains that these two numbers are the uptime in seconds, and idle time in seconds, respectively:

 % cat /proc/uptime
138346.07 93687.59

This C code reads the contents of /proc/uptime into a buffer, then uses sscanf to extract the uptime and idle time from this buffer.

#define UPTIME_FILE  "/proc/uptime"
...snip...
int uptime(double *restrict uptime_secs, double *restrict idle_secs) {
    double up=0, idle=0;
    char *savelocale;

    FILE_TO_BUF(UPTIME_FILE,uptime_fd);
    savelocale = strdup(setlocale(LC_NUMERIC, NULL));
    setlocale(LC_NUMERIC,"C");
    if (sscanf(buf, "%lf %lf", &up, &idle) < 2) {
        setlocale(LC_NUMERIC,savelocale);
        free(savelocale);
        fputs("bad data in " UPTIME_FILE "\n", stderr);
            return 0;
    }
    setlocale(LC_NUMERIC,savelocale);
    free(savelocale);
    SET_IF_DESIRED(uptime_secs, up);
    SET_IF_DESIRED(idle_secs, idle);
    return up;  /* assume never be zero seconds in practice */
}

/proc/PID/* and /proc/self/*

Each process will have its own directory as its PID in /proc.

 % ps ax --sort pid |head                             
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:19 /sbin/init splash
      2 ?        S      0:00 [kthreadd]
      3 ?        I<     0:00 [rcu_gp]
      4 ?        I<     0:00 [rcu_par_gp]
      6 ?        I<     0:00 [kworker/0:0H-kblockd]
      9 ?        I<     0:00 [mm_percpu_wq]
     10 ?        S      0:00 [ksoftirqd/0]
     11 ?        I      0:08 [rcu_sched]
     12 ?        S      0:00 [migration/0]
 % find /proc/[0-9]* -maxdepth 0 -type d |sort -V|head
/proc/1
/proc/2
/proc/3
/proc/4
/proc/6
/proc/9
/proc/10
/proc/11
/proc/12
/proc/13

/proc/self is a symbolic link to the current process. In this example, ls‘s /proc/self/exe is a symbolic link to /usr/bin/ls.

 % ls -l /proc/self/exe
lrwxrwxrwx 1 daniel daniel 0 Jan 29 16:53 /proc/self/exe -> /usr/bin/ls*

Each PID’s directory contains several files which may be forensically useful.

/proc/PID/comm

From the manual page:

              This file exposes the process's comm value—that is, the
              command name associated with the process.  Different
              threads in the same process may have different comm
              values, accessible via /proc/[pid]/task/[tid]/comm.  A
              thread may modify its comm value, or that of any of other
              thread in the same thread group (see the discussion of
              CLONE_THREAD in clone(2)), by writing to the file
              /proc/self/task/[tid]/comm.  Strings longer than
              TASK_COMM_LEN (16) characters (including the terminating
              null byte) are silently truncated.

              This file provides a superset of the prctl(2) PR_SET_NAME
              and PR_GET_NAME operations, and is employed by
              pthread_setname_np(3) when used to rename threads other
              than the caller.  The value in this file is used for the
              %e specifier in /proc/sys/kernel/core_pattern; see
              core(5).

/proc/PID/cmdline

From the manual page:

              This read-only file holds the complete command line for
              the process, unless the process is a zombie.  In the
              latter case, there is nothing in this file: that is, a
              read on this file will return 0 characters.  The command-
              line arguments appear in this file as a set of strings
              separated by null bytes ('\0'), with a further null byte
              after the last string.

              If, after an execve(2), the process modifies its argv
              strings, those changes will show up here.  This is not the
              same thing as modifying the argv array.

              Furthermore, a process may change the memory location that
              this file refers via prctl(2) operations such as
              PR_SET_MM_ARG_START.

              Think of this file as the command line that the process
              wants you to see.

/proc/PID/cwd

This is the current working directory of a process. Processes with current working directories of /tmp, /var/tmp, or /dev/shm may be worth exploring. I have used cwd to find “hidden” directories malware is running from.

/proc/PID/environ

This is the programs environment. You may be able to find secrets which are passed via the environment here.

OpenSSH sets forensically useful variables SSH_CLIENT, SSH_CONNECTION, and SSH_TTY which can be used to see where a command was initiated from.

/proc/PID/exe

This is a symbolic link to the process. you can copy these if the file has been deleted.

daniel@wildcat ~/code/delete % cat delete.c
#include <unistd.h>

int main()
{
    sleep(3000);
    return 0;
}

daniel@wildcat ~/code/delete % gcc -o delete delete.c
daniel@wildcat ~/code/delete % sha256sum delete
06782f63b8256fb1a86391cf0abde926f2ce9746a7b0dcd225fcad966f0bd712  delete
daniel@wildcat ~/code/delete % ./delete &
[7] 64521
daniel@wildcat ~/code/delete % ls -l /proc/64521/exe 
lrwxrwxrwx 1 daniel daniel 0 Jan 29 18:50 /proc/64521/exe -> /home/daniel/code/delete/delete*
daniel@wildcat ~/code/delete % rm delete
daniel@wildcat ~/code/delete % ls -l /proc/64521/exe 
lrwxrwxrwx 1 daniel daniel 0 Jan 29 18:50 /proc/64521/exe -> '/home/daniel/code/delete/delete (deleted)'
daniel@wildcat ~/code/delete % sha256sum /proc/64521/exe
06782f63b8256fb1a86391cf0abde926f2ce9746a7b0dcd225fcad966f0bd712  /proc/64521/exe
daniel@wildcat ~/code/delete % cat /proc/64521/exe > 64521
daniel@wildcat ~/code/delete % killall -9 delete

/proc/PID/fd/

heres zsh:

% ls -l /proc/18776/fd/
total 0
lrwx------ 1 daniel daniel 64 Jan 27 14:02 0 -> /dev/pts/0
lrwx------ 1 daniel daniel 64 Jan 27 14:02 1 -> /dev/pts/0
lrwx------ 1 daniel daniel 64 Jan 27 14:02 10 -> /dev/pts/0
lr-x------ 1 daniel daniel 64 Jan 27 14:02 12 -> /usr/share/zsh/functions/Completion.zwc
lr-x------ 1 daniel daniel 64 Jan 27 14:02 14 -> /usr/share/zsh/functions/Completion/Base.zwc
lr-x------ 1 daniel daniel 64 Jan 27 14:02 15 -> /usr/share/zsh/functions/Prompts.zwc
lr-x------ 1 daniel daniel 64 Jan 27 14:02 16 -> /usr/share/zsh/functions/Misc.zwc
lr-x------ 1 daniel daniel 64 Jan 27 14:02 17 -> /usr/share/zsh/functions/MIME.zwc
lrwx------ 1 daniel daniel 64 Jan 27 14:02 2 -> /dev/pts/0

/proc/PID/io

TODO: example of hunting with this

From the manual page:

              This file contains I/O statistics for the process, for
              example:

                  # cat /proc/3828/io
                  rchar: 323934931
                  wchar: 323929600
                  syscr: 632687
                  syscw: 632675
                  read_bytes: 0
                  write_bytes: 323932160
                  cancelled_write_bytes: 0

              The fields are as follows:

              rchar: characters read
                     The number of bytes which this task has caused to
                     be read from storage.  This is simply the sum of
                     bytes which this process passed to read(2) and
                     similar system calls.  It includes things such as
                     terminal I/O and is unaffected by whether or not
                     actual physical disk I/O was required (the read
                     might have been satisfied from pagecache).

              wchar: characters written
                     The number of bytes which this task has caused, or
                     shall cause to be written to disk.  Similar caveats
                     apply here as with rchar.

              syscr: read syscalls
                     Attempt to count the number of read I/O operations—
                     that is, system calls such as read(2) and pread(2).

              syscw: write syscalls
                     Attempt to count the number of write I/O
                     operations—that is, system calls such as write(2)
                     and pwrite(2).

              read_bytes: bytes read
                     Attempt to count the number of bytes which this
                     process really did cause to be fetched from the
                     storage layer.  This is accurate for block-backed
                     filesystems.

              write_bytes: bytes written
                     Attempt to count the number of bytes which this
                     process caused to be sent to the storage layer.

              cancelled_write_bytes:
                     The big inaccuracy here is truncate.  If a process
                     writes 1 MB to a file and then deletes the file, it
                     will in fact perform no writeout.  But it will have
                     been accounted as having caused 1 MB of write.  In
                     other words: this field represents the number of
                     bytes which this process caused to not happen, by
                     truncating pagecache.  A task can cause "negative"
                     I/O too.  If this task truncates some dirty
                     pagecache, some I/O which another task has been
                     accounted for (in its write_bytes) will not be
                     happening.

              Note: In the current implementation, things are a bit racy
              on 32-bit systems: if process A reads process B's
              /proc/[pid]/io while process B is updating one of these
              64-bit counters, process A could see an intermediate
              result.

              Permission to access this file is governed by a ptrace
              access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).

Example contents of a /proc/PID/io file

% cat /proc/64735/io 
rchar: 4292
wchar: 119
syscr: 12
syscw: 2
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

/proc/PID/maps

TODO: example

This can be used to see which libraries a process is using.

              A file containing the currently mapped memory regions and
              their access permissions.  See mmap(2) for some further
              information about memory mappings.

              Permission to access this file is governed by a ptrace
              access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).

              The format of the file is:

                  address           perms offset  dev   inode       pathname
                  00400000-00452000 r-xp 00000000 08:02 173521      /usr/bin/dbus-daemon
                  00651000-00652000 r--p 00051000 08:02 173521      /usr/bin/dbus-daemon
                  00652000-00655000 rw-p 00052000 08:02 173521      /usr/bin/dbus-daemon
                  00e03000-00e24000 rw-p 00000000 00:00 0           [heap]
                  00e24000-011f7000 rw-p 00000000 00:00 0           [heap]
                  ...
                  35b1800000-35b1820000 r-xp 00000000 08:02 135522  /usr/lib64/ld-2.15.so
                  35b1a1f000-35b1a20000 r--p 0001f000 08:02 135522  /usr/lib64/ld-2.15.so
                  35b1a20000-35b1a21000 rw-p 00020000 08:02 135522  /usr/lib64/ld-2.15.so
                  35b1a21000-35b1a22000 rw-p 00000000 00:00 0
                  35b1c00000-35b1dac000 r-xp 00000000 08:02 135870  /usr/lib64/libc-2.15.so
                  35b1dac000-35b1fac000 ---p 001ac000 08:02 135870  /usr/lib64/libc-2.15.so
                  35b1fac000-35b1fb0000 r--p 001ac000 08:02 135870  /usr/lib64/libc-2.15.so
                  35b1fb0000-35b1fb2000 rw-p 001b0000 08:02 135870  /usr/lib64/libc-2.15.so
                  ...
                  f2c6ff8c000-7f2c7078c000 rw-p 00000000 00:00 0    [stack:986]
                  ...
                  7fffb2c0d000-7fffb2c2e000 rw-p 00000000 00:00 0   [stack]
                  7fffb2d48000-7fffb2d49000 r-xp 00000000 00:00 0   [vdso]

              The address field is the address space in the process that
              the mapping occupies.  The perms field is a set of
              permissions:

                  r = read
                  w = write
                  x = execute
                  s = shared
                  p = private (copy on write)

              The offset field is the offset into the file/whatever; dev
              is the device (major:minor); inode is the inode on that
              device.  0 indicates that no inode is associated with the
              memory region, as would be the case with BSS
              (uninitialized data).

              The pathname field will usually be the file that is
              backing the mapping.  For ELF files, you can easily
              coordinate with the offset field by looking at the Offset
              field in the ELF program headers (readelf -l).

              There are additional helpful pseudo-paths:

              [stack]
                     The initial process's (also known as the main
                     thread's) stack.

              [stack:<tid>] (from Linux 3.4 to 4.4)
                     A thread's stack (where the <tid> is a thread ID).
                     It corresponds to the /proc/[pid]/task/[tid]/ path.
                     This field was removed in Linux 4.5, since
                     providing this information for a process with large
                     numbers of threads is expensive.

              [vdso] The virtual dynamically linked shared object.  See
                     vdso(7).

              [heap] The process's heap.

              If the pathname field is blank, this is an anonymous
              mapping as obtained via mmap(2).  There is no easy way to
              coordinate this back to a process's source, short of
              running it through gdb(1), strace(1), or similar.

              pathname is shown unescaped except for newline characters,
              which are replaced with an octal escape sequence.  As a
              result, it is not possible to determine whether the
              original pathname contained a newline character or the
              literal \012 character sequence.

              If the mapping is file-backed and the file has been
              deleted, the string " (deleted)" is appended to the
              pathname.  Note that this is ambiguous too.

              Under Linux 2.0, there is no field giving pathname.

/proc/PID/map_files/

TODO: example

From the manual page:

              This subdirectory contains entries corresponding to
              memory-mapped files (see mmap(2)).  Entries are named by
              memory region start and end address pair (expressed as
              hexadecimal numbers), and are symbolic links to the mapped
              files themselves.  Here is an example, with the output
              wrapped and reformatted to fit on an 80-column display:

                  # ls -l /proc/self/map_files/
                  lr--------. 1 root root 64 Apr 16 21:31
                              3252e00000-3252e20000 -> /usr/lib64/ld-2.15.so
                  ...

              Although these entries are present for memory regions that
              were mapped with the MAP_FILE flag, the way anonymous
              shared memory (regions created with the MAP_ANON |
              MAP_SHARED flags) is implemented in Linux means that such
              regions also appear on this directory.  Here is an example
              where the target file is the deleted /dev/zero one:

                  lrw-------. 1 root root 64 Apr 16 21:33
                              7fc075d2f000-7fc075e6f000 -> /dev/zero (deleted)

              Permission to access this file is governed by a ptrace
              access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).

              Until kernel version 4.3, this directory appeared only if
              the CONFIG_CHECKPOINT_RESTORE kernel configuration option
              was enabled.

              Capabilities are required to read the contents of the
              symbolic links in this directory: before Linux 5.9, the
              reading process requires CAP_SYS_ADMIN in the initial user
              namespace; since Linux 5.9, the reading process must have
              either CAP_SYS_ADMIN or CAP_CHECKPOINT_RESTORE in the user
              namespace where it resides.

/proc/PID/stat and /proc/PID/status

There are 50+ items to see in stat and status.

TODO: outline some useful items

              Status information about the process.  This is used by
              ps(1).  It is defined in the kernel source file
              fs/proc/array.c.

              The fields, in order, with their proper scanf(3) format
              specifiers, are listed below.  Whether or not certain of
              these fields display valid information is governed by a
              ptrace access mode
              PTRACE_MODE_READ_FSCREDS | PTRACE_MODE_NOAUDIT check
              (refer to ptrace(2)).  If the check denies access, then
              the field value is displayed as 0.  The affected fields
              are indicated with the marking [PT].
/proc/PID/status Provides much of the information in /proc/[pid]/stat and
              /proc/[pid]/statm in a format that's easier for humans to
              parse. 

/proc/PID/syscall

Syscall 230 is nanosleep(). You need to be root to do read syscall.

daniel@wildcat ~/code/delete % sudo cat /proc/64735/syscall
230 0x0 0x0 0x7ffd41fd1660 0x7ffd41fd1660 0x0 0x7f45a2e1fd50 0x7ffd41fd15e0 0x7f45a2ce2334

https://filippo.io/linux-syscall-table/

/proc/uptime

Here is a one-liner that will print the uptime in a human-readable form:

secs=$(cat /proc/uptime |awk {'print $1'});printf '%dd:%dh:%dm:%ds\n' $(($secs/86400)) $((($secs/3600)%24)) $(($secs%3600/60)) $(($secs%60))

/proc/modules

This provides output similar to lsmod

A text list of the modules that have been loaded by the system.

/proc/net/

This directory has information which may be used to implement commands such as netstat and route.

              This directory contains various files and subdirectories
              containing information about the networking layer.  The
              files contain ASCII structures and are, therefore,
              readable with cat(1).  However, the standard netstat(8)
              suite provides much cleaner access to these files.

/proc/version

This file contains kernel version information.

 % cat /proc/version
Linux version 5.6.0-1042-oem (buildd@lcy01-amd64-020) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #46-Ubuntu SMP Thu Jan 7 00:51:04 UTC 2021

 % uname -a
Linux wildcat 5.6.0-1042-oem #46-Ubuntu SMP Thu Jan 7 00:51:04 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux