
Reverse Engineering for Malware: Shellcodes and AV/API Hook Evasion
- On June 10, 2016
- In Blogs
- By Chris Howell
- 0 Comments
With more than a million malware threats being released every day, Cloudburst Security’s reverse engineers are focused on understanding how they work. Our malware reverse engineers explore samples of malicious code to understand how they were created, how they avoid detection and how they manipulate application programming interface (API) functions. APIs are how applications and programs communicate—from phones to computers.
Recently, a Cloudburst Security senior reverse engineer came across an interesting piece of malware shellcode that performs the API/AV (anti-virus) hook evasion.
Image A. Shellcode
[Note: The shellcode was copied directly out of the malicious payload and all addressing in IDA Pro (Interactive Dissassembler) is relative to the beginning of the shellcode. In memory, these addresses would be different.]
Initially, the purpose of this shellcode (see “Image A”, above) is not clear. Rather than any direct calls to any APIs, the shellcode instead calls this subroutine (sub_A7) with the address of the API to call passed in the EDX register.
Image B. Full Function Prolog
At the beginning of each API call, there contains a function prolog that takes up five bytes. The primary purpose of this function prolog is to set up a stack frame for the function’s local variables. In the past, this function prolog was only three bytes. The purpose of the added two byte instruction to the function prolog of “mov edi, edi” was to create enough space for the in-memory hot patching, without the need for a reboot (see “Image B”, above).
There is now enough room to overwrite the function prolog with useful code because AV and analysis tools could also utilize system-wide hooks. These first five bytes could be modified to jump to the API/AV hook evasion process without having to worry about overwriting the functions main code.
Here is the shellcode above, broken down by highlighted sections (see “#1”-“#6” in “Image A”, above).
- “ebx” register contains address after call at 0xC0 where the return address resides (see “seg000:000000A7” in “Image A” above).
- Pointer to the return address code is moved into “ecx” register (see “seg000:000000A8” in “Image A” above).
- “edx” register points to the API call, and five bytes are added to the return address to skip the function prolog (see “seg000:000000AB” in “Image A” above).
- Code implements its own function prolog using memory at “ecx” register (see “seg000:000000AF” in “Image A” above).
- Code is built at return address (see “seg000:000000B4” in “Image A” above).
- “edx” register points to code in API after the function prolog (see “seg000:000000BE” in “Image A” above).
Image C. “push”->”retn” Code on the Stack
By implementing its own return code routine which is a “push” instruction (see “seg 000:00000000” in “Image C”, above) followed by a “retn” instruction (see “seg000:00000005” in “Image C”, above), it bypasses any API/AV hooking code that may reside in the first five bytes of the API call.
Image D. Example: “CreateFileA()” in kernel32.dll
You can see in the enclosed red box the function prolog that precedes the shellcode (see “.text:7C801A28”,“.text:7C801A2A”; and “.text:7C801A2B” in “Image D”). After the function prolog, the “push” instruction is where the code would jump in—after already implementing its own function prolog, prior to the jump (see “.text:7C801A2D” in “Image D).
One consequence of not “call”-ing but rather “jmp”-ing (the previously noted “jump”) into a function prolog is that the return address is not automatically pushed to the stack. The code takes advantage of this by placing the “push”->”retn” code pointer at the location of where the return address would be. Once the function is complete, the API will return to the address of the built code. It will, then, push the return address back to the shellcode after the API/AV hook evasion code and return to it. It does this to avoid detection by any AV/API hooking code and continue with its nefarious work.
To learn more about Cloudburst Security, our operations and how we can help you combat malware, email us at: info@cloudburstsecurity.com