Endpoint Protection and Response is complicated for both offence and defence. In this blog we take a look at Kernel Callbacks, Hooks, and Thread Call Stacks from both perspectives.
Last updated
To recap the series so far, we've gone from looking at the high level purposes and intentions of a Command and Control Server (C2) in general to designing and implementing our first iteration of our implant and server. If you've been following along, you might think you've written a C2...
This is a common mindset. In our experience, getting to this point does not require much sophistication. All of our previous work could easily be achieved (and has been achieved!) using C#, Python, Go, in an evening's worth of frenetic caffeine-fuelled typing. Leading features of C2s can often be linked to pretty old solved concepts and patterns from software engineering, such as thread management, handling idle processes, and ensuring correction execution and program flow.
But as we found when writing our various C2s, and as numerous other offensive developers have found when writing their own implants and servers, once you have the code working and you can get a pingback, you stop running your implant on your development computer and try it on a second computer. This is where the questions start creeping in. Questions like "Why can't I access remote files?", "Why can I make outbound requests over this protocol, but not this?", "Why does this command just fail with no explanation", and for the cynical self-doubter with enough imposter syndrome "Why isn't Defender stopping me from doing this?".
This, personally, is the post we were looking forward to writing. It's going to be a discussion, with a few examples, of increasing common behaviours within environments with active endpoint protection. In 2022, implants face far more scrutiny - the implant and C2 operator must to be prepared to face or evade this scrutiny, and the defender must be aware of how it works so that it can be used to the best of its ability.
Whilst writing this, we also want to clear up the 'it avoids <insert company here> EDR' tweets. Just because the implant is able to execute, doesn't mean that Endpoint Protection is blind to it - it can mean that, but we want to demonstrate some techniques these solutions use to identify malicious behaviour and raise the suspicion of an implant.
In a nutshell, proof of execution is not proof of evasion.
This post will cover:
Setting up The Hunting ELK
Reviewing three ways EDRs can detect or block malicious execution:
Kernel Callbacks
Hooking
Thread Call Stacks
By the end of this post, we will have covered how modern EDRs can protect against malicious implants, and how these protections can be bypassed. We will move from having an implant which technically works to an awareness of how to write an implant which actually starts to work, and can achieve the goals of an operator.
As we've said many times, we are not creating an operational C2. The output from this series is poorly written and riddled with flaws - it only does enough to act as a broken proof-of-concept of the specific items we discuss in this series to avoid this code from being used by bad actors. For this same reason, we are trying to avoid discussing Red Team operational tactics in this series. However, as we go on, it will become obvious why blending in with the compromised users typical behaviour will work. This is something that xpn has discussed on Twitter:
Find Confluence, read Confluence.. become the employee!
— Adam Chester (@_xpn_) January 22, 2022
If your implant has been flagged by EDR, querying NetSessionEnum on every AD-joined computer to find active sessions is probably not typical user behaviour. You likely will not know your implant has been flagged until it stops responding. From here, it's a race until your implant is uploaded to VirusTotal and you have to go back to the drawing board.
We will be referring to the following programs a lot during this blog:
The Hunting ELK (HELK): HELK is an Elastic stack best summarised by themselves:
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack.
This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
PreEmpt: A pseudo-EDR which has the capability to digest EtwTi, memory scanners, hooks, and so on. Although, this is not public but code will be shared when necessary.
These two tools will allow us to generate proof-of-concept data when required.
Similar to Maelstrom: Writing a C2 Implant, we want to have a section dedicated to clearing up some topics we feel need some background before moving on.
Endpoint Detection and Response (EDR) software goes by a number of different acronyms, and there may well be distinctions between different companies programs and their functionality. For the sake of simplicity, we are call all programs that are limited to scanning files on disk statically "anti-virus", and all programs that go further and scan device memory, look at the behaviour of programs while they are running, and responding to threats as they happen "EDR"s. These may be called various names, including XDR, MDR, or just plain AV.
Throughout this series, as we have done so far, we will be sticking with "EDR".
A good overview of this is CrowdStrike's post "What is Endpoint Detection and Response (EDR)":
Endpoint Detection and Response (EDR), also referred to as endpoint detection and threat response (EDTR), is an endpoint security solution that continuously monitors end-user devices to detect and respond to cyber threats like ransomware and malware.
Coined by Gartner’s Anton Chuvakin, EDR is defined as a solution that “records and stores endpoint-system-level behaviors, uses various data analytics techniques to detect suspicious system behavior, provides contextual information, blocks malicious activity, and provides remediation suggestions to restore affected systems.”
Because it's relevant to this post, the next section will look at EDR architecture and comparing EDR behaviours across the various vendors. Without going hugely off-topic, we won't look at a number of also relevant areas, such as how Anti-Virus works, how disk-based protection may work to also stymie your implant execution (if you're still running on disk), and how AV and EDR actually goes about scanning files and their behaviours while they are doing so. Turns out, that's like, a whole field of study.
When discussing endpoint protection, it may help to be somewhat familiar with their architecture. The Symantec EDR Architecture looks something like this:
A similar approach can be seen for Defender for Endpoint. Essentially, a device with the product installed will have an agent which can consist of several drivers and processes, which gather telemetry from various aspects of the machine. Through this post and the next, we will go over a few of those.
As an aside, in a Windows environment, Microsoft inherently have an edge here. While this is currently aimed at "Large Enterprise" customers (or at least, we assume, given their price point for Azure!), Microsoft's Defender and new Defender MDE can both access Microsoft's knowledge of ... their own operating system, but also influence the development of new operating system functionality. Long-term, it wouldn't be a surprise to see Microsoft Defender MDE impact the EDR market in a similar way that Microsoft Defender impacted the anti-virus market.
The general gist of all EDR is that telemetry from the agent is sent to the cloud where it's run through various sandboxes and other test devices, and its behaviour can be further analysed by machine and human operators.
For the excessively curious reader, the following links go in to more detail about specific vendor approaches to EDR architecture:
Without going hugely off-topic, just as how not every red team assessment is a red team, not every EDR is an EDR.
The following "Gartner Magic Quadrant", from Gartner's May 2021 Report roughly maps out the EDR landscape. It's worth noting that CrowdStrike's hire of Alex Ionescu (a maintainer for the Kernel in ReactOS) demonstrates that the current best-in-class EDR's heavily leverage knowledge of internal Windows functionality to maximise their performance:
With so much of EDR functionality relying on implementing the methods we will discuss here such as custom-written direct behaviours like kernel callbacks and hooking, being able to quickly implement new Microsoft Windows features and develop your own custom ways of reliably interacting with and interrupting malicious processes seems to be the distinguishing feature of a modern EDR from its peers.
Another metric that EDR Vendors tend to use, especially because the reports are made so public, is the Mitre Enginuity. The Attack Evaluations is described as thus:
The MITRE Engenuity ATT&CK® Evaluations (Evals) program brings together product and service providers with MITRE experts to collaborate in evaluating security solutions. The Evals process applies a systematic methodology using a threat-informed purple teaming approach to capture critical context around a solution’s ability to detect or protect against known adversary behavior as defined by the ATT&CK knowledge base. Results from each evaluation are thoroughly documented and openly published.
For example, with SentinelOne, their results can be seen in: SentinelOne Overview. The overview goes through APT scenarios and marks whether or not the technique was detected and can be used as a tracker for its "effectiveness". However, some have expressed feelings online that this is not a thorough way to determine the effectiveness of the product.
When looking at EDRs from a purchasing perspective, there are a few methods of determining effectiveness and we wanted to briefly highlight them here. The main thing to consider is that some vendors do not necessarily provide more functionality than an anti-virus. As with any product, ensure that you purchase the right solution for your businesses needs.
When discussing the kernel and user-land model, the following architectural image familiar to any Computer Science graduate will be used:
A big majority of user activity will occur at ring 3, User Mode, surprisingly the Kernel operates within Kernel Mode.
For more information on this, see Windows Programming/User Mode vs Kernel Mode. A worthwhile note is that cross-over between user mode and kernel mode can and does happen. The following definitions from the previous link summarise the differences between these layers:
Ring 0 (also known as kernel mode) has full access to every resource. It is the mode in which the Windows kernel runs.
Rings 1 and 2 can be customized with levels of access but are generally unused unless there are virtual machines running.
Ring 3 (also known as user mode) has restricted access to resources.
Again, to save this post from being longer than it already is, see the Overview of Windows Components documentation for more detail on the following diagram. However, its simply showing the Windows architecture from processes, services, etc, crossing over to the Kernel. We will cover more on this shortly.
Applications that use the WinAPI will traverse through to the Native API (NTAPI) which operates within Kernel Mode.
As an example, API Monitor can be used to look at the calls being executed:
The above shows CreateThread being called and then, subsequently, NtCreateThreadEx being called shortly after.
When a function within KERNEL32.DLL is called, for example CreateThread, it will make a subsequent call to the NTAPI equivalent in NTDLL.DLL. For example, CreateThread calls NtCreateThreadEx. This function will then fill RAX register with the System Service Number (SSN). Finally, NTDLL.dll will then issue a SYSENTER instruction. This will then cause the processor to switch to kernel mode, and jumps to a predefined function, called the System Service Dispatcher. The following image is from Rootkits: Subverting the Windows Kernel, in the section on Userland Hooks:
A driver is a software component of Windows which allows the operating system and device to communicate with each other. Here is an example from What is a driver?:
For example, suppose an application needs to read some data from a device. The application calls a function implemented by the operating system, and the operating system calls a function implemented by the driver. The driver, which was written by the same company that designed and manufactured the device, knows how to communicate with the device hardware to get the data. After the driver gets the data from the device, it returns the data to the operating system, which returns it to the application.
In the case of Endpoint Protection, there are a few reasons why drivers are useful:
The use of Callback Objects which allows for a function to be called if an action occurs. For example, later on we will see the usage of PsSetLoadImageNotifyRoutine which is the call-back object for DLLs being loaded.
Access to privileged information from Event Tracing for Windows Threat Intelligence which is only accessible from the Kernel with an ELAM Driver.
DISCLAIMER: Before moving on, we highly recommend watching REcon 2015 - Hooking Nirvana (Alex Ionescu) Please come back to this post after.
Another common feature of EDR's are the Userland Hooking DLLs. Typically, these are loaded into a process on creation, and are used to proxy WinAPI Calls through themselves to assess the usage, then redirect onto whichever DLL is being used. As an example, if VirtualAlloc was being used, the flow would look something like this:
A hook allows for function instrumentation by intercepting WinAPI calls, by placing a jmp instruction in place of the function address. This jmp will redirect the flow of a call. We will take a look at this in action in the following section. By hooking a function call, it gives the author the ability to:
Assess arguments
Allowing Execution
Blocking Execution
This isn't an exhaustive list, but should serve to demonstrate the functionality which we will be coming across most when running our implants.
Examples of this in use are:
Understanding and Bypassing AMSI: Bypass AMSI by hooking the AmsiScanBuffer call
RDPThief: Intercept and read credentials from RDP
Windows API Hooking: Redirect MessageBoxA
Import Address Table (IAT) Hooking: Redirect MessageBoxA
Intercepting Logon Credentials by Hooking msv1_0!SpAcceptCredentials: Intercept and read credentials from msv1_0!SpAcceptCredentials
Protecting the Heap: Encryption & Hooks: Hook RtlAllocateHeap, RtlReAllocateHeap and RtlFreeHeap to monitor heap allocations
LdrLoadDll Hook: Prevent DLLs being loaded
To access our kernel callbacks without having to write all of that intimidating logic from scratch, we will be using [the] Hunting ELK (HELK):
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
We also use the following script is used from Exploring DLL Loads, Links, and Execution to search through the Sysmon logs:
Kernel Callbacks, according to Microsoft:
The kernel's callback mechanism provides a general way for drivers to request and provide notification when certain conditions are satisfied.
Essentially, they allow drivers to receive and handle notifications for specific events. From veil-ivy/block_create_process.cpp, here is an implementation of using the PsSetLoadImageNotifyRoutine Callback to BLOCK process creation:
In this instance, ObRegisterCallbacks is being used to block the creation of notepad. An Endpoint Protection solution may not use it in this way, but its very likely this type of callback will be used as telemetry to determine if malicious activity is occurring.
In this section, we are going to discuss PsSetLoadImageNotifyRoutine. This callback is responsible for exactly what it says: Sending a notification when an image is loaded into a process. For an example implementation, see Subscribing to Process Creation, Thread Creation and Image Load Notifications from a Kernel Driver.
To understand how PsSetLoadImageNotifyRoutine works, we need to determine what its trigger is.
Assuming the following code:
When LoadLibraryA is called, the function registers a callback to notify the driver than this has happened. In order to see this log in HELK, we use the script we mentioned earlier on.
If we filter for main.exe , which is the above code, we can see the winhttp.dll loaded:
In Elastic, we can also use the following KQL:
event_original_message holds the whole log:
To see what this is doing, we can float through the ReactOS source code:
This is good to get some familiarity with how this would work. However, in Bypassing Image Load Kernel Callbacks, by batsec, identifies that the trigger is in NtCreateSection call which is then called in the LdrpCreateDllSection. So, we don't need to spend too much time debugging to find this.
In the article from batsec, they show that the aforementioned events can be spammed with the following code:
The following screenshot is also from that blog post:
batsec identified that by making the call to NtCreateSection, the event can be spammed whilst not actually loading a DLL. Similarly, the spoof can be somewhat weaponised/manipulated to do other things by updating the LDR_DATA_TABLE_ENTRY struct:
In this example, we will use CertEnroll.dll for no reason at all:
Now we just need to step through the struct and fill out the required information.
Load Time:
Load Reason (LDR_DLL_LOAD_REASON):
Because the Loader needs a module base address, we'll just load shellcode for CALC.EXE here (we'll discuss this part more afterwards):
Hashed Base Name (RtlHashUnicodeString):
Fill out the rest of the struct:
Complete the DdagNode struct:
Here is it in action:
In the above, CertEnroll.dll can be seen loaded in the spoof-load.exe process. Remember, this is not loaded. The only thing that happened here is that a string for that DLL was passed in. We then told the loader than the base address of the DLL is that of the shellcode:
Looking at this technique, there are two obvious use cases:
Tie the implant base address (C2IMPLANT.REFLECTIVE.DLL) to a legitimate DLL (ADVAP32.DLL) causing it to appear less suspicious
Remove an IOC Library (WinHTTP.DLL) by loading ADVAPI32.DLL but pointing it to a WinHTTP.DLL base address.
We aren't going to reinvent the wheel here, its explained wonderfully in Bypassing Image Load Kernel Callbacks. Essentially, to cause the callback to not trigger, a full loader needs to be rewritten. The conclusion to that research was DarkLoadLibrary:
In essence,
DarkLoadLibraryis an implementation ofLoadLibrarythat will not trigger image load events. It also has a ton of extra features that will make life easier during malware development.
A proof-of-concept usage of this library was taken from DLL Shenanigans.
Let's inspect it:
Then the above 3 commands are ran:
dark-loader uses the LOAD_LOCAL_FILE flag to load a disk from disk, as LoadLibraryA does.
The Image Load logs are searched for Kernel32 to make sure logs were found.
winhttp.dll was searched, and nothing returned
To avoid the call to NtCreateSection which was identified to be registering the callback, the section mapping is done with NtAllocateVirtualMemory or VirtualAlloc, as seen in MapSections().
Obviously, PsSetLoadImageNotifyRoutine is not the only callback, and there are quite a few other callbacks readily available. Kernel Callback Functions has a (non-comprehensive!) list:
One that would be powerful would be PsSetCreateProcessNotifyRoutineEx() as the notification for process creation would be crippling for system telemetry. At the time of writing, we are not aware of any research in this space. Although to be totally honest, we haven't looked.
In this section, we are going to look at some popular, but elementary, hooking techniques.
Lets look at two examples before looking into some libraries - Manual Hooks in x86 and NtSetProcessInformation Callbacks.
Manual Hooks (x86)
Using Windows API Hooking as a x86 example (easier to demonstrate), we can adapt the code to look something like this:
Lets walk through this...
First off, MessageBoxA is in User32.dll so we load that:
Next, we need the address of USER32!MessageBoxA:
With that address, the bytes can now be read:
This will read the first 6 bytes of the function call which will later be updated to hold a push to the new function, resulting in a jmp.
The bytes:
Now, the patch needs to be built. This is done like so:
The hex produced from this:
Using defuse.ca to disassemble this, the above can be translated into Assembly:
Note that 0x00BD1212 being pushed is the address of the function we want to jump to INSTEAD of theUSER32!MessageBoxA call:
At this point, the patch is prepared. It's going to replace the first 6 bytes with a push to the new address.
The next thing is to actually write this new address in:
Then, in the disassembly:
A jmp is added to jump to the new function. Allowing this to run calls the hooked function and the arguments are printed:
Running it:
NtSetProcessInformation Callbacks
In REcon 2015 - Hooking Nirvana (Alex Ionescu), Alex Ionescu demonstrates Process Instrumentation with NtSetProcessInformation with the ProcessInstrumentationCallback flag. In this talk, Alex demonstrates hooking via callbacks with Process Instrumentation.
Setting up the callback is straight forward:
Where the callback function is included as follows:
CALLBACK_FUNCTION_GOES_HERE is a function to use as the callback and then ProcessInstrumentationCallback is:
An additional point is that by setting the callback to NULL, any callbacks sent will be removed. This was documented by modexp in Bypassing User-Mode Hooks and Direct Invocation of System Calls for Red Teams.
This talk was then built on by Secrary and again in Secrary's blog Hooking via InstrumentationCallback. The original code from Alex Ionescu can be found in the HookingNirvana repo.
Borrowing the hooks from Secrary gives access to the function and return value, giving us the following Assembly:
Hook: With the assembly written, we also need to write the function called by the assembly, allowing us to take in all of the provided registers and return their function names:
Then, in main, SymInitialize is called, then the instrumentation is set:
Running this completed example, we can now see all of the function names and return codes:
The hook could be updated to get access to the arguments for a full analysis, but we didn't feel the need to look into that for this initial proof-of-concept.
One final mention for this technique is that it can be used to enumerate the the System Service Number (SSN) for a given function call. This was documented by Paranoid Ninja in EtwTi-Syscall-Hook and Release v0.8 - Warfare Tactics, where the hook is significantly smaller (at the cost of doing far less):
And its companion assembly:
Back in 2019 Cneelis published Red Team Tactics: Combining Direct System Calls and sRDI to bypass AV/EDR which had a subsequent release of SysWhispers:
SysWhispers provides red teamers the ability to generate header/ASM pairs for any system call in the core kernel image (ntoskrnl.exe). The headers will also include the necessary type definitions.
Then modexp provided an update which corrected a shortcoming with version 1 and gave us SysWhispers2:
The specific implementation in SysWhispers2 is a variation of @modexpblog’s code. One difference is that the function name hashes are randomized on each generation. @ElephantSe4l, who had published this technique earlier, has another implementation based in C++17 which is also worth checking out.
The main change is the introduction of base.c which is a result of Bypassing User-Mode Hooks and Direct Invocation of System Calls for Red Teams.
And again, KlezVirus produced SysWhispers3:
The usage is pretty similar to SysWhispers2, with the following exceptions:
It also supports x86/WoW64
It supports syscalls instruction replacement with an EGG (to be dynamically replaced)
It supports direct jumps to syscalls in x86/x64 mode (in WOW64 it's almost standard)
It supports direct jumps to random syscalls (borrowing @ElephantSeal's idea)
A better explanation of these features are better outlined i the blog post SysWhispers is dead, long live SysWhispers!
This is just one suite of SysCall techniques, there's a whole other technique based on Heavens Gate.
See Gatekeeping Syscalls for a breakdown on these different techniques.
EVEN THEN! There are more:
RECAP!
With the ability to transition into Kernel-Mode, we have the ability to go unseen by the User-land hooks. So, lets build something.
For our example, we are going to use MinHook:
The Minimalistic x86/x64 API Hooking Library for Windows
The DLL
So, this is going to be a DLL which gets loaded into a process and then hooks functionality and makes some decision based on its behaviour. Here is DllMain:
When a DLL_PROCESS_ATTACH is the load reason, then we create a new thread and point it at our "main" function. This is where we initialise minhook, and set up some hooks:
MH_Initialize() is a mandatory call, so we start with that. Next, we create 3 hooks:
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtWriteVirtualMemory
Hooks are created with the MH_CreateHookApi() call:
To create a hook, 4 things are needed:
Module Name
Function Name
Function to "replace" the desired function
Somewhere to store the original function address
Below is an example:
NtAllocateVirtualMemory_Hook() is the function used to replace the original function:
The function is declared exactly the same as typedef for the function:
This is so that there are no issues with typing between hooks.
In the NtAllocateVirtualMemory_Hook function, the only thing we are checking here is if the protection type is PAGE_EXECUTE_READWRITE, RWX, because this is commonly a sign of malicious activity (COMMONLY). If it matches, we just print that we found something.
Then, we have a concept of blocking. This simply means that if BLOCKING is true, then it returns. If its false, then we return the pointer to the original function, allowing the function to execute as the user expects.
In NtProtectVirtualMemory, we just check for changes to PAGE_EXECUTE_READ as this is the common protection type to avoid RWX allocations:
In NtWriteVirtualMemory, no additional checks are made:
The Loader
In this instance, we have a PE which just calls LoadLibraryA on the DLL, and then runs a fake injection:
Detecting Functionality
Running this shows the calls being detected (in a non-blocking mode):
In the screenshot, we can see:
Moves to RX
RWX Allocations
Writes of 8 bytes
This is everything we planned on detecting. So, how would a bypass work here? Well, because of a lot of community development, its quite easy in practice. But before that, we need to discuss User-land and Kernel-land.
Bypassing the User-land hooks
For this example, we are going to use Tartarus Gate. All we have to do is make this one call in wmain:
And then change the payload in Payload:
Checking the loaded modules:
The DLL is loaded..
Running it, and setting a breakpoint on the thread creation because the payload is junk:
The above shows minhook being initialised, and then the hooks being enabled. Between this, there is a move to RX. However, it happens before the hooks are set up. So this is likely either minhook, or CRT doing something. We did not take the time to check this out.
As we've stated, this isn't a comprehensive review of every potential technique. Another which might specifically be worth exploring is Vectored Exception Handling (VEH). Two examples of this are ethicalchaos's post on In-Process Patchless AMSI Bypass and Countercept's CallStackSpoofer.
As with Kernel callbacks, this is a live field of study and there's far more to be explored than we have time or space in this post.
Another component of a process which gets interrogated is via the Threads Call Stack. As detailed in Viewing the Call Stack in WinDbg, the Call Stack is defined as:
The call stack is the chain of function calls that have led to the current location of the program counter. The top function on the call stack is the current function, the next function is the function that called the current function, and so on. The call stack that is displayed is based on the current program counter, unless you change the register context. For more information about how to change the register context, see Changing Contexts.
As the call stack can help determine the intention of a thread, it often undergoes scrutiny to determine its validity. In this section, we want to demonstrate how the call stack can be used to determine malicious behaviour (in a rudimentary example), and then discuss the offensive strategy for handling this.
Here, we have a an implant:
If we look at the processes (10792) threads, we can see a bunch of threads starting at the elusive TpReleaseCleanupGroupMembers
This is quite common amongst processes, here is an example of chrome.exe:
And then RuntimeBroker.exe:
This is a good side-note for attackers. If the implant in question is reliant on masquarading as something else, then this needs to be considered. For example, if the implant is operating out of browsers such as chrome, then the HTTP should be handled the same way, and then entry-point and call stack of the thread should be mimicked.
Back to the Vulpes implant, the call stack is primarily TpReleaseCleanupGroupMembers which is fine. However, if we go through some of the threads, here is the thread responsible for WinHTTP:
And here is a generic thread started by the process:
There are a few others, but lets focus on the second example because this is a call stack for a thread that will be found in a lot of processes. Lets look at how to programmatically read the thread stack and how a spoofed thread base address can look suspicious.
Here is the entry point:
In the above, we have two thread IDs.
1996: The spoofed thread
26084: A somewhat normal stack
With that, we need to write a function to enumerate the call stack of the thread. We can do that with the following code:
From the DbgHelp library, we are using:
StackWalk64: Obtains a stack trace.
SyGetModuleBase64: Retrieves the base address of the module that contains the specified address.
SymFromAddr: Retrieves symbol information for the specified address.
Pointing the code to the normal thread stack:
This matches what we saw earlier on. Changing this to point to the bad thread:
This now shows a different thread stack we haven't seen so far. Looking at frame 3, its CreateTimeQueueTimer which was described as the sleep obfuscation technique in:
https://pre.empt.blog/2023/obfuscating-reflective-dll-memory-regions-with-timers
As a disclaimer, this technique has full kudos to Peter Winter-Smith.
So, programmatically, its easy to find out the callstack of a thread. Let's expand this into something completely rudimentary that we can start to work with.
First, we'll define a hard-coded list of expected functions that we saw earlier on that we can use as an integrity check:
And then an empty one, to track everything we find:
Now, instead of just printing, lets add all the symbol names into a vector:
Once the code has ran, and found all the symbols, lets see if the vectors match:
Pointing this at the good thread:
We get the CLEAN message. And then the dirty thread:
Obviously, this code isn't production ready and the nuances of writing this kind of logic properly is extremely challenging. However, it is something that some EDR vendors are starting to pick up. Given the increase into research to confuse and blind endpoint protection, this is a good technique to have in the arsenal for both the blue and red teams.
Speaking of red teams, research into correcting this thread-mishap has already been ongoing.
This technique was first popularized by Peter Winter-Smith, who is a common reoccurrence in this space, and then reinterpreted by mgeeky in ThreadStackSpoofer. However, this proof-of-concept sets the return address to 0, removing references to memory addresses for shellcode injection.
This is an example implementation for Thread Stack Spoofing technique aiming to evade Malware Analysts, AVs and EDRs looking for references to shellcode's frames in an examined thread's call stack. The idea is to hide references to the shellcode on thread's call stack thus masquerading allocations containing malware's code.
If we remove the sleep masking from Vulpes, here is how the call stack looks:
The technique would aim to mask these addresses by storing the return address into a variable, setting the return address to 0, and then restoring the return address.
For a quick code example from the above repository:
In a more recent project, CallStackSpoofer by William Burgess was produced to take this a step further and fully mask the stack.
See: Spoofing Call Stacks to confused EDRs
By using predefined vectors of stacks, the project is able to mimic:
WMI
RPC
SVCHost
An example of a predefined vector for WMI:
By implementing this type of technique, it will make it extremely difficult to implement the callstack integrity checking we showed earlier (granted our demo was hard-coded values, but the point still stands).
This was a fairly long post in which we tried to provide some clarity into the mechanisms EDRs can use to not only identify malicious activity, but prevent it. Along the way we've discussed common pitfalls and some enhancements that can be made to protect against the bypasses.
Whilst doing this, we've tried to shed more light onto the 'X bypasses EDR' narrative in which, yes, the beacon might have comeback but there is likely logs of the activity.
The next episode will look at ETW and AMSI!
