Endpoint Protection and Response is complicated for both offence and defence. In this blog we take a look at Kernel Callbacks, Hooks, and Thread Call Stacks from both perspectives.
Last updated
To recap the series so far, we've gone from looking at the high level purposes and intentions of a Command and Control Server (C2) in general to designing and implementing our first iteration of our implant and server. If you've been following along, you might think you've written a C2...
This is a common mindset. In our experience, getting to this point does not require much sophistication. All of our previous work could easily be achieved (and has been achieved!) using C#, Python, Go, in an evening's worth of frenetic caffeine-fuelled typing. Leading features of C2s can often be linked to pretty old solved concepts and patterns from software engineering, such as thread management, handling idle processes, and ensuring correction execution and program flow.
But as we found when writing our various C2s, and as numerous other offensive developers have found when writing their own implants and servers, once you have the code working and you can get a pingback, you stop running your implant on your development computer and try it on a second computer. This is where the questions start creeping in. Questions like "Why can't I access remote files?", "Why can I make outbound requests over this protocol, but not this?", "Why does this command just fail with no explanation", and for the cynical self-doubter with enough imposter syndrome "Why isn't Defender stopping me from doing this?".
This, personally, is the post we were looking forward to writing. It's going to be a discussion, with a few examples, of increasing common behaviours within environments with active endpoint protection. In 2022, implants face far more scrutiny - the implant and C2 operator must to be prepared to face or evade this scrutiny, and the defender must be aware of how it works so that it can be used to the best of its ability.
Whilst writing this, we also want to clear up the 'it avoids <insert company here> EDR' tweets. Just because the implant is able to execute, doesn't mean that Endpoint Protection is blind to it - it can mean that, but we want to demonstrate some techniques these solutions use to identify malicious behaviour and raise the suspicion of an implant.
In a nutshell, proof of execution is not proof of evasion.
This post will cover:
Reviewing three ways EDRs can detect or block malicious execution:
Kernel Callbacks
Hooking
Thread Call Stacks
By the end of this post, we will have covered how modern EDRs can protect against malicious implants, and how these protections can be bypassed. We will move from having an implant which technically works to an awareness of how to write an implant which actually starts to work, and can achieve the goals of an operator.
Find Confluence, read Confluence.. become the employee!
We will be referring to the following programs a lot during this blog:
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack.
This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
These two tools will allow us to generate proof-of-concept data when required.
Endpoint Detection and Response (EDR) software goes by a number of different acronyms, and there may well be distinctions between different companies programs and their functionality. For the sake of simplicity, we are call all programs that are limited to scanning files on disk statically "anti-virus", and all programs that go further and scan device memory, look at the behaviour of programs while they are running, and responding to threats as they happen "EDR"s. These may be called various names, including XDR, MDR, or just plain AV.
Throughout this series, as we have done so far, we will be sticking with "EDR".
Endpoint Detection and Response (EDR), also referred to as endpoint detection and threat response (EDTR), is an endpoint security solution that continuously monitors end-user devices to detect and respond to cyber threats like ransomware and malware.
Because it's relevant to this post, the next section will look at EDR architecture and comparing EDR behaviours across the various vendors. Without going hugely off-topic, we won't look at a number of also relevant areas, such as how Anti-Virus works, how disk-based protection may work to also stymie your implant execution (if you're still running on disk), and how AV and EDR actually goes about scanning files and their behaviours while they are doing so. Turns out, that's like, a whole field of study.
The general gist of all EDR is that telemetry from the agent is sent to the cloud where it's run through various sandboxes and other test devices, and its behaviour can be further analysed by machine and human operators.
For the excessively curious reader, the following links go in to more detail about specific vendor approaches to EDR architecture:
Without going hugely off-topic, just as how not every red team assessment is a red team, not every EDR is an EDR.
With so much of EDR functionality relying on implementing the methods we will discuss here such as custom-written direct behaviours like kernel callbacks and hooking, being able to quickly implement new Microsoft Windows features and develop your own custom ways of reliably interacting with and interrupting malicious processes seems to be the distinguishing feature of a modern EDR from its peers.
When looking at EDRs from a purchasing perspective, there are a few methods of determining effectiveness and we wanted to briefly highlight them here. The main thing to consider is that some vendors do not necessarily provide more functionality than an anti-virus. As with any product, ensure that you purchase the right solution for your businesses needs.
When discussing the kernel and user-land model, the following architectural image familiar to any Computer Science graduate will be used:
A big majority of user activity will occur at ring 3, User Mode, surprisingly the Kernel operates within Kernel Mode.
Ring 0 (also known as kernel mode) has full access to every resource. It is the mode in which the Windows kernel runs.
Rings 1 and 2 can be customized with levels of access but are generally unused unless there are virtual machines running.
Ring 3 (also known as user mode) has restricted access to resources.
As an example, API Monitor can be used to look at the calls being executed:
The above shows CreateThread being called and then, subsequently, NtCreateThreadEx being called shortly after.
For example, suppose an application needs to read some data from a device. The application calls a function implemented by the operating system, and the operating system calls a function implemented by the driver. The driver, which was written by the same company that designed and manufactured the device, knows how to communicate with the device hardware to get the data. After the driver gets the data from the device, it returns the data to the operating system, which returns it to the application.
In the case of Endpoint Protection, there are a few reasons why drivers are useful:
Access to privileged information from Event Tracing for Windows Threat Intelligence which is only accessible from the Kernel with an ELAM Driver.
Another common feature of EDR's are the Userland Hooking DLLs. Typically, these are loaded into a process on creation, and are used to proxy WinAPI Calls through themselves to assess the usage, then redirect onto whichever DLL is being used. As an example, if VirtualAlloc was being used, the flow would look something like this:
A hook allows for function instrumentation by intercepting WinAPI calls, by placing a jmp instruction in place of the function address. This jmp will redirect the flow of a call. We will take a look at this in action in the following section. By hooking a function call, it gives the author the ability to:
Assess arguments
Allowing Execution
Blocking Execution
This isn't an exhaustive list, but should serve to demonstrate the functionality which we will be coming across most when running our implants.
Examples of this in use are:
To access our kernel callbacks without having to write all of that intimidating logic from scratch, we will be using [the] Hunting ELK (HELK):
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
The kernel's callback mechanism provides a general way for drivers to request and provide notification when certain conditions are satisfied.
In this instance, ObRegisterCallbacks is being used to block the creation of notepad. An Endpoint Protection solution may not use it in this way, but its very likely this type of callback will be used as telemetry to determine if malicious activity is occurring.
To understand how PsSetLoadImageNotifyRoutine works, we need to determine what its trigger is.
Assuming the following code:
If we filter for main.exe , which is the above code, we can see the winhttp.dll loaded:
In Elastic, we can also use the following KQL:
event_original_message holds the whole log:
In the article from batsec, they show that the aforementioned events can be spammed with the following code:
The following screenshot is also from that blog post:
In this example, we will use CertEnroll.dll for no reason at all:
Now we just need to step through the struct and fill out the required information.
Load Time:
Because the Loader needs a module base address, we'll just load shellcode for CALC.EXE here (we'll discuss this part more afterwards):
Fill out the rest of the struct:
Complete the DdagNode struct:
Here is it in action:
In the above, CertEnroll.dll can be seen loaded in the spoof-load.exe process. Remember, this is not loaded. The only thing that happened here is that a string for that DLL was passed in. We then told the loader than the base address of the DLL is that of the shellcode:
Looking at this technique, there are two obvious use cases:
Tie the implant base address (C2IMPLANT.REFLECTIVE.DLL) to a legitimate DLL (ADVAP32.DLL) causing it to appear less suspicious
Remove an IOC Library (WinHTTP.DLL) by loading ADVAPI32.DLL but pointing it to a WinHTTP.DLL base address.
In essence,
DarkLoadLibraryis an implementation ofLoadLibrarythat will not trigger image load events. It also has a ton of extra features that will make life easier during malware development.
Let's inspect it:
Then the above 3 commands are ran:
dark-loader uses the LOAD_LOCAL_FILE flag to load a disk from disk, as LoadLibraryA does.
The Image Load logs are searched for Kernel32 to make sure logs were found.
winhttp.dll was searched, and nothing returned
One that would be powerful would be PsSetCreateProcessNotifyRoutineEx() as the notification for process creation would be crippling for system telemetry. At the time of writing, we are not aware of any research in this space. Although to be totally honest, we haven't looked.
In this section, we are going to look at some popular, but elementary, hooking techniques.
Lets look at two examples before looking into some libraries - Manual Hooks in x86 and NtSetProcessInformation Callbacks.
Manual Hooks (x86)
Lets walk through this...
First off, MessageBoxA is in User32.dll so we load that:
Next, we need the address of USER32!MessageBoxA:
With that address, the bytes can now be read:
This will read the first 6 bytes of the function call which will later be updated to hold a push to the new function, resulting in a jmp.
The bytes:
Now, the patch needs to be built. This is done like so:
The hex produced from this:
Note that 0x00BD1212 being pushed is the address of the function we want to jump to INSTEAD of theUSER32!MessageBoxA call:
At this point, the patch is prepared. It's going to replace the first 6 bytes with a push to the new address.
The next thing is to actually write this new address in:
Then, in the disassembly:
A jmp is added to jump to the new function. Allowing this to run calls the hooked function and the arguments are printed:
Running it:
NtSetProcessInformation Callbacks
Setting up the callback is straight forward:
Where the callback function is included as follows:
CALLBACK_FUNCTION_GOES_HERE is a function to use as the callback and then ProcessInstrumentationCallback is:
Borrowing the hooks from Secrary gives access to the function and return value, giving us the following Assembly:
Hook: With the assembly written, we also need to write the function called by the assembly, allowing us to take in all of the provided registers and return their function names:
Running this completed example, we can now see all of the function names and return codes:
The hook could be updated to get access to the arguments for a full analysis, but we didn't feel the need to look into that for this initial proof-of-concept.
And its companion assembly:
SysWhispers provides red teamers the ability to generate header/ASM pairs for any system call in the core kernel image (ntoskrnl.exe). The headers will also include the necessary type definitions.
It also supports x86/WoW64
It supports syscalls instruction replacement with an EGG (to be dynamically replaced)
It supports direct jumps to syscalls in x86/x64 mode (in WOW64 it's almost standard)
This is just one suite of SysCall techniques, there's a whole other technique based on Heavens Gate.
EVEN THEN! There are more:
RECAP!
With the ability to transition into Kernel-Mode, we have the ability to go unseen by the User-land hooks. So, lets build something.
The Minimalistic x86/x64 API Hooking Library for Windows
The DLL
So, this is going to be a DLL which gets loaded into a process and then hooks functionality and makes some decision based on its behaviour. Here is DllMain:
When a DLL_PROCESS_ATTACH is the load reason, then we create a new thread and point it at our "main" function. This is where we initialise minhook, and set up some hooks:
MH_Initialize() is a mandatory call, so we start with that. Next, we create 3 hooks:
NtAllocateVirtualMemory
NtProtectVirtualMemory
NtWriteVirtualMemory
Hooks are created with the MH_CreateHookApi() call:
To create a hook, 4 things are needed:
Module Name
Function Name
Function to "replace" the desired function
Somewhere to store the original function address
Below is an example:
NtAllocateVirtualMemory_Hook() is the function used to replace the original function:
The function is declared exactly the same as typedef for the function:
This is so that there are no issues with typing between hooks.
In the NtAllocateVirtualMemory_Hook function, the only thing we are checking here is if the protection type is PAGE_EXECUTE_READWRITE, RWX, because this is commonly a sign of malicious activity (COMMONLY). If it matches, we just print that we found something.
Then, we have a concept of blocking. This simply means that if BLOCKING is true, then it returns. If its false, then we return the pointer to the original function, allowing the function to execute as the user expects.
In NtProtectVirtualMemory, we just check for changes to PAGE_EXECUTE_READ as this is the common protection type to avoid RWX allocations:
In NtWriteVirtualMemory, no additional checks are made:
The Loader
In this instance, we have a PE which just calls LoadLibraryA on the DLL, and then runs a fake injection:
Detecting Functionality
Running this shows the calls being detected (in a non-blocking mode):
In the screenshot, we can see:
Moves to RX
RWX Allocations
Writes of 8 bytes
This is everything we planned on detecting. So, how would a bypass work here? Well, because of a lot of community development, its quite easy in practice. But before that, we need to discuss User-land and Kernel-land.
Bypassing the User-land hooks
And then change the payload in Payload:
Checking the loaded modules:
The DLL is loaded..
Running it, and setting a breakpoint on the thread creation because the payload is junk:
The above shows minhook being initialised, and then the hooks being enabled. Between this, there is a move to RX. However, it happens before the hooks are set up. So this is likely either minhook, or CRT doing something. We did not take the time to check this out.
As with Kernel callbacks, this is a live field of study and there's far more to be explored than we have time or space in this post.
As the call stack can help determine the intention of a thread, it often undergoes scrutiny to determine its validity. In this section, we want to demonstrate how the call stack can be used to determine malicious behaviour (in a rudimentary example), and then discuss the offensive strategy for handling this.
Here, we have a an implant:
This is quite common amongst processes, here is an example of chrome.exe:
And then RuntimeBroker.exe:
This is a good side-note for attackers. If the implant in question is reliant on masquarading as something else, then this needs to be considered. For example, if the implant is operating out of browsers such as chrome, then the HTTP should be handled the same way, and then entry-point and call stack of the thread should be mimicked.
Back to the Vulpes implant, the call stack is primarily TpReleaseCleanupGroupMembers which is fine. However, if we go through some of the threads, here is the thread responsible for WinHTTP:
And here is a generic thread started by the process:
There are a few others, but lets focus on the second example because this is a call stack for a thread that will be found in a lot of processes. Lets look at how to programmatically read the thread stack and how a spoofed thread base address can look suspicious.
Here is the entry point:
In the above, we have two thread IDs.
1996: The spoofed thread
26084: A somewhat normal stack
With that, we need to write a function to enumerate the call stack of the thread. We can do that with the following code:
Pointing the code to the normal thread stack:
This matches what we saw earlier on. Changing this to point to the bad thread:
https://pre.empt.blog/2023/obfuscating-reflective-dll-memory-regions-with-timers
So, programmatically, its easy to find out the callstack of a thread. Let's expand this into something completely rudimentary that we can start to work with.
First, we'll define a hard-coded list of expected functions that we saw earlier on that we can use as an integrity check:
And then an empty one, to track everything we find:
Now, instead of just printing, lets add all the symbol names into a vector:
Once the code has ran, and found all the symbols, lets see if the vectors match:
Pointing this at the good thread:
We get the CLEAN message. And then the dirty thread:
Obviously, this code isn't production ready and the nuances of writing this kind of logic properly is extremely challenging. However, it is something that some EDR vendors are starting to pick up. Given the increase into research to confuse and blind endpoint protection, this is a good technique to have in the arsenal for both the blue and red teams.
Speaking of red teams, research into correcting this thread-mishap has already been ongoing.
This is an example implementation for Thread Stack Spoofing technique aiming to evade Malware Analysts, AVs and EDRs looking for references to shellcode's frames in an examined thread's call stack. The idea is to hide references to the shellcode on thread's call stack thus masquerading allocations containing malware's code.
If we remove the sleep masking from Vulpes, here is how the call stack looks:
The technique would aim to mask these addresses by storing the return address into a variable, setting the return address to 0, and then restoring the return address.
For a quick code example from the above repository:
By using predefined vectors of stacks, the project is able to mimic:
WMI
RPC
SVCHost
By implementing this type of technique, it will make it extremely difficult to implement the callstack integrity checking we showed earlier (granted our demo was hard-coded values, but the point still stands).
This was a fairly long post in which we tried to provide some clarity into the mechanisms EDRs can use to not only identify malicious activity, but prevent it. Along the way we've discussed common pitfalls and some enhancements that can be made to protect against the bypasses.
Whilst doing this, we've tried to shed more light onto the 'X bypasses EDR' narrative in which, yes, the beacon might have comeback but there is likely logs of the activity.
The next episode will look at ETW and AMSI!
Setting up
As we've said many times, we are not creating an operational C2. The output from this series is poorly written and riddled with flaws - it only does enough to act as a broken proof-of-concept of the specific items we discuss in this series to avoid this code from being used by bad actors. For this same reason, we are trying to avoid discussing Red Team operational tactics in this series. However, as we go on, it will become obvious why blending in with the compromised users typical behaviour will work. This is something that has discussed on Twitter:
— Adam Chester (@_xpn_)
If your implant has been flagged by EDR, querying on every AD-joined computer to find active sessions is probably not typical user behaviour. You likely will not know your implant has been flagged until it stops responding. From here, it's a race until your implant is uploaded to and you have to go back to the drawing board.
(HELK): HELK is an stack best summarised by themselves:
: A pseudo-EDR which has the capability to digest EtwTi, memory scanners, hooks, and so on. Although, this is not public but code will be shared when necessary.
Similar to , we want to have a section dedicated to clearing up some topics we feel need some background before moving on.
A good overview of this is :
EDR is defined as a solution that “records and stores endpoint-system-level behaviors, uses various data analytics techniques to detect suspicious system behavior, provides contextual information, blocks malicious activity, and provides remediation suggestions to restore affected systems.”
When discussing endpoint protection, it may help to be somewhat familiar with their architecture. The looks something like this:
A similar approach can be seen for . Essentially, a device with the product installed will have an agent which can consist of several drivers and processes, which gather telemetry from various aspects of the machine. Through this post and the next, we will go over a few of those.
As an aside, in a Windows environment, Microsoft inherently have an edge here. While this is (or at least, we assume, given their price point for Azure!), Microsoft's Defender and new Defender MDE can both access Microsoft's knowledge of ... their own operating system, but also influence the development of new operating system functionality. Long-term, it wouldn't be a surprise to see Microsoft Defender MDE impact the EDR market in a similar way that Microsoft Defender impacted the anti-virus market.
The following , from roughly maps out the EDR landscape. It's worth noting that of (a maintainer for the Kernel in ) demonstrates that the current best-in-class EDR's heavily leverage knowledge of internal Windows functionality to maximise their performance:
Another metric that EDR Vendors tend to use, especially because the reports are made so public, is the . The is described as thus:
The MITRE Engenuity ATT&CK® Evaluations (Evals) program brings together product and service providers with MITRE experts to collaborate in evaluating security solutions. The Evals process applies a using a threat-informed purple teaming approach to capture critical context around a solution’s ability to detect or protect against known adversary behavior as defined by the ATT&CK knowledge base. Results from each evaluation are thoroughly documented and openly published.
For example, with , their results can be seen in: . The overview goes through APT scenarios and marks whether or not the technique was detected and can be used as a tracker for its "effectiveness". However, some have expressed feelings online that this is not a thorough way to determine the effectiveness of the product.
For more information on this, see . A worthwhile note is that cross-over between user mode and kernel mode can and does happen. The following definitions from the previous link summarise the differences between these layers:
Again, to save this post from being longer than it already is, see the documentation for more detail on the following diagram. However, its simply showing the Windows architecture from processes, services, etc, crossing over to the Kernel. We will cover more on this shortly.
Applications that use the WinAPI will traverse through to the (NTAPI) which operates within Kernel Mode.
When a function within KERNEL32.DLL is called, for example CreateThread, it will make a subsequent call to the NTAPI equivalent in NTDLL.DLL. For example, calls . This function will then fill register with the System Service Number (SSN). Finally, NTDLL.dll will then issue a instruction. This will then cause the processor to switch to kernel mode, and jumps to a predefined function, called the System Service Dispatcher. The following image is from , in the section on :
A driver is a software component of Windows which allows the operating system and device to communicate with each other. Here is an example from :
The use of which allows for a function to be called if an action occurs. For example, later on we will see the usage of which is the call-back object for DLLs being loaded.
DISCLAIMER: Before moving on, we highly recommend watching Please come back to this post after.
: Bypass AMSI by hooking the AmsiScanBuffer call
: Intercept and read credentials from RDP
: Redirect MessageBoxA
: Redirect MessageBoxA
: Intercept and read credentials from msv1_0!SpAcceptCredentials
: Hook , and to monitor heap allocations
: Prevent DLLs being loaded
We also use the following script is used from to search through the Sysmon logs:
, according to Microsoft:
Essentially, they allow drivers to receive and handle notifications for specific events. From , here is an implementation of using the PsSetLoadImageNotifyRoutine Callback to BLOCK process creation:
In this section, we are going to discuss . This callback is responsible for exactly what it says: Sending a notification when an image is loaded into a process. For an example implementation, see .
When LoadLibraryA is called, the function registers a callback to notify the driver than this has happened. In order to see this log in HELK, we use .
To see what this is doing, we can float through the source code:
This is good to get some familiarity with how this would work. However, in , by , identifies that the trigger is in call which is then called in the LdrpCreateDllSection. So, we don't need to spend too much time debugging to find this.
batsec identified that by making the call to NtCreateSection, the event can be spammed whilst not actually loading a DLL. Similarly, the spoof can be somewhat weaponised/manipulated to do other things by updating the struct:
Load Reason ():
Hashed Base Name ():
We aren't going to reinvent the wheel here, its explained wonderfully in . Essentially, to cause the callback to not trigger, a full loader needs to be rewritten. The conclusion to that research was :
A proof-of-concept usage of this library was taken from .
To avoid the call to NtCreateSection which was identified to be registering the callback, the section mapping is done with NtAllocateVirtualMemory or VirtualAlloc, as seen in .
Obviously, PsSetLoadImageNotifyRoutine is not the only callback, and there are quite a few other callbacks readily available. has a (non-comprehensive!) list:
Using as a x86 example (easier to demonstrate), we can adapt the code to look something like this:
Using to disassemble this, the above can be translated into Assembly:
In , demonstrates Process Instrumentation with with the ProcessInstrumentationCallback flag. In this talk, Alex demonstrates hooking via callbacks with Process Instrumentation.
An additional point is that by setting the callback to NULL, any callbacks sent will be removed. This was documented by in .
This talk was then built on by and again in Secrary's blog . The original code from Alex Ionescu can be found in the repo.
Then, in main, is called, then the instrumentation is set:
One final mention for this technique is that it can be used to enumerate the the System Service Number (SSN) for a given function call. This was documented by in and , where the hook is significantly smaller (at the cost of doing far less):
Back in 2019 published which had a subsequent release of :
Then provided an which corrected a shortcoming with version 1 and gave us :
The specific implementation in SysWhispers2 is a variation of @modexpblog’s code. One difference is that the function name hashes are randomized on each generation. , who had this technique earlier, has another based in C++17 which is also worth checking out.
The main change is the introduction of which is a result of .
And again, produced :
The usage is pretty similar to , with the following exceptions:
It supports direct jumps to random syscalls (borrowing )
A better explanation of these features are better outlined i the blog post
See for a breakdown on these different techniques.
For our example, we are going to use :
For this example, we are going to use . All we have to do is make this one call in wmain:
As we've stated, this isn't a comprehensive review of every potential technique. Another which might specifically be worth exploring is . Two examples of this are 's post on and 's .
Another component of a process which gets interrogated is via the Call Stack. As detailed in , the Call Stack is defined as:
The call stack is the chain of function calls that have led to the current location of the program counter. The top function on the call stack is the current function, the next function is the function that called the current function, and so on. The call stack that is displayed is based on the current program counter, unless you change the register context. For more information about how to change the register context, see .
If we look at the processes (10792) threads, we can see a bunch of threads starting at the elusive
From the library, we are using:
: Obtains a stack trace.
: Retrieves the base address of the module that contains the specified address.
: Retrieves symbol information for the specified address.
This now shows a different thread stack we haven't seen so far. Looking at frame 3, its which was described as the sleep obfuscation technique in:
As a disclaimer, this technique has full kudos to .
This technique was first popularized by , who is a common reoccurrence in this space, and then reinterpreted by in . However, this proof-of-concept sets the return address to 0, removing references to memory addresses for shellcode injection.
In a more recent project, by was produced to take this a step further and fully mask the stack.
See:
An example of a :
