Maelstrom #5: EDR Kernel Callbacks, Hooks, and Call Stacks
Endpoint Protection and Response is complicated for both offence and defence. In this blog we take a look at Kernel Callbacks, Hooks, and Thread Call Stacks from both perspectives.
To recap the series so far, we've gone from looking at the high level purposes and intentions of a Command and Control Server (C2) in general to designing and implementing our first iteration of our implant and server. If you've been following along, you might think you've written a C2...
This is a common mindset. In our experience, getting to this point does not require much sophistication. All of our previous work could easily be achieved (and has been achieved!) using C#, Python, Go, in an evening's worth of frenetic caffeine-fuelled typing. Leading features of C2s can often be linked to pretty old solved concepts and patterns from software engineering, such as thread management, handling idle processes, and ensuring correction execution and program flow.
But as we found when writing our various C2s, and as numerous other offensive developers have found when writing their own implants and servers, once you have the code working and you can get a pingback, you stop running your implant on your development computer and try it on a second computer. This is where the questions start creeping in. Questions like "Why can't I access remote files?", "Why can I make outbound requests over this protocol, but not this?", "Why does this command just fail with no explanation", and for the cynical self-doubter with enough imposter syndrome "Why isn't Defender stopping me from doing this?".
This, personally, is the post we were looking forward to writing. It's going to be a discussion, with a few examples, of increasing common behaviours within environments with active endpoint protection. In 2022, implants face far more scrutiny - the implant and C2 operator must to be prepared to face or evade this scrutiny, and the defender must be aware of how it works so that it can be used to the best of its ability.
Whilst writing this, we also want to clear up the 'it avoids <insert company here> EDR' tweets. Just because the implant is able to execute, doesn't mean that Endpoint Protection is blind to it - it can mean that, but we want to demonstrate some techniques these solutions use to identify malicious behaviour and raise the suspicion of an implant.
In a nutshell, proof of execution is not proof of evasion.
Reviewing three ways EDRs can detect or block malicious execution:
Kernel Callbacks
Hooking
Thread Call Stacks
By the end of this post, we will have covered how modern EDRs can protect against malicious implants, and how these protections can be bypassed. We will move from having an implant which technically works to an awareness of how to write an implant which actually starts to work, and can achieve the goals of an operator.
As we've said many times, we are not creating an operational C2. The output from this series is poorly written and riddled with flaws - it only does enough to act as a broken proof-of-concept of the specific items we discuss in this series to avoid this code from being used by bad actors. For this same reason, we are trying to avoid discussing Red Team operational tactics in this series. However, as we go on, it will become obvious why blending in with the compromised users typical behaviour will work. This is something that xpn has discussed on Twitter:
Find Confluence, read Confluence.. become the employee!
If your implant has been flagged by EDR, querying NetSessionEnum on every AD-joined computer to find active sessions is probably not typical user behaviour. You likely will not know your implant has been flagged until it stops responding. From here, it's a race until your implant is uploaded to VirusTotal and you have to go back to the drawing board.
We will be referring to the following programs a lot during this blog:
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack.
This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
PreEmpt: A pseudo-EDR which has the capability to digest EtwTi, memory scanners, hooks, and so on. Although, this is not public but code will be shared when necessary.
These two tools will allow us to generate proof-of-concept data when required.
Important Concepts
Similar to Maelstrom: Writing a C2 Implant, we want to have a section dedicated to clearing up some topics we feel need some background before moving on.
What do we mean by Endpoint Detection and Response
Endpoint Detection and Response (EDR) software goes by a number of different acronyms, and there may well be distinctions between different companies programs and their functionality. For the sake of simplicity, we are call all programs that are limited to scanning files on disk statically "anti-virus", and all programs that go further and scan device memory, look at the behaviour of programs while they are running, and responding to threats as they happen "EDR"s. These may be called various names, including XDR, MDR, or just plain AV.
Throughout this series, as we have done so far, we will be sticking with "EDR".
Endpoint Detection and Response (EDR), also referred to as endpoint detection and threat response (EDTR), is an endpoint security solution that continuously monitors end-user devices to detect and respond to cyber threats like ransomware and malware.
Coined by Gartner’s Anton Chuvakin, EDR is defined as a solution that “records and stores endpoint-system-level behaviors, uses various data analytics techniques to detect suspicious system behavior, provides contextual information, blocks malicious activity, and provides remediation suggestions to restore affected systems.”
Because it's relevant to this post, the next section will look at EDR architecture and comparing EDR behaviours across the various vendors. Without going hugely off-topic, we won't look at a number of also relevant areas, such as how Anti-Virus works, how disk-based protection may work to also stymie your implant execution (if you're still running on disk), and how AV and EDR actually goes about scanning files and their behaviours while they are doing so. Turns out, that's like, a whole field of study.
Common EDR Architecture
When discussing endpoint protection, it may help to be somewhat familiar with their architecture. The Symantec EDR Architecture looks something like this:
A similar approach can be seen for Defender for Endpoint. Essentially, a device with the product installed will have an agent which can consist of several drivers and processes, which gather telemetry from various aspects of the machine. Through this post and the next, we will go over a few of those.
As an aside, in a Windows environment, Microsoft inherently have an edge here. While this is currently aimed at "Large Enterprise" customers (or at least, we assume, given their price point for Azure!), Microsoft's Defender and new Defender MDE can both access Microsoft's knowledge of ... their own operating system, but also influence the development of new operating system functionality. Long-term, it wouldn't be a surprise to see Microsoft Defender MDE impact the EDR market in a similar way that Microsoft Defender impacted the anti-virus market.
The general gist of all EDR is that telemetry from the agent is sent to the cloud where it's run through various sandboxes and other test devices, and its behaviour can be further analysed by machine and human operators.
For the excessively curious reader, the following links go in to more detail about specific vendor approaches to EDR architecture:
Briefly Reviewing and Comparing EDR Behaviour at a High Level
Without going hugely off-topic, just as how not every red team assessment is a red team, not every EDR is an EDR.
The following "Gartner Magic Quadrant", from Gartner's May 2021 Report roughly maps out the EDR landscape. It's worth noting that CrowdStrike's hire of Alex Ionescu (a maintainer for the Kernel in ReactOS) demonstrates that the current best-in-class EDR's heavily leverage knowledge of internal Windows functionality to maximise their performance:
With so much of EDR functionality relying on implementing the methods we will discuss here such as custom-written direct behaviours like kernel callbacks and hooking, being able to quickly implement new Microsoft Windows features and develop your own custom ways of reliably interacting with and interrupting malicious processes seems to be the distinguishing feature of a modern EDR from its peers.
Another metric that EDR Vendors tend to use, especially because the reports are made so public, is the Mitre Enginuity. The Attack Evaluations is described as thus:
The MITRE Engenuity ATT&CK® Evaluations (Evals) program brings together product and service providers with MITRE experts to collaborate in evaluating security solutions. The Evals process applies a systematic methodology using a threat-informed purple teaming approach to capture critical context around a solution’s ability to detect or protect against known adversary behavior as defined by the ATT&CK knowledge base. Results from each evaluation are thoroughly documented and openly published.
For example, with SentinelOne, their results can be seen in: SentinelOne Overview. The overview goes through APT scenarios and marks whether or not the technique was detected and can be used as a tracker for its "effectiveness". However, some have expressed feelings online that this is not a thorough way to determine the effectiveness of the product.
When looking at EDRs from a purchasing perspective, there are a few methods of determining effectiveness and we wanted to briefly highlight them here. The main thing to consider is that some vendors do not necessarily provide more functionality than an anti-virus. As with any product, ensure that you purchase the right solution for your businesses needs.
User-land and Kernel-land
When discussing the kernel and user-land model, the following architectural image familiar to any Computer Science graduate will be used:
A big majority of user activity will occur at ring 3, User Mode, surprisingly the Kernel operates within Kernel Mode.
For more information on this, see Windows Programming/User Mode vs Kernel Mode. A worthwhile note is that cross-over between user mode and kernel mode can and does happen. The following definitions from the previous link summarise the differences between these layers:
Ring 0 (also known as kernel mode) has full access to every resource. It is the mode in which the Windows kernel runs.
Rings 1 and 2 can be customized with levels of access but are generally unused unless there are virtual machines running.
Ring 3 (also known as user mode) has restricted access to resources.
Again, to save this post from being longer than it already is, see the Overview of Windows Components documentation for more detail on the following diagram. However, its simply showing the Windows architecture from processes, services, etc, crossing over to the Kernel. We will cover more on this shortly.
Applications that use the WinAPI will traverse through to the Native API (NTAPI) which operates within Kernel Mode.
As an example, API Monitor can be used to look at the calls being executed:
The above shows CreateThread being called and then, subsequently, NtCreateThreadEx being called shortly after.
When a function within KERNEL32.DLL is called, for example CreateThread, it will make a subsequent call to the NTAPI equivalent in NTDLL.DLL. For example, CreateThread calls NtCreateThreadEx. This function will then fill RAX register with the System Service Number (SSN). Finally, NTDLL.dll will then issue a SYSENTER instruction. This will then cause the processor to switch to kernel mode, and jumps to a predefined function, called the System Service Dispatcher. The following image is from Rootkits: Subverting the Windows Kernel, in the section on Userland Hooks:
Drivers
A driver is a software component of Windows which allows the operating system and device to communicate with each other. Here is an example from What is a driver?:
For example, suppose an application needs to read some data from a device. The application calls a function implemented by the operating system, and the operating system calls a function implemented by the driver. The driver, which was written by the same company that designed and manufactured the device, knows how to communicate with the device hardware to get the data. After the driver gets the data from the device, it returns the data to the operating system, which returns it to the application.
In the case of Endpoint Protection, there are a few reasons why drivers are useful:
The use of Callback Objects which allows for a function to be called if an action occurs. For example, later on we will see the usage of PsSetLoadImageNotifyRoutine which is the call-back object for DLLs being loaded.
Access to privileged information from Event Tracing for Windows Threat Intelligence which is only accessible from the Kernel with an ELAM Driver.
Another common feature of EDR's are the Userland Hooking DLLs. Typically, these are loaded into a process on creation, and are used to proxy WinAPI Calls through themselves to assess the usage, then redirect onto whichever DLL is being used. As an example, if VirtualAlloc was being used, the flow would look something like this:
A hook allows for function instrumentation by intercepting WinAPI calls, by placing a jmp instruction in place of the function address. This jmp will redirect the flow of a call. We will take a look at this in action in the following section. By hooking a function call, it gives the author the ability to:
Assess arguments
Allowing Execution
Blocking Execution
This isn't an exhaustive list, but should serve to demonstrate the functionality which we will be coming across most when running our implants.
To access our kernel callbacks without having to write all of that intimidating logic from scratch, we will be using [the] Hunting ELK (HELK):
The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.
The kernel's callback mechanism provides a general way for drivers to request and provide notification when certain conditions are satisfied.
Essentially, they allow drivers to receive and handle notifications for specific events. From veil-ivy/block_create_process.cpp, here is an implementation of using the PsSetLoadImageNotifyRoutine Callback to BLOCK process creation:
In this instance, ObRegisterCallbacks is being used to block the creation of notepad. An Endpoint Protection solution may not use it in this way, but its very likely this type of callback will be used as telemetry to determine if malicious activity is occurring.
When LoadLibraryA is called, the function registers a callback to notify the driver than this has happened. In order to see this log in HELK, we use the script we mentioned earlier on.
If we filter for main.exe , which is the above code, we can see the winhttp.dll loaded:
In Elastic, we can also use the following KQL:
process_name:"main.exe"and event_id:7 and ImageLoaded:winhttp.dll
This is good to get some familiarity with how this would work. However, in Bypassing Image Load Kernel Callbacks, by batsec, identifies that the trigger is in NtCreateSection call which is then called in the LdrpCreateDllSection. So, we don't need to spend too much time debugging to find this.
Spoofing Loads
In the article from batsec, they show that the aforementioned events can be spammed with the following code:
The following screenshot is also from that blog post:
batsec identified that by making the call to NtCreateSection, the event can be spammed whilst not actually loading a DLL. Similarly, the spoof can be somewhat weaponised/manipulated to do other things by updating the LDR_DATA_TABLE_ENTRY struct:
In the above, CertEnroll.dll can be seen loaded in the spoof-load.exe process. Remember, this is not loaded. The only thing that happened here is that a string for that DLL was passed in. We then told the loader than the base address of the DLL is that of the shellcode:
Looking at this technique, there are two obvious use cases:
Tie the implant base address (C2IMPLANT.REFLECTIVE.DLL) to a legitimate DLL (ADVAP32.DLL) causing it to appear less suspicious
Remove an IOC Library (WinHTTP.DLL) by loading ADVAPI32.DLL but pointing it to a WinHTTP.DLL base address.
Bypassing the Callback
We aren't going to reinvent the wheel here, its explained wonderfully in Bypassing Image Load Kernel Callbacks. Essentially, to cause the callback to not trigger, a full loader needs to be rewritten. The conclusion to that research was DarkLoadLibrary:
In essence, DarkLoadLibrary is an implementation of LoadLibrary that will not trigger image load events. It also has a ton of extra features that will make life easier during malware development.
A proof-of-concept usage of this library was taken from DLL Shenanigans.
Let's inspect it:
Then the above 3 commands are ran:
dark-loader uses the LOAD_LOCAL_FILE flag to load a disk from disk, as LoadLibraryA does.
The Image Load logs are searched for Kernel32 to make sure logs were found.
winhttp.dll was searched, and nothing returned
To avoid the call to NtCreateSection which was identified to be registering the callback, the section mapping is done with NtAllocateVirtualMemory or VirtualAlloc, as seen in MapSections().
Kernel Callback Conclusion
Obviously, PsSetLoadImageNotifyRoutine is not the only callback, and there are quite a few other callbacks readily available. Kernel Callback Functions has a (non-comprehensive!) list:
One that would be powerful would be PsSetCreateProcessNotifyRoutineEx() as the notification for process creation would be crippling for system telemetry. At the time of writing, we are not aware of any research in this space. Although to be totally honest, we haven't looked.
Hooking and Process Instrumentation
In this section, we are going to look at some popular, but elementary, hooking techniques.
Hooking Example
Lets look at two examples before looking into some libraries - Manual Hooks in x86 and NtSetProcessInformation Callbacks.
Manual Hooks (x86)
Using Windows API Hooking as a x86 example (easier to demonstrate), we can adapt the code to look something like this:
Borrowing the hooks from Secrary gives access to the function and return value, giving us the following Assembly:
.code
PUBLIC asmCallback
EXTERN Hook:PROC
asmCallback PROC
push rax ; return value
push rcx
push RBX
push RBP
push RDI
push RSI
push RSP
push R12
push R13
push R14
push R15
; without this it crashes :)
sub rsp, 1000h
mov rdx, rax
mov rcx, r10
call Hook
add rsp, 1000h
pop R15
pop R14
pop R13
pop R12
pop RSP
pop RSI
pop RDI
pop RBP
pop RBX
pop rcx
pop rax
jmp R10
asmCallback ENDP
end
Hook: With the assembly written, we also need to write the function called by the assembly, allowing us to take in all of the provided registers and return their function names:
DWORD64 counter =0;bool flag =false;EXTERN_C VOID Hook(DWORD64 R10, DWORD64 RAX/* ... */) {// This flag is there for prevent recursionif (!flag) { flag =true; counter++; CHAR buffer[sizeof(SYMBOL_INFO) + MAX_SYM_NAME] = { 0 }; PSYMBOL_INFO pSymbol = (PSYMBOL_INFO)buffer;pSymbol->SizeOfStruct =sizeof(SYMBOL_INFO);pSymbol->MaxNameLen = MAX_SYM_NAME; DWORD64 Displacement;// MSDN: Retrieves symbol information for the specified address. BOOLEAN result =SymFromAddr(GetCurrentProcess(), R10,&Displacement, pSymbol);if (result) {printf("%s => 0x%llx\n",pSymbol->Name, RAX); } flag =false; }}
Then, in main, SymInitialize is called, then the instrumentation is set:
Running this completed example, we can now see all of the function names and return codes:
The hook could be updated to get access to the arguments for a full analysis, but we didn't feel the need to look into that for this initial proof-of-concept.
One final mention for this technique is that it can be used to enumerate the the System Service Number (SSN) for a given function call. This was documented by Paranoid Ninja in EtwTi-Syscall-Hook and Release v0.8 - Warfare Tactics, where the hook is significantly smaller (at the cost of doing far less):
SysWhispers provides red teamers the ability to generate header/ASM pairs for any system call in the core kernel image (ntoskrnl.exe). The headers will also include the necessary type definitions.
Then modexp provided an update which corrected a shortcoming with version 1 and gave us SysWhispers2:
The specific implementation in SysWhispers2 is a variation of @modexpblog’s code. One difference is that the function name hashes are randomized on each generation. @ElephantSe4l, who had published this technique earlier, has another implementation based in C++17 which is also worth checking out.
The Minimalistic x86/x64 API Hooking Library for Windows
The DLL
So, this is going to be a DLL which gets loaded into a process and then hooks functionality and makes some decision based on its behaviour. Here is DllMain:
When a DLL_PROCESS_ATTACH is the load reason, then we create a new thread and point it at our "main" function. This is where we initialise minhook, and set up some hooks:
DWORD WINAPI SetupHooks(LPVOID param){ MH_STATUS status;if (MH_Initialize() != MH_OK) {return-1; } status =MH_CreateHookApi(L"ntdll","NtAllocateVirtualMemory", NtAllocateVirtualMemory_Hook,reinterpret_cast<LPVOID*>(&pNtAllocateVirtualMemory_Original) ); status =MH_CreateHookApi(L"ntdll","NtProtectVirtualMemory", NtProtectVirtualMemory_Hook,reinterpret_cast<LPVOID*>(&pNtProtectVirtualMemory_Original)