14. Debugging
The best way to debug Aurora is to run it directly in the command line and don't use it as service.
C:\Program Files\Aurora Agent\>aurora-agent-64.exe --rules-path .\my-rules --debug
The verbosity can even be increased by using the --trace
flag. With --trace
,
a log entry will be generated for every incoming event.
--debug
and --trace
apply to all outputs (log file, UDP / TCP, command line) except for the Windows Eventlog,
14.1. Status Information
The --status
can give you information from a running Aurora service.
1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status
2Aurora Agent
3Version: 0.1.6
4Build Revision: 5fef68a1
5Sigma Revision: 0.20-1884-ga4a26540
6Status: running
7Uptime (in hours): 0
8
9Active Outputs:
10Eventlog: enabled
11Stdout: enabled
12
13Rule Statistics:
14Loaded rules: 734
15Number of rule reloads: 0
16
17Event Statistics:
18Events observed so far: 89419
19Events lost so far: 0
20Sigma matches: 4
21Suppressed Sigma matches of those: 0
22
23Response Actions: disabled
It displays the number of events that the agent was able to see and process, the number of initialized rules and rule matches.
Adding the flag --trace
includes more information in the output, e.g. the number of processed events per ETW event channel.
1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status --trace
2Aurora Agent
3Version: 0.1.6
4Build Revision: 5fef68a1
5Sigma Revision: 0.20-1884-ga4a26540
6Status: running
7Uptime (in hours): 0
8
9Active Outputs:
10Eventlog: enabled
11Stdout: enabled
12
13Rule Statistics:
14Loaded rules: 734
15Number of rule reloads: 0
16
17Event Statistics:
18Events observed so far: 111138
19 88650 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
20 20094 events from WinEventLog:Microsoft-Antimalware-Engine
21 1265 events from PollNamedPipes
22 400 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
23 306 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
24 174 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
25 164 events from SystemLogger:Process
26 46 events from WinEventLog:Microsoft-Windows-DNS-Client
27 19 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
28 11 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
29 9 events from WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath
30Events lost so far: 0
31Sigma matches: 4
32 Run Whoami Showing Privileges: 2
33 Suspicious Certutil Command: 2
34Suppressed Sigma matches of those: 0
35
36Response Actions: disabled
The JSON output includes all the information plus the agent configuration settings.
1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status --json
2{
3 "Parameters": {
4 "SigmaFolders": [
5 "C:\\aurora\\rules"
6 ],
7 "AutoReload": false,
8 "LogFile": "",
9 "LogSources": [
10 "log-sources\\etw-log-sources-standard.yml"
11 ],
12 "Debug": false,
13 "Trace": false,
14 "NoEventlog": false,
15 "ReportingLevel": "high",
16 "Json": false,
17 "LicensePath": "C:\\aurora\\",
18 "UdpTarget": "",
19 "Silent": false,
20 "CpuLimit": 100,
21 "ReportStats": false,
22 "ReportStatsInterval": 3600000000000,
23 "LogRotateCount": 7,
24 "LogSize": 10485760,
25 "AgentName": "aurora-agent",
26 "ActivateModules": null,
27 "DeactivateModules": null,
28 "NoStdout": false,
29 "EventThrottling": 0,
30 "LowPrio": false,
31 "PrintEventId": false,
32 "ConsumerParameters": {
33 "ActivateResponses": false,
34 "DumpFolder": ".",
35 "SigmaMatchThrottling": 60000000000,
36 "SigmaMatchBurst": 5
37 },
38 "ProviderParameters": {
39 "NoHashes": false
40 }
41 },
42 "Uptime": 334601989600,
43 "Version": "0.1.6",
44 "SigmaRevision": "0.20-1884-ga4a26540",
45 "BuildRevision": "5fef68a1",
46 "CurrentAction": "running",
47 "LoadedRules": 734,
48 "ReloadCounter": 0,
49 "EventsProcessed": {
50 "PollNamedPipes": 1815,
51 "SystemLogger:Process": 175,
52 "WinEventLog:Microsoft-Antimalware-Engine": 27847,
53 "WinEventLog:Microsoft-Windows-DNS-Client": 57,
54 "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls": 124269,
55 "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE": 22,
56 "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH": 11,
57 "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE": 645,
58 "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS": 314,
59 "WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey": 342,
60 "WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath": 26
61 },
62 "EventsLost": 0,
63 "SigmaMatches": {
64 "Run Whoami Showing Privileges": 2,
65 "Suspicious Certutil Command": 2
66 },
67 "SuppressedSigmaMatches": {},
68 "ActiveModules": null
69}
14.2. Diagnostic information
14.2.1. Diagnostic pack
You can create a diagnostic pack to detect and debug performance problems.
Simply run:
C:\Program Files\Aurora Agent>aurora-agent-util.exe diagnostics
This creates a ZIP file with debugging information (such as heap usage, stack traces, ...) that we can use to analyze these issues.
14.2.2. Profiling server
If Aurora has been started with --pprof
, information can also be gathered manually via a web interface:
curl.exe http://localhost:8080/debug/pprof/profile?seconds=20 --output aurora-debug.pprof
curl.exe http://localhost:8080/debug/pprof/heap --output aurora-heap.pprof
curl.exe http://localhost:8080/debug/pprof/goroutine --output aurora-stack-traces.pprof
This is the same information that is included in the diagnostic pack.
14.3. Crashes
In cases of unexpected crashes, the following command lines can help you identify the source of the problem.
C:\Program Files\Aurora Agent>aurora-agent.exe -c agent-config.yml > aurora-crash.log 2>&1
C:\Program Files\Aurora Agent>aurora-agent.exe -c agent-config.yml --trace > aurora-crash-trace.log 2>&1
14.4. Error Messages
Check the configured log outputs for error messages. A faulty rule would e.g.
lead to error messages like this one in the Application
eventlog with EventID
Could not reload sigma rules
Module: Aurora-Agent
Changed_files: C:\Program Files\Aurora-Agent\myrules\my-ransomware.yml
Error: could not parse rule response in file "C:\\Program Files\\Aurora-Agent\\myrules\\my-ransomware.yml": invalid predefined response action kil
14.5. Performance Tuning
14.5.1. Event Source Tuning
14.5.1.1. Event Sources and Consumers
Internally, Aurora has a number of event consumers. The event consumers are:
Aurora's built-in modules
Sigma log sources
Each event consumer consists of:
A number of requested event sources
Logic to handle incoming events from these sources
Performance is primarily determined by the number of incoming events that Aurora has to process; The impact of Sigma rule matching, in comparison, is fairly low.
Therefore, to optimize performance, choose your event sources wisely and avoid event sources that produce an extreme number of events.
14.5.1.2. Event Source Analysis
When executing aurora-agent.exe --status --trace
while Aurora is running, an overview
of events that was received for each event source is generated. The performance impact of each source
scales roughly linear with the number of events.
To see which built-in modules requests which event source, the requested log sources can be listed with
aurora-agent.exe --module-info --trace
.
For sigma log sources, inspect the sigma configurations that are used by Aurora.
By default, etw-log-sources.yml
and default-log-sources.yml
from the Aurora directory are used.
Each sigma log source in these files that has a sources
element requests the event sources listed
in that element.
14.5.1.3. Event Source Definitions
Aurora Agent supports the following event source prefixes:
WinEventLog:
Events from an Eventlog channel or ETW provider.The schema for these sources is:
WinEventLog:Provider/Channel?Options
Channels and options are optional and add further restrictions on events from the provider that is requested. A full list of Eventlog channels on a system can be found using the Event Viewer. A full list of ETW providers on a system can be found using e.g. https://github.com/zodiacon/EtwExplorer.
SystemLogger:
Events from the System Trace Provider (see https://docs.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-a-systemtraceprovider-session for details).The schema is:
SystemLogger:SystemLoggerFlag
where the supportedSystemLoggerFlag
flags are:FileIO
Process
Thread
Registry
Image
Network-TCP-IP
Handle
PollHandles
: This event source is handled by a provider in Aurora that regularly creates an event for each handle that exists on a system.
14.5.1.4. Example: Disabling a Noisy Log Source
In this example, say that aurora-agent.exe --status --trace
results in this event overview:
Events observed so far: 50657
36783 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
7611 events from WinEventLog:Microsoft-Windows-Kernel-File?eventids=14
1842 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
1273 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
1058 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_THREAD
995 events from WinEventLog:Microsoft-Windows-DNS-Client
585 events from PollNamedPipes
235 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_FILENAME
169 events from SystemLogger:Process
39 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
32 events from WinEventLog:Microsoft-Windows-Sysmon/Operational
22 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
5 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
3 events from WinEventLog:Security
3 events from WinEventLog:Application
2 events from WinEventLog:Microsoft-Windows-Kernel-Registry/DeleteKey
As we can see, the by far noisiest event source is WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
.
If we want to disable this event source to lessen Aurora's CPU usage, we must find the event consumers that request it.
aurora-agent.exe --module-info --trace
shows these modules which use this event source:
Aurora Agent Modules:
LsassDumpCheck
Requested sources:
...
WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
Beaconhunter
Requested sources:
...
WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
...
Searching in etw-log-sources.yml
, we find that there is also a Sigma log source definition which uses this event source:
windows-api-call-auditing:
product: windows
service: api-call-auditing
sources:
- "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls"
To deactivate this log source, we therefore need to deactivate both modules which use this source using --deactivate-module and remove the log source definition from the sigma configuration.
Obviously, this will also impact Aurora's detection capabilities to some degree. Choose your trade-off between detection and performance carefully.
14.5.1.5. Examples
# Exclude a specific process
^C:\\Program Files\\My Antivirus\\antivirus\.exe$
# Exclude Windows Defender
^C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\[^\\]{5,20}\\MsMpEng\.exe$