13. Debugging

The best way to debug Aurora is to run it directly in the command line and don't use it as service.

C:\Program Files\Aurora Agent\>aurora-agent-64.exe --rules-path .\my-rules --debug

The verbosity can even be increased by using the --trace flag. With --trace, a log entry will be generated for every incoming event.

--debug and --trace apply to all outputs (log file, UDP / TCP, command line) except for the Windows Eventlog,

13.1. Status Information

The --status can give you information from a running Aurora service.

 1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status
 2Aurora Agent
 3Version: 0.1.6
 4Build Revision: 5fef68a1
 5Sigma Revision: 0.20-1884-ga4a26540
 6Status: running
 7Uptime (in hours): 0
 8
 9Active Outputs:
10Eventlog: enabled
11Stdout: enabled
12
13Rule Statistics:
14Loaded rules: 734
15Number of rule reloads: 0
16
17Event Statistics:
18Events observed so far: 89419
19Events lost so far: 0
20Sigma matches: 4
21Suppressed Sigma matches of those: 0
22
23Response Actions: disabled

It displays the number of events that the agent was able to see and process, the number of initialized rules and rule matches.

Adding the flag --trace includes more information in the output, e.g. the number of processed events per ETW event channel.

 1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status --trace
 2Aurora Agent
 3Version: 0.1.6
 4Build Revision: 5fef68a1
 5Sigma Revision: 0.20-1884-ga4a26540
 6Status: running
 7Uptime (in hours): 0
 8
 9Active Outputs:
10Eventlog: enabled
11Stdout: enabled
12
13Rule Statistics:
14Loaded rules: 734
15Number of rule reloads: 0
16
17Event Statistics:
18Events observed so far: 111138
19        88650 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
20        20094 events from WinEventLog:Microsoft-Antimalware-Engine
21        1265 events from PollNamedPipes
22        400 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
23        306 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
24        174 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
25        164 events from SystemLogger:Process
26        46 events from WinEventLog:Microsoft-Windows-DNS-Client
27        19 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
28        11 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
29        9 events from WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath
30Events lost so far: 0
31Sigma matches: 4
32        Run Whoami Showing Privileges: 2
33        Suspicious Certutil Command: 2
34Suppressed Sigma matches of those: 0
35
36Response Actions: disabled

The JSON output includes all the information plus the agent configuration settings.

 1C:\Program Files\Aurora Agent>aurora-agent-64.exe --status --json
 2{
 3    "Parameters": {
 4        "SigmaFolders": [
 5            "C:\\aurora\\rules"
 6        ],
 7        "AutoReload": false,
 8        "LogFile": "",
 9        "LogSources": [
10            "log-sources\\etw-log-sources-standard.yml"
11        ],
12        "Debug": false,
13        "Trace": false,
14        "NoEventlog": false,
15        "ReportingLevel": "high",
16        "Json": false,
17        "LicensePath": "C:\\aurora\\",
18        "UdpTarget": "",
19        "Silent": false,
20        "CpuLimit": 100,
21        "ReportStats": false,
22        "ReportStatsInterval": 3600000000000,
23        "LogRotateCount": 7,
24        "LogSize": 10485760,
25        "AgentName": "aurora-agent",
26        "ActivateModules": null,
27        "DeactivateModules": null,
28        "NoStdout": false,
29        "EventThrottling": 0,
30        "LowPrio": false,
31        "PrintEventId": false,
32        "ConsumerParameters": {
33            "ActivateResponses": false,
34            "DumpFolder": ".",
35            "SigmaMatchThrottling": 60000000000,
36            "SigmaMatchBurst": 5
37        },
38        "ProviderParameters": {
39            "NoHashes": false
40        }
41    },
42    "Uptime": 334601989600,
43    "Version": "0.1.6",
44    "SigmaRevision": "0.20-1884-ga4a26540",
45    "BuildRevision": "5fef68a1",
46    "CurrentAction": "running",
47    "LoadedRules": 734,
48    "ReloadCounter": 0,
49    "EventsProcessed": {
50        "PollNamedPipes": 1815,
51        "SystemLogger:Process": 175,
52        "WinEventLog:Microsoft-Antimalware-Engine": 27847,
53        "WinEventLog:Microsoft-Windows-DNS-Client": 57,
54        "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls": 124269,
55        "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE": 22,
56        "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH": 11,
57        "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE": 645,
58        "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS": 314,
59        "WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey": 342,
60        "WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath": 26
61    },
62    "EventsLost": 0,
63    "SigmaMatches": {
64        "Run Whoami Showing Privileges": 2,
65        "Suspicious Certutil Command": 2
66    },
67    "SuppressedSigmaMatches": {},
68    "ActiveModules": null
69}

13.2. Diagnostic information

13.2.1. Diagnostic pack

You can create a diagnostic pack to detect and debug performance problems.

Simply run:

C:\Program Files\Aurora Agent>aurora-agent-util.exe diagnostics

This creates a ZIP file with debugging information (such as heap usage, stack traces, ...) that we can use to analyze these issues.

13.2.2. Profiling server

If Aurora has been started with --pprof, information can also be gathered manually via a web interface:

curl.exe http://localhost:8080/debug/pprof/profile?seconds=20 --output aurora-debug.pprof
curl.exe http://localhost:8080/debug/pprof/heap --output aurora-heap.pprof
curl.exe http://localhost:8080/debug/pprof/goroutine --output aurora-stack-traces.pprof

This is the same information that is included in the diagnostic pack.

13.3. Crashes

In cases of unexpected crashes, the following command lines can help you identify the source of the problem.

C:\Program Files\Aurora Agent>aurora-agent.exe -c agent-config.yml > aurora-crash.log 2>&1

C:\Program Files\Aurora Agent>aurora-agent.exe -c agent-config.yml --trace > aurora-crash-trace.log 2>&1

13.4. Error Messages

Check the configured log outputs for error messages. A faulty rule would e.g. lead to error messages like this one in the Application eventlog with EventID

Could not reload sigma rules
Module: Aurora-Agent
Changed_files: C:\Program Files\Aurora-Agent\myrules\my-ransomware.yml
Error: could not parse rule response in file "C:\\Program Files\\Aurora-Agent\\myrules\\my-ransomware.yml": invalid predefined response action kil

13.5. Performance Tuning

13.5.1. Event Source Tuning

13.5.1.1. Event Sources and Consumers

Internally, Aurora has a number of event consumers. The event consumers are:

  • Aurora's built-in modules

  • Sigma log sources

Each event consumer consists of:

  • A number of requested event sources

  • Logic to handle incoming events from these sources

Performance is primarily determined by the number of incoming events that Aurora has to process; The impact of Sigma rule matching, in comparison, is fairly low.

Therefore, to optimize performance, choose your event sources wisely and avoid event sources that produce an extreme number of events.

13.5.1.2. Event Source Analysis

When executing aurora-agent.exe --status --trace while Aurora is running, an overview of events that was received for each event source is generated. The performance impact of each source scales roughly linear with the number of events.

To see which built-in modules requests which event source, the requested log sources can be listed with aurora-agent.exe --module-info --trace.

For sigma log sources, inspect the sigma configurations that are used by Aurora. By default, etw-log-sources.yml and default-log-sources.yml from the Aurora directory are used.

Each sigma log source in these files that has a sources element requests the event sources listed in that element.

13.5.1.3. Event Source Definitions

Aurora Agent supports the following event source prefixes:

  • WinEventLog: Events from an Eventlog channel or ETW provider.

    The schema for these sources is: WinEventLog:Provider/Channel?Options

    Channels and options are optional and add further restrictions on events from the provider that is requested. A full list of Eventlog channels on a system can be found using the Event Viewer. A full list of ETW providers on a system can be found using e.g. https://github.com/zodiacon/EtwExplorer.

  • SystemLogger: Events from the System Trace Provider (see https://docs.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-a-systemtraceprovider-session for details).

    The schema is: SystemLogger:SystemLoggerFlag where the supported SystemLoggerFlag flags are:

    • FileIO

    • Process

    • Thread

    • Registry

    • Image

    • Network-TCP-IP

    • Handle

  • PollHandles: This event source is handled by a provider in Aurora that regularly creates an event for each handle that exists on a system.

13.5.1.4. Example: Disabling a Noisy Log Source

In this example, say that aurora-agent.exe --status --trace results in this event overview:

Events observed so far: 50657
     36783 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
     7611 events from WinEventLog:Microsoft-Windows-Kernel-File?eventids=14
     1842 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
     1273 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
     1058 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_THREAD
     995 events from WinEventLog:Microsoft-Windows-DNS-Client
     585 events from PollNamedPipes
     235 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_FILENAME
     169 events from SystemLogger:Process
     39 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
     32 events from WinEventLog:Microsoft-Windows-Sysmon/Operational
     22 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
     5 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
     3 events from WinEventLog:Security
     3 events from WinEventLog:Application
     2 events from WinEventLog:Microsoft-Windows-Kernel-Registry/DeleteKey

As we can see, the by far noisiest event source is WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls.

If we want to disable this event source to lessen Aurora's CPU usage, we must find the event consumers that request it.

aurora-agent.exe --module-info --trace shows these modules which use this event source:

Aurora Agent Modules:
     LsassDumpCheck
             Requested sources:
             ...
             WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
     Beaconhunter
             Requested sources:
             ...
             WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
             ...

Searching in etw-log-sources.yml, we find that there is also a Sigma log source definition which uses this event source:

windows-api-call-auditing:
   product: windows
   service: api-call-auditing
   sources:
      - "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls"

To deactivate this log source, we therefore need to deactivate both modules which use this source using --deactivate-module and remove the log source definition from the sigma configuration.

Obviously, this will also impact Aurora's detection capabilities to some degree. Choose your trade-off between detection and performance carefully.

13.5.2. Process Exclusions

To exclude specific processes from analysis, you can configure Aurora to ignore all events from specific image paths.

In order to do so, the excluded images must be specified (as regexps) in a file that is passed to --process-excludes. By default, config\process-excludes.cfg is used. This file contains further examples on how to specify the excludes.

The process exclusion file is loaded at startup. If you change the file, you'll need to restart Aurora to apply those changes.

Please be aware that adding process exclusions can cause malware that uses process hollowing or similar techniques to mask themselves as an excluded process to go unreported.

13.5.2.1. Examples

# Exclude a specific process
^C:\\Program Files\\My Antivirus\\antivirus\.exe$

# Exclude Windows Defender
^C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\[^\\]{5,20}\\MsMpEng\.exe$