12. Debugging

The best way to debug Aurora is to run it directly in the command line and don't use it as service.

aurora-agent-64.exe --rules-path .\my-rules --debug

The verbosity can even be increased by using the --trace flag.

12.1. Status Information

The --status can give you information from a running Aurora service.

C:\aurora>aurora-agent-64.exe --status
Aurora Agent
Version: 0.1.6
Build Revision: 5fef68a1
Sigma Revision: 0.20-1884-ga4a26540
Status: running
Uptime (in hours): 0

Active Outputs:
Eventlog: enabled
Stdout: enabled

Rule Statistics:
Loaded rules: 734
Number of rule reloads: 0

Event Statistics:
Events observed so far: 89419
Events lost so far: 0
Sigma matches: 4
Suppressed Sigma matches of those: 0

Response Actions: disabled

It displays the number of events that the agent was able to see and process, the number of initialized rules and rule matches.

Adding the flag --trace includes more information in the output, e.g. the number of processed events per ETW event channel.

C:\aurora\>aurora-agent-64.exe --status --trace
Aurora Agent
Version: 0.1.6
Build Revision: 5fef68a1
Sigma Revision: 0.20-1884-ga4a26540
Status: running
Uptime (in hours): 0

Active Outputs:
Eventlog: enabled
Stdout: enabled

Rule Statistics:
Loaded rules: 734
Number of rule reloads: 0

Event Statistics:
Events observed so far: 111138
        88650 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
        20094 events from WinEventLog:Microsoft-Antimalware-Engine
        1265 events from PollNamedPipes
        400 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
        306 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
        174 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
        164 events from SystemLogger:Process
        46 events from WinEventLog:Microsoft-Windows-DNS-Client
        19 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
        11 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
        9 events from WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath
Events lost so far: 0
Sigma matches: 4
        Run Whoami Showing Privileges: 2
        Suspicious Certutil Command: 2
Suppressed Sigma matches of those: 0

Response Actions: disabled

The JSON output includes all the information plus the agent configuration settings.

C:\aurora\>aurora-agent-64.exe --status --json
{
        "Parameters": {
                "SigmaFolders": [
                        "C:\\aurora\\rules"
                ],
                "AutoReload": false,
                "LogFile": "",
                "LogSources": [
                        "log-sources\\etw-log-sources-standard.yml"
                ],
                "Debug": false,
                "Trace": false,
                "NoEventlog": false,
                "ReportingLevel": "high",
                "Json": false,
                "LicensePath": "C:\\aurora\\",
                "UdpTarget": "",
                "Silent": false,
                "CpuLimit": 100,
                "ReportStats": false,
                "ReportStatsInterval": 3600000000000,
                "LogRotateCount": 7,
                "LogSize": 10485760,
                "AgentName": "aurora-agent",
                "ActivateModules": null,
                "DeactivateModules": null,
                "NoStdout": false,
                "EventThrottling": 0,
                "LowPrio": false,
                "PrintEventId": false,
                "ConsumerParameters": {
                        "ActivateResponses": false,
                        "DumpFolder": ".",
                        "SigmaMatchThrottling": 60000000000,
                        "SigmaMatchBurst": 5
                },
                "ProviderParameters": {
                        "NoHashes": false
                }
        },
        "Uptime": 334601989600,
        "Version": "0.1.6",
        "SigmaRevision": "0.20-1884-ga4a26540",
        "BuildRevision": "5fef68a1",
        "CurrentAction": "running",
        "LoadedRules": 734,
        "ReloadCounter": 0,
        "EventsProcessed": {
                "PollNamedPipes": 1815,
                "SystemLogger:Process": 175,
                "WinEventLog:Microsoft-Antimalware-Engine": 27847,
                "WinEventLog:Microsoft-Windows-DNS-Client": 57,
                "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls": 124269,
                "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE": 22,
                "WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH": 11,
                "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE": 645,
                "WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS": 314,
                "WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey": 342,
                "WinEventLog:Microsoft-Windows-TCPIP/ut:ConnectPath": 26
        },
        "EventsLost": 0,
        "SigmaMatches": {
                "Run Whoami Showing Privileges": 2,
                "Suspicious Certutil Command": 2
        },
        "SuppressedSigmaMatches": {},
        "ActiveModules": null
}

12.2. Diagnostic information

12.2.1. Diagnostic pack

You can create a diagnostic pack to detect and debug performance problems.

Simply run:

.\aurora-agent-util.exe diagnostics

This creates a ZIP file with debugging information (such as heap usage, stack traces, ...) that we can use to analyze these issues.

12.2.2. Profiling server

If Aurora has been started with --pprof, information can also be gathered manually via a web interface:

curl.exe http://localhost:8080/debug/pprof/profile?seconds=20 --output aurora-debug.pprof
curl.exe http://localhost:8080/debug/pprof/heap --output aurora-heap.pprof
curl.exe http://localhost:8080/debug/pprof/goroutine --output aurora-stack-traces.pprof

This is the same information that is included in the diagnostic pack.

12.3. Crashes

In cases of unexpected crashes, the following command lines can help you identify the source of the problem.

C:\Program Files\Aurora Agent\>aurora-agent.exe -c agent-config.yml > aurora-crash.log 2>&1

C:\Program Files\Aurora Agent\>aurora-agent.exe -c agent-config.yml --trace > aurora-crash-trace.log 2>&1

12.4. Error Messages

Check the configured log outputs for error messages. A faulty rule would e.g. lead to error messages like this one in the Application eventlog with EventID

Could not reload sigma rules
Module: Aurora-Agent
Changed_files: C:\Program Files\Aurora-Agent\myrules\my-ransomware.yml
Error: could not parse rule response in file "C:\\Program Files\\Aurora-Agent\\myrules\\my-ransomware.yml": invalid predefined response action kil

12.5. Performance Tuning

12.5.1. Event Source Tuning

12.5.1.1. Event Sources and Consumers

Internally, Aurora has a number of event consumers. The event consumers are:

  • Aurora's built-in modules

  • Sigma log sources

Each event consumer consists of:

  • A number of requested event sources

  • Logic to handle incoming events from these sources

Performance is primarily determined by the number of incoming events that Aurora has to process; The impact of Sigma rule matching, in comparison, is fairly low.

Therefore, to optimize performance, choose your event sources wisely and avoid event sources that produce an extreme number of events.

12.5.1.2. Event Source Analysis

When executing aurora-agent.exe --status --trace while Aurora is running, an overview of events that was received for each event source is generated. The performance impact of each source scales roughly linear with the number of events.

To see which built-in modules requests which event source, the requested log sources can be listed with aurora-agent.exe --module-info --trace.

For sigma log sources, inspect the sigma configurations that are used by Aurora. By default, etw-log-sources.yml and default-log-sources.yml from the Aurora directory are used.

Each sigma log source in these files that has a sources element requests the event sources listed in that element.

12.5.1.3. Event Source Definitions

Aurora Agent supports the following event source prefixes:

  • WinEventLog: Events from an Eventlog channel or ETW provider.

    The schema for these sources is: WinEventLog:Provider/Channel?Options

    Channels and options are optional and add further restrictions on events from the provider that is requested. A full list of Eventlog channels on a system can be found using the Event Viewer. A full list of ETW providers on a system can be found using e.g. https://github.com/zodiacon/EtwExplorer.

  • SystemLogger: Events from the System Trace Provider (see https://docs.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-a-systemtraceprovider-session for details).

    The schema is: SystemLogger:SystemLoggerFlag where the supported SystemLoggerFlag flags are:

    • FileIO

    • Process

    • Thread

    • Registry

    • Image

    • Network-TCP-IP

    • Handle

  • PollHandles: This event source is handled by a provider in Aurora that regularly creates an event for each handle that exists on a system.

12.5.1.4. Example: Disabling a Noisy Log Source

In this example, say that aurora-agent.exe --status --trace results in this event overview:

Events observed so far: 50657
     36783 events from WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
     7611 events from WinEventLog:Microsoft-Windows-Kernel-File?eventids=14
     1842 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_IMAGE
     1273 events from WinEventLog:Microsoft-Windows-Kernel-Registry/CreateKey
     1058 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_THREAD
     995 events from WinEventLog:Microsoft-Windows-DNS-Client
     585 events from PollNamedPipes
     235 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_FILENAME
     169 events from SystemLogger:Process
     39 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_CREATE_NEW_FILE
     32 events from WinEventLog:Microsoft-Windows-Sysmon/Operational
     22 events from WinEventLog:Microsoft-Windows-Kernel-Process/WINEVENT_KEYWORD_PROCESS
     5 events from WinEventLog:Microsoft-Windows-Kernel-File/KERNEL_FILE_KEYWORD_DELETE_PATH
     3 events from WinEventLog:Security
     3 events from WinEventLog:Application
     2 events from WinEventLog:Microsoft-Windows-Kernel-Registry/DeleteKey

As we can see, the by far noisiest event source is WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls.

If we want to disable this event source to lessen Aurora's CPU usage, we must find the event consumers that request it.

aurora-agent.exe --module-info --trace shows these modules which use this event source:

Aurora Agent Modules:
     LsassDumpCheck
             Requested sources:
             ...
             WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
     Beaconhunter
             Requested sources:
             ...
             WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls
             ...

Searching in etw-log-sources.yml, we find that there is also a Sigma log source definition which uses this event source:

windows-api-call-auditing:
   product: windows
   service: api-call-auditing
   sources:
      - "WinEventLog:Microsoft-Windows-Kernel-Audit-API-Calls"

To deactivate this log source, we therefore need to deactivate both modules which use this source using --deactivate-module and remove the log source definition from the sigma configuration.

Obviously, this will also impact Aurora's detection capabilities to some degree. Choose your trade-off between detection and performance carefully.

12.5.2. Process Exclusions

To exclude specific processes from analysis, you can configure Aurora to ignore all events from specific image paths.

In order to do so, the excluded images must be specified (as regexps) in a file that is passed to --process-excludes. By default, config\process-excludes.cfg is used. This file contains further examples on how to specify the excludes.

Please be aware that adding process exclusions can cause malware that uses process hollowing or similar techniques to mask themselves as an excluded process to go unreported.

12.5.2.1. Examples

# Exclude a specific process
^C:\\Program Files\\My Antivirus\\antivirus\.exe$

# Exclude Windows Defender
^C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\[^\\]{5,20}\\MsMpEng\.exe$