SysFlow Specification

SysFlow is an open specification for system event-level telemetry. The main goal of SysFlow is to create a standard and extensible data format for both security and performance analytics for compute workloads. An open standard will enable researchers and practitioners to more easily work on a common data format, and focus on analytics using open source software.

The primary objective of SysFlow is to lift raw system call data into more semantic process behaviors which promote significant data reductions for longer term forensic storage of data which is crucial for security analyzes. Through an object relational model of entities, events and flows, we enable SysFlow users to configure the desired granularity of data collection and filtering in order to facilitate most types of analysis in big data frameworks.

Overview

SysFlow is an object relational model of entities, events and flows that describe the behaviors of processes on a system, and encode them into an open format. A SysFlow exporter is designed to monitor system events of a workload, convert them to SysFlow objects, and output them in a binary output file. We envision that one exporter will be deployed per host (or Virtual Machine) and will output one binary file over a particular time period. Figure 1 show a detailed view of the objects that the SysFlow exporter will export.

Entities represent the components on a system that we are interested in monitoring. In this version of SysFlow, we support three types of entities: Containers, Processes, and Files. As shown in Figure 1, Containers contain both Processes and Files, and the three are linked through object identifiers (more on this later).

Entity behaviors are modeled as events or flows. Events represent important individual behaviors of an entity that are broken out on their own due to their importance, their rarity, or because maintaining operation order is important. An example of an event would be a process clone or exec, or the deletion or renaming of a file. By contrast, a Flow represents an aggregation of multiple events that naturally fit together to describe a particular behavior. For example, we can model the network interactions of a process and a remote host as a bidirectional flow that is composed of several events, including connect, read, write, and close.

The idea behind SysFlow is to enable the user to configure the granularity of system-level data desired based on resource limitations and data analytics requirements. In this way, behaviors can be broken out into individual events or combined into smaller aggregated volumetric flows. The current version of the specification describes events and flows in three key behavioral areas: Files, Networks, and Processes. Figure 1 shows these events and flows with their attributes and relationships to entities, which are described in greater details in the following sections.

SF_Object_View.png

Figure 1: SysFlow Object Relational View

Entities

As mentioned above, entities are the components on a system that we are interested in monitoring. These include containers, processes, and files. We also support a special entity object called a Header, which stores information about the SysFlow version, and a unique ID representing the host or virtual machine monitored by the SysFlow exporter. The header is always the first record appearing in a SysFlow File. All other entities contain a timestamp, an object ID and a state. The timestamp is used to indicate the time at which the entity was exported to the SysFlow file.

Object ID

Object IDs allow events and flows to reference entities without having duplicate information stored in each record. Object IDs are not required to be globally unique across space and time. In fact, the only requirement for uniqueness is that no two objects managed by a SysFlow exporter can have the same ID simultaneously. Entities are always written to the binary output file before any events, and flows associated with them are exported. Since entities are exported first, each event, and flow is matched with the entity (with the same id) that is closest to it in the file. Furthermore, every binary output file must be self-contained, meaning that all entities referenced by flows/events must be present in every SysFlow file generated.

State

The state is an enumeration that indicates why an entity was written to disk. The state can currently be one of three values:

State Description
CREATED Indicates that the entity was recently created on the host/VM. For example, a process clone.
MODIFIED Indicates that some attributes of the entity were modified since the last time it was exported.
REUP Indicates that the entity already existed, but is being exported again, so that output files can be self-contained.

Each entity is defined below with recommendations on what to use for object identifiers, based on what is used in the current implementation of the SysFlow exporter.

Header

The Header entity is an object which appears at the beginning of each binary SysFlow file. It contains the current version of SysFlow as supported in the file, and the exporter ID.

Attribute Type Description Since (schema version)
version long The current SysFlow version. 1
exporter string Globally unique id representing the host monitored by SysFlow. 1
ip string IP address in dot notation representing the monitored host. 2

Container

The Container entity represents a system or application container such as docker or LXC. It contains important information about the container including its id, name, and whether it is privileged.

Attribute Type Description Since (schema version)
id string Unique string representing the Container Object as provided by docker, LXC, etc. 1
state enum state of the process (CREATED, MODIFIED, REUP). not implemented
timestamp (ts) int64 The timestamp when container object is exported (nanoseconds). not implemented
name string Container name as provided by docker, LXC, etc. 1
image string Image name associated with container as provided by docker, LXC, etc. 1
imageID string Image ID associated with container as provided by docker, LXC, etc. 1
type enum Can be one of: CT_DOCKER, CT_LXC, CT_LIBVIRT_LXC, CT_MESOS, CT_RKT, CT_CUSTOM 1
privileged boolean If true, the container is running with root privileges 1

Process

The process entity represents a running process on the system. It contains important information about the process including its host pid, creation time, oid id, as well as references to its parent id. When a process entity is exported to a SysFlow file, all its parent processes should be exported before the process, as well as the process’s Container entity. Processes are only exported to a SysFlow file if an event or flow associated with that process or any of its threads are exported. Threads are not explicitly exported in the process object but are represented in events and flows through a thread id field. Finally, a Process entity only needs to be exported to a file once, unless it’s been modified by an event or flow.

NOTE: In current implementation, the creation timestamp is the time at which the process is cloned. If the process was cloned before capture was started, this value is 0. The current implementation also has problems getting absolute paths for exes when relative paths are used to launch processes.
Attribute Type Description Since (schema version)
state enum state of the process (CREATED, MODIFIED, REUP) 1
OID:
host pid
create ts
struct
int64
int64
The Process OID contains the host pid of the project, and creation timestamp. 1
POID:
parent host pid
parent create ts
struct
int64
int64
The OID of the parent process can be NULL if not available or if a root process. 1
timestamp (ts) int64 The timestamp when process object is exported (nanoseconds). 1
exe string Full path (if available) of the executable used in the process launch; otherwise, it’s the name of the exe. 1
exeArgs string Concatenated list of args passed on process startup. 1
uid int32 User ID under which the process is running. 1
userName string User name under which the process is running. 1
gid int32 Group ID under which the process is running 1
groupName string Group Name under which the process is running 1
tty boolean If true, the process is tied to a shell 1
containerId string Unique string representing the Container Object to which the process resides. It can be NULL if process isn’t in a container. 1
entry boolean If true, the process is a container or system entrypoint (i.e., virtual pid = 1). 2

File

The File entity represents file-based resources on a system including files, directories, unix sockets, and pipes.

NOTE: Current implementation does not have access to inode related values, which would greatly improve object ids. Also, the current implementation has some issues with absolute paths when monitoring operations that use relative paths.
Attribute Type Description Since (schema version)
state enum state of the file (CREATED, MODIFIED, REUP) 1
FOID: string (128bit) File Identifier, is a SHA1 hash of the concatenation of the path + container ID 1
timestamp (ts) int64 The timestamp when file object is exported (nanoseconds). 1
restype enum Indicates the resource type. Currently support: SF_FILE, SF_DIR, SF_UNIX (unix socket), SF_PIPE, SF_UNKNOWN 1
path string Full path of the file/directory, or unique identifier for pipe, unix socket 1
containerId string Unique string representing the Container Object to which the file resides. Can be NULL if file isn’t in a container. 1

Events

Events represent important individual behaviors of an entity that are broken out on their own due to their importance, their rarity, or because maintaining operation order is important. In order to manage events and their differing attributes, we divide them into three different categories: Process, File, and Network events. These are described more in detail later on.

Each event and flow contains a process object id, a timestamp, a thread id, and a set of operation flags. The process object id represents the Process Entity on which the event occurred, while the thread id indicates which process thread was associated with the event.

Operation Flags

The operation flags describe the actual behavior associated with the event (or flow). The flags are represented in a single bitmap which enables multiple behaviors to be combined easily into a flow. An event will have a single bit active, while a flow could have several. The current supported flags are as follows:

Operation Numeric ID Description System Calls Evts/Flows Supported Since (schema version)
OP_CLONE (1 << 0) Process or thread cloned. clone() ProcessEvent 1
OP_EXEC (1 << 1) Execution of a file execve() ProcessEvent 1
OP_EXIT (1 << 2) Process or thread exit. exit() ProcessEvent 1
OP_SETUID (1 << 3) UID of process was changed setuid(), setresuid ProcessEvent 1
OP_SETNS (1 << 4) Process entering namespace setns() FileFlow 1
OP_ACCEPT (1 << 5) Process accepting network connections accept(), select() NetworkFlow 1
OP_CONNECT (1 << 6) Process connecting to remote host or process connect() NetworkFlow 1
OP_OPEN (1 << 7) Process opening a file/resource open(), openat(), create() FileFlow 1
OP_READ_RECV (1 << 8) Process reading from file, receiving network data read(),pread(),recv(),recvfrom(),recvmsg() NetworkFlow, FileFlow 1
OP_WRITE_SEND (1 << 9) Process writing to file, sending network data write(),pwrite(),send(),sendto(),sendmsg() NetworkFlow, FileFlow 1
OP_CLOSE (1 << 10) Process close resource close(),socketshutdown NetworkFlow, FileFlow 1
OP_TRUNCATE (1 << 11) Premature closing of a flow due to exporter shutdown N/A NetworkFlow, FileFlow 1
OP_SHUTDOWN (1 << 12) Shutdown all or part of a full duplex socket connection shutdown() NetworkFlow 1
OP_MMAP (1 << 13) Memory map of a file. mmap() FileFlow 1
OP_DIGEST (1 << 14) Summary flow information for long running flows N/A NetworkFlow, FileFlow 1
OP_MKDIR (1 << 15) Make directory mkdir(), mkdirat() FileEvent 1
OP_RMDIR (1 << 16) Remove directory rmdir() FileEvent 1
OP_LINK (1 << 17) Process creates hard link to existing file link(), linkat() FileEvent 1
OP_UNLINK (1 << 18) Process deletes file unlink(), unlinkat() FileEvent 1
OP_SYMLINK (1 << 19) Process creates sym link to existing file symlink(), symlinkat() FileEvent 1
OP_RENAME (1 << 20) File renamed rename(), renameat() FileEvent 1

Process Event

A Process Event is an event that creates or modifies a process in some way. Currently, we support four Process Events (referred to as operations), and their behavior in SysFlow is described below.

Operation Behavior
OP_CLONE Exported when a new process or thread is cloned. A new Process Entity should be exported prior to exporting the clone operation of a new process.
OP_EXEC Exported when a process calls an exec syscall. This event will modify an existing process, and should be accompanied by a modified Process Entity.
OP_EXIT Exported on a process or thread exit.
OP_SETUID Exported when a process’s UID is changed. This event will modify an existing process, and should be accompanied by a modified Process Entity.

The list of attributes for the Process Event are as follows:

Attribute Type Description Since (schema version)
OID:
host pid
create ts
struct
int64
int64
The OID of the process for which the event occurred. 1
timestamp (ts) int64 The timestamp when the event occurred (nanoseconds). 1
tid int64 The id of the thread associated with the ProcessEvent. If the running process is single threaded tid == pid 1
opFlags int64 The id of the syscall associated with the event. See list of Operation Flags for details. 1
args string[] An array of arguments encoded as string for the syscall. Sparingly implemented. Only really used with setuid for now.
ret int64 Syscall return value. 1

File Event

A File Event is an event that creates, deletes or modifies a File Entity. Currently, we support six File Events (referred to as operations), and their behavior in SysFlow is described below.

Operation Behavior
OP_MKDIR Exported when a new directory is created. Should be accompanied by a new File Entity representing the directory
OP_RMDIR Exported when a directory is deleted.
OP_LINK Exported when a process creates a hard link to an existing file. Should be accompanied by a new File Entity representing the new link.
OP_UNLINK Exported when a process deletes a file.
OP_SYMLINK Exported when a process creates a sym link to an existing file. Should be accompanied by a new File Entity representing the new link.
OP_RENAME Exported when a process creates renames an existing file. Should be accompanied by a new File Entity representing the renamed file.
NOTE: We’d like to also support chmod and chown but these two operations are not fully supported in sysdig. We’d also like to support umount and mount but these operations are not implemented. We anticipate supporting these in a future version.

The list of attributes for the File Event are as follows:

Attribute Type Description Since (schema version)
OID:
host pid
create ts
struct
int64
int64
The OID of the process for which the event occurred. 1
timestamp (ts) int64 The timestamp when the event occurred (nanoseconds). 1
tid int64 The id of the thread associated with the FileEvent. If the running process is single threaded tid == pid 1
opFlags int64 The id of the syscall associated with the event. See list of Operation Flags for details. 1
ret int64 Syscall return value. 1
FOID: string (128bit) The id of the file on which the system call was called. File Identifier, is a SHA1 hash of the concatenation of the path + container ID. 1
NewFOID: string (128bit) Some syscalls (link, symlink, etc.) convert one file into another requiring two files. This id is the id of the file secondary or new file on which the system call was called. File Identifier, is a SHA1 hash of the concatenation of the path + container ID. Can be NULL. 1

Network Event

Currently, not implemented.

Flows

A Flow represents an aggregation of multiple events that naturally fit together to describe a particular behavior. They are designed to reduce data and collect statistics. Examples of flows include an application reading or writing to a file, or sending and receiving data from another process or host. Flows represent a number of events occurring over a period of time, and as such each flow has a set of operations (encoded in a bitmap), a start and an end time. One can determine the operations in the flow by decoding the operation flags.

A flow can be started by any supported operation and are exported in one of two ways. First, they are exported on an exit, or close event signifying the end of a connection, file interaction, or process. Second, a long running flow is exported after a preconfigured time period. After a long running flow is exported, its counters and flags are reset. However, if there is no activity on the flow over a preconfigured period of time, that flow is no longer exported.

In this section, we describe three categories of Flows: Process, File and Network Flows.

Process Flow

A Process Flow represents a summarization of the number of threads created and destroyed over a time period. Process Flows are partially implemented in the collector and will be fully implemented in a later release. Since schema version 2. Currently we support the following operations:

Operation Behavior
OP_CLONE Recorded when a new thread is cloned.
OP_EXIT Recorded on a thread exit.

The list of attributes for the Process Flow are as follows:

Attribute Type Description Since (schema version)
OID:
host pid
create ts
struct
int64
int64
The OID of the process for which the flow occurred. 2
timestamp (ts) int64 The timestamp when the flow starts (nanoseconds). 2
numThreadsCloned int64 The number of threads cloned during the duration of the flow. 2
opFlags int64 (bitmap) The id of one or more syscalls associated with the ProcessFlow. See list of Operation Flags for details. 2
endTs int64 The timestamp when the process flow is exported (nanoseconds). 2
numThreadsExited int64 Number of threads exited during the duration of the flow. 2
numCloneErrors int64 Number of clone errors occuring during the duration of the flow. 2

File Flow

A File Flow represents a collection of operations on a file. Currently we support the following operations:

Operation Behavior
OP_SETNS Process entering namespace entry in mounted file related to reference File Entity
OP_OPEN Process opening a file/resource.
OP_READ_RECV Process reading from file/resource.
OP_WRITE_SEND Process writing to file.
OP_MMAP Processing memory mapping a file.
OP_CLOSE Process closing resource. This action will close corresponding FileFlow.
OP_TRUNCATE Indicates Premature closing of a flow due to exporter shutdown.
OP_DIGEST Summary flow information for long running flows (not implemented).

The list of attributes for the File Flow are as follows:

Attribute Type Description Since (schema version)
OID:
host pid
create ts
struct
int64
int64
The OID of the process for which the flow occurred. 1
timestamp (ts) int64 The timestamp when the flow starts (nanoseconds). 1
tid int64 The id of the thread associated with the flow. If the running process is single threaded tid == pid 1
opFlags int64 (bitmap) The id of one or more syscalls associated with the FileFlow. See list of Operation Flags for details. 1
openFlags int64 Flags associated with an open syscall if present. 1
endTs int64 The timestamp when the file flow is exported (nanoseconds). 1
FOID: string (128bit) The id of the file on which the system call was called. File Identifier, is a SHA1 hash of the concatenation of the path + container ID. 1
fd int32 The file descriptor associated with the flow. 1
numRRecvOps int64 Number of read operations performed during the duration of the flow. 1
numWSendOps int64 Number of write operations performed during the duration of the flow. 1
numRRecvBytes int64 Number of bytes read during the duration of the flow. 1
numWSendBytes int64 Number of bytes written during the duration of the flow. 1

Network Flow

A Network Flow represents a collection of operations on a network connection. Currently we support the following operations:

Operation Behavior
OP_ACCEPT Process accepted a new network connection.
OP_CONNECT Process connected to a remote host or process.
OP_READ_RECV Process receiving data from a remote host or process.
OP_WRITE_SEND Process sending data to a remote host or process.
OP_SHUTDOWN Process shutdown full or single duplex connections.
OP_CLOSE Process closing network connection. This action will close corresponding NetworkFlow.
OP_TRUNCATE Indicates Premature closing of a flow due to exporter shutdown.
OP_DIGEST Summary flow information for long running flows (not implemented).

The list of attributes for the Network Flow are as follows:

Attribute Type Description Since (schema version)
OID:
host pid
create ts
struct
int64
int64
The OID of the process for which the flow occurred. 1
timestamp (ts) int64 The timestamp when the flow starts (nanoseconds). 1
tid int64 The id of the thread associated with the flow. If the running process is single threaded tid == pid 1
opFlags int64 (bitmap) The id of one or more syscalls associated with the flow. See list of Operation Flags for details. 1
endTs int64 The timestamp when the flow is exported (nanoseconds). 1
sip int32 The source IP address. 1
sport int16 The source port. 1
dip int32 The destination IP address. 1
dport int16 The destination port. 1
proto enum The network protocol of the flow. Can be: TCP, UDP, ICMP, RAW 1
numRRecvOps int64 Number of receive operations performed during the duration of the flow. 1
numWSendOps int64 Number of send operations performed during the duration of the flow. 1
numRRecvBytes int64 Number of bytes received during the duration of the flow. 1
numWSendBytes int64 Number of bytes sent during the duration of the flow. 1
NOTE: The current implementation of NetworkFlow only supports ipv4.