SysFlow Python API Reference

SysFlow Reader API

class sysflow.reader.FlattenedSFReader(filename, retEntities=False)

FlattenedSFReader

This class loads a raw sysflow file, and links all Entities (header, process, container, files) with the current flow or event in the file. As a result, the user does not have to manage this information. This class supports the python iterator design pattern. Example Usage:

reader = FlattenedSFReader(trace)
head = 20 # max number of records to print
for i, (objtype, header, cont, pproc, proc, files, evt, flow) in enumerate(reader):
    exe = proc.exe
    pid = proc.oid.hpid if proc else ''
    evflow = evt or flow
    tid = evflow.tid if evflow else ''
    opFlags = utils.getOpFlagsStr(evflow.opFlags) if evflow else '' 
    sTime = utils.getTimeStr(evflow.ts) if evflow else ''
    eTime = utils.getTimeStr(evflow.endTs) if flow else ''
    ret = evflow.ret if evt else ''
    res1 = ''
    if objtype == ObjectTypes.FILE_FLOW or objtype == ObjectTypes.FILE_EVT:
        res1 = files[0].path
    elif objtype == ObjectTypes.NET_FLOW:
        res1 = utils.getNetFlowStr(flow) 
    numBReads = evflow.numRRecvBytes if flow else ''
    numBWrites = evflow.numWSendBytes if flow else ''
    res2 = files[1].path if files and files[1] else ''
    cont = cont.id if cont else '' 
    print("|{0:30}|{1:9}|{2:26}|{3:26}|{4:30}|{5:8}|{6:8}|".format(exe, opFlags, sTime, eTime, res1, numBReads, numBWrites))
    if i == head:
        break
Parameters:
  • filename (str) – the name of the sysflow file to be read.
  • retEntities (bool) – If True, the reader will return entity objects by themselves as they are seen in the sysflow file. In this case, all other objects will be set to None
Iterator

Reader returns a tuple of objects in the following order:

objtype (sysflow.objtypes.ObjectTypes) The type of entity or flow returned.

header (sysflow.entity.SFHeader) The header entity of the file.

cont (sysflow.entity.Container) The container associated with the flow/evt, or None if no container.

pproc (sysflow.entity.Process) The parent process associated with the flow/evt.

proc (sysflow.entity.Process) The process associated with the flow/evt.

files (tuple of sysflow.entity.File) Any files associated with the flow/evt.

evt (sysflow.event.{ProcessEvent,FileEvent}) If the record is an event, it will be returned here. Otherwise this variable will be None. objtype will indicate the type of event.

flow (sysflow.flow.{NetworkFlow,FileFlow}) If the record is a flow, it will be returned here. Otherwise this variable will be None. objtype will indicate the type of flow.

getProcess(oid)

Returns a Process Object given a process object id.

Parameters:oid (sysflow.type.OID) – the object id of the Process Object requested
Return type:sysflow.entity.Process
Returns:the desired process object or None if no process object is available.
class sysflow.reader.NestedNamespace(**kwargs)
class sysflow.reader.SFReader(filename)

SFReader

This class loads a raw sysflow file, and returns each entity/flow one by one. It is the user’s responsibility to link the related objects together through the OID. This class supports the python iterator design pattern. Example Usage:

reader = SFReader("./sysflowfile.sf")
for name, sf in reader:
    if name == "sysflow.entity.SFHeader":
       //do something with the header object
    elif name == "sysflow.entity.Container":
       //do something with the container object
    elif name == "sysflow.entity.Process":
       //do something with the Process object
    ....
Parameters:filename (str) – the name of the sysflow file to be read.

SysFlow Formatter API

class sysflow.formatter.SFFormatter(reader, defs=[])

SFFormatter

This class takes a FlattenedSFReader, and exports SysFlow as either JSON, CSV or Pretty Print . Example Usage:

reader = FlattenedSFReader(trace, False)
formatter = SFFormatter(reader)
fields=args.fields.split(',') if args.fields else None
if args.output == 'json':
    if args.file is not None:
        formatter.toJsonFile(args.file, fields=fields)
    else:
        formatter.toJsonStdOut(fields=fields)
elif args.output == 'csv' and args.file is not None:
    formatter.toCsvFile(args.file, fields=fields)
elif args.output == 'str':
    formatter.toStdOut(fields=fields)
Parameters:
applyFuncJson(func, fields=None, expr=None)

Enables a delegate function to be applied to each JSON record read.

Parameters:
  • func (function) – delegate function of the form func(str)
  • fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
getFields()

Returns a list with available SysFlow fields and their descriptions.

toCsvFile(path, fields=None, header=True, expr=None)

Writes SysFlow to CSV file.

Parameters:
  • path (str) – the full path of the output file.
  • fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
toDataframe(fields=None, expr=None)

Enables a delegate function to be applied to each JSON record read.

Parameters:
  • func (function) – delegate function of the form func(str)
  • fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
toJson(fields=None, flat=False, expr=None)

Writes SysFlow as JSON object.

Parameters:
  • fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
Flat:

specifies if JSON output should be flattened

toJsonFile(path, fields=None, flat=False, expr=None)

Writes SysFlow to JSON file.

Parameters:
  • path (str) – the full path of the output file.
  • fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
Flat:

specifies if JSON output should be flattened

toJsonStdOut(fields=None, flat=False, expr=None)

Writes SysFlow as JSON to stdout.

Parameters:
  • fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields
  • expr (str) – a sfql filter expression
Flat:

specifies if JSON output should be flattened

toStdOut(fields=['ts_uts', 'type', 'proc.exe', 'proc.args', 'pproc.pid', 'proc.pid', 'proc.tid', 'opflags', 'res', 'flow.rbytes', 'flow.wbytes', 'container.id'], pretty_headers=True, showindex=True, expr=None)

Writes SysFlow as a tabular pretty print form to stdout.

Parameters:
  • fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields
  • pretty_headers (bool) – print table headers in pretty format.
  • showindex (bool) – show record number.
  • expr (str) – a sfql filter expression

SysFlow Object Types

class sysflow.objtypes.ObjectTypes

ObjectTypes

Enumeration representing each of the object types:
HEADER = 0, CONT = 1, PROC = 2, FILE = 3, PROC_EVT = 4, NET_FLOW = 5, FILE_FLOW = 6, FILE_EVT = 7

SysFlow Utils API

sysflow.utils.getIpIntStr(ipInt)

Converts an IP address in host order integer to a string representation.

Parameters:ipInt – an IP address integer
Return type:str
Returns:A string representation of the IP address
sysflow.utils.getNetFlowStr(nf)

Converts a NetworkFlow into a string representation.

Parameters:nf (sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass) – a NetworkFlow object.
Return type:str
Returns:A string representation of the NetworkFlow in form (sip:sport-dip:dport).
sysflow.utils.getOpFlags(opFlags)

Converts a sysflow operations flag bitmap into a set representation.

Parameters:opflag (int) – An operations bitmap from a flow or event.
Return type:set
Returns:A set representation of the operations bitmap.
sysflow.utils.getOpFlagsStr(opFlags)

Converts a sysflow operations flag bitmap into a string representation.

Parameters:opflag (int) – An operations bitmap from a flow or event.
Return type:str
Returns:A string representation of the operations bitmap.
sysflow.utils.getOpStr(opFlags)

Converts a sysflow operations into a string representation.

Parameters:opflag (int) – An operations bitmap from a flow or event.
Return type:str
Returns:A string representation of the operations bitmap.
sysflow.utils.getOpenFlags(openFlags)

Converts a sysflow open modes flag bitmap into a set representation.

Parameters:opflag – An open modes bitmap from a flow or event.
Return type:set
Returns:A set representation of the open modes bitmap.
sysflow.utils.getTimeStr(ts)

Converts a nanosecond ts into a string representation.

Parameters:ts (int) – A nanosecond epoch.
Return type:str
Returns:A string representation of the timestamp in %m/%d/%YT%H:%M:%S.%f format.
sysflow.utils.getTimeStrIso8601(ts)

Converts a nanosecond ts into a string representation in UTC time zone.

Parameters:ts (int) – A nanosecond epoch.
Return type:str
Returns:A string representation of the timestamp in ISO 8601 format.

SysFlow Class

sysflow.sysflow.SysFlow

alias of sysflow.schema_classes.SchemaClasses.sysflow.SysFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.SysFlowClass(inner_dict=None)
rec
Return type:SchemaClasses.sysflow.entity.SFHeaderClass | SchemaClasses.sysflow.entity.ContainerClass | SchemaClasses.sysflow.entity.ProcessClass | SchemaClasses.sysflow.entity.FileClass | SchemaClasses.sysflow.event.ProcessEventClass | SchemaClasses.sysflow.flow.NetworkFlowClass | SchemaClasses.sysflow.flow.FileFlowClass | SchemaClasses.sysflow.event.FileEventClass | SchemaClasses.sysflow.event.NetworkEventClass | SchemaClasses.sysflow.flow.ProcessFlowClass

Container Class

sysflow.sysflow.entity.Container

alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.ContainerClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.ContainerClass(inner_dict=None)
id
Return type:str
image
Return type:str
imageid
Return type:str
name
Return type:str
privileged
Return type:bool
type
Return type:SchemaClasses.sysflow.type.ContainerTypeClass

File Class

sysflow.sysflow.entity.File

alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.FileClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.FileClass(inner_dict=None)
containerId
Return type:str
oid
Return type:bytes
path
Return type:str
restype
Return type:int
state
Return type:SchemaClasses.sysflow.type.SFObjectStateClass
ts
Return type:int

Header Class

sysflow.sysflow.entity.SFHeader

alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.SFHeaderClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.SFHeaderClass(inner_dict=None)
exporter
Return type:str
ip
Return type:str
version
Return type:int

Process Class

sysflow.sysflow.entity.Process

alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.ProcessClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.ProcessClass(inner_dict=None)
containerId
Return type:str
entry
Return type:bool
exe
Return type:str
exeArgs
Return type:str
gid
Return type:int
groupName
Return type:str
oid
Return type:SchemaClasses.sysflow.type.OIDClass
poid
Return type:SchemaClasses.sysflow.type.OIDClass
state
Return type:SchemaClasses.sysflow.type.SFObjectStateClass
ts
Return type:int
tty
Return type:bool
uid
Return type:int
userName
Return type:str

File Event

sysflow.sysflow.event.FileEvent

alias of sysflow.schema_classes.SchemaClasses.sysflow.event.FileEventClass

class sysflow.schema_classes.SchemaClasses.sysflow.event.FileEventClass(inner_dict=None)
fileOID
Return type:bytes
newFileOID
Return type:bytes
opFlags
Return type:int
procOID
Return type:SchemaClasses.sysflow.type.OIDClass
ret
Return type:int
tid
Return type:int
ts
Return type:int

Process Event

sysflow.sysflow.event.ProcessEvent

alias of sysflow.schema_classes.SchemaClasses.sysflow.event.ProcessEventClass

class sysflow.schema_classes.SchemaClasses.sysflow.event.ProcessEventClass(inner_dict=None)
args
Return type:list[str]
opFlags
Return type:int
procOID
Return type:SchemaClasses.sysflow.type.OIDClass
ret
Return type:int
tid
Return type:int
ts
Return type:int

File Flow

sysflow.sysflow.flow.FileFlow

alias of sysflow.schema_classes.SchemaClasses.sysflow.flow.FileFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.flow.FileFlowClass(inner_dict=None)
endTs
Return type:int
fd
Return type:int
fileOID
Return type:bytes
numRRecvBytes
Return type:int
numRRecvOps
Return type:int
numWSendBytes
Return type:int
numWSendOps
Return type:int
opFlags
Return type:int
openFlags
Return type:int
procOID
Return type:SchemaClasses.sysflow.type.OIDClass
tid
Return type:int
ts
Return type:int

Network Flow

sysflow.sysflow.flow.NetworkFlow

alias of sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass(inner_dict=None)
dip
Return type:int
dport
Return type:int
endTs
Return type:int
fd
Return type:int
numRRecvBytes
Return type:int
numRRecvOps
Return type:int
numWSendBytes
Return type:int
numWSendOps
Return type:int
opFlags
Return type:int
procOID
Return type:SchemaClasses.sysflow.type.OIDClass
proto
Return type:int
sip
Return type:int
sport
Return type:int
tid
Return type:int
ts
Return type:int