SysFlow Python API Reference¶

SysFlow Reader API¶

class sysflow.reader.FlattenedSFReader(filename, retEntities=False)¶

FlattenedSFReader

This class loads a raw sysflow file, and links all Entities (header, process, container, files) with the current flow or event in the file. As a result, the user does not have to manage this information. This class supports the python iterator design pattern. Example Usage:

reader = FlattenedSFReader(trace)
head = 20 # max number of records to print
for i, (objtype, header, cont, pproc, proc, files, evt, flow) in enumerate(reader):
    exe = proc.exe
    pid = proc.oid.hpid if proc else ''
    evflow = evt or flow
    tid = evflow.tid if evflow else ''
    opFlags = utils.getOpFlagsStr(evflow.opFlags) if evflow else '' 
    sTime = utils.getTimeStr(evflow.ts) if evflow else ''
    eTime = utils.getTimeStr(evflow.endTs) if flow else ''
    ret = evflow.ret if evt else ''
    res1 = ''
    if objtype == ObjectTypes.FILE_FLOW or objtype == ObjectTypes.FILE_EVT:
        res1 = files[0].path
    elif objtype == ObjectTypes.NET_FLOW:
        res1 = utils.getNetFlowStr(flow) 
    numBReads = evflow.numRRecvBytes if flow else ''
    numBWrites = evflow.numWSendBytes if flow else ''
    res2 = files[1].path if files and files[1] else ''
    cont = cont.id if cont else '' 
    print("|{0:30}|{1:9}|{2:26}|{3:26}|{4:30}|{5:8}|{6:8}|".format(exe, opFlags, sTime, eTime, res1, numBReads, numBWrites))
    if i == head:
        break

Parameters:	filename (str) – the name of the sysflow file to be read. retEntities (bool) – If True, the reader will return entity objects by themselves as they are seen in the sysflow file. In this case, all other objects will be set to None

Iterator

Reader returns a tuple of objects in the following order:

objtype (sysflow.objtypes.ObjectTypes) The type of entity or flow returned.

header (sysflow.entity.SFHeader) The header entity of the file.

cont (sysflow.entity.Container) The container associated with the flow/evt, or None if no container.

pproc (sysflow.entity.Process) The parent process associated with the flow/evt.

proc (sysflow.entity.Process) The process associated with the flow/evt.

files (tuple of sysflow.entity.File) Any files associated with the flow/evt.

evt (sysflow.event.{ProcessEvent,FileEvent}) If the record is an event, it will be returned here. Otherwise this variable will be None. objtype will indicate the type of event.

flow (sysflow.flow.{NetworkFlow,FileFlow}) If the record is a flow, it will be returned here. Otherwise this variable will be None. objtype will indicate the type of flow.

getProcess(oid)¶

Returns a Process Object given a process object id.

Parameters:	oid (sysflow.type.OID) – the object id of the Process Object requested
Return type:	sysflow.entity.Process
Returns:	the desired process object or None if no process object is available.

class sysflow.reader.NestedNamespace(**kwargs)¶

class sysflow.reader.SFReader(filename)¶

SFReader

This class loads a raw sysflow file, and returns each entity/flow one by one. It is the user’s responsibility to link the related objects together through the OID. This class supports the python iterator design pattern. Example Usage:

reader = SFReader("./sysflowfile.sf")
for name, sf in reader:
    if name == "sysflow.entity.SFHeader":
       //do something with the header object
    elif name == "sysflow.entity.Container":
       //do something with the container object
    elif name == "sysflow.entity.Process":
       //do something with the Process object
    ....

Parameters:	filename (str) – the name of the sysflow file to be read.

SysFlow Formatter API¶

class sysflow.formatter.SFFormatter(reader, defs=[])¶

SFFormatter

This class takes a FlattenedSFReader, and exports SysFlow as either JSON, CSV or Pretty Print . Example Usage:

reader = FlattenedSFReader(trace, False)
formatter = SFFormatter(reader)
fields=args.fields.split(',') if args.fields else None
if args.output == 'json':
    if args.file is not None:
        formatter.toJsonFile(args.file, fields=fields)
    else:
        formatter.toJsonStdOut(fields=fields)
elif args.output == 'csv' and args.file is not None:
    formatter.toCsvFile(args.file, fields=fields)
elif args.output == 'str':
    formatter.toStdOut(fields=fields)

Parameters:	reader (sysflow.reader.FlattenedSFReader) – A reader representing the sysflow file being read. defs (list) – A list of paths to filter definitions.

applyFuncJson(func, fields=None, expr=None)¶

Enables a delegate function to be applied to each JSON record read.

Parameters:	func (function) – delegate function of the form func(str) fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression

getFields()¶: Returns a list with available SysFlow fields and their descriptions.

toCsvFile(path, fields=None, header=True, expr=None)¶

Writes SysFlow to CSV file.

Parameters:	path (str) – the full path of the output file. fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression

toDataframe(fields=None, expr=None)¶

Enables a delegate function to be applied to each JSON record read.

Parameters:	func (function) – delegate function of the form func(str) fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression

toJson(fields=None, flat=False, expr=None)¶

Writes SysFlow as JSON object.

Parameters:	fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression
Flat:	specifies if JSON output should be flattened

toJsonFile(path, fields=None, flat=False, expr=None)¶

Writes SysFlow to JSON file.

Parameters:	path (str) – the full path of the output file. fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression
Flat:	specifies if JSON output should be flattened

toJsonStdOut(fields=None, flat=False, expr=None)¶

Writes SysFlow as JSON to stdout.

Parameters:	fields (list) – a list of the SysFlow fields to be exported in JSON. See formatter.py for a list of fields expr (str) – a sfql filter expression
Flat:	specifies if JSON output should be flattened

toStdOut(fields=['ts_uts', 'type', 'proc.exe', 'proc.args', 'pproc.pid', 'proc.pid', 'proc.tid', 'opflags', 'res', 'flow.rbytes', 'flow.wbytes', 'container.id'], pretty_headers=True, showindex=True, expr=None)¶

Writes SysFlow as a tabular pretty print form to stdout.

Parameters:	fields (list) – a list of the SysFlow fields to be exported in the JSON. See formatter.py for a list of fields pretty_headers (bool) – print table headers in pretty format. showindex (bool) – show record number. expr (str) – a sfql filter expression

SysFlow Object Types¶

class sysflow.objtypes.ObjectTypes¶

ObjectTypes

Enumeration representing each of the object types:: HEADER = 0, CONT = 1, PROC = 2, FILE = 3, PROC_EVT = 4, NET_FLOW = 5, FILE_FLOW = 6, FILE_EVT = 7

SysFlow Utils API¶

sysflow.utils.getIpIntStr(ipInt)¶

Converts an IP address in host order integer to a string representation.

Parameters:	ipInt – an IP address integer
Return type:	str
Returns:	A string representation of the IP address

sysflow.utils.getNetFlowStr(nf)¶

Converts a NetworkFlow into a string representation.

Parameters:	nf (sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass) – a NetworkFlow object.
Return type:	str
Returns:	A string representation of the NetworkFlow in form (sip:sport-dip:dport).

sysflow.utils.getOpFlags(opFlags)¶

Converts a sysflow operations flag bitmap into a set representation.

Parameters:	opflag (int) – An operations bitmap from a flow or event.
Return type:	set
Returns:	A set representation of the operations bitmap.

sysflow.utils.getOpFlagsStr(opFlags)¶

Converts a sysflow operations flag bitmap into a string representation.

Parameters:	opflag (int) – An operations bitmap from a flow or event.
Return type:	str
Returns:	A string representation of the operations bitmap.

sysflow.utils.getOpStr(opFlags)¶

Converts a sysflow operations into a string representation.

Parameters:	opflag (int) – An operations bitmap from a flow or event.
Return type:	str
Returns:	A string representation of the operations bitmap.

sysflow.utils.getOpenFlags(openFlags)¶

Converts a sysflow open modes flag bitmap into a set representation.

Parameters:	opflag – An open modes bitmap from a flow or event.
Return type:	set
Returns:	A set representation of the open modes bitmap.

sysflow.utils.getTimeStr(ts)¶

Converts a nanosecond ts into a string representation.

Parameters:	ts (int) – A nanosecond epoch.
Return type:	str
Returns:	A string representation of the timestamp in %m/%d/%YT%H:%M:%S.%f format.

sysflow.utils.getTimeStrIso8601(ts)¶

Converts a nanosecond ts into a string representation in UTC time zone.

Parameters:	ts (int) – A nanosecond epoch.
Return type:	str
Returns:	A string representation of the timestamp in ISO 8601 format.

SysFlow Class¶

sysflow.sysflow.SysFlow¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.SysFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.SysFlowClass(inner_dict=None)¶

rec¶

Return type: SchemaClasses.sysflow.entity.SFHeaderClass | SchemaClasses.sysflow.entity.ContainerClass | SchemaClasses.sysflow.entity.ProcessClass | SchemaClasses.sysflow.entity.FileClass | SchemaClasses.sysflow.event.ProcessEventClass | SchemaClasses.sysflow.flow.NetworkFlowClass | SchemaClasses.sysflow.flow.FileFlowClass | SchemaClasses.sysflow.event.FileEventClass | SchemaClasses.sysflow.event.NetworkEventClass | SchemaClasses.sysflow.flow.ProcessFlowClass

Container Class¶

sysflow.sysflow.entity.Container¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.ContainerClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.ContainerClass(inner_dict=None)¶

id¶

Return type:	str

image¶

Return type:	str

imageid¶

Return type:	str

name¶

Return type:	str

privileged¶

Return type:	bool

type¶

Return type:	SchemaClasses.sysflow.type.ContainerTypeClass

File Class¶

sysflow.sysflow.entity.File¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.FileClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.FileClass(inner_dict=None)¶

containerId¶

Return type:	str

oid¶

Return type:	bytes

path¶

Return type:	str

restype¶

Return type:	int

state¶

Return type:	SchemaClasses.sysflow.type.SFObjectStateClass

ts¶

Return type:	int

Header Class¶

sysflow.sysflow.entity.SFHeader¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.SFHeaderClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.SFHeaderClass(inner_dict=None)¶

exporter¶

Return type:	str

ip¶

Return type:	str

version¶

Return type:	int

Process Class¶

sysflow.sysflow.entity.Process¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.entity.ProcessClass

class sysflow.schema_classes.SchemaClasses.sysflow.entity.ProcessClass(inner_dict=None)¶

containerId¶

Return type:	str

entry¶

Return type:	bool

exe¶

Return type:	str

exeArgs¶

Return type:	str

gid¶

Return type:	int

groupName¶

Return type:	str

oid¶

Return type:	SchemaClasses.sysflow.type.OIDClass

poid¶

Return type:	SchemaClasses.sysflow.type.OIDClass

state¶

Return type:	SchemaClasses.sysflow.type.SFObjectStateClass

ts¶

Return type:	int

tty¶

Return type:	bool

uid¶

Return type:	int

userName¶

Return type:	str

File Event¶

sysflow.sysflow.event.FileEvent¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.event.FileEventClass

class sysflow.schema_classes.SchemaClasses.sysflow.event.FileEventClass(inner_dict=None)¶

fileOID¶

Return type:	bytes

newFileOID¶

Return type:	bytes

opFlags¶

Return type:	int

procOID¶

Return type:	SchemaClasses.sysflow.type.OIDClass

ret¶

Return type:	int

tid¶

Return type:	int

ts¶

Return type:	int

Process Event¶

sysflow.sysflow.event.ProcessEvent¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.event.ProcessEventClass

class sysflow.schema_classes.SchemaClasses.sysflow.event.ProcessEventClass(inner_dict=None)¶

args¶

Return type:	list[str]

opFlags¶

Return type:	int

procOID¶

Return type:	SchemaClasses.sysflow.type.OIDClass

ret¶

Return type:	int

tid¶

Return type:	int

ts¶

Return type:	int

File Flow¶

sysflow.sysflow.flow.FileFlow¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.flow.FileFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.flow.FileFlowClass(inner_dict=None)¶

endTs¶

Return type:	int

fd¶

Return type:	int

fileOID¶

Return type:	bytes

numRRecvBytes¶

Return type:	int

numRRecvOps¶

Return type:	int

numWSendBytes¶

Return type:	int

numWSendOps¶

Return type:	int

opFlags¶

Return type:	int

openFlags¶

Return type:	int

procOID¶

Return type:	SchemaClasses.sysflow.type.OIDClass

tid¶

Return type:	int

ts¶

Return type:	int

Network Flow¶

sysflow.sysflow.flow.NetworkFlow¶: alias of sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass

class sysflow.schema_classes.SchemaClasses.sysflow.flow.NetworkFlowClass(inner_dict=None)¶

dip¶

Return type:	int

dport¶

Return type:	int

endTs¶

Return type:	int

fd¶

Return type:	int

numRRecvBytes¶

Return type:	int

numRRecvOps¶

Return type:	int

numWSendBytes¶

Return type:	int

numWSendOps¶

Return type:	int

opFlags¶

Return type:	int

procOID¶

Return type:	SchemaClasses.sysflow.type.OIDClass

proto¶

Return type:	int

sip¶

Return type:	int

sport¶

Return type:	int

tid¶

Return type:	int

ts¶

Return type:	int