Installation¶
Cloning sources¶
This document describes how to build and run the SysFlow Collector both inside a docker container and on a linux host. Binary packages are also available in the deployments repository. Building and running the application inside a docker container is the easiest way to start. For convenience, skip the build step and pull pre-built images directly from Docker Hub.
To build the project, first clone the repository:
git clone --recursive https://github.com/sysflow-telemetry/sf-collector.git
Building as Docker container¶
To build as docker container, run:
make docker-build
Building directly on a host¶
First, install required dependencies.
On Rhel-based hosts:
scripts/installUBIDependency.sh
On Debian-based hosts:
apt install -y patch base-files binutils bzip2 libdpkg-perl perl make xz-utils libncurses5-dev libncursesw5-dev cmake libboost-all-dev g++ flex bison wget libelf-dev liblog4cxx-dev libapr1 libaprutil1 libsparsehash-dev libsnappy-dev libgoogle-glog-dev libjsoncpp-dev
To build the collector:
make
Running¶
Command line usage¶
To list command line options for the collector, run:
sysporter -h
Examples¶
To convert scap
files to SysFlow traces with an export id. The output will be written to output.sf
.
sysporter -r input.scap -w ./output.sf -e host
Trace a system live, and output SysFlow to files in a directory which are rotated every 30 seconds. The file name will be an epoch timestamp of when the file was initially written. Note that the trailing slash must be present. The example filter ensures that only SysFlow from containers is generated.
sysporter -G 30 -w ./output/ -e host -f "container.type!=host and container.type=docker"
Trace a system live, and output SysFlow to files in a directory which are rotated every 30 seconds. The file name will be an output.<epoch timestamp>
where the timestamp is of when the file was initially written. The example filter ensures that only SysFlow from containers is generated.
sysporter -G 30 -w ./output/output -e host -f "container.type!=host and container.type=docker" </code>`
Docker usage¶
The easiest way to run the SysFlow collector is from a Docker container, with host mount for the output trace files. The following command shows how to run sf-collector with trace files exported to /mnt/data
on the host.
docker run -d --privileged --name sf-collector \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-v /etc/:/host/etc:ro \
-v /mnt/data:/mnt/data \
-e INTERVAL=60 \
-e EXPORTER_ID=${HOSTNAME} \
-e OUTPUT=/mnt/data/ \
-e FILTER="container.name!=sf-collector and container.name!=sf-processor and container.name!=sf-exporter" \
--rm sysflowtelemetry/sf-collector
where INTERVAL
denotes the time in seconds before a new trace file is generated, EXPORTER_ID
sets the exporter name, OUTPUT
is the directory in which trace files are written, and FILTER
is the filter expression used to filter collected events.
Note append
container.type!=host
to FILTER expression to filter host events.
The key setting in the collector configuration is the FILTER
variable. The collector is built atop the Falco libs and it uses Falco’s filtering mechanism described here. It supports filtering on specific containers, processes, operations, etc. One of the most powerful filters is the container.type!=host
filter, which limits collection only to container monitoring. If you want to monitor the entire host, simply remove the container.type
operation from the filter.
Event rate optimization¶
The following environment variables can be set to reduce the number of events generated by the collector:
Drop mode (
ENABLE_DROP_MODE
=1): removes syscalls inside the kernel before they are passed up to the collector, resulting in much better performance, less spilled events, but does remove mmaps from output.Process flows (
ENABLE_PROC_FLOW
=1): enables the creation of process flows, aggregating thread events.File only (
FILE_ONLY
=1): filters out any descriptor that is not a file, including unix sockets and pipesFile read mode (
FILE_READ_MODE
=1): sets mode for file reads.0
enables recording all file reads as flows.1
disables all file reads.2
disables recording file reads to noisy directories: “/proc/”, “/dev/”, “/sys/”, “//sys/”, “/lib/”, “/lib64/”, “/usr/lib/”, “/usr/lib64/”.