Build
Cloning sources
This document describes how to build libSysFlow and run the SysFlow Collector both inside a docker container and on a linux host. Binary packages are also available in the deployments repository. Building and running the application inside a docker container is the easiest way to start. For convenience, skip the build step and pull pre-built images directly from Docker Hub.
To build the project, first clone the repository:
git clone --recursive https://github.com/sysflow-telemetry/sf-collector.git
Manifest
The manifest file contains the metadata and versions of dependencies used to build libSysFlow and the Collector. It can be modified to customize the build to specifc package versions.
Building using Docker
This is the simplest way of reliably building the collector. To build using docker, run:
make build
Note A musl build can be triggered using the
build/musl
target instead.
If this is your first time building the collector, run the build task in the background and go grab a coffee :) If you have cores to spare, the build time can be reduced by setting concurrent make jobs. For example,
make MAKE_JOBS=8 build
During the initial build, a number of base images are created. These are only needed once per dependency version set. Pre-built versions of these images are also available in Docker Hub and GHCR.
Image |
Tag |
Description |
Dockerfile |
---|---|---|---|
ghcr.io/sysflow-telemetry/ubi |
base- |
A UBI base image containing the build dependencies for Falco and libSysFlow |
|
mods- |
A UBI base image containing the pre-installed Falco Libs and tools for building libSysFlow |
||
driver- |
A UBI base image containing the Falco driver loader and container entrypoint for creating the SysFlow Collector released image |
||
ghcr.io/sysflow-telemetry/sf-collector |
libs- |
A UBI base image containing libSysFlow |
|
collector- |
A UBI base image containing the SysFlow Collector |
||
The SysFlow Collector image |
If building using musl, the following images are created instead.
Image |
Tag |
Description |
Dockerfile |
---|---|---|---|
ghcr.io/sysflow-telemetry/alpine |
base- |
An Alpine base image containing the musl build dependencies for Falco and libSysFlow |
|
mods- |
An Alpine base image containing the pre-installed Falco Libs and tools for building a musl-based libSysFlow |
||
ghcr.io/sysflow-telemetry/ubi |
driver- |
A UBI base image containing the Falco driver loader and container entrypoint for creating the SysFlow Collector released image |
|
ghcr.io/sysflow-telemetry/sf-collector-musl |
libs- |
An Alpine base image containing musl-based libSysFlow |
|
collector- |
An Alpine base image containing the musl-based SysFlow Collector |
||
The musl-based SysFlow Collector image |
Building directly on a host
First, install required dependencies.
On Rhel-based hosts:
scripts/installUBIDependency.sh
On Debian-based hosts:
apt install -y patch base-files binutils bzip2 libdpkg-perl perl make xz-utils libncurses5-dev libncursesw5-dev cmake libboost-all-dev g++ flex bison wget libelf-dev liblog4cxx-dev libapr1 libaprutil1 libsparsehash-dev libsnappy-dev libgoogle-glog-dev libjsoncpp-dev
To build the collector:
make
Binary Packaging
You can easily package libSysFlow and the Collector using cpack
. The deb, rpm, and tgz packages are generated in the sctrips/cpack
directory.
To package libSysFlow headers and static libraries, run:
make build
make package-libs
To package a musl-based libSysFlow headers and static libraries, run:
make build/musl
make package-libs/musl
To package the SysFlow collector and its systemd service specification, run:
make build/musl
make package
Running
Command line usage
To list command line options for the collector, run:
sysporter -h
Examples
To convert scap
files to SysFlow traces with an export id. The output will be written to output.sf
.
sysporter -r input.scap -w ./output.sf -e host
Trace a system live, and output SysFlow to files in a directory which are rotated every 30 seconds. The file name will be an epoch timestamp of when the file was initially written. Note that the trailing slash must be present. The example filter ensures that only SysFlow from containers is generated.
sysporter -G 30 -w ./output/ -e host -f "container.type!=host and container.type=docker"
Trace a system live, and output SysFlow to files in a directory which are rotated every 30 seconds. The file name will be an output.<epoch timestamp>
where the timestamp is of when the file was initially written. The example filter ensures that only SysFlow from containers is generated.
sysporter -G 30 -w ./output/output -e host -f "container.type!=host and container.type=docker" </code>`
Docker usage
The easiest way to run the SysFlow collector is from a Docker container, with host mount for the output trace files. The following command shows how to run sf-collector with trace files exported to /mnt/data
on the host.
docker run -d --privileged --name sf-collector \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-v /etc/:/host/etc:ro \
-v /var/lib/:/host/var/lib:ro \
-v /mnt/data:/mnt/data \
-e INTERVAL=60 \
-e EXPORTER_ID=${HOSTNAME} \
-e OUTPUT=/mnt/data/ \
-e FILTER="container.name!=sf-collector and container.name!=sf-processor and container.name!=sf-exporter" \
--rm sysflowtelemetry/sf-collector
where INTERVAL
denotes the time in seconds before a new trace file is generated, EXPORTER_ID
sets the exporter name, OUTPUT
is the directory in which trace files are written, and FILTER
is the filter expression used to filter collected events. The collector can also be setup to stream events through a unix domain socket (see sf-deployments for other deployment configurations).
Note append
container.type!=host
to FILTER expression to filter host events.
The key setting in the collector configuration is the FILTER
variable. The collector is built atop the Falco libs and it uses Falco’s filtering mechanism described here. It supports filtering on specific containers, processes, operations, etc. One of the most powerful filters is the container.type!=host
filter, which limits collection only to container monitoring. If you want to monitor the entire host, simply remove the container.type
operation from the filter.
Event rate optimization
The following environment variables can be set to reduce the number of events generated by the collector:
Drop mode (
ENABLE_DROP_MODE
=1): removes syscalls inside the kernel before they are passed up to the collector, resulting in much better performance, less spilled events, but does remove mmaps from output.Process flows (
ENABLE_PROC_FLOW
=1): enables the creation of process flows, aggregating thread events.File only (
FILE_ONLY
=1): filters out any descriptor that is not a file, including unix sockets and pipesFile read mode (
FILE_READ_MODE
=1): sets mode for file reads.0
enables recording all file reads as flows.1
disables all file reads.2
disables recording file reads to noisy directories: “/proc/”, “/dev/”, “/sys/”, “//sys/”, “/lib/”, “/lib64/”, “/usr/lib/”, “/usr/lib64/”.