SysFlow Telemetry Pipeline
The SysFlow Telemetry Pipeline is a framework for monitoring cloud and enterprise workloads. The framework builds the plumbing required for system telemetry so that users can focus on writing and sharing analytics on a scalable, common open-source platform.
If in a hurry, skip to our quick start guide.
The backbone of the telemetry pipeline is a new data format which lifts raw system event information into an abstraction that describes process behaviors, and their relationships with containers, files, and network activity. This object-relational format is highly compact, yet it provides broad visibility into legacy endpoints and container clouds.
The platform is designed as a pluggable edge processing architecture which includes a policy engine that accepts declarative policies that support edge filtering, tagging, and alerting on SysFlow streams. It also offers several APIs that allow users to process SysFlow with their favorite toolkits.
The pipeline can be deployed using Docker, Kubernetes, OpenShift, and bare metal/VMs. The SysFlow agent can be configured as an edge analytics pipeline to stream SysFlow records through rsyslog, or as a batch exporter of raw SysFlow traces to S3-compatible object stores.
An integrated Jupyter environment makes it easy to perform log hunting on collected traces. There are also Apache Avro schema files for SysFlow so that users can generate APIs for other programming languages. C++, Python, and Golang APIs are available, allowing users to interact with SysFlow traces programmatically.
To learn more about SysFlow, check the table of contents below.
We welcome feedback, bug reports, and contributions!
Keep in touch
Please connect with us on our Slack community!
Bugs & Feature requests
For bugs and feature requests, please check our issue tracker.
SysFlow and all projects are released under the Apache v2.0 license.
- Quick Start
- SysFlow Specification
- SysFlow Collector (sf-collector repo)
- SysFlow Processor (sf-processor repo)
- SysFlow Exporter (sf-exporter repo)
- SysFlow APIs and Utilities (sf-apis repo)
- Deployments (sf-deployments repo)
- Code of Conduct
- Contributor Covenant Code of Conduct
- Talks & Publications