Using Go for Scalable Operating System Analytics

At Kolide, part of what we do is build a client-server system that facilitates the management of large, isolated installations of an open-source tool called osquery. While I’ve written and spoken about osquery in the past, in this post I want to focus on how we use Go to solve many of the challenges we face building software for the osquery community and for internal distributed systems.

Osquery is a operating system analytics tool that allows you to articulate operating system state using SQL queries. You then provide osquery with a declarative configuration of queries that you want to use to monitor a host. With each query, you use options to define how the results of these queries should be logged (i.e.: log whenever the result-set changes, log a snapshot of the data-set at a set interval, etc). An operator can then use this data to create and annotate a current, evolving data-model of your infrastructure. You can also store this raw data (JSON) and load it into a data warehouse like HDFS or BigQuery. Analysis of this data can be useful for accomplishing objectives related to security, operations, device administration, and compliance.

At Kolide, we have a lot of experience with osquery and managing osquery infrastructure. In order for an osquery deployment across your fleet to be successful, you need to have excellent packaging tooling: osquery provides a lot of capabilities, but in-order to take advantage of it’s power, configuration can be rather complex. As of osquery version 2.10.1, there are 115 configurable options. Configuration via the filesystem is most effectively provided with a directory structure of files. You may use custom extensions, which are to be distributed as platform-specific binary executables. To accomplish your specific objectives, you must have good tooling for packaging and distributing osquery.

Managing osquery also requires diligent update hygiene: in order to take advantage of the most reliable software that allows you to monitor for emergent threats, you should really be keeping osqueryd very current. Osquery is a fast-moving project and a lot of great work is happening all throughout the community. Using an osquery manager which is capable of auto-updating and managing osquery as per an “update channel” (i.e.: “beta”, “stable”) makes this process much easier, especially in environments where clients and package management may be unreliable.

The Osquery Launcher

At Kolide, the tool we use for managing osquery instances, autoupdating osquery, and establishing remote communication with a specified server is called “Launcher”. You can read about Launcher on our website and on GitHub. Launcher is written in Go and includes a number of niceties that make scaling osquery deployments easier. For the Go developer, some of my favorite features / parts of the codebase include:

• An osquery runtime which exposes a programmatic Go interface that takes advantage of the Functional Option API pattern made popular by Dave Cheney (blog, video). Once @groob introduced me to this pattern, I fell in love with it. It definitely takes more time and boilerplate, but the beauty of the resultant API and the additional type safety make it a must-have for these kinds of APIs.

• A modern, type-safe gRPC server specification. gRPC, a CNCF project created by Google, is an excellent RPC framework that is basically Protocol Buffers over HTTP2 with a whole bunch of fun capabilities. We write a lot of Launcher Servers, so having a well-defined, versioned server specification is useful.

• A secure osquery auto-update system using The Update Framework. TUF, a CNCF project created by Docker, is a specification that outlines a peer-reviewed update mechanism that is robust and complete. The specification articulates a client/server model. Kolide hosts a TUF server using the Notary tool. Notary, a CNCF project created by Docker, provides a TUF server and we wrote a purpose-built TUF client called Updater at Kolide (in Go of course)

All of these components are wired together in the Launcher project in the main file of the Launcher repository (cmd/launcher/launcher.go).

The Launcher is designed to communicate with a server that fulfills the provided gRPC server specification. Package Builder makes it easy to distribute the Launcher and Osquery but a server is necessary to complete the transport. For this, there is Kolide Fleet, an open-source osquery fleet manager. We have previously written about Fleet on this blog and one of the excellent features of Fleet is that it supports both the existing osquery TLS server API as well as the new osquery gRPC server API. This makes converting easy for users with an existing osquery deployment.

The Launcher and Package Builder, and Fleet all follow similar patterns that we like to follow for all Go projects at Kolide:

• The minimal program files should always be in cmd/program/.

• Use Go-Kit whenever you need to map an idiomatic program interface to a remote transport (like gRPC or HTTP).

• Use the new Go Dep tool to manage dependencies.

• Use Dave Cheney’s error package to wrap errors.

• Use the command parsing pattern established by Peter Bourgon in OKLog instead of command-parsing libraries if the CLI is not extremely complex.

• Don’t use init and don’t use package-level variables. Peter Bourgon has written about this rather convincingly. Peter says:

tl;dr: magic is bad; global state is magic → no package level vars; no func init.

• Propagate a context.Context throughout your APIs. Use this context to pass things like request UUIDs, which are useful when doing distributed tracing of requests.

• Supply a Dockerfile with each project to facilitate Cloud Native testing and deployment strategies.

Kolide Kit & Style Guide

We have written about these patterns and more in our Go Style Guide which lives in the repository where we keep many Go libraries and helpers that we frequently use across our projects: kolide/kit. We have many utilities and helpers that make writing modern TLS servers easier, helpers on top of the Go-Kit Logger, and more. We think these libraries can be generically useful and we plan on blogging more about them in the future.

A Consistent Developer Experience using GNU Make

In addition to Fleet, Launcher, Updater, and Kit, we also have several projects in this domain that are internal. Having clearly not gone the way of the monorepo, having some consistencies across all these projects really adds to the developer experience.

Whenever a developer at Kolide creates a new Go repo, they will add a Makefile that follows a similar format.

The basic Makefile structure that every Kolide project uses requires the following commands:

• make deps

• make test

• make build

• ./build/template --help

• ./build/template version

# install the dependency manager and invoke it
# all kolide projects use dep now, but we used to always use glide
# hiding this behind `make deps` made the transition transparent

make deps

# run the full test-suite
# this is (hopefully) just a light wrapper on top of go test
# sometimes we also add linters and static analyzers here too

make test

# build the binary for your platform
# you can also run `make xp` to cross-compile a mac and linux binary

make build

## now you can run the program, get help, and version information

./build/template --help
./build/template version

An example Makefile that implements this above functionality is included here. Note the usage of the version package from kolide/kit which is used to easily add git-based version tooling to a program.

ifndef ($(GOPATH))
    GOPATH = $(HOME)/go
endif

REPO_NAME=template

PATH := $(GOPATH)/bin:$(PATH)
VERSION = $(shell git describe --tags --always --dirty)
BRANCH = $(shell git rev-parse --abbrev-ref HEAD)
REVISION = $(shell git rev-parse HEAD)
REVSHORT = $(shell git rev-parse --short HEAD)
USER = $(shell whoami)

KIT_VERSION = "\
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.appName=${APP_NAME} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.version=${VERSION} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.branch=${BRANCH} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.revision=${REVISION} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.buildDate=${NOW} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.buildUser=${USER} \
    -X github.com/kolide/${REPO_NAME}/vendor/github.com/kolide/kit/version.goVersion=${GOVERSION}"

ifneq ($(OS), Windows_NT)
    CURRENT_PLATFORM = linux

    # If on macOS, set the shell to bash explicitly
    ifeq ($(shell uname), Darwin)
        SHELL := /bin/bash
        CURRENT_PLATFORM = darwin
    endif

    # To populate version metadata, we use unix tools to get certain data
    GOVERSION = $(shell go version | awk '{print $$3}')
    NOW = $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
else
    CURRENT_PLATFORM = windows
    # To populate version metadata, we use windows tools to get the certain data
    GOVERSION_CMD = "(go version).Split()[2]"
    GOVERSION = $(shell powershell $(GOVERSION_CMD))
    NOW = $(shell powershell Get-Date -format s)
endif

all: build
.PHONY: build
build: .pre-build .pre-template
    go build -i -o build/template -ldflags ${KIT_VERSION} ./cmd/template/

deps:
    go get -u github.com/golang/dep/cmd/dep
    dep ensure -vendor-only

.pre-build:
    mkdir -p build/darwin
    mkdir -p build/linux

.pre-template:
    $(eval APP_NAME = template)

xp: .pre-build .pre-template
    GOOS=darwin CGO_ENABLED=0 go build -i -o build/darwin/template -ldflags ${KIT_VERSION} ./cmd/template/
    GOOS=linux CGO_ENABLED=0 go build -i -o build/linux/template -ldflags ${KIT_VERSION} ./cmd/template/
    ln -f build/$(CURRENT_PLATFORM)/template build/template

test:
    go test -cover -race -v ./...

Additionally, consider the minimal main file of this program, which uses the kolide/kit version package and the OKLog command parsing pattern.

These files should live at Makefile and cmd/template/template.go for this example, but obviously paths and the template string would need to be replaced for your usage.

Development Infrastructure: Docker Compose and Minikube

Many of our Go (and non-Go) projects require some sort of infrastructure to enable effective local development. For example, Kolide Fleet requires MySQL and Redis. MailHog is also useful for testing SMTP features. To solve this requirement for local development, we usually use Docker Compose. For example, check out Fleet’s docker-compose.yml.

In the future, I would like to start defining all of this as a Kubernetes Deployment or a ReplicaSet and manage local development infrastructure via Minikube. Since we us Kubernetes in production at Kolide, this parity with the local environment would be an interesting way to further shrink the gap between development and production.

Conclusion

At Kolide, we create and manage client-server software to make operating system analytics easy, modern, and scalable. This domain lends itself very well to what Go is really good at. We love writing the best Go that we can and we’re constantly trying to improve.

If you’d like to read more tutorial content like this, sign up for our biweekly newsletter.