Retrieving Cloudflare logs via Logpull API with feron, and pushing them into Elasticsearch with Filebeat.

The whole shebang can be installed onto K8s cluster with included Helm chart. Chart is published to https://charts.random.io, and registered with Artifact Hub

docker build -t get-logs .

Before launching, make sure to set your CF credentials as environment variables

export CF_ZONE_ID=51e241f08e014feb95d1b2760228d12a
export CF_AUTH_EMAIL=admin@example.com
export CF_AUTH_KEY=51e241f08e014feb95d1b2760228d12a2df50

or modify docker-compose.yaml appropriately (see docs on env_file, and environment usage)

After launching local environment, access Kibana via http://localhost:5601/app/kibana#/discover.

# (re)build
docker-compose build

# launch Elasticsearch, Kibana, and get-logs container instances
docker-compose up -d

# keep an eye on the logs
docker-compose logs -f get-logs

After launching local environment, access Kibana via http://localhost:5601/app/kibana#/discover.

NOTE: since log collection is running on schedule, data will not appear in ES immediately. Keen an eye on the logs, and ./logs directory. ES index will receive data shortly after you see new file appearing, and / or Harvester started for file:... log message in the logs. Create an index pattern while you are waiting.

# launch Elasticsearch container instance
docker run -d \
  --name es \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  docker.elastic.co/elasticsearch/elasticsearch:7.6.2

# launch Kibana container instance
docker run -d \
  --name ki \
  -p 5601:5601 \
  --link es:elasticsearch \
  docker.elastic.co/kibana/kibana:7.6.2

# launch Cloudflare Logpull container instance
docker run -it --rm \
  -e CF_AUTH_EMAIL \
  -e CF_AUTH_KEY \
  -e CF_ZONE_ID \
  -e SAMPLE_RATE="0.01" \
  -e ES_HOST="http://elasticsearch:9200" \
  -e ES_INDEX="cloudflare-test" \
  -e ES_INDEX_SHARD=5 \
  -e ES_INDEX_REPLICAS=0 \
  -e ES_INDEX_REFRESH=10s \
  --link es:elasticsearch \
  get-logs

Open-sourced software licensed under the MIT license.

This repo includes and relies on go-tasks, created by Martin Fabrizzio Vilche. Thank you 🙏, Martin.