To make labels exploitable in Kibana

[cabrinoob]

Hi,
I'am using Rancher to orchestrate my containers deployed on many hosts. I choosed Dockbeat as a very lightweight solution to log system metrics in elasticsearch. Thank you for your nice job!

Some of information very usefull to me in the logged data is some rancher labels like "stack name" or "container id". But the way they are stored in ElasticSearch makes very difficult to exploit them correctly in Kibana. Array of objects, even indexed with a nested type index, is'nt exploitable.

So, I modified a little bit the buildLabelArray function in /event/generator.go file like this :

func (d *EventGenerator) buildLabelArray(labels map[string]string) common.MapStr {
	output_labels := make(common.MapStr)
	i := 0
	for k, v := range labels {
		label := strings.Replace(k, ".", "_", -1)
		current := common.MapStr{
			label:v,
		}
		
		output_labels.Update(current);
		i++
	}
	return output_labels
}

Now, I have a flat map of labels like this :

labels.io_rancher_container_ip	:10.42.117.226/16

labels.io_rancher_container_name:Collecte_Gollector_1

labels.io_rancher_container_pull_image:always

labels.io_rancher_container_uuid:8c17d2c0-0c4f-4e8f-8317-370cfa9a8a4f

labels.io_rancher_project_name:Collecte

labels.io_rancher_project_service_name:Collecte/Gollector

labels.io_rancher_service_deployment_unit:08a4a13b-e12e-402d-841a-119a94d7846a

labels.io_rancher_service_launch_config:io.rancher.service.primary.launch.config

labels.io_rancher_stack_name:Collecte

labels.io_rancher_stack_service_name:Collecte/Gollector

Thanks to this I can have aggregated visualizations per Stack in my Dashboards.

Will you consider to "flatten" the labels array in future versions ?

Thank you for your work. That helps us a lot.

[marminthibaut]

IMO it's a good idea 😄

[ruflin]

We did almost the same in the metricbeat docker module: elastic/beats#3024

[dang3r]

Good idea! Can it be incorporated into a new Docker image with a tag indicating it is a breaking change?

[cabrinoob]

If you want to test it I pushed a docker image of my version here : https://hub.docker.com/r/fmeriot/dockbeat-labels/

I use it as a base image like this :

FROM fmeriot/dockbeat-labels:v1
COPY dockbeat.yml /etc/dockbeat/
CMD []

Don't forget the "v1" version

[jdel]

I would be very interested in this too !

[ruflin]

The docker module is now also available in the metricbeat 5.1 release with the label refactoring.

[alangibson]

Any update on this? It would be very useful for our Grafana dashboards.

Apparently related: #133

[ruflin]

@alangibson The docker module in metricbeat has this change. The docker module is dockbeat as a module in metricbeat and was contributed from this code base.