Docker Swarm / FluentD - Removing a Key

If you are setting up FluentD within a Docker Swarm, with the docker metadata filter plugin you may come across this error when there’s a swarm label with an empty value. In my case, the problem was with this key: docker.labels.com.docker.swarm.task.

Error Message

{
	"type": "illegal_argument_exception",
	"reason": "Can't merge a non object mapping [docker.labels.com.docker.swarm.task] with an object mapping [docker.labels.com.docker.swarm.task]"
}

Example of an event that has this empty label, triggering the error

{
	"message": "2018-06-23T18:18:05+0000 INFO takes the value and converts it to string.\n",
	"type": "http-bulk",
	"fluentd_tags": "final",
	"docker": {
		"id": "611e37729095550c20c8d464ccf6a5fcbdae3db7859441d5b45fdfbe26b74168",
		"name": "testing_random-logger.1.cwx24otbnqhgmszjyhonsp8wo",
		"container_hostname": "611e37729095",
		"image": "chentex/random-logger:[email protected]:f287d3ff193fc99211943ddfcf4fcfcff6acbe2ac7dd8385acb07bcaaec6a41f",
		"image_id": "sha256:759976efd70726fd508dfe7795ebd55cea9b96585f615c558e4ccc28e5d33f12",
		"labels": {
			"com.docker.stack.namespace": "testing",
			"com.docker.swarm.node.id": "gz4fb30qwy2lhq4htazaeyypw",
			"com.docker.swarm.service.id": "qa72pgmzc26akxclz99j170d3",
			"com.docker.swarm.service.name": "testing_random-logger",
			"com.docker.swarm.task": "",
			"com.docker.swarm.task.id": "cwx24otbnqhgmszjyhonsp8wo",
			"com.docker.swarm.task.name": "testing_random-logger.1.cwx24otbnqhgmszjyhonsp8wo"
		}
	},
	"@timestamp": "2018-06-23T18:18:05.275+00:00",
	"stream": "stdout"
}

Config block that removes the JSON key that has an empty value and corrects the error

<filter docker.**>
  @type record_transformer
  remove_keys $['docker']['labels']['com.docker.swarm.task']
</filter>

Victory.

References

  • https://github.com/moby/moby/issues/28806
  • https://discuss.elastic.co/t/using-autodiscover-feature-for-docker-does-not-work-when-running-in-swarm-mode/120342/2

© 2019. All rights reserved.

Powered by Hydejack v8.5.2