Fluent-Bit, HTTP Output, and JSON Structure

Background

I recently went deep into tweaking and deploying the fluent bit helm chart into my Kubernetes Cluster. Most importantly for this post, I was using the HTTP Output Plugin to send logs to Logz.io (They have a free tier). But every time fluent-bit would push logs, I’d get a HTTP 400 response code.

And so it begins…

Warning

In its current state, the Fluent-Bit Helm Chart must be customized (not just values.yaml), to interpret the additional HTTP output plugin configuration properties.

JSON

When using backend.type.http with Format json set in config, fluent-bit will group multiple messages into a JSON array, and that array composes the body of the HTTP POST request. In this case, Logz.io can’t parse that JSON structure correctly (Their support person confirmed). I tested different structures with cURL:

‘Malformed’ JSON Array

Notice the brackets encapsulating multiple JSON objects, denoting an array.

$ echo "[{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}]" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=UTF-8
< 
* Connection #0 to host example.com left intact
{"malformedLines":1,"oversizedLines":0,"successfulLines":0}

Acceptable JSON Format

$ echo "{\"iamkey\":\"iamvalue\"},{\"who\":\"you\"}" | curl -X POST "https://listener.logz.io:8071" -v --data-binary @-
...
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8

The way to get this working with fluent-bit is using Format json_stream, like so:

[OUTPUT]
        Name  http
        Match *
        Host listener.logz.io
        Port 8071
        URI "?token=PUT_TOKEN_HERE&type=json"
        # Proxy http://proxy-server.internal.com
        tls on
        Format json_stream

References

  • https://github.com/fluent/fluent-bit/issues/680

© 2019. All rights reserved.

Powered by Hydejack v8.5.2