Skip to content

Fluentd implementation on Azure Kubernetes Service

Prerequisite

  • Env variables setup, to be done before running terraform
export ARM_CLIENT_ID=<client id>
export ARM_CLIENT_SECRET=<client secret>
export ARM_TENANT_ID=<tenant id>
export ARM_SUBSCRIPTION_ID=<subscription id>

  • Resource group should be already created in the Azure, to setup a resource group you can:

    • goto portal and create manually
    • add resource group as a resource in terraform, to read more about it goto: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group
    • create resource group via az commands, to read mote about it goto: https://learn.microsoft.com/en-us/cli/azure/group?view=azure-cli-latest#az-group-create

  • Do the neccessary changes as per your need in the fluentd-config.yaml

Steps to create

STEP 1. Create Aks from either azure portal or az commands or terraform. - setup context for AKS

az aks get-credentials -g $RESOURCE_GROUP -n $CLUSTER_NAME -a --overwrite-existing
  • add url for helm repo 
helm repo add helm-virtual https://repo1.uhc.com/artifactory/helm-virtual
  • repo update 
helm repo update

STEP 2. Attach aks and acr with each other, either while creating aks from portal or with this command:

az aks update -n {namespace} -g {resource group} --attach-acr {acr name}

STEP 3. Change source plugin and match plugin in the fluentd-config.yaml as per your need (currently its setup for sending ness app logs to a particular eventhub).
STEP 4. Setup fluentd with appropriate config file with the below command

helm install fluent bitnami/fluentd -f fluentd-config.yaml

STEP 5. Create container and pods of application, here we have taken a simple .sh file which gives logs every 5 seconds.

# This is a simple pod with sh file which echos a message at every 5 second
kubectl apply -f dummy-app/dummy-ness-app.yaml

# This is sample java implementation of ness format logs
kubectl apply -f ness-app.yaml
  • you can checkout bitnami's fluentd implementation here: (https://github.com/bitnami/charts/tree/main/bitnami/fluentd/#installing-the-chart)
  • To verify that Fluentd has started, run:
kubectl get all -l "app.kubernetes.io/name=fluentd,app.kubernetes.io/instance=fluent"

Logs are captured on each node by the forwarder pods and sent to the standard output by default. You can see all the logs by running this command:

    kubectl logs -l "app.kubernetes.io/component=forwarder"

STEP 6. Observe the logs going to configured eventhub

Config for fluentd


  fluentd.conf: |-
    # Ignore fluentd own events
    <match fluent.**>
      @type null
    </match>

    @include fluentd-inputs.conf
    @include fluentd-output.conf
    @include metrics.conf
  fluentd-output.conf: |-
    # Throw the healthcheck to the standard output instead of forwarding it
    <match kubernetes.**>
      @type kafka2

      # list of seed brokers
      # put your eventhub name instead of !!EventhubNameSpace!!
      brokers !!EventhubNameSpace!!.servicebus.windows.net:9093

      # topic settings: put eventhub instance name here.
      default_topic "!!EventhubInstance!!"

      <format>
         @type json
      </format>

      ssl_ca_certs_from_system true

      # producer settings
      max_send_retries 1
      required_acks -1

      username $ConnectionString

      # put your eventhub connection string instead of !!EventhubConnectionString!!
      password !!EventhubConnectionString!!
    </match>
    <match fluentd.healthcheck>
      @type stdout
    </match>
    # Send the logs to the standard output
    # <match **>
    #   @type stdout
    # </match>
  metrics.conf: |-
    # Prometheus Exporter Plugin
    # input plugin that exports metrics
    <source>
      @type prometheus
      port 24231
    </source>
    # input plugin that collects metrics from MonitorAgent
    <source>
      @type prometheus_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    # input plugin that collects metrics for output plugin
    <source>
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
    # input plugin that collects metrics for in_tail plugin
    <source>
      @type prometheus_tail_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>
  fluentd-inputs.conf: |-
    # HTTP input for the liveness and readiness probes
    <source>
      @type http
      port 9880
    </source>
    # Get the logs from the containers running in the node
    <source>
      @type tail

      # change path based on your need
      path /var/log/containers/*.log
      # exclude Fluentd logs
      exclude_path /var/log/containers/*fluentd*.log
      pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type regexp
        expression ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[^Z]*Z)\s(?<stream>[^\s]+)\s(?<character>[^\s])\s(?<message>.*)$
      </parse>
    </source>
    <filter kubernetes.**>
      enable_ruby
      @type record_transformer
      renew_record true
      <record>
        type kubernetes-logs
        agg app
        data ${record}
      </record>
    </filter>
    <filter kubernetes.**>
      @type stdout
    </filter>

Ness formatted logs in the eventhub

For java app use the following command: - First go into the app using below command

kubectl exec {podname} -it /bin/sh
  • Paste the below command after you are inside the pod
wget http://localhost:8080/echo

Troubleshooting

STEP 1. Check if fluentd pod is up and running and no visible logs are there. STEP 2. Check the if the below logs are successfully coming in the fluentd pod.

2023-03-15 18:08:24 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"

2023-03-20 10:28:11 +0000 [info]: #0 following tail of /var/log/containers/ness-app_default_ness-app-39490fb7550a8f56c12e83ede69c60da483bc228c1ac4843a4065840cfe1f8ca.log

If non or one of these logs are missing its probably issue with your config.

STEP 3. Check if your file is followed but its coming in "not matched" logs, then your regex pattern in fluentd config is wrong.
STEP 4. Match pattern for kafka should also be registered, which looks like this.

2023-03-20 10:26:10 +0000 [info]: adding match pattern="kubernetes.**" type="kafka2"

STEP 5. Eventhub should be kafka enabled.

Notes

  • If you already have infrastructure created and want to just install fluentd, you can use this command directly
sh ./fluentd-deploy.sh $ARM_CLIENT_ID $ARM_CLIENT_SECRET $ARM_TENANT_ID $SUBSCRIPTION_ID $RESOURCE_GROUP $CLUSTER_NAME $EVENTHUB_NAMESPACE $EVENTHUB_INSTANCE_NAME