Skip to content

Infrastructure setup and Application setup

In order to deploy FluentD agent to VM for logging purposes, we're making the assumption that your VM is already created and that you will be installing using the shell script.

In order to see the results at an end-user level, your subscription will need to be onboarded into the OSGS-cloud (information on how to do that is coming in April).

Subscription Level Resources

We need to gather the information about the OSGS-central-logging EventHub within your subscription. From the Azure portal, navigate to your subscription and look in the list of resource groups for the one labelled "osgs-central-logging".
Within that resource group, look for a resource of type "Event Hubs Namespace" that is in the form of "osgs-cl-REGION-eventhub-SUBSCRIPTION_FIRST_BLOCK" where the `REGION` is the region that matches the region where the VM resides and the `SUBSCRIPTION_FIRST_BLOCK` corresponds to the first block of characters in your subscription ID before the first '-' character.

Click into that Event Hubs Namespace, and click on the "Event Hubs" element on the left hand side of the Portal window.

Click on the "application-logs" Event Hub, and click on the "Shared access policies" element on the left hand side of the portal.

Since it's always a bad idea to re-use credentials, and always a good idea to give the credentials only the roles that they will need, create a new set by clicking on "+Add" at the top of the pane, and selecting "Send" as the only permission needed for this set of credentials.

Copy the "Connection string - primary key", as you will need to put that in the configuration file in your copy of the example repo.

Modifying the parameters for the target subscription

Follow below steps to update td-agent.conf fie configuration :

  • Login to VM.
  • Navigate to /etc/td-agent/td-agent.conf.
  • Stop the fluentD agent with sudo systemctl stop td-agent.
  • Update the Source, Filter and Match configurations as per below given directions.
  • Start the fluentD agent with sudo systemctl start td-agent.
The values in the in td-agent.conf file at the /etc/td-agent/ that need to be changed to source, filter and match your subscription information are shown in the block below indicated by the various `TAGS` that need to be replaced:

Source:

Path <APPLICATION_LOG_PATH>

<APPLICATION_LOG_PATH> - The file path for application logs which deployed on virtual machine where fluentD agent configured.

Filter: 

agg <AGG-NAME>

type <LOG-TYPE>

<AGG-NAME> - The name of the aggregator that will be a value for the "agg" key when searching in log analytics workspace.

<LOG-TYPE> - A user selectable additional tag that can be set to whatever your team wants it to be. Maybe something like "app-logs".

Match: 

brokers lp-cl-<REGION>-eventhub-<SUBSCRIPTION_FIRST_BLOCK>.servicebus.windows.net:9093

username $ConnectionString

password Endpoint=sb://lp-cl-<REGION>-eventhub-<SUBSCRIPTION_FIRST_BLOCK>.servicebus.windows.net/;SharedAccessKeyName=<LP-SAS-NAME>;SharedAccessKey=<SAS-PRIMARY-KEY>;

default_topic application-logs

The whole connection string can be copied directly from the portal as opposed to replacing the elements. Here is a description of what the different elements represent:

REGION - the region where the target VM resides, usually centralus

SUBSCRIPTION_FIRST_BLOCK - normally the first 8 characters of the target subscription ID

LP-SAS-NAME - the Event Hub Shared Access Policy name

SAS-PRIMARY-KEY - the primary SAS key associated with the Shared Access Policy

Accessing the data processed by OSGS central logging with Log analytics

Application teams can use parse, parseWith and project keywords in query to debug and analyse logs from log analytics workspace.

Application team needs to put config such that it will send resourceId along with each log. It should be inside "additionalFields" key of ness format. For reference visit here