Running Grafana in Azure Kubernetes Service

This session will show you how to get started with Azure Container Service (AKS), one of the most powerful ways of running containerized applications in Azure. You'll learn how to set up a Kubernetes cluster using the Azure Cloud Shell and deploy a highly available and scalable web application with only a few commands. We'll then show how easy it is to install Grafana into your cluster using Helm and then scale it up. We will then connect Grafana to our Azure subscription and use to build some great dashboards from our Azure Analytics.

Grafana is an open platform for beautiful analytics and monitoring and also provides an Azure Monitoring plugin.

Daniel Scott-Raynsford

Continuous Delivery Practice Lead, IAG NZ

Microsoft Cloud and Datacenter MVP | @dscottraynsford | Linked-In | Email | GitHub

Content

• What You Will Need
• Prerequisite Knowledge
• What You Will Learn
• Part 1 - Opening a Cloud Shell - 5 min
• Part 2 - Create an Azure Container Service - 30 min
• Introduction to Containers, Docker and Kubernetes - 30 min
• Part 3 - Manage Cluster with Cloud Shell - 5 min
• Part 4 - Configure Helm Tiller - 5 min
• Part 5 - Install Grafana using Helm - 10 min
• Part 6 - Connect Grafana to Azure - 5 min
• Part 7 - Add Some More Workloads - 5 min
• Part 8 - Cleanup After the Workshop - 5 min

Estimated workshop time: 90 min Estimated Azure credit usage: USD 3.00 (as long as you delete the infrastructure straight after completion of the workshop)

Prerequisite Knowledge

• Basic knowledge of Compute and Virutalization (Hyper-V, VMWare, Cloud compute)
• Basic knowledge of text editors (Notepad++ or Vim or VS Code)
• Basic knowledge of using text based consoles (Bash, Cmd, PowerShell)

What You Will Learn

You'll learn the basics in the following skills:

• Create and use the Azure Cloud Shell.
• Use the Azure CLI (az) to create and delete an Azure Container Service.
• Use the Kubernetes tools (kubectl) to deploy and manage highly available container applications.
• Use the Helm tool (helm) to deploy and manage a highly available Grafana cluster and connect it to monitor an Azure VM.

What You Will Need

To complete this workshop you'll need the following:

• A Microsoft Azure Account. You can sign up for a free trial here.
• A computer running Windows, OSX or Linux (desktop OS) with an up-to-date version of either Chrome, Firefox, Edge or Opera.

Part 1 - Opening a Cloud Shell

Azure Cloud Shell is an interactive, browser-accessible shell for managing Azure resources. It provides the flexibility of choosing the shell experience that best suits the way you work. Linux users can opt for a Bash experience, while Windows users can opt for PowerShell.

1. Open Cloud Shell by clicking the Cloud Shell icon:

If you have not previously used Azure Cloud Shell:

1. Click Bash (Linux)

   When you first create a Cloud Shell a storage account will get created for you to store your settings, scripts and other files you might create. This enables you to have access to your own environment no matter what device you're using.

   

2. Select the subscription to create the Storage Account in and click Create storage.

3. The Storage Account will be created and the Cloud Shell will be started:

   

If you have previously used Azure Cloud Shell:

1. Select Bash from the shell drop down:

   

Part 2 - Create an Azure Kubernetes Service

We will now use the Cloud Shell to create a new Azure Kubernetes Service (AKS) cluster that will be used to host our containers.

Any AKS service you create will be publically accessible on the internet. A URL will be automatically assigned to your AKS service that you will be able to use to access your containers and manage your cluster.

1. Launch an Cloud Shell in the Azure Portal or as a standalone console:

   

2. Depending on your type of subscription (Free, Azure Pass etc.) you may have to register the required resource providers. This is because by default many resource providers (types of resource providers) are not registered by default.

   This only needs to be done once for a subscription. To do this, run the following commands in Cloud Shell:

   az provider register --namespace Microsoft.Network
   az provider register --namespace Microsoft.Compute
   az provider register --namespace Microsoft.Storage
   az provider register --namespace Microsoft.ContainerService --wait

   

3. Come up with a name for your Azure Kubernetes Service. The name must contain only letters and numbers and be globally unique because it will be used for the public URLs of your cluster.

4. Run this command in Cloud Shell, but change <set me please> to the name that you specified above. You can also change the location variable to configure the Azure data center to install the AKS cluster into. The version and nodevmsize variables can be used to set the Kubernetes and VM size of the nodes:

   name="<set me please>"
   location="EastUS"
   version="1.10.5"
   nodevmsize="Standard_D2s_v3"

   Important: Please note this value and command down, because if your Cloud Shell closes the value will be removed and you'll have to define it in your Cloud Shell session again by re-running this command.

5. Run this command your Cloud Shell to create a resource group:

   az group create --name $name-rgp --location $location

6. Run this command in Cloud Shell to create a Azure Kubernetes Service cluster:

   az aks create --name $name --resource-group $name-rgp --location $location --dns-name-prefix $name --generate-ssh-keys --node-count 2 --node-vm-size $nodevmsize

   | Note: If you have an Azure Log Analytics workspace you'd like to configure your cluster to log to, you can include the --workspace-resource-id and --enable-addons monitoring parameters in the create command:

   workspaceId="$(az resource list --resource-type Microsoft.OperationalInsights/workspaces --query '[0].id' --o tsv)"
   az aks create --name $name --resource-group $name-rgp --location $location --dns-name-prefix $name --generate-ssh-keys --node-count 2 --node-vm-size $nodevmsize --workspace-resource-id $workspaceId --enable-addons monitoring --kubernetes-version $version

   | To enable automatic HTTP application routing:

   workspaceId="$(az resource list --resource-type Microsoft.OperationalInsights/workspaces --query '[0].id' --o tsv)"
   az aks create --name $name --resource-group $name-rgp --location $location --dns-name-prefix $name --generate-ssh-keys --node-count 2 --node-vm-size $nodevmsize --workspace-resource-id $workspaceId --enable-addons monitoring,http_application_routing --kubernetes-version $version

The AKS cluster will be created in your Azure subscription. This will take at least 10 minutes to complete creation of the AKS.

Part 3 - Manage Cluster with Cloud Shell

Once your AKS has been created you will be able to review the AKS serivce resource that has been created:

The actual infrastructure running the cluster gets installed into another resource group:

The cluster itself should be managed by the AKS resource, not within the actual infrastructure deployed.

Now that our cluster is deployed we need to configure Cloud Shell to be able to manage it.

Kubernetes clusters always expose a management endpoint that the Kubernetes tools and other software can use to control and monitor the cluster with. The FQDN for this endpoint can be located by selecting the AKS cluster resource in the resource group that we deployed to contain our cluster:

We can then configure the kubectl tool to manage this cluster. We could do this manually, but the Azure CLI in Cloud Shell provides a handy way to do this for us.

1. Configure your Cloud Shell to manage your AKS by running the command:

   az aks get-credentials --resource-group $name-rgp --name $name

   

2. Validate our cluster is running by running the command:

   kubectl cluster-info

   

3. Check all nodes in the cluster by running the command:

   kubectl get nodes

Browse the Cluster Dashboard

If you are using the AZ CLI on a Windows machine with a compatible web browser you can use the AZ CLI to browse the cluster dashboard. This will start a small proxy service on your computer that will allow you to access the Cluster Dashboard.

The Cluster Dashboard is a container called kubernetes-dashboard that runs inside the AKS on one of cluster nodes in the kube-system namespace.

| Note: This does not work on Azure Cloud Shell because it lacks a compatible web browser.

1. To run the dashboard a ClusterRole has to be first added because we enabled RBAC on the cluster.

   kubectl create -f https://raw.githubusercontent.com/PlagueHO/Workshop-AKS/master/src/helm/dashboard-clusterrole.yaml

2. To open the Cluster Dashboard:

   az aks browse --resource-group $name-rgp --name $name

Part 4 - Configure Helm Tiller

Helm is essentially a package management system for Kubernetes. It makes locating and installing services in Kubernetes even easier as well as simplifying the updating and upgrading of deployed services. The Helm management tool is already installed in the Azure Cloud Shell, but can also be installed onto a client machine (Windows, MacOS and Linux).

Normally, Helm can just be enabled and installed using helm init. However, this will install into the default Kubernetes namespace which we can't enable role based access (RBAC) control on. The Grafana Helm chart requires RBAC so we'll need to create a new Kubernetes namespace called tiller-world to install it into.

1. Initialize the Helm Tiller with role based access control enabled into a the tiller-world namespace with:

   kubectl create -f https://raw.githubusercontent.com/PlagueHO/Workshop-AKS/master/src/helm/cluster-rbac.yaml
   helm init --service-account tiller

Part 5 - Install Grafana using Helm

Next, we'll install an instance of Grafana into our cluster using Helm and then scale up the replica set to run 2 replicas. We need to pass some configuration information to Helm to tell it how to configure our Grafana service, such as the plugins to include.

1. To install and run Grfana:

   helm install stable/grafana --set "service.type=LoadBalancer,persistence.enabled=true,persistence.size=10Gi,persistence.accessModes[0]=ReadWriteOnce,plugins=grafana-azure-monitor-datasource\,grafana-kubernetes-app"

   | Note: If you've enabled your AKS cluster including HTTP application routing, you can have Grafana automatically register a DNS name in the Azure DNS Zone that was created for your AKS cluster:

   dnsName=$(az aks show -n $name -g $name-rgp --query 'addonProfiles.httpApplicationRouting.config.HTTPApplicationRoutingZoneName' -o tsv)
   helm install stable/grafana --set "service.type=LoadBalancer,persistence.enabled=true,persistence.size=10Gi,persistence.accessModes[0]=ReadWriteOnce,plugins=grafana-azure-monitor-datasource\,grafana-kubernetes-app,ingress.enabled=true,ingress.annotations.kubernetes\.io/ingress\.class=addon-http-application-routing,ingress.hosts[0]=grafana.$dnsName"

   

   

2. Set a variable name from the name of the Grafana service that was started by helm:

   serviceName="<the name of the Kubernetes service that was created>"

3. To get IP Address of the Grafana server:

   kubectl get service $serviceName -o jsonpath="{.status.loadBalancer.ingress[0].ip}"; echo

4. To get admin account password Grafana server:

   kubectl get secret $serviceName -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

5. Open the IP address from step 4 in a web browser and enter the username admin and the password from step 5.

   

Part 6 - Connect Grafana to Azure

Now that our Highly Available Grafana service is running in our AKS cluster we need to connect it to our Azure subscription. This will allow Grafana to ingest monitoring information from Azure.

The Grafana Azure Monitor datasource will require an Azure Service Principal details it can use to connect to Azure.

1. To create a new Service Principal run the following commands in the Azure Cloud Shell:

   clientKey="<set this to a long password>"
   appName="GrafanaAzureMonitor"
   az ad app create --display-name $appName --homepage "http://localhost/$appName" --identifier-uris "http://localhost/$appName"
   appId=$(az ad app list --display-name $appName --query [].appId -o tsv)
   az ad sp create-for-rbac --name $appId --password $clientKey

   

2. You will need to record the following four values from the service principal:

   a. Subsription Id - can be found by running:

   az account list

   b. Tenant Id - returned as the tenant from the output in Step 1. c. Client Id - returned as the appId from the output in Step 1. d. Client Secret - set in the clientKey variable in Step 1.

3. In Grafana, click the settings (cog) icon and then click Add data source:

   

4. Set Azure Monitor as the Type:

   

5. Enter a Name for the datasource and complete the Azure Monitor API Details as per the values found in Step 2.

   | Note: You don't need to complete the Application Insights details | unless you have one you want to ingest information into.

   

6. Click the Save & Test button:

   

If everything has gone to plan, then you have now connected your Grafana service to Azure and you can now create dashboards with it.

Now you're able to create some dashboards to monitor your services in Azure:

If you're new to Grafana, then it is best to look through the Grafana getting started guide.

Part 7 - Add Some More Workloads

One of the most useful and powerful things about AKS and Kubernetes in general is you can simply continue to add workloads (apps/services) to your cluster if you've got spare capacity on it. And of course you can continue to add Nodes if you start to run low on capacity.

Adding another workload to an existing cluster is simple with Helm.

1. Add a Wordpress Application instance to your cluster:

   helm install stable/wordpress

2. Scale up the Wordpress Deployment to have 2 replicas:

   kubectl scale deployment vehement-angelfish-wordpress --replicas=2

3. Configure Autoscaling for the Wordpress Deployment:

   kubectl autoscale deployment vehement-angelfish-wordpress --min=2 --max=4 --cpu-percent=80

4. Add a Redmine (Project Management) Application to your cluster:

   helm install stable/redmine

Part 8 - Cleanup After the Workshop

This step is optional and only needs to be done if you're finished with your cluster and want to get rid of it to save some Azure credit.

Note: If you just want to pause running your cluster, you can actually go in and shut each of the cluster VMs down. This will reduce some compute costs but won't completely delete the cluster. You will still pay for some components.

1. Delete the cluster by running the following command in the Azure Cloud Shell:

   az aks delete --resource-group $name-rgp --name $name --yes

2. Delete the resource group by running this command in the Azure Cloud Shell:

   az group delete --name $name-rgp --yes

Everything will now be cleaned up and deleted and you won't be paying to run an AKS cluster.

Well done! You have taken your first steps into the amazingly powerful world of Containers, Kubernetes and Azure Kubernetes Service. This technology is increadibly powerful and can allow your applications to run virtually anywhere and they will always run the same way.

Thank you!