Grafana and AWS - Implementation and Usage

Grafana and AWS - Implementation and Usage

• 1.
  Grafana and AWS- Implementation and Usage Table of Contents Grafana and AWS - Implementation and Usage ....................................................................... 1 What is Grafana?......................................................................................................................... 2 Exploring metrics and logs ......................................................................................................... 2 Grafana Implementation ............................................................................................................. 2 Adding AWS CloudWatch as Datasource in Grafana ................................................................ 4 Grafana Dashboard for Amazon AWS ECS............................................................................... 7 Importing the Amazon AWS ECS Dashboard in Grafana.......................................................... 7 Adding load on SampleApp-Web service................................................................................. 10 Grafana Dashboard for AWS Cloudwatch Data....................................................................... 11 Grafana CloudWatch Logs Insights Custom Queries............................................................... 14 Grafana in Production............................................................................................................... 16 Conclusion ................................................................................................................................ 17 References................................................................................................................................. 17
• 2.
  What is Grafana? Grafanais an open source visualization and analytics software. It allows you to query, alert, visualize, and explore your metrics no matter where they are stored. It provides you with the tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations. Exploring metrics and logs Explore your data through ad-hoc queries and dynamic drilldown. Split view and compare different time ranges, queries and data sources side by side. Grafana Implementation Though there are several methods of implementing Grafana, we shall broadly look into its installation and usage on:  Linux  AWS  Docker Implementing Grafana is relatively simple and involves few steps in the installation. Its usage is rather complex and can involve several steps to accomplish stages from data collection, data sources, database connections and finally querying and visualizations. For Linux, visit Install Grafana webpage covers the most popular Linux distributions like CentOS, Fedora, OpenSuse, RedHat and others. For AWS, one can follow Amazon Managed Service for Grafana and Grafana Integration with CloudWatch sections. For Docker, it is also given on the link above in “Install Grafana” for Linux. Grafana is only for visualizations and requires a database connection to fetch data to display several types of graphs. There are several exercises available for monitoring and visualization.
• 3.
  Following are afew images taken from such exercises carried out for Grafana implementation. | Figure: Grafana dashboard connected to influxDB Figure: Prometheus dashboard
• 4.
  Adding AWS CloudWatchas Datasource in Grafana Run the following docker-compose.yml to start the Grafana container: version: '3' services: grafana: image: grafana/grafana container_name: grafana networks: - grafana environment: - GF_PATHS_DATA=/var/lib/grafana ports: - 3000:3000 volumes: - ./data:/var/lib/grafana networks: grafana: {} # docker-compose up -d Creating grafana ... done This should show output as here: [root@ns3078348 cloudwatch1]# docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a49b3eee6e74 grafana/grafana "/run.sh" About an hour ago Up About an hour 0.0.0.0:3000->3000/tcp grafana Access Grafana in Web UI Server Link : http://188.165.220.196:3000/login Username : admin Password : admin
• 5.
  Figure: Grafana DashboardLogin Screen Now let’s add AWS CloudWatch as Datasource in Grafana. Figure: Click on Add data source and select CloudWatch
• 6.
  Figure: Provide AWSIAM Access key ID, Secret access key, and Region Figure: Click on Save & Test to check if Cloudwatch datasource is working After adding the Cloudwatch datasource, we can now get data from Amazon AWS ECS as has been configured in the last few steps.
• 7.
  Grafana Dashboard forAmazon AWS ECS Many Grafana Dashboards are readily available here https://grafana.com/grafana/dashboards that can be imported inside the Grafana user interface to perform specific analytics operations and visualizations. Figure: Dashboard for Amazon AWS ECS Source: https://grafana.com/grafana/dashboards/551 Importing the Amazon AWS ECS Dashboard in Grafana To import the above given dashboard in Grafana, ensure that the AWS Cloudwatch data source has been added already.
• 8.
  Figure: On theleft side, click on the + icon and import Figure: Provide the Dashboard number, here which is 551 for AWS ECS
• 9.
  Figure: Verify thedashboard being imported Figure: Dashboard fetches ECS clusters and services
• 10.
  Adding load onSampleApp-Web service Create the script given below. [root@ns3078348 cloudwatch1]# vi ab.sh ab -n 100000 -c 1000 http://example.com/ ab -n 100000 -c 1000 http://example.com/ ab -n 100000 -c 1000 http://example.com/ ab -n 100000 -c 1000 http://example.com/ ab -n 100000 -c 1000 http://example.com/ ab -n 100000 -c 1000 http://example.com/ Run the script. It will run the commands in batch mode, just to make the test run continuously for long enough. ################# OUTPUT ################# [root@ns3078348 cloudwatch1]# sh ab.sh This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking example.com (be patient) Completed 10000 requests Completed 20000 requests Completed 30000 requests Completed 40000 requests Completed 50000 requests Completed 60000 requests Completed 70000 requests Completed 80000 requests Completed 90000 requests Completed 100000 requests Finished 100000 requests Server Software: nginx/1.16.0 Server Hostname: example.com Server Port: 80 Document Path: / Document Length: 3032 bytes Concurrency Level: 1000 Time taken for tests: 21.355 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 326600000 bytes HTML transferred: 303200000 bytes Requests per second: 4682.65 [#/sec] (mean) Time per request: 213.554 [ms] (mean) Time per request: 0.214 [ms] (mean, across all concurrent requests) Transfer rate: 14935.08 [Kbytes/sec] received
• 11.
  Connection Times (ms) minmean[+/-sd] median max Connect: 98 104 40.1 102 1136 Processing: 99 106 22.7 103 959 Waiting: 99 106 20.2 103 728 Total: 197 210 46.4 205 1245 Percentage of the requests served within a certain time (ms) 50% 205 66% 207 75% 209 80% 210 90% 214 95% 219 98% 246 99% 271 100% 1245 (longest request) Grafana Dashboard for AWS Cloudwatch Data After running the script, here is what we see on the Grafana dashboard. Figure: On the top right, change interval to Last 5 minutes and refresh rate to 5 seconds
• 12.
  Figure: The loadstarted at 15:44 as per this graph Figure: CPU utilisation starts linearly, becomes constant and decreases in 3 minutes
• 13.
  Figure: Graph afterChanging interval to Last 1 hour on top right Figure: SampleApp-Web service Logs on CloudWatch
• 14.
  Figure: Dashboard viewof CloudWatch ECS Container Insights After we are done with the Grafana’s testing of metrics and visualisation, it may be desirable to stop the container service. Run the following command: [root@ns3078348 cloudwatch1]# docker-compose down Stopping grafana ... done Removing grafana ... done Removing network cloudwatch1_grafana This will also close the ports open for public access. Grafana CloudWatch Logs Insights Custom Queries We require monitoring several metrics from Example applications. Writing custom queries is the starting point to reach several such scenarios, like:  monitoring the number of successful events per minute  monitoring the number of unsuccessful events per minute  monitoring the time taken by an event’s iteration  monitoring the total time taken by an API call Using the same Cloudwatch Data Source in Grafana UI open the explore window on the left side tool bar, and browse Cloud Watch logs.
• 15.
  Run the GrafanaCloudWatch Logs Insights Query as given below for SampleApp-API Logs Group : fields @timestamp, @message | filter @message like /WARN/ | sort @timestamp desc | limit 25 Figure: Custom CloudWatch Logs Insights Query for SampleApp-API Logs Group The query here is to extract all lines in the logs with a “WARN“ message and display the time on the graph when the log message was generated.
• 16.
  Grafana in Production Grafanacan be setup in production by following any of the methods of implementation as given in the beginning of this article. Once you set it up, considering that we are using the example.com domain with SSL, Grafana will be available on the following URL: Web URL : https://grafana.example.com:3000/login Login user : admin Password : your-password Figure: Grafana in Production
• 17.
  Figure: Grafana Dashboards Conclusion Grafanais flexible to provide a wide variety of visualizations using data from several types of sources. Activities on Grafana is a continuous process and we are likely to see more action with this tool in future. References Grafana documentation https://grafana.com/docs/grafana/latest AWS CloudWatch data source https://grafana.com/docs/grafana/latest/datasources/cloudwatch Analyzing Log Data with CloudWatch Logs Insights https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html CloudWatch Logs Insights Query Syntax https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html