This article is about ELK (buzz word now) implementation.
The ELK stack consists of Elasticsearch, Logstash, and Kibana.
Logstash is a tool for log data intake, processing, and output. This includes virtually any type of log that you manage: system logs, webserver logs, error logs, and app logs.
Here in this post, 'Logstash' will be replaced by 'AWS CloudWatch' and 'AWS Kinesis Firehose'.
Elasticsearch - Is a NoSQL database that is based on the Lucene search engine. Is a popular open-source search and analytics engine. It is designed to be distributed across multiple nodes enabling work with large datasets. Handle use cases as : Log Analytics, Real-time application monitoring, Click Stream Analytics and Text Search
Here in this post, 'AWS Elasticsearch' Service will be used for 'Elasticsearch' component.
Kibana is your log-data dashboard. It’s a stylish interface for visualizing logs and other time-stamped data.
Enable better grip on your large data stores with point-and-click pie charts, bar graphs, trendlines, maps and scatter plots.
First Implementation – ELK With CloudTrail/CloudWatch (as LogStash)
We’ll try to list few easy steps to do so:
- Go to AWS Elastic Search
- Create ES Domain – amelasticsearchdomain
o Set Access Policy to Allow All/Your Id
- Go to AWS CloudTrail Service
- Create Cloud Trail - amElasticSearchCloudTrail
o Create S3 Bucket – amelasticsearchbucket (Used to hold cloudtrail data)
o Create CloudWatch Group - amElasticSearchCloudWatchGroup
o In order to deliver CloudTrail events to CloudWatch Logs log group, CloudTrail will assume role with below two permissions
§ CreateLogStream: Create a CloudWatch Logs log stream in the CloudWatch Logs log group you specify
§ PutLogEvents: Deliver CloudTrail events to the CloudWatch Logs log stream
- Go & Setup Cloud Watch,
- Select Group and Then Action to Stream data to Elastic Search Domain
o Create New Role - AM_lambda_elasticsearch_execution
o Create Lambda (Automatically) LogsToElasticsearch_amelasticsearchdomain - CloudWatch Logs uses Lambda to deliver log data to Amazon Elasticsearch Service / Amazon Elasticsearch Service Cluster.
- Go to Elastic Search
o Hit Kibana link
o On Kibana - Configure an index pattern
Second Implementation – ELK With AWS KinesisFirehose/CloudWatch (as LogStash)
We’ll try to list few easy steps to do so:
- Go to AWS Elastic Search
- Create ES Domain - amelasticsearchdomain
o Set Access Policy to Allow All/Your Id
- Create Kinesis Firehose Delivery Stream - amelasticsearchkinesisfirehosestream
o Attach it to above ES Domain
o Create Lambda (Optional) - amelasticsearchkinesisfirehoselambda
o Create S3 Bucket for Backup - amelasticsearchkinesisfirehosebucket
o Create Role - am_kinesisfirehose_delivery_role
- Create EC2 System - (To send log data to above configured Kinesis Firehose)
o This will be using 1995 NASA Apache Log (http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html) to feed into Kinesis Firehose.
o EC2 used the Amazon Kinesis Agent to flow data from my file system into my Firehose stream.
o Amazon Kinesis Agent is a standalone Java software application that offers an easy way to collect and send data to Amazon Kinesis and to Firehose
- Steps:
- Launch an EC2 Instance (t2.micro) running the Amazon Linux Amazon Machine Image (AMI)
- Putty into instance/etc/aws-kinesis/agent
- Install Kinesis Agent - sudo yum install –y aws-kinesis-agent
- Go to directory - /etc/aws-kinesis/
- Open file - nano agent.json
- Make sure it has this data:
{
"cloudwatch.emitMetrics": true,
"firehose.endpoint": "https://firehose.us-east-1.amazonaws.com",
"flows": [
{
"filePattern": "/tmp/mylog.txt",
"deliveryStream": "amelasticsearchkinesisfirehosestream",
"initialPosition": "START_OF_FILE"
}
]
}
- Now Download NASA access log file in your local desktop and Upload to S3
- URL - http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html
- File download - Jul 01 to Jul 31, ASCII format, 20.7 MB gzip compressed,
- Unzip and uplaod this file to any S3 bucket (other than any used above)
- Make sure file is Public
- Again go to EC2 Putty
- Go to directory - /etc/aws-kinesis/
- Downlaod file from S3 - wget https://s3-us-west-1.amazonaws.com/arunm/access_log_Jul95
- Concatenate this file to mylog.txt - cat access_log_Jul95 >> /tmp/mylog.txt
- Again go to EC2 Putty
- Come to root - cd ~
- Go to directory - /var/log/aws-kinesis-agent/
- Monitor the agent’s log at /var/logs/aws-kinesis-agent/aws-kinesis-agent.log.
- Open file - nano aws-kinesis-agent.log
- You’ll find log lines like : 2017-03-01 21:46:38.476+0000 ip-10-0-0-55 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 1891715 records parsed (205242369 bytes), and 1891715 records sent successfully to destinations. Uptime: 630024ms
- Create Kibana ( To Visualize data)
o Go to AWS Elasticsearch
o Click on link to Kibana
o The first thing you need to do is configure an index pattern. Use the index root you set when you created the Firehose stream (in our case, logs*).
o Kibana should recognize the logs indexes and let you set the Time-field name value. Firehose provides two possibilities:
§ @timestamp – the time as recorded in the file
§ @timestamp_utc – available when time zone information is present in the log data
o Choose either one, and you should see a summary of the fields detected.
o Select the discover tab, and you see a graph of events by time along with some expandable details for each event.
o As we are using the NASA dataset, we get a message that there are no results. That’s because the data is way back in 1995.
o Expand the time selector in the top right of the Kibana dashboard and choose an absolute time. Pick a start of June 30, 1995, and an end of August 1, 1995. You’ll see something like this.
Hope this helps.
Regards,
Arun Manglick
