Elastic, Logstash, Kibana and Beans Getting Startted

About Me

DevOps@RajeshKumar.XYZ

ELBK STACK ESSENTIAL


What is the Elastic Stack?

Previously Known as the ELK Stack

Elasticsearch Useful for Many Cases


  • Highly Scalable
  • Built in search, aggregation, and sharding
  • Used by Microsoft Azure, Wordpress, and Stack Exchange

Your Role Today

Take monitoring situation from non-
existent to fully-fledged enterprise-ready

Web-based monitoring and historical searching

Proactive alerting solution

Elasticsearch


Distributed, fast, highly scalable document database

Created by Shay Banon in 2010

We'll use a simple single node cluster

Getting Started with Elasticsearch for .NET Developers

Administering an Elasticsearch Cluster

Logstash


Aggregates, filters, and supplements log data

Forwards altered logs to Elasticsearch

Sending logs directly to Elasticsearch without Logstash can lead to inconsistent data

Kibana


Web-based front-end

Work easily with Elasticsearch for charts, graph, and visualizing data

Free form the Elastic Company

Beats


Small, lightweight utilities for reading logs from a variety of sources. Usually sends data to Logstash

Filebeat: Text log files

Metricbeat: OS and applications

Packetbeat: Network monitoring

Winglogbeat: Windows Event log

Libbeat: Write your own

Alerting


Help track conditions based on Elasticsearch data

Continually monitors log data for pre-configured conditions

Send notifications to email, stack, Hipchat, and PagerDuty out of the box

Summary


Discussed tools needed and how we'll build out the infrastructure

Let's begin building our Elastic Stack and installing software

You should have some experience with Windows & Linux administration to get the most out of the course

Configuring Elasticsearch


Globomantics Is Worldwide

Beats

System Buildout


Start from back, work forward

Usually Elasticsearch clusters comprise many nodes

We're keeping things simple with one Elasticsearch node

What kind of OS should we use?
Elasticsearch runs file on Linux & Windows

We're going to choose Linux and use distribution packages

Ubuntu 16.10 Server Edition

We'll also demonstrate a Windows install

Linux intall looks good, let's do a Windows install

If you're not in mixed / heterogeneous OS environment, this section is for you

yum install java-1.8.0-openjdk-devel

sysctl -w vm.max_map_count=262144

https://goo.gl/621XMQ

Change following in yaml file "elasticsearch.yml" under "config" directory


cluster.name: ibm
node.name
network.host: 172.31.26.75

./elasticsearch

Verify
curl -X GET 'http://localhost:9200'

You can add your first entry with the command:



curl -X POST 'http://localhost:9200/scmgalaxy/helloworld/1' -d '{ "message": "Hello World!" }'

					

You can retrieve this first entry with an HTTP GET request like this:


curl -X GET 'http://localhost:9200/scmgalaxy/helloworld/1'

To modify an existing entry you can use an HTTP PUT request like this:



curl -X PUT 'localhost:9200/tutorial/helloworld/1?pretty' -d '
{
  "message": "Hello People!"
}'

					

It enables human readable format


curl -X GET 'http://localhost:9200/tutorial/helloworld/1?pretty'

IP address:9200

We can override either the cluster or node name



./elasticsearch -Ecluster.name=my_cluster_name -Enode.name=my_node_name 

					

Verify


curl -X GET 'http://localhost:9200'

To retrieve this first entry with an HTTP GET request like this:



$ curl -X GET 'http://localhost:9200/scmgalaxy/helloworld/1'

					

It enables human readable format



$ curl -X GET 'http://localhost:9200/tutorial/helloworld/1?pretty'

					

Summary


Challenges to the monitoring steup

Figured out our plan

Demonstrated how to intall Elasticsearch on Linux and Windows

Next: Logstash

Installing Logstash


Logstash Is a Data Collection Engine

Logstash Configuration

Logstash Plugins


Out of the box can read apache logs, log4j files, Windows Event log, and more...

Included filters can read raw text, parse csv, or look up geo/location information by IP address, or reading json

Dzones of filters are included by default

Logstash Filters

Geoip Filter

Let's create our Logstash server

Ubuntu Linux Server

Located in US EAST

https://goo.gl/HxN7go


drwxrwxr-x 4 logstash logstash 4096 Nov 21 23:08 vender/
root@g1obo-1ogstashe01:/usr/share/1ogstash# bin/logstash -e 'input { stdin { ) ) output { elasticsearch { hosts => ["192.168.0.1
2:9200] } }'
					

rootglobo-logstashe01 :/usr/share/logstash# syste.ctl enable logstash
Created symlink /etc/systemd/system/multi-user.target.wants/logstash.service - /etc/systemd/system/logstash.service.
rootglobo-1ogstashe1 : /usr/share/logstash# service logstash start
rootglobo-logstashe1 : /usr/share/logstash# |
					

To test your Logstash installation, run the most basic Logstash pipeline. For example:



cd logstash-5.4.0 bin/logstash -e 'input { stdin { } } output { stdout {} }' 

					

curl -X GET http://localhost:9200/logstash-*/_search

Summary


Logstash Pipelines and how they will help us

Built a Logstash server and tested the data flow through to Elasticsearch

Next: Kibana

Visualizing with Kibana


Almost Complete

General graphing and visualization tool written in Node.js

Free, works great with Elasticsearch, includes a ton of visualization options and widgets

Easy to create useful dashboards and share them with coworkers

Installing Kibana on Ubuntu is pretty easy

Elastic company maintains .deb packages for Debian-based systems

https://goo.gl/VZgnlz


Processing triggers for ureadahead (0.100.0-19)...
root@globo-kibana01:/home/jtoto# namo / etc/kibana.yaml |
					

# To allow connection from remc
server.host 192.168.0.15?
					

# The Kibana server's name.
server.name: "globo-kibana01"
					

# The URL of the Elasticsearch instance to use for
elasticsearch.url: "http://192.168.e.149200"
					

PS C:winlogbeat-5.1.1-windows-x86_64> start-service winlogbeat 
PS C: winlogbeat -5.1.1 -windows -x86_64> Get -Content . logs\wlnlogbeat -Wait
2016-12-17106:40:12-08:00 INFO Metrics logging every 30s;
2016-12-17T06:40:12-0€:00 INFO winlogbeat start running.
2016-12-1fl06:40:13-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused it.
2016-12-17T06:40:IS-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused it.
2016.1217T06:40:1808:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused It.
2016-12-17T06:40:23-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection
refused It.
					

localhost:5601

Installed and configured Kibana

Kibana will be our primary view into the log data

Next: Configuring client server

Instrumenting Windows Servers


Instrumenting Windows Servers

A Complete Picture

Golang


Go programs are static binaries, no need for JVM or other runtimes

Can be "cross-compiled" to work on Windows, Linux, macOS, and BSD

Usually pretty small and lightweight-great for system uitilities

Usually large companies have dozens, hundreds, or even thousands of server

For our purposes, we're going to use two Windows web servers and one Windows file server

Will keep the data diverse enough for our demonstrations

Download and unpack Winlogbeat

Configure it to use logstash and add some custom fields and data

Set it up to run as a Windows service


6_64> Invoke-WebRequest -Method PUT -InFile . \winlogbeat.template. json -Uri http://192. 168.0. 12:9200/_temp1ate/winlogbeat
					

		s-x86_64> .\install-service-winlogbeat.ps1
					

			_64> start-service winlogbeat
					

PS C:winlogbeat-5.1.1-windows-x86_64> start-service winlogbeat 
PS C: winlogbeat -5.1.1 -windows -x86_64> Get -Content . logs\wlnlogbeat -Wait
2016-12-17106:40:12-08:00 INFO Metrics logging every 30s;
2016-12-17T06:40:12-0€:00 INFO winlogbeat start running.
2016-12-1fl06:40:13-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused it.
2016-12-17T06:40:IS-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused it.
2016.1217T06:40:1808:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection 
refused It.
2016-12-17T06:40:23-08:00 ERR Connecting error publishing events (retrying): dial tcp 192.168.0.14:5043: connectex: No connection
refused It.
					

Configure Logstash to read Beats data and forward it to Elasticsearch


Learn how basic Kibana configuration works

General querying

Visualizations and dashboards

Configure Metricbeat it to use logstash and add some custom fields and data

Set it up to run as a Windows service

Build a new dashboard in Kibana

Summary


Installed and configured Winlogbeat and Metricbeat

Build our first Logstash configuration

Created dashboards in Kibana

Next: Instrumenting Linux

Instrumenting Linux Servers


Beats for Linux

Filebeat


Build for consuming and shipping text-based logs and data

Outputs to Elasticsearch or Logstash

Most Linux logs are text-based so it's a good fit for monitoring

Download,install, and configure Filebeat

Setup Filebeat to read syslog files and forward to Logstash for syslog

Add a filter configuration to Logstash for syslog

Verify data is arriving in Elasticsearch from Filebeat

Setup first Linux dashboard

Summary


Instrumented our first "client" Linux server with Filebeat and Metricbeat

Added complex filtering to Logstash and setup a Kibana dashboard for Linux

Next: Packetbeat

Instrumenting Network Traffic


Examining Network Traffic

Packetbeat


Realtime network packet analyzer

Bundles packets of traffic into transactions which can be charted

Can listen for ICMP, DNS,HTTP, Cassandra, MySQL, PostgreSQL, Redis, Thrift, MongoDB, and Memcache

Download,install, and configure Packetbeat


Examine Packetbeat traffic in Kibana Discovery screen

Create web traffic dashboard

Summary


Installed and configured Packetbeat on Windows.

Learned how to use Packetbeat to visualize network traffic and tracking load

Next: Filebeat, Logstash, and IIS

Instrumenting IIS Logs


Windows Server 2016


Windows Server 2016, IIS

IIS uses the common w3c log formate which is easy to read

IIS Log Example

How Should We Parse IIS Logs


Filebeat

Not just good for reading syslog data

Great for reading any text log data

Configure Filebeat to read IIS logs

Update the Logstash configuration to parse IIS data

https://www.elastic.co/products/beats/filebeat

Examine IIS log data in Kibana

Create a geo location dashboard for request

Summary


Configured Filebeat to read IIS log data and modified Logstash accordingly

Graphed location data of the requesting IP address

Next: Alerting

Alerting with Watcher


Alerting with Watcher

Watcher is a Plugin for Elasticsearch

Watcher Workflow

Install X-Pack onto Elasticsearch

X-Pack must be installed on each Elasticsearch Node

Configure SMTP settings


xpack.notification.email.account:
	globo:
		email defaults:
			from: alerts@globornanticz . corn
			cc: adininiztrator@globornanticz . corn
		smtp:
			auth:false
			host:192.168.0.17
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher—history*
xpack.security.enabled: false
					

./elasticsearch-plugin install x-pack

Setup a Watch inside Elasticsearch using Postman

Scan winlogbeat indices for Windows Eventlog Errors

Questions

Thank You