{"id":4916,"date":"2018-04-17T10:42:45","date_gmt":"2018-04-17T10:42:45","guid":{"rendered":"http:\/\/www.scmgalaxy.com\/tutorials\/?p=4916"},"modified":"2018-06-22T06:14:04","modified_gmt":"2018-06-22T06:14:04","slug":"logstash-explained-in-5-mins","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/logstash-explained-in-5-mins\/","title":{"rendered":"Logstash explained in 5 mins"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-5035\" src=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash.png\" alt=\"\" width=\"600\" height=\"350\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash.png 600w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-300x175.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/p>\n<p><strong>What is Logstash?<\/strong><br \/>\nLogstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite \u201cstash.\u201d<\/p>\n<p><a href=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash.jpg\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-4917\" src=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash-300x195.jpg\" alt=\"\" width=\"300\" height=\"195\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-300x195.jpg 300w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash.jpg 534w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><strong>Logstash Benefits<\/strong><\/p>\n<ul>\n<li>Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.<\/li>\n<li>Logstash offers pre-built filters, so you can readily transform common data types, index them in Elasticsearch, and start querying without having to build custom data transformation pipelines.<\/li>\n<li>With over 200 plugins already available on Github, it is likely that someone has already built the plugin you need to customize your data pipeline.<\/li>\n<\/ul>\n<p><strong>Logstash work in 3 phases&#8230;.<\/strong><\/p>\n<p><a href=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash-architecture1.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-4918\" src=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash-architecture1-300x88.png\" alt=\"\" width=\"300\" height=\"88\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture1-300x88.png 300w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture1-768x225.png 768w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture1-1024x299.png 1024w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture1.png 1204w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><strong>Phase 1 &#8211;<\/strong> When it comes from ingests data from a multitude of sources simultaneously, which includes files, s3,beats, kafka etc. Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time.<br \/>\nList of sources from where logstash can ingest the data are as follows;<br \/>\nhttps:\/\/www.elastic.co\/guide\/en\/logstash\/current\/input-plugins.html<\/p>\n<p><strong>Phase 2 &#8211;<\/strong> Next, It Parse &amp; Transform Your Data On the Fly. As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for easier, accelerated analysis and business value. Logstash dynamically transforms and prepare your data regardless of format or complexity<\/p>\n<p><strong>Phase 3 &#8211;<\/strong> Last, Logstash stored the parsed data into Elasticsearch, aws,hadoop, Mongodb and go-to output that opens up a world of search and analytics possibilities.Logstash has a variety of outputs that let you route data where you want, giving you the flexibility to unlock a slew of downstream use cases. Some of these are given below;<br \/>\nhttps:\/\/www.elastic.co\/guide\/en\/logstash\/current\/output-plugins.html<\/p>\n<p><strong>Where you can use the Logstash?<\/strong><\/p>\n<ol>\n<li><strong>Log Analytics &#8211;\u00a0<\/strong>Ingest un-structured and semi-structured logs generated by servers, applications, mobile devices, and more for a wide variety of applications such as digital marketing, application monitoring, fraud detection, ad tech, gaming, and IoT. Logstash provides plugins to quickly load data from a variety of data sources.<\/li>\n<li><strong>IT Operations Monitoring &#8211;<\/strong> Capture server logs and push them into your Elasticsearch cluster using Logstash. Elasticsearch indexes the data and makes it available for analysis in near real-time (less than one second). You can then use Kibana to visualize the data and perform operational analyses like identifying network issues and disk I\/O problems. Your on-call teams can perform statistical aggregations to identify root cause and fix issues.<a href=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash-architecture-elk.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4919\" src=\"http:\/\/www.scmgalaxy.com\/tutorials\/wp-content\/uploads\/2018\/04\/logstash-architecture-elk.png\" alt=\"\" width=\"748\" height=\"201\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture-elk.png 748w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2018\/04\/logstash-architecture-elk-300x81.png 300w\" sizes=\"auto, (max-width: 748px) 100vw, 748px\" \/><\/a><iframe loading=\"lazy\" width=\"760\" height=\"428\" src=\"https:\/\/www.youtube.com\/embed\/gUJvP2OZENk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>What is Logstash? Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite&#8230; <\/p>\n","protected":false},"author":1,"featured_media":5036,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[4898],"tags":[987],"class_list":["post-4916","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-logstash","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4916","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4916"}],"version-history":[{"count":3,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4916\/revisions"}],"predecessor-version":[{"id":5038,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4916\/revisions\/5038"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media\/5036"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4916"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4916"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4916"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}