{"id":23857,"date":"2021-09-25T07:25:42","date_gmt":"2021-09-25T07:25:42","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=23857"},"modified":"2025-05-04T06:12:23","modified_gmt":"2025-05-04T06:12:23","slug":"how-to-install-and-configure-elastic-logstash","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/how-to-install-and-configure-elastic-logstash\/","title":{"rendered":"How to Install and configure elastic Logstash?"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Step 1 &#8211; Download a Latest version of logstash.<\/h2>\n\n\n\n<p>https:\/\/www.elastic.co\/downloads\/logstash<\/p>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-1\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">$ wget https:<span class=\"hljs-comment\">\/\/artifacts.elastic.co\/downloads\/logstash\/logstash-7.15.0-linux-x86_64.tar.gz<\/span>\n$ tar -zxvf logstash<span class=\"hljs-number\">-7.15<\/span><span class=\"hljs-number\">.0<\/span>-linux-x86_64.tar.gz\n$ cd logstash<span class=\"hljs-number\">-7.15<\/span><span class=\"hljs-number\">.0<\/span>\n\n<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-1\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<h2 class=\"wp-block-heading\">Step 2 &#8211; Configure first pipeline<\/h2>\n\n\n\n<p>$ vi logstash-7.2.0\/first-pipeline.conf<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/devops-school\/ae31d652a28c4529fda361f005652c10.js\"><\/script>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3 &#8211; To verify your configuration, run the following command:<\/h2>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-2\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">$ bin\/logstash -f first-pipeline.conf --config.test_and_exit\nThe --config.test_and_exit option parses your configuration file and reports any errors.\n\nIf the configuration file passes the configuration test, start Logstash <span class=\"hljs-keyword\">with<\/span> the following command:\n\n$ bin\/logstash -f first-pipeline.conf --config.reload.automatic\n\nThe --config.reload.automatic option enables automatic config reloading so that you don\u2019t have to stop and restart Logstash every time you modify the configuration file.\n<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-2\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Example Commands<\/h2>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-3\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">$ bin\/logstash -e <span class=\"hljs-string\">'input { stdin { } } output { stdout {} }'<\/span>\n\nAfter starting Logstash, wait until you see <span class=\"hljs-string\">\"Pipeline main started\"<\/span> <span class=\"hljs-keyword\">and<\/span> then enter hello world at the command prompt:\n\nhello world\n<span class=\"hljs-number\">2013<\/span><span class=\"hljs-number\">-11<\/span><span class=\"hljs-number\">-21<\/span>T01:<span class=\"hljs-number\">22<\/span>:<span class=\"hljs-number\">14.405<\/span>+<span class=\"hljs-number\">0000<\/span> <span class=\"hljs-number\">0.0<\/span><span class=\"hljs-number\">.0<\/span><span class=\"hljs-number\">.0<\/span> hello world\n\nLogstash adds timestamp <span class=\"hljs-keyword\">and<\/span> IP address information to the message. <span class=\"hljs-keyword\">Exit<\/span> Logstash by issuing a CTRL-D command in the shell where Logstash is running.\n<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-3\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Configure Logstash with Elastic Search<\/h2>\n\n\n\n<p>The Logstash pipeline can index the data into an Elasticsearch cluster. Edit the first-pipeline.conf file and replace the entire output section with the following text:<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-4\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">output {\n    elasticsearch {\n        hosts =&gt; &#91; <span class=\"hljs-string\">\"localhost:9200\"<\/span> ]\n    }\n}\n\noutput {\n    elasticsearch {\n        hosts =&gt; &#91; <span class=\"hljs-string\">\"172.31.26.247:9200\"<\/span> ]\n    }\n}<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-4\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">-e flag<\/h2>\n\n\n\n<p>The -e flag enables you to specify a configuration directly from the command line. Specifying configurations at the command line lets you quickly test configurations without having to edit a file between iterations. The pipeline in the example takes input from the standard input, stdin, and moves that input to the standard output, stdout, in a structured format.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Import &amp; Parse CSV Data using Logstash<\/h2>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-5\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">$ mkdir -p csv-data\n\n$ cd csv-data &amp;&amp; { curl -O https:<span class=\"hljs-comment\">\/\/raw.githubusercontent.com\/devopsschool-demo-labs-projects\/elasticsearch\/master\/sample-data\/csv-schema-short-numerical.csv ; cd -; }<\/span>\n\n$ sudo wget -P \/home\/centos\/logstash<span class=\"hljs-number\">-7.15<\/span><span class=\"hljs-number\">.0<\/span> https:<span class=\"hljs-comment\">\/\/raw.githubusercontent.com\/devopsschool-demo-labs-projects\/elasticsearch\/master\/logstash-config-file\/csv-read.conf<\/span>\n\n$ bin\/logstash -f csv-read.conf --config.reload.automatic\n\n$ curl -XGET localhost:<span class=\"hljs-number\">9200<\/span>\/demo-csv\/_search?pretty=trueCopied!<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-5\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-6\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">input {\n  file {\n    path =&gt; <span class=\"hljs-string\">\"\/home\/student\/csv-data\/csv-schema-short-numerical.csv\"<\/span>\n    start_position =&gt; <span class=\"hljs-string\">\"beginning\"<\/span>\n    sincedb_path =&gt; <span class=\"hljs-string\">\"\/dev\/null\"<\/span>\n  }\n}<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-6\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p>Here, the \u201c<strong>file<\/strong>\u201d subsection indicates that a file will be used for the input. Within, we use these options:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>path<\/strong>&nbsp;which indicates the full path to the input file (in our case, the CSV file we prepared earlier).<\/li><li><strong><a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/plugins-inputs-file.html#plugins-inputs-file-start_position\" target=\"_blank\" rel=\"noreferrer noopener\">Start_position<\/a><\/strong>&nbsp;is where we instruct Logstash to read the file from the beginning. This setting only applies to files that are read for the first time. By default, the program would prefer to read from the end, since it expects to process a file that would have data added to it periodically.&nbsp; This way, it could import only the new data whenever it\u2019s added to the end of the CSV file. If the file has been seen before, the next parameter will be used to decide what to do.<\/li><li><strong>sincedb_path<\/strong>&nbsp;points to a database file that keeps track of the last line parsed in an input file (in this scenario, the CSV file). The next time the input file would be parsed, the process would continue from the position recorded in the sincedb file. Here, we set this to \u201c\/dev\/null\u201d so that Logstash won\u2019t be able to record the last line it previously read for that particular file. This way, it will process the entire file each time, from beginning, rather than continue where it left off.<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-7\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">filter {\n  csv {\n      separator =&gt; <span class=\"hljs-string\">\",\"<\/span>\n      skip_header =&gt; <span class=\"hljs-string\">\"true\"<\/span>\n      columns =&gt; &#91;<span class=\"hljs-string\">\"id\"<\/span>,<span class=\"hljs-string\">\"timestamp\"<\/span>,<span class=\"hljs-string\">\"paymentType\"<\/span>,<span class=\"hljs-string\">\"name\"<\/span>,<span class=\"hljs-string\">\"gender\"<\/span>,<span class=\"hljs-string\">\"ip_address\"<\/span>,<span class=\"hljs-string\">\"purpose\"<\/span>,<span class=\"hljs-string\">\"country\"<\/span>,<span class=\"hljs-string\">\"age\"<\/span>]\n  }\n}<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-7\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p>We\u2019re going to use \u201c<strong>csv<\/strong>\u201d as a subsection to specify that this is the type of file we intend to parse. Within that, we declare the following options:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>separator&nbsp;<\/strong>where we pass the comma character \u201c,\u201d. This way, Logstash will know that commas are used as a delimiter to separate fields.<\/li><li><strong>skip_header<\/strong>&nbsp;is important here, telling the parser to ignore the first line, which only indicates column names and not the actual data fields we intend to import.<\/li><li><strong>columns<\/strong>&nbsp;where we specify the names of the columns that will be imported. In this case, we are telling our parser that the first field on each line is an ID, the second is a timestamp, and so on, until the last one which is an age.<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-8\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">output {\nelasticsearch {\nhosts =&gt; <span class=\"hljs-string\">\"http:\/\/localhost:9200\"<\/span>\nindex =&gt; <span class=\"hljs-string\">\"demo-csv\"<\/span>\n}\nstdout {}\n}<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-8\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>elasticsearch<\/strong>&nbsp;subsection instructs our program that we intend to send the data to Elasticsearch.<\/li><li><strong>hosts<\/strong>&nbsp;option specifies the host and port where our Elasticsearch accepts incoming connections.<\/li><li><strong>index<\/strong>&nbsp;option defines the name of the Elasticsearch index where imported data will be stored.<\/li><li><strong>stdout<\/strong>&nbsp;will make the import action display its status output and log information in the terminal.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"674\" height=\"524\" src=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-3.png\" alt=\"\" class=\"wp-image-23876\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-3.png 674w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-3-300x233.png 300w\" sizes=\"auto, (max-width: 674px) 100vw, 674px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Logstash Mutate Filter Plugin<\/h2>\n\n\n\n<p>So far, we\u2019ve only played around with the basics of importing CSV files but we can already see that it\u2019s pretty straightforward.<\/p>\n\n\n\n<p>But that\u2019s only because we haven\u2019t been picky about the exact nature of the data. Other times though, we may need to be specific with what data we use and how. For example, we may need to skip importing some fields that are unnecessary, change the mapping of the selected fields, convert the data type of some values or process them in some other way, by changing all characters to uppercase, for example.<\/p>\n\n\n\n<p>These kinds of scenarios can be configured by adding another entry to our Logstash configuration located in the&nbsp;<strong>filter<\/strong>&nbsp;section which is called&nbsp;<strong>mutate<\/strong>.<\/p>\n\n\n\n<p>To make the next part easier, you can download the configuration file which we are going to be working with. You can do so by typing in the following command:<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-9\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">$ sudo wget -P \/home\/centos\/logstash<span class=\"hljs-number\">-7.15<\/span><span class=\"hljs-number\">.0<\/span> https:<span class=\"hljs-comment\">\/\/raw.githubusercontent.com\/devopsschool-demo-labs-projects\/elasticsearch\/master\/logstash-config-file\/csv-read-drop.conf<\/span>\r\n$ bin\/logstash -f csv-read-drop.conf --config.reload.automatic\r\n$ curl -XGET <span class=\"hljs-string\">\"http:\/\/localhost:9200\/demo-csv-drop\/_search?pretty=true\"<\/span> -H <span class=\"hljs-string\">'Content-Type: application\/json'<\/span> -d<span class=\"hljs-string\">'{  \"size\":1}'<\/span>\r\n$ curl -XGET <span class=\"hljs-string\">\"http:\/\/localhost:9200\/demo-csv-drop\/_mapping\/field\/age?pretty=true\"<\/span>Copied!<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-9\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-10\" data-shcb-language-name=\"PHP\" data-shcb-language-slug=\"php\"><span><code class=\"hljs language-php\">filter {\r\n  csv {\r\n      separator =&gt; <span class=\"hljs-string\">\",\"<\/span>\r\n      skip_header =&gt; <span class=\"hljs-string\">\"true\"<\/span>\r\n      columns =&gt; &#91;<span class=\"hljs-string\">\"id\"<\/span>,<span class=\"hljs-string\">\"timestamp\"<\/span>,<span class=\"hljs-string\">\"paymentType\"<\/span>,<span class=\"hljs-string\">\"name\"<\/span>,<span class=\"hljs-string\">\"gender\"<\/span>,<span class=\"hljs-string\">\"ip_address\"<\/span>,<span class=\"hljs-string\">\"purpose\"<\/span>,<span class=\"hljs-string\">\"country\"<\/span>,<span class=\"hljs-string\">\"age\"<\/span>]\r\n  }\r\n  mutate {\r\n      convert =&gt; {\r\n          age =&gt; <span class=\"hljs-string\">\"integer\"<\/span>\r\n      }\r\n        remove_field =&gt; &#91;<span class=\"hljs-string\">\"message\"<\/span>,<span class=\"hljs-string\">\"@timestamp\"<\/span>,<span class=\"hljs-string\">\"path\"<\/span>,<span class=\"hljs-string\">\"host\"<\/span>,<span class=\"hljs-string\">\"@version\"<\/span>]\r\n  }\r\n}<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-10\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">PHP<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">php<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p>In this example, the filter section has two main entries: \u201c<strong>csv<\/strong>\u201d and \u201c<strong>mutate<\/strong>\u201c.<\/p>\n\n\n\n<p>Keep in mind that the order in which these appear is important. In this case, the parameters from the \u201ccsv\u201d section will be applied first, and only afterwards will the ones from \u201cmutate\u201d be applied.<\/p>\n\n\n\n<p>The&nbsp;<strong>convert<\/strong>&nbsp;section is pretty straight forward. All values in the&nbsp;<strong>age<\/strong>&nbsp;field will be converted to&nbsp;<strong>integer<\/strong>&nbsp;values. For example, the age value of \u201c30.5\u201d will be converted to the integer value \u201c30\u201d.<\/p>\n\n\n\n<p>Under\u00a0<strong>remove_field<\/strong>\u00a0we specify all the fields that we want to remove.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"663\" height=\"437\" src=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-4.png\" alt=\"\" class=\"wp-image-23880\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-4.png 663w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-4-300x198.png 300w\" sizes=\"auto, (max-width: 663px) 100vw, 663px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"670\" height=\"467\" src=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-5.png\" alt=\"\" class=\"wp-image-23882\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-5.png 670w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/image-5-300x209.png 300w\" sizes=\"auto, (max-width: 670px) 100vw, 670px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Step 1 &#8211; Download a Latest version of logstash. https:\/\/www.elastic.co\/downloads\/logstash Step 2 &#8211; Configure first pipeline $ vi logstash-7.2.0\/first-pipeline.conf Step 3 &#8211; To verify your configuration, run&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[5936],"tags":[],"class_list":["post-23857","post","type-post","status-publish","format-standard","hentry","category-elastic"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23857","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=23857"}],"version-history":[{"count":12,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23857\/revisions"}],"predecessor-version":[{"id":23883,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23857\/revisions\/23883"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=23857"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=23857"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=23857"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}