Analyzing Machine Data with Splunk
WHAT IS SPLUNK?
About Me
DevOps@RajeshKumar.XYZ
Analyzing Log Files
Why Learn Splunk?
Basic Windows Administration
Basic Linux Administration
Windows Environment
Hadoop Sandbox
Overview
Why machine data?
Splunk Terms
Careers in Splunk
Machine Data
Machine Data
Data generated by machines, computer processing, applications and sensor data.
Machine data is everywhere. In fact you are generating it right now!
Server & Workstation Logs
- Linux/Windows
- Log files
- Access
- File system
Networks
- Firewall
- Warnings
- Alerts
- IP addresses
Database
- Audit logs
- Configurations
- Schemas
- Tables
- Queries
Web
- Transactions
- Click-stream
- Location
- Browser
- Time
DevOps
- Test logs
- log4j alerts
- Event logs
- Code check-in
IOT
- GPS
- RFID
- Biometric
- Temperature
- Limitless
Splunk Terms
Splunk Workflow
- Index
- Events
- Search
- Pivot
- Dashboard
- Forwarder
Splunk Careers
Security is one of the fastest growing sectors in IT
Who Is Splunk For?
Summary
Definition of Splunk
Understand machine data
A look at Splunkarchitecture
Careers in Splunk
Setting up the Splunk Environment
Overview
How Splunk is licensed
Where to get Splunk
Installing Splunk
Running Splunk
Splunk Licensing
Flavors of Splunk
Capped at 500MB of data
Splunk Cost
Get Splunk
Demo
Splunk Website
Register with Splunk.com
Installing Splunk for Windows
Demo
Installing Splunk
Example Data
Demo
Logging into Splunk
Testing the environment
Summary
- Splunk documentation
- Explained Splunk cost
- Walked through installing Splunk
- Up and running with Splunk
Basic Splunking Techniques
Overview
- Adding more data
- Deep dive into Splunk search
- Reporting in Splunk
- Alert based reports
More Data
Demo
Windows System Logs
Windows Security Logs
Search in Splunk
- Analyze data
- Date Time
- Event IDs
- Etc
- Testing Data
- Setup alerts
- Create dashboards
Demo
Searching in Splunk
Search Commands
SPL
SplunkProcessing Language
Search Commands
source="WinEventLog:*" host="Henson-Lap"Chaining Commands
source="WinEventLog:*" host="Henson-Lap"| command 1 | command 2...Filtering Results
source="WinEventLog:*" host="Henson-Lap"| search EventCode=100Allows for users to filter results in query. For example show results where event code = 100
Remove Duplicates
source="WinEventLog:*" host="Henson-Lap"| dedup EventCodeOnly shows unique events. For example show only Event Codes once
Reports in Splunk
- Ongoing Analysis
- Trends
- Daily Awareness
- Management
- Status
- Pattern
Demo
Developing saved reports
Custom Alerts
- Warnings
- Real-time
- Scheduled
- Problems
- Quicker resolution
- Actionable
Demo
Alerting in Splunk
Summary
- Added more data from local machine
- Understanding of search
- Created reports based of searches
- Developed custom alerts
Splunking in the Enterprise
Overview
- Move logs in Splunk
- Different forwarding options
- Enterprise architecture
- Walk through setting up forwarder
Forwarders
Forwarder
Instance of Splunk that sends data to another instance of Splunk.
|
Universal Forwarder
|
Heavy Forwarder
|
Light forwarder is deprecated as of Splunk6.0
Enterprise Splunk Architecture
Benefits of Forwarding
Load Balancer
Distributing data across multiple Splunk environments
Installing Forwarders
 |
|
Demo
- Download Forwarder
- Ubuntu Server
Summary
- Explained Splunk Forwarders
- Discussed Forwarder Architecture
- Installed Forwarder in VM
Splunking for DevOps and Security
Overview
- Devops optimization with Splunk
- Security strength with Splunk
- Splunk Use Cases in Enterprise
Splunk in DevOps
DevOps
Increased communication between software developers, QA and IT operations.
- Quicker development time
- Less down time
- Faster release of patches
- Enhanced culture
Demo
Uploading DevOps log file
Splunk in Security
- Fraud Detection
- Outside threats
- Data breaches
- Insider threats
Security Monitoring
|
|
Enterprise Use Cases
Data Monitoring
|
|
Summary
- How to use Splunk for DevOps
- Demo analyzing log4j file
- Talked about Security
- Splunk in Marketing
Application Development in Splunkbase
Overview
- Splunkbase defined
- Walk through Splunkbase universe
- Creating apps in Splunkbase
- Setting up Splunkbase environment
What Is Splunkbase?
New Workflows
Splunkbase
Market place for Splunk plug-ins and application. Community driven application with licensed and non-licensed options for Splunk application.
- Microsoft Exchange App
- Isilon Splunk App
- Splunk App for Dropbox
Demo
Navigating the Splunkbase
(https://splunkbase.splunk.com/)
Creating Apps for Splunk
- Dashboard Editor
- Dashboard Editor
- XML
- Dashboard Editor
- XML Editor
- HTML Dashboards
- Dashboard Editor
- XML Editor
- HTML Dashboards
- SplunkJS
SDK Option
Demo
Installing Splunkbase API
Benefits Building in Splunkbase
Splunkbase
Limitless Splunk
Summary
- Learned what Splunkbase
- Toured the Splunkbase
- Discussed benefits of Splunkbase
- Looked at how to create Splunk App
Splunking on Hadoop with Hunk
Overview
- Explain Hadoop
- Walk through Hadoop Environment
- Hunk
- Setup Hunk Environment
- Analyze data in HDFS with Hunk
What Is Hadoop?
Hadoop
Programing framework that processes large data sets in a distributed environment. Two major components MapReduce and HDFS.
MapReduce
HDFS Hadoop Distributed File System
|
Schema on Write
|
Schema on Read
|
Demo
- Hadoop Distributions
- HDFS Demo
- Hortonworks
- Cloudera
- Pluralsight
- HDFS Getting Started
Hunk Defined
Hunk
- Apache Hadoop
- Hortonworks
- MapR
- Pivotal HD
- Cloudera
- Amazon EMR
Demo
- Hunk download
- Hunk install
Demo
HDFS data in Hunk
- Setup Splunk Cluster
- Splunk.com
- Learn Hadoop
- Other analytic tools
Summary
- Understanding of Splunk platform
- Setup dev Splunk environment
- Comfortable in Splunk search
- Value of forwarding in Splunk
- Non IT operations Splunk
- Splunk Marketplace
- Hadoop and Splunk
Questions?
