Splunk

Splunk手册学习之Search Tutorial

一、The Content

  1. Getting started
  2. Uploading the tutorial data
  3. Using the Splunk Search app
  4. Searching the tutorial data
  5. Enriching events with lookups
  6. Creating reports and charts
  7. Creating dashboards

二、Start

2.1.Getting started

下载Splunk Enterprise,免费使用60天,每天有500MB的流量

2.2.Uploading the tutorial data

  • The kind of data
    The datas which is indexed are like the content of next graphic
    078
  • the place of datas which are stored
    When you add data to splunk deployment,the data is processed and transform into a series of individual events that you can view,search,analyze.
    The process of transforming the data is called indexing.During indexing,the incoming data is processed to enable fast searching and analysis.The processed results are stored in the index as events.
    Events are stored in the index as a group of files that fall into two categories:
  1. Raw data ,which is the data that you add to the splunk deployment.The raw data is stored in a compresseed format
  2. index files,which include some metadata files that point to the raw data
  • what is the tutorial data
  1. access.log
175.44.24.82 - - [22/Feb/2019:18:44:40] "POST /product.screen?productId=WC-SH-A01&JSESSIONID=SD7SL9FF5ADFF5066 HTTP 1.1" 200 3067 "http://www.buttercupgames.com/product.screen?productId=WC-SH-A01" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; BOIE9;ENUS)" 307
142.233.200.21 - - [22/Feb/2019:19:20:13] "GET show.do?productId=SF-BVS-01&JSESSIONID=SD6SL8FF4ADFF5218 HTTP 1.1" 404 1329 "http://www.buttercupgames.com/cart.do?action=purchase&itemId=EST-13" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 674
  1. secure.log
Thu Mar 22 2019 00:15:06 mailsv1 sshd[60445]: pam_unix(sshd:session): session opened for user mdubios by (uid=0)
Thu Mar 22 2019 00:15:06 mailsv1 sshd[3759]: Failed password for djohnson from 194.8.74.23 port 3769 ssh2
Thu Mar 22 2019 00:15:08 mailsv1 sshd[5276]: Failed password for invalid user appserver from 194.8.74.23 port 3351 
  1. vendor_sales.log
[13/Apr/2019:18:23:07] VendorID=5037 Code=C AcctID=5317605039838520
[13/Apr/2019:18:23:22] VendorID=9108 Code=A AcctID=2194850084423218
[13/Apr/2019:18:23:49] VendorID=1285 Code=F AcctID=8560077531775179
[13/Apr/2019:18:23:59] VendorID=1153 Code=D AcctID=4433276107716482
  • Use the Add Data wizard
    07100
    07101
    07102
    07103
    07104
    07105
    07106

2.3.using the splunk search app

07107
simple instruction:
1、app menu
2、splunk bar
3、app bar
4、search bar
5、time range picker
6、how to search
7、what to search
8、history search
07108
Host,Source,Sourcetype describle the original data

tutorialdata contains next three sourcetypes:

  • access_combined_wcookie
  • secure
  • vendor_sales

we run a search and display next graphic:
1、app bar
2、search bar
3、time range picker
4、search action buttons which you can printing、sharing and exporting by
5、search results tab
6、search mode menu
7、timeline
8、fields sidebar
9、events viewer
10、save as menu

2.4.specifying time ranges

explore data from Buttercup Games online store using different time ranges
The tutorial data for the Buttercup Games store contains events for a seven day period. The dates of the events are based on the date that you downloaded the tutorial data file
Presets time
07110

Because the data for the Buttercup Games online store is a snapshot of historical data, you will not use the '''Real-time''' preset time ranges in this tutorial

Custom time ranges
07111
if select option "latest" is beginning of today,display 12:00:00AM

2.5.search the tutorialdata

The search assistant
The search assistant will display the terms which contain you content you type in the search bar
retrieve data from index

buttercuogames ( error OR fail* OR severe)

07112
Understanding the results
the result bar contains four kinds of result tab
events patterns statistics visualization
The Events tab displays the Timeline of events, the Display options, the Fields sidebar, and the Events viewer
07113
change display of the event view
List is default
you can change it to table or raw
field sidebar
it contains three part
selected fields
interested fields
all fields
Timeline
it can make you zomm the data

Other resukt tabs
patterns(样式) :by the search results,summary the common patterns
statistics: get thesearch result by searching with transforming commands
visualization: Searches with transforming commands also populate the Visualization tab. The results area of the Visualizations tab includes a chart and the statistics table that is used to generate the chart

2.6.using the fields to search

a field can be multivalued in single event
extracted fields
index time
Default fields and timestamps are extracted, and transforms are applied.
search time
search time field extraction, field aliasing, source type renaming, event type matching, and so on.
search with fields
the filed should apply next three criteria:
filed name is sensitive,but field name is not
filed value can contain wildzard
quotation marks should be needed if filed value contain space

When you first run a search the Selected Fields list contains the default fields host, source, and sourcetype

Run targeted searches

sourcetype=access_combined* status=200 action=purchase
sourcetype=access_* status!=200 action=purchase
(error OR fail* OR severe) OR (status=404 OR status=500 OR status=503)
sourcetype=access_* status=200 action=purchase categoryId=simulation

2.7.using the search languages

view results in statistics tab

sourcetype=access_* status=200 action=purchase | top categoryId

07114

format results on visualization tab
07115
click part "STRATEGY" of above graphic
07116

2.8.using a subsearch

requirements: find the frequent shopper who access the Buttercup Game store
search without a subsearch

sourcetype=access_* status=200 action=purchase | top limit=1 clientip
sourcetype=access_* status=200 action=purchase  clientip="87.194.216.51" | stats count,dc(productId),values(productId) by clientip

07117

search with a subsearch

sourcetype=access_* status=200 action=purchase | top limit=1 clientip | table clientip

07118
Because you specified only the clientip field with the table command, that is the only field returned. The count and percent fields that the top command generated are discarded from the output

sourcetype=access_* status=200 action=purchase [search sourcetype=access_* status=200 action=purchase | top limit=1 clientip | table clientip] | stats count, distinct_count(productId), values(productId) by clientip

By default, subsearches return a maximum of 10,000 results and have a maximum runtime of 60 seconds
Make the search syntax easier to read
ctrl + \
Make the search results easier to understand

sourcetype=access_* status=200 action=purchase [search sourcetype=access_* status=200 action=purchase | top limit=1 clientip | table clientip] | stats count AS "Total Purchased", distinct_count(productId) AS "Total Products", values(productId) AS "Product IDs" by clientip | rename clientip AS "VIP Customer"

2.9.Enabling field lookups

There are five key steps to enabling field lookups:
1、Upload the lookup file
2、Share the uploaded file with the applications
3、Create a lookup definition
4、Share the lookup definition with the applications
5、Optional. Make the lookup definition automatic
Download and uncompress the tutorial lookup file
Prices.csv
07119
Find lookups manager
07120
upload lookup table file
07122
07121

share the lookup table file
07123

add the field lookup definition
07124
07125

sharing the lookup definition with all apps
07126

make the lookup automatic
07127
07128

3.0.search with lookup

  • Display the product names and prices
sourcetype=access_* | stats values(price) AS Price By productName | rename productName AS "Product Name"

07129

  • Display the VIP client purchases
sourcetype=access_* status=200 action=purchase 
    [ search sourcetype=access_* status=200 action=purchase 
    | top limit=1 clientip 
    | table clientip] | stats count AS "Total Purchased",dc(productId) AS "Total Products",values(productId) AS "Product IDs" By clientip | rename clientip AS "VIP Customer"

07130

3.1.create reports and charts

Save a search as a report
run a search

sourcetype=access_* status=200 action=purchase 
    [ search sourcetype=access_* status=200 action=purchase 
    | top limit=1 clientip 
    | table clientip] | stats count AS "Total Purchased",dc(productId) AS "Total Products",values(productName) AS "Product IDs" By clientip | rename clientip AS "VIP Customer"

07131
edit report
07132
07133
find and share report
07135
Sharing is set to Private. Only you can view and edit the report. You can allow other apps to view, edit, or both view and edit the report by changing the report permission
07137
07136

3.2.create a basic chart

customer requirements:

  • The number of times each product is viewed
  • The number of times each product is added to the cart
  • The number of times each product is purchased
sourcetype=access_* status=200 | chart count AS views count(eval(action="addtocart")) AS addtocart count(eval(action="purchase")) AS purchases by productName | rename productName AS "Product Name", views AS "Views", addtocart AS "Adds to Cart", purchases AS "Purchases"

07138
07139

3.3.Create an overlay chart and explore visualization options

use the stats command to count the user actions. The eval command is used to calculate the conversion rates for those actions. For example, how often someone who viewed a product also added the product to their cart

sourcetype=access_* status=200 | stats count AS views count(eval(action="addtocart")) AS addtocart count(eval(action="purchase")) AS purchases by productName | eval viewsToPurchases=(purchases/views)*100 | eval cartToPurchases=(purchases/addtocart)*100 | table productName views addtocart purchases viewsToPurchases cartToPurchases | rename productName AS "Product Name", views AS "Views", addtocart as "Adds To Cart", purchases AS "Purchases"

07140
Format the X-Axis and Y-Axis labels
07141
07142
07145

Format the second Y-Axis
conversion rates such as viewsToPurchases are shown separately
07143
07144

save the revised chart as report
The save operation is the same as above operation of saving the result as a report

3.4.Create a report from a custom chart

sourcetype=access_* | timechart count(eval(action="purchase")) by productName usenull=f useother=f

uses the usenull and useother arguments to ensure that the timechart command counts events that have a value for productName Events that have null values for productName are not included
07146
07147

3.5.Create a report from a sparkline chart

For searches that use the stats and chart commands, you can add sparkline charts to the results table

sourcetype=access_* status=200 action=purchase| chart sparkline(count) AS "Purchases Trend" count AS Total BY categoryId | rename categoryId AS Category

07148

3.6.Create dashboards and panels

create dashboard

sourcetype=access_* status=200 action=purchase | top categoryId

07149
07150
07151
edit dashboard
there two ways to edit dashboard

  • UI
    07152
    you can use add Panel option to create new panel
    you can use add Input option to choose from a list of controls to add to the dashboard, including text, a checkbox, and a time range picker
    you can use the Dark Theme option to change the background appearance of the dashboard
  • Source
    you can edit the XML source for the panel directly

Add controls to a dashboard
07153
07154
07155
07156
07157

3.7.add more panels to dashboards

you add those saved reports to an existing dashboard
Add saved reports to a dashboard
07158
07159
07160
Adding a search to an existing dashboard
07161
07162

If the panel is based on an ad hoc search, you can connect the panel to the shared Time Range Picker. If the panel is a report, you cannot connect it to the shared Time Range Picker. Reports can be scheduled to run at a set time interval

三、Source Download

tutorial data download
prices.csv download

支付宝扫码打赏 微信打赏

若你觉得我的文章对你有帮助,欢迎点击上方按钮对我打赏

扫描二维码,分享此文章

linuxwt's Picture
linuxwt

我叫王腾,来自武汉,2016年毕业后在上海做了一年helpdesk,自学了linux后回武汉从事系统运维的工作,从2017年开始写博客记录自己的学习工作,现在正在进行数据迁移到此博客,目前就职于北京神州新桥科技有限公司,个人的座右铭是:逃脱舒适区才能在闲暇的时候惬意的玩耍。

武汉光谷 https://linuxwt.com

Subscribe to 今晚打老虎

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!

Comments