Splunk手册学习之Knowledge Manager Manual-1

一、The Content

  1. Welcome to knowledge management
  2. Get started with knowledge objects
  3. Fields and field extractions
  4. Use the field extractor in Splunk Web
  5. Use the settings pages for field extractions in Splunk Web
  6. Use the configuration files to configure field extractions\
  7. Calculated fields
  8. Event types
  9. Transactions
  10. Use lookups in Splunk Web
  11. Use the configuration files to configure lookups
  12. Workflow actions
  13. Tags
  14. Field aliases
  15. Search macros
  16. Manage and explore datasets
  17. Create and edit table datasets
  18. Build a data model
  19. Define data model dataset fields
  20. Use data summaries to accelerate searches


2.1.Welcome to knowledge management

What is Splunk knowledge?
Some of this information is extracted at index time, as Splunk software indexes your IT data. But the bulk of this information is created at "search time," both by Splunk software and its users
You can think of Splunk software knowledge as a multitool that you use to discover and analyze various aspects of your IT data
Splunk software knowledge is grouped into five categorie:

  • Data interpretation: Fields and field extractions
  • Data classification: Event types and transactions
  • Data enrichment: Lookups and workflow action
  • Data normalization: Tags and aliases
  • Data models

TIP1:Data models are representations of one or more datasets, and they drive the Pivot tool, enabling Pivot users to quickly generate useful tables, complex visualizations, and robust reports without needing to interact with the Splunk software search language
TIP2: A typical data model makes use of other knowledge object types discussed in this manual, including lookups, transactions, search-time field extractions, and calculated fields

  • Summary-based report and data model acceleration
    report acceleration (for searches), data model acceleration (for pivots)

Why manage Splunk knowledge?
Splunk knowledge managers provide centralized oversight of Splunk software knowledge. The benefits that knowledge managers can provide include:

  • Oversight of knowledge object creation and usage across teams, departments, and deployments
  • Normalization of event data、
  • Management of knowledge objects through configuration files
  • Creation of data models for Pivot users
  • Manage setup and usage of summary-based search and pivot acceleration tools

Prerequisites for knowledge management
Most knowledge management tasks are centered around search time event manipulation
a typical knowledge manager usually doesn't focus their attention on work that takes place before events are indexed, such as setting up data inputs, adjusting event processing activities, correcting default field extraction issues, creating and maintaining indexes, setting up forwarding and receiving

2.2.Get started with knowledge objects

Manage knowledge objects through Settings pages
1、Save and schedule searches
2、Add tags to fields
3、Define event types and transactions that group together sets of events
4、Create lookups and workflow actions
study with document
The sequence of search-time operations
When you run a search, the Splunk software runs several operations to derive various knowledge objects and apply them to the events returned by the search. These knowledge objects include extracted fields, calculated fields, lookup fields, field aliases, tags, and event types
Inline field extractions:do not include a field transform reference、explicit
Field extraction that uses a field transform:include a field transform reference、explicit
Automatic key-value field extraction:not explicit
Field aliasing
Calculated fields:Configurations that create one or more fields through the calculation of eval expressions and add those fields to events
Lookups:here are four distinct types of lookup configurations:CSV lookups, external lookups, KV Store lookups, and geospatial lookups
Event types
Give knowledge objects of the same type unique names
you should not have two inline field extraction configurations that have the same <class> value ,but you can have an inline field extraction, a transform field extraction, and a lookup that share the same name, because they belong to different knowledge object types
Configurations sharing a host, source, or source type

LOOKUP-table = logs_per_day host OUTPUTNEW average_logs AS logs_per_day
LOOKUP-table = location host OUTPUTNEW building AS location

the last LOOKUP-table configuration in that stanza overrides the one that precedes it. The Splunk software adds the location field to your matching events, but does not add the logs_per_day field to any of them
modify the configuration

LOOKUP-table = logs_per_day host OUTPUTNEW average_logs AS logs_per_day
LOOKUP-location = location host OUTPUTNEW building AS location

Configurations belonging to different hosts, sources or source types

LOOKUP-splk_host = splk_global_lookup search_name OUTPUTNEW global_code

LOOKUP-splk_host = splk_searcher_lookup search_name OUTPUTNEW search_code

Any events that overlap between these two lookups are only affected by one of them

  • Events that match the host get the host lookup
  • Events that match the source type get the source type lookup
  • Events that match both get the host lookup

Develop naming conventions for knowledge objects
Naming conventions can help with object organization, but they can also help users differentiate between groups of reports, event types, and tags that have similar uses
And they can help identify a variety of things about the object that may not even be in the object definition, such as what teams or locations use the object, what technology it involves, and what it is designed to do
Set up a naming convention for reports
Possible reports using this naming convention:

  • SEG_Alert_Windows_Eventlog_15m_Failures
  • SEG_Report_iSeries_Jobs_12hr_Failed_Batch
  • NOC_Summary_Network_Security_24hr_Top_src_ip

Manage knowledge object permissions
When a Splunk user first creates a new report, event type, transaction, or similar knowledge object, it is only available to that user

  • Make the knowledge object available globally to users of all apps (also referred to as "promoting" an object).
  • Make the knowledge object available to all users of an app
  • Restrict (or expand) access to global or app-specific objects by user or role
  • Set read/write permissions at the app level for roles, to enable users to share or delete objects they do not own

TIP:By default, only users with a power or admin role can share and promote knowledge objects

Make an object available to users of all apps
need select ALL apps
Make an object available to all users of its app
need select app
Moving or cloning a knowledge object
you want users of an app to be able to access a particular knowledge object that belongs to a different app, but you do not want to share that object globally with all apps

Restrict knowledge object access by app and role

  • Be able to use the object and update its definition, give that role Read and Write access.
  • Be able to use the object but be unable to update it, give that role Read access only (and make sure that Write is unchecked for the Everyone role)
  • Be unable to see or use the knowledge object at all, leave Read and Write unchecked for that role (and unchecked for the Everyone role as well)

Enable a role other than Admin and Power to set permissions and share objects
By default, only the Power and Admin roles can set permissions for knowledge objects. Follow these steps to give another role the ability to set knowledge object permissions

  • Click on the Applications menu in the Splunk bar, and select Manage Apps
  • On the Permissions page for the app, give the role Read and Write permissions

Set permissions for categories of knowledge objects
making changes to the default.meta file
About deleting users who own knowledge objects
If you delete a user from your Splunk deployment, the objects that user owns become orphaned. Orphaned objects can have serious implications
To prevent this from happening, reassign knowledge objects to another user

Manage orphaned knowledge objects
The Splunk software provides several methods of detecting orphaned knowledge objects. Once you have found orphaned knowledge objects, you have several options for resolving their orphaned status
Find orphaned knowledge objects:

  • Review orphaned scheduled search notifications
  • Look at the Orphaned Scheduled Searches, Reports, and Alerts dashboard and report
  • Run the Monitoring Console health check
  • Use the Reassign Knowledge Objects page in Settings

Reassign one or more shared knowledge objects to a new owner:
Only users with the Admin role can reassign knowledge objects to new owners

  • Settings > All configurations
  • Reassign Knowledge Objects
  • Filter by Owner
  • Orphaned
  • Object type
  • App
  • Enter the string into the filter field and click Return

Reassign a single knowledge object to another owner

  • Reassign
  • Select an owner

Reassign unshared, orphaned knowledge objects

  • Temporarily recreate the invalid owner
  • Perform a knowledge object stanza copy and paste operation between two .conf files

About resolving orphaned scheduled searches
Keep the search running on its schedule:Reassign the search to a new owner
Let the search run on an ad-hoc basis:Remove the schedule for the search from its definition
Keep the search from running again under any circumstances:Disable the search, or delete it

Turn off notifications of orphaned searches
open limits.conf, look for the [system_checks] stanza, and set orphan_searches to disabled

Disable or delete knowledge objects

  • You cannot delete default knowledge objects that were delivered with Splunk software (or with an app)
  • You can always delete knowledge objects that you have created, and which haven't been shared by you or someone with admin-level permissions
  • To delete any other knowledge object, your role must have write permissions for the app to which the knowledge object belongs and the knowledge object itself

App-level write permissions are usually only granted to users with admin-equivalent roles
If a role does not have write permissions for an app but does have write permissions for knowledge objects belonging to that app, it can disable those knowledge objects. Clicking Disable for a knowledge object has the same function as knowledge object deletion, with the exception that Splunk software does not remove disabled knowledge objects from the system. A role with write permissions for a disabled knowledge object can re-enable it at any time

Grant a role write permissions for an app
Users whose roles have write permissions to an app can also edit knowledge objects that are associated with that app.
Grant a role with app write permissions the ability to delete a knowledge object that belongs to that app

  • Your role has admin-level permissions.
  • The role that you are setting object-level permissions for has the ability to write to the app that the object belongs to

Deleting knowledge objects with downstream dependencies
Deleting knowledge objects in configuration files

About Splunk regular expressions
Character types
Groups, quantifiers, and alternation
example: fail admin_user

  • (<?\S+) (?\S+) (?\S+)
  • (?\d+.\d+.\d+.\d+) (?\w+) (?.*)

Modular regular expressions
Modular regular expressions are defined in transforms.conf.
you can define an integer and then use that regular expression definition to define a float

# matches an integer or a hex number
REGEX = 0x[a-fA-F0-9]+|\d+

# matches a float (or an int)
REGEX = \d*\.\d+|[[int]]

You can also use the modular regular expression in field extractions

# this would match only numbers from 0-255 (one octet in an ip)
REGEX = (?:2(?:5[0-5]|[0-4][0-9])|[0-1][0-9][0-9]|[0-9][0-9]?)

# matches a valid IPv4 optionally followed by :port_num the 
# octets in the ip would also be validated 0-255 range
# Extracts: ip, port
REGEX = (?<ip>[[octet]](?:\.[[octet]]){3})(?::[[int:port]])?