Category: Splunk

Splunk user roles decide what the user can see, do, interact with Splunk with the access. Three General roles are mentioned below:

1. 1. Administrator: role is the powerful role in the list of roles; – install app- ingest data- create knowledge objects for all users
   2. Power User: create and share knowledge objects for users of an app and do real time searches. – this is in general people get to create alert and dashboards.
   3. User role – can see only their own knowledge objects and those shared with them.

Each Organization has a config on having the types of user roles based on the hierarchy, billing, usage and may not be same across!

Classic Dashboard, Dashboard Studio

and the 2nd dashboard is

created from Search

Absolute/Grid is the 2 view types

Grid layout has the view mode first and then

can see edit the search, visualization, code in json for the visualization.

Data Sources, inputs, general layout -> order in which dashboard source code is ordered

json code/file has .viz and .splunk prefixes

dashboard from web page, create dashboard

set private/all aps

Classic or Dashboard studio can be selected.

Edit mode is default from absolute layout.

icons, shapes text options along with background color can be changed.

Grid to absolute or vice versa can be made, but it’s not an innovative idea to change the dashboard studio dashboard.

Grid layout has charts and rows.

In Search there are certain commands like each other but have unique functionality.

SEARCH JOB INSPECTOR – trouble shooting the searches.

EXECUTION COSTS – Based on the more time the search time, more the components.

Comments in general used to make a note of what eh search code is intended to be executed as ”’is a sample comment”’

Indexer vs Search head, Splunk uses bloom to find the search terms

Search head is for the centralized streaming and transforming commands

Transforming commands – time charts, stats, chart, top, Rare

Centralized– is made in search head – stateful Streaming commands

Distributable streaming command- eval- executes on indexer

Notes: Rename- distributive-streaming command | Rename to added before stats

Search tokens- event tokens from Segmentation – affect search performances, either improve or not.

Major breakers – Space-new line-carriage return, Comma, exclamation mark

Minor breakers – Symbols like:

Searches– tokens-> Search in address- click search log

Splunk uses lispy expressions to create bloom filters.

AND OR NOT

Hot bucket- stores data as it arrives

Warm bucket- Read only then cold then

Frozen bucket – deletion and archiving

Bucket– journal has the raw data, and the tsi (timeStamp index) file – index keys to journal file has a unique lexicon

1. Which of the following commands generates temporary search results? make results
2. Where should the make results command be placed in the search? In the beginning of the search
3. Which if the following command signified the comment in spl? ”’triple single quotes”’
4. Where are comments to be placed in the Search? comment can be placed anywhere – in the search
5. Which component of the search job Inspector shows how long a search took to execute? Header
6. When is a bucket bloom filter created? When the bucket changes from hot to warm
7. Which architectural component of a Splunk deployment initiates a search? initiates in Search head – sent to indexer peers!
8. Which component of a bucket stores raw event data? Journal
9. Where in the search pipeline are the transforming commands executed? Search head
10. If a search begins with a distributable streaming command, where is it first executed? indexer– if search head then
11. After the Splunk tokenizes terms at the index time, where are the tokens stored? tsidx files
12. Which of the following breakers would be used first in segmentation? major breakers – spaces, new lines, carriage returns, tabs, [],  ! , commas?
13. Which of the following expressions builds a search-time bloom filter? lispy
14. Which of the following could cause a lispy expression to not create tokens? wild card in beginning
15. Which directive is used in a search to bypass minor breakers inside the supplied argument? lispy?\:/-$ — term

Intro to knowledge objects

Fields – when a search is made the fields are displayed. This helps in narrowing the data which contains only that specific field/keyword. the events,
All the fields can be found in the field side bar, on hover and clicking on the fields specifically, it can be added to the search.

Field extractions

field aliases – normalize data to provide a normal name if the current name is like another field or common!

calculated fields – perform calculations based on existing fields.

lookups – sources like csv can be configured to additional fields and events in search

Event types – same combination over and over- save the search as event type to categorize the data

Tags – key value pairs can be made into tags, can be used in search as event type

note: event types and tags can be also noted in the field side bar

Workflow Actions – interact with external resources to narrow the searches, http get and post to external sources and back to Splunk for secondary searches.

Reports – Repeated search can be saved as reports

Alerts – to receive a notification, search can be saved as alerts

both are used for scheduling

Macros – similar or more complicated syntax – can store entire search strings including commands,

Data models – data sets can be events, can be used in pivot

Name ur objects using Six segmented keys

1. Group
2. type
3. platform
4. category
5. time
6. Description

Primary type of knowledge objects

permissions lay a key role in creating and sharing KO,

1. private –
2. Specific App
3. All apps

When a user creates a ko, it automatically private and only available to that user,

when a power user or admin creates a Ko, it is shared across all users, and has control on other roles by hide/give permissions

only admin has access to allow ko to access across all apps, can edit permissions, can read or edit private objects created by anyone.

Admin can only reassign a ko to user- when the user leaves the org but has knowledge.

 

1. Primary functions of workflow actions? interact with external resources to narrow the searches
2. which knowledge objects can have an eval expression? calculated fields
3. Which knowledge object can communicate to external sources using the HTTP GET and POST methods? workflow actions
4. knowledge objects to be scheduled and executed at specific times? alerts/reports
5. By default, what user role is required to make a knowledge object available to all apps? any user
6. Where can you find the list of fields returned from events? fields sidebar
7. When a user has left the organization, what haens to the knowledge objects? admin can edit the permission, user level info
8. which kind of knowledge objects can be searched in pivot? data models
9. Which are used to manually extract the fields? regex, delimiters like commas, spaces.
10. Which of the knowledge objects can contain eval?  calculated fields? macros?
11. Which of the user roles can create knowledge objects? Power user, admin, user
12. When a ko is crated wo can access its contents? user created and admin
13. Which of the types can be uploaded to create a lookup? csv
14. What are the three predefined sharing options for a knowledge object? private, shared in all apps, shared in app
15. which knowledge object type can store entire search strings including commands? macros

Few important tips to take note of

Here we can see about Field Extraction and table formatting

| fields – to make searches more efficient, main part of search

• -fieldname fieldname2 is different for – fieldname fieldname2

it makes search more efficient if the fields are included or excluded.

| table – like fields- but changes data to tabulated format (transforming command)

fields to be used before using the table command

| dedup – to remove duplicate values form the values displayed in the events/rows

1. count ex:
2. top, limit: top vendor limit=5
3. rare
4. showperc=true/false
5. otheruser=true

Few of the Stats commands

1. count ex: stats count as “column name” by field name
2. distinct count ex:
3. sum
4. average
5. min
6. max
7. list
8. values

Chart

| chart count over status

count of a field can be noted in x axis, y always numeric as the data mentioned in x axis can be displayed based on the count.

Questions and Answers:

1. Which of the following removes the duplicate? dedup.
2. In a single series data table, which column provides the x-axis values for the visualization? 1st column?
3. Which optional argument of the addtotals command changes the label for row totals in a table? label.
4. Which clause can be used with the top command to change the name of the count column? countfield.
5. Which clause can be used with the top command to specify a number of values to return? limit.
6. Which command changes the appearance of field values? fieldformat.
7. How many columns are displayed by default when using the chart command? 10.
8. Which type of default map visualization uses shading to represent relative metrics? chloropleth.
9. Which of the following commands can return a count of all events matching a search criteria over a specific time period? trendline sma/ema/wma
10. When using the time chart command, which axis represents time? x-axis.
11. How can the order of columns in a table be changed? By changing the order of fields specified in the table command.
12. Which argument can be used with the time chart command to specify the time range to use when grouping events? span.
13. Which clause can be used with the rare command to specify whether a percentage column is created? showperc.
14. Which command can be used to exclude fields from search results? fields.
15. Which argument can be used with geostats command to control the column count? globallimit.

Business Analyst – Splunk related project

• Excellent troubleshooting and problem-solving skills to identify problems from a functional perspective, specifically when supporting end-user testing and training. Includes proficiency with such tools as Splunk, AppDynamics, etc.
• Create reports, dashboards, and visualizations to understand business performance
• Analyze process issues and bottlenecks and to make improvements
• Communicate and validate requirements with relevant stakeholders
• Develop and maintain reporting tools
• Perform data discovery, analysis, and modeling
• Collaborate with product manager on roadmap planning and prioritization

Splunk introduction – notes!

Splunk is considered Google like search engine for the logs. Correlation of data is one of the key features considered to use Splunk.

Flexible data pipeline – any type of data can be roped into the platform, extract, and format it and make it searchable

Quick search, time normalization and powerful query language makes it stand top across competitors

ADHOC Search- considered in general inefficient on comparison with other types of searches.
As u are trying to find the problem- may-may not find it after the search – if many people do at the same time, efficiency may be impacted. It is done to make a feel for the data, to pin down the issue we are looking for! the discovery of issues which are already known and few which the user sees for the first time as well.

Scheduled Search-you know the problem- search in time intervals and make it efficient. During this time, we must make sure- the impact on the system is high| real time Search- real time search as it happens – heavy impact on environment- do not perform any real time without approval.

Licensing model – earlier- charge on amount of data bringing in- usually- filter the data – so correlating gets impacted-

Workload pricing model- computation charges- based on the compute on platform the charge is made and not for the data loaded-in. more computation, the licensing will be decided.

1. Major features of Splunk enterprise.
2. index – bucket of data -> as data enters its inspected and match to a source type and make it as a single event – timestamped and stored in the Splunk indexes so it can be searched. a particular level of access ex: network logs to a index, application logs to another index etc.
3. index can be considered for the data retention ex: 30 days, 60 days
4. by searching in the Splunk- diff source type can be searched
5. Search – monitor- alert
6. one can create alerts and monitor specific conditions
7. allows you to collect reports in visualization and dashboard

Web Interface

1. Apps – sit on top of Splunk instance, can also be called as workspace
2. Roles- decide what the user can see, do or interact with
   1. Administrator: role is the powerful role in the list of roles; – install app- ingest data- create knowledge objects for all users
   2. Power User: create and share knowledge objects for users of an app and do real time searches. – this is in general people get to create alert and dashboards.
   3. User role – can see only their own knowledge objects and those shared with them.

Once logging in Splunk enterprise, it has 2 apps by default- and there are so many apps which can be picked from the Splunk base!

1. Home app – manage other apps- gives quick space to – create custom dashboard as a default
   Admin can also add apps from home app
2. Search & Reporting app: provides a default interphase for searching and analyzing the data and has 8 components
   1. Splunk bar-> edit->view messages->monitor the progress of search jobs
   2. App bar
   3. Search bar- used to run searches
   4. time range picker – events for specific time ex: 60 min, 1 day, 4 day – – do not perform long time search
   5. histogram – the events occurring in the specific period is not here.
   6. how to search panel
   7. Data Summary button
      1. host (IP address, domain name)
      2. source(path/filename)
      3. source type (classification of data)
3. Table view-
4. Search history- old search history can be searched with the filter option, can be re-run again across specific timeline on how many runs have been made.
5. rolling over events- makes it highlighted- can add that to search
6. failed password to the search – can remove the data from search by clicking on the highlight
7. drop down for event actions.
8. The vents can be extracted by clicking on the arrow in the recent. The data is in the key value pair, one point t
9. The admin team will have to do the field extractions, only the key value pairs are extracted and made int he proper format. Field extractions to be done, manual extractions base done expressions limit the filtering on the later part of the search- so as a best practice do as must as search and filter in the base search as possible
10. Key word search example “error” keyword is given to Splunk- it searches across all events for the keyword
11. text from the pdf when updated may not format as expected. for format- properly – control | is used to format the results.
12. Table commands- the field mentioned will allow you to see the results in the format of a table
13. fields command- to remove fileds or order fields in a particular way
14. top- finds the most common values of the given field and % distribution and count
15. top is easy to make the visualization- in the results just cluck o visualization from the data searched.
16. rare – opposite to top
17. Stats – enables users to calculate the statistics
18. Sum –
19. As –
20. Group by – count by
21. eval – used to create an extra column with a default value or a formula evaluation of the values. ex: eval abdc=if(x<‘5000, 8000, abdc) 
22. Time chart- takes results and formulate in the time selected in the time picker.
23. span – can be used in time chart command to chunk the time intervals- for trends etc.
24. Stats –
25. Base search – the search before the pipe, mostly index, source, source type, host.
26. transforming search – everything after the pipe | symbol written after the base search

Search Processing language

1. wild card- * ex: fail* leads to search of failed or failure or fails – used after the string is more efficient than at front.
2. AND NOT OR –
   1. ex: failed password is like failed AND password
   2. ex: failed OR password displays all combinations
   3. Order is NOT OR AND
   4. parenthesis is used to control the order of evaluation
   5. “Failed password ” in general used with quotes to search

Features and terms used on Splunk on day 2-day use!

• Shared Search jobs
• Export Results- raw- csv, xml, Json
• Search mode- fast (no field discovery)
• verbose- discovering all data as can
• default mode- –
• Timeline- visual rep of segments on the time- on clicking the timeline- we ca see the event generates on that time.

What is an event? – time index- based on time zone in user account bottom row has the selected fields, rolling

Other factors used can be noted below:

Add to Search

icon- to open in new browse window

Clicking on highlighted text can add or remove to search

event actions

field actions

Search Processing language

Wild cards – *

search terms are not case sensitive

AND OR NOT can be used for multiple familiar words like US or CA

Order Role evaluation

not or and (Preference)

“

\”

What are commands, functions, clauses, arguments in search terms?

how we want to search- a site’s foundation of search queries.

Commands -what we need to do with the searches results- create charts, computing statistics and formatting

Functions – explains how we want to compute and evaluate the result

Arguments – variables we need to apply for the functions

Clauses – how we want results (group or defined)

Below terms can be used in the search

Index

host

Source type

Stats

Count

visits

search Visits >1

There are certain admin consoles not all the users may have access to.

Splunk Specialist with good IT infrastructure skills, in multi-platform environments, ideally familiar with Linux. There are several innovative projects in Splunk, and various companies are looking for qualified administrators with Splunk experience and/or certification.

Main responsibilities:

• Participated in all Splunk company initiatives, both internal projects and customer mandates.
• Install and configure the necessary components to collect data from DB, log files, API, etc. to Splunk.
• Install, configure, administer Splunk Enterprise on Windows and Linux.
• Support Splunk updates.
• Monitor and identify performance issues.
• Perform data onboarding in Splunk: data collection, filtering, and transformation (source types, inputs, transforms, etc.);
• Build use cases: advanced SPL, dashboards, reports, alerts, etc.
• Always continue to develop product knowledge and act as a product expert.
• Document best practices.

Qualifications required:

• Integrating data from various sources (DB, log files, APIs, etc.) into Splunk (on prem or cloud);
• Experience in CIM modeling in Splunk.
• Experience in managing indexes and knowledge objects in Splunk.
• Experience working with cloud offerings such as Azure or AWS.
• Knowledge of basic security concepts.
• Experience in access management (RBAC model) in Splunk.
• Valuable experience in AIX, Linux (RedHat, CentOS) systems administration (permissions management, security (including TLS/SSL), debugging, etc.);
• Exceptionally good experience in Splunk user support and training.
• Good knowledge of system virtualization.
• Good knowledge of server infrastructure.
• Knowledge of storage, operating systems and networking.
• Knowledge of Splunk Enterprise Security is an asset.