The easiest way to get your Splunk Enterprise, Splunk Splunk Cloud, ITSI, and Enterprise Security data into Splunk Observability!
This Splunk Alert Action and ES Adaptive Response Action will send an event to Splunk Observability based on an alert created in Splunk Enterprise, Splunk Cloud, ITSI, or Enterprise Security.
Includes the ability to match and include field data from your Splunk Alerts in Splunk Observability events.
- Get important security data from Splunk Core, ITIS, and Enterprise Security in front of developers in Splunk Observability
- Notify Splunk Observability users of issues uncovered in Splunk Core, ITIS, and Enterprise Security log data
- Close the loop between Splunk Core, ITIS, and Enterprise Security users and Splunk Observability users
- Tie your Splunk Ecosystem together
See Details for detailed setup and usage instructions
Example Use Cases:
- Send security events from locked down security indexes in Splunk to Splunk Observability.
- Choose any fields you want to include in the Splunk Observability event to pass on needed context (even if Developers normally do not have access to the index containing the security information)
- Allow alerting from ITOps teams using Splunk to Developers in Splunk Observability who may or may not have access to Splunk at all.
- Create event markers in Splunk Observability from CI/CD processes being tracked in Splunk.
- Events for Start of Deployment and End of Deployment (including if it was a success or failure) can help Dev teams track their CI/CD processes all in one place
- Notify Splunk Observability users of issues being tracked in Splunk. This could include business processes, ITSI service/KPI alerts, network outages (on-prem or in cloud), etc
- Any other use cases that requires stitching together context between Splunk Enterprise/Cloud and Splunk Observability
Instructions
Prerequisites
Before installing this app, the following needs to be addressed.
You must have access to Splunk and a valid Splunk Observability Cloud API Ingest token.
Install
Install the Alert Action from the "Manage Apps" section of Splunk Enterprise / Splunk Cloud Platform:
For detailed instructions see these docs on installing Add-Ons in various Splunk Environments
- Click the "Apps" drop down and select "Manage Apps"
- Choose "Install app from file" and upload the Splunk Observability Cloud Alert Action for Splunk archive file from Splunk Base. You may be prompted to restart Splunk.
- Click the "Apps" drop down and select "Splunk Observability Events" to configure your Splunk Observability API Token.
- Note: Make sure you are using an Ingest token and not a personal API token.
Setup your new Alert Action and send Events to Splunk Observability Cloud
- Create an alert as normal or open an existing alert to add your Alert Action.
- Click the "Add Actions" button and choose "Observability Events" from the drop down
- Input your realm (by default this will have us0 filled in.)
- If you would like to add an Info dimension and value to your Event fill in the value in the Info Field text box.
- Input a comma separated list of fields to pull from events found by your Splunk Alert
- On your Splunk Observability Cloud Dashboard enter the name of your Alert in the "Event Overlay" text box.
- You may also search the Events finder for your Alert Action name or any associated dimensions including your "info" field's value.
Troubleshooting Steps
For Add-ons, follow this troubleshooting-addon document to troubleshoot the issue.
For documentation on obtaining a Splunk Observability Cloud token from your Org see the Org Token docs for detailed steps.
“Data does not appear to be going into Splunk Observability”:
- Verify your Alert Action has matching events in the time period you are investigating.
- Verify that the Alert Action is firing in the Alerts Page
- Verify Alert Action settings and perms in the Alert Action Manager
- Verify that your results contain a
source field. If they do not, eval a source field onto the end of your search (E.G. | eval source=if(isnotnull(source), source, "no source")
- This is most common when using a
| stats command with no source output and can be diagnosed from index=_internal AND signature="Unexpected error: 'source'."
- Search:
index=_internal sourcetype="tasplunkobservabilityevents:log" ("Response was" OR "observability_events") look for non-200 responses.
- 400 means there was bad data sent to the Events API
- Check for any quotation marks or other special characters in your fields.
- Escape any special characters with an eval. Example removing quotation marks
| eval groupingid=replace(groupingid,"\"","")
- 401 means their token is unauthorized
- 500 indicates an Observability side error
- Search: index=_internal (eventtype=splunkd-access OR eventtype=splunkd-log) "observability_events" and look for non-200 status codes.
- 500 indicates Splunk Observability may be down
- 404 indicates sending to the wrong Splunk Observability endpoint. Check and verify your realm is correct in the Alert Action setup.
- 401 indicates your Splunk Observability token is likely expired or dead. Verify token (and that it is an INGEST capable token) or try a new token.
“I can’t seem to add a specific field to my alert action events”:
- Verify the spelling and naming of the field you wish to include
- If the name includes a period character (E.G.
security.repo) try using an eval statement to rename the field replacing the . character with an _ character then match to the renamed field
“I can’t make an INGEST token in Splunk Observability”:
- Check your permissions level. You may require an Admin to create the INGEST token.
“I dont see my events overlaid on my charts in my Splunk Observability dashboard”:
- Verify that you have setup your Event Overlay for the dashboard https://docs.splunk.com/Observability/data-visualization/dashboards/dashboards-add.html#overlay-event-markers-on-charts-in-a-dashboard
Hint: Using * characters as wildcards can be helpful to match events.
Verify that events with matching info dimensions exist within the time period you are investigating in your dashboard
Copyright 2022 Splunk Inc.
