Community edition App for monitoring Rundeck Core Community Edition
SPLUNK_HOME/etc/apps
directoryIn a distributed Splunk installation you will typically install the App components across Heavy Forwarders and Search Heads.
As this App requires that you have access to the App's Setup page in order to configure hosts and encrypted auth tokens , you will have to use a Heavy Forwarder(HF) rather than a Universal Forwarder if you want to split out the data collection (REST polling) logic to a forwarder tier.
The App's setup page will only automatically enable the REST polling stanzas if this App is running on a Splunk instance that is an Indexer or Forwarder server role.
Your HF will forward data into your indexer cluster as per standard Splunk administation of outputs.conf
configurations.
The Search Heads will then only require the UI,Knowledge Object and Custom Alert Action components of the App release.
The App release ships with an app.manifest
file , so you can utilise Splunk's Packaging Toolkit (http://dev.splunk.com/view/packaging-toolkit/SP-CAAAE9V) to build the Search Head App bundle.
If you install the App on a Heavy Forwarder , the Modular Input (for polling REST Endpoints) needs to have access to your Rundeck index.This is because it executes Splunk searches to dynamically create endpoints to poll based on values in the search results ie: list of Project names.
Browse to Rundeck App Community Version
in the lefthand side Apps panel and enter various global settings as well as your authentication token.These global settings are used by both the Rundeck Modular Input and the Rundeck Custom Alert Action.
The authentication token gets securely encrypted in local/passwords.conf
Upon saving the setup screen , the REST inputs will automatically enable and start polling for data if the Splunk instance is an Indexer or Forwarder server role only.
Check that the REST API data is being received by browsing to the Data -> Data Sources
dashboard , it may take a minute or 2.
The setup page can be browsed to at any time for configuration edits via the navigation menu: Admin -> Configuration -> Setup Rundeck Environment
The REST inputs can be manually viewed/edited/controlled via the navigation menu : Admin -> Configuration -> REST Endpoints
Descriptions of the input configuration fields can be found in README/inputs.conf.spec
and also in the UI.
Your Rundeck host must be entered in the format : foo.myrundeck.com (don't include 'https://' , this is hardcoded by default to satisfy Splunk certification requirements for secure network communications).You can also specify an alternative port with the host , foo.myrundeck.com:1234
When you create a scheduled alert in Splunk (http://docs.splunk.com/Documentation/Splunk/7.1.1/Alert/Definescheduledalerts) , Rundeck Job
will be presented as one of the available Trigger Actions.
The UI will then guide you to fill in any additional fields.
For a detailed list of the fields , refer to README/alert_actions.spec
and README/savedsearches.spec
The Rundeck host that you specify for your Alert Action must match 1 of the hosts that you setup on the App's setup page.
main (this is the default , change this if your Splunk admin has directed you to use a different index)
see below for the log sourcetypes to use for the various log sources.It is important that you use the correct sourcetype name.
the host you specify must match what you setup in the Rundeck App setup screen.
Configure Splunk receiving in your Splunk Server.Default port 9997 is usually fine (unless your network admin says otherwise) , https://docs.splunk.com/Documentation/Splunk/7.1.2/Forwarding/Enableareceiver
Ensure your network is open for TCP 9997 from your Rundeck Server to your Splunk server
Download a Splunk Universal Forwarder(UF) https://www.splunk.com/en_us/download/universal-forwarder.html
Install the UF on your Rundeck Server to monitor the Rundeck log files, http://docs.splunk.com/Documentation/Forwarder/7.1.2/Forwarder/Abouttheuniversalforwarder. You can just untar the UF tarball release to your Rundeck install directory.
Setup the inputs.conf
and outputs.conf
files
5.1. Copy the example local
directory from SPLUNK_HOME/etc/apps/rundeck_app/forwarder_config
to your UF's SPLUNK_HOME/etc/apps/search directory
5.2. Replace placeholders
5.2.1. inputs.conf
* host (to match what you specified in the app's setup page).For a cluster , use the cluster address in the host field.
* log paths , if these are different to the examples (which are the Rundeck defaults)
* index , if using a different index to the default of main
5.2.2. outputs.conf
* your Splunk server host/IP and receiving port
Restart the UF
Check that the log data is being received by browsing to the Data -> Data Sources
dashboard
Rundeck uses log4j as it's logging framework , http://rundeck.org/docs/administration/configuration-file-reference.html#log4j.properties
An alternative to installing a Splunk Universal Forwarder is to install a Splunk log4j appender to forward log data directly to Splunk via raw TCP or HTTP Event Collector.There are 3rd party libraries availble for this purpose , such as https://splunkbase.splunk.com/app/1715/
You can request your authentication token from your Rundeck administrator.This app has a background task that will automatically check if the authentication token is going to expire within 2 days time(by default) , and automatically generate a new token for you. The new token will start being used and the old token discarded (but not deleted from Rundeck).
Token generation settings are configurable via the rundeck://authtokenrefresh
stanza in inputs.conf
duration
: the duration in days of new token ie: 30d , 120dexpiry_window_secs
: if the current token is due to expire within this window in seconds , generate a new oneBy default ,this App is set to be Globally accessible by all users and apps
This App does not create any indexes by default.If you want to change the default index (main), ensure you also change the global index used for searches in default/macros.conf
in the 'rundeck_index' stanza. This is done for you automatically if you change the index via the App's setup page.
You will also need to update the index in inputs.conf
on your Splunk Forwarders.
This App uses 2 sources of data :
http://rundeck.org/docs/api/#system-info
http://rundeck.org/docs/api/#list-users
http://rundeck.org/docs/api/#log-storage-info
http://rundeck.org/docs/api/#list-executions-with-incomplete-log-storage
http://rundeck.org/docs/api/#listing-projects
http://rundeck.org/docs/api/#listing-history
http://rundeck.org/docs/api/#listing-resources
http://rundeck.org/docs/api/#listing-jobs
http://rundeck.org/docs/api/index.html#execution-query
Below are the sourcetypes you must use for monitoring your Rundeck log files.The example stanzas below can be added to inputs.conf on your Splunk Forwarder instance.
Ensure that your host
and index
match what you specified in the App setup page.
For a cluster , use the cluster address in the host field.
The monitoring path is the default Rundeck log path, change this if necessary.
[monitor:///var/log/rundeck/rundeck.access.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_access
[monitor:///var/log/rundeck/rundeck.api.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_api
[monitor:///var/log/rundeck/rundeck.audit.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_audit
[monitor:///var/log/rundeck/rundeck.executions.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_executions
[monitor:///var/log/rundeck/rundeck.jobs.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_jobs
[monitor:///var/log/rundeck/rundeck.log*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_main
[monitor:///var/log/rundeck/rundeck.options.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_options
[monitor:///var/log/rundeck/rundeck.storage.*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_storage
[monitor:///var/log/rundeck/service.log*]
disabled = false
index = main
host = yourhost
sourcetype = rundeck_logs_service
Any log entries/errors will get written to SPLUNK_HOME/var/log/splunk/rundeck_app*.log
rundeck_app_alertaction.log
: alert action scriptrundeck_app_modularinput.log
: modular input scriptrundeck_app_setuphandler.log
: custom setup handler scriptThis log file is rotated daily.
You can set the logging level globally via the App's setup page.
You can then easily search for logs in Splunk : index=_internal source=*rundeck_app*.log ERROR
This App was developed by BaboonBones, Ltd. for Rundeck, Inc. ( www.rundeck.com ) * www.baboonbones.com * info@baboonbones.com
Ensured API token will not be revealed in logs (https://answers.splunk.com/answers/688471/rundeck-app-for-splunk-token-value-exposed-in-log.html)
Now enabling the normal Splunk menu bar inside Rundeck App for Splunk views (https://answers.splunk.com/answers/688472/how-come-the-splunk-top-level-menu-bar-is-not-visi.html)
Clarified documentation
Initial release of the community application
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.