AppDynamics

The AppDynamics app allows you to mine application performance monitoring data from AppDynamics using its REST API. You can then slice and dice the data within Splunk using Search Processing Language (SPL). It also contains a notification client that can be extracted to an AppDynamics controller that will relay event and policy violation notifications in AppDynamics to Splunk and has the ability to cross-launch into AppDynamics from Splunk.

Release Notes

How To Use the AppDynamics App in Splunk

What is AppDynamics?

AppDynamics is an Application Performance Management (APM) solution that helps you:

  • Identify problems such as slow and stalled user requests and errors in a production environment.
  • Troubleshoot and isolate the root cause of such problems by:
    • Mining performance data from AppDynamics into Splunk using the AppDynamics Controller REST API. See Seeing Performance Data from AppDynamics in Splunk.
    • Pushing notifications on policy violations and events from AppDynamics to Splunk so that a Splunk user can use those to launch deep dives in AppDynamics. See Getting Notifications From and Launching AppDynamics in Splunk.

You can mine application performance monitoring data from AppDynamics using its REST API. You can then process the data within Splunk using Search Processing Language (SPL). It also contains a notification client that can be extracted to an AppDynamics Controller that will relay event and policy violation notifications in AppDynamics to Splunk and has the ability to cross-launch into AppDynamics from Splunk.

Installation

These instructions assume that you are familiar with using both AppDynamics and Splunk.

Links within this file go to AppDynamics 3.7 documentation. If you are running an earlier version, use the Search feature to find the associated topics.

Prerequisites

  • You have installed AppDynamics version 3.5 or newer. If you do not already have a license, you can sign up for a trial license. You can choose either a SaaS solution or an On-Premise installation.
  • You have installed Splunk version 4.x or newer.
  • You have installed the AppDynamics App for Splunk from Splunkbase.
  • You have access to the following AppDynamics Controller information, which is required to set up the integration:
  • hostname/IP address
  • port number
  • account name
  • user name
  • password

    If you use a SaaS account, AppDynamics provides you with the required information. - You have access to the AppDynamics documentation. When you trial or buy the product, AppDynamics provides access credentials to you. - $SPLUNK_HOME is set to the directory where Splunk is installed.

Steps

  1. Locate and edit the files: $SPLUNK_HOME/etc/apps/appdynamics/local/metrics.conf and $SPLUNK_HOME/etc/apps/appdynamics/local/events.conf
  2. In the metrics.conf file, add one section for each individual metric you want to mine from AppDynamics. You need the following:

    • AppDynamics metric name, to name the section in the metrics.conf file, and for use as as unique identifier in Splunk
    • REST URL of the metric from the AppDynamics Metric Browser, see the AppDynamics REST documentation (login required).
    • polling interval - how frequently, in seconds, Splunk will run the script to get this metric
    • username and password
      • for On-Premise or SaaS multi-tenant, <username>@<account> and the password
      • for On-Premise or SaaS single-tenant, <username>@customer1 and the password

    For example, if you want to mine a metric called AverageResponseTime for the ViewCart.sendItems business transaction, the entry would be similar to this:

    [ViewCart.sendItems_AverageResponseTime]  
    url = http://<controller-host>:<port>/controller/rest/applications/Acme%20Online%20Book%20Store/metric-data?metricpath=Business%20Transaction%20Performance%7CBusiness%20Transactions%7CECommerce%7CViewCart.sendItems%7CAverage%20Response%20Time%20(ms)&time-range-type=BEFORE_NOW&duration-in-mins=15  
    interval = 60  
    username = user1@customer1  
    password = welcome
    
  3. In the events.conf file, add one section for each individual event type you want to mine from AppDynamics. You need the following:

    • AppDynamics event type, to name the section in the events.conf file, specify the event query for the REST URL, and for use as as unique identifier in Splunk
    • AppDynamics event severity, to specify the event query for the REST URL
    • REST URL of the event type from the AppDynamics Metric Browser, see the AppDynamics REST documentation (login required).
    • polling interval - how frequently, in seconds, Splunk will run the script to get this metric
    • username and password
      • for On-Premise or SaaS multi-tenant, <username>@<account> and the password
      • for On-Premise or SaaS single-tenant, <username>@customer1 and the password

    For example, if you want to mine events caused by application changes, the entry would look similar to this:

    [Server.application_Changes]  
    url = http://<controller-host>:<port>/controller/rest/applications/Acme%20Online%20Book%20Store/events?time-range-type=BEFORE_NOW&duration-in-mins=15&event-types=APP_SERVER_RESTART,APPLICATION_CONFIG_CHANGE,APPLICATION_DEPLOYMENT&severities=INFO,WARN,ERROR  
    interval = 60  
    username = user1@customer1  
    password = welcome
    

Note: Workaround for an existing Splunk bug. When you make changes to metrics.conf or events.conf, you need to restart Splunk for your changes to take effect. However, when you restart Splunk, it does not properly clean up the running instances of the Python scripts, metrics.py and events.py, which continue to run as zombies. So, after you stop Spunk, if you do a "ps -ef | grep python" and kill all the zombie processes and then restart Splunk, then your changes should be seen immediately.

Metrics

  1. Launch the AppDynamics App in Splunk.
  2. Enter index=appdynamics in the Search field of the AppDynamics App in Splunk.

100353

Events

  1. Launch the AppDynamics App in Splunk.
  2. Enter index=appdynamics_events in the Search field of the AppDynamics App in Splunk.

120359

Pushing AppDynamics Notifications to Splunk

Note: This feature is currently available only for single-tenant Controllers.

For AppDynamics SaaS customers:

Contact AppDynamics Support and ask them to extract the splunkClient zip file on your behalf.

For AppDynamics On-Premise Controllers:

  1. Locate and copy the splunkClient zip file:

    • For Linux: $SPLUNK_HOME/etc/apps/appdynamics/splunkClient/splunkClient-3.7-linux.zip
    • For Windows: $SPLUNK_HOME/etc/apps/appdynamics/splunkClient/splunkClient-3.7-windows.zip
  2. Extract the splunkClient zip file to the Controller installation directory on the machine where the AppDynamics Controller is installed

Note: The splunkClient.zip includes a custom.xml file containing notifications. If your Controller already has a custom.xml file, edit it and merge the contents.

Setting up .splunkrc file

A sample.splunkrc file is part of the splunkClient zip file.

  1. Edit the .splunkrc file to add information that allows the Controller to communicate with Splunk. If you do not already have a .splunkrc file, edit the file <controller-home>/custom/conf/.splunkrc and change the properties to suit your Splunk installation

       # Host at which Splunk is reachable (OPTIONAL)  
       host=localhost  
       # Port at which Splunk is reachable (OPTIONAL)  
       # Use the admin port, which is 8089 by default.  
       port=8089  
       # Splunk username
       # Note: This user must have tcp_edit capability as defined
       # in the file $SPLUNK_HOME/etc/system/local/authorize.conf
       username=admin  
       # Splunk password  
       password=changeme  
       # Access scheme (OPTIONAL)  
       scheme=https  
       # Namespace to use (OPTIONAL)  
       namespace=*:*
    
  2. Copy the .splunkrc file to the platform home directory of the user that started the Controller. In Linux, this is the environment variable $HOME location; in Windows, it is the environment variable %USERPROFILE% location.

Custom Notifications

Custom Notifications in AppDynamics

  1. Use the AppDynamics Controller UI to configure the custom actions notify-splunk-of-event and notify-splunk-of-policy-violation in the Global Notifications and Policy Notifications screens. Details are available in Configure Custom Notifications (login required).
  2. Add the following field extraction section to your $SPLUNK_HOME/etc/apps/search/default/props.conf file:

    [source::http-simple]  
    EXTRACT-AppD = url="http[s]*://(?<nurl>[^"|]+)"
    
  3. Add the following workflow action to your $SPLUNK_HOME/etc/apps/search/default/workflow_actions.conf file:

    [LaunchAppD]  
    display_location = both  
    fields = url  
    label = Launch in AppDynamics  
    link.method = get  
    link.target = blank  
    link.uri = http://$!nurl$  
    type = link
    

Custom Notifications in Splunk from AppDynamics

100352

Launching AppDynamics from Splunk

On an event in the Splunk Search App, click the blue pulldown and choose Launch in AppDynamics. See the screenshot above.

Contributing

Always feel free to fork and contribute any changes directly via GitHub.

Community

Find out more in the AppSphere community.

Support

For any questions or feature request, please contact AppDynamics Center of Excellence.

3 ratings

Built by Pranta Das