icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading GUI for SysStat
SHA256 checksum (gui-for-sysstat_006.tgz) 936527b84d386e15fdb8ce251af7c74d0f33af809ec63841e704455f26085dea SHA256 checksum (gui-for-sysstat_005.tgz) bd44d7c91e9fd9ac8cfca0d2552c3dce85ce46573ca17bf02d957033cf048d18 SHA256 checksum (gui-for-sysstat_03.tgz) 978638254386e91d2d9730aa211bbbcbbb3917aeba4847d2232628b249f6b8fc
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

splunk

GUI for SysStat

Splunk Cloud
Overview
Details
Splunk GUI for SysStat

The Splunk GUI for SysStat provides visualization of the system activity collected with the SysStat package. This includes I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization among others. The statistics can be collected in near real-time from as many hosts as you like (potentially thousands). The SysStat package is already present in all major Linux distributions and likely already installed, is a non-intrusive, secure and lightweight way to collect system activity information.

Imagine being able to see output of vmstat, sar or iostat for any period in the past!

A unique feature of the app is the ability to show any available parameters of your choice in one graph.

It is suitable for an ad-hoc or permanent advanced performance troubleshooting and profiling.

The analysis of historical performance data and the detection of bottlenecks allows the establishment of a baseline of the resource usage, predicts utilization and reduces costs, especially in cloud environment like AWS.

The UI has contextual help for every graph, which provides explanations of the parameters and suggests optimizations.

Splunk GUI for SysStat

The Splunk GUI for SysStat provides visualization of the system activity collected with the SysStat package. This includes I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization among others. The statistics can be collected in near real-time from as many hosts as you like (potentially thousands). The SysStat package is already present in all major Linux distributions and likely already installed, is a non-intrusive, secure and lightweight way to collect system activity information.

Prerequisites:

  • Splunk 6/7/8/9
  • RedHat/CentOS 6/7/8/9
  • Debian 7/8/9
  • Ubuntu 14+
  • sysstat package

Upgrade from the version 0.3

The app installation folder was changed from "sysstat" to "Sysstat" to be indentical with the app id (Splunkbase's requirement). You need to disable old "sysstat" app. NOTE: please do a backup before an upgrade.

Single host installation:

  • install the sysstat package

RPM-based distros (Redhat, CentOS):

yum install sysstat

DPKG-based distros (Debian, Ubuntu):

apt-get install sysstat
  • Decrease the interval between samples and enable collection of all information.

Redhat, Centos:
Edit /etc/cron.d/sysstat and change

`*/10 * * * * root /usr/lib64/sa/sa1 -S DISK 1 1`

to:

`* * * * * root /usr/lib64/sa/sa1 -S XALL 1 1`

Debian, Ubuntu:
Edit /etc/cron.d/sysstat and change

`5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1`

to:

`* * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1`

Edit /etc/default/sysstat and change

`ENABLED="false"`

to

`ENABLED="true"`

Edit /etc/sysstat/sysstat and change

`SADC_OPTIONS="-S DISK"`

to

`SADC_OPTIONS="-S XALL"`
  • Install splunk.
  • Install this app.
  • Configure inputs.conf
    local/inputs.conf or configure it using UI:
[script://$SPLUNK_HOME/etc/apps/Sysstat/bin/sadf.sh]
disabled=false
interval=60
sourcetype=sysstat
  • Right after the installation the dropdown menus are inactive - it can take up to 1 minute to get dropdown menus populated.

Distributed multihost installation (simplified):

  • Install sysstat package on each host which need to be monitored
  • Configure inputs.conf
  • Decrease the interval between samples and enable collection of all information
  • Install a central splunk instance (Search Head + Indexer) to collect the logs from the hosts
  • Install this app on the central splunk server
  • Install Splunk forwarder on each host that need to be monitored and configure to send logs to the Indexer
  • Install this app on each forwarder

Tuning:

  • Per default all possible statistics are collected. You can deactivate some less important or very verbose counters. For example on some systems the interrupt statistics contains more than 2500 individual counters, so you can tune the script to report a commulative interrupt statistics instead of ~2500 individual counters. To do so read 'man sar' and find a line in the man page which looks similar to '-A This is equivalent to specifying -bBdFHqSuvwWy -I SUM -I ALL -m ALL -n ALL -r ALL -u ALL -P ALL' (your line can differ). Then replace "-A" parameter in the sadf.sh script in the $SPLUNK_HOME/etc/apps/Sysstat/bin directory with a string from the manual page.

Per default the sadf.sh script looks like this:

LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -A

Find a line in the man page and modify the script to:

LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -bBdFHqSuvwWy -I SUM -I ALL -m ALL -n ALL -r ALL -u ALL -P ALL

then remove parameters which you don't need, for example "-I ALL", so the final string in the script will be similar to (your string can vary!):

LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -bBdFHqSuvwWy -I SUM -m ALL -n ALL -r ALL -u ALL -P ALL

If after this change the sadf/sar stops working, then you need to remove system activity files, please read the sysstat FAQ: http://sebastien.godard.pagesperso-orange.fr/faq.html#sar : "The list of activities that are saved in a file can no longer be modified once the file has been created. So it is important to use the proper options the first time sadc is executed (whether via a crontab, a script like sa1(8) or even the script used to insert a RESTART message when the machine is rebooted)."

This optimisation is not enabled by default because the "-A" options includes different parameters depending on the version of sysstat package and used linux distribution. If you include parameters that your system doesn't support, the script will fail.

  • Reduce the interval for of scheduled searches, which collect monitored hosts names and parameters. Per default the searches run every minute to prepopulate the dropdown lists as soon as possible. But after the setup is completed the interval can be increased to 10 or more minutes. The downside of this change is that you have to wait up to 10 minutes to see your host in the dropdown menu.

Troubleshooting:

  • check /opt/splunk/var/log/splunk/splunkd.log
  • run /opt/splunk/etc/apps/Sysstat/bin/sadf.sh manually
  • search via Splunk UI "sourcetype=sysstat"

For sysstat/sar related problems check http://sebastien.godard.pagesperso-orange.fr/faq.html

Examples of use:

Let's simulate an out of memory (OOM) condition and analyse it using sysstat.

  • Install vmtouch
  • Run this oneliner (modify the number of runs as needed, here 200 for 200*100MB=20GB memory allocation):

    for i in $(seq 1 200); do echo $i; date; free -m; dd if=/dev/zero of=/tmp/${i}_100MB.dat bs=1024 count=100000; vmtouch -d -l /tmp/${i}_100MB.dat; sleep 60; done
    
  • Sooner or later the OOM Killer should be triggered - check it with grep -i killer /var/log/messages

  • Go to Playground, choose the test system and some of the following parameters: %system, kbmemfree, kbbuffers, majflt/s, pgscank/s, bread/s, %util, runq-sz, ldavg-1

Splunk GUI for Sysstat - Out of Memory Killer Investigation

More information:

Contact:

splunk@compek.net

Release Notes

Version 0.0.6
March 22, 2024

Removed inputs.conf from default folder, read documentation how to configure inputs.conf. Renamed saved search 'Search' to 'Sysstat_Search' to avoid name clashing. Upgraded forms and dashboards.

Version 0.0.5
Nov. 25, 2019
  • a major upgrade - the app will be installed in a "Sysstat" folder instead of "sysstat" to be consistent with the app id (splunkbase requirement)
  • no configuration of sadf.sh script required anymore - it will work with any sysstat version and any modern linux out of the box
  • side by side comparison of performance data coming from different hosts is possible now
  • UI is improved
Version 0.3
June 13, 2014

initial public release


Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk Inc. in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.