The Splunk GUI for SysStat provides visualization of the system activity collected with the SysStat package. This includes I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization among others. The statistics can be collected in near real-time from as many hosts as you like (potentially thousands). The SysStat package is already present in all major Linux distributions and likely already installed, is a non-intrusive, secure and lightweight way to collect system activity information.
The app installation folder was changed from "sysstat" to "Sysstat" to be indentical with the app id (Splunkbase's requirement). You need to disable old "sysstat" app. NOTE: please do a backup before an upgrade.
RPM-based distros (Redhat, CentOS):
yum install sysstat
DPKG-based distros (Debian, Ubuntu):
apt-get install sysstat
Redhat, Centos:
Edit /etc/cron.d/sysstat and change
`*/10 * * * * root /usr/lib64/sa/sa1 -S DISK 1 1`
to:
`* * * * * root /usr/lib64/sa/sa1 -S XALL 1 1`
Debian, Ubuntu:
Edit /etc/cron.d/sysstat and change
`5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1`
to:
`* * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1`
Edit /etc/default/sysstat and change
`ENABLED="false"`
to
`ENABLED="true"`
Edit /etc/sysstat/sysstat and change
`SADC_OPTIONS="-S DISK"`
to
`SADC_OPTIONS="-S XALL"`
[script://$SPLUNK_HOME/etc/apps/Sysstat/bin/sadf.sh] disabled=false interval=60 sourcetype=sysstat
'-A This is equivalent to specifying -bBdFHqSuvwWy -I SUM -I ALL -m ALL -n ALL -r ALL -u ALL -P ALL'
(your line can differ). Then replace "-A
" parameter in the sadf.sh script in the $SPLUNK_HOME/etc/apps/Sysstat/bin directory with a string from the manual page.Per default the sadf.sh script looks like this:
LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -A
Find a line in the man page and modify the script to:
LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -bBdFHqSuvwWy -I SUM -I ALL -m ALL -n ALL -r ALL -u ALL -P ALL
then remove parameters which you don't need, for example "-I ALL", so the final string in the script will be similar to (your string can vary!):
LC_ALL=C sadf -t -s $(date -d "2 min ago" +%H:%M:%S) -- -bBdFHqSuvwWy -I SUM -m ALL -n ALL -r ALL -u ALL -P ALL
If after this change the sadf/sar stops working, then you need to remove system activity files, please read the sysstat FAQ: http://sebastien.godard.pagesperso-orange.fr/faq.html#sar : "The list of activities that are saved in a file can no longer be modified once the file has been created. So it is important to use the proper options the first time sadc is executed (whether via a crontab, a script like sa1(8) or even the script used to insert a RESTART message when the machine is rebooted)."
This optimisation is not enabled by default because the "-A" options includes different parameters depending on the version of sysstat package and used linux distribution. If you include parameters that your system doesn't support, the script will fail.
For sysstat/sar related problems check http://sebastien.godard.pagesperso-orange.fr/faq.html
Let's simulate an out of memory (OOM) condition and analyse it using sysstat.
Run this oneliner (modify the number of runs as needed, here 200 for 200*100MB=20GB memory allocation):
for i in $(seq 1 200); do echo $i; date; free -m; dd if=/dev/zero of=/tmp/${i}_100MB.dat bs=1024 count=100000; vmtouch -d -l /tmp/${i}_100MB.dat; sleep 60; done
Sooner or later the OOM Killer should be triggered - check it with grep -i killer /var/log/messages
%system, kbmemfree, kbbuffers, majflt/s, pgscank/s, bread/s, %util, runq-sz, ldavg-1
Removed inputs.conf from default folder, read documentation how to configure inputs.conf. Renamed saved search 'Search' to 'Sysstat_Search' to avoid name clashing. Upgraded forms and dashboards.
initial public release
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.