Shuttl provides data archive management for Splunk. It supports backend storage solutions such as: ApacheHDFS, Amazon S3, or NFS attached storage. Shuttl works on the bucket level, and leverages the standard Splunk mechanism for archiving data based on total data size or time expiration. Use of Shuttl eliminates the need for Splunk users to implement their own homegrown solution for bulk-moving data to storage backends.
In addition to Archiving, Shuttl is useful for both compliance needs of data retention, as well as improving performance of Splunk. Shuttl also supports archiving the data in CSV format, and therefore, when data is moved to HDFS, it opens up the data to other tools such as Apache Hive and Hadoop Map Reduce to do further data processing and analysis.
For more information see the following blog articles:
Source code is available here: https://github.com/splunk/splunk-shuttl
Quickstart Guide is available here: https://github.com/splunk/splunk-shuttl/wiki/Quickstart-Guide
Setup video is available here: http://www.youtube.com/watch?v=OP7IYNVR5ms
For feedback, please email shuttl-dev at splunk.com.
Now comes with both coldToFrozen AND warmToCold transfer retries!
This means that all failed transfers/shuttl's will be retried periodically (default every 60 secs), instead of "everytime any bucket is shuttl'ed".
User happiness is expected to go up by 137%!
Automatically retries to transfer coldToFrozen failures every 60 secs (configurable)
Splunk Shuttl 0.8.0 release is another major milestone release.
The main new feature is distributed Shuttl operations from the Splunk Search Head. This means that all Shuttl operations will operate on the entire cluster, and not on a per-indexer basis.
Actions include:
Listing of archived buckets
Thawing of archived buckets
* Flushing of archived buckets
There is no special configuration that needs to be done on the Search Head. Shuttl when installed on a Search Head will query the Search Head for all Search Peers (the Search Peers should all have Shuttl installed of course), and all Shuttl operations will be distributed operations.
Extensive testing has been done for distributed failure scenarios. However, if problems are encountered, please report them ASAP for them to be addressed.
Happy Shuttling!
Shuttl version 0.7.2 represents a significant leap in functionality. Thanks to our robust user base for the feature request ideas. Now, Shuttl is easier to setup, supports a new backend (Glacier), reduced archive latency, and full support for the latest Splunk 5.0.
In summary, new features include:
Shuttl tested on CDH3. There is a wirelevel incompatibility with CDH and Apache, so there's documents on how to configure Shuttl to work with CDH3 on the github wiki. https://github.com/splunk/splunk-shuttl/wiki/System-Requirements
Fix spurious error message when a thaw request is issued while thaw is in progress. No duplicates happen, and all requests are serviced.
Simplified the failed bucket transfer dashboard.
When doing thaw operation, the UI can "return" and list a table, before the thaw action completes - Fix is to do thaw operation asynchronously, so the UI returns immediately, while job happens in background.
Amazon S3 tested, and works with no code change - slight display issue for bucket sizes in the UI, bug filed.
Updates:
* Support for Flushing of Thawed Splunk Buckets via the UI
Known Issues being worked on:
Quirks in the Thaw request UI
S3 is not yet tested
* Failed buckets dashboard may report false failures
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.