Splunk App for AWS Billing allows you to collect Detailed Billing data from which in-depth analysis of usage patterns and spending becomes available for Amazon Web Services environment.
It provides a base for you to extend and articulate your own spending and usage patterns.
Install as you normally would by unpacking all the files in the SPLUNK_HOME/etc/apps directory or uploading using the app manager.
You will need to fill in the following pieces of information in the local directory aws.yaml file. (an example is provided)
Splunk App for AWS Billing allows you to collect Detailed Billing data from which in-depth analysis of usage patterns and spending becomes available for Amazon Web Services environment.
It provides a base for you to extend and articulate your own spending and usage patterns. It converts your billing line items into JSON which is imported and can be processed in SPlunk.
Any feedback, including requests for enhancement are most welcome. Email: magic.monkee.magic@gmail.com
Splunk App for AWS Billing is hosted at GitHub:
Feel free to submit a pull request:
https://github.com/monk-ee/SplunkAppforAWSBilling
Also feel fre to raise any issues here:
https://github.com/monk-ee/SplunkAppforAWSBilling/issues
The application supports the straight Cost model as well as the Blended/Unblended Cost Model. Dashboards for both are provided; use the one that suits your billing settings.
The detailed billing report enables customers to perform detailed cost analyses on their usage.
AWS meters usage in hourly increments; for each product resource in use, a rate is applied for operations performed by usage type in that hour, with each operation comprising a
line item. The detailed billing report shows both blended and unblended rates for each line item. An unblended rate is the cost per hour for a product, usage type, and the
operation performed. A blended rate is an average rate calculated for identical instance usage in an Availability Zone for members of a Consolidated Billing family.
If you have a single AWS account and you don't use Consolidated billing your report only contains the Rate and Cost Fields.
eg NON-CONSOLIDATED: Your search should look like this:
index=aws-bill | timechart sum(Cost) as $ by ItemDescription
If you have have multiple AWS accounts and use Consolidated billing your report contains BlendedRate, BlendedCost, UnBlendedRate and UnBlendedCost fields BUT not the Cost and Rate fields.
eg CONSOLIDATED: Your search should look like this:
index=aws-bill | timechart sum(BlendedCost) as $ by ItemDescription
This means you need different searches and dashboards for the different billing reports eg a Consolidated OR a Single billing account report.
v2.0 Indexes are not compatible with earlier versions of the app, you will have to rerun all your log data and flush the older indexes - the data has changed from csv items to JSON.
You will need to ensure that you have set your billing preferences correctly.
If you are not sure go to Billing Preferences:
You will then need to make sure that you have subscribed to "Recieve Billing Reports" and have setup a valid S3 bucket for your bills.
Now make sure you have also ticked the Detailed Billing Report (with resources and tags):
It's also a great idea to setup an IAM user account that only has readonly rights to your file bucket.
The app is provided as an spl an can be installed using the GUI - feel free to unpack it if you want:
It's file system location is $SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling.
You will need to take the aws.yaml.example and put in your own configuration details then save it as aws.yaml in the $SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/local directory.
For a single account use the following style of aws.yaml:
accounts:
- account_number : 123456
billing_bucket : company-billing
aws_access_key : AAAAAAAAAAAA
aws_secret_key : AAAAAAAAAAAABBBBBBBBBBBBBBCC
For multiple accounts use the following style of aws.yaml:
accounts:
- account_number : 123456
billing_bucket : company-one-billing
aws_access_key : AAAAAAAAAAAA
aws_secret_key : AAAAAAAAAAAABBBBBBBBBBBBBBCC
- account_number : 654321
billing_bucket : company-two-billing
aws_access_key : AAAAAAAAAAAA
aws_secret_key : AAAAAAAAAAAABBBBBBBBBBBBBBCC
Your bucket policy for the programmatic user (the user that accesses the bucket) should look something like this; as a minimum:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetObject", "s3:GetObjectVersion"],
"Resource": [
"arn:aws:s3:::billing-bucket",
"arn:aws:s3:::billing-bucket/*"
]
}]
}
The best practice for this is to create a new policy and user for this, NEVER EVER EVER EVER use your root credentials for this.
The index is called aws-bill
There are three main views:
When the app imports your billing report it adds all of the custom fields from your report.
They appear like this in your report:
You can then use them in a search or to customize a dashboard like so:
index=aws-bill | timechart sum(BlendedCost) as $ by user:Customer
The error log file system location is $SPLUNK_HOME/var/log/splunk/SplunkAppforAWSBilling.log.
Special thanks to Nilesh Khetia who's module I borrowed to make this one http://answers.splunk.com/users/114849/nkhetia
2.0.5
- Supports multiple accounts
- set driven duplicate protection for line items
- events are streamed in json
- file positions are tracked using pickle binary files, to reduce the likelihood of double ups
- all functions have been moved to classes
- boto has been updated to 2.38.0
- old reports are posted to splunk using splunklib index post
- proxy support is introduced - a must for corporate network fun
- pivot tables on LinkedAccountId
2.0.6
- automated history fetching - last 12 months
- automated report processing - last 12 months
- no date fudging
- checksums of s3 billing reports are checked so only changed files are downloaded
- all original zip files are downloaded locally
- the raw data files are unzipped on the system
- fix for UsageStartDate 'fudge' misfiring Issue #5 https://github.com/monk-ee/SplunkAppforAWSBilling/issues/5
- fix for Double Indexed Items - Remove KVMODE Issue #6 https://github.com/monk-ee/SplunkAppforAWSBilling/issues/6
- be aware the application needs more space as it gets a years worth of data and unzips it
- maintenance script will clean up files older than a year (but it is still disabled by default)
2.0.7
- fix recurring position bug
2.0.8
- clear duplicates from positional set - this a cracker - sorry about that folks
2.0.9
- changed the import to KV_MODE json - this is more flexible than commiting to an index at import time
2.0.10
- remove deprecated files
- add doco abput upgrading
Here are some example searches, I hope you find them useful:
index=aws-bill | timechart sum(BlendedCost) as $ by ItemDescription
index=aws-bill BlendedCost !="" | timechart span=1day sum(BlendedCost) as $ by ProductName useother=f
Search for Duplicate Record Id
index=aws-bill | stats count by RecordId | where count > 1
Usage Quantity
Account
Dashboard
- spend summary - include subscriptions
- month-to-date spend by service
- consolidated bill charges by service charges
- bill details by account ****
Cost explorer
- monthly spend by service view
- monthly spend by linked account view
- daily spend view
Pivot
- API Operation
- Availability Zone
- Linked Account
- Purchase Option
- Service
- Tag
The scripts have changed to go back at least 12 months to fecth older reports, this means the old method for getting
reports and processing has been deprecated. DO NOT under any circumstances run the fetch older or process older reports
scripts. At best they will not work at worst they will cause data duplicates.
Currently on upgrade these files will be in your bin directory, delete them for your own safety and sanity.
Versions moving forward will not have these files.
It would be better if you have the time, to flush all your indexes and install the application
"as if for the first time", these will avoid any of the nasty import duplication bugs that were present in the
2.0.5 - 2.0.8 releases.
If you would like older reports than just a year (it defaults to 12), you can change the history value in aws.yaml file:
history: 12
In the $SPLUNK_HOME/etc/apps/SplunkAppforAWSBilling/local directory are the *.p files that keep track of processed records. If you find that you have got twisted or need to
reimport a file you may need to clear the aws-bill index or delete these files. They are basically sets of imported RecordIds - to avoid duplicates.
2.0.11
Issues fixes
2.0.9
- changed the import to KV_MODE json - this is more flexible than commiting to an index at import time
2.0.6
- automated history fetching - last 12 months
- automated report processing - last 12 months
- no date fudging
- checksums of s3 billing reports are checked so only changed files are downloaded
- all original zip files are downloaded locally
- the raw data files are unzipped on the system
- fix for UsageStartDate 'fudge' misfiring Issue #5 https://github.com/monk-ee/SplunkAppforAWSBilling/issues/5
- fix for Double Indexed Items - Remove KVMODE Issue #6 https://github.com/monk-ee/SplunkAppforAWSBilling/issues/6
- be aware the application needs more space as it gets a years worth of data and unzips it
- maintenance script will clean up files older than a year (but it is still disabled by default)
Highlights include
+ more efficient json objects
+ refactored import classes
+ catch blank date fields
+ multiple account set resets
+ new ui pages
Supports multiple accounts
set driven duplicate protection for line items
events are streamed in json
file positions are tracked using yaml files, to reduce the likelihood of double ups
all functions have been moved to classes
boto has been updated to 2.38.0
proxy support is introduced but not implemented - coming soon!
The application supports the straight Cost model as well as the Blended/Unblended Cost Model. Dashboards for both are provided; use the one that suits your billing settings.
Minor bucket fix - changed method to get_bucket() from create_bucket()
Removed .pyc files
1.0 Initial Version
1.1 Bug fixes for boto pathing, error handlers and configuration
Initial Release
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.