GCP Application Template for Splunk

This application template provides visualizations, reports, and searches for Google Cloud Platform data gathered utilizing the Splunk Add-on for Google Cloud Platform, Cloud Functions (see https://github.com/splunk/splunk-gcp-functions) or GCP DataFlow Splunk Template. The purpose of this application template is to provide a starting point for various use cases involving GCP data. Add to, delete from, and modify this template to fit your own requirements.
Correlate other data sources with GCP Data to provide greater Operational or Security Intelligence.
Note - Metrics data can be collected in 3 ways depending on your requirements - GCP Add-On (Event), Cloud Functions (Event or Metrics Store), Splunk Infrastructure Monitoring Add-On.

Each dashboard has the flexibility to allow for different collection and indexing methods. The reason for the need for this flexibility is that the format of the json data collected from PubSub differs between the Add-on and DataFlow - an additional "data." prefix layer is present with json content from the Add-On. Also, for performance reasons, it may be beneficial to use indexed extractions for the json content, which will allow for faster searches with tstats - the dashboards can be configured to use either standard and tstats searches. (see below for setup instructions)
Where metrics are required, it is possible to use metrics collected via the Add-On, Cloud functions (into metrics store), or by pulling from Splunk Infrastructure Monitoring using the SIM Add-On.

Pre-requisites / Dependencies:

Install the Google Cloud Platform Add-on for the knowledge objects. https://splunkbase.splunk.com/app/3088/

Add-Ons / Collection methods

This GCP App supports data to be collected into Splunk via the following methods:

1) Splunk Add-On for Google Cloud Platform (https://splunkbase.splunk.com/app/3088/)

All Data collection methods/sources supported (PubSub, Metrics, GCS)

2) GCP DataFlow. https://cloud.google.com/blog/products/data-analytics/connect-to-splunk-with-a-dataflow-template

All Data collected supported. (pre-install the GCP Add-On, and ensure all sourcetypes for HEC are set to the correct values)

3) Cloud Functions https://github.com/splunk/splunk-gcp-functions

All Data collected supported. (pre-install the GCP Add-On, and ensure all sourcetypes for HEC are set to the correct values)

4) Splunk Infrastructure Monitoring Add-on (SIM) https://splunkbase.splunk.com/app/5247/

All Metrics collected by SIM supported

Follow the instructions for these collection methods before setting up this app.

You will need assets information via Cloud Functions (see https://github.com/splunk/splunk-gcp-functions), or by using gcloud command / cron schedule (https://cloud.google.com/asset-inventory/docs/exporting-to-cloud-storage). For assets data, make sure you use the assets sourcetype below. (additional information is available here - https://www.splunk.com/en_us/blog/partners/taking-inventory-of-your-google-cloud.html)

Setup / Installation

Macros

The GCP app requires some initial setup of macros to work with your Splunk environment. Set the macro settings according to the tables below:

(Note that the current macro settings for the app can be viewed on the "setup" page on the App - if these are blank/empty, you will need to set them to the correct values before the dashboards work)

Splunk Add-On for Google Cloud Platform

If you are collecting the GCP Pub-Sub data via the GCP Add-On, and also collecting metrics via the Add-On, then you will need to set the following :

(note that if you are collecting metrics via Cloud Functions into Metrics store or using SIM, then the two metrics macros need to be set per the instructions below)

DataFlow

If you are using DataFlow (without transformations) to collect the data from PubSub with no compatibilty settings, then change the following. (note if you use compatibilty, you do not need to change this, and keep as addon):

SIM / SignalFX

If you are using the SIM / SignalFX (Splunk Infrastructure Monitoring) to collect metrics from GCP, you will need to use and configure connections on the SIM Add-On - https://splunkbase.splunk.com/app/5247/

Then set the following:

Cloud Functions

PubSub: Cloud Functions can send data in either Add-On or DataFlow formats - refer to the function documentation for details, and set the macros accordingly.
Metrics: Cloud Functions can send metrics in either Add-On format (event index) or into Metrics Store. If you are collecting metrics into the event index, use the same settings as the Add-On, otherwise if using the metrics store, use the following:

Search Performance

If you want to have significantly faster searches using indexed json extractions with tstats, you will need to set the following:
(default is notstats)

Note also that you will need to apply props.conf and transforms.conf updates to your local GCP-Add-on settings to apply this setting, as it requires indexed extractions. (see below).

Setting this macro (keeping default) to "notstats" will use standard searches, but will provide slower "standard" search performance, but will not require any changes to your GCP Add-On configuration.

Props/Transforms

If you want to use tstats based searches for faster performance, you will need to apply these changes to your props.conf / transforms.conf in the local directory of the GCP Add-On.

props.conf

[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false
SHOULD_LINEMERGE = false
TRUNCATE = 300000
CHARSET=UTF-8

[google:gcp:assets]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
KV_MODE=none
INDEXED_EXTRACTIONS = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
disabled = false
TRANSFORMS-sourcetype_splunk_gcp_compute_instance=gcp_compute_instance
TRUNCATE=50000

[google:gcp:buckets:jsondata]
KV_MODE = none
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false

[google:gcp:compute:instance]
KV_MODE = none
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false

[google:gcp:compute:vpc_flows]
KV_MODE = none
INDEXED_EXTRACTIONS = json
AUTO_KV_JSON = false

limits.conf

As some of the json in GCP's message payloads are large, you will need to apply this update to your limits.conf: ($SPLUNK_HOME$/etc/system/local/limits.conf)

[kv]
limit = 0
indexed_kv_limit = 0
maxchars = 20480
maxcols = 0
max_extractor_time = 2000

Release Notes

Version 1.1 Jan 2021

Initial release

Version 1.2

(minor updates and bugfixes)

Bugfixes (Compute Overview, vpc overview, iam overview)
Update to VPC & "Security Overview: Public Access" dashboards to align with other dashboards with datatags (compatability with Dataflow)
Update adding additional index for assets - gcp_assets_index. Backwards compatible
Update to default setting of usetstats macro - default is NOT to use this, i.e. the dashboards work without changing the default add-on sourcetype settings
Documentation update to transforms.conf and props.conf descriptions in the documentation:
- remove unused changes/ fix to props definitions which causes error
- added props.conf indexed extraction for google:gcp:buckets:jsondata
- added sourcetype descriptions
Added Service Account Activity External members access activity to IAM Activity Dashboard
Added VMs created by Default Service Account into IAM Activity Dashboard
Added Live Migrated Hosts to Compute Engine Overview