Netskope Add-on For Splunk
Overview
Details
PLEASE GO TO DETAILS AND READ THE INSTALLATION INSTRUCTIONS
!! UPGRADING IS SUPPORTED WITHIN THE >= 2.X CHAIN. IF COMING FROM 1.X, PLEASE REMOVE 1.X BEFORE INSTALLING ?=2.X !!
The Add-on typically imports and enriches data from Netskope API, creating a rich data set ready for direct analysis or use in an App. The Netskope Add-on for Splunk will provide the below functionalities:
* Collect data from Netskope via REST endpoints and store it in Splunk indexes
* Categorize the data in different source types
* Parse the data and extract important fields
## DEPRECATION WARNING
* Inputs named "Events (Deprecated)" and "Alerts (Deprecated)" will be removed soon in future releases. So, it is recommended to move to "Events (Iterator)" and "Alerts (Iterator)" inputs.
## DEPRECATION INFO
* Input named "Web Transaction V1" has been been removed from v3.4.0. So, it is recommended to move to the "Web Transaction V2" input.
Splunk Add-on for Netskope
This is an add-on powered by the Splunk Add-on Builder
OVERVIEW
- The Add-on typically imports and enriches data from Netskope API, creating a rich data set ready for direct analysis or use in an App.
- Author - Netskope, Inc.
-
Version - 3.4.0
-
Compatible with:
- Splunk Enterprise version: 9.1.x, 9.0.x, 8.2.x
- OS: Platform independent
- Browser: Google Chrome, Firefox
NOTE: Compatible OS with Web transactions V2 Input: Latest Ubuntu, CentOS, RHEL and Windows
DEPRECATION WARNING
- Inputs named "Events (Deprecated)" and "Alerts (Deprecated)" will be removed soon in future releases. So, it is recommended to move to "Events (Iterator)" and "Alerts (Iterator)" inputs.
DEPRECATION INFO
- Input named "Web Transaction V1" has been been removed from v3.4.0. So, it is recommended to move to the "Web Transaction V2" input.
PREREQUISITES
Netskope Webtransactions V2 input:
- The Web transactions v2 is GA; it is a licensed feature. You must be a web transaction v2 customer to use the WebTxv2 endpoint functionality. Please ensure that web transaction v2 logs are enabled, this is not done by default. Support can ensure this is done. Splunk will get a 404 for the web transactions polling response if they are not enabled.
- If you are using Web transactions V2 input, make sure that the underlying Operating System is as latest as possible, as it is using OS-dependent functionalities while collecting events that might not work in older OS.
- For admins restricting access to the internet for their Splunk heavy forwarders or Cloud IDM, please add a bi-directional firewall to allow the rule to reach "pubsublite.googleapis.com" on TCP:443 and all GCP (Google Cloud Platform) hosts. The Add-on will first send the request to "pubsublite.googleapis.com:443" and fetch a list of GCP hosts to later request them for web transaction events. Check out the below links for getting a list of CIDRs of GCP hosts for setting firewall rules.
- Make sure that SSL inspection is disabled for the mentioned list of IP ranges.
- As Webtx2 input is not supporting HTTPS_PROXY, make sure that it is not configured (in TA, OS, and Network level) Splunk environment.
Events & Alerts Input
- If the Netskope and Splunk admin wants to leverage API v2, they should create a new v2 API token that is fully entitled with every read scope (Refer to section "Details > #SUPPORTED TOKENS AND REQUIRED PERMISSIONS").
NOTE: If Splunk is stopped or input is disabled or some intermediate error occurs from API while TA is collecting data, then it might result in data duplication during that period.
Orchestration Actions (Populate URL list or Filehash list)
- File Hash Alert Action supports only the V1 token which doesn't require setting any scope.
- To use the URL list, Netskope admin will need to increase the APIv2 token's scope to include read AND write scope (Refer to section "Details > #SUPPORTED TOKENS AND REQUIRED PERMISSIONS").
RECOMMENDED SYSTEM CONFIGURATION
- Refer to the Splunk Enterprise system requirements: https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements
TOPOLOGY AND SETTING UP SPLUNK ENVIRONMENT
This Add-On can be set up in two ways:
1) Standalone Mode: Install the Add-on app on a single machine. This single machine would serve as a Search Head + Indexer + Heavy Forwarder for this setup.
2) Distributed Environment: Install Add-on on Search Head and Heavy Forwarde (or Cloud IDM)
* Add-on resides on search head machine need not require any configuration unless user wants to use alert action.
* If alert actions are used, configure account, proxy and logging in configuration page on Search Head.
* Add-on needs to be installed and configured on the Heavy Forwarder system.
* Execute the following command on Heavy Forwarder to forward the collected data to the indexer.
$SPLUNK_HOME/bin/splunk add forward-server <indexer_ip_address>:9997
* On the Indexer machine, enable event listening on port 9997 (recommended by Splunk).
* Add-on needs to be installed on search head for CIM mapping.
NOTE: Here $SPLUNK_HOME is the absolute path where splunk is installed.
SUPPORTED TOKENS AND REQUIRED PERMISSIONS
- Ensure you are configuring inputs or alert actions with an account configured with tokens as shown below.
Event types: Connection, Application, Network, Audit, Infrastructure, Incident
Alert Types: compromisedcredential, ctep, dlp, malsite, malware, policy, quarantine, remediation, securityassessment, uba, watchlist
| Input Name / Alert Action | Supported Tokens / Key |
|---|---|
| Events (Iterator) / (Data Export) | V2 Token |
| Alerts (Iterator) / (Data Export) | V2 Token |
| Events (Deprecated) | V1/V2 Token (V2 Token will be used if both are configured) |
| Alerts (Deprecated) | V1/V2 Token (V2 Token will be used if both are configured) |
| Clients | V1 Token |
| Web Transactions V2 | V2 Token |
| URL List Alert Action | V1/V2 Token (V2 Token will be used if both are configured) |
| File Hash Alert Action | V1 Token |
- If V2 Token is configured, ensure it has sufficient endpoint permissions for respective inputs or alert actions as listed in the table.
| Input Name / Alert Action | Endpoint Permissions |
|---|---|
| Events (Iterator) / (Data Export) | /api/v2/events/data/page, /api/v2/events/dataexport/events/* |
| Alerts (Iterator) / (Data Export) | /api/v2/events/data/page, /api/v2/events/dataexport/events/audit, /api/v2/events/dataexport/events/alert , /api/v2/events/dataexport/alerts/* |
| Web Transactions V2 | /api/v2/events/token/transaction_events |
| Deprecated inputs [ Events (Deprecated) / Alerts (Deprecated) ] | /api/v2/events/data/* |
| URL List Alert Action | /api/v2/policy/* (Read & Write Both) |
Note: * Here "/api/v2/events/dataexport/events/*" means all endpoints starting with the "/api/v2/events/dataexport/events/" prefix. * To update the V2 Token permission, login to the netskope portal and then navigate to Settings > Tools > REST API v2. * V1 Token doesn't require any additional steps of adding permissions. * When configuring "Token V2" on the account page, "/api/v2/events/dataexport/events/audit" or "/api/v2/events/data/page" endpoint permissions will be required for Iterator or Deprecated inputs, respectively.
CAUTION
Note: Webtx2 input is not supporting HTTPS_PROXY.
- This component is required and must be addressed before installing the Netskope TA.
- The TA only supports polling Netskope events at 10,000 items per page for events, alerts or clients input.
- Upgrading is supported within the 2.x version chain. If you are upgrading from TA v1.x, please remove it before installing version 2.x.
- If Splunk is stopped or input is disabled or some intermediate error occurs from API while TA is collecting data, then it might result in data duplication during that period.
CONFIGURATION
Follow the below steps for configuring Netskope Add-on for Splunk
Account
- From the Splunk Home Page, click on Netskope Add-on for Splunk and navigate to the Configuration section.
- In the Account tab, click on the
Addbutton to configure a new Account. - Enter the required details like Account Name (To uniquely identify accounts in Splunk), Hostname, Token V1, Token V2. Click on Save to save the configuration.
Logging
- To configure the Log Level, go to the Configuration -> Logging tab.
Proxy
- To use a proxy as part of the connection to Netskope, go to the Configuration -> Proxy tab and provide the required details. Don't forget to check the Enable option.
Advance Settings (For custom Splunk index usage)
- To change the Base Event Type, go to Add-on Settings tab. It will be the base search for all event types(All the Dashboards of Netskope App for Splunk will refer to this Base Event Type to populate the data in the Dashboards)
Events, Alerts & Clients Input
- To manage the Modular Inputs, navigate to the Inputs section.
- Click on the
Create New Inputbutton provided on the top right and select specific Input Name. - Specify all required parameters needed to configure inputs.
- For Events, Alerts & Clients related input, provide details like Name, Interval, Index, Netskope Account, Start Date Time, Query, Event Types, Alert Types, etc, and click on
Saveto save the input configuration. - If you are collecting data in the custom index then don't forget to update the
Base Event Typeunder Add-on Settings of Configuration Section. E.g. index=main OR index=custom_index_1 OR index=custom_index_2 - User can enable/disable/edit/delete Modular Input by selecting specific action.
Note:
* If token is updated in existing account then existing inputs will start using new token within 30 minutes. If you want to make it use new token, then disable and enable the existing inputs.
* If v1 & v2 both tokens are configured in selected account, then Inputs or Alert Action will first Prioritise v2 token.
* In "Alerts (Iterator)" input, select the specific alert type instead of "All" to get detailed alert events. Selecting each alert type creates a separate thread to collect alerts, which might increase resource consumption.
* The "Alerts (Iterator)" input may cause resource usage to increase if separate alert types are selected (instead of All), since the threading mechanism will collect all types of alerts simultaneously.
* Netskope recommends using the dataexport iterator endpoints whenever possible as they provide greater reliability in log delivery and better performance overall.
* Splunk admins should not query the same event or alert data export endpoint from multiple Splunk instances. Admin can reduce HA capacity requirements by splitting endpoints up between multiple TA, each of which is solely responsible for polling those particular endpoints.
Netskope Webtransaction V2 Input Configuration
Follow the below steps for configuring Netskope Web transaction V2 * Navigate to Inputs -> Create New Input -> Web Transactions V2. * Enter the Input Name, Index & Netskope Account (which has V2 Token with rewuired permission mentioned in ## SUPPORTED TOKENS AND REQUIRED PERMISSIONS). * Click on Save.
CONFIGURATION OF MULTIPLE PARALLEL INGESTION PIPELINE STEPS
Note: * Adding multiple ingestion pipelines, requires significant hardware resources. Reference: https://docs.splunk.com/Documentation/Splunk/9.0.3/Capacity/Referencehardware
If you are having significant input data for Web Transaction V2 input and Splunk is not able to match the ingestion rate, then follow the below steps to configure multiple ingestion pipelines in Splunk to achieve a higher ingestion rate for Web Transaction V2 Input.
1. Disable web transaction V2 input.
Note: If web transaction V2 input is not created then first create it from Inputs -> Netskope Web Transactions V2
2. From backend, navigate to $SPLUNK_HOME/etc/system/local/server.conf and modify [general] stanza to make "parallelIngestionPipelines" property value with the required parallel pipelines value (Recommended max parallel pipelines are 2).
Note: If server.conf files do not exist at the specified location then create it and apply changes.
3. From backend navigate to $SPLUNK_HOME/etc/apps/TA-NetSkopeAppForSplunk/local/inputs.conf and in [netskope_webtransactions_v2://<<web txn V2 input name>>] stanza add "parallel_ingestion_pipeline" property with same value as provided in [general] stanza in step #2.
4. In $SPLUNK_HOME/etc/apps/TA-NetSkopeAppForSplunk/local/inputs.conf there will be [batch://$SPLUNK_HOME/var/spool/splunk/webtxn1/<account_name><input_name>__web_transactions_v2.gz] stanza for the Web Transactions V2 input. Make replica of same batch stanza and its property values in same inputs.conf file.
5. In this replicated batch stanza change the directory name "webtxn1" with "webtxn2" (i.e. [batch://$SPLUNK_HOME/var/spool/splunk/webtxn1/<account_name><input_name>__web_transactions_v2.gz] to [batch://$SPLUNK_HOME/var/spool/splunk/webtxn2/<account_name>_<input_name>__web_transactions_v2.gz]). Do the same and make number of batch stanzas same as the number of ingestion pipelines.
Note: Give the names for webtxn{N} in batch stanza sequentially.
6. Restart Splunk.
Note:
* In case of editing or deleting of Web Transactions V2 input, apply the changes for all of the batch stanzas property in $SPLUNK_HOME/etc/apps/TA-NetSkopeAppForSplunk/local/inputs.conf from the backend.
ALERT ACTION
Two types of Alert Actions * Netskope File Hash Alert Action * Netskope URL Alert Action
Warning:
* If you want to use API v2 token then configure API v2 token in the existing account which is used in URL list alert action. If a new account is created for the existing URL list alert action, then the old URL list will be removed & it will start filling from that point.
END USER LICENSE AGREEMENT
https://www.netskope.com/software-eula
Support
- Support Email: support@netskope.com
- Responses vary on working days between working hours.
- Supported TA: v3.x.x
COPYRIGHT INFORMATION
Copyright (C) 2023 Netskope, Inc. All rights reserved.
Release Notes
Version 3.4.0
- NOTE: This version v3.4.0 has some breaking changes, so please check the 'UPGRADE' section in the Installation tab before proceeding with the update.
- Changed the configuration logic for the "Web Transaction V2" input to obtain the Subscription path and Subscription Key using the V2 token instead.
- Added back pressure mechanism for 'Web Transaction V2' input, to handle the situation of too many files in the spool location.
- Added End Datetime support for the "Events (Iterator)" and "Alerts (Iterator)" inputs.
- Added retry logic for the validation of the "Events (Iterator)" and "Alerts (Iterator)" inputs.
- Changed retry count to be configurable for the "Events (Iterator)" and "Alerts (Iterator)" inputs.
- Updated logging time to GMT/UTC timezone for all inputs.
- Upgraded Netskope SDK to 0.0.33.
Version 3.3.1
- Improved the data collection logic for "Web Transaction V2" input to prevent slow ingestion during header format changes.
- Retry logic for the "Events (Iterator)" and "Alerts (Iterator)" inputs has been enhanced by adding a backoff factor between each retry calls.
- Added validation for the selected events/alerts type on the "Events (Iterator)" and "Alerts (Iterator)" input pages.
- Added feature for auto-read newly configured account tokens without manual changes.
- Security fix (CVE-2022-23491) - Upgraded the certifi library to the latest version.
Version 3.3.0
Version 3.3.0
- Added the support of the "Alert Type" filter for "Alerts (Iterator)" input.
- Enhanced data collection logic for "Alerts (Iterator)" input.
- Added the tooltips for various configuration fields.
- Enhanced the validation in the account page for the "Input Types" field.
Version 3.2.0
Version 3.2.0
- Added support for incident data collection in "Events (Iterator)" input.
- Added CIM mapping for incident data.
- Updated thread allocation logic for iterator inputs.
- Added dynamic wait time between consecutive API calls for iterator inputs.
- Upgraded AOB to v4.1.1
- Added API rate limiting feature for iterator inputs.
- Enhanced Account token validation.
Version 3.1.2
- Enhanced the logic to provide a seamless upgrade
Version 3.0.0
- Updated the data collection approach for Events and Alerts input.
- Updated the checkpoint format for Events, Alerts, Clients Input.
- Added support of API Token version 2 in Events, Alerts & URL List Alert Action.
- Invalid URLs handling is added in URL List Alert Action.
- Upgraded AOB to v4.0.0
- Upgraded internal libraries
- Added new input named "Netskope Webtransactions V2"
- Added feature to reconnect on Idle Connection Timeout in Web Transaction V2.
- Added feature to reconnect in case of any error during event processing in Web Transaction V2.
- Added support for multiple ingestion pipelines in Web Transaction V2 input for better performance.
- Only TA v3.0 is supported by Netskope