Secure eStreamer Client (f.k.a. eNcore) for Splunk is a plugin based eStreamer client built from scratch in Python designed to deliver fully qualified event data to Splunk from Secure Firewall (f.k.a. Firepower) 6.x platforms. If you have experienced problems getting the Cisco eStreamer for Splunk app version 2.2.1 and 2.2.2 working with Secure Firewall 6.x you should move to this new application so that you can leverage the many improvements listed below.
Secyre eStreamer Client for Splunk is a Technical Add-on (TA) designed purely to collect data and be installed on a forwarder.
New in this solution:
Resolved Issues in V 3.5.0
v5.2.9
*Fixed JSON output in several fields including: Connection Events: User Data, Iface/Egress Interface, converted byte string outputs to regular strings in the ssl context, id fields
Fixed "BlockedReasonId" key error present in FMC < 7.0 builds which was causing the program to stop parsing during IDS enrichment.
*Various bug fixes including:
- Fixed an issue with Metadata events, record type 112, contained logic to handle blocked reason ids that were not defined as part of the record type
- Removed offset warnings issue in malware events for ingress/egress vrf fields
- Corrected Syntax error in default cache values
Added CIM compliant event types and tags to default conf file definitions, CIM modeled event types include authentication, ids, malware, network resolution, network traffic and ssl events. For more details please view the eventtypes.conf file [https://github.com/CiscoSecurity/fp-05-firepower-cli/blob/master/assets/splunk-collector/default/eventtypes.conf]
Supports FMC 7.x
Removed byte hex encoding for file hash fields, malware event fields (records 125,502 and 511) no longer contain the b'<file_hash>' wrapper encoding.
Added additional configuration options for packet records, you can now select whether or not to include the original packet in the record which contains the payload and the packet header, this configured using the following variable in the estreamer.conf
includeOriginalPacket: true in (https://github.com/CiscoSecurity/fp-05-firepower-cli/blob/master/default.conf#L56) estreamer.conf
Enabled configuration of output types for packet data (record type = 2), the following types are now supported in the estreamer.conf
Hex (default)
Ascii
*UTF-8
configuration settings are case sensitive ('hex', 'ascii', 'utf-8')
To configure the packet data output type modify the 'packetEncoding' variable in the estreamer.conf, under subscriptions
estreamer.conf --> packet settings
Fixed issue with 110 events
Fixed an issue with malware and file events, record types 500 and 502. This effects 6.x and 7.x FMC versions. The issue was a bug within the Splunk adapter that did not convert the fileAnalysisStatus field properly. This would cause version 5.x to hang on parsing
Additionally, 7.0x requests for the newer event format for malware events were not implemented, still requesting the old format, now version 7 for malware events will retrieve block type 79 in the estreamer guide, which includes ingress and egress network routing information.
eStreamer 7.1 guide
https://www.cisco.com/c/en/us/td/docs/security/firepower/710/api/estreamer/EventStreamerIntegrationGuide.pdf
*Fixed original src ip naming conflict in legacy record 110 events (only effects versions below 7.0)
*Fixed parsing issue with legacy user event record types, type 62
*Fixed bug with user record type 98, corrected parsing issues and removed additional binary characters present at the end of usernames
Updated support for new Intrusion and Connection Event structures in FMC 7.0/7.1
This release handles the removal of XFF (record type 110) events and provides original client proxy Ip information directly within IPS events
*Additional Fields added include:
Intrusion Events:
Original Source IP (Proxy IP's0
Blocked Reason ID
Connection Events:
SourceSecurityGroupTag
SouceSecurityGroupTagType,
DestinationSecurityGroupTag
DestinationSecurityGroupTagType
ThreatIntelligenceCategory
IngressVRFName
EgressVRFName
SourceIpDynamicAttributes
DestIpDynamicAttributes
x5.0.1 - Reverted constants file to prior for python3 messaging during certificate install
Added support for Splunk status in .splencore.sh script
Fixed App Inspect failure due to no default settings in app.conf for estreamer.conf reload triggers
*Added additional error handling for local cache storage
*Fixed bug associated with "JSON settings error" in estreamer.log which would periodically stop the encore process since the configuration file could not be read into memory.
*Specifically modified the 'stop' command to safely terminate existing estreamer python connections to the FMC
More Detail: This latest update, v4.8.1 in Splunkbase, should provide more robust handling of the FMC connection, and eliminate the "JSON settings error" from the encore logs. Additionally, we have created CSCvz51007 to address the potential FMC condition of reaching too many client connections. I would highly encourage an update to 4.8.1 if you are seeing any timeout issues, as always please reach out to our mailer here if you are experiencing problems or create a Cisco Support TAC case so we can promptly investigate.
*Modified the ./splencore.sh stop command to wait for TCP termination vs pkill, this is likely the issue behind "JSON settings" error and additional idle timeouts as the FMC server was receiving multiple session requests before older sessions were properly terminated
*Fixed incorrect value in the disposition field for malware events
*Fixed incorrect disposition field value in malware events
Fixed errors in the stop command which not consistently terminate encore processes and remove pid files
Modified handling of XFF events
*Modified clean script which is believed to cause issues with the bug "Invalid JSON settings"
*Updated stop script for multi-thread mode
Added additional transforms to the props.conf to support dashboard 1.60 updates
Modified the stop utility to remove temporary pid file and terminate all estreamer client processes, whereas in versions this only worked in single thread mode
Added support for VPN connection events, record types 170/171, user login/logoff attempts
Added additional fields to support XFF HTTP URI events
Fixed encoding bug "UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)"
Updated multiple rec types with src_host_ip field, this requires FMC host discovery policy to be set
Added "src_host" field to multiple record types, this field provides traceability do the original host ip.
*4.0.9 Update
Removed ORIGINAL SOURCE IP error, this potentially would cause applications errors due to the reference of a non-existent data type. Please reference the full setup guide for Splunk 8.0/Cloud changes
Splunk 8.0 release, please see full guide for install/setup details
4.0.8 includes
reordered event format to include time in the beginning of the record, reset default MAX_LOOKAHEAD accordingly for increased performance
Encore version 3.x support only legacy Splunk 7.x with python2
Modified event structure to include event_sec in the beginning of the record for increased performance
*Added additional ipv6 formatting for XFF events
Corrected performance issues associated with outputting various pcap data types, this feature will be revised and re-released in a future version
Modified definition of event_sec for connection events, in the FMC all connection events start at first packet time which is now the value used for indexing. The first_pkt_sec field is still preserved to support backward compatibility.
*Fixed bug with the initiatorIpAddress field in correlation events
This is the rewrite for the SourceFire eStreamer client.
The Cisco Event Streamer (also known as eStreamer) allows you to stream System intrusion,
discovery, and connection data from Firepower Management Center or managed device (also
referred to as the eStreamer server) to external client applications.
eStreamer responds to client requests with terse, compact, binary encoded messages – this
keeps it fast
eNcore is a new all-purpose client which requests all possible events from eStreamer, parses
the binary content and outputs events in various formats to support other SIEMs
*Updates for 3.5.8
*Bug fix - microseconds on pcap data now use the proper field name 'upacket_sec', and seconds use 'packet_sec'
Version 3.5.7 Updates
*Added back aliasing for action/blocked fields
Version 3.5.6 Updates
Removed default disabling of the cisco:estreamer:data source type
Removed duplicate aliasing for action/blocked fields
eNcore v3.5.3 resolves issues with previous v3.5.x versions where eNcore would crash under certain conditions.
All eNcore v3.5.x versions provide significant performance enhancements over pre-v3.5 versions.
eNcore version 3.5.0 features performance improvements – the ability to process a significantly higher event rate. This improvement requires no additional configuration on the part of the user.
However, if the eNcore platform has four or more cores, then additional performance improvement can be gained by adjusting a parameter in the configuration file, estreamer.conf. This parameter is called “workerProcesses” and is highlighted below:
{
"connectTimeout": 10,
"enabled": true,
"workerProcesses": 4,
"handler": {
--- rest of config file omitted ---
The highlighted line shows the “workerProcesses” to be set at 4. It can be set anywhere from 4-12, but with four or more cores, testing showed the best performance when set to 12.
See Details section for more Release Notes on Version 3.5.0
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.