| Author | Oluwaseun Remi-Omosowon |
|---|---|
| App Version | 1.5.4 |
| Vendor Products |
|
The TA-mailclient add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python, should work on any Splunk
installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
This only fetches emails from the 'inbox' folder when using POP3. Additional mailbox folders can be indexed when using IMAP.
Be sure to set the interval to run this as frequently as required.
It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:
'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'
Images, videos and executables are not indexed.
Includes:
- Splunk SDK for Python (1.6.14)
- Six python 2/3 compatibility (1.15.0)
- mail_lib - supports the calculation of vincenty distances which is used by default
- constants.py - A number of constants / defaults used throughout the mail_lib module.
- mail_common.py - Shared functions used to parse emails and attachments
- exceptions raised by functions used in the mail_lib module.
Version 1.5.4 of the TA-mailclient is compatible with:
| Splunk Enterprise versions | 8.x, 7.x |
|---|---|
| CIM | Not Applicable |
| Platforms | Platform independent |
| Lookup file changes | No lookups included in this app |
This version removes support for unencrypted connections to mailboxes to allow the app pass Splunk Certification.
The is_secure is no longer required and should be removed from the config.
The administrator is responsible for setting the sourcetype to whatever is desired,
as well as extracting CIM fields for the sourcetype.
This app already includes several extractions for different parts of the message that can be reused.
This app will not work on a universal forwarder,
as it requires Python which comes with an HF or a full Splunk install.
Note: Travis CI includes tests for both secure versions of POP3 / IMAP.
TA-mailclient includes the following new features:
This is currently tested against 7.3, 8.0 and the latest version of Splunk Enterprise (v8.1 as at the time of this writing).
Issues can be reported and tracked on Github at this time.
This uses the inbuilt poplib and imaplib that comes with Python by default.
Contributions on github are welcome and will be incorporated into the main release.
Current contributors are listed in AUTHORS.md.
v1.3.0
v0.5.1
v0.5.0
v0.4.9
v0.4.8
v0.4.7
v0.4.6
v0.4.5
v0.4.4
v0.4.3
Mail.save_password() to allow reuse of code when writing other modular inputs.v0.4.2
v0.4.1
v0.4
Note: filename and filecontent are multi-valve fields.
v0.3
v0.2
Questions and answers
Access questions and answers specific to the TA-mailclient at (https://answers.splunk.com/).
Support
This Splunk support add-on is community / developer supported.
Questions asked on Splunk answers will be answered either by the community of users or by the developer when available.
All support questions should include the version of Splunk and OS.
You can also contact the developer directly via Splunkbase.
Feedback and feature requests can also be sent via Splunkbase.
Issues can also be submitted at the TA-mailclient repo via on Github
Future release will support
1. Support for configuration of mail limits in inputs.conf
2. Recursive option to read all folders inside Inbox, and not just emails within inbox.
3. Support indexing mails from additional folders in a mailbox
Note : This has not been tested against an exhaustive list of mail servers, so I'll welcome the feedback.
Also, feel free to send me a list of well known servers that you 're using this with without problems.
Rate the add-on on Splunkbase if you use it and are happy with it,
and share your feedback. Thanks!
TA-mailclient supports the following server platforms in the versions supported by Splunk Enterprise:
The app was developed to be platform agnostic, but tests are mostly run on Linix.
Please contact the developer with issues running this on Windows. See the Splunk documentation for hardware
requirements for running a heavy forwarder.
To function properly, TA-mailclient has no external requirements but needs to be installed on a full Splunk
install which provides python and the required libraries (poplib and imaplib).
Because this add-on runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.
Download the TA-mailclient at one of the following locaitons:
- Splunkbase
- Github
To install and configure this app on your supported standalone platform, do one of the following:
$SPLUNK_HOME/etc/apps/ and restart SplunkInstall to search head - (Standalone or Search head cluster)
Install to indexers
Install to forwarders
Follow the steps to install the TA-mailclient on a heavy forwarder.
More instructions available at the following URL
Configure an email input by going to the setup page or configuring inputs.conf.
For Splunk cloud installations, install TA-mailclient on a heavy forwarder that has been configured to forward
events to your Splunk Cloud instance.
The sourcetype is set by the administrator of the heavy forwarder when configuring the inputs.
You can work with Splunk Support on installing the Support add-on on Splunk Cloud for parsing the mails collected.
This app adds a mail:// modular input and supports a variety of parameters in inputs.conf.
[mail://email_address@domain.com]
interval = 600
mailserver = imap.domain.com
password = mypassword
protocol = IMAP|POP3
disabled = 0
mailbox_cleanup = delete
additional_folder = test,rfc,spam
Once the input is read, the password gets replaced and shows as 'encrypted'.
As such, the password for the mailbox must not be set to 'encrypted'.
The input can be edited if the password needs to be updated, and the password stored in a password
storage endpoint would get updated automatically. Passwords are never stored in clear text.
A different sourcetype can be specified for each input, thus making it possible to have different sourcetypes
for every mailbox. Mailbox cleanup is also managed automatically, and emails are deleted once it has been
indexed.
mailserver - This is a mandatory field and should be the hostname or
IP address for the mail server or client access server with support for retrieving emails via POP3 or IMAP
protocol - This must be set to either POP3 or IMAP
password - Passwords must be set for every account,
or the input will get disabled.
mailbox_cleanup = This indicates if every email should be deleted as it is read,
or delayed until the next interval.
Setting this to readonly prevents mails from being deleted.
The default is readonly. Supported options are:
delayed|delete|readonly
interval - This should be configured to run as frequent as required
to retreive emails. This modular input retrieves up to 20 emails at each run.
A future release to this input might allow the limit to be configured as a parameter to the modular input.
This modular input supports multiple instances, and each input runs at separate intervals.
include_headers - This determines if email headers should be included.
additional_folders - This is an optional parameter containing a comma-separated list of additional folders to be indexed if IMAP is configured for the mailbox.
A copy of the Creative Commons Legal code has been added to the add-on detailing its license.
Data is indexed using a sourcetype specified by the administrator when configuring the inputs.
If nothing is specified, events will get indexed with a sourcetype of mail.
Once an email is indexed, it will not be re-indexed except the checkpoint directory is emptied.
This can be achieved by running the following command:
splunk clean inputdata mail
Logs can be found by searching Splunk internal logs
index=_internal sourcetype=splunkd (component=ModularInputs OR component=ExecProcessor) mail.py
Additional logging can be enabled by turning on debug logging for ExecProcessor and ModInputs.
set the logging level of the ExecProcessor to Debug
/opt/splunk/bin/splunk set log-level ExecProcessor -level DEBUG
/opt/splunk/bin/splunk set log-level ModInputs -level DEBUG
You can find additional ways to enable debug logging on
here.
The TA-mailclient add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text, and replaces it with a placeholder while storing it encrypted within Splunk.
This is developed using the Splunk SDK for Python, should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
This only fetches emails from the 'inbox' folder when using POP3. Additional mailbox folders can be indexed when using IMAP.
This release fixes the key error where additional_folder has not been set.
It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:
'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'The TA-mailclient add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text, and replaces it with a placeholder while storing it encrypted within Splunk.
This is developed using the Splunk SDK for Python, should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
This only fetches emails from the 'inbox' folder when using POP3. Additional mailbox folders can be indexed when using IMAP.
This release fixes the key error where additional_folder has not been set.
It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:
'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'The TA-mailclient add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text, and replaces it with a placeholder while storing it encrypted within Splunk.
This is developed using the Splunk SDK for Python, should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
This only fetches emails from the 'inbox' folder when using POP3. Additional mailbox folders can be indexed when using IMAP.
This release fixes the key error where additional_folder has not been set.
It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:
'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'The TA-mailclient add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text, and replaces it with a place holder, while storing it encrypted within Splunk.
This is developed using the Splunk SDK for Python, should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
This only fetches emails from the 'inbox' folder when using POP3. Additional mailbox folders can be indexed when using IMAP.
This release fixes the key error where additional_folder has not been set.
It supports all 'text/*' content types and several well known scripts (.bat, .js, .sh) detailed below:
'application/xml'
'application/xhtml'
'application/x-sh'
'application/x-csh',
'application/javascript'
'application/bat'
'application/x-bat'
'application/x-msdos-program'
'application/textedit'TA-mailclient v1.5.0 includes the following new features:
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
- Added support for zips and text files within zips
Version 1.1.* added optimised the app and added the following
- Fixed unicode conversion of emails following contributions from Francois Lacombe on GitHub
- Also added static mail preamble for line break. Event breaking configuration may not be
required since the modular input writes individual events separately, but it's always a good idea.
- Additional logging from pop3 / imap
- Removed interval from inputs.conf.spec
- Added additional test cases on Travis CI to test that functionality work
- modularized storage/password functions to make them reusable and simpler
- Also fixed exception handling when dealing with storage/password
- Fixed type casting for boolean parameters (is_secure, include_headers) and port validation
- Merged functions from poputils/imaputils into main code and added additional logs from connection
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
- Fixed unicode conversion of emails following contributions from Francois Lacombe on GitHub
- Also added static mail preamble for line break. Event breaking configuration may not be
required since the modular input writes individual events separately, but it's always a good idea.
- Added delineations and extractions to multipart content
- Removed interval from inputs.conf.spec
- Upgraded Splunk SDK to 1.6.2
- Added additional test cases on Travis CI to test that functionality works
- Fix loading local exceptions
- modularized storage/password functions to make them reusable and simpler
- Also fixed exception handling when dealing with storage/password
- Fixed type casting for boolean parameters (is_secure, include_headers)
- Rewrote sections of mail_common
- Merged functions from poputils/imaputils into main code and added additional logs from connection
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
TA-mailclient includes the following new features:
V1.0.3 is missing some exception handling when writing / reading from storage/passwords. This could cause unexpected results when trying to add an input, or update the encrypted password as a non-privileged user.
TA-mailclient includes the following new features:
This technology adapter add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python,
should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
At present, this only fetches emails form the 'inbox' folder.
Images and attachments are not indexed.
Finally fixed header inclusion/exclusion, and bug in 0.4.5
This technology adapter add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python,
should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
At present, this only fetches emails form the 'inbox' folder.
Images and attachments are not indexed.
Finally fixed header inclusion/exclusion, and bug in 0.4.5
This technology adapter add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python,
should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
At present, this only fetches emails form the 'inbox' folder.
Images and attachments are not indexed.
Finally fixed header inclusion/exclusion, and bug in 0.4.5
This technology adapter add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python,
should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
At present, this only fetches emails form the 'inbox' folder.
Images and attachments are not indexed.
Finally fixed header inclusion/exclusion, and bug in 0.4.5
This technology adapter add-on fetches emails for Splunk to index from mailboxes
using either POP3 or IMAP, with or without SSL.
The modular input also stores takes the password from inputs.conf in plain text,
and replaces it with a place holder, while storing it encrypted within Splunk.
This is built using the Splunk SDK for Python,
should work on any Splunk installation with Python available including SHC.
Passwords should also get replicated between search heard peer members.
For IMAP, this only fetches emails form the 'inbox' folder.
For multipart emails, only 'text/plain' and 'text/html' are indexed.
Images and attachments are not indexed.
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.