Outcold Solutions provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses to reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications to help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer a unique solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance and cluster health.
We provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.
All our solutions are powered by the Collectord, a container-native software built by Outcold Solutions that provides capabilities for discovering, transforming and forwarding logs, collecting system metrics, collecting metrics from the control plane of the orchestration frameworks and forwarding network activity. Collectord provides flexible and powerful tools for transforming logs. With our software you can hide sensitive information from the loglines before forwarding them. With Collectord you can reduce the licensing costs associated with logging aggregation by choosing which data you want to forward from the log streams. Collectord forwards container logs, host logs and can discover logs written by the containerized applications.
See detailed metrics from containers and processes, including performance metrics, utilization metrics and security insights. Forward application-specific metrics, exported in Prometheus format. Use prebuilt Splunk dashboards for a comprehensive overview.
Aggregate logs from containers, applications, and servers. Use flexible mappings to filter logs enriched with container metadata, correlate logs with metrics, and leverage Splunk capabilities for analyzing logs. Use Collectord to transform logs before they reach Splunk, remove sensitive information, remove PII data to help keep your logs GDPR compliant. With Collectord you can reduce licensing and storage costs by choosing which loglines you want to forward.
Diagnose cluster issues by looking at historical events, monitoring allocations, and regulating cluster capacity. Leverage pre-built alerts for monitoring the health of the clusters out of the box.
Define access to the data by clusters, namespaces and even pods or containers. Review network activities, happening inside your cluster, and outside connections. Verify containers running with elevated security permissions. Use audit logs for monitoring changes in deployments.
Use one tool to collect and forward logs and metrics required by developers for reviewing performance and health of their applications. With the annotations developers can define how they want to see the data in log aggregation tool, specify multiline log patterns, removing terminal escape codes, override types, sources and indexes.
Supports collectorforkubernetes version 5.23.x and below (see https://www.outcoldsolutions.com for latest configuration)
- To better support installations with large number of nodes and containers, default behavior for most of the dashboards
is to require pressing a Submit button after selecting filters.
- Overview Dashboard - new table with Not Ready Containers.
- Pod Dashboard - include container statuses table.
- Audit Dashboard - include user agent, and update compatibility with latest audit formats.
- Audit Dashboards - small performance improvement for the new installations.
- Host dashboard - show node conditions table.
- Host dashboard - show only external eht* interfaces in network stats.
Collectord updates:
- Implement new and improved watch mechanism for Kubernetes resources to handle large clusters.
- Change the default pipe join configuration to have max size of 1MB instead of 100KB.
- Allow...
Collectord updates:
Supports collectorforkubernetes version 5.22.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Workload dashboard - add Pod OwnerKind and OwnerName, PriorityClass, and Pod Requests/Limits
- Address too many data points in host and workload dashboard in network graphs
- Additional CPU Metrics: CPU IOWait, Steal and Idle in Top Hosts dashboards.
- Showing CPU IOWait in Host dashboard.
- Alert Container CPU Throttled - exclude container with low CPU usage.
- New dashboard Review->Disk Stats for the host.
- Exclude virtual ethernet interfaces from host dashboard.
- Support memory limits and requests expressed in milli-bytes.
Collectord updates:
- Allow disabling IP address Lookup in net_socket_table input.
- Better handling of zombie processes in proc_stats input.
- Allow configuring user Splunk outputs using CDR SplunkOutput.
- Allow blacklisting labels from forwarded metadata.
...
Supports collectorforkubernetes version 5.21.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Compatibility updates for the version 5.21 of Collectord
- New Dashboard: Review -> CPU (Throttled, Limits, Requests)
- Alert update: High amount of GRPC errors
- Alert update: Container CPU Throttled
- Network tables update: show UDP connections for Host, Workloads, Containers, and Pods
- Network Connection Dashboard: allows filtering by namespaces
- Show maximum and average number of Pods per cluster in Clusters (Allocations and usage) dashboard
- Update Resource Quota dashboard to support comparing milli-cores and cores
Collectord updates:
- Support for global replace configurations for Collectord, allowing to sanitize data before forwarding to Splunk
- Support journald as logging driver for container logs
- When both volatile and persistent journald destination exist, ...
Supports collectorforkubernetes version 5.20.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Show Pod conditions on the Pod dashboard
- Bug fix: Pods dashboard filters out pods not on the host network.
- Compatibility updates for the version 5.20 of Collectord
Collectord updates:
- Multi-architecture images for amd64 and arm64
- Allow sending logs to multiple Splunk HEC endpoints simultaneously
- New annotation collectord.io/volume.{N}-logs-onvolumedatabase
to keep acknowledgement information about forwarded logs on the volume
- Allow including placeholder templates in the annotation collectord.io/volume.{N}-logs-glob
- Support for new outputs (ElasticSearch and OpenSearch)
- Collectord produces diag file without performance data, if flag --include-performance-profiles
is not set
- Use IMDSv2 for AWS metadata
...
5.19.390 - 2022-10-17
Supports collectorforkubernetes version 5.19.x and below (see https://www.outcoldsolutions.com for the latest configuration)
- Update dashboards for the latest changes in the metric names for API Server, Controller, and Scheduler
- Update the Kubelet dashboard to support various container runtimes
- Audit (users and namespaces) dashboard: show access to non-namespaces resources
- Logs dashboard: show container and pod as separate filters
- New alert for Collectord alarms for node diagnostics (reboot required and entropy)
- Bug fix: misprint in "Cluster Warning: container CPU is throttled" alert
Collectord updates:
- Splunk output supports maximumMessageLength to truncate messages exceeding this size
- Splunk output supports requireExplicitIndex to ignore all events that don't have explicit index defined
- Collectord monitors if node requires reboot
....
Collectord updates:
- Support cgroupv2
- New ability to specify the message field name for the logs extraction with annotations extractionMessageField
- Collectord improves grace period for expired licenses allowing to bootstrap new nodes for 14 days
- Support of journald database written with systemd library 247+
- Upgrade go runtime to 1.17.9
- Bug fix: cleanup the diag, exclude the real license key
- Bug fix: collectord reports high CPU usage for just started containers or hosts
- Bug fix: update pods/container labels when user updates them (prior restart was required)
- Bug fix: set now as a date for container logs with corrupted log files instead of 0 timestamp
- Bug fix: include the values of whitelists and blacklists in diag
...
Requires collectorforkubernetes version 5.16.363 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Collectord updates:
5.16.360 - 2021-03-16
Collectord updates:
Collectord updates:
\0
symbol from the labels values in the prometheus metricsRequires collectorforkubernetes version 5.15.300 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Collectord updates:
Requires collectorforkubernetes version 5.12.271 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
- Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
- Bug fix: reduce warnings failed to get the new event in pipeline - submitted
- Stability improvements
Collectord updates:
Requires collectorforkubernetes version 5.11.260 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Collectord updates:
Requires collectorforkubernetes version 5.9.240 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Requires collectorforkubernetes version 5.8.230 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Requires collectorforkubernetes version 5.7.220 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
Requires collectorforkubernetes version 5.6.212 or above (see https://www.outcoldsolutions.com for latest configuration)
Read more https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
Requires collectorforkubernetes version 5.5.202 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
- Fixed: Interval 0 in prometheus input can crash the collectord.
- Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from
finding the files in the volume.
Requires collectorforkubernetes version 5.4.201 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
- Fixed: Better handling rotated files (less open fd)
- Fixed: Events input can hang in the err loop.
Requires collectorforopenshift version 5.4 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
- New: Attach EC2 metadata fields
- New: Basic Auth for Proxy (License Server and Splunk)
- Fixed: Collectord verifies reports CRI-O as unsupported runtime.
- Fixed: Rare crash on Prometheus metrics definition.
- Fixed: Better handling of acknowledgment database corruption.
- Fixed: When handling incorrect indexes, collectord can send index with an empty string, that Splunk recognize as an incorrect index
Requires collectorforkubernetes version 5.3 or above (see https://www.outcoldsolutions.com for latest configuration)
https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
5.2.180 - 2018-10-29
- Fixed: misprint in the search for showing alerts
5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC
5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations
Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC
5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations
Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations
Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
Highlights:
- Application logs
- Annotations for fields extraction, hiding sensitive information, time extraction, redirecting to /dev/null, stripping terminal colors and more
For more details:
https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/
3.0.23 - bug fixes release
3.0.22
New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.
Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/upgrade-2-to-3/
Requires collectorforkubernetes version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)
New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.
Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/upgrade-2-to-3/
Requires collectorforkubernetes version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)
Requires collectorforkubernetes version 2.1.59.171209 or above
- Updated author and description
Requires collectorforkubernetes version 2.1.59.171209 or above
Requires collectorforkubernetes version 2.1.59.171209 or above
Requires collectorforkubernetes version 2.0.37.171023 or above
kubernetes_node_labels_LABEL1=VALUE1
to kubernetes_node_labels=[LABEL1=VALUE1,LABEL2=VALUE2]
.Updated links to official documentation for installation instructions.
Fix labels on Kubernetes Dashboards (Most of the filters has incorrect label Daemon Sets)
Monitoring Kubernetes
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.