icon/x Created with Sketch.

Splunk Cookie Policy

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website. Learn more (including how to update your settings) here.
Accept Cookie Policy

We are working on something new...

A Fresh New Splunkbase
We are designing a New Splunkbase to improve search and discoverability of apps. Check out our new and improved features like Categories and Collections. New Splunkbase is currently in preview mode, as it is under active development. We welcome you to navigate New Splunkbase and give us feedback.

Accept License Agreements

This app is provided by a third party and your right to use the app is in accordance with the license provided by that third-party licensor. Splunk is not responsible for any third-party apps and does not provide any warranty or support. If you have any questions, complaints or claims with respect to this app, please contact the licensor directly.

Thank You

Downloading Monitoring Kubernetes - Metrics and Log Forwarding
SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_523430.tgz) ec2e50eda09241e124ac2a35278e19b2d16a83dbc0de601d98bdf3817315c56d SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_522422.tgz) 75a035c4b18f9e40548e246039a3e56708a793536ba91ab101bf09c6896b989c SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_522420.tgz) f484e540972faad0c8d3cd5f026a63d9ed43457aa453d2ac817eb38e4e08a0f8 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_521410.tgz) 142ac5481ca163c4671d12239abac23b8b849d68cf619cb71ed5fe062eaf050d SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_520400.tgz) 3735a610df336a1cf61b0c67b7a35c7e6a9262070afb3fd6ee5111457b41d516 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_519390.tgz) 3b6604a934d07762a9fe000db3ff837ed4ece0a616421b6732b8f2fc2238a7c4 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_518380.tgz) 1c58fb096e6dabcadcf356231894938d7dcc144d01ef7a33d3cd356cb6267c9a SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_517372.tgz) 7fd26532afba8cb4329ad0fd5198c024b87710728d32325a4271a440934aedc3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_517370.tgz) 55c84b4c50b052225c17ebd0768c86f886b4975e18f60fc6005c4a64c556a788 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_516363.tgz) b261b2fa09133dc18a1b713ab38fa3e621fc4749b165384a0fa94c6011bbad97 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_516361.tgz) 278800aee92a0a6e3a3f2ae56ef68c2f6ee730cdde2d7b73e0324dc264e29299 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_516360.tgz) bbed8f7d50e9648fae8ee37ef0eff5abca4396d381bc3e167c9e8f6d4bcb8b77 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_516352.tgz) 2ddfdd008bc302e07400c5fe19e566cdb992664e0e3d1b74d123f13e4e0ee967 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_516351.tgz) a77558dfedc6541204c5e7dfc146359ce6ce5c722da8c65708e8f08dbad4733e SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_515300.tgz) 79ad9ef3e7826db4b78190a4edd102d709340ff424d748cf8c7ee24aee0db1a3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_514280.tgz) fc4f391503082446166a2cdfa7988a78cd0d65063016f831617ee03b9ca5a402 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_512272.tgz) 467080ed1272a963bccbc1c8a8446bac5ead1ba3c1637145367def41615926cc SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_512271.tgz) dd52412890a4b4853837ece0a63d659ac299bea71e39480dbd79ceb6619ec1a3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_512270.tgz) 30ae09503d3523a6e4b004d5283356d5710b7e92fb12516312b7b1a192189c97 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_511260.tgz) 309eb9f67a891c517a7e862430d7205ecbf0b26587ad49679d4d312a9a9e48ff SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_510250.tgz) 23207fb025d7198d49b5a0c4153f7c6699e45b70a66db52ccbaacdb5b535efe4 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_59240.tgz) eee17eb4922895d47cbc97492e0fcd52e53a3082a358d259165fe96a864063c3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_58231.tgz) 63ac5aaf5a82028d25a7adb348ce2f641e61d859799c39e01760ad3d44f711c6 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_58230.tgz) fb740976c4bcb2e3e0f292c8a8a92d1b5fe96843e34a31a21730a2111a10d892 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_57220.tgz) b6c13fb8619de9c9933e89dad26f92d6d347450e7abf636593b72d37ed6ea245 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_56212.tgz) 0cc3509f956730487f8929ba49c505808a43ff1816d407cd200f810787cda906 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_55202.tgz) 2780f36ab72946185cef6e483c2e4c225417e1c229af215c33e1e9ade93cd110 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_54201.tgz) 57d29a324d3bf5d97c1730ab3d1814fd083a1a80c33a2cfa1e59068c8ed36c34 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_54200.tgz) 7a9c07bcb1b12c91eb63ad2771cff86c343af87c0504e5de729e8398b5c2ff30 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_53190.tgz) 34984c8b10d1ba523e65d4236872deac98429a765aea8a477150f590bf1be080 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_52181.tgz) 68a8731ea6eb1aedc04c7b60005be6f052555bb70731c711cf350e8cdf4860b1 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_52180.tgz) 5c83c3399c9270350ec744c531820b9995e9abf1cd758872007427c9ed6defa8 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_52179.tgz) 1af2291a12c385272a20ca160c58115ac745ffd39d3e92911f318e6694e3ec9b SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_52176.tgz) a0d1057c4c267a938702db283b025d6972f0b6f047eb28a1297612954de202dd SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_51175.tgz) d19de02f57582c4ecb3bab970987618a1338fa1282e3053c36ecde002e40db57 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_50174.tgz) c46f04681fe9daaa94bfcfcae54b78cf693a75a5a35201f500f9dd98af8584f3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_4024.tgz) a57ce4d09b3a67480c448970facdac46796eee7f7d0bc32c2d81d1b287b6d623 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_3023.tgz) d9866904f01904d95e62afe4552878e45f1bf51f5f92a7528ccc86743a09ff7e SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_3022.tgz) 297b929e46e5e3dc3744a063688d6a67f7162d2d509e98b91a32a03928e22fd4 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_2121.tgz) 2bec0f9f968ac72faf83870520734c0ad552e1e07a65bfd7c32a831c6d85f108 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_2120.tgz) adb3681c604a4b3f43f3b23b0f3693aaab4011ec1e96e122a5c99ff926f044a1 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_2017.tgz) ab2ba3cdf2451dcff7c7e4899a802b192697434803aa3f4b12c1a8e1bc82f4b9 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_103.tgz) 1f8a84414c6da88c42574cf4f3a4cb625577718f8a6416f7594103610e336ac2 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_102.tgz) ac1997b547e8abd41284271fee7691b8e27526d1aa2c81ddc57cbaecf1aa8ab3 SHA256 checksum (monitoring-kubernetes-metrics-and-log-forwarding_101.tgz) ce40ddfcfea3603b9ea7b5d85ba756a02c01c3a69ca19f99f5ecc0b3fbb5f387
To install your download
For instructions specific to your download, click the Details tab after closing this window.

Flag As Inappropriate

splunk

Monitoring Kubernetes - Metrics and Log Forwarding

Splunk Cloud
Overview
Details
Focus on your applications, we will take care of infrastructure monitoring and logs forwarding.

Use cases

- Application Monitoring
- Log Aggregation
- Cluster Health Monitoring
- Security and Audit
- Reduce complexity and improve productivity

With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.

An application requires Collectord built by Outcold Solutions, see https://www.outcoldsolutions.com/docs/monitoring-kubernetes/

Overview

Outcold Solutions provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses to reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications to help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer a unique solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance and cluster health.

Description

We provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.
All our solutions are powered by the Collectord, a container-native software built by Outcold Solutions that provides capabilities for discovering, transforming and forwarding logs, collecting system metrics, collecting metrics from the control plane of the orchestration frameworks and forwarding network activity. Collectord provides flexible and powerful tools for transforming logs. With our software you can hide sensitive information from the loglines before forwarding them. With Collectord you can reduce the licensing costs associated with logging aggregation by choosing which data you want to forward from the log streams. Collectord forwards container logs, host logs and can discover logs written by the containerized applications.

Use cases

Application Monitoring

See detailed metrics from containers and processes, including performance metrics, utilization metrics and security insights. Forward application-specific metrics, exported in Prometheus format. Use prebuilt Splunk dashboards for a comprehensive overview.

Log Aggregation

Aggregate logs from containers, applications, and servers. Use flexible mappings to filter logs enriched with container metadata, correlate logs with metrics, and leverage Splunk capabilities for analyzing logs. Use Collectord to transform logs before they reach Splunk, remove sensitive information, remove PII data to help keep your logs GDPR compliant. With Collectord you can reduce licensing and storage costs by choosing which loglines you want to forward.

Cluster Health Monitoring

Diagnose cluster issues by looking at historical events, monitoring allocations, and regulating cluster capacity. Leverage pre-built alerts for monitoring the health of the clusters out of the box.

Security and Audit

Define access to the data by clusters, namespaces and even pods or containers. Review network activities, happening inside your cluster, and outside connections. Verify containers running with elevated security permissions. Use audit logs for monitoring changes in deployments.

Reduce complexity and improve productivity

Use one tool to collect and forward logs and metrics required by developers for reviewing performance and health of their applications. With the annotations developers can define how they want to see the data in log aggregation tool, specify multiline log patterns, removing terminal escape codes, override types, sources and indexes.

Links

Release Notes

Version 5.23.430
Oct. 23, 2024

5.23.430 - 2024-10-28

Supports collectorforkubernetes version 5.23.x and below (see https://www.outcoldsolutions.com for latest configuration)
- To better support installations with large number of nodes and containers, default behavior for most of the dashboards
is to require pressing a Submit button after selecting filters.
- Overview Dashboard - new table with Not Ready Containers.
- Pod Dashboard - include container statuses table.
- Audit Dashboard - include user agent, and update compatibility with latest audit formats.
- Audit Dashboards - small performance improvement for the new installations.
- Host dashboard - show node conditions table.
- Host dashboard - show only external eht* interfaces in network stats.

Collectord updates:
- Implement new and improved watch mechanism for Kubernetes resources to handle large clusters.
- Change the default pipe join configuration to have max size of 1MB instead of 100KB.
- Allow...

Version 5.22.422
June 17, 2024

5.22.422 - 2024-06-17

  • Bug fix: Fix issue with calculating values on Resource Quota dashboard.

Collectord updates:

  • Upgrade SQLite to 3.46.0.
  • Upgrade golang to 1.22.4.
Version 5.22.420
April 21, 2024

5.22.420 - 2024-04-22

Supports collectorforkubernetes version 5.22.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Workload dashboard - add Pod OwnerKind and OwnerName, PriorityClass, and Pod Requests/Limits
- Address too many data points in host and workload dashboard in network graphs
- Additional CPU Metrics: CPU IOWait, Steal and Idle in Top Hosts dashboards.
- Showing CPU IOWait in Host dashboard.
- Alert Container CPU Throttled - exclude container with low CPU usage.
- New dashboard Review->Disk Stats for the host.
- Exclude virtual ethernet interfaces from host dashboard.
- Support memory limits and requests expressed in milli-bytes.

Collectord updates:
- Allow disabling IP address Lookup in net_socket_table input.
- Better handling of zombie processes in proc_stats input.
- Allow configuring user Splunk outputs using CDR SplunkOutput.
- Allow blacklisting labels from forwarded metadata.
...

Version 5.21.410
Oct. 14, 2023

5.21.410 - 2023-10-16

Supports collectorforkubernetes version 5.21.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Compatibility updates for the version 5.21 of Collectord
- New Dashboard: Review -> CPU (Throttled, Limits, Requests)
- Alert update: High amount of GRPC errors
- Alert update: Container CPU Throttled
- Network tables update: show UDP connections for Host, Workloads, Containers, and Pods
- Network Connection Dashboard: allows filtering by namespaces
- Show maximum and average number of Pods per cluster in Clusters (Allocations and usage) dashboard
- Update Resource Quota dashboard to support comparing milli-cores and cores

Collectord updates:
- Support for global replace configurations for Collectord, allowing to sanitize data before forwarding to Splunk
- Support journald as logging driver for container logs
- When both volatile and persistent journald destination exist, ...

Version 5.20.400
April 9, 2023

5.20.400 - 2023-04-17

Supports collectorforkubernetes version 5.20.x and below (see https://www.outcoldsolutions.com for latest configuration)
- Show Pod conditions on the Pod dashboard
- Bug fix: Pods dashboard filters out pods not on the host network.
- Compatibility updates for the version 5.20 of Collectord

Collectord updates:
- Multi-architecture images for amd64 and arm64
- Allow sending logs to multiple Splunk HEC endpoints simultaneously
- New annotation collectord.io/volume.{N}-logs-onvolumedatabase to keep acknowledgement information about forwarded logs on the volume
- Allow including placeholder templates in the annotation collectord.io/volume.{N}-logs-glob
- Support for new outputs (ElasticSearch and OpenSearch)
- Collectord produces diag file without performance data, if flag --include-performance-profiles is not set
- Use IMDSv2 for AWS metadata
...

Version 5.19.390
Oct. 16, 2022

5.19.390 - 2022-10-17

Supports collectorforkubernetes version 5.19.x and below (see https://www.outcoldsolutions.com for the latest configuration)
- Update dashboards for the latest changes in the metric names for API Server, Controller, and Scheduler
- Update the Kubelet dashboard to support various container runtimes
- Audit (users and namespaces) dashboard: show access to non-namespaces resources
- Logs dashboard: show container and pod as separate filters
- New alert for Collectord alarms for node diagnostics (reboot required and entropy)
- Bug fix: misprint in "Cluster Warning: container CPU is throttled" alert

Collectord updates:
- Splunk output supports maximumMessageLength to truncate messages exceeding this size
- Splunk output supports requireExplicitIndex to ignore all events that don't have explicit index defined
- Collectord monitors if node requires reboot
....

Version 5.18.380
April 19, 2022
  • Cluster filter on Events dashboard
  • Rewrite CPU throttled alert to make it less verbose
  • Memory usage now reports memory without caches and memory that can be freed.
  • Support cgroupv2

Collectord updates:
- Support cgroupv2
- New ability to specify the message field name for the logs extraction with annotations extractionMessageField
- Collectord improves grace period for expired licenses allowing to bootstrap new nodes for 14 days
- Support of journald database written with systemd library 247+
- Upgrade go runtime to 1.17.9
- Bug fix: cleanup the diag, exclude the real license key
- Bug fix: collectord reports high CPU usage for just started containers or hosts
- Bug fix: update pods/container labels when user updates them (prior restart was required)
- Bug fix: set now as a date for container logs with corrupted log files instead of 0 timestamp
- Bug fix: include the values of whitelists and blacklists in diag
...

Version 5.17.372
Jan. 27, 2022
  • Added version=1.1 to all dashboard for compatibility with Splunk Cloud and pass AppInspector
Version 5.17.370
Oct. 15, 2021

5.17.370 - 2021-10-20

  • Show milicores/cores CPU usage instead of percents
  • New dashboard: Review - Resource Quotas
  • Review - Projects: filter by project name
  • Review - Clusters: filter by node label
  • Review - Clusters: include max and avg usage
  • Bug fix: storage dashboard might not render in some Splunk versions
  • Bug fix: Namespaces dashboard shows only one namespace label
    Collectord updates:
  • Upgrade to Go 1.17.2
  • Support query in Prometheus URLs for metrics
  • Collectord now reports source and source type for the events with incorrect index
  • Support for licensing server
  • Support for CPU-based licenses
  • Allow to specify multiple values for blacklist and whitelist for host logs
  • Bug fix: Collectord clogs the output with WARN messages for stopped containers running with Containerd
    ...
Version 5.16.363
May 27, 2021

5.16.363 - 2021-05-26

Requires collectorforkubernetes version 5.16.363 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Bug fix: Put in parentheses source selection in macro_openshift_prometheus_metrics

Collectord updates:

  • Upgrade go runtime to 1.16.3
  • Bug fix: fix verbose logging for docker watcher with messages "failed to get next event"
  • Bug fix: NetworkPolicy cannot be watched, as Collectord does not convert it in plural form properly
  • Bug fix: Verify command fails on Containerd runtime
  • Bug fix: DefaultIdleConnTimeout is ignored for HTTP clients
Version 5.16.361
March 17, 2021
  • Overview dashboard filters respect filters (show only namespaces from selected cluster)
  • Bug fix: use correct units for Memory and Storage (MiB, MB, Mb)
  • Bug fix: compatibility with new format of Events from API server (FirstSeen, LastSeen, Source could be shown as null)
  • Bug fix: Collectord metrics request time shows the summary on the period, not the individual request times

Collectord updates:

  • ARM64 image
  • Allow removing managed fields from events (enabled with new configurations by default)
  • Upgrade to Go 1.16.2
  • Bug fix: precise time to Splunk HEC, sending with milliseconds instead of nanoseconds (which are incorrectly ronded by HEC)
  • Bug fix: first sample of the container can record above 100% of the CPU usage, as the values are pretty low
  • Bug fix: verify command does not respect glob patterns for Prometheus inputs (certs, tokens)
  • Bug fix: trim spaces in token value for Prometheus inputs
  • Bug fix: better recovery from corrupted journald
Version 5.16.360
March 17, 2021

5.16.360 - 2021-03-16

  • Overview dashboard filters respect filters (show only namespaces from selected cluster)
  • Bug fix: use correct units for Memory and Storage (MiB, MB, Mb)
  • Bug fix: compatibility with new format of Events from API server (FirstSeen, LastSeen, Source could be shown as null)

Collectord updates:

  • ARM64 image
  • Allow removing managed fields from events (enabled with new configurations by default)
  • Upgrade to Go 1.16.2
  • Bug fix: precise time to Splunk HEC, sending with milliseconds instead of nanoseconds (which are incorrectly ronded by HEC)
  • Bug fix: first sample of the container can record above 100% of the CPU usage, as the values are pretty low
  • Bug fix: verify command does not respect glob patterns for Prometheus inputs (certs, tokens)
  • Bug fix: trim spaces in token value for Prome
Version 5.16.352
Feb. 22, 2021
  • Bug fix: events dashboard does not filter by the namespace name
Version 5.16.351
Dec. 15, 2020
  • New dashboard: Collectord metrics
  • Compatibility for Kubernetes 1.20
  • Bug fix: broken link in Allocatable Resources dashboard

Collectord updates:

  • Annotations for collecting prometheus metrics: authorization keys and CAName for SSL certificates
  • Improvement for DNS resolutions of Splunk output FQDN
  • Export internal collectord metrics in Prometheus format
  • Forwarding internal collectord metrics to Splunk
  • For the watch objects inputs being able to hide management fields
  • In the diag include all open file descriptors
  • Upgrade go runtime to 1.14.13
  • Remove \0 symbol from the labels values in the prometheus metrics
  • Allow to filter host logs with blacklist and whitelist
  • Bug fix: less verbose warnings about not being able to load resources from API server
  • Bug fix: performance improvements for Ack DB
  • Bug fix: custom prometheus metrics forwarded by Collectord do not include cluster field or custom user fields
  • Bug fix: addon pod terminates faster
    ....
Version 5.15.300
June 1, 2020

5.15.300 - 2020-06-01

Requires collectorforkubernetes version 5.15.300 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Events dashboard: filters depend on selection of cluster and node labels
  • Improvements for supporting Kubernetes 1.14 and higher (OpenShift 4.2+)
  • Improvement for alert "Cluster Warning: high number of errors to Kubernetes API" (only alert on 5xx errors)
  • Bug fix: node events aren't visible in Events tab

Collectord updates:

  • Support for annotations to add custom user fields to data
  • Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
  • Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
  • Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
  • Bug fix: occasionally addon can report warnings about trying to delete expired keys
    ..
Version 5.14.280
Jan. 24, 2020
  • Logs dashboard: filters depend on selection
  • Overview dashboard: namespace counter for list of projects

Collectord updates:

  • Support templates in the index, source and sourcetype
  • Allow to exclude indexed fields when forwarding to Splunk
  • Support annotation for stats interval for containers
  • Support containerd runtime
  • Bug fix: verify command can show incorrect error about verifying journald input
  • Bug fix: index on namespace should set index for application logs
  • Bug fix: warning about not being able to retrieve node information
Version 5.12.272
Nov. 12, 2019
  • Improvement for the alerts verifying that control components or nodes are down.
Version 5.12.271
Nov. 7, 2019

5.12.271 - 2019-11-07

Requires collectorforkubernetes version 5.12.271 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Improvements for the macros for backward compatibility

Collectord updates:
- Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
- Bug fix: reduce warnings failed to get the new event in pipeline - submitted
- Stability improvements

Version 5.12.270
Oct. 18, 2019
  • Compact metrics (pre-calculated on Collectord side)
  • Switched stats for host and cgroup in different macros
  • Use base macro for alerts
  • Improved command extraction for exec in Audit Logs
  • Add cluster name in the alert results

Collectord updates:

  • Watch namespaces and workloads for changes
  • Global configurations with Custom Resources and selectors
  • Describe command to see applied annotations for pods
  • Bug fix: panic when pipe join configuration is removed
  • Bug fix: panic when proc stats is enabled and cgroup stats is disabled
  • Bug fix: support ProxyBasicAuthorization for license server checks
  • Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
  • Bug fix: if list of URLs is used for Splunk output, the empty URL is still required
  • Beta: dynamic index, source and sourcetype names based on the metafields
  • Beta: cluster diagnostics with one rule: node entropy
Version 5.11.260
Sept. 9, 2019

5.11.260 - 2019-09-09

Requires collectorforkubernetes version 5.11.260 or above (see https://www.outcoldsolutions.com for latest configuration)

  • GPU Monitoring (NVIDIA)

Collectord updates:

  • Support for PVC volumes for application logs
  • Bug fix: small memory leak in addon
  • Bug fix: duplicate events then pipeline is getting throttled
  • Bug fix: don't use throttling for devnull output
  • Bug fix: better recovery for ack db corruption
  • Bug fix: crash on journald input initialization when ack db is corrupted
  • Bug fix: annotations joinmultiline requires joinpartial
  • Bug fix: configurations for stdout only with annotations can crash collectord
  • Set events = 50 by default for Splunk output batches
Version 5.10.250
June 15, 2019
  • Security dashboard: Access: access to host via ssh, sudo, exec commands, failed access
  • Security dashboard: Audit (users and namespaces)
  • Security dashboard: Network (traffic)
  • Security dashboard: Network (connections)
  • Security dashboard: Objects (pods) - review pods with host network, age of pods, image pull policy, attached host paths, security context and restart policies
  • Review dashboard: Clusters (allocations and usage)
  • Cluster field filters
  • Base macro for overriding macros for other macros

Collectord updates:

  • Support for volatile and persistent journald storage with default configuration
  • Updated YAML configuration to include most common resources
  • Better support for overriding sourcetype, that does not require to update the Splunk macros
  • Bug fix: rarely when collectord fails to post to HEC it can panic
  • Bug fix: better support for Kubernetes 1.14 and CRI-O storage
  • Bug fix: space characters in index annotations can break the pipeline
Version 5.9.240
May 13, 2019

5.9.240 - 2019-05-14

Requires collectorforkubernetes version 5.9.240 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Visual improvements on the graphs for the number of logs and events
  • New alerts for the CPU and Memory reservation

Collectord updates:

  • Support for multiple Splunk destinations (outputs)
  • Support subdomains for annotations (to deploy multiple collectord instances)
  • Support for streaming objects from Kubernetes API to Splunk
  • Bug fix: journald input keeps fd open to the rotated files
  • Bug fix: fix in the annotation parser for the interval annotations
  • Bug fix: fix splunk url selection configuration for multiple splunk URLs
Version 5.8.231
April 25, 2019

5.8.231 - 2019-04-25

  • Bug fix: Collectord usage report shows trial licenses for all instances
Version 5.8.230
April 20, 2019

5.8.230 - 2019-04-22

Requires collectorforkubernetes version 5.8.230 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Use multiselect filters for most dashboards and filters with possibility to input custom filters.
  • Reduce dedup usage to improve performance on dashboards.
  • Add critical pod annotations for Kubernetes ...1.13, and priority class for Kubernetes 1.14...
  • Fix: statefulset dashboard does not show data with filters.
  • Add graph of number of pods per namespace on Overview dashboard.

Collectord updates:

  • Bug fix: clogging collectord output with errors when incorrect index is used.
  • Bug fix: short lived containers can results in duplicating logs.
  • Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
  • Bug fix: annotations cannot override timestamp location for fields extraction.
  • Bug fix: verify command reports Journald input in incorrect place.
Version 5.7.220
March 16, 2019

Requires collectorforkubernetes version 5.7.220 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
  • Workload dashboard - change CPU (of host) in table to real CPU
  • Fixed single value memory panel on host dashboard (missed span)
  • Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)

Collectord updates:

  • Support hostname formatting with environment variables in configuration
  • New rotated file logic uses less file descriptors and frees rotated files quicker
  • Allow to specify a default sampling value for container logs
  • Reimplemented shutdown sequence to stop collectord faster
  • Allow to override sampling percent with annotations
  • New Input: journald
Version 5.6.212
Feb. 19, 2019

5.6.212 - 2019-02-19

Requires collectorforkubernetes version 5.6.212 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Alert: high CPU usage on the host.
  • Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
  • New: Support Dark theme.
  • New: Free text search in Logs dashboard.
  • New: Add auto-refresh options to the dashboard.
  • Fixed: Revisited CPU limits and requests for Pods and Containers.
  • New: add CPU Max, Memory Max and Project/Namespace labels to the Review-Namespaces dashboard.
  • Fixed: Show deleted events

Read more https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.5.202
Jan. 23, 2019

5.5.202 - 2019-01-24

Requires collectorforkubernetes version 5.5.202 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Dashboard Review -> namespaces. Review allocations and requests for namespaces and pods.
  • Fixed: kubernetes_stats_cpu_request_percent - is divided by the number of CPU.

Collectord updates:
- Fixed: Interval 0 in prometheus input can crash the collectord.
- Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from
finding the files in the volume.

Version 5.4.201
Dec. 20, 2018

5.4.201 - 2018-12-19

Requires collectorforkubernetes version 5.4.201 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Fixed: Alerts for licenses issued with AWS Subscriptions

Collectord updates:
- Fixed: Better handling rotated files (less open fd)
- Fixed: Events input can hang in the err loop.

Version 5.4.200
Dec. 16, 2018

5.4 - 2018-12-17

Requires collectorforopenshift version 5.4 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Improved: etcd metrics representation for bucket values.
  • Fixed: API latency alert - exclude imagestreamimports.
  • Compatibility update for collectord 5.4.

Collectord updates:
- New: Attach EC2 metadata fields
- New: Basic Auth for Proxy (License Server and Splunk)
- Fixed: Collectord verifies reports CRI-O as unsupported runtime.
- Fixed: Rare crash on Prometheus metrics definition.
- Fixed: Better handling of acknowledgment database corruption.
- Fixed: When handling incorrect indexes, collectord can send index with an empty string, that Splunk recognize as an incorrect index

Version 5.3.190
Nov. 17, 2018

5.3 - 2018-11-19

Requires collectorforkubernetes version 5.3 or above (see https://www.outcoldsolutions.com for latest configuration)

  • Fixed: Improved Workload dashboard. Allows to filter by namespace, see all Pods in a specific namespace, filter by workload label.
  • New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
  • New: Alert for showing when Collectord reports warnings.
  • Fixed: Add node labels filter to Storage Dashboard and Control Plane Dashboards.
  • New: Alert if lag in the indexing of the data.
  • New: Splunk Usage (License usage, number of events) report under Setup.
  • Fixed: adjusted high amount of errors to Kubernetes API dashboard to make it less verbose.

https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.181
Oct. 29, 2018

5.2.180 - 2018-10-29
- Fixed: misprint in the search for showing alerts

5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Review/Storage dashboard based on storage metrics and PVC metrics.
  • New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
  • Fixed: Performance improvements
    ...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.180
Oct. 28, 2018

5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Review/Storage dashboard based on storage metrics and PVC metrics.
  • New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
  • Fixed: Performance improvements
    ...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.179
Oct. 17, 2018

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Review/Storage dashboard based on storage metrics and PVC metrics.
  • New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
  • Fixed: Performance improvements
    ...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.176
Oct. 15, 2018

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

  • New: Review/Storage dashboard based on storage metrics and PVC metrics.
  • New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
  • Fixed: Performance improvements
    ...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.1.175
Sept. 17, 2018
  • New: Network metrics (MB, Packets, Drops, ​and Errors) for host and containers.
  • New: Network socket tables (list of the port that containers and hosts are listening on, connections to external resources).
  • New: Network review dashboard to see the list of connection to public services and in private network.
  • Improvement: Replace python-based lookup with a macro written with eval.
  • Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
    ... and more

https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.0.174
Sept. 4, 2018

Highlights:
- Application logs
- Annotations for fields extraction, hiding sensitive information, time extraction, redirecting to /dev/null, stripping terminal colors and more

For more details:
https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 4.0.24
May 16, 2018
  • New dashboard: Cluster/Audit
  • New dashboard: Cluster/Kubernetes API Server
  • New dashboard: Cluster/Kubelet
  • New dashboard: Cluster/etcd
  • New dashboard: Cluster/Scheduler
  • New dashboard: Cluster/Controller Manager.
  • Include image name, when list containers.
  • Added syslog component to the list of host logs.
  • Fixed: Include Daemon Set on Overview dashboard, list of namespaces.
Version 3.0.23
Feb. 17, 2018

3.0.23 - bug fixes release

3.0.22

New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.

Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/upgrade-2-to-3/
Requires collectorforkubernetes version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)

Version 3.0.22
Feb. 8, 2018

New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.

Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/upgrade-2-to-3/
Requires collectorforkubernetes version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)

Version 2.1.21
Jan. 2, 2018

2.1.21 - 2018-01-02

Requires collectorforkubernetes version 2.1.59.171209 or above
- Updated author and description

2.1.20 - 2017-12-09

  • Fixed link to setup / installation instructions.

2.1.18 - 2017-12-09

  • Implemented collectors dashboard to track number of collectors, their versions
    and used licenses.
  • Fallback to the process IO statistics when blkio is not available.
  • Fix IO statistic graphs, showed average, when sum should be used.
  • Fields extraction support for nginx ingress 0.9 and above.
  • [collector] Improved resistance for storage failures.
  • [collector] License checks reporting.
  • [collector] Better support for openshift environment (default configuration).
Version 2.1.20
Dec. 17, 2017

2.1.20 - 2017-12-09

Requires collectorforkubernetes version 2.1.59.171209 or above

  • Fixed link to setup / installation instructions.

2.1.18 - 2017-12-09

Requires collectorforkubernetes version 2.1.59.171209 or above

  • Implemented collectors dashboard to track number of collectors, their versions
    and used licenses.
  • Fallback to the process IO statistics when blkio is not available.
  • Fix IO statistic graphs, showed average, when sum should be used.
  • Fields extraction support for nginx ingress 0.9 and above.
  • [collector] Improved resistance for storage failures.
  • [collector] License checks reporting.
  • [collector] Better support for openshift environment (default configuration).
Version 2.0.17
Oct. 24, 2017

2.0 - 2017-10-22

Requires collectorforkubernetes version 2.0.37.171023 or above

  • Better labels support in Dashboards.
    Collector has a breaking feature, replacing format for labels from
    kubernetes_node_labels_LABEL1=VALUE1 to kubernetes_node_labels=[LABEL1=VALUE1,LABEL2=VALUE2].
  • Process level metrics.
  • Uptime for hosts and processes.
  • Fields extraction for kubernetes controller manager and scheduler.
  • Fields extraction and support in dashboards for main kubernetes components (setup
    host logs collection with collector).
  • New top-like dashboards allow to monitor Hosts/Pods/Containers/Processes in realtime.
  • Rewritten Kubernetes Objects Dashboards with support of Events and Labels.
  • Improved dashboards navigation.
  • Support for host logs.
  • Other bugs and improvements based on user feedback.
Version 1.0.3
Oct. 13, 2017

Updated links to official documentation for installation instructions.

Version 1.0.2
Oct. 5, 2017

Fix labels on Kubernetes Dashboards (Most of the filters has incorrect label Daemon Sets)

Version 1.0.1
Oct. 4, 2017

Monitoring Kubernetes


Subscribe Share

Are you a developer?

As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.

Follow Us:
Splunk, Splunk>,Turn Data Into Doing, Data-to-Everything, and D2E are trademarks or registered trademarks of Splunk LLC in the United States and other countries. All other brand names,product names,or trademarks belong to their respective owners.