Keptn Metrics

The Keptn Metrics Operator provides a single entry point to all metrics in the cluster and allows you to define metrics based on multiple data platforms and multiple instances of any data platform. Metrics are fetched independently and can be used for an evaluation at workload- and application-level, or for scaling your workloads.

This data can be displayed on Grafana or another standard dashboard application that you configure or can be retrieved using standard Kubernetes commands.

For an introduction to Keptn metrics, see Getting started with Keptn metrics.

Keptn metric basics

Keptn metrics are implemented with two resources:

• KeptnMetric – define the metric to report
• KeptnMetricsProvider – define the configuration for a data provider

Define KeptnMetricsProvider resources

You must define a KeptnMetricsProvider resource for each instance of each data provider you are using.

Note the following:

• Each KeptnMetricsProvider resource is bound to a specific namespace.
• Each KeptnMetric resource must be located in the same namespace as the associated KeptnMetricsProvider resource.
• KeptnEvaluationDefinition resources can reference metrics from any namespace in the cluster.
• To define metrics that can be used in evaluations on all namespaces in the cluster, create KeptnMetricsProvider and KeptnMetric resources in a centralized namespace such as keptn-lifecycle-toolkit-system.

To configure a data provider into your Keptn cluster:

1. Create a secret if your data provider uses one. See Create secret text.
2. Install and configure each instance of each data provider into your Keptn cluster, following the instructions provided by the data source provider. See Prepare your cluster for Keptn for links. Keptn supports using multiple instances of multiple data providers.
3. Define a KeptnMetricsProvider resource for each data source.

For example, the KeptnMetricProvider resource for a Prometheus data source that does not use a secret could look like:

apiVersion: metrics.keptn.sh/v1alpha2
kind: KeptnMetricsProvider
metadata:
  name: prometheus-provider
  namespace: simplenode-dev
spec:
  type: prometheus
  targetServer: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"

The KeptnMetricProvider resource for a Dynatrace provider that uses a secret could look like:

apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetricsProvider
metadata:
  name: dynatrace-provider
  namespace: podtato-kubectl
spec:
  type: dynatrace
  targetServer: "<dynatrace-tenant-url>"
  secretKeyRef:
    name: dt-api-token
    key: DT_TOKEN

Accessing Metrics via the Kubernetes Custom Metrics API

KeptnMetrics can also be retrieved via the Kubernetes Custom Metrics API.

Retrieve KeptnMetric values with kubectl

Use the kubectl get --raw command to retrieve the values of a KeptnMetric, as in the following example:

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/podtato-kubectl/keptnmetrics.metrics.sh/keptnmetric-sample/keptnmetric-sample" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta2",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "KeptnMetric",
        "namespace": "podtato-kubectl",
        "name": "keptnmetric-sample",
        "apiVersion": "metrics.keptn.sh/v1alpha1"
      },
      "metric": {
        "name": "keptnmetric-sample",
        "selector": {
          "matchLabels": {
            "app": "frontend"
          }
        }
      },
      "timestamp": "2023-01-25T09:26:15Z",
      "value": "10"
    }
  ]
}

Filter on matching labels

You can filter based on matching labels. For example, to retrieve all metrics that are labelled with app=frontend, use the following command:

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/podtato-kubectl/keptnmetrics.metrics.sh/*/*?labelSelector=app%3Dfrontend" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta2",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "KeptnMetric",
        "namespace": "keptn-lifecycle-toolkit-system",
        "name": "keptnmetric-sample",
        "apiVersion": "metrics.keptn.sh/v1alpha3"
      },
      "metric": {
        "name": "keptnmetric-sample",
        "selector": {
          "matchLabels": {
            "app": "frontend"
          }
        }
      },
      "timestamp": "2023-01-25T09:26:15Z",
      "value": "10"
    }
  ]
}

Querying Metrics over a Timerange

You can query metrics over a specified timeframe. Let’s suppose you set the range.interval field to be 3m, the Keptn Metrics Operator would query the metrics for the last 3 minutes which means the from = currentTime - range.interval and to = currentTime.

The default value is set to be 5m if the range.interval is not set.

apiVersion: metrics.keptn.sh/v1alpha3
kind: KeptnMetric
metadata:
  name: good-metric
spec:
  provider:
    name: my-provider
  query: "sum(kube_pod_container_resource_limits{resource='cpu'})"
  fetchIntervalSeconds: 10
  range:
    interval: "3m"

Using the HorizontalPodAutoscaler

Use the Kubernetes Custom Metrics API to refer to KeptnMetric via the Kubernetes HorizontalPodAutoscaler (HPA), as in the following example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: podtato-head-entry
  namespace: podtato-kubectl
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podtato-head-entry
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Object
      object:
        metric:
          name: keptnmetric-sample
        describedObject:
          apiVersion: metrics.keptn.sh/v1alpha1
          kind: KeptnMetric
          name: keptnmetric-sample
        target:
          type: Value
          value: "10"

See the Scaling Kubernetes Workloads based on Dynatrace Metrics blog post for a detailed discussion of doing this with Dynatrace metrics. The same approach could be used to implement HPA with other data providers.