Talos - Enable ETCD Metrics Scraping for the Kube-Prometheus-Stack

When deploying the kube-prometheus-stack on Talos Linux, you might notice that ETCD metrics are missing by default. This occurs because Talos secures ETCD using mTLS, and the default Prometheus configuration does not have the necessary certificates to authenticate against the ETCD endpoints.

Here is a quick guide on how to extract the necessary certificates and configure the monitoring stack to scrape ETCD metrics successfully.

First, we need to export the client certificates from a Talos control-plane node. These certificates are required for Prometheus to authenticate with ETCD. Run the following commands to copy the certificate authority, server certificate, and key to your local machine:

 [archy@admin42 ~]$ mkdir -p -m 700 ~/etcd  
 [archy@admin42 ~]$ MASTER_NODE=master01.talos.archyslife.lan  
 [archy@admin42 ~]$ talosctl -e ${MASTER_NODE} -n ${MASTER_NODE} copy /system/secrets/etcd/ca.crt ~/etcd  
 [archy@admin42 ~]$ talosctl -e ${MASTER_NODE} -n ${MASTER_NODE} copy /system/secrets/etcd/server.crt ~/etcd  
 [archy@admin42 ~]$ talosctl -e ${MASTER_NODE} -n ${MASTER_NODE} copy /system/secrets/etcd/server.key ~/etcd  

Next, create a generic secret in the namespace where your monitoring stack resides ('monitoring' in my case). This secret will house the certificates we just downloaded:

 [archy@admin42 ~]$ kubectl -n monitoring create secret generic etcd-client-cert \  
                        --from-file ca.crt=etcd/ca.crt \  
                        --from-file server.crt=etcd/server.crt \
                        --from-file server.key=etcd/server.key  

We now need to configure the kube-prometheus-stack to use these secrets. Update your values.yaml file to enable the ETCD service monitor and mount the certificates into the Prometheus container:

 kubeEtcd:  
  enabled: true  
  endpoints:  
   - 172.31.10.81    # IP of 1st master node  
   - 172.31.10.82    # IP of 2nd master node  
   - 172.31.10.83    # IP of 3rd master node  
  service:  
   enabled: true  
   port: 2379  
   targetPort: 2379  
  serviceMonitor:  
   scheme: https  
   insecureSkipVerify: false  
   # certs are valid for 'localhost' only  
   # so we'll have to use that one here
   serverName: localhost  
   caFile: "/etc/prometheus/secrets/etcd-client-cert/ca.crt"  
   certFile: "/etc/prometheus/secrets/etcd-client-cert/server.crt"  
   keyFile: "/etc/prometheus/secrets/etcd-client-cert/server.key"  

 prometheus:  
  prometheusSpec:  
   secrets:  
    - etcd-client-cert  

Finally, apply the changes using Helm:

 [archy@admin42 ~]$ helm -n monitoring upgrade monitoring \  
                        oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack:80.10.0 \  
                        -f values.yaml  

Once the upgrade completes, check your Prometheus or Grafana dashboard. You should now see data populating in the ETCD dashboards, confirming that the metrics are being scraped successfully from your Talos Linux control-plane nodes.

Feel free to comment and / or suggest a topic.