Improve heartbeat workload name/kind detection (pod-sandbox labels + configurable label keys)

[artursarlo]

Improve heartbeat workload name/kind detection (pod-sandbox labels + configurable label keys)

Summary

The heartbeat workload inventory previously derived each container's workload_name /
workload_kind from a small set of container-level labels plus a brittle pod-name
regex. Investigation across three staging clusters showed this is unreliable:

• On EKS-style nodes the container-runtime (CRI) labels expose only the four
  io.kubernetes.* keys — the pod's real identity labels (app.kubernetes.io/name,
  app, vendor CRD labels, …) live on the pod sandbox, not on the container.
• The workload-name label keys were company-specific and hardcoded, which is not
  acceptable for an open-source agent.
• The pod-name regex did not cover DaemonSets and used an incorrect hex character class
  for the ReplicaSet hash.

This PR makes workload detection accurate and cluster-agnostic by (1) consuming
pod-sandbox labels (added in the companion granulate-utils PR), (2) improving the
label priority list and pod-name normalization, and (3) making the vendor-specific label
keys configurable via CLI instead of hardcoded.

Companion change: granulate-utils — Expose pod-sandbox labels on Container.
This PR depends on the new Container.pod_labels field it introduces.

Motivation

Empirical findings from live nodes (crictl):

Cluster	Container labels	Sandbox labels	Workload identity source
containerd (vendor-propagated)	io.kubernetes.* + vendor CRD labels	rich	container or sandbox
EKS	only io.kubernetes.*	app, app.kubernetes.io/name, k8s-app, vendor CRD, …	sandbox only
GPU/ML EKS	only io.kubernetes.*	same as EKS (k8s-app-only pods like kube-proxy)	sandbox only

Conclusion: there is no single container-label convention across clusters. The pod
sandbox is the reliable source, and the exact label keys carrying the workload name/kind
vary per deployment, so they must be configurable.

Changes

gprofiler/metadata/heartbeat_metadata.py

• Vendor-neutral defaults. Replaced the hardcoded label constants with:
  • DEFAULT_WORKLOAD_NAME_LABELS = app.kubernetes.io/name, app.kubernetes.io/instance,
    app, k8s-app (standard Kubernetes labels only).
  • DEFAULT_WORKLOAD_KIND_LABELS = empty (kind is inferred from the pod-name shape by
    default).
• Pod-sandbox labels are preferred. _best_effort_workload_name /
  _best_effort_workload_kind now probe pod_labels (sandbox) first, then container
  labels, then fall back to pod-name normalization. Placeholder values
  (unknown / none / empty) are skipped.
• New _best_effort_workload_kind mirrors the name function: label lookup →
  pod-name-shape inference (Deployment / StatefulSet / DaemonSet) →
  generic k8s / container fallback. Shared label-probing logic factored into
  _first_label_value.
• Better pod-name regexes. Fixed the ReplicaSet hash class (k8s uses a vowel-free
  "safe" alphabet, not hex), added a DaemonSet/generateName single-suffix pattern
  (previously unhandled), and constrained the random suffix to the safe alphabet so real
  name tails (e.g. service-redis) are not stripped.
• Configurable keys. HeartbeatMetadataCollector.__init__ accepts
  workload_name_labels / workload_kind_labels; configured keys are probed before
  the built-in defaults, so custom keys take priority without losing standard k8s coverage.

gprofiler/main.py

Two new CLI arguments in the heartbeat argument group (settable via config file / env var
through configargparse):

Flag	dest	Default
--heartbeat-workload-name-labels	heartbeat_workload_name_labels	[]
--heartbeat-workload-kind-labels	heartbeat_workload_kind_labels	[]

Both accept a comma-separated list of label keys (priority order) and are threaded into
HeartbeatClient.

gprofiler/dynamic_profiling_management/heartbeat.py

HeartbeatClient accepts workload_name_labels / workload_kind_labels and forwards them
to HeartbeatMetadataCollector.

gprofiler/gprofiler_types.py

Added comma_separated_list argparse type helper (splits/strips a comma-separated string).

Usage

Deployments that expose workload identity under vendor keys can opt in, e.g.:

--heartbeat-workload-name-labels mycompany.com/workload-name
--heartbeat-workload-kind-labels mycompany.com/workload-kind

With no configuration, the agent uses the standard Kubernetes labels and pod-name
inference — no company-specific behavior is baked in.

Backwards compatibility

• workload_kind is now richer than before (Deployment / StatefulSet / DaemonSet /
  configured kind, instead of only k8s). Consumers that keyed off the literal "k8s"
  should accept these values. container is still emitted for non-k8s containers.
• No hardcoded vendor labels remain in the codebase.

Testing

tests_fast/test_workload_inventory_spec.py extended to cover: sandbox-vs-container label
precedence, configured-vs-unconfigured vendor keys (name and kind), placeholder-value
skipping, DaemonSet/ReplicaSet/StatefulSet inference, and the vowel-tail guard.

.venv/bin/python -m pytest tests_fast/ -q
50 passed

black, isort, flake8, and mypy pass on the changed files.