Support autoscaling the API on GKE - #95
Conversation
Adds optional Horizontal Pod Autoscaling on GKE using native AutoscalingMetric support. When turned on, the API scales on CPU, event-loop utilization, and mean event-loop delay (same signals we use on AWS ECS), and Prometheus metrics are enabled automatically. Requires GKE 1.35.1+. See https://docs.cloud.google.com/kubernetes-engine/docs/how-to/expose-custom-metrics-autoscaling
…oscaling-api-on-gke
Erik Weathers (erikdw)
left a comment
Erik Weathers (erikdw)
left a comment
There was a problem hiding this comment.
Two small follow-ups. Neither blocks merge.
| {{- end }} | ||
| {{- if .Values.api.extraEnvVars }} | ||
| {{- toYaml .Values.api.extraEnvVars | nindent 12 }} | ||
| {{- end }} |
There was a problem hiding this comment.
When autoscaling is on, the chart sets ENABLE_PROMETHEUS_METRICS=true before extraEnvVars.
Kubernetes keeps both env entries. The last entry wins.
If extraEnvVars sets ENABLE_PROMETHEUS_METRICS to "false", the gauges do not publish. The HPA then cannot read the custom metrics. Helm does not report an error.
Move this env entry after extraEnvVars, so autoscaling keeps the value "true".
There was a problem hiding this comment.
Is this behavior consistent with the rest of the helm chart? I feel like we usually merge anything "extra" in last
There was a problem hiding this comment.
For this one it’s chart/Braintrust-owned, not a user knob. If extraEnvVars sets ENABLE_PROMETHEUS_METRICS=false, they silently kill autoscaling (no Helm error). Fine for extras to win elsewhere, but here the autoscaling-owned true should come last, or we could be in a world of pain.
There was a problem hiding this comment.
Updated
| - GKE **1.35.1-gke.1396000** or later | ||
| - The Performance HPA profile and the Autoscaling API enabled on the cluster | ||
| - `roles/autoscaling.metricsWriter` granted to all node service accounts | ||
| - The Autoscaling API included in your service perimeter when using VPC Service Controls |
There was a problem hiding this comment.
Google documents this Preview feature for GKE 1.35.1-gke.1396000 or later on the Rapid release channel.
This list names the version and the Performance HPA profile. It does not name the Rapid channel.
Add Rapid to the list here and in values.yaml. Clusters on Regular or Stable will not get this feature.
There was a problem hiding this comment.
I think rapid channel was a launch constraint from when this feature initially became a preview. I was on the stable regular channel while testing this and all required functionality existed.
There was a problem hiding this comment.
Are you sure the cluster itself was on the stable channel? When I looked right now it showed Rapid in the GCP console.
There was a problem hiding this comment.
Oops yeah I mis-spoke re: stable. I tested on regular (the channel we default to in the gcp terraform module), and then switched to rapid to test against the newest version. At least in us-central1, these are the channels that meet the requirements
$ gcloud container get-server-config --flatten="channels" --format="yaml(channels.channel,channels.defaultVersion)" --location=us-central1
Fetching server config for us-central1
---
channels:
channel: EXTENDED
defaultVersion: 1.35.6-gke.1250000
---
channels:
channel: RAPID
defaultVersion: 1.36.2-gke.2064000
---
channels:
channel: REGULAR
defaultVersion: 1.35.6-gke.1250000
---
channels:
channel: STABLE
defaultVersion: 1.34.9-gke.1065000
I'm not in love with documenting a channel requirement, as a) these change over time (docs burden) and b) there's no guarantee that a cluster in a channel is at the most current version.
How about we see if helm can ensure the CRD is available and fail with a friendly error if not?
There was a problem hiding this comment.
Added an explicit check for the required CRD in 7acb818
There was a problem hiding this comment.
wow this channel stuff is so complicated! I like the solution you came up with.
| # See: https://docs.cloud.google.com/kubernetes-engine/docs/how-to/expose-custom-metrics-autoscaling | ||
| autoscaling: | ||
| enabled: false | ||
| minReplicas: 3 |
There was a problem hiding this comment.
api.replicas defaults to 4. minReplicas defaults to 3.
If you turn on autoscaling on an existing deploy and load is low, the HPA can scale the API from 4 pods to 3 pods with no load change.
Set the minReplicas default to 4 so that this does not happen.
There was a problem hiding this comment.
The autoscaling sizing is meant to match what we have for ECS: https://github.com/braintrustdata/terraform-aws-braintrust-data-plane/blob/615110c0f4279aa173c61f630fa9d6f23db0b2d8/modules/api-ecs/variables.tf#L68.
There was a problem hiding this comment.
Hmm — so we'll be inconsistent somewhere. Options:
- Drop
api.replicasdefault to3to match ECS /minReplicas, or - Keep k8s at a higher floor (
minReplicas: 4) because the pod footprint is different (sidecar/helper daemons, etc.).
I'd slightly prefer (2) + a one-line README note that enabling autoscaling may change replica count unless minReplicas is set explicitly — but matching ECS min=3 is also fine if we document the 4→3 step-down.
There was a problem hiding this comment.
Okay bumped to 4 minReplicas
2 participants
Adds optional Horizontal Pod Autoscaling on GKE using native AutoscalingMetric support. When turned on, the API scales on CPU, event-loop utilization, and mean event-loop delay (same signals we use on AWS ECS), and Prometheus metrics are enabled automatically. Requires GKE 1.35.1+.
This has a hard requirement on v2.9.0+ of the API, as this introduced the internal
/metricsendpoint capability.See https://docs.cloud.google.com/kubernetes-engine/docs/how-to/expose-custom-metrics-autoscaling for docs on the GKE preview feature that backs this.