[release-5.0] OCPBUGS-114671: fall back to intermediate ciphers during etcd bootstrap - #1690
Conversation
During bootstrap the EnvVarController reads servingInfo.cipherSuites from the operator's observedConfig, but the config observer cannot populate it until the control plane is running - and etcd cannot start until this controller produces ETCD_CIPHER_SUITES. Until now getCipherSuites() hard-errored on the empty observedConfig, setting EnvVarControllerDegraded and blocking the first etcd revision (StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0). On slower platforms (e.g. the gcd sovereign-cloud region) this convergence race can exceed the installer's 45-minute bootstrap window, causing intermittent install failures. Break the chicken-and-egg deadlock by falling back, only at bootstrap (LatestAvailableRevision == 0), to the same TLSProfileIntermediateType ciphers the render path bakes into the bootstrap etcd member (pkg/cmd/render/env.go:getTLSCipherSuites). The fallback is cryptographically identical to the cipher set the bootstrap etcd is already running, and is self-correcting: once observedConfig converges the observed ciphers take over. Past bootstrap (revision > 0) an empty observedConfig remains a genuine convergence failure and still surfaces as degraded. The previously merged mitigation (openshift#1659, adding the APIServer informer to the EnvVarController cache sync) only waits for the informer cache to sync, not for the config observer to populate observedConfig, so the race persisted on slow environments; this change addresses the root cause. Assisted-By: Claude Opus 4.8 Signed-off-by: Mat Kowalski <mko@redhat.com>
Address two review findings on the bootstrap cipher-suites fallback:
1. Restore the original getCipherSuites error string ("no supported
cipherSuites not found in observedConfig"). The earlier reword broke the
EtcdBootstrapRev0CipherSuitesOCPBUGS94106 CI symptom matcher and was
unrelated to the fix.
2. Replace the imprecise "LatestAvailableRevision == 0" bootstrap check with
bootstrap.IsBootstrapComplete (the kube-system/bootstrap configmap), the
same idiom used by the etcdcertsigner/bootstrapteardown/etcdendpoints
controllers. LatestAvailableRevision can read 0 outside a fresh install
(Etcd CR delete/recreate, DR restore, status reset); gating on it risked
silently downgrading an established cluster's configured ciphers to the
intermediate profile. The configmap signal only allows the fallback while
bootstrap is genuinely in progress, and covers the whole bootstrap phase
rather than just revision 0.
Wire a kube-system configmap lister/informer into the EnvVarController for the
check. Tests: drive TestGetCipherSuites via the bootstrap configmap state and
assert the restored error string; add a TestEnvVarController bootstrap scenario
proving sync() does not degrade when observedConfig is empty during bootstrap.
Assisted-By: Claude Opus 4.8
Signed-off-by: Mat Kowalski <mko@redhat.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
@openshift-cherrypick-robot: Detected clone of Jira Issue OCPBUGS-94106 with correct target version. Will retitle the PR to link to the clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
openshift-ci
Bot
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-114671, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/label backport-risk-assessed |
openshift-ci
Bot
added
the
backport-risk-assessed
|
Scheduling required tests: |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tjungblu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
4 participants
This is an automated cherry-pick of #1689
/assign openshift-ci-robot