diff --git a/mission-control-chart b/mission-control-chart index 201b29dc..005d2aa9 160000 --- a/mission-control-chart +++ b/mission-control-chart @@ -1 +1 @@ -Subproject commit 201b29dce9be9f145bc4c3c24317c4d723ccd2ee +Subproject commit 005d2aa99683ed5c5fd6c278ef30b4271ece1507 diff --git a/mission-control/docs/guide/notifications/concepts/silences.mdx b/mission-control/docs/guide/notifications/concepts/silences.mdx index 41b1d655..0cf4e17e 100644 --- a/mission-control/docs/guide/notifications/concepts/silences.mdx +++ b/mission-control/docs/guide/notifications/concepts/silences.mdx @@ -6,30 +6,90 @@ sidebar_custom_props: import Silence from "@site/docs/reference/notifications/_silence.mdx" -A silence is a way to temporarily suppress notifications. Each silence has: +Silences temporarily suppress notifications. Each silence has: -- A description - explaining why the silence was created -- A duration - specified by `from` and `until` timestamps in RFC3339 format, datetime or datemath expressions (e.g., `now`, `now+2d`) -- A scope - defined through selectors and filters to target specific resources -- A recursive flag - to apply silence to child resources +- A description explaining why you created the silence +- A duration specified by `from` and `until` timestamps in RFC3339 format, datetime or datemath expressions (e.g., `now`, `now+2d`) +- A scope that selectors and filters define to target specific resources +- A recursive flag to apply silence to child resources :::note Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes. ::: +## Examples -```yaml title="" file=/modules/mission-control/fixtures/silences/silence-test-env.yaml -``` +### Selector-based silences + +#### Silence notifications in test and stage namespaces + +```yaml title="silence-test-env.yaml" file=/modules/mission-control/fixtures/silences/silence-test-env.yaml +``` + +This example: + +1. Uses `selectors` to target all resources in the `test` and `stage` namespaces +2. Sets a time window from January 1 to February 1, 2025 +3. Silences all notification events for resources in those namespaces + +#### Silence low-severity Kubernetes Jobs + +```yaml title="silence-test-deployments.yaml" file=/modules/mission-control/fixtures/silences/silence-test-deployments.yaml +``` + +This example: + +1. Uses `types` to target Kubernetes Job resources +2. Uses `tagSelector` to match resources with the `severity=low` tag +3. Silences notifications from matching Jobs indefinitely (no time window specified) + +### Filter-based silences + +#### Silence canary unhealthy events in a daily time window + +```yaml title="checks-date-window.yaml" file=/modules/mission-control/fixtures/silences/checks-date-window.yaml +``` + +This example: + +1. Uses a CEL `filter` expression to match canary unhealthy events +2. Uses `source_event` to check the event type is `config.unhealthy` +3. Uses `time.InTimeRange()` with `event_time` to silence events between 6:00 AM and 7:00 AM daily +4. Matches only resources of type `MissionControl::Canary` + +#### Silence notifications from PostgreSQL StatefulSet + +```yaml title="postgresql-sts.yaml" file=/modules/mission-control/fixtures/silences/postgresql-sts.yaml +``` + +This example: + +1. Uses a CEL `filter` expression to match a specific resource +2. Checks the config name is `postgresql` and the type is `Kubernetes::StatefulSet` +3. Silences all notification events for this resource + +#### Silence RDS PostgreSQL maintenance in flanksource account + +```yaml title="rds.yaml" file=/modules/mission-control/fixtures/silences/rds.yaml +``` + +This example: + +1. Uses a CEL `filter` to target AWS RDS database instances +2. Filters by the `account-name` tag matching `flanksource` +3. Filters by the config `Engine` property matching `postgres` +4. Silences notifications during maintenance windows and brief status flaps ## Use cases -- Planned maintenance or deployments - Silence notifications from a namespace or helm release and optionally all their children -- Non-critical resources - Suppress notifications from resources that routinely trigger expected and harmless alerts -- Known issues - Temporarily silence alerts for known issues that can't be immediately resolved due to dependencies or resource constraints +- **Planned maintenance or deployments** - Silence notifications from a namespace or helm release and optionally all their children +- **Non-critical resources** - Suppress notifications from resources that routinely trigger expected and harmless alerts +- **Known issues** - Temporarily silence alerts for known issues that you can't immediately resolve due to dependencies or resource constraints ## Creating Silences Silences can be created in multiple ways: + 1. Through the notification page UI 2. Using the silence button on Slack notifications (when using default templates) 3. By applying a NotificationSilence custom resource @@ -38,11 +98,11 @@ Silences can be created in multiple ways: ### Resource Selection -Silences can target resources using selectors, filters, or a specific resource. At least one targeting method must be specified. +You can target resources using selectors, filters, or a specific resource. At least one targeting method must be specified. -1. **Selectors**: Direct resource matching using types, names, namespaces, labels, and tags -2. **Filters**: Complex matching using CEL expressions -3. **Specific Resource**: Selecting a specific config, check, canary, or component (via UI) +1. **Selectors**: Match resources directly using types, names, namespaces, labels, and tags +2. **Filters**: Match resources using complex CEL expressions +3. **Specific Resource**: Select a specific config, check, canary, or component (via UI) :::info When both `filter` and `selectors` are specified, the filter is evaluated first. If the filter matches, the silence is applied. Selectors are only evaluated if the filter does not match. @@ -54,25 +114,36 @@ Selectors use [Resource Selectors](/docs/reference/resource-selector) to target #### Filters -A filter is a CEL expression that evaluates to a boolean value. The notification is silenced when the filter returns true. Filters provide powerful, flexible matching capabilities. +Filters use CEL expressions that evaluate to a boolean value. Mission Control silences the notification when the filter returns true. Filters provide powerful, flexible matching capabilities. ##### Filter Examples -| Filter | Description | -|--------|-------------| -| `check.type == 'http'` | Silences HTTP check notifications | -| `regexp.Match("^check-[0-9]+", check.name)` | Matches checks with prefix `check-` | -| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | Silences notifications from a specific StatefulSet | -| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming").size > 0 && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | Matches pods in production namespaces | +| Filter | Description | +| ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | +| `check.type == 'http'` | Silences HTTP check notifications | +| `regexp.Match("^check-[0-9]+", check.name)` | Matches checks with prefix `check-` | +| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | Silences notifications from a specific StatefulSet | +| `config.health == "unhealthy" && config.name.endsWith("-canary") && time.Since(timestamp(config.updated_at)) < duration("15m")` | Silences recent `config.unhealthy` canary updates | +| `source_event == 'config.unhealthy' && config.type == 'MissionControl::Canary' && time.InTimeRange(event_time, "06:00", "07:00")` | Silences canary unhealthy events during a daily time window | +| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | Matches pods in production namespaces | ##### Available Template Variables -Filters can reference these variables: +In addition to the resource-specific variables below, the following common variables are available in silence filters: + +| Variable | Description | Schema | +| -------------- | ------------------------------------------------------------------------------------------------ | ----------- | +| `source_event` | The event that triggered the notification (e.g., `config.unhealthy`, `check.passed`) | `string` | +| `event_time` | The timestamp when the event was created | `time.Time` | + +Resource-specific variables: + - [CheckEvents](/docs/reference/notifications/template_vars/checks) - [ConfigEvents](/docs/reference/notifications/template_vars/config) ### Recursive Mode When `recursive: true` is set, the silence applies to all child resources of the matched resources. For example: + - Silencing a namespace affects all deployments, statefulsets, pods, etc. within it - Silencing a statefulset affects all its pods diff --git a/mission-control/docs/reference/notifications/_env_vars.mdx b/mission-control/docs/reference/notifications/_env_vars.mdx index c9bad802..3c837c34 100644 --- a/mission-control/docs/reference/notifications/_env_vars.mdx +++ b/mission-control/docs/reference/notifications/_env_vars.mdx @@ -36,6 +36,16 @@ export function CheckHealthEnv() { "scheme": "string", "description": "The notification channel, e.g. `slack`, `email`" }, + { + "field": "source_event", + "scheme": "string", + "description": "The event that triggered the notification (e.g. `check.passed`, `check.failed`)" + }, + { + "field": "event_time", + "scheme": "`time.Time`", + "description": "The timestamp when the event was created" + }, { "field": "groupedResources", "scheme": "`[]string`", @@ -83,6 +93,16 @@ export function ConfigEventsTemplateVars() { "scheme": "string", "description": "The notification channel, e.g. `slack`, `email`" }, + { + "field": "source_event", + "scheme": "string", + "description": "The event that triggered the notification (e.g. `config.unhealthy`, `config.created`, `config.updated`, `config.deleted`)" + }, + { + "field": "event_time", + "scheme": "`time.Time`", + "description": "The timestamp when the event was created" + }, { "field": "groupedResources", "scheme": "`[]string`", diff --git a/modules/canary-checker b/modules/canary-checker index e60eb7d9..4daf3f63 160000 --- a/modules/canary-checker +++ b/modules/canary-checker @@ -1 +1 @@ -Subproject commit e60eb7d9a6b9fd1fa3f4736df3366a05be0100e9 +Subproject commit 4daf3f63deca68e735381464b1ead61358793c82 diff --git a/modules/config-db b/modules/config-db index f54265d6..e358b38f 160000 --- a/modules/config-db +++ b/modules/config-db @@ -1 +1 @@ -Subproject commit f54265d6d72d94b1d8f7c5ae4391dfb2d4e45c87 +Subproject commit e358b38f7f33ef0f83fdb7cbd3c1c8f4166c8417 diff --git a/modules/duty b/modules/duty index 69c8f7a5..db23703c 160000 --- a/modules/duty +++ b/modules/duty @@ -1 +1 @@ -Subproject commit 69c8f7a5aa1545d3959c2182412da88d2be8dd63 +Subproject commit db23703c038b01dab6f0b6e6aaeda4dbf6263302 diff --git a/modules/mission-control b/modules/mission-control index abe692d1..ef38fe35 160000 --- a/modules/mission-control +++ b/modules/mission-control @@ -1 +1 @@ -Subproject commit abe692d1922ef06e069dbbaa833cebbceaaff4e1 +Subproject commit ef38fe354de5c2e4ecaae7187f0488fdc42ce93f diff --git a/modules/mission-control-chart b/modules/mission-control-chart index 5c0862ce..005d2aa9 160000 --- a/modules/mission-control-chart +++ b/modules/mission-control-chart @@ -1 +1 @@ -Subproject commit 5c0862ce262114d9f3dd8900407ff8df643fcc27 +Subproject commit 005d2aa99683ed5c5fd6c278ef30b4271ece1507