Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion mission-control-chart
Submodule mission-control-chart updated 30 files
+68 −0 .github/workflows/bump-chart-deps.yml
+22 −18 .github/workflows/release.yml
+1 −1 Makefile
+3 −3 agent-chart/Chart.yaml
+7 −4 agent-chart/README.md
+1 −1 agent-chart/values.schema.json
+1 −1 agent-chart/values.yaml
+8 −5 chart/Chart.lock
+7 −3 chart/Chart.yaml
+247 −211 chart/README.md
+33 −7 chart/crds/mission-control.flanksource.com_applications.yaml
+877 −30 chart/crds/mission-control.flanksource.com_connections.yaml
+304 −2 chart/crds/mission-control.flanksource.com_notifications.yaml
+677 −23 chart/crds/mission-control.flanksource.com_playbooks.yaml
+300 −0 chart/crds/mission-control.flanksource.com_views.yaml
+0 −72 chart/templates/views/dashboard/recent-changes.yaml
+51 −0 chart/templates/views/system/database.yaml
+2 −2 chart/templates/views/system/job_history.yaml
+7 −0 chart/templates/views/system/system.yaml
+26 −8 chart/values.schema.json
+18 −1 chart/values.yaml
+72 −3 crd-chart/templates/canary-checker.flanksource.com_Canary.yaml
+43 −0 crd-chart/templates/canary-checker.flanksource.com_Topology.yaml
+2,865 −113 crd-chart/templates/configs.flanksource.com_scrapeconfigs.yaml
+120 −5 crd-chart/templates/configs.flanksource.com_scrapeplugins.yaml
+33 −7 crd-chart/templates/mission-control.flanksource.com_applications.yaml
+877 −30 crd-chart/templates/mission-control.flanksource.com_connections.yaml
+304 −2 crd-chart/templates/mission-control.flanksource.com_notifications.yaml
+677 −23 crd-chart/templates/mission-control.flanksource.com_playbooks.yaml
+300 −0 crd-chart/templates/mission-control.flanksource.com_views.yaml
115 changes: 93 additions & 22 deletions mission-control/docs/guide/notifications/concepts/silences.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,90 @@ sidebar_custom_props:

import Silence from "@site/docs/reference/notifications/_silence.mdx"

A silence is a way to temporarily suppress notifications. Each silence has:
Silences temporarily suppress notifications. Each silence has:

- A description - explaining why the silence was created
- A duration - specified by `from` and `until` timestamps in RFC3339 format, datetime or datemath expressions (e.g., `now`, `now+2d`)
- A scope - defined through selectors and filters to target specific resources
- A recursive flag - to apply silence to child resources
- A description explaining why you created the silence
- A duration specified by `from` and `until` timestamps in RFC3339 format, datetime or datemath expressions (e.g., `now`, `now+2d`)
- A scope that selectors and filters define to target specific resources
- A recursive flag to apply silence to child resources

:::note
Notifications that aren't sent due to silence are still visible in the notification history for auditing purposes.
:::

## Examples

```yaml title="" file=<rootDir>/modules/mission-control/fixtures/silences/silence-test-env.yaml
```
### Selector-based silences

#### Silence notifications in test and stage namespaces

```yaml title="silence-test-env.yaml" file=<rootDir>/modules/mission-control/fixtures/silences/silence-test-env.yaml
```

This example:

1. Uses `selectors` to target all resources in the `test` and `stage` namespaces
2. Sets a time window from January 1 to February 1, 2025
3. Silences all notification events for resources in those namespaces

#### Silence low-severity Kubernetes Jobs

```yaml title="silence-test-deployments.yaml" file=<rootDir>/modules/mission-control/fixtures/silences/silence-test-deployments.yaml
```

This example:

1. Uses `types` to target Kubernetes Job resources
2. Uses `tagSelector` to match resources with the `severity=low` tag
3. Silences notifications from matching Jobs indefinitely (no time window specified)

### Filter-based silences

#### Silence canary unhealthy events in a daily time window

```yaml title="checks-date-window.yaml" file=<rootDir>/modules/mission-control/fixtures/silences/checks-date-window.yaml
```

This example:

1. Uses a CEL `filter` expression to match canary unhealthy events
2. Uses `source_event` to check the event type is `config.unhealthy`
3. Uses `time.InTimeRange()` with `event_time` to silence events between 6:00 AM and 7:00 AM daily
4. Matches only resources of type `MissionControl::Canary`

#### Silence notifications from PostgreSQL StatefulSet

```yaml title="postgresql-sts.yaml" file=<rootDir>/modules/mission-control/fixtures/silences/postgresql-sts.yaml
```

This example:

1. Uses a CEL `filter` expression to match a specific resource
2. Checks the config name is `postgresql` and the type is `Kubernetes::StatefulSet`
3. Silences all notification events for this resource

#### Silence RDS PostgreSQL maintenance in flanksource account

```yaml title="rds.yaml" file=<rootDir>/modules/mission-control/fixtures/silences/rds.yaml
```

This example:

1. Uses a CEL `filter` to target AWS RDS database instances
2. Filters by the `account-name` tag matching `flanksource`
3. Filters by the config `Engine` property matching `postgres`
4. Silences notifications during maintenance windows and brief status flaps

## Use cases

- Planned maintenance or deployments - Silence notifications from a namespace or helm release and optionally all their children
- Non-critical resources - Suppress notifications from resources that routinely trigger expected and harmless alerts
- Known issues - Temporarily silence alerts for known issues that can't be immediately resolved due to dependencies or resource constraints
- **Planned maintenance or deployments** - Silence notifications from a namespace or helm release and optionally all their children
- **Non-critical resources** - Suppress notifications from resources that routinely trigger expected and harmless alerts
- **Known issues** - Temporarily silence alerts for known issues that you can't immediately resolve due to dependencies or resource constraints

## Creating Silences

Silences can be created in multiple ways:

1. Through the notification page UI
2. Using the silence button on Slack notifications (when using default templates)
3. By applying a NotificationSilence custom resource
Expand All @@ -38,11 +98,11 @@ Silences can be created in multiple ways:

### Resource Selection

Silences can target resources using selectors, filters, or a specific resource. At least one targeting method must be specified.
You can target resources using selectors, filters, or a specific resource. At least one targeting method must be specified.

1. **Selectors**: Direct resource matching using types, names, namespaces, labels, and tags
2. **Filters**: Complex matching using CEL expressions
3. **Specific Resource**: Selecting a specific config, check, canary, or component (via UI)
1. **Selectors**: Match resources directly using types, names, namespaces, labels, and tags
2. **Filters**: Match resources using complex CEL expressions
3. **Specific Resource**: Select a specific config, check, canary, or component (via UI)

:::info
When both `filter` and `selectors` are specified, the filter is evaluated first. If the filter matches, the silence is applied. Selectors are only evaluated if the filter does not match.
Expand All @@ -54,25 +114,36 @@ Selectors use [Resource Selectors](/docs/reference/resource-selector) to target

#### Filters

A filter is a CEL expression that evaluates to a boolean value. The notification is silenced when the filter returns true. Filters provide powerful, flexible matching capabilities.
Filters use CEL expressions that evaluate to a boolean value. Mission Control silences the notification when the filter returns true. Filters provide powerful, flexible matching capabilities.

##### Filter Examples

| Filter | Description |
|--------|-------------|
| `check.type == 'http'` | Silences HTTP check notifications |
| `regexp.Match("^check-[0-9]+", check.name)` | Matches checks with prefix `check-` |
| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | Silences notifications from a specific StatefulSet |
| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming").size > 0 && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | Matches pods in production namespaces |
| Filter | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| `check.type == 'http'` | Silences HTTP check notifications |
| `regexp.Match("^check-[0-9]+", check.name)` | Matches checks with prefix `check-` |
| `config.name == "postgresql" && config.type == "Kubernetes::StatefulSet"` | Silences notifications from a specific StatefulSet |
| `config.health == "unhealthy" && config.name.endsWith("-canary") && time.Since(timestamp(config.updated_at)) < duration("15m")` | Silences recent `config.unhealthy` canary updates |
| `source_event == 'config.unhealthy' && config.type == 'MissionControl::Canary' && time.InTimeRange(event_time, "06:00", "07:00")` | Silences canary unhealthy events during a daily time window |
| `config.type == "Kubernetes::Pod" && catalog.traverse(config.id, "Kubernetes::Namespace", "incoming")[0].tags.?env.orValue("") == "prod"` | Matches pods in production namespaces |

##### Available Template Variables

Filters can reference these variables:
In addition to the resource-specific variables below, the following common variables are available in silence filters:

| Variable | Description | Schema |
| -------------- | ------------------------------------------------------------------------------------------------ | ----------- |
| `source_event` | The event that triggered the notification (e.g., `config.unhealthy`, `check.passed`) | `string` |
| `event_time` | The timestamp when the event was created | `time.Time` |

Resource-specific variables:

- [CheckEvents](/docs/reference/notifications/template_vars/checks)
- [ConfigEvents](/docs/reference/notifications/template_vars/config)

### Recursive Mode

When `recursive: true` is set, the silence applies to all child resources of the matched resources. For example:

- Silencing a namespace affects all deployments, statefulsets, pods, etc. within it
- Silencing a statefulset affects all its pods
20 changes: 20 additions & 0 deletions mission-control/docs/reference/notifications/_env_vars.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,16 @@ export function CheckHealthEnv() {
"scheme": "string",
"description": "The notification channel, e.g. `slack`, `email`"
},
{
"field": "source_event",
"scheme": "string",
"description": "The event that triggered the notification (e.g. `check.passed`, `check.failed`)"
},
{
"field": "event_time",
"scheme": "`time.Time`",
"description": "The timestamp when the event was created"
},
{
"field": "groupedResources",
"scheme": "`[]string`",
Expand Down Expand Up @@ -83,6 +93,16 @@ export function ConfigEventsTemplateVars() {
"scheme": "string",
"description": "The notification channel, e.g. `slack`, `email`"
},
{
"field": "source_event",
"scheme": "string",
"description": "The event that triggered the notification (e.g. `config.unhealthy`, `config.created`, `config.updated`, `config.deleted`)"
},
{
"field": "event_time",
"scheme": "`time.Time`",
"description": "The timestamp when the event was created"
},
{
"field": "groupedResources",
"scheme": "`[]string`",
Expand Down
2 changes: 1 addition & 1 deletion modules/config-db
Submodule config-db updated 174 files
2 changes: 1 addition & 1 deletion modules/duty
Submodule duty updated 118 files
2 changes: 1 addition & 1 deletion modules/mission-control
Submodule mission-control updated 93 files
+9 −1 .gitignore
+1 −1 Dockerfile
+2 −2 Makefile
+592 −0 api/application-data.yaml
+45 −2 api/application.go
+330 −0 api/application_pretty.go
+187 −0 api/application_pretty_test.go
+48 −0 api/openapi.yaml
+223 −0 api/v1/connection_smtp.go
+53 −247 api/v1/connection_types.go
+4 −1 api/v1/notification_types.go
+3 −17 api/v1/playbook_actions.go
+9 −0 api/v1/playbook_types.go
+81 −23 api/v1/zz_generated.deepcopy.go
+65 −3 application/application.go
+44 −15 application/controller.go
+112 −2 application/job_test.go
+124 −0 application/print.go
+161 −0 application/render_facet.go
+60 −0 application/render_facet_test.go
+34 −0 application/testdata/backups-view.yaml
+61 −0 application/testdata/incident-commander.yaml
+94 −0 cmd/application.go
+53 −4 cmd/catalog.go
+465 −0 cmd/connection_crd.go
+319 −0 cmd/connection_crd_test.go
+82 −6 cmd/connections.go
+1 −1 cmd/root.go
+877 −30 config/crds/mission-control.flanksource.com_connections.yaml
+304 −2 config/crds/mission-control.flanksource.com_notifications.yaml
+622 −28 config/crds/mission-control.flanksource.com_playbooks.yaml
+300 −0 config/crds/mission-control.flanksource.com_views.yaml
+124 −0 config/schemas/connection.schema.json
+38 −2 config/schemas/notification.schema.json
+69 −16 config/schemas/playbook-spec.schema.json
+111 −17 config/schemas/playbook.schema.json
+39 −0 config/schemas/view.schema.json
+43 −17 connection/check.go
+185 −0 db/applications.go
+42 −0 db/applications_test.go
+70 −9 db/connections.go
+2 −1 db/notifications.go
+52 −0 fixtures/applications/azdo-pipeline-app.yaml
+56 −0 fixtures/applications/incident-commander.yaml
+58 −0 fixtures/applications/kubernetes-app.yaml
+66 −0 fixtures/applications/mssql-registry.yaml
+58 −0 fixtures/applications/pop-api.yaml
+103 −0 fixtures/playbooks/ams-admin-access-grant.yaml
+13 −0 fixtures/silences/checks-date-window.yaml
+50 −58 go.mod
+92 −117 go.sum
+8 −3 notification/cel.go
+12 −1 notification/controllers.go
+4 −1 notification/events.go
+13 −0 notification/send.go
+1 −0 notification/silence.go
+35 −11 playbook/actions/actions.go
+70 −0 playbook/actions/actions_test.go
+19 −48 playbook/actions/http.go
+19 −0 playbook/actions/logs.go
+0 −1 playbook/actions/notification.go
+37 −0 playbook/playbook_test.go
+21 −0 playbook/runner/exec.go
+27 −15 playbook/runner/runner.go
+82 −0 playbook/runner/scrub_test.go
+7 −7 playbook/runner/template.go
+14 −0 playbook/testdata/action-secret-params.yaml
+2 −2 playbook/testdata/connections/httpbin.yaml
+48 −29 rbac/adapter/permission.go
+4 −0 report/.gitignore
+154 −0 report/Application.tsx
+54 −0 report/components/AccessControlSection.tsx
+35 −0 report/components/ApplicationDetails.tsx
+130 −0 report/components/BackupsSection.tsx
+150 −0 report/components/DynamicSection.tsx
+100 −0 report/components/FindingsSection.tsx
+54 −0 report/components/IncidentsSection.tsx
+47 −0 report/components/LocationsSection.tsx
+80 −0 report/components/utils.ts
+7 −0 report/embed.go
+309 −0 report/fixtures/mission-control.yaml
+11 −0 report/mission-control.ts
+3,090 −0 report/package-lock.json
+21 −0 report/package.json
+2,331 −0 report/test1.css
+2,340 −0 report/test1.html
+ report/test1.pdf
+21 −0 report/tsconfig.json
+175 −0 report/types.ts
+1 −1 tests/e2e/playbooks_test.go
+6 −2 tests/e2e/suite_test.go
+59 −0 tests/e2e/views_endpoint_test.go
+20 −0 utils/map.go