fix(aws_elb_metrics_otel): filter TargetGroup/AvailabilityZone rollup…

[mrg-elastic]

… rows to prevent metric overcounting

Proposed commit message

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[agithomas]

nit:
Try to include AND stat == "Sum" or as to reduce the documents being scanned.

Example: Single metric panel, Detailed dashboard

Diff (original → suggested):
        --- original
        +++ suggested
        @@ -1,7 +1,7 @@
         SET unmapped_fields = "nullify";
         FROM metrics-aws.elb.otel-*
        -| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "app/")
        +| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "app/") AND stat == "Sum"
        -| STATS last_seen = MAX(@timestamp) BY LoadBalancer
        +| STATS last_seen = MAX(@timestamp) BY LoadBalancer, cloud.region
         | INLINE STATS latest_global = MAX(last_seen)
         | WHERE DATE_DIFF("minutes", last_seen, latest_global) <= 5
         | STATS count = COUNT(*), names = VALUES(LoadBalancer)

[agithomas]

Detailed dashboard / Account ID:

        -| STATS account = MAX(cloud.account.id)
        +| STATS account = VALUES(cloud.account.id)

[agithomas]

Please increase the value precision instead of using human-friendly values.

[agithomas]

The backend 5xx error count is 0, but the graph next to it shows the values.

Check if the values of Healthy hosts and unhealthy hosts (4 panels are loading). In the Discover view, the value appears, but not in the kibana dashboard. Please recheck

Check possible double-counting issue with Active connections

SET unmapped_fields = "nullify";
        -TS metrics-aws.elb.otel-* | WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "app/") | WHERE stat == "Sum" AND `metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount` IS NOT NULL | STATS col_activeconn = SUM(sum_over_time(`metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount`)) BY timestamp = TBUCKET(50, ?_tstart, ?_tend), LoadBalancer, cloud.region | SORT timestamp ASC
        +TS metrics-aws.elb.otel-* | WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "app/") AND TargetGroup IS NULL AND AvailabilityZone IS NULL | WHERE `metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount` IS NOT NULL | STATS col_activeconn = SUM(sum_over_time(CASE(stat == "Sum", `metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount`, NULL))) BY timestamp = TBUCKET(50, ?_tstart, ?_tend), LoadBalancer, cloud.region | SORT timestamp ASC, col_activeconn DESC | LIMIT 10 BY timestamp

[agithomas]

Panel title: Active connections over time (sum)

Qn? Do you want to consider removing the suffix (sum) from the title?

Do you want to consider

-TS metrics-aws.elb.otel-* | WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND TargetGroup IS NULL AND AvailabilityZone IS NULL | WHERE stat == "Sum" | STATS active_conn = SUM(sum_over_time(`metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount`)) BY ts = TBUCKET(50, ?_tstart, ?_tend) | SORT ts ASC
 
+TS metrics-aws.elb.otel-* | WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND TargetGroup IS NULL AND AvailabilityZone IS NULL | WHERE stat == "Average" | STATS active_conn = SUM(avg_over_time(`metrics.amazonaws.com/AWS/ApplicationELB/ActiveConnectionCount`)) BY ts = TBUCKET(50, ?_tstart, ?_tend) | SORT ts ASC

[agithomas]

Panel name : Primary metric: Browse other AWS OTel dashboards

--- original
+++ suggested
@@ -1,2 +1 @@
-SET unmapped_fields = "nullify";
 ROW value = "AWS-Managed-OTel" | KEEP value

[agithomas]

Every NLB detail panel queries the ALB stream metrics-aws.elb.otel-. You may have to retarget them to metrics-aws.elb_network.otel- and the matching dataset. Please check for the gateway load balancer.

This has an impact on the Overview dashboards, as these are different datasets - the data shown in the Overview dashboard (count, table view, hexmap) presently is ALB specific, might not be covering other LB types. So, you may have to revisit the Overview dashboard as well.

[agithomas]

ELB - NLB Details dashboard

1. Load balancer (single metric) / Account / Region

Please add the stat == "Sum"

  --- original
  +++ suggested
   FROM metrics-aws.elb_network.otel-*
  -| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "net/")
  +| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "net/") AND stat == "Sum"

-| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "net/")
+| WHERE LoadBalancer IS NOT NULL AND LoadBalancer != "" AND STARTS_WITH(LoadBalancer, "net/") AND stat == "Sum"

4. Status timeline:

-| STATS unhealthy_max = MAX(max_over_time(CASE(stat == "Maximum", `metrics.amazonaws.com/AWS/NetworkELB/UnHealthyHostCount`, NULL)))
+| STATS unhealthy_max = MAX(max_over_time(CASE(stat == "Maximum" AND TargetGroup IS NOT NULL, `metrics.amazonaws.com/AWS/NetworkELB/UnHealthyHostCount`, NULL)))

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 0074c0d

History

• 💚 Build #48640 succeeded eddd6af
• 💚 Build #48489 succeeded 16be14c
• 💚 Build #48446 succeeded c88e1f3
• 💚 Build #48423 succeeded be29aac