feat(connectivity_check): add automated monitoring with CloudWatch metrics and alarms

[sai-praveen-os]

JIRA: PLATEXP-11106

Add Monitoring Features to connectivity_check Module

Problem

The connectivity_check module currently supports on-demand connectivity testing via script invocation. We need automated scheduled monitoring with CloudWatch metrics and alarms for proactive alerting.

Solution

Extend the module with optional monitoring capabilities while maintaining backward compatibility.

Changes

monitoring.tf (new):

• EventBridge rule for scheduled Lambda invocation (configurable, default: 5 min)
• CloudWatch alarms for endpoint failures (per-critical-endpoint + aggregate)
• CloudWatch alarm for Lambda execution errors
• IAM permissions for CloudWatch metrics and EventBridge

lambda/handler.ts:

• Publish CloudWatch metrics: EndpointConnectivity (1=up, 0=down), EndpointLatency (ms)
• Metrics include dimensions: FunctionName, Endpoint, Critical

lambda/package.json:

• Added @aws-sdk/client-cloudwatch dependency

variables.tf:

• Added monitoring configuration variables: enable_monitoring, monitoring_schedule, monitoring_targets, alarm_sns_topic_arns, alarm_evaluation_periods

Usage Example

module "connectivity_monitor" {
  source = "path/to/connectivity_check"
  
  enable_monitoring    = true
  monitoring_schedule  = "rate(5 minutes)"
  cloudwatch_namespace = "myapp/connectivity/dev"
  
  monitoring_targets = [
    { host = "api.example.com", port = 443, protocol = "https", critical = true }
  ]
  
  alarm_sns_topic_arns = [aws_sns_topic.alerts.arn]
}

Backward Compatibility

All monitoring features are optional. Existing module usage continues to work unchanged.

[smayberry]

Since critical is included in target, and testTcp() and testHttp() already copy fields from target into their result, they could also include critical in their results and avoid having to add it after the fact.

[kevin-secrist]

More granularity might be a good idea. As configured it'll take 10 minutes for the alarm to fire. Running it every minute and requiring 3 or 4 evaluation periods would invoke an alarm much sooner and have a smaller false positive rate.

[kevin-secrist]

Suggested change

	default = "janus/connectivity"
	default = "connectivity"

Or make the default empty string and it can be the switch to turn metrics on/off

[kevin-secrist]

Another option is hard-coding it so that all consumers have the same namespace, so when they get exported to datadog they'll all be consistent which might be useful for making cross-account/cross-team dashboards, since the metrics at that point will also (I think) be tagged with the account number, environment, etc.

[kevin-secrist]

The variables in this file can be removed

[kevin-secrist]

If this is always false we don't need the code for it. For now I think it makes sense to just remove it, if it's not being used. We're generally going to be using datadog for monitoring the metrics anyway, since those will be available.

[kevin-secrist]

This should be configurable, or it should depend on if cloudwatch_namespace is empty.

[kevin-secrist]

Might be good to try out what it'd look like to consume these in datadog as a PoC. I think we'd probably want monitors on all the hosts, but have different severities based on if the host is critical or not. Inevitably we may end up making a dashboard that aggregates all of these across all accounts.

Also might be worth looking into if these are necessary at all. We export metrics to datadog, so we could use that do the alerting on that side. Maybe skip making the cloudwatch alarms if var.alarm_sns_topic_arns is empty? That gives consumers a choice if they want to put datadog into the equation. Without datadog, I guess you'd put a SNS topic that hits alertops or something here.