feat: Notifications, Observability & Production Hardening

[ray-cod]

Summary

Adds transactional outbox processing, durable event dispatch, RabbitMQ publisher confirms, retry and dead-letter handling, notification domain and preferences, in-app and email notifications, SMTP delivery, persistent security audit events, health/readiness improvements, OpenTelemetry traces, metrics and OTLP export, correlation and trace propagation, HTTP resilience, graceful shutdown, worker failure isolation, data retention, database indexes, security hardening, production configuration, Docker observability tooling, integration and resilience tests, XML documentation enforcement, and updates the project's technical documentation to reflect the completed architecture and production capabilities.

Type of Change

• Feature
• Bug fix
• Refactoring
• Documentation
• CI/CD
• Security
• Performance
• Tests

Validation

• dotnet restore
• dotnet build
• dotnet test
• dotnet format --verify-no-changes
• Security/dependency checks reviewed

Architecture Impact

Describe any changes to:

• Domain: Added notification, notification preference, outbox, and persistent audit domain models required for production-oriented behavior.
• Application: Added outbox, notification, observability, and delivery abstractions and refactored event publication so business state and integration events are persisted within the same database transaction.
• Infrastructure: Added transactional outbox persistence and dispatching, RabbitMQ publisher confirms, retry/dead-letter handling, SMTP notification delivery, development-safe notification delivery, observability, HTTP resilience, persistent auditing, data retention, and production hardening.
• API: Added notification capabilities and strengthened health/readiness, correlation, observability, security, and production configuration behavior.
• Workers: Added outbox dispatching, notification delivery, failure isolation, retry handling, and graceful shutdown behavior for background processing.
• Events: Changed integration-event delivery from direct RabbitMQ publication to transactional outbox dispatch with durable retry and dead-letter handling.
• Database: Added outbox, notification, notification preference, and persistent audit storage, together with required indexes, retention support, and atomic business-state/event persistence.
• External integrations: Added SMTP email delivery, reliable RabbitMQ publishing, OpenTelemetry OTLP export, and resilient HTTP dependency handling.

Security Considerations

Improves production security through persistent security auditing, safer event delivery, production configuration hardening, controlled notification delivery, HTTP resilience, and protection against accidental development email delivery. Secrets remain configuration-managed and are not committed to source control or exposed through application logging.

Breaking Changes

• No
• Yes — explain below

Notes

This phase is the final backend phase and completes the transition toward a production-oriented FinSight architecture. The project documentation has also been updated to reflect the completed system architecture, capabilities, operational components, and engineering practices.