TRAC-2821 RCA – IOT Direct – Degradation of ALH Connectivity and Watchtower enforcement.
Event Start Date/Time UTC: 10-Dec-2025 11:20 UTC
Event End Date/Time UTC: 10-Dec-2025 21:15 UTC
Date/Time Reported UTC: 10-Dec-2025 13:44 UTC
Severity: High
Services Affected: ALH Connectivity and Watchtower Enforcement
Case Number: 159601
Duration: 9 hours 55 minutes
Description of Failure:
On December 10, 2025, at 11:20 UTC, the virtual machines supporting ALH services experienced a failure in the underlying storage subsystem, resulting in approximately 50% degradation of ALH data services and ALH WatchTower enforcement services. ALH data services were restored at 16:50 UTC, while ALH WatchTower enforcement services continued to be impacted until 21:15 UTC, when full service restoration was achieved
Impairment Cause:
Approximately 50% degradation of ALH data services and ALH WatchTower enforcement services.
Impairment Resolution:
Service restoration was achieved by bypassing the impacted ALH nodes, followed by the reset of an affected virtual machine.
Corrective Action Items:
Perform a comprehensive investigation of the storage subsystem failure to confirm root cause and implement the most resilient architecture to prevent future impact.
Due Date:
December 31, 2025