Incident Summary
On 30 October 2026, GreenFlux experienced a service degradation affecting communication between charging stations and the GreenFlux platform.
Starting at approximately 10:50 CET, some charging stations experienced delayed or missing responses to OCPP messages. This affected communication required for station registration and charging-session processing, including start and stop transaction messages.
As responses were delayed, affected charging stations automatically retried messages, reconnected, or rebooted. These retries increased the amount of traffic that needed to be processed and contributed to the overall duration and impact of the incident.
The underlying messaging-service throttling subsided at approximately 16:30 CET. The accumulated message backlog continued to be processed afterwards as the platform progressively returned to normal operating conditions.
Customer Impact
During the incident, customers may have experienced:
- Delayed communication between charging stations and the GreenFlux platform.
- Delayed responses to OCPP messages.
- Charging stations reconnecting or rebooting after not receiving a response within their expected timeframe.
- Delays in processing charging-session information, including start and stop transaction messages.
- Delayed visibility of charging station or charging-session information in connected services.
There was no fleet-wide connectivity loss, and the majority of charging station connections remained established.
Timeline
Time (CET) | Event |
Before 10:50 | The platform and charging station communication were operating normally. |
10:50 | The first communication interruption occurred within the Azure messaging infrastructure. Microsoft later identified a backend infrastructure transition around this time. Some OCPP messages began experiencing delays or timeouts. |
10:52 onwards | Messaging activity started to recover from the initial interruption, but processing remained slower than normal. This resulted in increasing delays in communication between some charging stations and the GreenFlux platform. |
Approx. 11:30 | Start of storage-related throttling within Azure Service Bus. Message processing was restricted while the service attempted to process accumulated internal storage operations. |
11:00–14:15 | Processing delays continued and a significant message backlog developed. Some affected charging stations automatically retried messages, reconnected, or rebooted when responses were not received within the expected timeframe. These additional requests further increased processing demand. |
13:30 | Repeated charger communication reached its highest observed level. |
14:00 onwards | GreenFlux began additional recovery measures, including significantly increasing messaging capacity and adding capacity to supporting application services. |
14:14–16:17 | Affected application services were restarted where necessary as part of the recovery process, while additional capacity was introduced to support message processing. |
Approx. 16:30 | Internal storage backlog had reduced sufficiently and the Azure Service Bus throttling stopped. Platform processing progressively improved. |
After 16:30 | GreenFlux continued processing the accumulated messages and monitoring end-to-end charging station communication as services recovered. |
Evening | The remaining message backlog continued to decrease as the platform returned to normal operating conditions. |
Root Cause
The incident involved the Microsoft Azure messaging infrastructure used by GreenFlux to process communication between charging stations and backend services.
Microsoft's investigation confirmed that the messaging service experienced storage-related throttling. A buildup of internal storage processing operations exceeded a protection threshold within the Azure Service Bus service. As a result, Azure temporarily limited message processing to allow its internal storage operations to recover.
Microsoft also identified a backend infrastructure transition around 10:50 CET, which coincided with the start of the initial communication interruption.
The exact reason for this infrastructure transition, and whether it directly contributed to the subsequent storage-processing slowdown, remains under investigation by Microsoft.
GreenFlux identified several conditions that increased the impact after the initial disruption:
- Delayed responses caused charging stations to retry messages and reconnect, increasing processing demand.
- The application architecture had a dependency on the affected messaging service, meaning messaging delays could also delay subsequent OCPP processing.
- Increased traffic placed additional load on downstream application and database services.
- Existing capacity controls were primarily based on CPU utilisation and did not detect the storage-related bottleneck that occurred during this incident.
There was no GreenFlux production deployment or relevant messaging configuration change immediately before the start of the incident that was identified as the trigger.
Resolution and Recovery
During the incident, GreenFlux took several measures to restore normal processing:
- Messaging capacity was significantly increased to provide additional processing capacity during recovery.
- Affected application services were restarted where necessary to restore processing.
- Additional application capacity was added to address increased processing demand.
- GreenFlux worked with its cloud support partners and Microsoft Support to investigate the Azure Service Bus behaviour.
- Microsoft observed the internal storage-processing backlog beginning to decline, and the associated throttling stopped at approximately 16:30 CET.
- GreenFlux continued processing the accumulated message backlog after the throttling had ended to progressively restore normal processing across the platform.
Because multiple recovery measures were performed while the Microsoft-managed service was also recovering, it has not been possible to attribute recovery to one individual action.
Corrective and Preventive Actions
Following the incident, GreenFlux is implementing and evaluating improvements in several areas.
1. Earlier Detection
We are reviewing monitoring and alerting for:
- OCPP response times.
- Messaging latency and throttling.
- Unexpected message-backlog growth.
- End-to-end charging station communication performance.
The objective is to identify similar degradation earlier, before a significant backlog develops.
2. Improved Recovery Procedures
Our incident recovery procedures are being reviewed to better identify where congestion occurs across the full communication chain.
The updated procedures will focus on limiting additional system load during recovery and verifying that charging station communication has fully recovered before an incident is considered resolved.
3. Increased Messaging Resilience
GreenFlux is evaluating additional resilience and failover capabilities for its Azure messaging infrastructure.
This includes assessing Azure's replication and recovery capabilities and validating how applications handle message redelivery or duplicate messages during a failover scenario.
4. Improved Diagnostic Monitoring
Additional Azure Service Bus diagnostic logging will be introduced to provide greater visibility into messaging-service behaviour and allow faster investigation of similar incidents.
5. Isolation of Critical Charging Communication
GreenFlux is assessing options to further separate time-critical charging station communication from less time-sensitive processing.
This reduces the risk that congestion or disruption in one workload can affect critical OCPP communication.
6. OCPP Platform Architecture
GreenFlux is already working on the next generation of its OCPP communication architecture.
As part of this work, we will evaluate how communication with charging stations can be made less dependent on individual backend messaging components, helping prevent a messaging-service disruption from directly delaying charger responses.
Relevant lessons from this incident will be incorporated into the design of the new OCPP Gateway where appropriate.