Root cause:
Part of occupancy engine responsible for calculating footfall occupancy received data which violated specification in a rare case (only 4 readings from only one device).
The device which sent this data is currently investigated.
Gap in validation caused processing to stop.
Impact:
Occupancy data not available in real time for retrieval.
No data was lost.
Monitoring check:
Monitoring allowed for an immediate reaction after issue started as well as for a quick analysis and fix
Immediate mitigation
Extended validation to skip wrong data (applied)
Short term action
Add automatic fallback for occupancy engine to skip wrong data.
Additional action
Investigate root cause of generating wrong data