Insite LMS is down
Resolved
Oct 29 at 09:55pm CET
The issue seems to be resolved. Smaller hickups may still occur, but we are successfully able to access all our systems.
We will post the Azure post mortem once it is available under
https://docs.insitelms.net/general-information/hosting-information
We apologize for any inconveniences and hope, Microsoft will be able to fix their Front Door deployments.
Affected services
Updated
Oct 29 at 09:53pm CET
We are seeing recovery. Thank you all for your patience!
Affected services
Updated
Oct 29 at 08:21pm CET
Update from Azure:
We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.
At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. We will provide another update on our progress within two hours, or sooner if warranted.
Affected services
Updated
Oct 29 at 07:57pm CET
Update from Azure:
We have initiated the deployment of our last known good configuration. This deployment was initially expected to complete within 45 minutes; however, due to protective blocks we have put in place to safeguard the AFD service, we are encountering some delays. While progress is ongoing, these safeguards are extending the overall deployment time. Once the rollout is complete, we will begin recovering nodes and re-routing traffic through healthy nodes to accelerate recovery.
Affected services
Updated
Oct 29 at 07:13pm CET
An interesting update from Azure:
Current status:
We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.
Affected services
Updated
Oct 29 at 06:20pm CET
Microsoft is still working to resolve the Azure Front Door issues, please check their website:
https://azure.status.microsoft/en-us/status
This is noted on their website at the moment:
Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.
We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.
We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.
Affected services
Updated
Oct 29 at 05:38pm CET
At this stage we see that not just Azure, all big providers are having issues. This seems like a much bigger problem.
https://downdetector.com/
Affected services
Updated
Oct 29 at 05:22pm CET
Finally Microsoft has acknowledged the issue on their official status page:
https://azure.status.microsoft/en-us/status
Affected services
Updated
Oct 29 at 05:12pm CET
Azure Front Door, the component to manage all incoming connections is clearly down / having issues.
We can't really access the cental Azure portal as well, and reports from around the world show, they are having the same issues.
We can only apologize at this stage for the inconvenience.
Affected services
Created
Oct 29 at 04:56pm CET
Unfortunately we see Microsoft Azure having issues again:
https://statusgator.com/services/azure
We are investigating
Affected services