Back to overview
Downtime

Insite LMS is down

Oct 29 at 04:56pm CET
Affected services
Authenticator API
API
Report API
Import API
Sync API
portal.insitelms.net
E-Mail Interface
Insite LMS Web
Public API

Resolved
Oct 29 at 09:55pm CET

The issue seems to be resolved. Smaller hickups may still occur, but we are successfully able to access all our systems.

We will post the Azure post mortem once it is available under
https://docs.insitelms.net/general-information/hosting-information

We apologize for any inconveniences and hope, Microsoft will be able to fix their Front Door deployments.

Updated
Oct 29 at 09:53pm CET

We are seeing recovery. Thank you all for your patience!

Updated
Oct 29 at 08:21pm CET

Update from Azure:

We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. We will provide another update on our progress within two hours, or sooner if warranted.

Updated
Oct 29 at 07:57pm CET

Update from Azure:

We have initiated the deployment of our last known good configuration. This deployment was initially expected to complete within 45 minutes; however, due to protective blocks we have put in place to safeguard the AFD service, we are encountering some delays. While progress is ongoing, these safeguards are extending the overall deployment time. Once the rollout is complete, we will begin recovering nodes and re-routing traffic through healthy nodes to accelerate recovery.

Updated
Oct 29 at 07:13pm CET

An interesting update from Azure:
Current status:

We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.

Updated
Oct 29 at 06:20pm CET

Microsoft is still working to resolve the Azure Front Door issues, please check their website:
https://azure.status.microsoft/en-us/status

This is noted on their website at the moment:
Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.

We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.

We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.

Updated
Oct 29 at 05:38pm CET

At this stage we see that not just Azure, all big providers are having issues. This seems like a much bigger problem.
https://downdetector.com/

Updated
Oct 29 at 05:22pm CET

Finally Microsoft has acknowledged the issue on their official status page:
https://azure.status.microsoft/en-us/status

Updated
Oct 29 at 05:12pm CET

Azure Front Door, the component to manage all incoming connections is clearly down / having issues.

We can't really access the cental Azure portal as well, and reports from around the world show, they are having the same issues.

We can only apologize at this stage for the inconvenience.

Created
Oct 29 at 04:56pm CET

Unfortunately we see Microsoft Azure having issues again:
https://statusgator.com/services/azure

We are investigating