Investigating an issue in Visma.net
Incident Report for Visma Cloud Services
Resolved
We checked this out with the product teams mostly affected by our incident and they confirm all is back to normal also on their side. So then we close this incident.
We still have some smaller improvements and we are monitoring the situation carefully.
Posted Jun 23, 2022 - 12:31 CEST
Update
According to our monitoring, we conclude the latest fixes solved the main performance problems.
We change status now to monitoring and we will continue to work toward addressing some improvements in other components of our platform.
Posted Jun 23, 2022 - 08:05 CEST
Update
With the fixes released yesterday (improvements on the application code) plus some additional infrastructure updates is clearly we're having a significant improvement. We continue the work toward solving this incident, we have several improvements in progress.
We are monitoring the situation carefully.
Posted Jun 22, 2022 - 08:27 CEST
Update
We released today the fixes, but unfortunately without significant improvements for end-customer/integration. On internal infrastructure & architecture we definitively see improvements and we need some steps more toward overall improvement.
We will keep monitoring and informing periodically about the status and progress.
Posted Jun 21, 2022 - 12:48 CEST
Update
We have several fixes in testing (incl. stress testing) which we need to ensure quality before reaching production. These include improvements both in the application code and infrastructure. Hopefully today these will reach production.
We are monitoring the situation carefully.
Posted Jun 21, 2022 - 07:41 CEST
Update
We still experience slowness, and we continue to work very hard to solve this.
There are several tasks in progress, including a stress test which hopefully will provide us with more valuable information. The logging has been improved, aiming to find where we have the bottlenecks.
Posted Jun 20, 2022 - 09:10 CEST
Update
Our logs indicate slight improvements compared to yesterday, but since different services are impacted differently, we will continue to monitor and gather feedback from impacted services.
We have several tasks in progress, and we are currently conducting extensive (stress) testing on a fix we hope to deploy early next week.
Posted Jun 17, 2022 - 08:51 CEST
Update
Several fixes have been deployed during the last 24 hours. We need more time to monitor to know whether this will help.
Several tasks are in progress, so the search for the root cause and a fix for full it is continuing with a fully staffed task force.
Posted Jun 16, 2022 - 09:05 CEST
Update
We are still having technical issues and we are working hard to fix them. Scaling up the database and the infrastructure didn't improve too much and now the bottlenecks we're experiencing are more down the process. We're looking to improve more in other areas, both on the infrastructure and applications.
We will keep monitoring and informing periodically about the status and progress.
Posted Jun 15, 2022 - 08:46 CEST
Update
We are deploying new improvements for reducing the load on our application. The load is causing periodically the Single sign-on between applications to fail.
We will keep monitoring and informing periodically with the status.
Posted Jun 14, 2022 - 09:57 CEST
Update
In order to fix this issue, we will scale up the production database on the 11th of June at 03:00 UTC.
We are expecting a few seconds of interruptions, that could be caused by DNS propagation when the switch to the new upgraded instance is done.
Posted Jun 10, 2022 - 11:09 CEST
Monitoring
We have some issues when is to handling authentication and authorization of Visma.net Services.
We have applied 2 infrastructure changes to handle this situation. We did some improvements to scale out part of our infrastructure, which so far shows improvements, but we are now monitoring to see if the periodic spikes in target response time will reoccur.
It seems that the fix that was applied didn't fix the root problem.
We are continuing to monitor and investigate the situation and we will update you when we have findings and actions.
Posted May 20, 2022 - 08:30 CEST
This incident affected: Visma Cloud Platform (Visma.net ODP).