Problems with white page when opening eAccounting for some Customers
Incident Report for Visma Cloud Services
Postmortem

After looking into and following up on this incident, we can now give more information about what happened and what has been done/is planned to be done to minimize and/or prevent it from happening in the future.

Chain of events
2019-07-30

  • Around 11:00 we got the first reports from the support department that customers got a white page trying to log in to Visma eAccounting
  • 11:45 we restarted some of our web instances based on the result from our monitoring tools (a restart means that logged in customers are transferred to another instance - an error message might appear but you should not have to log in again). Logfile and process dump was taken from that instance for investigation.
  • During the afternoon we got the problem on some other instances that were then restarted. We also re-deployed the last version of Visma eAccounting just to make sure that nothing had gone wrong with the latest release package

2019-07-31

  • Developers were working together trying to find the root cause.
  • We decided to revert to an earlier version but since (with all facts) the error was in this version as well, this had no effect. Note that the revert of the client version will NOT affect any data.

2019-08-01

  • We kept on investigating and released a new version with some possible fixes but we saw no improvement regarding the issue we had.

2019-08-02

  • Microsoft did a version upgrade on their cloud services in Azure (patch) close to these dates and we also tried to release with a degraded version (we had indications that this could be a possible cause) but still no effect on the problem.
  • Around lunchtime we did another release with some extended logging.
  • During the afternoon the problem was identified and around 16:00 we started a new deploy. After monitoring the systems we saw that this correction had expected result and the service went back to fully operational.

During the incident, we continuously did monitor the service also outside office hours to be able to restart affected instances as soon as possible to minimize customer impact.

Root cause
The Visma eAccounting incident affecting our customers from July 30 to August 2 was caused by our own code. This was just triggered during specific circumstances and was not something that we at the time were able to identify before releasing the code into production. The code causing this was released during July 29 and was in production around 24 hours before we got the first indication that something was wrong.

Technical details
The intention of the new code was to log information of particular database exceptions.

In the context where the error happens a container with a number of objects was accidentally disposed after the database error was logged. Since the container is used in several places in the code the dispose causes a failure of the running web server instance and it will no longer respond to any requests.

To summarize this - the occurrence of a particular and quite unusual database error causes the instance running the query to fail and stop responding to any following requests.

Impact
This was not something that did impact all of our customers. Most logged in users had a fully working Visma eAccounting during these days. When the exception that caused the problem did occur this was affecting a specific server and customers trying to log in or already logged into this server got a white page. This means that between 10-15% were affected each and every time the issue was triggered. This happened a few times during these days so some customers might have been affected one or more times, but most customers did probably not notice the incident.

Summary/Follow up

  • Internal incident meeting will be held to see if we can find even more action points to help prevent similar things in the future.
  • We will implement the solution described in “Technical details” below.
  • No data was lost or compromised and we were not attacked in any way - it was our own code causing the issue preventing some of you from logging in or stopping you from working when logged in.

Technical details
Since the container is such a critical component, invalid use will be prohibited by wrapping it in another object only exposing the safe to use functionality. This means that the same type of error cannot be released to the production environment because the code does not even compile on invalid or prohibited usage.

We are very sorry for any problems this might have caused you. We constantly try to learn and improve, and our mission is of course to deliver a service which is the obvious choice for you and your company with great features and high availability.

The Visma eAccounting Development Team

Posted 2 months ago. Aug 16, 2019 - 15:25 CEST

Resolved
The issue is now resolved and eAccounting is up and running as normal.
Posted 3 months ago. Aug 02, 2019 - 20:15 CEST
Monitoring
The service is back to normal and we are monitoring the result.
Posted 3 months ago. Aug 02, 2019 - 15:56 CEST
Update
We are still investigating the root cause of the problem. We are working together with our hosting provider, to find a permanent fix to the problem. A temporary fix has been deployed, and we are monitoring the result.
Posted 3 months ago. Aug 02, 2019 - 09:20 CEST
Update
We are still investigating to find out what is causing the problem/disruption. Try to restart the browser and sign in again, this can help.
Posted 3 months ago. Aug 01, 2019 - 13:42 CEST
Update
We still have some customers who get a white page problem when they log into eAccounting. We will continue to investigate what causes the problem/disruption. We are sorry for the inconvenience this may cause you.
Posted 3 months ago. Aug 01, 2019 - 07:27 CEST
Investigating
We are still investigating to find out what is causing the problem/disruption. We are sorry for the inconvenience this may cause you.
Posted 3 months ago. Jul 31, 2019 - 10:18 CEST
Monitoring
The fix is implemented and we are monitoring the result. We will update again if needed.
Posted 3 months ago. Jul 30, 2019 - 16:56 CEST
Investigating
Some customer is​ still getting a white page when accessing eAccounting. We are still investigating to find out what is causing the problem/disruption.
Posted 3 months ago. Jul 30, 2019 - 15:22 CEST
Monitoring
The fix is implemented and we are monitoring the result. We will update again if needed.
Posted 3 months ago. Jul 30, 2019 - 13:35 CEST
Update
We are still investigating to find the root cause of the problem/disruption. We will provide more information shortly.
Posted 3 months ago. Jul 30, 2019 - 11:49 CEST
Investigating
We are currently investigating a service disruption in Visma eAccounting.
Posted 3 months ago. Jul 30, 2019 - 11:17 CEST
This incident affected: Visma eAccounting / Visma eEkonomi / ePasseli (Visma eAccounting, Visma eAccounting Payslip / Visma eEkonomi Lönebesked).