After looking into and following up on this incident, we can now give more information about what happened and what has been done/is planned to be done to minimize and/or prevent it from happening in the future.
Chain of events
2019-07-30
2019-07-31
2019-08-01
2019-08-02
During the incident, we continuously did monitor the service also outside office hours to be able to restart affected instances as soon as possible to minimize customer impact.
Root cause
The Visma eAccounting incident affecting our customers from July 30 to August 2 was caused by our own code. This was just triggered during specific circumstances and was not something that we at the time were able to identify before releasing the code into production. The code causing this was released during July 29 and was in production around 24 hours before we got the first indication that something was wrong.
Technical details
The intention of the new code was to log information of particular database exceptions.
In the context where the error happens a container with a number of objects was accidentally disposed after the database error was logged. Since the container is used in several places in the code the dispose causes a failure of the running web server instance and it will no longer respond to any requests.
To summarize this - the occurrence of a particular and quite unusual database error causes the instance running the query to fail and stop responding to any following requests.
Impact
This was not something that did impact all of our customers. Most logged in users had a fully working Visma eAccounting during these days. When the exception that caused the problem did occur this was affecting a specific server and customers trying to log in or already logged into this server got a white page. This means that between 10-15% were affected each and every time the issue was triggered. This happened a few times during these days so some customers might have been affected one or more times, but most customers did probably not notice the incident.
Summary/Follow up
Technical details
Since the container is such a critical component, invalid use will be prohibited by wrapping it in another object only exposing the safe to use functionality. This means that the same type of error cannot be released to the production environment because the code does not even compile on invalid or prohibited usage.
We are very sorry for any problems this might have caused you. We constantly try to learn and improve, and our mission is of course to deliver a service which is the obvious choice for you and your company with great features and high availability.
The Visma eAccounting Development Team