Investigating an issue in Visma.net ERP
Incident Report for Visma Cloud Services
Postmortem

After deployment of Version 8.19, we experienced instability in our infrastructure. We released four changes to Visma.net ERP:

  • Customer Invoice endpoint: On object creation in code-behind for POST and PUT we do a select all ARAdjustments which is later used as lookup. Select all work with 100 documents. Problem occurs when you have customers with large data set.
  • Scheduler: When running some Scheduled tasks, filters was not included.
  • AutoInvoice: Customer obliged to VAT, but missing MVA at the end of MVA_reg_number got an error. This was affecting only Norway companies.
  • Inventory endpoint: If customer used "uom" field, the endpoint failed.
Posted Mar 10, 2020 - 14:01 CET

Resolved
The issue is now resolved and the service is up and running as normal.
Posted Feb 26, 2020 - 10:44 CET
Update
Monitoring shows that everything is good. We will continue to monitor the application.
Posted Feb 23, 2020 - 12:02 CET
Update
Monitoring shows that everything works as expected. We will continue to monitor the application.
Posted Feb 22, 2020 - 16:36 CET
Update
Monitoring shows that everything works as expected. We will continue to monitor the application throughout the weekend.
Posted Feb 22, 2020 - 05:35 CET
Update
The scheduled jobs are now enabled.
Posted Feb 21, 2020 - 21:35 CET
Monitoring
Application has been patched and all systems are up and running again. Monitoring shows that everything works as expected. We will continue to monitor the application throughout the weekend.
Posted Feb 21, 2020 - 20:52 CET
Update
Deployment preparations done, start patching all Scheduler and UI servers (AutoInvoice and CustomerInvoice).
Posted Feb 21, 2020 - 20:10 CET
Update
All API servers have been patched and monitoring shows that CustomerInvoice endpoint works as expected. We continue to monitor the application.
Posted Feb 21, 2020 - 16:25 CET
Update
We have identified and corrected the issues related to AutoInvoice, Scheduler and CustomerInvoice endpoint. We will deploy the API (CustomerInvoice) within 1 hour. We will deploy the Scheduler and AutoInvoice change at 20:00 CET. The deployment at 20:00 CET will create a 15 min downtime for user (not API). We continue to monitor the application throughout the weekend.
Posted Feb 21, 2020 - 15:42 CET
Update
We continue to test CustomerInvoice endpoint and AutoInvoice. Preliminary results of the tests looks good. We will later today give a time for a patch together with Scheduler fix.
Posted Feb 21, 2020 - 12:41 CET
Update
We are continuing to test the code changes. We will give a new status before 1300.
Posted Feb 21, 2020 - 11:32 CET
Identified
The issues related to Auto Invoice and CustomerInvoice endpoint are identified and corrected. We are currently testing the changes done. We will come back with more information related to status on testing and release time.
Posted Feb 21, 2020 - 09:35 CET
Update
We have identified the issues related to service disruption. It is connected to post/put on CustomerInvoice API endpoint. We are working on correcting the issues.
Posted Feb 20, 2020 - 13:20 CET
Update
We are currently monitoring the application.
Posted Feb 20, 2020 - 08:15 CET
Update
Database servers scale out completed. We will continue to monitor the application and investigate.
Posted Feb 19, 2020 - 21:40 CET
Update
We will scale out the database servers at 21:30 CET, there is a chance that some customers will experience up to 5 minutes of downtime.
Posted Feb 19, 2020 - 19:55 CET
Update
We are continuing to investigate this issue. The infrastructure fix that we applied had a temporary effect.
Posted Feb 19, 2020 - 14:00 CET
Update
Infrastructure change was implemented at 11:30 CET. We are continuing to monitor the application and investigate this issue.
Posted Feb 19, 2020 - 11:46 CET
Update
We will do an infrastructure change at 11:30 CET, it is possible that some of the customers will experience a small downtime.
Posted Feb 19, 2020 - 11:02 CET
Update
We are continuing to investigate this issue.
Posted Feb 19, 2020 - 09:51 CET
Investigating
We are currently investigating a service disruption in Visma.net ERP. Some of our customers are experiencing performance issues after last night release.
Posted Feb 19, 2020 - 09:38 CET
This incident affected: Visma.net ERP (Visma.net ERP, Visma.net ERP API).