WHAT: From June 1st 4:06 AM CEST to June 2nd 4:31 PM some customers and integration of Visma.net ERP were unable to use the APIs. The error message they got was 401 Unauthorized. Customers reporting this challenge were using our API to integrate Visma.net ERP with other solutions.
ROOT CAUSE: The Platform team was doing maintenance and security improvements cleaning up some old authentication tokens for these APIs. Unfortunately a script was not 100% correct and some active tokens were deleted also.
RESOLUTION: The Platform team recovered the authentication tokens from a backup. Unfortunately this took a while to recover the correct tokens - both on writing the correct script and running it - considering the size of data. There were some million tokens to restore - we prioritized the integrations having errors first - around lunch time on June 2nd these were restored, while until the end of the working day all tokens were restored.
PREVENTIVE ACTIONS: The affected integrations are using an old version of this service, where there is a project ongoing to lift them to an improved and more secure solution. Meanwhile the Platform team will keep this solution available and will do only improvements on caching and not deleting more the tokens.
Posted Jun 13, 2022 - 15:40 CEST
Resolved
The issue is now resolved and the service is up and running as normal.
Posted Jun 02, 2022 - 16:31 CEST
Update
We are now in a normal operational status, but will continue to monitor. To avoid misunderstanding: no customer data is restored or affected in this incident, only structural data.
Posted Jun 02, 2022 - 14:42 CEST
Update
75% full restore done by Platform team! We will try to have them run all today, even if it takes after the working hours
Posted Jun 02, 2022 - 14:41 CEST
Update
We are still working with restoring data and have received confirmation from several customers that their integrations are now up and running. We are about halfway through and next update will come when we are 100% done.
Posted Jun 02, 2022 - 12:31 CEST
Monitoring
We have now restored all data for the active integrations according to our logs. Please check if your active (daily) integrations are up and running. Some integrations are not active every day, but are scheduled to certain days etc - next step for us is now to restore all data to ensure that scheduled integrations will work.
Posted Jun 02, 2022 - 09:50 CEST
Update
Last night we prepared for and tested restoring of the data from a backup, and we have now started to roll-out the solution. Some of the affected integrations is now running, but a total restore are estimated to take about 3 hours. For now our focus is to make sure the roll-out is completed, then we will have an incident review to see how this could happen, and what we will do to make sure it is not happening again.
Posted Jun 02, 2022 - 08:46 CEST
Update
Rolling back the changes is taking longer. More information we will update in the morning. Again, sorry for the inconvenience.
Posted Jun 02, 2022 - 08:42 CEST
Investigating
Rolling back the changes is taking longer. More information we will update in the morning. Again, sorry for the inconvenience.
Posted Jun 01, 2022 - 19:11 CEST
Update
We are still working on rolling back the changes. More information will be published when available.
Posted Jun 01, 2022 - 14:34 CEST
Identified
We have found the root cause for the issue and are working on rolling back the changes. We are aiming of having this done in a couple of hours. More information will be published when available. Again, sorry for the inconvenience.
Posted Jun 01, 2022 - 11:34 CEST
Investigating
We are currently investigating an issue with API connection in Visma.net ERP. Since 0400 CET today, a lot of clients have been getting 401 Unauthorized on all their API calls. A possible workaround for the issue is to re-generate the bearer token. We are looking into the root cause for the issue, and will come back with more information when available. We will continue to monitor the environment closely. Sorry for the inconvenience.
Posted Jun 01, 2022 - 10:06 CEST
This incident affected: Visma.net ERP (Visma.net ERP API).