Introduction

On 5th of June 2023 between 09:14 CEST and 10:00 CEST Tink’s services experienced a degraded performance affecting multiple products (Payments, Risk Insights, Expense Check, Income Check, Account Check). The degradation resulted in decreased success rates during the time of the incident. 

We know how important it is to our customers that we provide a highly available and reliable service and we sincerely apologize for the disturbances this incident caused.

‌

Root Cause Analysis

The incident was caused by an incompatible library version unintentionally introduced in one service, causing degraded performance of that component. While this change was tested in pre-production environments and passed existing health checks, lack of coverage for the particular failure scenario led to a faulty production release and the subsequent degradation.

‌

Remediation

At 09:46 CEST Tink rolled back its services to the previous versions and by 10:00 CEST performance was back to normal. 

‌

Follow-up actions

As a consequence of this incident we will focus on the following actions to prevent similar scenarios from occurring again: 

• Introduce stricter release procedures for the affected component
• Expand testing methods to discover possible dependency issues regardless of build environment
• Expand existing deployment health checks to include more scenarios