Issue affecting several Tink products

Incident Report for Tink

Postmortem

Introduction

On 12th February 2024 between 00:45 AM and 02:00 CET Tink experienced a technical issue which impacted all of our Products’ (with degraded performance experienced until 11.00 CET for the following products: Account Check, Business Account Check, Risk Insights, Income Check and Expense Check).

Root Cause Analysis

After a thorough investigation, it was found out that a database upgrade due to be conducted on a no-downtime basis failed and led to our services being unavailable from 00.45 AM CET to 02.00 AM CET.

Additionally, databases required by Account Check and Risk products remained unavailable until 11.00 CET and led to a partial degradation of those products.

Remediation

Once the issue was identified, we performed all the necessary steps to complete the required upgrades and bring up our services into operational status again.

Follow-up actions

To prevent similar issues in the future Tink will be implementing the following actions:

Implement tighter DB upgrade mechanisms and disable automatic upgrades from our cloud provider.
Perform all DB upgrades in a supervised setup by performing upgrades on replicas and then switching our services to point to them.

Posted Feb 13, 2024 - 13:01 CET

Resolved

After monitoring the situation for a while we haven't encountered any additional issues. We will go ahead and close this incident.

Posted Feb 12, 2024 - 13:15 CET

Monitoring

We have successfully resolved the underlying issue that caused Identity Data to be unavailable. We can see Account Check, Income Check, Expense Check and Risk Insights returning to full operational performance. We will monitor the situation for some time. Thank you again for your patience as we working on the resolution. We apologise for any inconvenience this issue has caused.

Posted Feb 12, 2024 - 11:30 CET

Update

Tink is actively working on resolving the remaining issue with Identity Data which is currently causing degraded performance for Expense Check, Income Check, Risk Insights and Account Check products. We made significant progress towards completely remedying the issue, however an exact ETA is not available yet. We will communicate it as soon as it becomes available. Thank you for your patience.

Posted Feb 12, 2024 - 10:14 CET

Update

We are continuing to work on a fix for this issue. All products are fully functional since 02:00 AM CET with the exception of Account Check that still experiences degraded performance due to Identity Data not being available.

Posted Feb 12, 2024 - 07:01 CET

Update

Posted Feb 12, 2024 - 05:32 CET

Identified

The issue has been identified and we are currently working on a solution.

Posted Feb 12, 2024 - 03:22 CET

Investigating

We are currently investigating an issue that is affecting several Tink products.

See details for the full list of affected products.

Posted Feb 12, 2024 - 00:45 CET

This incident affected: Payments, Account Check, Business Account Check, Income Check, Risk Insights, Expense Check, and Money Manager.