Transaction Elasticsearch storage issues in the production environment
Incident Report for Tink
Postmortem

Impact:

Public APIs depending directly on reading or writing transactions were impacted for the duration of the incident, ~08:30 - 09:20 CET on 2022-01-25. The behaviour also impacted other domains like e.g. Money Manager that is depending on transaction retrieval.

The issue re-appeared the next day, 2022-01-26, ~12:28 - 12:56 CET while we were still monitoring the issue.

Posted Feb 02, 2022 - 15:22 CET

Resolved
This incident has been resolved.
Posted Feb 01, 2022 - 12:03 CET
Update
Please note that we're still monitoring and investigating the incident.
Posted Jan 27, 2022 - 09:10 CET
Monitoring
The last disturbances happened between 12:28 and 12:56 CET. We are now back to monitoring the situation.
Posted Jan 26, 2022 - 14:11 CET
Investigating
Since around 12:30 CET, we're seeing the higher latencies and the higher error rates again. We've additionally reached out to AWS to find a solution.
Posted Jan 26, 2022 - 12:48 CET
Update
Please not that we're still taking care about this issue.
Posted Jan 25, 2022 - 12:34 CET
Monitoring
The following endpoint is/was also affected by this incident:
GET /data/v2/transactions

However, we're already seeing the issues go down. We'll monitor the situation for a while.
Posted Jan 25, 2022 - 09:48 CET
Update
We are doing our best to get this issue resolved as soon as possible.
Posted Jan 25, 2022 - 09:18 CET
Update
These endpoints might receive an error:

GET /api/v1/budgets/summaries
POST /api/v1/user-report-bundles/
DELETE /api/v1/credentials/{id}
PUT /api/v1/accounts/{id}
POST /api/v1/search/

These might experience high latencies:

DELETE /api/v1/credentials/{id}
POST /api/v1/search/
PUT /api/v1/accounts/{id}
POST /api/v1/user-report-bundles/
GET /api/v1/budgets/summaries
GET /api/v1/transactions/
PUT /connector/users/{externalUserId}/transactions/{externalTransactionId}
Posted Jan 25, 2022 - 09:14 CET
Investigating
The issue leads to higher latencies and higher error rates in the Transaction APIs. We're just looking into the details, an update is coming soon.
Posted Jan 25, 2022 - 08:59 CET