Upstash

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-09-29 15:12 UTC

   Between 15:12 and 15:16 UTC, we observed a brief increase in load on the connection proxy layer in us-east-1, which may have affected some existing and new connections. The team responded quickly and resolved the issue. All systems are now fully operational.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-09-09 12:00 UTC

   Upstash Vector experienced a brief DNS resolution issue in the US East (us-east-1) region lasting approximately 20 minutes. During this period, new DNS resolution attempts may have failed. Existing connections and clients with cached DNS records were not affected. The issue has...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Login, tokens, certificates or credentials failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-09-03 08:00 UTC

   We experienced a brief issue affecting access to the Upstash Console. Our team quickly identified the cause and resolved the issue within minutes. The console is now operating normally.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-08-28 22:20 UTC

   Between 22:20 and 22:33 UTC, QStash clients in us-east-1 experienced disruptions related to message persistence. The team applied a fix, and service has been restored.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-07-22 08:50 UTC

   We are investigating connectivity issues affecting a subset of Redis databases hosted on Fly.io. The issue is related to an ongoing Fly.io infrastructure disruption in the DFW region (https://status.flyio.net/incidents/n4z2my6qw4sb). We are working to restore availability and...

2. monitoring 2026-07-22 09:01 UTC

   The affected Fly.io-hosted machines recovered at 08:48 UTC, and service availability has been restored. We are monitoring the systems to confirm continued stability.

3. resolved 2026-07-22 10:05 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-07-16 06:30 UTC

   Partial outage due to http parsing errors. Problem was resolved couple minutes later. If problem continues, refreshing the DNS cache is recommended.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1), N. California, USA (us-west-1), Frankfurt, Germany (eu-central-1)

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2026-06-25 15:23 UTC

   We are currently experiencing a partial service disruption affecting Upstash Redis in the following regions: * eu-central-1 * us-east-1 * us-west-1 Our team is investigating the issue and working to restore full service as soon as possible. We will share updates as more...

2. monitoring 2026-06-25 15:31 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-06-25 16:36 UTC

   This incident has been resolved, we will publish RCA soon.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-06-23 16:22 UTC

   We are currently investigating this issue.

2. monitoring 2026-06-23 16:32 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-06-23 17:14 UTC

   This incident has been resolved.

4. postmortem 2026-06-24 12:28 UTC

   ### **Summary** On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Box

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2026-06-23 06:05 UTC

   Between 01:18 and 04:03 UTC, customers were unable to access running boxes or perform create, read, update, or delete operations. Running containers were not affected and continued operating throughout. The issue has been fully resolved and all box operations are functioning...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-06-19 09:11 UTC

   We identified an outbound networking issue affecting a subset of QStash server instances in the EU region between 2026-06-19 09:11 UTC and 2026-06-19 09:45 UTC. During this period, some requests dispatched by QStash may have failed. The issue has been resolved, and QStash is...

2. postmortem 2026-06-19 13:52 UTC

   ## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-06-04 10:10 UTC

   We are currently experiencing a service disruption affecting Upstash Vector in the IAD region. Initial investigation indicates that the issue is related to a Fly.io networking/routing problem impacting connectivity in the region. The Fly.io team is actively investigating the...

2. monitoring 2026-06-04 12:56 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-06-04 14:09 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: AWS - US-EAST-1

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-29 20:18 UTC

   Some Upstash Vector indexes in the US-EAST-1 region may be experiencing intermittent slowness. Initial investigation indicates this may be related to memory pressure on the affected servers. Our team is actively investigating the root cause and working to restore normal...

2. monitoring 2026-05-29 23:44 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-05-30 07:51 UTC

   The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1), N. California, USA (us-west-1), Oregon, USA (us-west-2), Frankfurt, Germany (eu-central-1), Ireland (eu-west-1), Singapore (ap-southeast-1), Sydney, Australia (ap-southeast-2), Mumbai, India (ap-south-1), Tokyo, Japan (ap-northeast-1), São Paulo, Brazil (sa-east-1), Ohio, USA (us-east-2), London, UK (eu-west-2), Council Bluffs, Iowa, North America (us-central1), Ashburn, Virginia, North America (us-east4), Tokyo, Japan, APAC (asia-northeast1), St. Ghislain, Belgium, Europe (europe-west1)

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-22 23:27 UTC

   New database provisioning is currently failing due to an ongoing upstream provider issue affecting DNS. Existing databases and connections are unaffected — only the creation flow is impacted. We're monitoring the situation and will update once the upstream issue is resolved.

2. monitoring 2026-05-22 23:48 UTC

   Upstream issue resolved — new database creation is working again. We're monitoring to confirm full recovery.

3. resolved 2026-05-23 00:00 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1), N. California, USA (us-west-1), Sydney, Australia (ap-southeast-2), Mumbai, India (ap-south-1)

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. monitoring 2026-05-14 15:47 UTC

   We identified intermittent connection issues affecting some regions. Fix is being deployed. We'll update the list of affected regions shortly.

2. identified 2026-05-14 15:49 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2026-05-14 16:00 UTC

   A fix has been implemented and we are monitoring.

4. resolved 2026-05-14 17:31 UTC

   Earlier today, unexpected load on our proxies caused intermittent connection issues for Upstash Redis in the following regions: us-east-1, us-west-1, ap-southeast-2, and ap-south-1. During this period, some clients may have seen connection timeouts or elevated error rates when...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-12 09:49 UTC

   Some regions are experiencing connectivity issues due to an ongoing network problem. We are currently investigating

2. identified 2026-05-12 10:50 UTC

   The issue has been identified and the fix is being implemented.

3. monitoring 2026-05-12 12:05 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2026-05-12 13:48 UTC

   This incident has been resolved.

5. postmortem 2026-05-15 12:43 UTC

   On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-11 15:05 UTC

   Some databases may experience increased latency or timeouts in Fly.io’s FRA region.

2. investigating 2026-05-11 15:07 UTC

   We are continuing to investigate the issue.

3. investigating 2026-05-11 17:19 UTC

   We are working with Fly team to investigate the root cause.

4. resolved 2026-05-11 18:22 UTC

   The incident has been resolved. We are working with Fly team on RCA.

5. postmortem 2026-05-15 12:49 UTC

   On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: US-EAST-1

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-08 09:46 UTC

   We are currently investigating the issue.

2. investigating 2026-05-08 09:46 UTC

   We are continuing to investigate this issue.

3. monitoring 2026-05-08 10:06 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2026-05-08 10:08 UTC

   This incident has been resolved, we will publish RCA soon.

5. postmortem 2026-05-12 08:25 UTC

   **Root Cause Analysis** On **April 24**, we deployed a more optimized scheduler implementation in the **US East \(N. Virginia\)** region. On **May 8**, a user who had active schedules deleted their account. Under normal behavior, scheduled tasks associated with a deleted account...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: US-EAST-1

Failure mode: A schema or data migration caused the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-05-05 14:24 UTC

   We are currently experiencing issues in the US region. - Duplicate Deliveries: During this period, some scheduled jobs may be executed twice. - Schedule Disruption: Schedules created between April 24, 2026 and May 2, 2026 are currently not running. Our team is actively working...

2. identified 2026-05-05 14:29 UTC

   We are continuing to work on a fix for this issue.

3. monitoring 2026-05-05 15:44 UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring 2026-05-05 21:31 UTC

   Main schedule functionality is back to normal. We are currently checking if previously created schedules are delivered as expected before marking the incident as resolved.

5. resolved 2026-05-06 08:05 UTC

   This incident has been resolved.

6. postmortem 2026-05-06 08:55 UTC

   # **Incident Postmortem: Scheduled Jobs Inconsistency in US Region** On May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update. The issue has been resolved, and all affected schedules have been restored....

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-04-02 18:45 UTC

   Servers in the iad region experienced unexpected disk load, resulting in elevated latencies and a temporary read-only state. We are migrating replicas to new instances to mitigate the issue and expect to have it fully resolved shortly.

2. resolved 2026-04-02 19:53 UTC

   Replication complete, incident resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: Google Cloud.

1. resolved 2026-03-27 15:30 UTC

   Due to a race condition in a process that attaches static IPs to nodes, some of the IPs in the dns were detached from the nodes, causing timeouts.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Tokyo, Japan (ap-northeast-1)

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-03-06 13:50 UTC

   We are currently investigating this issue.

2. monitoring 2026-03-06 14:08 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-03-06 14:49 UTC

   This incident has been resolved.

4. postmortem 2026-03-09 10:07 UTC

   On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2026-01-23 15:30 UTC

   We identified the cause of elevated latency impacting some databases in us-east-1 region between 15:30–15:35 UTC as a sudden surge of connection attempts that hit OS-level connection limits on our proxy layer. This resulted in slower new connection establishment and increased...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: AWS US-EAST-1

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-12-10 11:50 UTC

   We identified an issue in Regional Databases in US-East-1 where database replicase have connectivity issues with each other. Other regions are not impacted. Global databases are not impacted. We are working on the issue.

2. resolved 2025-12-10 12:08 UTC

   Issue has been identified and replicas were successfully reconnected.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Upstash Console, Context7 Console

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-12-05 09:03 UTC

   Upstream provider confirmed an incident. We are investigating the impact and potential resolutions.

2. resolved 2025-12-05 09:18 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2025-12-01 07:00 UTC

   We identified and fixed a bug that could cause messages with Flow Control enabled to be delayed longer than their configured delay, resulting in unexpectedly long pending times. The fix is in place and the issue should not recur. If you’re still seeing unusually long-delayed...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Upstash Console

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-10-20 14:52 UTC

   We are currently experiencing errors in the Upstash Console due to issues with one of our upstream providers. This may affect access to the dashboard and related operations. Our team is actively monitoring the situation and working to mitigate the impact. We will provide updates...

2. monitoring 2025-10-20 15:05 UTC

   Console is back to normal again. We are currently monitoring.

3. monitoring 2025-10-20 18:07 UTC

   Developer API (api.upstash.com) is having availability issues alongside Upstash Console. While we are monitoring the underlying cloud provider's status updates, we are also working on a remediation.

4. resolved 2025-10-20 18:50 UTC

   A fix has been deployed as a workaround so that our systems are not affected from the ongoing incident of the cloud provider

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2025-10-20 07:00 UTC

   As a side effect of an incident on the underlying cloud provider, Upstash Console has had availability issues between 07:00UTC and 09:23UTC. Only Upstash Console is impacted, Upstash products remained operational.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2025-10-10 15:46 UTC

   Between 15:46–15:55 UTC, some client connection attempts to databases in us-east-1 timed out due to unexpected high load on a server. The node was recovered at 15:50 UTC, and the updated DNS record propagated by 15:55 UTC. Services are operating normally.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2025-09-09 12:00 UTC

   Impact: A subset of clients connecting through the eu-central-1 region experienced increased error rates and timeouts when accessing certain databases. Clients in us-west-2 were also briefly affected. The issue was limited in scope and did not impact other regions. Root Cause:...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1)

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2025-08-18 12:37 UTC

   Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial...

2. postmortem 2025-08-18 13:30 UTC

   Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Upstash Console

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2025-06-26 06:53 UTC

   Some users may experience login issues due to a disruption in our authentication provider. We’re actively monitoring the situation.

2. monitoring 2025-06-26 07:12 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-06-26 07:34 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-06-12 14:28 UTC

   We are currently investigating this issue.

2. monitoring 2025-06-12 14:40 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-06-12 16:07 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1), N. California, USA (us-west-1), Oregon, USA (us-west-2), Frankfurt, Germany (eu-central-1), Ireland (eu-west-1), Singapore (ap-southeast-1), Sydney, Australia (ap-southeast-2), Mumbai, India (ap-south-1), Tokyo, Japan (ap-northeast-1), São Paulo, Brazil (sa-east-1), Ohio, USA (us-east-2), London, UK (eu-west-2)

Failure mode: A code release introduced the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2025-06-11 06:51 UTC

   We are currently investigating this issue.

2. investigating 2025-06-11 08:24 UTC

   We are continuing to investigate this issue.

3. monitoring 2025-06-11 09:20 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2025-06-11 10:02 UTC

   This incident has been resolved.

5. postmortem 2025-06-11 11:49 UTC

   A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts. As a result,...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: N. Virginia, USA (us-east-1)

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2025-06-10 07:23 UTC

   We are currently investigating this issue.

2. monitoring 2025-06-10 07:45 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-06-10 12:08 UTC

   This incident has been resolved.

4. postmortem 2025-06-11 11:37 UTC

   A routine system maintenance operation at the OS level led to the application of system updates across multiple EC2 instances in our clusters in several AWS regions. These updates included changes to networking components, which inadvertently triggered restarts. As a result,...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: Ireland (eu-west-1)

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. monitoring 2025-05-22 16:44 UTC

   A fix has been implemented and we are monitoring the results.

2. resolved 2025-05-22 17:41 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-04-18 12:36 UTC

   We are currently investigating this issue.

2. investigating 2025-04-18 12:36 UTC

   We are continuing to investigate this issue.

3. monitoring 2025-04-18 12:46 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2025-04-18 13:18 UTC

   This incident has been resolved.

5. postmortem 2025-04-18 13:19 UTC

   An internal cleanup task for QStash events, coinciding with disc layer compaction task has caused performance degradation to some users. ‌ Mitigation: Team has mitigated the event by pausing some of these tasks and monitored the status for a while. ‌ Fixes: Improvements are...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-04-02 07:15 UTC

   We are currently investigating this issue.

2. monitoring 2025-04-02 08:15 UTC

   A fix has been implemented and we are monitoring the results.

3. investigating 2025-04-02 16:21 UTC

   We are currently investigating this issue.

4. investigating 2025-04-02 16:29 UTC

   We are continuing to investigate this issue.

5. investigating 2025-04-02 16:33 UTC

   We are continuing to investigate this issue.

6. monitoring 2025-04-02 17:33 UTC

   Main QStash service is back to normal. Event logs are still temporarily unavailable.

7. monitoring 2025-04-02 20:20 UTC

   Monitoring: Main QStash service is back to normal. Event logs service is back online but events will be lagging a few mins.

8. resolved 2025-04-02 21:12 UTC

   QStash service and Event logs are fully functional without any remaning issues.

9. postmortem 2025-04-03 15:49 UTC

   ### Product: QStash ### Incident Summary Due to high load, the volume of QStash event logs reached to a point which caused latency in the underlying data store operations. Event log creation was slowed down and lead to performance degradation in QStash request processing. In...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-03-12 09:43 UTC

   We are currently investigating this issue.

2. monitoring 2025-03-12 10:08 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-03-12 10:16 UTC

   This incident has been resolved.

4. postmortem 2025-03-12 12:26 UTC

   On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: AWS EU-WEST-1

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. identified 2025-02-25 14:03 UTC

   Regional AWS eu-west-1 cluster is experiencing performance degradation, and we are adding more resources to the cluster.

2. monitoring 2025-02-25 20:54 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-02-25 21:23 UTC

   This incident has been resolved.

4. postmortem 2025-02-26 12:29 UTC

   ### Incident Summary During a maintenance update to the regional Upstash Redis databases in AWS eu-west-1, several databases hosted in that region has unnecessarily triggered a full synchronisation between their primary and backup replicas. ### Root Cause A full synchronisation...

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. resolved 2025-02-20 08:00 UTC

   We experienced a very short period of API downtime for the incoming requests to QStash due to urgent maintenance to ensure the stability and performance of our services. Our team acted quickly to address the issue, and everything is now fully operational.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2025-02-06 10:10 UTC

   Latest QStash release caused some Workflow runs to fail due to a bug in Workflow URL detection mechanism. Affected workflows did not start at all and moved directly to DLQ. These workflow runs, which show "detected non-workflow destination" message in the response body, can be...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: AWS US-EAST-1

Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2025-01-29 18:03 UTC

   We observed a disk failure on some instances on 16:52 UTC and restarted affected components to bring them back online. Issue is resolved on 17:13 UTC. During the time of the issue databases reachability are affected, no data is lost.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-11-13 14:14 UTC

   We are currently investigating this issue.

2. identified 2024-11-13 15:38 UTC

   The issue has been identified and a fix is being implemented.

3. resolved 2024-11-13 17:27 UTC

   We will be sharing a postmortem about the incident soon.

4. postmortem 2024-11-13 20:38 UTC

   **Product:** QStash **Impact:** Degraded performance, delayed processing of events, and duplicate event deliveries for some customers ## Incident Summary QStash experienced an incident marked by a sudden and extreme load on our servers. This caused a degradation in performance,...

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2024-11-13 09:31 UTC

   We are currently investigating this issue.

2. identified 2024-11-13 10:28 UTC

   The issue has been identified and a fix is being implemented.

3. identified 2024-11-13 10:28 UTC

   We are continuing to work on a fix for this issue.

4. monitoring 2024-11-13 12:12 UTC

   A fix has been implemented and we are monitoring the results.

5. resolved 2024-11-13 12:49 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2024-09-02 12:18 UTC

   Our data processing infrastructure is running behind. No data has been lost and the system should be caught up shortly.

2. monitoring 2024-09-02 14:19 UTC

   We are continuing to monitor for any further issues.

3. resolved 2024-09-02 14:53 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Affected: EU-CENTRAL-1

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-08-23 05:32 UTC

   The issue has been identified and a fix is being implemented.

2. resolved 2024-08-23 06:01 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. resolved 2024-06-05 14:02 UTC

   Some of the databases in us-east-1 region might have experienced increased latencies.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. identified 2024-06-03 08:52 UTC

   The issue has been identified and a fix is being implemented.

2. monitoring 2024-06-03 09:06 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2024-06-03 09:20 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2024-05-09 14:39 UTC

   We are currently investigating this issue.

2. monitoring 2024-05-10 09:42 UTC

   We are working on a fix and monitoring the cluster performance.

3. resolved 2024-05-13 08:30 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on Upstash's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2024-03-13 08:30 UTC

   We have taken actions to increase the capacity of the region. During the operation, clients might have experienced higher than usual latencies for about 15 minutes.

Excerpts are shortened. Read the full incident on Upstash's status page.