">
sevanalysis

PubNub

Infrastructure. Source: PubNub status page.

Incidents, last 90 days
3
Median to mitigation
0m
Median to resolution
0m
p95 to resolution
1h 29m
Naming an upstream provider
0%
With a postmortem on the status page
100%

What failed, last 90 days

Traffic surge or denial-of-service attack2
Orchestration, control plane, service discovery or load balancing failed1

Tagged by claude-haiku-5-5 from each incident's updates. See the methodology.

Incidents per month

All recorded incidents

Increased Subscribe error rate and latency observed globally none 2026-10-09 17:30 UTC 0m

Failure mode: Traffic surge or denial-of-service attack. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    Starting at 17:32 UTC on October 9, 2026 we observed elevated 5XX error rates and latency on the Subscribe endpoint globally. The issue was resolved by 18:18 UTC and services have returned to normal levels. A limited number of customers may have been affected. A root cause...

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On October 9, 2026, between 17:30 and 18:20 UTC, we observed elevated HTTP 5xx errors affecting the Subscribe endpoint for a subset of customers. Some subscribe requests may have failed, and a subset of messages may have been...

Excerpts are shortened. Read the full incident on PubNub's status page.

Potential for some missed messages for subscribers in US East PoP minor 2026-08-25 13:22 UTC 1h 29m

Affected: Publish/Subscribe Service, North America Points of Presence

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    As of 12:15 UTC, Users in our US East PoP are currently experiencing an issue with Publish and Subscribe service that is causing some messages to not appear in calls to that service. No errors are being returned from the service, despite messages not being returned correctly. We...

  2. monitoring

    As of 13:55 UTC, the issue affecting Publish and Subscribe services in the US East PoP has been mitigated. Synthetic monitoring has returned to normal, and no missed messages have been detected. The incident is currently believed to have been related to a monitoring failure....

  3. resolved

    With no further issues observed for the past 30 minutes, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  4. postmortem

    ### **Problem Description, Impact, and Resolution** On Tuesday, August 25th, 2026, at 12:10 UTC, our internal monitoring alerted us to an issue where a very small subset of messages within one availability zone of one region \(our US East point of presence\) may not have been...

Excerpts are shortened. Read the full incident on PubNub's status page.

Global Errors and Failures with Publish and Functions, Delays with Events & Actions none 2026-08-14 14:29 UTC 0m

Affected: Publish/Subscribe Service, Functions Service, North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence

Failure mode: Traffic surge or denial-of-service attack. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    From 13:55 to 14:15 UTC, users globally may have experienced errors with failure on Publish, and published messages may not have been received by subscribers. We also experienced errors and failures with PubNub Functions, and delays with Events & Actions during the period....

  2. postmortem

    ### **Problem Description, Impact, and Resolution** At approximately **13:55 UTC on Aug 14, 2026**, we observed elevated publish errors and message replication failures in our publish/subscribe service, which also caused latency in other PubNub services globally. Customers may...

Excerpts are shortened. Read the full incident on PubNub's status page.

Replication failures none 2026-06-10 20:14 UTC 31m

Affected: Publish/Subscribe Service, North America Points of Presence

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    Starting at 17:00 UTC on June 10, a subset of publishes originating from the North America POP failed to replicate to subscribers globally. PubNub Technical Staff is investigating, and more information will be posted as it becomes available.

  2. monitoring

    The PubNub Technical Staff identified that failures are limited to publishes from US-East and has identified, have applied a fix, and are now monitoring the results. If you are experiencing issues that you believe to be related to this incident, please report the details to...

  3. resolved

    Services have returned to normal. A root cause analysis will be published in the coming days. If you believe you were impacted and would like to speak with us, please report impact to support@pubnub.com.

  4. postmortem

    ## Problem Description, Impact, and Resolution At 19:50 UTC on June 10, 2026, we observed a small fraction of publishes originating from US-EAST-1 failing to replicate to subscribers globally. We removed the degraded publisher pod from service and the issue was resolved at 21:21...

Excerpts are shortened. Read the full incident on PubNub's status page.

Global Errors and Failures with Publish and Functions, Delays with Events & Actions none 2026-06-09 14:00 UTC 0m

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    From 13:20 to 13:48 UTC, users globally may have experienced errors with failure on Publish, and published messages may not have been received by subscribers. We also experienced errors and failures with PubNub Functions, and delays with Events & Actions during the period....

  2. postmortem

    ### **Problem Description, Impact, and Resolution** At approximately **13:20 UTC on June 9, 2026**, we observed elevated publish errors and message replication failures in our publish/subscribe service, which also caused latency in other PubNub services globally. Customers may...

Excerpts are shortened. Read the full incident on PubNub's status page.

Connectivity Issues Affecting a Subset of Subscriptions none 2026-03-24 20:21 UTC 52m

Affected: Publish/Subscribe Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    As of 19:27 UTC, we are investigating reports of connectivity issues affecting a subset of subscriptions across several global regions. While the majority of the PubNub network is operating normally, users on the affected segment may experience: - Connection delays and increased...

  2. resolved

    The incident has been resolved. If you believe you were impacted by the incident and wish to discuss it with our team, please contact us by email at support@pubnub.com

  3. postmortem

    ### **Problem Description, Impact, and Resolution** On March 24, 2026, at 19:27 UTC, one network shard experienced intermittent connectivity affecting a subset of customers. The affected users may have experienced elevated latency and temporary error responses related to their...

Excerpts are shortened. Read the full incident on PubNub's status page.

Delay in Publishing Messages to Storage Globally minor 2026-01-01 00:25 UTC 35m

Affected: North America Points of Presence, Storage and Playback Service, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Starting at 12:13AM GMT, we began seeing delays in published messages being written to history storage. Users may be unable to access recently stored messages through History API calls. The messages will eventually be published. Our engineers are actively working to address the...

  2. monitoring

    A fix has been applied and we are monitoring the results. Published messages waiting on queue to be written to history have now been successfully written. We will continue to monitor for the next 15 minutes.

  3. resolved

    The incident has been resolved. Published messages continue to be written to history storage normally. No messages were lost during the incident. If you believe you were impacted by the incident and wish to discuss it with our team, please contact us by email at...

  4. postmortem

    ### **Problem Description, Impact, and Resolution** On January 1, 2026 at 00:00 UTC, we observed elevated latency in our History service across multiple regions. Customers may have experienced delays in message persistence and history availability during this period. The issue...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors observed and resolved minor 2025-11-20 16:30 UTC 0m

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. resolved

    Increased errors (5XX) observed On Nov. 20, 2025 between 16:36 UTC and 16:48 UTC, publish and replication errors (5xx) were observed in North American and Asia Pacific regions, and immediately resolved. Please contact support@pubnub.com if you believe you were impacted by the...

  2. postmortem

    **Problem Description, Impact, and Resolution** Starting at 16:46 UTC on Nov. 20, 2025, we noticed a small number of errors with the publish API in the North American and Asia Pacific regions. The system automatically recovered with all functionality fully restored by 16:50 UTC...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased latency and errors observed in US-West minor 2025-10-20 13:37 UTC 10h 11m

Affected: Publish/Subscribe Service, North America Points of Presence, Presence Service, Access Manager Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    Beginning at 12:40 UTC, we observed increased latency and errors in our US-West region for multiple services. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. investigating

    We are continuing to investigate this issue.

  3. investigating

    Apologies, but we are continuing to investigate this issue.

  4. identified

    The issue has been identified and a fix is being implemented.

  5. identified

    We are still working with our infrastructure provider to mitigate the effects of the ongoing issues. We have been monitoring and handling traffic to minimize disruptions. While points of presence in the US may see slightly higher latencies, we are stable. Please reach out to...

  6. identified

    We are still stable, but seeing minor issues. We are still working with our infrastructure provider to address the remaining issues. Please reach out to PubNub support with any issues.

  7. monitoring

    PubNub remains stable and fully operational. Our infrastructure provider is continuing to apply mitigation steps to restore normal health across their network and related services. We are closely monitoring the situation and coordinating with them to ensure continued stability....

  8. identified

    The issue has been identified and a fix is being implemented.

  9. identified

    PubNub continues to remains stable, with only minor issues being observed. Our infrastructure provider is making progress in restoring their services, with early signs of recovery in several regions. We continue to work closely with them as they complete mitigations across the...

  10. identified

    PubNub remains stable, with only minor issues being observed. Our infrastructure provider is making progress in restoring their services, with early signs of recovery in several regions. We continue to work closely with them as they complete mitigations across the remaining...

  11. identified

    Our status is the same: stable, and working with our infrastructure provider to restore services to their normal state. Our normal policy is to update every 30 minutes, but we will stop issuing this status; we will update this page when a change to the status occurs. As always,...

  12. monitoring

    PubNub remains stable, and our infrastructure provider continues to make steady progress in restoring services. System performance and reliability are improving, and functionality across affected components is returning to normal levels. We continue to monitor closely and will...

  13. monitoring

    PubNub remains stable, and our infrastructure provider has restored normal service capacity. Systems are now operating at pre-event levels, and most dependent services are performing as expected. We continue to monitor closely and will share further updates as needed. Please...

  14. resolved

    This incident has been fully resolved. Our infrastructure provider has restored normal operations, and all systems are functioning as expected. Service performance and reliability have returned to pre-event levels. PubNub continues to monitor systems closely to ensure ongoing...

  15. postmortem

    ### **Problem Description, Impact, and Resolution** On October 20th, 2025 at 07:06 UTC, our monitoring systems alerted us to elevated error levels across multiple PubNub services in the IAD region \(US-East\). Some customers may have experienced increased error rates and...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated latencies and errors for multiple services in US-west and US East minor 2025-10-20 07:32 UTC 1h 47m

Affected: Publish/Subscribe Service, North America Points of Presence, Storage and Playback Service, Stream Controller Service, Presence Service, Access Manager Service, DNS Service, Mobile Push Gateway, App Context Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    At approximately 07:02 UTC, PubNub services began experiencing elevated latencies and server errors in the US-West and US-East regions. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    With no further issues observed for the past 30 minutes, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** On October 20th, 2025 at 07:06 UTC, our monitoring systems alerted us to elevated error levels across multiple PubNub services in the IAD region \(US-East\). Some customers may have experienced increased error rates and...

Excerpts are shortened. Read the full incident on PubNub's status page.

Potential for some missed messages for subscribers in IAD major 2025-10-17 06:42 UTC 1h 24m

Affected: Publish/Subscribe Service, North America Points of Presence

Failure mode: A code release introduced the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We are currently investigating an incident that could lead to some missed messages by subscribers in the IAD region. All messages are being received and persisted, and can be retrieved from the Storage service. This incident started around 22:07 UTC (03:07 PDT) on 16th Oct 2025....

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    There have been no further issues for the past 45 minutes. We are resolving this issue, and we will follow up with a post-mortem soon.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** On October 17, 2025 at 04:51 UTC, some customers may have experienced elevated latency and error rates with the Pub/Sub service in the IAD region \(US-East\). Our engineering teams began immediate investigation and identified a...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated Event & Action Error In FRA Region minor 2025-10-06 10:13 UTC 47m

Affected: European Points of Presence

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    At about 08:40 UTC, the Event & Action publish operation began to experience elevated error rates. Our Technical Staff is actively investigating, and more information will be posted as it becomes available. If you are experiencing issues that you believe are related to this...

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** At 08:40 UTC on October 6, 2025, we observed elevated error rates in the Events & Actions service in our EU-Central \(FRA\) region, which led to delays in processing publish-triggered events. Some customers may have experienced...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased Error Rate and Latency for Presence none 2025-09-07 18:45 UTC 18m

Affected: Presence Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. monitoring

    We experienced increased latencies and error rates in our Tokyo, San Jose, and Virginia points-of-presence from 18:14 - 18:17 UTC. A fix was deployed and we are monitoring the situation.

  2. resolved

    The issues affected the Presence service from 18:14 - 18:17 UTC. We will follow up with a post-mortem soon. We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact...

  3. postmortem

    ### **Problem Description, Impact, and Resolution** At 18:14 UTC on September 7, 2025 we observed increased error rates and latency for our Presence service in our San Jose, Virginia, and Tokyo regions. We increased capacity in those regions and the issue was resolved at 18:17...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence is experiencing elevated latencies and error rates none 2025-09-02 18:34 UTC 16m

Affected: North America Points of Presence, Asia Pacific Points of Presence, Presence Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. monitoring

    At about 11:09, the Presence began to experience elevated latencies and error rates. PubNub Technical Staff is investigating and more information will be posted as it becomes available. If you are experiencing issues that you believe to be related to this incident, please report...

  2. resolved

    The issues affected the Presence service from 18:08 to 18:16 UTC. We will follow up with a post-mortem soon. We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact...

  3. postmortem

    ### **Problem Description, Impact, and Resolution** At 18:09 UTC on September 2, 2025 we observed increased error rates and latency for our Presence service in our San Jose, Virginia, and Tokyo regions. We increased capacity in those regions and the issue was resolved at 18:16...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence service errors in multiple regions minor 2025-07-31 22:14 UTC 59m

Affected: North America Points of Presence, Asia Pacific Points of Presence, Presence Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We have detected elevated error levels with the Presence service in multiple regions. Our Engineers are actively working to mitigate the issues and return service to normal levels. We will provide updates here. If you believe you have been impacted by the issue, please report...

  2. identified

    The issue has been identified and a fix is being implemented. We will provide updates here as progress is made.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** On August 1, 2025 at 22:15 UTC, we observed elevated 5xx errors across the Presence service in multiple regions. Customers may have experienced intermittent failures when attempting to receive or update presence messages. We...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated Presence Latency and Errors in FRA none 2025-07-24 15:30 UTC 0m

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    Between 15:33-15:59 UTC we observed elevated levels of latency and errors with our presence service in our FRA region. The issue has been resolved and remains stable. We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub...

  2. postmortem

    ### **Problem Description, Impact, and Resolution** At 14:43 UTC on July 22, 2025, we observed elevated latency and service degradation in our Presence system in the FRA region, impacting customers' ability to receive timely presence events. We implemented changes in our...

Excerpts are shortened. Read the full incident on PubNub's status page.

Intermittent Presence Latency Spikes in FRA minor 2025-07-22 18:18 UTC 0m

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    Between 14:43-15:06 UTC we observed intermittent spikes in errors and latency with our presence service in our FRA region. The issue has been resolved and remains stable.

  2. postmortem

    ### **Problem Description, Impact, and Resolution** At 14:43 UTC on July 22, 2025, we observed elevated latency and service degradation in our Presence system in the FRA region, impacting customers' ability to receive timely presence events. We implemented changes in our...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased Latency and Errors in Presence in the US (East and West) and AP-North regions minor 2025-07-08 17:36 UTC 40m

Affected: North America Points of Presence, Asia Pacific Points of Presence, Presence Service

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    At approximately 17:18 UTC, PubNub Presence services began experiencing elevated latencies and server errors in the Asia Pacific-North and US regions. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. investigating

    We are continuing to investigate this issue.

  3. monitoring

    Services have returned to normal and we will continue monitoring for the next 30 minutes.

  4. resolved

    With no further issues observed for the past 30 minutes, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** On July 8, 2025 at 17:15 UTC on, we observed elevated errors and increased latency for customers using our Presence service in the US East, US West, and Tokyo regions. We identified an abnormal concentration of traffic and...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in FRA region minor 2025-06-30 13:39 UTC 1h 22m

Affected: North America Points of Presence, European Points of Presence

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    At approximately 13:11 UTC, PubNub services began experiencing elevated latencies and server errors in the Europe & North US region. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  5. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in FRA region minor 2025-06-30 08:12 UTC 1h 19m

Affected: European Points of Presence

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    At approximately 07:30 UTC, PubNub services began experiencing elevated latencies and server errors in the Europe region. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. monitoring

    A fix has been implemented and we are monitoring the results.

  3. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  4. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased global errors and latency for all services none 2025-06-29 17:15 UTC 0m

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    On June 29, 2025 around 16:00 UTC we began seeing elevated latency and increased errors globally across all services. Our Engineers scaled-up resources and PubNub services returned to normal levels by 16:28 UTC.

  2. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in IAD region minor 2025-06-27 15:52 UTC 0m

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    At approximately 14:02 - 14:09 UTC, PubNub services began experiencing elevated latencies and server errors in the Europe region. PubNub Technical Staff is investigated and the issue has resolved

  2. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in IAD region minor 2025-06-27 11:00 UTC 0m

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    At approximately 11:10 - 11:20 UTC, PubNub services began experiencing elevated latencies and server errors in the Virginia region. PubNub Technical Staff investigated, and the issues is no longer occuring.

  2. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in FRA region minor 2025-06-27 09:21 UTC 1h 16m

Affected: European Points of Presence

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    At approximately 08:15 UTC, PubNub services began experiencing elevated latencies and server errors in the Europe region. PubNub Technical Staff is currently investigating, and more updates will follow once available.

  2. investigating

    The PubNub Technical Staff continues to investigate. More updates will follow once available.

  3. identified

    The issue has been identified and a fix is being implemented.

  4. monitoring

    A fix has been implemented and we are monitoring the results.

  5. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced an impact related to this incident, please report it to PubNub Support at support@pubnub.com.

  6. postmortem

    Beginning on Friday, June 27, 2025 at 08:15 UTC, there were occasional, intermittent increases in latency and errors in three of our services: Pub/Sub, History, and Presence. The root cause discussed in this analysis was identified and corrected on Monday, June 30. ### **Problem...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated latency and errors for History service in US-East none 2025-06-10 18:30 UTC 0m

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    Beginning at 18:33 UTC we observed increased latency and errors for our History service in one of our North America regions. Users in the affected region may have experienced failures with error or latency when making calls to the History endpoint. The issue has been resolved as...

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On June 10, 2025 at 18:33 UTC, we observed elevated error rates and increased latency for a subset of traffic for the History service in the US East region. Customers using the History API in this region may have encountered...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency for presence in FRA and BOM minor 2025-06-02 14:43 UTC 0m

Affected: European Points of Presence, Presence Service, Southern Asia Points of Presence

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    Starting at 12:20 UTC, we noticed an increase in errors and latency affecting the Presence service in the BOM and FRA regions. Our engineers investigated the issue and successfully restored the service completely, which has remained stable since 13:51 UTC.

  2. postmortem

    ### Problem Description, Impact, and Resolution Starting at 12:20 pm UTC on June 2, 2025, we noticed an increase in errors and latency affecting the Presence service in multiple regions. After investigating the increase in errors and latency, we identified the cause to be a...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency in US East none 2025-05-22 21:34 UTC 0m

Affected: Publish/Subscribe Service, North America Points of Presence, Storage and Playback Service, Stream Controller Service, Presence Service, Access Manager Service, Mobile Push Gateway

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    Beginning at 20:51 UTC, we detected increased errors and latency for traffic within a single availability zone in the US East region. Our engineers investigated the issue and successfully restored service, which has remained stable since 21:04 UTC.

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On May 23, 2025 at 20:51 UTC, PubNub experienced increased errors and latency for a subset of traffic within a single availability zone in the US East region. The root cause was an operational error during an infrastructure...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence API errors & latency in US and AP North minor 2025-05-16 09:59 UTC 24m

Affected: North America Points of Presence, Asia Pacific Points of Presence, Presence Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. monitoring

    At 09:05 UTC we detected increased errors and latency for the Presence service in US and AP-North regions. The service has stablized as of 9:30 UTC. Our engineers are monitoring the system.

  2. resolved

    This incident has been resolved.

  3. postmortem

    ### **Problem Description, Impact, and Resolution** On May 16, 2025 at 09:05 AM UTC, some customers in US and APAC regions experienced elevated error rates and increased latency when using the Presence service. This disruption was caused by an interruption of underlying...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increase Latency and Errors in Presence in the US (East and West) and AP-North regions minor 2025-05-13 00:00 UTC 43m

Affected: Presence Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    Beginning at 23:22 UTC we detected increased errors and latency for the Presence service in US (East and West) and AP-North regions. Our engineers are investigating the issue and the service has stablized as of 23:52 UTC.

  2. monitoring

    The issue has resolved and we are monitoring the system for stability.

  3. resolved

    This incident has been resolved.

  4. postmortem

    ## **Problem Description, Impact, and Resolution** On May 12, 2025 at 23:22 UTC, we observed elevated errors and increased latency for customers using our Presence service in the Virginia \(US-East\), California \(US-West\), and Tokyo \(AP-North\) regions. After discovering the...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased errors and latency for presence in FRA and BOM minor 2025-05-02 15:23 UTC 0m

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    Beginning at 15:40 UTC we detected increased errors and latency for the Presence service in the BOM and FRA regions. Our engineers investigated the issue and were able to restore the service which remains stable as of 15:52 UTC.

  2. postmortem

    ## **Problem Description, Impact, and Resolution** On May 2, 2025 at 15:40 UTC, we observed elevated errors and increased latency for customers using our Presence service in the Frankfurt \(FRA\) and Mumbai \(BOM\) regions. After observing the errors and latency, we increased...

Excerpts are shortened. Read the full incident on PubNub's status page.

Server Errors and Elevated Latency minor 2025-04-25 00:51 UTC 1h 41m

Affected: Publish/Subscribe Service, North America Points of Presence, Asia Pacific Points of Presence, Presence Service, Access Manager Service, Mobile Push Gateway

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    At about 12:17 AM UTC, we started to experience elevated latencies and server errors in all PoPs. PubNub Technical Staff is currently investigating and more updates will follow once available. If you are experiencing issues and believe them to be related to this incident, please...

  2. investigating

    The PubNub Technical Staff is investigating. More updates will follow once available.

  3. investigating

    The PubNub Technical Staff continues to investigate. Errors and elevated latency are with some Presence customers. The real-time network services are operational.

  4. identified

    The issue has been identified and a fix is being implemented.

  5. monitoring

    A fix has been implemented and we are monitoring the results.

  6. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced impact related to this incident, please report them to PubNub Support at support@pubnub.com.

  7. postmortem

    ### Problem Description, Impact, and Resolution At approximately 00:17 UTC on April 25, 2025, we observed elevated server errors and increased latency impacting multiple API endpoints, most notably the Presence service. Our engineering team immediately began investigating the...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence Server Errors and Elevated Latency minor 2025-03-10 20:25 UTC 1h 12m

Affected: North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Presence Service, Southern Asia Points of Presence

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    At about 6:26 PM UTC, Presence service started to experience elevated latencies and server errors in all PoPs. PubNub Technical Staff is currently investigating and more updates will follow once available. If you are experiencing issues and believe them to be related to this...

  2. monitoring

    We have identified the issue and have taken effective remediation actions, and our engineers are diligently monitoring the situation to guarantee that stability is fully restored.

  3. monitoring

    We continue to monitor the situation to guarantee that stability is fully restored.

  4. resolved

    With no further issues observed, the incident has been resolved. We will follow up soon with a root cause analysis. If you believe you experienced impact related to this incident, please report them to PubNub Support at support@pubnub.com.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** On March 10, 2025, we observed elevated latency and errors in the Presence service across our global points of presence from 18:26 to 18:40 UTC and again from 20:05 to 20:27 UTC. After investigating, we found that a short spike...

Excerpts are shortened. Read the full incident on PubNub's status page.

Errors and Latency Across All Services in our US-East and US-West Points of Presence minor 2024-12-11 22:04 UTC 18m

Affected: Publish/Subscribe Service, Website, Functions Service, North America Points of Presence, Storage and Playback Service, Administration Portal, Vault, Stream Controller Service, SDK Documentation, Key Value store, Presence Service, PubNub Support Portal, Scheduler Service, Access Manager Service, DNS Service, Mobile Push Gateway, App Context Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We are currently investigating an issue that is causing requests in our US-East points-of-presence to fail or respond slowly.

  2. monitoring

    A fix has been implemented and we are monitoring the results.

  3. resolved

    This incident has been resolved. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching out to PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service. An RCA will be provided soon.

  4. postmortem

    ### **Problem Description, Impact, and Resolution** At 21:50 UTC on 2024-12-11 we observed increased latencies and error rates across all services in our US-East point-of-presence and, a few minutes later, in US-West as well. We observed that the PubNub Access Manager \(PAM\)...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated History Error/Latency in US West Region minor 2024-12-09 08:42 UTC 1h 18m

Affected: North America Points of Presence, Storage and Playback Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Around 07:23 UTC we began to notice increasing errors and latency for History in SJC region(US West).

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix has been implemented and we are monitoring the results.

  4. resolved

    This incident has been resolved with no errors observed for the last 30 minutes. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching out to PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service....

  5. postmortem

    ### **Problem Description, Impact, and Resolution** At 07:23 UTC on December 9th, we received alerts indicating high error levels related to storage writer operations in one of our data centers. Shortly after, one of our third-party service providers reported service disruption...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence Server Errors and Elevated Latency none 2024-10-24 12:01 UTC 2h 57m

Affected: Presence Service

Failure mode: Traffic surge or denial-of-service attack. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    At about 11:00 AM UTC, Presence service started to experience elevated latencies and server errors in all PoPs. PubNub Technical Staff is currently investigating and more updates will follow once available. If you are experiencing issues and believe them to be related to this...

  2. investigating

    We are continuing to investigate this issue.

  3. investigating

    We are continuing to investigate this issue.

  4. investigating

    We are continuing to investigate this issue.

  5. identified

    We have successfully identified the issue, and our dedicated engineers are actively working to resolve it. We are seeing positive trends, with both latency and error rates improving significantly.

  6. monitoring

    We have taken effective remediation actions, and our engineers are diligently monitoring the situation to guarantee that stability is fully restored.

  7. resolved

    Beginning at around 11:00 UTC we observed elevated latency and server errors for our Presence service in all of our server endpoints. The issue has been resolved as of 14:11 UTC. We will continue to monitor the incident to ensure service stability has been fully restored. Your...

  8. postmortem

    ### **Problem Description, Impact, and Resolution** On October 24, 2024 at 11:15AM UTC, we observed elevated latency and errors in the Presence service across our global points of presence. Affected customers may have experienced a slowdown in Presence request responses and/or...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated latency and errors for history service in us-east-1 region minor 2024-10-05 11:16 UTC 2h 51m

Affected: North America Points of Presence, Storage and Playback Service

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    At about 8:00 AM UTC, History service started to experience elevated latencies and errors in North America PoP. PubNub Technical Staff is currently investigating and more updates will follow once available. If you are experiencing issues and believe them to be related to this...

  2. identified

    The issue has been identified and our engineers are engaged and continue to work on the issue. Latency and errors rates are improving.

  3. identified

    PubNub Technical Staff still working on fixing the issue.

  4. identified

    We are continuing to work on a fix for this issue.

  5. monitoring

    Remediation actions have been taken. Our engineers are currently monitoring the incident to ensure the stability has been restored.

  6. resolved

    Beginning at around 8:00 UTC we observed increased latency and errors for our History service in one of our North America regions. The issue has been resolved as of 14:05 UTC. We will continue to monitor the incident to ensure service stability has been fully restored.

  7. postmortem

    ### **Problem Description, Impact, and Resolution** At 7:35 UTC on October 5, 2024, we received a report of intermittent failures \(5xx errors\) for History API requests. The issue was triggered by an unexpectedly high volume of data requests processed through our shared...

Excerpts are shortened. Read the full incident on PubNub's status page.

Presence Latency minor 2024-08-22 22:32 UTC 0m

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    Beginning at 22:42 UTC some customers may have observed increased latency and errors for our Presence service in our Frankfurt and Mumbai regions. The issue has been resolved as of 22:50 UTC. We will continue to monitor the incident to ensure service stability has been fully...

  2. postmortem

    ### **Problem Description, Impact, and Resolution** At 22:42 UTC on August 22, 2024, we observed increased latency and errors for our Presence service. We found evidence of network issues from our own testing and monitoring, and we created a ticket with our network service...

Excerpts are shortened. Read the full incident on PubNub's status page.

Push Notifications Latency in Multiple PoPs major 2024-06-13 18:51 UTC 3h 2m

Affected: North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence, Mobile Push Gateway

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We have discovered an issue where push notifications are being delivered latently in our Frankfurt point-of-presence since the last 30 minutes. Our Engineering teams are actively investigating the issue and we will provide updates here. If you believe you have experienced...

  2. investigating

    We have discovered that push notifications in our Mumbai point-of-presence are also affected, with push notifications being delivered latently. Push notifications in our Frankfurt point-of-presence are also being delivered latently. We continue to investigate and will provide...

  3. investigating

    We are continuing to investigate this issue.

  4. investigating

    We have discovered that push notifications in our US-East point-of-presence are also affected, with push notifications being delivered latently. We continue to investigate and will provide updates here.

  5. investigating

    We are continuing to investigate this issue.

  6. investigating

    We are continuing to investigate this issue.

  7. investigating

    Our Engineering teams continue actively investigating the issue. We will continue to provide updates here.

  8. investigating

    Our investigation continues and we will continue to provide updates here.

  9. investigating

    We continue investigating delayed push notifications in our Frankfurt point-of-presence. Push notifications are now being delivered normally in our other regions. We will continue to provide updates here.

  10. investigating

    We are continuing to investigate this issue.

  11. monitoring

    Push notifications are now being delivered normally. We are monitoring the system to ensure no further issues.

  12. resolved

    This incident has been resolved, and mobile push notifications continue to be delivered normally. We will follow up with a root cause analysis soon. We sincerely apologize for any impact on our customers and their users. If you believe you have experienced production impact due...

  13. postmortem

    **Problem Description, Impact, and Resolution** At 18:16 UTC on June 13, 2024 we observed increased latency for delivery of mobile push messages in our Frankfurt and US-East points of presence. In response, we increased the resources available to the services and redeployed the...

Excerpts are shortened. Read the full incident on PubNub's status page.

Delayed Push Notifications in Frankfurt PoP none 2024-06-07 19:25 UTC 1h 10m

Affected: Mobile Push Gateway

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We have discovered an issue where push notifications were not being delivered in our Frankfurt point-of-presence. Those notifications were then delivered about twenty minutes late. We are investigating the issue.

  2. investigating

    We are still experiencing some delayed delivery of push messages in our Frankfurt point-of-presence. We continue to investigate the issue.

  3. monitoring

    We have processed all push messages in our backlog and stabilized the system in Frankfurt. We are monitoring the system to ensure no further issues.

  4. resolved

    This incident has been resolved, and mobile push notifications are being delivered normally. We will follow up with a root cause analysis.

  5. postmortem

    At 18:53 UTC on June 7, 2024, we observed excessively latent deliveries of mobile push messages in our Frankfurt point-of-presence. We discovered that a previously undetected bug was being triggered by malformed messages being sent to the service. We increased the resources...

Excerpts are shortened. Read the full incident on PubNub's status page.

Increased latency and errors across all services in US East none 2024-05-12 15:41 UTC 0m

Affected: Publish/Subscribe Service, Storage and Playback Service, Stream Controller Service, Presence Service, Access Manager Service, DNS Service, Mobile Push Gateway, App Context Service

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. resolved

    From 14:16 UTC to 15:20 UTC on May 12, 2024, users may have experienced increased latency and errors across all PubNub services in our US East PoP. Our Engineering teams applied a fix and the issue has been resolved since 15:20 UTC. A root cause analysis will be posted soon.

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On Sunday, May 12, 2024 at 14:16 UTC, customers using PubNub’s legacy Access Manager Version 2 may have experienced increased errors and latency across all services in our US-East point of presence. After investigation, we...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated History Error/Latency in Tokyo Region none 2024-03-17 00:43 UTC 41m

Affected: Storage and Playback Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Around 00:06 UTC we began to notice increasing errors and latency for History in Tokyo region. We are investigating this incident.

  2. monitoring

    The elevated state of History errors and latency has returned to normal. We will continue to monitor the incident

  3. monitoring

    We have not seen any errors or increased latency for ~45 minutes. We will continue to monitor history to validate the resolution.

  4. resolved

    This incident has been resolved.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** At 00:06 UTC on March 17, 2024, we observed increased error rates and latency in our Tokyo region for History calls. We then identified the source of latency and errors were due to our third-party provider for storage. We...

Excerpts are shortened. Read the full incident on PubNub's status page.

Failures for Presence Webhooks minor 2024-03-02 01:26 UTC 1h 13m

Affected: North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Presence Service, Southern Asia Points of Presence

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    `Around 14:45 UTC (06:45 PT) March 01, the Presence service began to experience missed Presence webhook calls for some users. PubNub Technical Staff is investigating and more information will be posted as it becomes available. If you are experiencing issues that you believe to...

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    A fix was implemented at 01:45 UTC. We are monitoring the results for the next 30 minutes.

  4. resolved

    This incident has been resolved with no errors observed for the last 30 minutes. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service. An RCA...

  5. postmortem

    ### **Problem Description, Impact, and Resolution** Around 14:45 UTC, March 1, 2024, we observed the Presence service started experiencing missed Presence webhook calls. The missed webhook calls were due to a migration of the Presence webhooks across multiple customers. We...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated Subscribe Latency and Errors in US-East none 2023-10-30 18:11 UTC 48m

Affected: Publish/Subscribe Service, North America Points of Presence

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Starting around 13:45 UTC, we began observing elevated latency and errors in calls made to our Subscribe endpoint in our US East point of presence. PubNub Technical Staff is investigating, and more information will be posted as it becomes available. We apologize for any impact...

  2. identified

    The issue has been identified and a fix is being implemented. We will continue to provide regular updates.

  3. resolved

    This incident has been resolved with no errors observed for the last 30 minutes. We apologize for any impact this may have had on your service. Don't hesitate to contact us by reaching PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service. An RCA...

  4. postmortem

    ### **Problem Description, Impact, and Resolution** On October 30, 2023 at 17:52 UTC, we observed a high number of errors and increased latency in our Subscribe service, mostly in our US-East point of presence. This was a result of increased Subscribe traffic in the region...

Excerpts are shortened. Read the full incident on PubNub's status page.

Proactive Reconfiguration of PubNub Services none 2023-09-28 21:18 UTC 1h 24m

Affected: Publish/Subscribe Service, Website, Functions Service, North America Points of Presence, Storage and Playback Service, Administration Portal, Vault, Stream Controller Service, SDK Documentation, Key Value store, Presence Service, PubNub Support Portal, Scheduler Service, Access Manager Service, DNS Service, Mobile Push Gateway, App Context Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We are proactively reconfiguring our services, based on advice from an infrastructure provider that is experiencing issues in our US-East point-of-presence. There is no customer impact at this time, and we will update this page if we see any impact during this process. Please...

  2. identified

    The issue has been identified and a fix is being implemented.

  3. monitoring

    We have completed our changes, with no customer impact reported. We will continue to monitor the situation closely.

  4. resolved

    We've had no customer impact from this configuration change, and are closing this incident. As always, contact PubNub support with any questions.

  5. postmortem

    We received a notification from an infrastructure provider that they were having issues that could affect our US-East Point of Presence. On that recommendation, we shifted our infrastructure to avoid any potential problems. We posted this incident in advance of shifting, in...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated Channel Groups Subscribe latency in the US-West PoP minor 2023-08-12 01:05 UTC 1h 50m

Affected: Publish/Subscribe Service, North America Points of Presence

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    On August 11 at 23:45 UTC, we began to observe increased latency for channel group subscribes in our US-West PoP, which could result in delays in receiving messages. PubNub Technical Staff is investigating, and more information will be posted as it becomes available. We...

  2. identified

    We believe the issue has been identified and a fix is being implemented. We will provide updates as they become available.

  3. monitoring

    This issue has been resolved and latency has returned to normal levels. We will continue to monitor services for the next 30-60 minutes. We will continue to provide updates here.

  4. monitoring

    We are continuing to monitor for any further issues for the next 30-60 minutes. We will continue to provide updates here.

  5. resolved

    This incident has been resolved. The incident was declared with an overabundance of caution, and was determined to be limited to a handful of customers. Please contact PubNub Support (support@pubnub.com) if you wish to discuss the incident.

  6. postmortem

    ### **Problem Description, Impact, and Resolution** On Friday, August 11th at 23:45 UTC, we observed a delay in message delivery for subscribe requests using our Channel Groups service. After identifying the delay, we restarted the affected pods, and the issue was resolved at...

Excerpts are shortened. Read the full incident on PubNub's status page.

Some Services Across US East Are Suffering From Partial Outage major 2023-06-13 19:30 UTC 3h 28m

Affected: North America Points of Presence, Access Manager Service, App Context Service

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. identified

    This is particularly effecting Authorization and downstream Objects.

  2. identified

    We have taken steps to mitigate this issue. Error rates and latency have be reduced.

  3. monitoring

    We are continuing to monitor the changes that we have implemented to mitigate this incident.

  4. resolved

    This incident has been resolved, all services fully restored.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** At 19:20 UTC on Tuesday June 13th, 2023 we observed increased error rates and latency for our Authorization services at our US East facility. In response, we redirected authorization services from US East to US West, and the...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated latencies in the EU PoP Subscribe Service none 2023-06-01 15:18 UTC 0m

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    On June 1, 2023 between 08:18 and 09:06 UTC, customers using Subscribe service in our EU Central PoP may have experienced some errors, timeouts, and delays in message delivery. We will publish a root cause analysis soon.

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On June 1, 2023, we observed two subscribe latency spikes in our Europe Point of Presence. The spikes occurred from 08:18 AM UTC through 09:06 AM UTC, and from 09:44 AM UTC through 09:54 AM UTC. During these times, users in the...

Excerpts are shortened. Read the full incident on PubNub's status page.

Subscribers experiencing errors in all regions none 2023-05-18 16:48 UTC 1h 36m

Affected: Publish/Subscribe Service, North America Points of Presence, European Points of Presence, Asia Pacific Points of Presence, Southern Asia Points of Presence

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Subscribers were experiencing sporadic errors on May 18 between 2:35 PM and 4:11 PM UTC. We are investigating the cause.

  2. identified

    We have identified the issue, and are still investigating further. All systems are operational.

  3. monitoring

    We have been operating with all systems normal for more than an hour. We are monitoring the situation at this point and investigating the root cause.

  4. resolved

    We are resolving this issue, and we will follow up with a post-mortem soon. We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss the impact on your service.

  5. postmortem

    ### **Problem Description, Impact, and Resolution** At 14:35 UTC on May 18, 2023 we observed some errors being served to subscribers globally. We noted a large, unusual traffic pattern that was putting memory pressure on parts of our infrastructure faster than our normal...

Excerpts are shortened. Read the full incident on PubNub's status page.

Events and Actions Outage in US East PoP. Events are not Being Dispatched. minor 2023-05-02 22:01 UTC 52m

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    Events and Actions messages are not currently being dispatched for customers in our US-East PoP.

  2. monitoring

    The Events and Actions service in US-East has been restored and any messages that would trigger actions have now been processed.

  3. resolved

    This service incident has been resolved since 22:00 UTC

  4. postmortem

    ### **Problem Description, Impact, and Resolution** On Tuesday May 2, 2023 at 21:23 UTC, we observed that Events and Actions messages were not being processed on our US PoP or EU Central PoP. Shortly after observing the issue, we redeployed a processing schema, and the issue was...

Excerpts are shortened. Read the full incident on PubNub's status page.

Elevated rate of latency in US East Point of Presence none 2023-05-02 14:30 UTC 0m

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. resolved

    On Tuesday, May 2, 2023 between 14:20 UTC and 14:28 UTC, customers in our US-East PoP may have experienced some errors, timeouts, or delays across multiple services. RCA will be provided soon.

  2. postmortem

    ### **Problem Description, Impact, and Resolution** On Tuesday May 2, 2023 at 14:26 UTC observed errors and latency on all services in US East PoP. We modified the configuration to scale faster during high load occurrences, and the issue was resolved on May 2, 2023 at 14:34 UTC....

Excerpts are shortened. Read the full incident on PubNub's status page.