LiveKit

Affected: US Central - Real Time Communication, US East - Real Time Communication

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-20 23:09 UTC

   We are currently investigating higher than normal join latencies affecting a small percentage of participants in our US East and US Central regions. The latency increase appears to have begun at 21:30 UTC. Affected participants may notice increased join latencies higher than 5...

2. monitoring 2026-10-20 23:25 UTC

   We identified connection drops at load balancers in our US Central and US East regions that caused a subset of connection latencies to take 5 seconds or longer. We applied a mitigation at 23:15 UTC, and we are seeing connection speeds return to normal. We are continuing to...

3. resolved 2026-10-20 23:50 UTC

   Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US Central - Real Time Communication

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-19 17:02 UTC

   We are currently investigating reports of elevated API errors across our services in the Chicago region.

2. monitoring 2026-10-19 17:11 UTC

   A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.

3. resolved 2026-10-19 17:34 UTC

   Between 16:35-17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. We faced another wave of the same issue between 17:47-17:53 UTC where the failure rates were less than 10%. Room Service APIs...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US West - SIP

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-10-16 16:04 UTC

   We are investigating degraded SIP performance in US West starting at 14:49 UTC affecting new outbound calls.

2. identified 2026-10-16 16:12 UTC

   We have identified the source of the issue and are working to route around the impact.

3. monitoring 2026-10-16 16:35 UTC

   We applied mitigation as of 15:58 UTC and SIP outbound connection success has returned to normal. We are continuing to monitor before marking this resolved.

4. monitoring 2026-10-16 16:57 UTC

   We estimate that approximately 14% of outbound calls and 5% of inbound calls in US West failed between 14:49 UTC and 15:58 UTC. We will follow up as soon as possible with more details.

5. resolved 2026-10-16 17:30 UTC

   This incident has been resolved. Between 14:49 UTC and 15:58 UTC, approximately 14% of outbound calls and 5% of inbound calls in US West failed. We will follow up with more details surrounding the root cause and preventative actions in a postmortem.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Europe Central - Real Time Communication

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-16 10:55 UTC

   We are investigating intermittent timeouts when establishing new connections to our Frankfurt (EU) region. A small percentage of connection attempts may time out. We are working on a mitigation and will post an update as soon as possible.

2. monitoring 2026-10-16 11:15 UTC

   We have applied a mitigation and new connections should no longer time out. We are monitoring to confirm that connection times have returned to normal.

3. resolved 2026-10-16 11:59 UTC

   This incident is resolved. Connections to the Frankfurt region have returned to normal.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-13 10:01 UTC

   We are investigating an issue with Agent Insights where agents are unable to upload their observability data at the end of their sessions. Some recent agent sessions may be missing from the dashboard's sessions list or shown as “In progress”. Agent sessions themselves are not...

2. identified 2026-10-13 10:45 UTC

   We have identified the cause of the Agent Insights issue and are working on a fix.

3. identified 2026-10-13 11:40 UTC

   We are continuing to work on a fix for this issue.

4. identified 2026-10-13 13:15 UTC

   Agent Insights uploads are now operational and session data has been storing normally since 12:40 UTC. We are still investigating an issue with Simulations. Simulation runs may complete without a transcript, summary, or verdict. We will post an update shortly.

5. resolved 2026-10-13 13:44 UTC

   This incident has been resolved. Between 08:15 and 12:35 UTC, agents were not able to upload agent observability data to LiveKit's servers in all regions. Agent sessions completed normally, but, unfortunately, any Agent Insights data which attempted to upload within that window...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US East - Cloud Agents

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-10-06 03:46 UTC

   We are investigating an issue causing some cloud agent builds to be backed up in US East. This issue only impacts new builds for a small number of customers, existing agents are running without any impact.

2. resolved 2026-10-06 04:52 UTC

   A fix was applied and all builds are back to normal.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Europe Central - Cloud Agents

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-10-05 13:34 UTC

   We are investigating potential scaling issues with LiveKit Cloud hosted agents in eu-central resulting in increased join latencies. Existing agents are still functional and can serve sessions normally, but they may be slower to join new sessions. We are actively working on a...

2. monitoring 2026-10-05 13:40 UTC

   A fix has been implemented and we are monitoring to confirm full recovery.

3. resolved 2026-10-05 14:21 UTC

   This incident has been resolved, and we have confirmed full recovery.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Egress, Global Ingress, Global SIP

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. monitoring 2026-09-15 22:10 UTC

   From 21:11 to 21:21 UTC we observed an occurrence of the same incident experienced earlier today impacting Global SIP, Ingress, and Egress due to the same underlying issue. We are currently operating normally and are continuing to monitor.

2. resolved 2026-09-15 23:15 UTC

   We have resolved the underlying issue and have not observed any further occurrences of this incident. We will follow up with a detailed postmortem in the original incident.

3. postmortem 2026-09-17 13:53 UTC

   Please see a full postmortem [here](https://status.livekit.io/incidents/dbqb4jhxcg9h).

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Egress, Global Ingress, Global SIP

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-09-15 17:04 UTC

   We are investigating reports of intermittently increased API latencies starting at 13:30 UTC. Users may observe increased latency spikes across all APIs.

2. monitoring 2026-09-15 17:37 UTC

   We are still investigating, but believe that customer impact should be mostly mitigated - although some spikes may still be occurring. We are investigating reports of aborted egresses which we believe were also related to this issue. We will post another update as soon as...

3. monitoring 2026-09-15 18:03 UTC

   We are continuing to monitor, but want to keep users up to date with how we believe impact may have materialized. The first impact window started at 13:40 UTC and impacted Ingress and Egress services for about 3 minutes. The second impact window started at 16:11 UTC and impacted...

4. resolved 2026-09-15 18:16 UTC

   We are resolving this incident as the underlying issue has been resolved. Thank you for your patience and apologies for the disruption. We will follow up with a postmortem as soon as possible.

5. postmortem 2026-09-17 14:26 UTC

   ### Summary On 15 September, LiveKit Cloud experienced three periods of degraded performance between 13:34–13:41, 16:06–16:11 and 21:10–21:22 UTC. Ingress creation was the most affected: more than half of `CreateIngress` requests failed at the worst point. A number of...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io), US West - Analytics Ingestion

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-15 12:08 UTC

   We are investigating an issue where users may be unexpectedly signed out of the LiveKit Cloud dashboard and unable to sign back in.

2. investigating 2026-09-15 12:31 UTC

   We are continuing to investigate this issue affecting LiveKit Cloud dashboard, Analytic Services APIs and Billing APIs. Users may be unexpectedly signed out and unable to sign back in, and API requests may return 522 errors.

3. investigating 2026-09-15 12:49 UTC

   We are continuing to investigate this issue.

4. identified 2026-09-15 13:04 UTC

   We have identified the cause of the issue affecting the Cloud Dashboard and Analytics Services APIs, and recovery work is underway. Customers may continue to see errors accessing the Cloud Dashboard and Analytics Services APIs during this time. Room APIs, RTC APIs, and all other...

5. monitoring 2026-09-15 14:29 UTC

   A fix has been deployed as of 14:23 UTC, and access to the Cloud Dashboard and Analytics Services APIs is returning to normal. We are continuing to monitor before marking this resolved. Recovery may take a few additional minutes for some clients as the change fully propagates;...

6. resolved 2026-09-15 15:30 UTC

   This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs. The cause was a control plane failure in our last remaining legacy Kubernetes cluster, which took out a dependency of those...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Europe Central - Cloud Agents

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-09-05 23:36 UTC

   We have identified an issue with LiveKit Cloud hosted agents in eu-central. Existing agents in eu-central are facing scaling issues resulting in increased join latencies. Existing agents are still functional and can serve sessions normally, but they may be slower to join new...

2. monitoring 2026-09-06 00:24 UTC

   We have applied a mitigation and believe that symptoms should be improving. We will update again with confirmation as soon as possible.

3. resolved 2026-09-06 01:12 UTC

   The mitigation was successful and users shouldn't notice any issues with join latencies or creating new agents. We will follow up as soon as possible with a postmortem.

4. postmortem 2026-09-08 21:59 UTC

   **Root Cause** A configuration change introduced a formatting error in the startup settings for the server pools that run hosted agents in our eu-central region. Newly provisioned servers failed to start, so the region could not add capacity. Agents that were already running...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-09-04 10:00 UTC

   A subset of room sessions that ended around 4 September 09:57 UTC were not recorded with an end time. These sessions continue to appear as active in the Cloud dashboard and in session records, and no `room_ended` end time is reflected for them. This was a reporting issue only....

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global SIP

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-09-02 21:03 UTC

   We are investigating elevated issues in calls connected to LiveKit Phone numbers where callee audio is not being heard by the caller. This is currently only experienced for numbers issues by LiveKit, and no other SIP provider connectivity should be affected. Our team is actively...

2. monitoring 2026-09-02 21:25 UTC

   An upstream carrier reverted a codec change that was causing one-way audio issues on calls to LiveKit-issued phone numbers. As of 21:10 UTC calls are connecting normally and we are no longer seeing the issue. The upstream codec change has impacted calls from T-Mobile and AT&T...

3. resolved 2026-09-02 21:45 UTC

   This incident has been resolved. We will follow up with a detailed postmortem as soon as possible.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-02 08:15 UTC

   We are currently investigating reports of our dashboard failing to load across multiple regions, beginning at 07:00 UTC. Other services remain unaffected and the impact is limited to the dashboard.

2. identified 2026-09-02 09:01 UTC

   We have identified the cause: a problematic node in US-East and are deploying a fix. Impact continues to be limited to the dashboard.

3. monitoring 2026-09-02 10:55 UTC

   A fix has been deployed as of 10:55 UTC, and the dashboard is now loading successfully. We are continuing to monitor before marking this resolved.

4. resolved 2026-09-02 11:57 UTC

   This incident has been resolved and dashboard load times returned to normal by 10:55 UTC. As a note, our beta agent simulations feature was also impacted from approximately 06:07 - 07:52 UTC.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-26 10:44 UTC

   We are investigating a period of elevated error rates and timeouts (08:47:00-08:53:35 UTC) for LiveKit Inference requests in our US East region. Service has since recovered and inference requests are completing normally. We are continuing to investigate the cause and confirm the...

2. resolved 2026-08-26 12:22 UTC

   Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: India - Analytics Ingestion, Europe Central - Analytics Ingestion

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-08-24 10:11 UTC

   We identified an issue where agent session analytics in the Cloud dashboard displayed zero concurrent sessions, beginning late on August 21 (UTC). This was a reporting issue only — agent sessions, calls, and all real-time services in these regions operated normally throughout....

2. monitoring 2026-08-24 13:55 UTC

   We are continuing to monitor while the historical data backfill completes, and will resolve this incident once all analytics for the affected window are fully restored.

3. resolved 2026-08-24 14:14 UTC

   Agent session analytics in the Cloud dashboard have recovered and current data is reporting correctly for all affected projects. Some gaps may remain in historical data from the affected window (August 21–24 UTC); these will be backfilled.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-08-23 21:00 UTC

   From 09:18 UTC on 21 August to 21:00 UTC on 23 August, 0.008% of egress recordings processed in our US East region were terminated before completion. Affected recordings were marked EGRESS_FAILED ("egress timed out"). For file outputs, media was not written to the configured...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Egress

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-20 21:10 UTC

   We are currently investigating reports of elevated API errors across the Egress service. We will post another update in 5 minutes.

2. investigating 2026-08-20 21:24 UTC

   We had a spike of errors between 20:52 and 21:01 UTC, and the errors have subsided to baseline as of 21:01 UTC. We are actively monitoring and will share more updates in 30 minutes.

3. resolved 2026-08-20 21:47 UTC

   This incident has been resolved. Between 20:52 and 21:01 UTC, we observed an increase in failed egress and connector API requests. Errors have returned to baseline as of 21:01 UTC. We will follow up with a postmortem with further details.

4. postmortem 2026-09-10 19:08 UTC

   On August 20, 2026, customers in our US East region experienced approximately 10 minutes of 503 errors on the LiveKit Ingress API and failures launching media track egress, between 20:52 and 21:01 UTC. All other regions were unaffected. ‌ **Root Cause** An internal routing...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-08-19 19:33 UTC

   We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.

2. identified 2026-08-19 21:22 UTC

   Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards. We've identified the source of the slow ingest and are working on a mitigation. We will update again in 30 mins.

3. identified 2026-08-19 21:25 UTC

   We are continuing to work on a fix for this issue.

4. identified 2026-08-19 22:07 UTC

   The amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. We are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.

5. monitoring 2026-08-20 00:01 UTC

   Mitigations have been deployed and we are now catching up to realtime.

6. resolved 2026-08-20 00:30 UTC

   This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US East - SIP

Failure mode: A configuration change caused the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-17 18:10 UTC

   We're investigating an elevated rate of failures when transferring active SIP calls in the US East region. SIP calls themselves remain connected, and inbound and outbound calling are otherwise operating normally.

2. investigating 2026-08-17 18:22 UTC

   No transfer failures has been observed since 18:05 UTC and SIP transfers are currently completing normally. Impact was limited to a small percentage of calls between 15:48-16:08 UTC and 17:24-18:05 UTC. We're actively monitoring while we investigate the cause and put safeguards...

3. resolved 2026-08-17 19:22 UTC

   We have identified the root cause of both spikes and have ensured safeguards to prevent a recurrence. We've been monitoring since 18:05 UTC and have seen no further transfer failures, and have confirmed that SIP transfers are operating normally. We will follow up with a detailed...

4. postmortem 2026-09-01 11:44 UTC

   ## Summary On 17 August, between 17:24 and 18:05 UTC, a small percentage of SIP call transfers failed in our US East region during a routine configuration rollout. An issue in the automation that manages our SIP signaling servers prevented outgoing servers from being taken out...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US East - Cloud Agents

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2026-08-14 21:10 UTC

   New hosted agent builds are failing in US East. Existing agent workloads are not affected. We are currently deploying a mitigation and will follow up with more information shortly.

2. monitoring 2026-08-14 21:15 UTC

   We have deployed a mitigation and builds are now successful. We will continue monitoring while we collect any further details to share.

3. monitoring 2026-08-14 21:41 UTC

   The total build error rate has decreased significantly due to the mitigation, but a small percentage of builds in US East are still failing. We are continuing to monitor and will follow up with another update as soon as possible.

4. resolved 2026-08-14 23:01 UTC

   We pushed a second mitigation at 22:14 UTC and all US East builds have been successful since 22:25 UTC. We will follow up with a postmortem describing the timeline, root cause, and follow ups as soon as possible.

5. postmortem 2026-08-17 17:14 UTC

   ### Summary An issue with the image registry which powers our hosted agents offering temporarily caused new hosted agent builds in US East to fail. Existing agent workloads were not affected, and no sessions or calls were impacted. ### Timeline \(UTC\) * 19:54 - Hosted agent...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-13 20:42 UTC

   We are currently investigating longer than usual dashboard loading times. No services appear to be impacted and we don't expect any data to be lost.

2. monitoring 2026-08-13 21:34 UTC

   We are seeing a 5-10 minute delay in loading some sessions, but we are confident that no data is being lost. We are currently monitoring and will update again once ingestion times have returned to baseline.

3. resolved 2026-08-14 00:15 UTC

   We have resolved the incident as session load times have returned to baseline.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-12 18:34 UTC

   We are currently investigating alerts for elevated server error rates on ListParticipants APIs across multiple regions. The impact appears to be intermittent and limited to a subset of room-management API requests. We will follow up with more details as soon possible.

2. monitoring 2026-08-12 19:20 UTC

   Error rates on ListParticipants API requests returned to normal as of 18:58 UTC. We will follow up with more details as soon possible.

3. resolved 2026-08-12 20:09 UTC

   Error rates on ListParticipants API requests recovered at 18:58 UTC and have remained stable since.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-08-07 21:22 UTC

   We are currently investigating automated alerting which triggered for degraded Room service performance in the US West and US Central regions beginning around 19:45 UTC. We are working to determine if there is any customer-facing impact. If there is impact, we don't currently...

2. monitoring 2026-08-07 21:57 UTC

   We have applied a mitigation to the affected regions and are now monitoring to ensure that performance signals return to baseline. We are still working on determining impact (including how users can determine if they were affected) and will share those details as soon as...

3. resolved 2026-08-07 22:27 UTC

   This incident has been resolved. Between 19:45 and 20:00 UTC, calls crossing the US West US Central link experienced intermittent degradation in Room Service and SIP performance. The underlying network between the two regions had episodes of high packet loss during this time...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Inference

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2026-08-06 19:47 UTC

   Between approximately 18:45 and 19:15 UTC today, a subset of LiveKit Inference requests using the google/gemma-4-31b-it model returned errors. Affected agent sessions would have seen an LLM request error on those requests. Other models and other LiveKit services were not...

2. identified 2026-08-06 19:51 UTC

   We are actively investigating elevated error rate on LiveKit Inference (google/gemma-4-31b-it)

3. monitoring 2026-08-06 21:08 UTC

   A burst of requests increased the failure rate. Both primary and secondary model providers failed to fulfill the requests at the time; we are root-causing the issue. We are continuing to monitor.

4. resolved 2026-08-07 02:51 UTC

   Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Inference

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2026-08-04 20:45 UTC

   We are monitoring elevated timeouts that affected LiveKit Inference between 14:05 and approximately 15:00 UTC today. During that window, roughly 3% of LLM requests using the google/gemma-4-31b-it model timed out. Impacted customers would have seen LLM request timeouts in their...

2. resolved 2026-08-04 22:11 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-08-03 15:30 UTC

   We are currently investigating an issue where the sessions view does not populate in the Cloud Dashboard. This issue is specific to the dashboard display; all LiveKit services are operating normally and no sessions or sessions data are affected.

2. investigating 2026-08-03 16:51 UTC

   We have traced the underlying issue to a backlog in our data pipeline, and we are continuing to investigate the root cause while working to clear the backlog. This data ingest delay results in the Sessions view returning no results for any time range ending at the current time,...

3. resolved 2026-08-03 19:48 UTC

   We have fixed the data pipeline delay and all sessions should be loading now in the Cloud Dashboard.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A configuration change caused the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-07-30 22:55 UTC

   Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period.

2. postmortem 2026-07-30 22:56 UTC

   ## Summary Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period. ## Root Cause A bug in an internal configuration service caused webhook settings to...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-07-30 18:53 UTC

   We are currently investigating longer than usual dashboard loading times. No services appear to be impacted and we don't expect any data to be lost.

2. resolved 2026-07-30 22:25 UTC

   An internal one-off analysis job ran some expensive DB queries which slowed down load times of the dashboard's sessions page. The job was removed and load times are now back to baseline.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-07-28 14:23 UTC

   Between approximately 10:30 and 11:55 UTC on July 28, customers with telephony workloads in our London region experienced elevated error rates on SIP APIs. Small number of call transfers were affected. We mitigated by removing the affected infrastructure from service at 11:48...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-07-27 17:00 UTC

   On July 27, our San Jose region experienced a minor networking incident beginning at 16:14 UTC. Some users connected to this region may have experienced disconnects and increased session start times. Traffic was routed away from this region by 16:39 UTC and errors have since...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2026-07-23 08:00 UTC

   On July 23, between 07:50 and 08:07 UTC, media servers in our London region experienced a host-level network fault, causing them to fail health checks and restart. Users connected to these servers may have experienced brief connection interruptions, and egress operations may...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: United Kingdom - Real Time Communication

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. monitoring 2026-07-14 09:49 UTC

   Between 08:54 am and 09:06 am UTC, 5.4% of room service api requests in our London region failed with errors. A fix was applied and error rates returned to normal by 09:06 am UTC. We are continuing to monitor.

2. resolved 2026-07-14 10:41 UTC

   Error rates for room service api requests in our London region have remained normal since 09:06 am UTC and we are no longer seeing any issues.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Japan - Real Time Communication

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-07-06 18:40 UTC

   We are currently investigating increased connection failures affecting our Japan region beginning at 18:18 UTC.

2. resolved 2026-07-07 10:30 UTC

   Users connecting to our Japan region experienced elevated connection failures between approximately 18:18 and 18:43 UTC, caused by a network disruption affecting connectivity between our Tokyo data centres. Connectivity was restored by 18:43 UTC. Update (July 7): After the...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US Central - SIP, US Central - Real Time Communication

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2026-06-22 13:05 UTC

   We have begun routing traffic away from US Central after our monitoring system triggered due to a high API error rate. Customer impact may include WebRTC connection failures and SIP transfer errors.

2. monitoring 2026-06-22 13:59 UTC

   Update: As soon as the US Central region became unresponsive around 12:25 UTC, traffic was automatically re-routed to the nearest healthy region. Customers may have noticed failed API requests for approximately 1 minute while the re-route took place. Failed SIP Transfers...

3. resolved 2026-06-22 14:40 UTC

   We are resolving this incident as the mitigation is in place and successful. We have received some reports of self-hosted agents which connect to US Central needing to be restarted in order to continue receiving dispatches and will investigate opportunities to improve this...

4. postmortem 2026-06-23 08:32 UTC

   ## Summary LiveKit Cloud is a globally distributed system. Our data centers continuously probe one another, and our overlay network uses those probes to choose healthy paths between regions. When a brief network disruption occurs in or between data centers, the overlay is...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-18 17:49 UTC

   Due to networking issues faced in our US East region, certain features of Cloud Dashboard are not accessible, including Sessions page and Agent Insights. There is no data loss or ongoing session impact associated with this. Our team is actively working on restoring the service.

2. investigating 2026-06-18 18:39 UTC

   Session data (except Agent Insights) on the dashboard can still be accessed by changing the timeline on the Sessions page to "Past 7 days" or greater. We're still working to restore full access.

3. resolved 2026-06-18 19:10 UTC

   As of 18:45 UTC, all features of Cloud Dashboard are fully restored and functional.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US East - Real Time Communication

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-18 17:09 UTC

   We are currently investigating connection failure alerts in US East.

2. monitoring 2026-06-18 17:32 UTC

   US East has experienced a networking failure. Traffic was automatically rerouted away from US East within a minute and connection failure errors are back to baseline. The disruption started happening as of 16:47 UTC, and traffic was rerouted at 16:48. We are continuing to...

3. resolved 2026-06-18 17:40 UTC

   TransferSIPParticipant for calls happening in US East required a manual intervention to redirect. That API is fully restored as of 17:18 UTC. We will follow up with a postmortem as soon as possible.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: India - Egress

Failure mode: Network, DNS, routing or connectivity failure. Root cause: Network or DNS. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2026-06-16 09:49 UTC

   We identified an issue in the India region where a small number of active sessions may have ended earlier than expected. Affected sessions could experience egress recordings stopping mid-session. We've deployed a mitigation and are monitoring to confirm resolution. Other regions...

2. resolved 2026-06-16 20:42 UTC

   We have not seen further instances of this issue since the mitigation was applied. We will follow up with a postmortem as soon as possible.

3. postmortem 2026-06-23 17:46 UTC

   **Summary** Between June 5 and June 16, 2026, an intermittent loss of connectivity between internal systems caused room state to be altered incorrectly in a single region in India. As a result, a small number of active sessions in that region ended earlier than expected, and...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global Inference

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-06-11 17:52 UTC

   We are currently investigating intermittent 429 errors for Google Gemini models routed through LiveKit Inference.

2. identified 2026-06-11 19:09 UTC

   We have identified the cause as an upstream limit on our Google Gemini API account. Automatic failover to an alternate Gemini deployment is in place and has reduced the impact, but a subset of requests to Google Gemini models may still intermittently return errors when failover...

3. resolved 2026-06-11 23:47 UTC

   This incident has been resolved. The errors were caused by an upstream issue at our model provider (Google) that incorrectly triggered an account-level usage cap on our Gemini API access, returning rate-limit errors for a subset of Gemini requests routed through LiveKit...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-06-11 15:30 UTC

   Between 13:00 and 15:05 UTC, a subset of requests to certain Google Gemini preview models (gemini-3.1-flash-lite and gemini-3-flash-preview) routed through LiveKit Inference returned errors due to a project-level misconfiguration. These preview models did not yet have model...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-06-11 13:00 UTC

   On 2026-06-11 around 13:00 UTC, a serialization bug was introduced via frontend components which rendered some services unavailable via the dashboard for some users. These services included SIP trunk and dispatch rule creation/management as well as several advanced features in...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: India - SIP, India - Ingress

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-05 09:06 UTC

   We're investigating elevated latency affecting Ingress and SIP services in our Hyderabad region.

2. investigating 2026-06-05 10:05 UTC

   As of 09:21 UTC, we've applied a mitigation by routing traffic away from the Hyderabad region. Latency rates are returning to normal levels. We're continuing to monitor the issue.

3. resolved 2026-06-05 11:08 UTC

   We identified a connectivity problem in the Hyderabad cluster. As mitigation, we've drained the cluster to route traffic away from it. Mumbai traffic is unaffected and serving normally. We'll undrain Hyderabad once we're confident it's functioning normally.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A zone, region or data center became unavailable. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-06-03 20:00 UTC

   Summary On June 2, 2026, a subset of users placing or receiving SIP calls through LiveKit Cloud's India region experienced call failures between 12:00 and 12:40 UTC. During a routine maintenance, both of our SIP-serving regions in India were temporarily disabled in error. With...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US Central - SIP

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-06-02 14:05 UTC

   We're investigating elevated SIP call connection failures in our US Central region beginning at ~12:06 UTC. We are working to mitigate this issue.

2. monitoring 2026-06-02 14:17 UTC

   We have routed the traffic away from the US Central region at 14:00 UTC and are seeing the connection failures returning to normal levels. We are continuing to monitor the issue.

3. monitoring 2026-06-02 14:58 UTC

   We've identified the root cause as a single problematic media node in US Central, which has been suspended at 14:38 UTC. We are continuing to monitor before marking this resolved.

4. resolved 2026-06-02 15:18 UTC

   This incident has been resolved.

5. postmortem 2026-06-04 14:01 UTC

   **Summary** On June 2, 2026, a small subset of users connected to LiveKit Cloud US Central region experienced elevated connection failures between 12:06 and 14:38 UTC. A single node in the region entered a degraded state in which it continued to accept new connections but could...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: US East - Real Time Communication

Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-28 14:53 UTC

   We are currently investigating reports of spikes in participant connection latency and errors

2. identified 2026-05-28 15:35 UTC

   We are observing database connection timeouts in the US East region and are currently seeing an impact to all services in the region. We are currently draining the region and routing away all traffic to other regions.

3. monitoring 2026-05-28 15:50 UTC

   US East drain is complete and all new traffic is now routing to other regions. Error rate is trending down and we'll continue monitoring.

4. monitoring 2026-05-28 16:27 UTC

   Connection errors have fully cleared since the US East drain completed at 15:44 UTC. We'll continue monitoring. We will be sharing a detailed RCA.

5. monitoring 2026-05-28 18:27 UTC

   Service is operating normally with traffic routing through other regions. We are working on bringing US East back online and will share a detailed RCA once complete.

6. resolved 2026-05-28 20:13 UTC

   Connection errors have remained cleared since the US East drain at 15:44 UTC, and as of 19:30 UTC, US East is back online. We'll share a detailed RCA in the coming days.

7. postmortem 2026-05-30 18:18 UTC

   ## Summary LiveKit Cloud validates incoming API and connection requests via an internal authentication service, accessed over gRPC. The service has been in place since 2022 and is designed for resilience: it runs as multiple redundant instances, has pod-level health monitoring...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Cloud Dashboard (cloud.livekit.io)

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-26 19:22 UTC

   We are investigating reports of Agent Observability not accessible for some sessions.

2. identified 2026-05-26 19:53 UTC

   We have identified an issue causing partial data unavailability in LiveKit Cloud Dashboard: - For some users, Agent Observability is temporarily unavailable for some sessions. - Some sessions in the 24-hour filter are showing as Active even though they have ended. A fix has been...

3. monitoring 2026-05-26 20:13 UTC

   A fix has been implemented and we are seeing no more data outage for newly created sessions. We are monitoring the fix and are currently working on backfilling the missing data.

4. resolved 2026-05-26 21:32 UTC

   All missing data has now been backfilled and the incident is resolved.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-05-22 20:26 UTC

   We have observed network instability impacting cross-region traffic from Frankfurt beginning at 18:15 UTC. Traffic has been redirected from Frankfurt as of 19:55 UTC. Participants originally connecting via Frankfurt may see slightly increased latency until the region is restored.

2. monitoring 2026-05-23 00:17 UTC

   We have re-enabled our Frankfurt cluster and are monitoring for further disruptions.

3. resolved 2026-05-23 16:36 UTC

   We have been monitoring the clusters for the past 16 hours and have not found any further disruptions.

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-15 16:57 UTC

   We are investigating intermittent connectivity disruptions affecting a subset of sessions, caused by internal network degradation between several of our US clusters (most heavily impacting US East). Affected sessions may see brief interruptions (less than 5 minutes) to media...

2. monitoring 2026-05-15 21:16 UTC

   We have not observed further instances of connectivity disruptions since 13:19 UTC. We are continuing to investigate why our reroute mechanism did not activate in these isolated instances and are monitoring for further disruptions.

3. resolved 2026-05-16 02:50 UTC

   We are resolving this incident as we have not observed any further connectivity disruptions. We will follow up with a postmortem once available.

4. postmortem 2026-05-22 07:36 UTC

   ## Summary LiveKit Cloud runs as a distributed realtime network, with data centers around the world interconnected via dedicated networking. Even with dedicated fiber, momentary disruptions between any two data centers can and do occur. To handle these blips, we have built a...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2026-05-07 21:15 UTC

   Between ~17:16 and 17:21 UTC on May 7, 2026, inter-region connectivity between our US East region and other regions was briefly degraded due to a network connectivity issue in our underlying cloud provider's data center. During this window, sessions that relied on cross-region...

Excerpts are shortened. Read the full incident on LiveKit's status page.

Affected: Global SIP

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-05-06 19:46 UTC

   We have noted that outbound calls with both TLS and media encryption enabled via Twilio are failing. We are currently investigating and will update here as soon as we know more.

2. monitoring 2026-05-06 20:05 UTC

   A fix has been implemented and we are monitoring the results. The issue persisted from 18:10 - 19:58 UTC.

3. monitoring 2026-05-06 20:32 UTC

   We believe this incident was not provider-specific and as such have removed Twilio from the incident title.

4. resolved 2026-05-06 23:04 UTC

   We are resolving this incident as we have not observed any additional errors since the fix was rolled out. We will follow up with a postmortem as soon as possible.

5. postmortem 2026-05-10 15:11 UTC

   **Summary** On 2026-05-06 between 18:07 and 19:56 UTC, a code change to the API server which handles CreateSIPParticipant calls caused outbound calls to incorrectly inherit their encryption settings from outbound trunks. This resulted in calls that didn't specify an encryption...

Excerpts are shortened. Read the full incident on LiveKit's status page.