">
sevanalysis

Daily

Infrastructure. Source: Daily status page.

Incidents, last 90 days
0
Median to mitigation
none resolved
Median to resolution
none resolved
p95 to resolution
none resolved
Naming an upstream provider
no incidents
With a postmortem on the status page
no incidents

Incidents per month

All recorded incidents

Continued issues with joining calls from certain ISPs/Regions minor 2026-06-10 19:20 UTC 7h 27m

Affected: Core Call Experience

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: GitHub.

  1. identified

    We're seeing a recurrence of the problems with RegistryDNS.co and AT&T DNS from Monday. Users trying to join calls from affected regions using ISP DNS may get "Unable to join call" errors. We know specifically that AT&T DNS servers 68.94.157.1 and 68.94.156.1 are affected. We've...

  2. identified

    We expect to have daily-js 0.91.0 available within the next 60 to 90 minutes. The update includes automated failover to `dailywebrtc.com` or `dailywebrtc.net` if `daily.co` is unreachable. If you haven't updated daily-js in a while, you can do preliminary testing with 0.90.0 in...

  3. monitoring

    daily-js 0.91.0 is now available on npm and GitHub. If your app uses call object mode, you can update your dependency to that version and redeploy, and this issue should be resolved for you. We're publishing an update to our networking guide soon, but in the meantime, anything...

  4. resolved

    The RegistryDNS.co servers continue to be intermittently unavailable, but AT&T's DNS servers seem to be more responsive. We encourage you to update to daily-js 0.91.0 as soon as possible to avoid future issues for your users.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with joining calls from certain ISPs/regions minor 2026-06-08 14:32 UTC 24h 25m

Affected: Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: Cloudflare.

  1. identified

    Users that are using their ISP's DNS service (specifically AT&T users in the southeast US and/or Texas) are experiencing intermittent problems accessing any .co domain, including daily.co. This may result in those users seeing an “Unable to join call” error in the browser when...

  2. identified

    We are working on implementing a fallback to a .com domain so that users on ISPs that are not fixing this will not see any issues starting calls.

  3. resolved

    CentralNIC confirmed that there was indeed an issue with the .co nameservers, and they resolved it at approximately 06:30 UTC today. Our monitoring of both their DNS servers and downstream providers confirmed this. We're continuing to work urgently on updating our infrastructure...

Excerpts are shortened. Read the full incident on Daily's status page.

API and call connection issues minor 2026-04-17 13:33 UTC 9h 33m

Affected: API, Core Call Experience

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS, Cloudflare.

  1. investigating

    We're receiving reports of some users having problems connecting to Daily calls. It seems to be localized to the Texas area. We're investigating.

  2. identified

    We've identified an issue that's causing some users to fail to join calls. The daily-js library has to download a bundle of additional JavaScript as part of joining a call. This bundle is downloaded from c.daily.co, which is using Amazon's CloudFront CDN. They aren't reporting...

  3. identified

    We're receiving reports of a few other regions experiencing similar connection issues. If you're monitoring client errors and you see messages that start with "Failed to load call object bundle https://c.daily.co/....", you're being affected by this. We're working with AWS to...

  4. identified

    We've confirmed through several different end users that this is a DNS issue. Affected users from multiple regions have been able to join calls by pointing DNS to Google or CloudFlare, using IPs like 1.1.1.1 or 8.8.8.8 for DNS. Obviously, this solution doesn't scale; we're...

  5. identified

    Cloudflare has posted an incident concerning intermittent DNS failures for .co domains: https://www.cloudflarestatus.com/incidents/z3b5zxjtp6g1 This aligns with the troubleshooting we've done so far. Some internet discussions suggest that it can be somewhat ISP-dependent, but...

  6. identified

    The .co registry appears to be experiencing issues. Cloudflare's recursive DNS is affected (and they've acknowledged it). Many regional ISPs, such as AT&T in the southeast US, rely on Cloudflare's DNS to power their own DNS. Unlike Cloudflare, other DNS providers, like Google...

  7. identified

    Cloudflare has posted that they've implemented a fix, and they are monitoring the results. We're still not certain that Cloudflare is the actual root cause of 100% of our affected users, but we'll be monitoring error rates.

  8. identified

    Cloudflare has resolved their status incident, but we're unsure if that means the underlying issue is resolved. We're continuing to monitor our own error rates and health checks.

  9. identified

    The rate of bundle download failures has decreased, but it's also tracking our overall usage volume through the day. The number of affected users appears to be very small, but we still have tests that can replicate the failure. Unfortunately this is completely out of our...

  10. identified

    Our tests just confirmed that DNS resolution is no longer returning errors for .co domains in our tests. This issue has been resolved!

  11. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues joining calls minor 2026-04-10 15:36 UTC 54m

Affected: API, Core Call Experience

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We're seeing an elevated rate of errors for API requests and room joins. We're addressing it right now.

  2. monitoring

    There was a sudden increase in activity across several databases. We've addressed the cause of the issue, and platform metrics are returning to normal. We're continuing to monitor for any further issues.

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with inbound phone calls minor 2026-03-23 20:18 UTC 58m

Affected: SIP/PSTN

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating elevated error rates for incoming PSTN/SIP calls.

  2. investigating

    SignalWire has identified an issue causing problems with incoming and outgoing PSTN/SIP calls for Daily customers. If you're making dialout calls, you may see an error that says "DialOut stopped: Remote busy or Remote did not answer or Remote ended the call". If you're using...

  3. monitoring

    It seems that SignalWire has resolved their issue. We're monitoring for any other problems.

  4. resolved

    This incident has been resolved. We're still collecting more information on the root cause here; we'll update this status post when we have more info.

Excerpts are shortened. Read the full incident on Daily's status page.

Webhook delivery delays minor 2026-03-16 17:12 UTC 4h 45m

Affected: Webhook Delivery

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue that may be causing delayed delivery for some webhooks.

  2. monitoring

    We've identified an issue with the webhook system that was causing brief delays when finding meeting start and end events for webhooks. We've addressed that issue. Webhook delivery time was only minimally impacted during the event, and it has returned to normal as of about 15...

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

SIP / PSTN Disruption critical 2026-02-18 11:31 UTC 4h 24m

Affected: SIP/PSTN

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We are currently investigating an issue affecting SIP / PSTN inbound calls. Customers may encounter 480 “Temporarily Unavailable” errors when attempting to receive calls. Our engineering team is actively working to identify the root cause and restore service.

  2. identified

    Our upstream carrier partner has identified the root cause of the issue impacting SIP/PSTN dial-in (inbound) calling and has implemented a fix. Service is beginning to recover, though some degradation may still be present while the fix fully propagates. We are actively...

  3. monitoring

    The fix from our upstream carrier partner has been applied, and service has been restored for SIP/PSTN dial-in (inbound) calling. We are seeing successful inbound call connections again, and our internal testing confirms recovery. We are continuing to monitor closely to ensure...

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Increase in "Failed to load call object bundle" errors minor 2026-02-04 23:59 UTC 8h 16m

Affected: Core Call Experience

Failure mode: CDN or edge layer failed or served bad content. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. identified

    We've seen an increase in "Failed to load call object bundle" errors within the past hour. This happens when the browser is unable to download our JavaScript bundle from the CDN. If this happens to you or your users, you should be able to refresh the page and join your call...

  2. monitoring

    The initial increase in error rates has subsided. Based on reports from a few customers, it seems like the impact was primarily in the Texas area. We're continuing to work with AWS to determine the root cause.

  3. resolved

    The issue has been resolved. We’ve seen the number of “Failed to load call object bundle” errors return to normal levels, and call joins are operating as expected.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with transcriptions and captions minor 2026-02-02 15:45 UTC 6h 52m

Affected: Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. identified

    We're tracking a Deepgram incident that is causing problems with transcription events, as well as captions in Daily Prebuilt: https://status.deepgram.com/incidents/rp0jzltym9kw

  2. identified

    We're continuing to see very brief occurrences of slow or missing transcription data from the Deepgram streaming endpoint. We're talking to Deepgram engineers and continuing to monitor the situation.

  3. monitoring

    We're seeing extremely low error rates, but Deepgram hasn't resolved their incident yet, so we'll keep monitoring.

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Webhook deliveries delayed minor 2026-01-26 16:47 UTC 4h 37m

Affected: Webhook Delivery

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We are currently experiencing an abnormally large queue of webhooks awaiting delivery. Our team is investigating and working to restore normal processing.

  2. identified

    We’ve identified the cause of the issue. While a backlog of webhooks remains, we’re actively working to reduce it. Webhook deliveries will continue to be delayed for some time, but no data is being lost.

  3. identified

    We're continuing to work through a backlog of webhook events, and the delay is starting to decrease. It will still take some time to get fully caught up.

  4. monitoring

    We've worked through the webhook backlog, and delivery times should be normal. We'll be monitoring for any further issues.

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues starting SIP/PSTN calls minor 2026-01-15 21:32 UTC 39m

Affected: SIP/PSTN

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. identified

    We're seeing an increased rate of errors when dialing out SIP/PSTN calls.

  2. identified

    We've rolled back the release that was in process when we started seeing the errors, which seems to have resolved the issue. We'll be monitoring for any regressions.

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Webhook deliveries delayed minor 2026-01-13 22:13 UTC 1h 33m

Affected: Webhook Delivery

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. identified

    We're seeing an abnormally large queue of webhooks to be delivered, so we've scaled up to work through the backlog. We'll post another update when delivery times return to normal.

  2. identified

    We're continuing to work through the backlog of webhooks.

  3. resolved

    Webhook delivery times have returned to normal.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with PSTN/SIP availability none 2026-01-13 00:16 UTC 0m

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. resolved

    SignalWire experienced an outage earlier this morning. Their status site post is here: https://status.signalwire.com/incidents/MJVYwKKYYwbG Our first customer report of problems was at 1:51 AM PST (09:51 UTC). During the incident, many SignalWire SIP calls were returning 603...

Excerpts are shortened. Read the full incident on Daily's status page.

Impact from the ongoing AWS us-east-1 outage minor 2025-10-20 12:57 UTC 26h 25m

Affected: Dashboard, Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. identified

    AWS is addressing an issue with DynamoDB that is causing problems with many of their services in us-east-1. We have seen issues loading the dashboard and docs sites, and for a few hours we've been unable to log in to Statuspage to update this site. We're sorry about that delay....

  2. identified

    AWS is continuing to address issues in us-east-1. We're still seeing normal activity levels across our APIs and call servers for this time of day, so we have no reason to believe that any your customers' calls are actually being affected. Part of AWS's mitigation efforts involve...

  3. monitoring

    AWS has resolved their incident. We're continuing to run the same call servers in us-east-1 as we were before the incident started, but we're still routing the majority of US traffic through us-west-2. We plan to return things to normal tomorrow.

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Dashboard API and webhook logs temporarily disabled minor 2025-09-24 18:17 UTC 27h 4m

Affected: Dashboard

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We've temporarily disabled displaying the API and webhook log views on the Dashboard Developer page. The Daily API is still fully operational, and these logs are still being collected. We expect to re-enable these log views in the next few hours. If you have an urgent need for...

  2. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Delayed data from the /logs API endpoint minor 2025-09-04 14:48 UTC 1h 57m

Affected: API

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're experiencing delays in making log data available via the /logs REST API endpoint. We're not losing any log data, but the process that ingests logs for the REST API is bottlenecked. This means that the most recent data available from the /logs endpoint is about 6 hours old....

  2. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with SIP dialin/dialout minor 2025-07-17 20:52 UTC 48m

Affected: SIP/PSTN

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We're investigating reports of SIP/PSTN dialin and dialout failing, or stopping immediately after starting.

  2. identified

    We're in the process of rolling back a release to address this issue.

  3. monitoring

    We've rolled back a server release, and the SIP error messages have stopped. We're monitoring for any further problems.

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Webhook delivery delays minor 2025-07-14 16:43 UTC 42m

Affected: API, Webhook Delivery

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We're seeing alerts that webhooks are taking longer than normal to deliver.

  2. identified

    We're aware of an infrastructure issue that may be causing some intermittent failures for Daily API requests.

  3. resolved

    We're continuing to investigate the root cause, but we've resolved all impact to our running services.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with logging and telemetry minor 2025-06-12 18:39 UTC 3h 8m

Affected: Dashboard, Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: Google Cloud.

  1. investigating

    Daily is being affected by widespread internet outages in authentication services. This is currently affecting our ability to collect or view some call metrics. We'll post more as soon as we have more information.

  2. identified

    This appears to be related to the Google Cloud outage posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW We'll post here as we learn more.

  3. identified

    We're seeing some increases in response time from our internal API endpoint that returns connection information to daily-js. This may be causing "room lookup timed out" errors for users attempting to join calls. We believe this is related to the ongoing GCP incident.

  4. identified

    We had a brief spike in response time, but platform operations appear normal since approximately 20:05 UTC. We're continuing to monitor for any follow-on effects, and we'll leave this status post open until various upstream providers close their respective status incidents.

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Elevated error rates for SIP/PSTN minor 2025-04-17 19:33 UTC 1h 3m

Affected: SIP/PSTN

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We're investigating elevated error rates from SignalWire when provisioning SIP/PSTN resources. If you're creating rooms with dial-in or dial-out enabled and it isn't absolutely necessary, you can remove those params from your room creation request to successfully create rooms.

  2. identified

    SignalWire has confirmed that they made an unannounced change that's causing the problem, and they're working to revert it. We're also working on a quick production update to work around the issue if they can't revert quickly enough.

  3. monitoring

    It appears that SignalWire rolled back the change, and the errors have stopped. We are monitoring for any further issues.

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues with SIP/PSTN audio quality minor 2025-04-15 14:11 UTC 7h 45m

Affected: SIP/PSTN

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating reports of poor quality audio for SIP/PSTN participants in some calls.

  2. identified

    We've identified an issue causing degraded ("broken up" or "choppy") audio between some Daily sessions and SignalWire SIP/PSTN endpoints. PSTN dialout and PIN dialin do not seem to be affected. SIP dialout seems to be moderately affected. PIN-less PSTN dialin and SIP dialin seem...

  3. identified

    We've identified an issue with how Daily and SignalWire are negotiating audio codecs for incoming SIP/PSTN calls. We're working on an update that will change the default audio codec used between Daily and SignalWire to Opus. Once this is deployed, you may still experience audio...

  4. monitoring

    We've deployed an update to production servers to remediate the issue, and we're testing to ensure everything is fixed.

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Elevated SIP/PSTN error rates critical 2025-04-09 12:15 UTC 9h 33m

Affected: SIP/PSTN

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We're seeing elevated error rates from our SIP/PSTN provider. If you're creating rooms with dial-in support and getting errors, you may want to retry creating those rooms without dial-in, and then add dial-in with an update REST request.

  2. identified

    Our SIP/PSTN provider has identified an issue and they're deploying a fix. In the meantime, you should still be able to create Daily rooms without provisioning SIP/PSTN.

  3. identified

    We're continuing to monitor an issue from our SIP/PSTN provider. The rest of the Daily platform is unaffected. If you're getting an error when trying to create a room, you can remove the dialout_enabled property from the room creation request and try again.

  4. identified

    We're still monitoring. In addition to dialout_enabled, you'll need to remove other room properties related to SIP/PSTN, dialin, and/or dialout to create rooms.

  5. identified

    We're still waiting on Signalwire to restore service.

  6. identified

    We've been told we should be back online after a hotfix at approximately 18:00 UTC, or 15 minutes from now. We'll post another update as soon as we have more info.

  7. monitoring

    Signalwire has restored service, but we're still seeing some error responses from their API. We believe the majority of room creation failure issues have been resolved. We'll post here again when we've handled this last issue.

  8. monitoring

    We're still waiting on some additional fixes from Signalwire. Room creation with dial-in/dial-out is working, but you still may experience problems updating dial-in/dial-out settings for existing rooms. Daily Bots and Pipecat Cloud users should be unaffected.

  9. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Delayed audio for some SIP/PSTN dial-in calls minor 2025-03-12 15:19 UTC 5h 27m

Affected: SIP/PSTN

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue that's causing delays in audio connection for a some SIP/PSTN calls.

  2. identified

    We're getting reports of delayed audio from some customers using SIP dialin and dialout. When the phone user joins the call, they can talk and others will hear them, but the phone user won't hear any audio from other call participants (bot or human) for the first 20-30 seconds....

  3. identified

    We've identified the issue, and we're testing a fix in our staging infrastructure. We'll post another update when we've deployed the fix.

  4. monitoring

    We've deployed a fix, and we're monitoring for any further issues.

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues connecting to rooms minor 2025-01-28 16:21 UTC 38m

Affected: Core Call Experience

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue that may be preventing some users from joining meeting rooms.

  2. monitoring

    We've resolved the issue and we're monitoring to ensure the platform is operating normally.

  3. resolved

    This issue has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Networking issues major 2024-11-25 16:48 UTC 2h 38m

Affected: API, Core Call Experience

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating issues that may be causing problems with network connections between regions.

  2. investigating

    We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and...

  3. identified

    We've temporarily disabled the component that sends webhooks.

  4. identified

    We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves....

  5. monitoring

    We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.

  6. monitoring

    Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.

  7. resolved

    We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues starting recordings minor 2024-10-30 18:59 UTC 2h 4m

Affected: Recording/Live Streaming

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. identified

    We've identified an issue causing some recordings to fail to start, specifically in the Oracle Cloud San Jose region. We've already made some infrastructure changes that should be routing new recording requests to other regions. If you've seen a recording fail to start, you can...

  2. resolved

    We're still making a few small infrastructure changes, but our internal metrics have been back at normal levels for some time.

Excerpts are shortened. Read the full incident on Daily's status page.

Delayed API calls minor 2024-10-22 18:46 UTC 3h 16m

Affected: API

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue that's causing delays with some API operations, such as creating rooms and starting recordings. We'll post more info as soon as we have it.

  2. identified

    We've identified an issue causing some slowdowns in one of our databases, leading to some delayed or failed API responses. We've solved the root cause of the issue, but we're being cautious about restoring the database to full functionality, so we expect the delays to continue...

  3. identified

    API error levels have decreased considerably, but we're still working on full remediation. More updates to come.

  4. monitoring

    API latency and errors have stayed at normal levels for a while now, but we're continuing to monitor for any further impact.

  5. resolved

    This issue has been resolved. We will post more information about this incident in the near future.

  6. postmortem

    On Tuesday, October 22, around 17:15 UTC \(9:15 AM PDT\), a Daily customer started running a series of load tests. Their test involved rapidly creating and deleting a large number of rooms that used PSTN dial-out, cloud recording, and webhooks. This eventually caused several...

Excerpts are shortened. Read the full incident on Daily's status page.

Missing meeting webhook deliveries none 2024-10-21 17:50 UTC 8h 49m

Affected: API

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. monitoring

    Between 14:01 UTC and 17:12 UTC webhooks for meeting.started and meeting.ended events were not delivered. We have applied a mitigation and are continuing to monitor. The underlying cause for the missing deliveries is still under investigation.

  2. resolved

    This incident has been resolved. Customers needing assistance with missing webhook deliveries should contact support via help@daily.co.

Excerpts are shortened. Read the full incident on Daily's status page.

dashboard.daily.co availability minor 2024-08-07 19:41 UTC 35m

Affected: Dashboard

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. investigating

    Some customers are seeing 400 BAD_REQUEST messages when trying to load dashboard.daily.co. This is likely related to a Vercel incident: https://www.vercel-status.com/incidents/f6b2blrl5f5f

  2. monitoring

    The upstream issue has been resolved, and we're monitoring for any more issues.

  3. monitoring

    The upstream issue has been resolved, and we're monitoring for any more issues.

  4. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Elevated latency on some API endpoints none 2024-05-24 08:03 UTC 53m

Affected: API

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We are currently investigating increased latency affecting some of our APIs.

  2. monitoring

    We have applied a mitigation and are continuing to monitor the situation.

  3. resolved

    API latency has returned to normal levels.

Excerpts are shortened. Read the full incident on Daily's status page.

Degraded logging and metrics API performance minor 2023-12-21 14:04 UTC 9h 45m

Affected: API

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We are currently investigating an issue with degraded performance with the logging and metrics API.

  2. monitoring

    The degraded logging and metrics API performance was the result of an impaired database system. The initial impact was resolved earlier today, but we continue to monitor the system as recovery completes.

  3. resolved

    The impaired database system has fully recovered and is operating normally. API performance has returned to normal levels.

Excerpts are shortened. Read the full incident on Daily's status page.

Elevated latency / intermittent failures on API endpoints minor 2023-09-18 20:21 UTC 14h 15m

Affected: API, Recording/Live Streaming

Failure mode: Network, DNS, routing or connectivity failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're currently investigating this issue.

  2. monitoring

    Network-level mitigations have been applied and we are seeing latency back to normal levels.

  3. resolved

    Network-level issues are resolved and service is operating nominally.

Excerpts are shortened. Read the full incident on Daily's status page.

Issue with sessions in ap-northeast-2 major 2023-08-10 14:13 UTC 29m

Affected: Core Call Experience

Failure mode: A zone, region or data center became unavailable. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue preventing some users from joining call sessions in the ap-northeast-2 region.

  2. investigating

    We've confirmed an issue preventing some users from joining calls hosted in the ap-northeast-2 region. Sessions in other regions are unaffected. If you've set the 'geo' property on your domain or a specific room to 'ap-northeast-2', you may want to temporarily change it to...

  3. resolved

    This issue has been resolved, and call sessions in ap-northeast-2 are working normally.

Excerpts are shortened. Read the full incident on Daily's status page.

Problems connecting to rooms minor 2023-03-13 20:36 UTC 2h 34m

Affected: Core Call Experience

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating reports of problems when trying to join calls.

  2. monitoring

    We've identified an issue that was causing some users to receive an error when trying to join a call. Affected users would see an error in the console starting with "web socket connection failed". We've rolled back a platform update from earlier today, and the errors have...

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues connecting to calls minor 2023-02-07 14:54 UTC 2d 8h

Affected: API, Dashboard, Core Call Experience

Failure mode: A schema or data migration caused the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    We are investigating elevated platform error rates. Users may get websocket connection errors when trying to join calls.

  2. identified

    We've identified an issue with one of our databases that coordinates activity between call servers. This is causing elevated rates of "meeting moves", which is when an ongoing call session has to move from one call server to a different one. If you're in a call when this...

  3. monitoring

    We’ve addressed the issue with the database, and platform operations have returned to normal. We are monitoring alerts and metrics for any further issues.

  4. monitoring

    While we were able to restore platform functionality earlier today, we've continued to troubleshoot the underlying issue that caused the problem. As a precautionary measure, we've temporarily enabled rate limiting on the REST API endpoint used to create rooms. The limit for POST...

  5. monitoring

    We've deployed a platform update with a few improvements designed to mitigate the impact of the current database performance issue. The only thing you may notice is that you'll no longer see 429 rate limit responses in your Dashboard API logs. Our database metrics have remained...

  6. resolved

    We've identified the issue that caused the incident on Tuesday morning. While we've already deployed fixes that helped prevent the problem from reoccurring, we still need to perform one more database update that will require a short scheduled maintenance. That will likely happen...

  7. postmortem

    On Tuesday, February 7 at 9:47 AM Eastern time \(14:47 UTC\), our database reported a performance issue under normal operational load. We had upgraded the database server over the weekend, but it had been operating normally since Monday. The alerts indicated a high level of lock...

Excerpts are shortened. Read the full incident on Daily's status page.

Issues connecting to calls. critical 2023-02-06 15:01 UTC 2h 32m

Affected: API, Core Call Experience, Recording/Live Streaming

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. investigating

    We're investigating an issue preventing some users from connecting to calls.

  2. identified

    The issue has been identified and a fix is being implemented.

  3. identified

    We have identified the issue and are applying a fix.

  4. monitoring

    A fix has been applied and we are monitoring to be sure that all underlying issues are resolved.

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Missing metrics in call participant logs minor 2023-01-25 15:18 UTC 4h 50m

Affected: Dashboard

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating reports of missing metrics data in participant logs.

  2. investigating

    We're investigating reports of missing metrics data in participant logs from a small number of users. This may date back to some time around 2023-01-23 17:00 UTC (9:00 AM PST on Monday, Jan 23).

  3. resolved

    We've confirmed the initial report that there are a small number of recent call sessions that didn't log any metrics data. This can happen if your app has multiple call object instances running on the same page. Your app may do this if you are calling createCallObject() more...

Excerpts are shortened. Read the full incident on Daily's status page.

Problems creating raw-tracks recordings minor 2023-01-18 20:54 UTC 7h 33m

Affected: Recording/Live Streaming

Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We're investigating reports of errors from customers trying to create "raw-tracks" recordings.

  2. identified

    We've confirmed an issue preventing the creation of raw-tracks recordings. Other recording types are unaffected, including "cloud" recordings to your own S3 bucket. If you need to record an important call during this incident, you can change the enable_recording property on your...

  3. identified

    We're in the process of deploying updates to resolve this issue. We'll resolve the incident as soon as the fix is live in production.

  4. monitoring

    We've deployed new call servers and recording infrastructure to resolve the issue. You should be able to start a raw-tracks recording from any call session that started on or after approximately 02:15 UTC. Existing long-running call sessions may still be running on older call...

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Intermittent issues with cloud recordings minor 2022-12-06 00:37 UTC 45m

Affected: API, Recording/Live Streaming

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. monitoring

    We are experiencing an issue where cloud recordings are intermittently returning all black frames. We have pushed a fix to production and are currently monitoring the situation.

  2. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Intermittent issues starting cloud recordings and livestreams major 2022-11-29 13:17 UTC 3h 30m

Affected: API, Recording/Live Streaming

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. investigating

    We are experiencing an issue where customers attempting to start livestreams or cloud recordings are intermittently receiving a temporarily-unavailable error.

  2. identified

    We have identified the issue and are currently testing a fix.

  3. monitoring

    We have pushed a fix to production and are currently monitoring the situation.

  4. monitoring

    Daily’s auto scaling system experienced a failure to communicate with some internal services, preventing it from adding capacity for cloud recording and live streaming quickly enough to keep up with demand. We resolved the issue, and we're monitoring platform operations to...

  5. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Problems joining rooms in us-east-1 minor 2022-10-17 22:14 UTC 27m

Affected: Core Call Experience

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    We're investigating an issue preventing some users from joining meetings hosted in the us-east-1 region.

  2. monitoring

    We identified a brief issue with DNS while deploying a call server (sigh, it's always DNS). This would have caused intermittent join problems for some users for several minutes. We resolved the issue, and we're monitoring platform operations to ensure that everything has...

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Issues connecting to rooms minor 2022-09-28 16:57 UTC 6h 43m

Affected: Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    We're investigating an issue preventing some users from connecting to rooms in the us-west-2 region.

  2. identified

    We're routing around a possible networking issue to our API gateways in us-west-2 . This should allow your users to connect to calls, but we're still watching for other networking problems or follow-on effects .

  3. monitoring

    AWS has acknowledged an issue with API Gateway in the us-west-2 region. We're routing API requests to other regions for now, so everything should be operating normally for you and your users. We'll leave this issue open until AWS has resolved their underlying issue and our...

  4. monitoring

    We're still watching the ongoing AWS status incident until it's resolved. We'll provide another update if anything changes in the meantime.

  5. resolved

    AWS has resolved their issue, and our operations have returned to normal.

Excerpts are shortened. Read the full incident on Daily's status page.

Problems connecting to rooms minor 2022-08-24 17:16 UTC 56m

Affected: Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    We are investigating an AWS issue with API Gateways in the us-west-2 region that is preventing some users from joining Daily sessions.

  2. investigating

    We've temporarily removed our affected us-west-2 API Gateways while AWS works to resolve the underlying issues. That should solve the problem that was preventing users from joining calls. Existing call sessions should be unaffected. We're closely monitoring other parts of our...

  3. monitoring

    We've confirmed that the issues with joining calls were a result of an AWS incident posted on their status site. AWS has resolved that incident, and we're seeing successful responses from our us-west-2 resources in our staging environment. We should be re-enabling our us-west-2...

  4. resolved

    We've re-enabled our API Gateways in us-west-2, and users are connecting to rooms successfully. This incident is resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Users may be unable to view meeting session data in dashboard minor 2022-06-02 11:41 UTC 6h 12m

Affected: Dashboard

Failure mode: The text does not say enough to tell. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. identified

    We've identified an error impacting the ability for some users to view meeting session data in the Daily Dashboard. A fix is being implemented.

  2. monitoring

    A fix has been deployed, and dashboard users should now be able to access meeting session data. We're continuing to monitor the situation.

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Customers may experience difficulty downloading meeting recordings. minor 2022-04-06 13:45 UTC 46m

Affected: API, Dashboard

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

  1. identified

    Daily has identified a problem that is impacting the ability to download meeting recordings via the dashboard and access-link APIs, and is implementing a fix.

  2. monitoring

    Daily has resolved the issue impacting downloads of meeting recordings, and continues to monitor the situation.

  3. resolved

    The issue impacting downloads of meeting recordings is resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Connectivity issues major 2021-12-15 15:32 UTC 1h 3m

Affected: API, Dashboard, Core Call Experience, Recording/Live Streaming

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    We're investigating reports of problems connecting to calls.

  2. identified

    We're experiencing network delays and timeouts throughout our infrastructure as a result of a larger-scale AWS incident. You may experience problems connecting to calls, viewing your Dashboard, or making REST API requests. We'll update this incident as we know more.

  3. identified

    We're continuing to see issues across our platform as a result of the ongoing AWS outage. You'll likely experience problems joining calls, accessing the Dashboard, or using the REST API. We'll continue to post more information here as we have it.

  4. identified

    We've seen overall error rates decrease as AWS has been working to resolve the networking issue. Things are improving, but you may still experience delays and errors until this issue is fully resolved.

  5. resolved

    Error rates and network metrics have returned to normal levels, so we're considering this issue resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Degraded audio and video call experience minor 2021-12-10 21:05 UTC 2d 16h

Affected: Core Call Experience

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

  1. identified

    We are currently experiencing an issue with one of our service providers that may be affecting connection to calls (slow connections or timeouts). We are implementing a fix.

  2. identified

    We are continuing to work on a fix for this issue.

  3. monitoring

    We have temporarily routed traffic through another service provider, which should resolve call connection issues for most users.

  4. resolved

    We have restored our service provider configuration to its nominal state after confirming that all providers are operating normally.

Excerpts are shortened. Read the full incident on Daily's status page.

Audio and video calls impacted by an ongoing incident in us-east-1 major 2021-12-07 17:43 UTC 19h 30m

Affected: Core Call Experience

Failure mode: A zone, region or data center became unavailable. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. investigating

    Users close to the us-east-1 (Northern Virginia) region of AWS may be unable to join video calls. We are investigating.

  2. monitoring

    We have attempted to work around this AWS issue for users that would normally be routed to our us-east-1 resources by temporarily removing all of our us-east-1 DNS records from our AWS API Gateway configurations whilst we wait for AWS to recover the region. We'll continue to...

  3. resolved

    AWS has resolved the regional issues in us-east-1. We have re-activated the us-east-1 region for new audio and video calls.

Excerpts are shortened. Read the full incident on Daily's status page.

Internet connectivity - South America region minor 2021-11-09 01:15 UTC 3h 23m

Affected: Core Call Experience

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

Names an upstream provider: AWS.

  1. identified

    AWS is experiencing intermittent Internet connectivity issues in the South America - Brazil region. Users may experience a degraded experience connecting to audio and video calls in the region. Connecting to calls may take longer, and users in the region may connect to a server...

  2. monitoring

    Connectivity issues in the South America - Brazil region have been resolved. We are continuing to monitor the situation.

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.

Dashboard degraded: logging and telemetry impacted minor 2021-10-11 18:04 UTC 3h 10m

Affected: Dashboard

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

  1. identified

    We have identified a problem retrieving audio and video call logging and telemetry from the dashboard. Customers may experience a 'Session not found' error message when attempting to view call logs and telemetry.

  2. monitoring

    The impacted data repository has returned to normal operation. Audio and video call logs and metrics should now be available in the dashboard.

  3. resolved

    This incident has been resolved.

Excerpts are shortened. Read the full incident on Daily's status page.