PyPI (Python Infrastructure)

Affected: pypi.org - Backends

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2026-10-02 01:15 UTC

   infrastructure maintenance is having a larger impact on availability of PyPI's backends than anticipated. we are working to stabilize.

2. resolved 2026-10-02 01:32 UTC

   the maintenance is complete and everything has stabilized

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: docs.python.org - Backends

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2026-10-01 11:13 UTC

   Multiple python.org services were impacted by expiration of internal TLS certificates caused by use of APIs deprecated by a recent library update. We have resolved the issue and are monitoring as services come back online after renewing their internal certificates.

2. resolved 2026-10-01 11:18 UTC

   All services are restored.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General, pypi.org - Backends

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-09-28 12:42 UTC

   We are currently investigating this issue.

2. investigating 2026-09-28 13:47 UTC

   We are continuing to investigate this issue.

3. monitoring 2026-09-28 13:48 UTC

   We have resolved the issue and are monitoring for changes.

4. resolved 2026-09-29 01:24 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General, pypi.org - Backends

Failure mode: Traffic surge or denial-of-service attack. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2026-09-23 11:02 UTC

   We are currently investigating this issue.

2. monitoring 2026-09-23 14:06 UTC

   The traffic has subsided and the cluster is stable. A performance fix has been drafted.

3. resolved 2026-09-23 17:05 UTC

   A fix has been implemented and the incident is closed.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: python.org - CDN, python.org - Backends, python.org - Downloads Backends, bugs.python.org, us.pycon.org

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2026-02-26 20:04 UTC

   We are currently investigating this issue

2. investigating 2026-02-26 20:40 UTC

   The root cause has been identified as an erroneous scaling operation that affected some application workloads. All services have been restored and are coming back online. We are monitoring to confirm full recovery.

3. resolved 2026-02-26 23:26 UTC

   The incident has been resolved.

4. postmortem 2026-02-26 23:26 UTC

   The incident has been resolved. ## Incident Report: Cluster-wide service outage **Duration**: ~36 minutes \(20:04–20:40 UTC\) **Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats,...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General, pypi.org - CDN, pypi.org - Backends

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-02-19 22:46 UTC

   We are currently investigating this issue.

2. monitoring 2026-02-19 22:56 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2026-02-19 23:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2026-01-07 22:42 UTC

   We are investigating why PyPI uploads are failing.

2. monitoring 2026-01-07 22:59 UTC

   We identified a database lock causing contention and stopped all upload backends to release all locks and are monitoring for recovery.

3. resolved 2026-01-07 23:00 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General, pypi.org - CDN, pypi.org - Backends, pypi.org - Email

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-11-12 15:46 UTC

   We are currently observing a slowdown in background requests processing for events on PyPI.org This could lead to stale web pages and API responses as the events system is not able to purge the caches. Our team is continuing to investigate.

2. monitoring 2025-11-12 15:50 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2025-11-12 16:26 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: Data delayed, stale, lost or corrupted, or needing replay after recovery. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. monitoring 2025-07-29 14:41 UTC

   The PyPI search index has degraded and is not returning results. A reindexing job has begun, and results will be available once it completes.

2. resolved 2025-07-29 19:58 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: Traffic surge or denial-of-service attack. Root cause: An external attack. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2025-05-24 22:57 UTC

   We are working to investigate a partial outage in the PyPI backends.

2. identified 2025-05-24 23:11 UTC

   We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.

3. identified 2025-05-24 23:37 UTC

   We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse. This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.

4. monitoring 2025-05-24 23:54 UTC

   Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact admin@pypi.org.

5. monitoring 2025-05-25 12:16 UTC

   We have removed the temporary block and are continuing to monitor.

6. resolved 2025-05-27 12:53 UTC

   We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. investigating 2025-04-24 12:02 UTC

   We're investigating elevated error rates for pypi.org

2. identified 2025-04-24 12:09 UTC

   We've identified an issue with database contention arising from locking.

3. monitoring 2025-04-24 12:09 UTC

   The locking process has been terminated and we're monitoring as things recover.

4. resolved 2025-04-24 12:18 UTC

   This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN, pypi.org - Backends

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2025-04-08 10:23 UTC

   We are working to resolve an outage impacting our /simple and /json apis.

2. monitoring 2025-04-08 10:26 UTC

   We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover.

3. monitoring 2025-04-08 10:31 UTC

   Errors have subsided, and we are monitoring as things continue to recover.

4. resolved 2025-04-08 11:00 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: files.pythonhosted.org - Files

Failure mode: CDN or edge layer failed or served bad content. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2025-03-11 12:13 UTC

   We are investigating issues with files from files.pythonhosted.org being served with corrupted data. A support ticket has been opened with our CDN provider and we are awaiting their response.

2. identified 2025-03-11 12:47 UTC

   This issue appears to be isolated to a single edge node in our CDN provider's network. They have validated the issue and are working to resolve.

3. monitoring 2025-03-11 12:50 UTC

   Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports.

4. monitoring 2025-03-11 13:05 UTC

   We are continuing to monitor for any further issues.

5. resolved 2025-03-11 13:39 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: docs.python.org - Backends, speed.pypy.org, python.org - Downloads Backends, bugs.python.org, wiki.python.org

Failure mode: Network, DNS, routing or connectivity failure. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-12-13 13:44 UTC

   We are currently investigating an infrastructure outage impacting python.org/downloads backends, docs.python.org backends, bugs.python.org backends, wiki.python.org backends, as well as other services.

2. investigating 2024-12-13 13:48 UTC

   We are continuing to investigate this issue.

3. investigating 2024-12-13 13:59 UTC

   We are continuing to investigate this issue.

4. identified 2024-12-13 14:00 UTC

   Private networking in our datacenter for these services is currently intermittent, we have opened a ticket with the provider requesting assistance.

5. identified 2024-12-13 14:23 UTC

   We are continuing to work on a fix for this issue.

6. identified 2024-12-13 14:45 UTC

   We are continuing to work on a fix for this issue.

7. monitoring 2024-12-13 14:58 UTC

   We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy.

8. resolved 2024-12-13 15:09 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2024-08-21 19:53 UTC

   An error serving /simple requests has caused timeouts for some requests. We have identified the issue and are working on a revert.

2. resolved 2024-08-21 20:23 UTC

   This incident has been resolved.

3. postmortem 2024-08-22 20:47 UTC

   **What happened?** PyPI experienced an outage which resulted in serving 500-level responses for certain requests, including requests to a subset of the [Simple Repository API](https://peps.python.org/pep-0503/), from approximately 19:00:00 UTC to 20:15:00 UTC. As a result, some...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: The text does not say enough to tell. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-08-09 18:22 UTC

   We are currently investigating an issue with PyPI uploads causing uploads to fail.

2. identified 2024-08-09 19:08 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2024-08-09 19:23 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2024-08-09 19:26 UTC

   Uploads are now being accepted properly.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN, pypi.org - Backends

Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-07-31 03:21 UTC

   We are currently investigating an issue with the PyPI backend due to a recent deploy.

2. resolved 2024-07-31 03:34 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-07-12 14:28 UTC

   We are performing upgrades to PyPI's database storage which may impact performance and availability of the service as they rollout.

2. resolved 2024-07-12 16:18 UTC

   This maintenance is complete.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General, pypi.org - CDN, pypi.org - Backends

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-06-08 13:43 UTC

   We will be performing database maintenance including version upgrades and storage re-configuration. This will result in several brief outages for the service.

2. monitoring 2024-06-08 13:58 UTC

   Underlying upgrades are complete, we are running database statistics updates to optimize performance.

3. monitoring 2024-06-08 14:26 UTC

   We are in the final steps of the upgrade and should see performance and availability stabilize in the next 30 minutes.

4. monitoring 2024-06-08 14:42 UTC

   All upgrade steps are complete and performance is stabilizing. A final maintenance task is taking place in the background which will minimally impact performance until it is complete.

5. resolved 2024-06-08 14:52 UTC

   All maintenance is complete and telemetry around database and application performance have stabilized. This incident is complete.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: python.org - Backends, us.pycon.org

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2024-06-05 15:38 UTC

   An outage in our infrastructure is impacting availability of us.pycon.org and python.org

2. monitoring 2024-06-05 15:57 UTC

   A cascading failure due to an expired certificate led to this outage. The certificate has been expired and systems are coming back online.

3. resolved 2024-06-05 16:05 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2024-03-28 02:16 UTC

   We have temporarily suspended new project creation and new user registration to mitigate an ongoing malware upload campaign.

2. resolved 2024-03-28 12:56 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-03-12 20:00 UTC

   A maintenance event requires us to pause our workers, which will delay cache purges and email delivery. We anticipate this will persist for up to 30 minutes.

2. monitoring 2024-03-12 20:19 UTC

   Workers have been restarted and are clearing backlog of enqueued tasks.

3. resolved 2024-03-12 20:41 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Failure mode: Login, tokens, certificates or credentials failed. Root cause: A configuration change. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2024-02-12 17:50 UTC

   A site wide issue is impacting many services hosted in the PSF infrastructure.

2. identified 2024-02-12 18:00 UTC

   Our internal Certificate Authority certificate has expired causing cascading outages across multiple services. An updated CA has been created and we are working to roll it out to restore service.

3. monitoring 2024-02-12 18:07 UTC

   Our new internal CA has been rolled out across the infrastructure and we are monitoring for remaining service impact.

4. resolved 2024-02-12 21:06 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: bugs.python.org

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2024-01-19 11:31 UTC

   bugs.python.org (and bugs.jython.org, and issues.roundup-tracker.org) is currently under maintenance.

2. monitoring 2024-01-19 12:04 UTC

   Maintenance and migration is complete. DNS may take some time to propagate. We will monitor as that occurs.

3. resolved 2024-01-19 12:18 UTC

   Maintenance is complete.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2023-12-27 20:11 UTC

   New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.

2. resolved 2024-01-02 15:59 UTC

   New user registrations are now permitted.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2023-11-24 23:14 UTC

   New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.

2. resolved 2023-12-04 13:47 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: python.org - Backends, pypi.org - Backends

Failure mode: Exhausted memory, disk, CPU, queues, connection pools or quotas. Root cause: Not enough capacity for the load. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-10-17 19:32 UTC

   We are investigating and responding to widespread issues with databases in our infrastructure showing poor performance due to cloud block storage.

2. identified 2023-10-17 19:44 UTC

   A burst balance for our cloud storage disks that host our databases was exhausted. We are going to migrate to storage that will allow us to respond by adding more IOPS.

3. resolved 2023-10-17 22:44 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-09-20 21:25 UTC

   We're receiving reports from users that attempts to download packages from PyPIs files archive may be failing when clients use IPv6 as their transport. Our network provider Fastly is also investigating the issue, see their status page: https://www.fastlystatus.com/incident/376079

2. monitoring 2023-09-20 22:17 UTC

   Fastly has routed traffic away from an impacted point of presence. We will monitor for further impact.

3. resolved 2023-09-20 22:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A schema or data migration caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2023-08-17 15:00 UTC

   A slow migration PyPI admins are attempting to apply is causing database contention leading to intermittent outages in the PyPI backends. We are working on mitigating the impact to end users.

2. monitoring 2023-08-17 15:05 UTC

   The migration has been completed and we are monitoring to ensure stability.

3. resolved 2023-08-17 15:08 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2023-08-01 19:42 UTC

   An update to our database connection library has caused some web requests including uploads to fail. We have identified the issue and are working to resolve.

2. identified 2023-08-01 19:56 UTC

   We have initiated a rollback of the change that introduced this outage.

3. monitoring 2023-08-01 20:03 UTC

   The revert has been deployed and we are monitoring the result.

4. resolved 2023-08-01 20:09 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: files.pythonhosted.org - Files

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-06-21 21:38 UTC

   We are currently investigating an issue with our object storage provider.

2. identified 2023-06-21 22:46 UTC

   We have identified an issue that is impacting latencies for files.pythonhosted.org, resulting in timeouts for a portion of requests.

3. identified 2023-06-21 22:52 UTC

   We're in communication with the object storage provider and are waiting for them to rollout a fix.

4. identified 2023-06-21 22:57 UTC

   We are also implementing our own change to guard against excessive latencies from the provider.

5. monitoring 2023-06-21 23:45 UTC

   Our mitigation has decreased overall latencies, but they remain elevated over normal operation. As our provider's service recovers we expect to see a return to baseline.

6. resolved 2023-06-22 01:59 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2023-05-20 16:02 UTC

   New user and new project name registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI...

2. resolved 2023-05-21 21:57 UTC

   Suspension has been lifted.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2023-05-11 22:12 UTC

   One of our providers is returning 500 responses during new package file uploads. They have completed their work, and errors should be subsiding. We are monitoring the situation.

2. resolved 2023-05-11 23:13 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2023-04-25 20:44 UTC

   Our object storage provider is currently experiencing an outage, causing PyPI uploads to fail.

2. monitoring 2023-04-25 21:51 UTC

   Our object storage provider has recovered from the outage and we are monitoring for any regressions.

3. resolved 2023-04-26 15:52 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2023-03-27 14:34 UTC

   We are currently investigating an intermittent outage with our upstream object storage provider, causing uploads for new releases to fail intermittently with a "503: Service Unavailable".

2. investigating 2023-03-27 17:29 UTC

   We are still observing intermittent upload failures and have contacted our upstream provider for support.

3. monitoring 2023-03-27 20:34 UTC

   The upstream provider has identified the issue as being resolved and we have seen failures subside over the last hour. We are continuing to monitor for regressions.

4. monitoring 2023-03-27 20:35 UTC

   We are continuing to monitor for any further issues.

5. resolved 2023-03-28 13:59 UTC

   We have seen errors subside over the last 16 hours, and uploads are succeeding as usual.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: python.org - CDN, python.org - Backends

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2023-03-20 12:55 UTC

   We are working to resolve a backend outage on python.org due to an outage in our search infrastructure.

2. monitoring 2023-03-20 13:00 UTC

   A fix has been implemented and we are monitoring the results.

3. resolved 2023-03-20 13:08 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Email

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. identified 2023-03-17 21:04 UTC

   All emails sent by PyPI are currently not being sent. Our email provider has suspended our account. We are currently determining how to proceed.

2. identified 2023-03-17 21:34 UTC

   We are working through the remediation process requested by our provider.

3. identified 2023-03-17 21:47 UTC

   We have identified the contributing factor that led to our complaint rate going above the acceptable threshold of our provider and are working to implement a fix that will satisfy their requirements.

4. monitoring 2023-03-17 22:28 UTC

   We have submitted our response with remediation efforts to the provider and are awaiting their response/resolution. Detail: PyPI replaced reCaptcha on our signup form with a less robust form of automation defense in 2018 due to reports from users in China not being able to sign...

5. monitoring 2023-03-17 23:12 UTC

   We have validated that failed email sends are queuing for delivery when our account suspension is lifted. For the moment we are holding off on mitigating by failing over to another provider. If we do not see resolution in 12 hours we will begin work to add another provider as a...

6. resolved 2023-03-18 01:00 UTC

   Our vendor has reinstated our sending ability after reviewing our remediation. Emails that were enqueued during the outage will send over the next 1-2 hours.

7. postmortem 2023-03-18 11:38 UTC

   ## Summary At 20:36 UTC on 2023-03-17, PyPI’s email provider enforced a suspension against our account. After our remediation efforts and response, our account was restored at 00:36 UTC 2023-03-18. In response, the PyPI Administrators did the following: * Audited our email logs,...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: files.pythonhosted.org - Files

Failure mode: A provider or third-party service this vendor depends on failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2023-01-10 13:59 UTC

   We are investigating issues with files.pythonhosted.org causing larger files to experience large latencies and timeouts.

2. investigating 2023-01-10 14:15 UTC

   We are working with our CDN provider to determine what is contributing to these latencies.

3. investigating 2023-01-10 16:26 UTC

   We're continuing to investigate with our CDN, but found a potentially related outage published by our backend that we are now tracking as well.

4. monitoring 2023-01-10 17:37 UTC

   We're seeing improvements after reports from our backend providers that their incident is resolved.

5. resolved 2023-01-10 17:43 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - General

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2022-11-11 22:24 UTC

   Beginning on Nov 6th, 2022, PyPI began to become inundated with failed authentication attempts due to a dictionary attack (https://en.wikipedia.org/wiki/Dictionary_attack). At its peak, the attack was attempting more than 260K passwords per hour, and at the time of writing has...

2. resolved 2022-11-17 17:47 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends, files.pythonhosted.org - Redirects, files.pythonhosted.org - Redirects Backends

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

Names an upstream provider: AWS.

1. investigating 2022-10-24 14:51 UTC

   The backend that hosts PyPI and associated services is experiencing a major outage, we are investigating.

2. monitoring 2022-10-24 15:05 UTC

   We have identified and resolved the reason for the outage and are monitoring to ensure it remains stable.

3. resolved 2022-10-24 15:44 UTC

   This incident has been resolved.

4. postmortem 2022-10-24 15:44 UTC

   ## Summary The cluster that hosts PyPI’s backends as well as multiple ancillary services experienced an outage during maintenance that interrupted access to services over HTTPS. ## Details From 2022-10-24 14:43 UTC until 2022-10-24 15:03 UTC, PyPI’s backends were not accessible...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - Backends

Failure mode: A code release introduced the failure. Root cause: The vendor does not say why it happened. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. investigating 2022-10-11 16:05 UTC

   We are currently investigating this issue.

2. monitoring 2022-10-11 16:08 UTC

   Some users experienced missing CSS files during a deployment of PyPI. Those files now appear to be serving, we are monitoring for further reports.

3. resolved 2022-10-11 16:25 UTC

   All CSS appears to be loading, we will investigate what occurred that caused CSS to fail to load during deployment.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2022-09-23 14:44 UTC

   TestPyPI may be unaccessible for up to an hour while we complete migration of the CDN configurations. Services may also be impacted after the migration due to changes in the configuration to better match PyPI.org. We will be monitoring throughout and after these changes.

2. resolved 2022-09-23 15:17 UTC

   This maintenance is complete. Please file an issue at https://github.com/pypa/pypi-support/issues/new?assignees=&labels=network&template=access-issues.yml if you experience issues with test.pypi.org after this migration.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Failure mode: CDN or edge layer failed or served bad content. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with low confidence (taxonomy v0.2).

1. identified 2022-09-12 18:24 UTC

   https://test.pypi.org's infrastructure is currently under maintenance. File downloads may be impacted while we migrate our CDN for the service.

2. resolved 2022-09-12 20:00 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: docs.python.org - Backends

Failure mode: Hardware, power, cooling or another physical failure. Root cause: Hardware or facility. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2022-08-26 13:19 UTC

   The backends for docs.python.org are currently unavailable. They became unresponsive at approximately 12:00 UTC and we have been unable to bring the cloud server back online via administration panel. We are working with our hosting provider to bring the host back online.

2. identified 2022-08-26 13:20 UTC

   Our CDN will continue to serve pages that are cached, but some docs.python.org URLs will not be able to load. In the event of cache expiry some pages that were previously cached may become unavailable.

3. monitoring 2022-08-26 13:43 UTC

   Host is back online and we are monitoring the status of the services it backs.

4. identified 2022-08-26 14:03 UTC

   The underlying cause is that the host is running on a host with degraded storage performance. We will be taking the host back offline to migrate it to a new underlying system.

5. resolved 2022-08-26 16:18 UTC

   This incident has been resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN, pypi.org - Backends

Failure mode: CDN or edge layer failed or served bad content. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. monitoring 2022-07-29 15:19 UTC

   In order to resolve an issue in our CDN cache caused by a misconfiguration of Surrogate-Keys, PyPI will need to issue a "Purge All" of our CDN to clear out lingering cached objects that are not accessible to purge individually. This purge is very likely to impact performance...

2. monitoring 2022-07-29 15:30 UTC

   Purge All has been issued and we are monitoring as the backends attempt to meet the request volume. We anticipate that the service will continue to stabilize and reach steady state over the next 30-90 minutes.

3. monitoring 2022-07-29 16:03 UTC

   Our backends have begun to stabilize as the caches are refilled. During the peak of the backend traffic from our CDN we recognized a few bottlenecks in our database layer and have provisioned additional storage IOPs in order to reduce impact of surges in the future.

4. monitoring 2022-07-29 16:30 UTC

   Error rates have returned to baseline, and latencies continue to return to normal across our services. We will continue to monitor as the database upgrades complete and caches re-fill.

5. resolved 2022-07-29 19:01 UTC

   Our CDN hit rate and latencies have recovered, this event is now resolved.

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Failure mode: Orchestration, control plane, service discovery or load balancing failed. Root cause: A failure at a provider or vendor they depend on. Tagged by claude-haiku-5-5 with medium confidence (taxonomy v0.2).

1. resolved 2022-07-28 19:00 UTC

   PyPI experienced an outage in its backends from 18:46-18:53 UTC on 2022-07-28. This outage stemmed from a cascading failure in the backend infrastructure that disrupted the management layer that deploys and secures PyPI's application servers. It is now resolved.

2. postmortem 2022-07-28 19:00 UTC

   PyPI experienced an outage in its backends from 18:46-18:53 UTC on 2022-07-28. This outage stemmed from a cascading failure in the backend infrastructure that disrupted the management layer that deploys and secures PyPI's application servers. ## Details Beginning at 17:24 UTC...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN, pypi.org - Email

Failure mode: A code release introduced the failure. Root cause: A bug in the vendor's own code. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2022-06-27 14:20 UTC

   A delay in our job queue is affecting CDN purges causing public pages to be stale and delays in email sending for notifications, email validation, password resets, etc.

2. investigating 2022-06-27 14:20 UTC

   We are continuing to investigate this issue.

3. identified 2022-06-27 15:05 UTC

   We've identified specific jobs in our queue that are causing processing delays and are working to remove them.

4. monitoring 2022-06-27 16:23 UTC

   We've disabled the specific tasks causing delays in the queue and are continuing to monitor the queue as it drains.

5. monitoring 2022-06-27 16:52 UTC

   A long running migration has caused our backends to become unhealthy, affecting all web traffic.

6. monitoring 2022-06-27 16:56 UTC

   Backends are recovering

7. resolved 2022-06-27 18:38 UTC

   This incident has been resolved.

8. postmortem 2022-06-28 14:21 UTC

   ### Summary PyPI initially experienced delays in features which used a centralized task queue \(such as purging the CDN cache and sending emails\) due to a poorly performing task type. This was followed by an outage due to the combination of the steps taken to mitigate the queue...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: pypi.org - CDN, pypi.org - Backends, files.pythonhosted.org - Redirects, files.pythonhosted.org - Redirects Backends

Failure mode: A configuration change caused the failure. Root cause: A configuration change. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. identified 2022-06-10 11:08 UTC

   Some cached responses are causing redirect loops for endpoints on the JSON API. We are working to determine how to clear these cached values without impacting the overall health of PyPI.

2. identified 2022-06-10 11:34 UTC

   We have started a task which will iterate over all projects and purge the cache for each individually. This will keep the PyPI backends from being overloaded by a completely bare cache. This process will take some time to complete, current estimate is 1-2 hours.

3. identified 2022-06-10 12:38 UTC

   Our mass purge operation is continuing. Based on the current rate that we're able to process all purges the processes should be complete in 45-60 minutes.

4. identified 2022-06-10 13:54 UTC

   The cache purge has cleared all but projects starting with the letter `p`. Our estimates failed to take into consideration the popularity of project names on PyPI starting with p 🙃

5. monitoring 2022-06-10 16:01 UTC

   All purges of JSON API documents have completed. Our backends are recovering from the added load of repopulating the entire cache. Any failed purges may result in latent redirect loops for specific projects or releases in the JSON API, these will self-resolve within 24 hours as...

6. resolved 2022-06-10 16:32 UTC

   This incident has been resolved.

7. postmortem 2022-06-10 16:32 UTC

   ## Summary PyPI's JSON API experienced an incident that caused redirect loops to occur for all clients when requesting many of the endpoints available on the service. These endpoints provide JSON documents describing projects and specific releases of projects hosted on...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Affected: files.pythonhosted.org - Files

Failure mode: A configuration change caused the failure. Root cause: A human mistake during operations. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. investigating 2022-06-02 22:20 UTC

   We are currently investigating this issue.

2. identified 2022-06-02 22:40 UTC

   The issue has been identified and a fix is being implemented.

3. monitoring 2022-06-02 22:40 UTC

   A fix has been implemented and we are monitoring the results.

4. resolved 2022-06-03 12:07 UTC

   We have resolved this incident, identified the cause of the breakage, and completed ensuring that it will not regress. An incident report will be provided shortly.

5. postmortem 2022-06-03 12:07 UTC

   ## Summary The cloud infrastructure that runs PyPI is managed via an automation tool which allows for configuration changes to be version controlled, reviewed, and deployed automatically. This tool includes multiple backends for state sharing and locking to facilitate...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.

Failure mode: Security incident or emergency response to a vulnerability. Root cause: An external attack. Tagged by claude-haiku-5-5 with high confidence (taxonomy v0.2).

1. resolved 2022-05-24 10:00 UTC

   This incident has been resolved.

2. postmortem 2022-05-24 17:33 UTC

   Takeover of the ctx project was reported on multiple channels overnight and was mitigated as of 6:07 AM Eastern. We confirmed via investigation that this compromise was of a single user account due to re-registration over an expired domain. The domain that hosted the users email...

Excerpts are shortened. Read the full incident on PyPI (Python Infrastructure)'s status page.