The Python Package Index Blog

PyPI has completed its second audit

Mike Fiedler — Fri, 10 Apr 2026 19:38:00 +0000

In 2023 PyPI completed its first security audit, and I am proud to announce that we have now completed our second external security audit.

This work was funded by the Sovereign Tech Agency, a supporter of Open Source security-related improvements, partnering with Trail of Bits to perform the audit. Thanks to ongoing support from Alpha-Omega, my role at the PSF enabled me to focus on rapid remediation of the findings.

This time around, there's no three-part series, as the scope was narrower, focused only on PyPI's codebase and behaviors. Read on for a summary of issues identified, their resolutions, and more details about the audit process.

The full audit report can be found on the Trail of Bits publication page. I highly recommend reading that for the fullest context first.

Findings

Here's a table of the findings, status, and links to the relevant pull requests where applicable:

ID	Title	Severity	Difficulty	Status
TOB-PYPI26-1	OIDC JTI anti-replay lock expires before JWT leeway window closes	Medium	High	Remediated
TOB-PYPI26-2	OIDC token minting is vulnerable to a TOCTOU race in JTI anti-replay	Low	High	Remediated
TOB-PYPI26-3	Verification badge bypass on the home page and download URLs	Low	Low	Remediated
TOB-PYPI26-4	Project-level token deletion audit events silently dropped due to data structure mismatch	Low	Low	Remediated
TOB-PYPI26-5	Password reset leaks privileged account status	Low	High	Remediated
TOB-PYPI26-6	IP ban bypass via macaroon API token authentication	Informational	High	Accepted
TOB-PYPI26-7	Moderators can modify organization applications due to a missing write permission check	Low	High	Remediated
TOB-PYPI26-8	Organization members can invite new owners due to a missing manage permission check	High	Medium	Remediated
TOB-PYPI26-9	TOTP replay prevention bypass via space normalization mismatch between validation and storage	Informational	High	Remediated
TOB-PYPI26-10	Wheel METADATA is served to installers without validation against upload metadata	Low	Low	Accepted
TOB-PYPI26-11	IDOR in API Token Deletion Allows Any Authenticated User to Delete Other Users' Macaroons	Low	High	Remediated
TOB-PYPI26-12	GitHub OIDC publisher lookup lacks issuer URL isolation for custom GHES issuers	Informational	High	Remediated 1, 2
TOB-PYPI26-13	Organization-scoped project associations persist after project transfer or removal	High	High	Remediated
TOB-PYPI26-14	Admin flag changes lack audit logging	Informational	High	Remediated

Of the 14 findings, I used a combination of Severity and Difficulty to determine which ones to work on first, and which ones to accept for now.

There were no Critical severity findings, 2 High, 1 Medium, 7 Low, and 3 Informational severity findings.

All but 2 findings have been remediated, and the remaining 2 are accepted for now. More details on the accepted findings below, but in general these were accepted because they require significant effort to remediate, and the risk they pose is relatively low.

To reiterate, the published report PDF goes into deeper detail about each finding, so I recommend reading that for the fullest context first.

Details

For some of the Remediated entries and all the Accepted ones, I'll go into more detail below.

TOB-PYPI26-1: OIDC JTI anti-replay lock expires before JWT leeway window closes

PyPI's Trusted Publishing flow uses OIDC JWTs issued by CI providers to mint short-lived upload tokens. Each JWT contains a jti (JWT Token Identifier) claim that should be single-use. To enforce this, we store each jti in cache (Redis) with an expiration of exp + 5 seconds, and check whether it already exists before accepting a new token.

The problem: PyJWT is configured with leeway=30, meaning it accepts tokens up to 30 seconds past their exp claim. This created a 25-second window (from exp + 5 to exp + 30) where the cache key had already been evicted, but the JWT still passed signature verification. During that window, a replayed token would pass both the signature check and the jti uniqueness check.

The fix was straightforward -- align the cache TTL to outlive the full leeway window by setting the expiration to exp + leeway + margin. I also took the opportunity to centralize these time-window constants so they're derived from a shared configuration, preventing future drift when one value is updated without the other.

TOB-PYPI26-6: IP ban bypass via macaroon API token authentication

Accepted for now.

PyPI administrators can ban IP addresses through the admin dashboard. The session authentication policy enforces this by checking the IP against the ban list before returning an identity. However, the macaroon (API token) authentication policy doesn't perform this same check. This means a user with a valid API token could continue uploading packages from a banned IP address.

I've accepted this finding for now. IP bans are a relatively blunt tool that we use sparingly, introduced late last year to mitigate a specific wave of abuse. The practical risk here is low - if we've identified a malicious actor, we have other mechanisms to disable their account entirely. That said, it's a gap worth closing, and we'll likely address it as part of broader work on making security controls consistent across all authentication methods.

TOB-PYPI26-8: Organization members can invite new owners due to a missing manage permission check

This was the highest-severity finding in the audit, and one I prioritized immediately.

The manage_organization_roles view handled both GET (viewing the people page) and POST (sending invitations) under a single @view_config decorator that only required OrganizationsRead permission. This meant any organization member could send invitations with any role - including Owner - to any PyPI user.

The irony is that we already had the correct pattern elsewhere in the codebase. Views like resend_organization_invitation and change_organization_role correctly use separate @view_config decorators for GET and POST with distinct permission requirements. This one was simply missed.

The fix was to split the view configuration: GET requires OrganizationsRead, POST requires OrganizationsManage. As part of the audit, Trail of Bits also developed a custom CodeQL query to detect this class of issue - views that handle state-changing POST requests under a read-only permission check. I'll integrate that into our CI to catch this pattern going forward.

TOB-PYPI26-10: Wheel METADATA is served to installers without validation against upload metadata

Accepted for now.

This is a nuanced one. When a wheel is uploaded to PyPI, we store two independent sources of metadata: the form-declared metadata from the upload request (which populates the database and the JSON API), and the embedded .dist-info/METADATA file extracted from the wheel itself (which is served via PEP 658 to pip for dependency resolution).

These two sources are never compared. In theory, an attacker could embed hidden dependencies in the wheel's METADATA that pip would install, but that security tools querying the JSON API would never see.

We've accepted this for now because the fix is non-trivial. Properly validating embedded metadata against upload metadata touches a core part of how we handle uploads, and requires careful consideration of edge cases across the ecosystem. This is something we want to get right rather than rush, and involves a fair amount of database changes, including data backfills.

TOB-PYPI26-13: Organization-scoped project associations persist after project transfer or removal

This was the other High-severity finding, and a subtle one.

When a project is transferred between organizations, the OrganizationProject junction record is correctly deleted and recreated. However, the TeamProjectRole records - which grant a team's members access to specific projects - were not cleaned up during the transfer.

This meant that if LexCorp Organization had a "release-engineers" team with Owner-level access to a project, and that project was transferred to Organization OsCorp, the LexCorp team's members would silently retain full access to the project. Worse, the receiving organization had no visibility into these stale associations - team-granted permissions are resolved at ACL evaluation time and don't appear as individual collaborator entries in the UI.

The fix in pypi/warehouse#19749 ensures that TeamProjectRole records belonging to the departing organization are cleaned up when a project is transferred. Auditing database records proved that this has not happened in the past, so I am confident there have been no such transfers with dangling permissions. I also added defensive filters in the project's ACL computation to verify that a team's organization matches the project's current organization before granting permissions, so stale records can't grant access regardless of how they're orphaned.

Summary

Working with Trail of Bits was again a pleasure. The team were thorough, communicative, and clearly understood the nuances of a system like PyPI - where the threat model spans everything from CI/CD token replay to metadata integrity for millions of downstream users.

Beyond the 14 findings, the audit also produced proposal reviews for features I'm considering (per-org Trusted Publishers, TOTP hardening, and more), as well as custom CodeQL queries to integrate into our CI/CD pipeline.

This audit was funded in partnership with the Sovereign Tech Agency, which continues to support security improvements across the Open Source ecosystem.

My work at the Python Software Foundation is supported by Alpha-Omega.

Incident Report: LiteLLM/Telnyx supply-chain attacks, with guidance

Seth Larson — Thu, 02 Apr 2026 15:04:45 +0000

This post will drill deeper into two recent supply chain exploits, targeting users of popular PyPI packages - litellm & telnyx. We also provide Python developers and maintainers with guidance on what they can do to prepare and protect themselves from future incidents.

What happened with LiteLLM and telnyx?

After an API token exposure from an exploited Trivy dependency releases of the packages litellm and telnyx were published to PyPI containing credential harvesting malware.

The malware ran on install, harvesting sensitive credentials and files, and exfiltrating to a remote API. More details published in the advisory for LiteLLM (PYSEC-2026-2), the LiteLLM blog post about the incident, the advisory for telnyx (PYSEC-2026-3) and the telnyx notice.

After contacting the litellm and telnyx maintainers, Mike and Seth collaborated with each team on steps forward, including token rotation, release removals, and recommendations around further security practices like using Trusted Publishers which both projects have since adopted.

Why is this malware different?

This class of malware is different from most malware published to PyPI, which are mostly published as new packages, either as typosquats or with a plan to share the package with others and hope they install it. This malware is "injected" into open source packages that are already in widespread use. The malware injection occurs one of two ways:

Targeting open source projects with insecure repositories, release workflows, or authentication
Targeting developers installing the latest versions of open source projects and exfiltrating API tokens and keys

Using the API tokens and keys gathered from developer machines, the malware authors are able to further compromise other open source packages if API tokens for PyPI or GitHub are exfiltrated. This cycle continues as long as it is effective in exfiltrating more credentials.

What is PyPI doing to mitigate malware?

With daily volume of ~700-800 new projects created daily on PyPI, this poses a scaling challenge. PyPI partners with security researchers from our community who regularly scan and report malware through elevated channels to facilitate quicker remediation times.

Below see the timeline of events.

During the window of attack, the exploited versions of litellm were downloaded over 119k times.

PyPI received 13 inbound reports from concerned users, leveraging the "Report project as malware" feature, added back in 2024, accelerating the review/action.

From upload to first report: 1h 19m
First report to quarantine: 1h 12m
From upload to quarantine (total exposure time): 2h 32m

LiteLLM is typically installed ~15-20 million times per week. Averaging this out to an "installs per minute" rate nets a value between ~1700 installs per minute. This means between ~40-50% of all installs of LiteLLM were unpinned and fetching the latest version on each install invocation.

Because these systems are pulling latest, this leaves very little time for researchers and PyPI admins to report, triage, and quarantine malware when published to a popular package. Read on for information on "dependency cooldowns"

PyPI's remediation response to telnyx was autonomously taken thanks to our pool of trusted reporters. These reporters add weight to any given report, triggering an automated quarantine feature. Subscribe to the PyPI blog for a future update about our automatic quarantining system.

From upload to first report: 1h 45m
First report to quarantine: 1h 57m
Form upload to quarantine (total exposure time): 3h 42m

Below are a few methods to make your usage of Python packages from PyPI more secure and to avoid installing malware.

Protecting yourself as a developer

Dependency Cooldowns

One method to avoid "drinking from the firehose" and allow time for malware to be detected and remediated is using "Dependency Cooldowns". Dependency cooldowns is a strategy for package installers (like pip, uv, etc) to avoid installing packages that have very recently been published to PyPI. By doing so, this allows for human operators like security researchers and PyPI admins a chance to respond to reports of malware.

Dependency cooldowns work best when they are configured "globally" on a developer machine and then passively protect developers from compromises on every invocation of pip or uv. Setting a relative value like "3 days" ("P3D" per RFC 3339) means packages that are newer than 3 days will not be installed.

uv already supports setting relative dependency cooldown via --exclude-newer. You can configure the option globally (in ~/.config/uv/uv.toml) or per-project in your pyproject.toml:

[tool.uv] exclude-newer = "P3D" # "3 days" in RFC 3339 format

Relative dependency cooldowns are coming soon to pip v26.1 which should be available in April of this year. Once they are available you can set the option in your pip.conf file (~/.config/pip/pip.conf):

[install] uploaded-prior-to = P3D

Starting in pip v26.0 you can set absolute dependency cooldowns with pip from the command-line, likely paired with another tool like date to calculate an absolute date from a relative offset like "3 days":

python -m pip install \ --uploaded-prior-to=$(date -d '-3days' -Idate) \ simplepackage

Applying dependency cooldowns everywhere isn't a silver bullet, though! There are certain situations where you do want the latest version of a package as soon as possible, like when applying patches for vulnerabilities. Dependency cooldowns should be paired with a vulnerability scanning strategy so security updates for your application's dependencies aren't waiting to be deployed. For example, Dependabot and Renovate both bypass dependency cooldowns by default for security updates.

You can manually bypass a dependency cooldown in pip and uv by setting a value of "the current day" to get the actual latest release:

python -m pip install \ --uploaded-prior-to=P0D \ simplepackage==26.3.31

Locking Dependencies

Installing a package from PyPI without a "lock" means that it's possible to receive new code every time you run pip install. This leaves the door open for a compromise of a package to immediately get installed and execute malware on the installing systems.

If you're a developer of an application using PyPI packages you should be using lock files both for security and reproducibility of your application. Some examples of tools which produce lock files for applications are:

uv lock
pip-compile --generate-hashes
pipenv

Note that pip freeze doesn't create a lock file: a lock file must include checksums / hashes of the package archives to be secure and reproducible. pip freeze only records packages and their versions. pip is working on experimental support for the pylock.toml standard through the pip lock sub-command.

Protecting your project as an open source maintainer

If you are a maintainer of an open source project on PyPI, you can do your part to protect your users from compromises. There are three approaches we recommend:

securing your release workflows
using Trusted Publishers
adding 2FA to all accounts associated with open source development

Securing release workflows

If you're using a continuous deployment system to publish packages to Python: these workflows are targets for attackers. You can prevent most of the danger by applying a handful of security recommendations:

Avoid insecure triggers. Workflows that can be triggered by an attacker, especially with inputs they control (such as PR titles, branch titles) have been used in the past to inject commands. The trigger pull_request_target from GitHub Actions in particular is difficult to use securely and should be avoided.
Sanitize parameters and inputs. Any workflow parameter or input that can expand into an executed command carries potential to be used by attackers. Sanitize values by passing them as environment variables to commands to avoid template injection attacks.
Avoid mutable references. Lock or pin your dependencies in workflows. Favor using Git commit SHAs instead of Git tags, as tags are writeable. Maintain a lock file for PyPI dependencies used in workflows.
Use reviewable deployments. Trusted Publishers for GitHub supports "GitHub Environments" as a required step. This makes publishing your package to PyPI require a review from your GitHub account, meaning a higher bar for an attacker to compromise.

If you are using GitHub Actions as your continuous deployment provider, we highly recommend the tool "Zizmor" for detecting and fixing insecure workflows.

Trusted Publishers over API tokens

If the platform you use to publish to PyPI supports Trusted Publishers (GitHub, GitLab, Google Cloud Build, ActiveState) then you should use Trusted Publishers instead of API tokens.

PyPI API tokens are "long-lived", meaning if they are exfiltrated by an attacker that attacker can use the token at a much later date even if you don't detect the initial compromise. Trusted Publishers comparatively uses short-lived tokens, meaning they need to be used immediately and don't require manual "rotating" in the case of compromise.

Trusted Publishers also provides a valuable signal to downstream users through Digital Attestations. This means users can detect when a release hasn't been published using the typical release workflow, likely drawing more scrutiny.

Adding 2FA to open source development accounts

We may be starting to sound like a broken record, but 2FA should be used for all accounts associated with open source development: not just PyPI. Think about accounts like version control / software forges (GitHub, GitLab, Codeberg, Forgejo) and your email provider. PyPI has required 2FA to be enabled to publish packages since the beginning of 2024, but enabling phishing-resistant 2FA like a hardware key can protect you further.

How can you support this kind of work?

Security work isn't free. You can support security work on the Python Package Index by supporting the Python Software Foundation (PSF). If you or your organization is interested in sponsoring or donating to the PSF so we can continue supporting Python, PyPI, and its community, check out the PSF's sponsorship program, donate directly, or contact our team at sponsors@python.org!

Mike Fiedler and Seth Larson's roles as PyPI Safety & Security Engineer and Security Developer-in-Residence at the Python Software Foundation are supported by Alpha Omega.

Welcome to the PyPI Blog

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

Today, we're excited to launch blog.pypi.org, the official blog of the Python Package Index.

One of the most common refrains I hear from Python community members, irrespective of if they have been around for days or years, is "I didn't realize that PyPI...". Followed by something along the lines of:

Could do that
Is operated with so few resources¹
Only had 3 administrators²
Needs more help

That is certainly in part due to the fact that the people³ who make PyPI "happen" are humble, and do so with the limited time they have to commit to the project among their other responsibilities. But its also because writing is hard. Harder still when there is not a convenient tool at hand. We're hoping that a modern and extensible stack⁴ for this blog will make writing about and sharing our work more frictionless.

The PyPI team will use this space to communicate with PyPI users, provide announcements about new features and updates, interesting technology, as well as share information and context around PyPI and related efforts of the Python Software Foundation.

Stay tuned for more, we're very glad you're here! 💜

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

We're thrifty and picky about what we run, but Fastly is the single most important factor allowing us to get by with comparatively few resources for our backends. ↩
That was until we welcomed Mike Fiedler as an admin in 2023. ↩
The PyPI Administrator team is Donald Stufft, Dustin Ingram, Mike Fiedler, and Ee Durbin. The PyPI Moderator team is Yeray Díaz, Cristián Maureira-Fredes, Joachim Jablon, Jason Madden, Pradyun Gedam, and Stargirl Flowers. ↩
Shout-out: This blog is hosted on Read the Docs, using the MkDocs static site generator, and the Material for MkDocs theme. Many thanks to the maintainers of those projects for your contributions, we're very excited to have our blog contents live side-by-side with our other documentation and code in pypi/warehouse! ↩

Introducing 'Trusted Publishers'

Dustin Ingram — Wed, 18 Mar 2026 18:14:42 +0000

Starting today, PyPI package maintainers can adopt a new, more secure publishing method that does not require long-lived passwords or API tokens to be shared with external systems.

About trusted publishing

"Trusted publishing" is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use username/password combinations or manually generated API tokens to authenticate with PyPI when publishing.

Instead, PyPI maintainers can configure PyPI to trust an identity provided by a given OpenID Connect Identity Provider (IdP). This allows allows PyPI to verify and delegate trust to that identity, which is then authorized to request short-lived, tightly-scoped API tokens from PyPI. These API tokens never need to be stored or shared, rotate automatically by expiring quickly, and provide a verifiable link between a published package and its source.

Using trusted publishing with GitHub Actions

PyPI currently supports trusted publishing with GitHub Actions, using their support for OpenID Connect.

After configuring PyPI to trust a given GitHub repository and workflow, users of the PyPA's 'pypi-publish' GitHub Action can adopt trusted publishing by removing the username and password fields from their workflow configuration, and adding permissions to generate an identity token:

jobs: pypi-publish: name: upload release to PyPI runs-on: ubuntu-latest + permissions: + # IMPORTANT: this permission is mandatory for trusted publishing + id-token: write steps: # retrieve your distributions here - name: Publish package distributions to PyPI uses: pypa/gh-action-pypi-publish@release/v1 - with: - username: __token__ - password: ${{ secrets.PYPI_TOKEN }}

Using the PyPA's GitHub action is strongly recommended, but not required. More details on how to manually exchange tokens are available in our documentation.

Additional security hardening is available

PyPI package maintainers can further increase the security of their release workflows by configuring trusted publishers to only release from a specific GitHub Actions environment.

Configuring an environment is optional, but strongly recommended: with a GitHub environment, you can apply additional restrictions to your trusted GitHub Actions workflow, such as requiring manual approval on each run by a trusted subset of repository maintainers.

Unblocking future security improvements

In addition to making publishing more secure now, the availability of trusted publishers unblocks additional future security improvements for PyPI.

Configuring and using a trusted publisher provides a 'strong link' between a project and its source repository, which can allow PyPI to verify related metadata, like the URL of a source repository for a project¹. Additionally, publishing with a trusted publisher allows PyPI to correlate more information about where a given file was published from in a verifiable way.

Finally, although trusted publishers is currently limited to GitHub Actions, much of the underlying work that went into making this feature possible is generalizable and not specific to a single publisher. We're interested in supporting the ability to publish from additional services that provide OpenID Connect identities.

Get started today

To get started with using trusted publishers on PyPI, see our documentation here: https://docs.pypi.org/trusted-publishers/.

Acknowledgements

Funding for this work was provided by the Google Open Source Security Team, and much of the development work was performed by Trail of Bits, with special thanks to contributor William Woodruff.

Many thanks as well to Sviatoslav Sydorenko, maintainer of the PyPA's 'pypi-publish' GitHub Action for his quick and timely work to add support for trusted publishers in the action.

Finally, we want to thank all our beta testers, including GitHub staff, for working with us to ensure this feature is intuitive and useful, and for providing valuable feedback to improve this feature along the way.

Dustin Ingram is a maintainer of the Python Package Index.

Currently, information such as this are provided by the uploader and are not verified as accurate by PyPI. ↩

Introducing PyPI Organizations

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

Today, we are rolling out the first step in our plan to build financial support and long-term sustainability of the Python Packaging Index (PyPI), while simultaneously giving our users one of our most requested features: organization accounts.

Introducing Organizations

Organizations on PyPI are self-managed teams, with their own exclusive branded web addresses. Our goal is to make PyPI easier to use for large community projects, organizations, or companies who manage multiple sub-teams and multiple packages. We’re making organizations available to community projects for free, forever, and to corporate projects for a small fee. Additional priority support agreements will be available to all paid subscribers, and all revenue will go right back into PyPI to continue building better support and infrastructure for all our users.

Increasing sustainability and support

In the last year PyPI served 235.7 billion downloads for the 448,941 projects hosted there. This means that since the previous period, PyPI saw a 57% annual growth in download counts and bandwidth alike. Having more people using and contributing to Python every year is an fantastic problem to have, but it is one we must increase organizational capacity to accommodate.

Increased revenue for PyPI allows it to become a staffed platform that can respond to support requests and attend to issues in a timeframe that is significantly faster than what our excellent (but thinly spread) largely volunteer team could reasonably handle.

Organizations are opt-in

We want to be very clear—these new features are completely optional. If features for larger projects don't sound like something that would be useful to you as a PyPI maintainer, then there is no obligation to create an organization and absolutely nothing about your PyPI experience will change for you.

A basis for future features

We look forward to discussing what other features PyPI users would like to see tackled next. We know there are lots of ideas out there around safety, security and usability and we’re looking forward to hearing about anything that you think would benefit the community. And in the spirit of open source, we welcome your feedback on what's on offer so far.

Get started today

Both community projects (non-profits, NGO’s, hobbyists, etc) and corporate teams can sign up to request their organization name starting today. Submissions will begin seeing review and approval in the coming weeks, and corporate teams will be able to finalize their signup with billing details in May.

Acknowledgements

Organization feature development was approved by the Packaging Working Group and funded through the Python Software Foundation's sponsorship program -- thanks to our sponsors for making this work possible.

We especially want to thank Bloomberg for funding our Packaging Project Manager, Shamika Mohanan.

We are also grateful for the many generous PyPI users who shared their perspectives with Shamika, which laid the foundation for these new features.

And thanks as well to our beta testers, including the following ✨new✨ PyPI organizations:

Python Cryptographic Authority (https://pypi.org/org/pyca/)
The Pallets Project (https://pypi.org/org/pallets/)
certifi (https://pypi.org/org/certifi)

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

Announcing the PyPI Safety & Security Engineer role

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

We are pleased to announce Amazon Web Services (AWS) as the inaugural Security Sponsor for PyPI, investing $144,000 over one year to fund key enhancements to PyPI infrastructure and operations, including the creation of a new “PyPI Safety & Security Engineer” role.

This role builds on our existing long term partnership with AWS as one of the top sponsors of the Python Software Foundation for the last five years, which has included in-kind donations of cloud computing infrastructure and services to support PyPI. The role will complement our previously announced role for a PSF Security Developer in Residence and will work closely with the person hired for that role (to be announced soon).

This funding also builds on previously successful project-focused funding efforts, such as the 2018 full-stack rewrite of PyPI, the introduction of internationalization and localization for PyPI, as well as 2FA and WebAuthn support.

We expect this partnership to tangibly improve the experience for all PyPI users, from consumers downloading packages, to package maintainers, to large corporate teams. Some of the outcomes we are working toward over the next year include increased support for package maintainers including multi-maintainer projects, improvements to reporting infrastructure for malicious projects, as well as a reduced response time for malware reports and account recovery requests.

The job posting can be found here, and applications for the role are open until June 1st. Similar to existing developer-in-residence roles, the contract for this role will be for a one year period, and the PSF will be actively engaging with our sponsors and supporters to renew funding for the role in subsequent years.

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

Removing PGP from PyPI

Donald Stufft — Wed, 18 Mar 2026 18:14:42 +0000

If you are someone who is currently uploading signatures, your package uploads will continue to succeed, but any PGP signatures will be silently ignored. If you are someone who is currently downloading PGP signatures, existing signatures SHOULD continue to be available ¹, but no new signatures will be made available. The related API fields such as has_sig have all been hardcoded to always be False.

Historically, PyPI has supported uploading PGP signatures alongside the release artifacts in an attempt to provide some level of package signing. However, the approach used had long standing, documented issues which had previously lead us to deemphasize the support for PGP signatures over time by removing them from the PyPI web user interface.

PyPI has continued to support uploading these signatures in the hope that there might be some systems out there that found them useful. Recently though, an examination of the signatures on PyPI has revealed to us that the current support for PGP signatures is not proving useful.

In the last 3 years, about 50k signatures had been uploaded to PyPI by 1069 unique keys. Of those 1069 unique keys, about 30% of them were not discoverable on major public keyservers, making it difficult or impossible to meaningfully verify those signatures. Of the remaining 71%, nearly half of them were unable to be meaningfully verified at the time of the audit (2023-05-19) ².

In other words, out of all of the unique keys that had uploaded signatures to PyPI, only 36% of them were capable of being meaningfully verified ³ at the time of audit. Even if all of those signatures uploaded in that 3 year period of time were made by one of those 36% of keys that are able to be meaningfully verified, that would still represent only 0.3% of all of those files.

Given all of this, the continued support of uploading PGP signatures to PyPI is no longer defensible. While it doesn't represent a massive operational burden to continue to support it, it does require any new features that touch the storage of files to be made aware of and capable of handling these PGP signatures, which is a non zero cost on the maintainers and contributors of PyPI.

Donald Stufft is a PyPI administrator and maintainer of the Python Package Index since 2013.

For now, but they may be removed in the future. ↩
These could be because the original signature was made incorrectly and never had a binding signature to a associated key identity, or because the signature was present but had since expired. ↩
We use meaningfully verified to mean that the signature was valid and the key that made it was not expired and had binding identify information that could tell us if this key was the correct key. ↩

PyPI was subpoenaed

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

In March and April 2023, the Python Software Foundation (PSF) received three (3) subpoenas for PyPI user data. All three subpoenas were issued by the United States Department of Justice. The PSF was not provided with context on the legal circumstances surrounding these subpoenas. In total, user data related to five (5) PyPI usernames were requested.

The data request was:

"Names (including subscriber names, user names, and screen names);"
"Addresses (including mailing, residential addresses, business addresses, and email addresses);"
"Connection records;"
"Records of session times and durations, and the temporarily assigned network address (such as Internet Protocol addresses) associated with those sessions;"
"Length of service (including start date) and type of services utilized;"
"Telephone or instrument numbers (including the registration Internet Protocol address);"
"Means and source of payment of any such services (including any credit card or bank account number) and billing records;"
"Records of all Python Package Index (PyPI) packages uploaded by..." given usernames
"IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames

The privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators, and we are committed to protecting user data from disclosure whenever possible. In this case, however, PSF determined with the advice of counsel that our only course of action was to provide the requested data. I, as Director of Infrastructure of the Python Software Foundation, fulfilled the requests in consultation with PSF's counsel.

We have waited for the string of subpoenas to subside, though we were committed from the beginning to write and publish this post as a matter of transparency, and as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023.

Next Steps

PyPI and the PSF are committed to the freedom, security, and privacy of our users.

This process has offered time to revisit our current data and privacy standards, which are minimal, to ensure they take into account the varied interests of the Python community. Though we collect very little personal data from PyPI users, any unnecessarily held data are still subject to these kinds of requests in addition to the baseline risk of data compromise via malice or operator error.

As a result we are currently developing new data retention and disclosure policies. These policies will relate to our procedures for future government data requests, how and for what duration we store personally identifiable information such as user access records, and policies that make these explicit for our users and community.

Please continue to watch this blog for related announcements as policies are finalized, published, and implemented.

Details

In order to provide as much transparency as possible, the following will detail the shape of and extent of the data that was contained in the responses to these subpoenas.

We will not be releasing the usernames involved publicly or to the users themselves.

1) Names (including subscriber names, user names, and screen names);

These were confirmed via our database records

select id, username, name from users where username = '{USERNAME}';

Returning:

id | UUID for USERNAME username | PyPI username name | Display name for user

And are also publicly available at https://pypi.org/user/{USERNAME}/.

PyPI allows users to delete their accounts.

PyPI does not allow for the username field to be changed without admin intervention, and no such intervention has occurred for the users in question. If they had, we would have provided records of those changes.

PyPI does allow for user to update their display names and keeps no record of the history of these changes.

2) Addresses (including mailing, residential addresses, business addresses, and email addresses);

PyPI only stores email addresses for individual users. No physical addresses are stored. Organization accounts who have signed up for billing (not yet live) will be required to provide a billing address that validates to their payment method.

These were sourced from our database records and is private to PyPI.

select email, user_id from user_emails where user_id ='{UUID for USERNAME}';

Returning:

email | An email address user_id | UUID for USERNAME

PyPI allows for users to add and remove email addresses without admin intervention. Records of these changes are kept, and no such changes were observed in our records for the usernames in question. If they had, we would have provided records of those changes.

3. Connection records

3a. Project events

PyPI retains records of all changes to projects on the index, and has since 2002-11-01 17:11:36 UTC.

These were confirmed via our database records

select * from journals where submitted_by='{USERNAME}' order by submitted_date;

Returning:

id | An auto incrementing integer representing the "Serial" name | Name of a PyPI Project version | Version of a PyPI Release if relevant action | Description of the action taken against the Project/Release submitted_date | ISO-8601 datetime in UTC submitted_by | PyPI Username submitted_from | IP Address

and with the exception of the submitted_by (PyPI username) and submitted_from (IP Address) columns are publicly available via our XMLRPC API.

3b. User events

PyPI retains records of critical user events including account creation, emails sent, email address changes, logins, and login failures. See this list for the comprehensive set of events recorded.

These were sourced from our database records

select * from user_events where source_id = '{UUID for USERNAME}' order by time desc;

Returning:

id | UUID of the event source_id | UUID for USERNAME tag | EventTag time | ISO-8601 datetime in UTC ip_address_string | IP Address additional | JSONB metadata about the event ip_address_id | UUID of associated IPAddress object

and are private to PyPI.

4. Records of session times and durations, and the temporarily assigned network address (such as Internet Protocol addresses) associated with those sessions;

PyPI does not record session durations.

Session creation (Login) times were provided as a synopsis of the data in 3b.

Sessions are not created for uploads, but the associated login events for project uploads were provided as a synopsis of the data in 3a.

5. Length of service (including start date) and type of services utilized;

PyPI retains records of the date that a user account was created, as well as the last time it was successfully logged in by any method (web UI or command line tool for upload).

select date_joined, last_login from users where username = {USERNAME}

Returning:

date_joined | ISO-8601 datetime in UTC last_login | ISO-8601 datetime in UTC

These were sourced from our database records and are private to PyPI.

Types of service utilized are "standard" to PyPI and include the ability to create projects, releases, and distribution files for downloads.

6. Telephone or instrument numbers (including the registration Internet Protocol address);

A synopsis of all IP Addresses for each username from previous records were shared.

These were sourced from our database records and are private to PyPI.

7. Means and source of payment of any such services (including any credit card or bank account number) and billing records;

PyPI has no cost to use for individual users and no such payment records or billing records exist.

8. Records of all Python Package Index (PyPI) packages uploaded by the given usernames

A list of all past and any current projects associated with each username was provided.

These were sourced from our database records and for past projects, are private to PyPI.

9. IP download logs of any Python Package Index (PyPI) packages uploaded by the given usernames

PyPI does not retain download logs for packages which include IP addresses. Download logs are processed by a pipeline which includes GeoIP information reported by our CDN only.

These records were sourced from the Google BigQuery Public dataset with the following queries:

SELECT * FROM `bigquery-public-data.pypi.file_downloads` WHERE project IN ({LIST OF PROJECT NAMES FROM 8}) AND timestamp > '{START OF PERIOD IN QUESTION}';

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

Securing PyPI accounts via Two-Factor Authentication

Donald Stufft — Wed, 18 Mar 2026 18:14:42 +0000

One of the key security promises that PyPI makes is that when you're downloading something, that only the people associated with that project are going to be able to upload, delete, or otherwise modify a project. That when you look at that project and see that it is owned by someone that you trust, that you can be assured that nobody else is making changes to that package on PyPI.

This promise is predicated on the security of each and every individual account on PyPI used to create and maintain a Python project. In the past we've taken steps to safeguard these accounts by blocking compromised passwords, strong 2FA support using TOTP and WebAuthN, support for API tokens with offline attenuation, enrolling the most downloaded projects into mandatory 2FA, and enabling short lived tokens for upload.

Today, as part of that long term effort to secure the Python ecosystem, we are announcing that every account that maintains any project or organization on PyPI will be required to enable 2FA on their account by the end of 2023.

Between now and the end of the year, PyPI will begin gating access to certain site functionality based on 2FA usage. In addition, we may begin selecting certain users or projects for early enforcement.

What can I do to prepare?

The most important things you can do to prepare are to enable 2FA for your account as soon as possible, either with a security device (preferred) or an authentication app and to switch to using either Trusted Publishers (preferred) or API tokens to upload to PyPI.

Why Use 2FA?

Account takeover attacks typically stem from someone using an insecure password: perhaps it was easy to guess, or it was reused and appeared in a breach. With that insecure password, an attacker is able to gain control over a maintainers account and begin to take actions as if they were that user.

This is particularly problematic on a site like PyPI, where the actions that a person can take include releasing software that might be used by people world wide, allowing an attacker to install and execute software on unsuspecting user's computers. Account takeover attacks have been previously used to compromise PyPI users in this way.

Two-factor authentication immediately neutralizes the risk associated with a compromised password. If an attacker has someone's password, that is no longer enough to give them access to that account.

Why every project or organization maintainer?

There's two ways to think about this question:

Why every project and organization maintainer instead of just some subset of them (based on popularity, purpose, whether that user uses their account, etc)?
Why only maintainers and not every single user?

Not every account on PyPI has the same value to an attacker. An account with access to the most downloaded project on PyPI can be used to attack far more people than an account with access to the least downloaded project.

However, it only takes one compromised project in someone's dependency set to compromise their computer. The attacker doesn't care if they get you from a widely used or a niche project, just that they got you. Even worse, once compromised, an attacker can extend that attack to attack other systems, including other projects on PyPI that the now compromised person maintains.

Given that it only takes one compromised project, no matter how many downloads it gets ¹, to compromise someone we want to ensure that every project is being protected by 2FA.

On the flipside, an account without access to any project cannot be used to attack anyone ² so it is a very low value target.

We recognize that enabling 2FA for an account imposes a non zero cost both for the owner of that account and for PyPI ³, so forcing that on accounts that cannot affect anyone but themselves is not an effective use of limited resources. In addition, from a practical view, the standard 2FA flow that most people are used to and that PyPI implements also involves adding a 2FA token to an existing account rather than forcing it to be part of the registration flow.

Our expectation is that for users who currently are not a project maintainer or organization member, the ultimate experience will be whenever they attempt to take some action that would require them to add 2FA (creating a new project, accepting an invite, making an organization, etc) they will be prompted to add 2FA to their account before they can proceed.

Why now?

The direction of many projects in or serving the Open Source community in the last 5-10 years has been an increasing focus on supply chain security, preventing attacks that are being delivered through the "supply chain", namely the services and tooling used to create and consume software.

In July of 2022, we announced a security key giveway in conjunction with a plan to begin mandating 2FA for the top 1% of projects on PyPI by download count.

The initial announcement of that mandate to the top 1% of projects and the related giveaway was met with mixed reactions, ranging from people applauding the effort to people deciding to distribute their code on places other than PyPI. We planned to limit this to the projects in the top 1% of downloads because those are likely he highest value targets for an attacker, and because we were concerned about the impact of such a mandate on both the maintainers of projects on PyPI, and on the support burden of the PyPI team itself.

At that time last year, we did not have any plans or expectations on widening our net of users that would fall under that mandate, other than would occur naturally as projects rose in popularity to be part the top 1%.

However, in the year since then a few things have changed.

We've gotten a lot more confident in our 2FA implementation and in what the impact to enabling it is for both people publishing to PyPI, and to the PyPI team itself.
We've shipped features like Trusted Publishing that help remove some of the overheard of 2FA has on publishing (by far the most common action users do).
GitHub has furthered it's plans to mandate 2FA for all contributors on their platform, making more people already (or soon will be) prepared to cope with the requirements of having 2FA.
The PSF has received funding to hire a PyPI Safety and Security Engineer. While the role is not meant to purely handle support requests, having dedicated focus on the overall security posture, as well as the implementation specifics will improve PyPI as a whole for both users and project maintainers, and will help alleviate some of the pressures on PyPI's volunteers.
The workload to support end users relies heavily on a very small group of volunteers. When an user account report is seen by our trusted admins, we have to take time to properly investigate. These are often reported as an emergency, red-alert-level urgency. By mandating 2FA for project maintainers, the likelihood of account takeovers drop significantly, reserving the emergency status for truly extraordinary circumstances. Account recovery becomes part of normal routine support efforts instead of admin-level urgency.

All of these together help lead us to the conclusion that we can widen our mandate to all project maintainers on PyPI, while minimizing the impact on both project maintainers and PyPI administrators, and to do so in a way that improves the sustainability of PyPI and the wider Python ecosystem.

Isn't supply chain security a corporate concern?

There are some people who believe that efforts to improve supply chain security benefits only corporate or business users, and that individual developers should not be asked to take on a uncompensated burden for their benefit.

We believe this is shortsighted.

A compromise in the supply chain can be used to attack individual developers the same as it able to attack corporate and business users. In fact, we believe that individual developers, are in a more vulnerable position than corporate and business users. While businesses are generally able to hire staff and devote resources to vetting their dependencies, individual developers generally are not, and must expend their own limited free time to do so ⁴.

To make matters worse for the individual, in the case of a compromise a business is more likely going to have experts available to them to detect and remediate the compromise, while the individual has to do this on their own. At the extreme ends, businesses often have insurance to compensate them for any losses incurred while the individual almost always does not.

We recognize that supply chain security effects everyone, no matter how big or how small they are, and we are dedicated to protecting all our users.

Donald Stufft is a PyPI administrator and maintainer of the Python Package Index since 2013.

Technically a project with 0 downloads is effectively the same as a non-existent project, but it's easier to draw the line between non-existent and existent than it is to draw the line between 0 and 1 downloads. This is particularly true on PyPI, where a large network of mirrors and scanners mean that no projects truly get downloaded exactly 0 times. ↩
Except maybe the account owner themselves, by denying them access to their account. ↩
For end users it forces them to purchase some kind of hardware token OR to use some sort of TOTP application. In both cases it forces them to keep track of something else besides their password and changes the login flow from what they are used to. For PyPI it increases the chance that someone may get locked out of their account, requiring intervention by administrators. ↩
Not for nothing, but PyPI is also an Open Source project, run largely by volunteers, and cleaning up after a compromise on PyPI is something that affects those volunteers significantly. ↩

Reducing Stored IP Data in PyPI

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

Hi there! I'm Mike, the newest member of the PyPI admin team. Nice to meet you!

TL;DR

We've been working on reducing the amount of IP address data we store, and we're making progress.

What's this about?

If you've read some of the other blogs here, you may have noticed that we've been working on a number of security and privacy improvements.

A few months ago we started exploring what it would take to remove the concept of IP addresses from our stack, and retain the ability to safely manage the platform.

Some places of where IP data was historically used:

web access logs
user events (login,. logout, password reset, et al)
project events (creation, new releases, new file uploads, yanks, et al)
organization/team membership events (new!)
journal entries (private to PyPI admins)

Security is a spectrum - where on one extreme, it's the wild west, no security. On the other extreme, everything is locked down, impossible to use. We constantly have to balance the needs and desires of our users with the needs of running a sustainable, trusted platform.

A similar mindset can be applied to privacy - where we must strike the balance between providing a manageable service, and protecting the privacy of our users.

The two main approaches we've pursued in the short term are:

Evaluate whether we need to store IP data at all
Whenever possible, use alternatives to IP data

I'll provide a couple of examples demonstrating the above.

Do we need the IP data?

As we evaluated the different places we stored IP data, we learned that our Journal entries (similar to an append-only transaction log) were never exposed beyond admin users, and even then, used for admin display only.

We never share that via API, or used it for operational purposed. So we audited the code, removed calls to stop writing the column, and dropped it.

Woohoo!

Other places where we currently still need IP data include rate limiting, and fallbacks until we have backfilled the IP data with hashes and geo data. Our modern approach has evolved from using the IP data at display time to find the relevant geo data, to storing the geo data directly in the database.

Another use case is for handling abuse - we need to be able to identify the source of abuse, and take action to prevent it. We're thinking about how to manage that without storing IP data, but we're not there yet.

Alternatives to IP data

For the other places where we stored IP data, we asked ourselves - could we use an alternative?

We can't store what we don't see, so we explored what we could do to reduce the amount of IP data we see.

Pretty much every request to PyPI is served via our CDN partner, Fastly. They provide a number of features, including the ability to add custom headers. We leverage this ability already for a number of things, like informing the warehouse code of the inbound encoding of the request or language.

We explored whether we could use this to add a hash of the IP address, and use that instead of the IP address itself. Fastly can also provide some basic geographic info about the IP address, which served our purpose of showing the user where they had connected from.

Using this approach, we have Fastly pass along an already-hashed IP address, as well as the geographic data, to our backend, and store those for later use.

Another spot we identified was web access logs. We don't need real IP addresses there, as we rarely use them for anything other than low-level debugging, so a stable, hashed value serves the purpose of correlating requests.

For the requests that Fastly doesn't serve, we're already hashing the IP address ourselves prior to storage, so we could "see" the IP address briefly, but we try to avoid storing it. We don't get get the geo data for these requests, we're thinking of creative and sustainable solutions already.

Questions and Answers

I tried to think up some questions you might have, and answer them here. If you have more, please feel free to reach out to us!

Q: Is a hashed IP address secure?

A: It's more secure than a plain IP address, for sure. We apply a salt (known value) to the IP address before hashing it. It's not a perfect solution, but it's a step in the right direction.

The hash is non-reversible, but since the known address space is relatively small, it's possible to brute force the hash to determine the original IP address. By applying a salt, we require someone to possess both the salt and the hashed IP addresses to brute force the value. Our salt is not stored in the database while the hashed IP addresses are, we protect against leaks revealing this information.

Q: Is this a response to the subpoenas?

A: No, we started exploring this back in 2020, with a long term goal to increase end-user security, while retaining the ability to effectively steer the platform. We picked it up recently as we explored our CDN partner's options for geo IP data.

Q: Are we done?

A: Nope! Security is an ongoing journey, and we're making strong strides to accomplish this goal. We still have some work to do to replace IP data in our models, after we've backfilled our models with the hashed IP data and relevant geo data, and clean up some of the code.

Q: What's next?

A: I can't predict every future step we're likely to take, but some things we're considering:

Reevaluate the need for IP data in Event history forever, remove it after a period of time
Explore whether we can use a CDN for all requests
Determine if there's a better mechanism than Journal Entries, and replace them

We believe the steps we're taking are in the right direction, and we're excited to share our progress with you. Hopefully this enriches your understanding of the work we're doing, in support of maintaining a secure, trusted platform.

Thanks for reading!

Mike Fiedler is a PyPI administrator and maintainer of the Python Package Index since 2022.

Enforcement of 2FA for upload.pypi.org begins today

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

Beginning today, all uploads from user accounts with 2FA enabled will be required to use an API Token or Trusted Publisher configuration in place of their password.

This change has been planned since 2FA was rolled out in 2019. In February of 2022 we began notifying users on upload that this change was coming.

If you have 2FA enabled and have been using only your password to upload, the following email is likely familiar to you:

A sample notice email sent when users with 2FA enabled upload using only their password.

Initially, we intended for this notice to live for six months before we began enforcement.

However, some valid concerns were raised regarding the use of user-scoped API tokens for new project creation.

With the introduction of Trusted Publishers PyPI now provides a way for users to publish new projects without provisioning a user-scoped token, and to continue publishing without ever provisioning a long lived API token whatsoever.

Given this, and our commitment to further rolling out 2FA across PyPI, we are now enforcing this policy.

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

Announcing the launch of PyPI Malware Reporting and Response project

Shamika Monahan — Wed, 18 Mar 2026 18:14:42 +0000

We are pleased to announce that the PSF has received funding from the Center for Security and Emerging Technology (CSET) to develop and improve the infrastructure for malware reporting and response on PyPI. This project will be executed over the coming year.

Currently, malware reports are submitted to PyPI admins by email before being manually triaged and responded to. There is an opportunity for improvement in streamlining the report submission process and the tools used to triage and respond to them. The current process cannot scale easily or handle duplication of reports. It is not easy to measure time to remediation and is currently impossible to implement automated takedown of threats.

This project has the following aims:

Develop an API that allows malware reporting
Extend PyPI admin tools to view, collate and handle security reports
Collect metadata as required and identify trusted reporters
Define metrics that allow us to define good reporting practices and time to handle a security issue
Define the criteria for automated consensus based takedown and soft-deletes of packages
Highlight trusted reporters and report quality

As PyPI is an integral part of the Python ecosystem, this project is crucial in ensuring the security of over 450,000 packages that are trusted by millions of Python developers. Over the next few weeks, we will be working with security reporters to identify key elements that should be supported by the API and useful metrics that would add value to PyPI security reporting. If you or your colleagues are currently performing malware analysis of PyPI uploads, we would love to hear from you at https://forms.gle/ixRoNJEPVNekFN7H7.

Shamika Mohanan is the Packaging Project Manager at the PSF since 2021.

Deprecation of bdist_egg uploads to PyPI

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

PEP 715, deprecating bdist_egg/.egg uploads to PyPI has been accepted. We'll begin the process of implementing this today.

Please note that this does NOT remove any existing uploaded eggs from PyPI.

The deprecation timeline is as follows:

Today, June 26, 2023: All maintainers of projects which have uploaded one or more eggs since January 1, 2023 will receive a one-time email informing them of this change.
Today, June 26, 2023: Each upload of an egg to PyPI will result in a notice being sent to all Owners and Maintainers for the project.
August 1, 2023: Uploads of eggs will be rejected by PyPI.

You can read more detailed rationale in PEP 715. Thanks to contributor William Woodruff for his work to author and propose PEP 715, as well as support the rollout of the implementation.

Ee Durbin is the Director of Infrastructure at the Python Software Foundation. They have been contributing to keeping PyPI online, available, and secure since 2013.

PyPI hires a Safety & Security Engineer

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

👋 Hi there! I'm Mike Fiedler (@miketheman) I've been a Python Package Index (PyPI) contributor since early 2021, and became a maintainer in 2022. Now I'm joining the PSF to work on PyPI full-time as the first PyPI Safety & Security Engineer.

What is that, you ask? We had posted about this opening in May, and I'm happy to be joining the team to help improve the safety and security of PyPI.

What does that mean, Mike? As safety and security are pretty broad topics, it boils down to working on any initiatives that we believe will increase the safety and security of the Package Index for all users - end users, package publishers, maintainers, and PyPI moderators and administrators. That's a huge audience!

All of us deserve to have a safe and secure experience when using PyPI, and I'm excited to be able to work on this full-time.

Major thanks to Amazon Web Services (AWS) for their role in funding this position, and to the PSF for their continued support of the global Python community.

Mike Fiedler is the inaugural PyPI Safety & Security Engineer since 2023.

2FA Enforcement for New User Registrations

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

What's changing?

Starting today, newly registered users must enable 2FA before they can perform any management actions on PyPI. This change comes after we've also added a rule for accounts to have a verified, primary email address for the same set of management actions.

As a reminder, PyPI has supported adding 2FA since 2019.

This change is continuing along the path of enforcing 2FA for all users. In May of this year we announced that by the end of 2023 PyPI will require all users to enable Two-Factor Authentication (2FA). That post has a wealth of information on what enforcement means, and how folks can prepare for the change before end of year.

What are management actions?

Management actions may include any of the following:

Creating/managing Projects
Adding/removing API Tokens
Uploading/removing Releases
Adding/removing Collaborators
Requesting/managing Organizations
Adding/managing Trusted Publishers

This is not an exhaustive list, but should provide a good idea of the actions we're talking about.

How does this affect me?

If you only need to browse, download, and install packages from PyPI then a PyPI account isn't needed so this change doesn't affect you.

If you've already enabled 2FA on your PyPI account, this change will not affect you. Thanks for doing your part to keep the Python ecosystem safe!

If you recently registered a new PyPI account, you are required to enable 2FA before you can perform any management actions. When attempting to perform a management action, you may see a red banner flash at the top of the page, and be redirected to the 2FA setup page for your account.

You will still be able to log in, browse, and download packages without 2FA. But to perform any management actions, you'll need to enable 2FA.

Is this the end?

As a reminder, we will enforce the 2FA requirement for all PyPI users at the end of 2023.

These changes intend to mitigate scenarios like account takeovers, where an attacker may be able to gain access to a user's account using only an email and password (either via phishing or credential stuffing). If a user's email account access is compromised, and the attacker is able to successfully request a password reset, the attacker would still need to bypass the 2FA requirement.

The more users using a Two Factor Authentication methods available, the safer we all are. Today PyPI offers both Time-based One-Time Passwords (TOTP) and WebAuthn.

Security is a spectrum. As long as we continue to make incremental progress, we'll improve the overall posture for all users of PyPI.

Mike Fiedler is the PyPI Safety & Security Engineer since 2023.

GitHub now scans public issues for PyPI secrets

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

Back in 2019 we kicked off efforts to integrate with GitHub secret scanning. Due to the complexity in nature, the completed integration launched in 2021, with the volunteer-led effort by Joachim Jablon (@ewjoachim) and the GitHub team.

PyPI didn't have a blog back then, but GitHub did! Here's a link their post.

The completed integration increased security for all PyPI users. If a user accidentally made their PyPI token public by committing it and pushing it to a GitHub public repository, GitHub would notify us and we would automatically revoke the token to prevent any misuse. This process often completes within seconds.

Cool, Mike, that was two years ago, so what?

GitHub announced yesterday that they will now notify users about any secrets exposed in:

... issue's title, description, or comments, including historical revisions ...

This is a great enhancement to their existing secret scanning capabilities, previously only scanning code.

And even better, we don't have to change anything, as we continue to receive the same notifications from our existing integration with GitHub. 🎉

When implementing the integration, we set up some metrics to track the number of tokens revoked. Here's a visualization showing the amount of inbound notifications from GitHub, and the number of tokens we've revoked (click for larger image):

Bar charts of values over period of 1 year, and total counts of: received (5.87k), valid (537), processed (535)

We show 15 months of data, thanks to our infrastructure sponsor Datadog.

The large discrepancy between "received" and "valid" can be due to GitHub reporting the same token multiple times. This can happen if the token is used in multiple repositories, forks, etc.

The small discrepancy between "valid" and "processed" is due to that we may raise an exception when attempting to email the owner of a token that has been revoked, after which we don't mark it as "processed" when exception is raised. The second time the task runs, the token is now invalid, and doesn't increment the "valid" count.

Read GitHub's notice on The Changelog for more details.

If your organization performs this kind of secret scanning, we'd love to hear from you and integrate with your tooling.

Inbound Malware Volume Report

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

Background

The current PyPI security reporting procedure directs reporters to send an email to security@pypi.org with details. security@ was previously an email alias for admin@, a Google Group that contains all current PyPI Administrators (4 people).

I'll refer to the security@ address as the Security Inbox herein, despite it not being a traditional inbox, and what changes we've made to it.

The inbound reporting workflow was roughly:

Admins receive an email to the Security Inbox
Any admin reads the email
Admin inspects the indicators of compromise (IOC), package and user history. Often using inspector.pypi.io to investigate package contents
Admin takes an action (oftentimes to remove as malware)
Admin responds to reporter and CCs the Security Inbox for tracking

Here's a rough sequence diagram demonstrating the notification flow:

sequenceDiagram participant External participant googleMail as Google Mail Routing participant googleGroup as Security Inbox<br/>(Google Group) participant pypiAdmins as PyPI Admins External->>googleMail: sends an email report googleMail->>googleGroup: sends an email report googleGroup->>+pypiAdmins: sends an email report pypiAdmins-->>+External: Hi, thanks for your report... pypiAdmins-->>-googleGroup: CC response to group title Previous Inbound Flow

I wanted to answer a couple of questions, so as to have some data to work with:

How many inbound malware reports do we receive? (daily/weekly/monthly)
How long does it take for a response from an administrator to remove the reported malware?

Answering these questions with the email-based system will not be 100% accurate, as there are some conditions that lead to inaccuracies, but since we're looking at large volumes of records, it's unlikely that the inaccuracies will lead to material differences in the numbers. Some examples:

A reporter submits multiple reports on a single message
A reporter replies to their original email for a new report, instead of starting a new thread/conversation
Folks contacting admins for support may email the Security Inbox
Security-related issues that are not malware reports
Spam

This was largely to establish a baseline of where we are today. With this measurement, we can then observe whether changes to our process have a positive or negative impact on the volume and response times.

Note: This analysis is not an accurate measurement of distinct malware packages reported, as multiple researchers may report the same package, which will show up as distinct conversation threads. We could try to normalize the packages, however since the emails are unstructured, that may take more effort than is worthwhile. In any case, admins respond to duplicates, so there's still non-zero effort being done.

Earlier this year one of our admins posted some removals stats on Twitter (before we had a blog!):

in 2022, the @pypi team removed >12,000 unique projects. each were instances of spam, typosquatting, dependency confusion, exfiltration and/or malware.

2022: ~12K (mostly malware)
2021: ~27K (mostly dep confusion)
2020: ~500
2019: 65
2018: 137
2017: 38
— Dustin Ingram (@di_codes) January 4, 2023

What is the frequency of inbound malware reports?

To determine this answer, I elected to use the emails themselves, since that's the feed we have of details. While this is an imperfect data source, it should be good enough to make some early assertions. We can also leverage any statistics generated thus far to assist in measuring future progress.

My PyPI Google Workspace account was created on 2023-03-02 and began receiving security@pypi.org emails after that. Google Groups does not surface any APIs I could find that allows an authenticated user to list/read conversations directly from the group.

This approach provides a cleaner signal-to-noise ratio, as I often delete the random emails that are sent to the Security Inbox if they are not useful (spam, marketing, etc). We can either accept this approach of data collection for the past few months, or pursue other methods for collecting longer-term data from older accounts/mailing lists, all subject to different data quality issues.

Using the data based on 1,303 email threads sent to the Security Inbox by 2023-08-14, we can produce this chart:

The same data, grouped by week number:

One observation is that post-PyPI Weekend Suspension in May (Week 20), the overall volume drops for a while. There's no hard evidence as to why, but it's interesting that a brief disruption reduced some of the toil maintainers currently handle.

Form completeness, here's the monthly view:

How long does it take to respond?

Why is response time interesting? The longer a malicious package is available for end users to install, the more people and systems it may affect. This is further complicated to any package mirrors that capture the malware, and may not remove it as quickly as PyPI admins. We have received anecdotal evidence from reporters that PyPI admins are already quite fast at handling inbound reports (#humblebrag), but let's see if we can get data out of the same emails.

Again, since the nature of email isn't 100% accurate in this case, we'll rely on calculating the duration of time (in minutes) between the first message of a thread and the last message of a thread. This doesn't account for the occasional behavior of a reporter re-using the same thread to report more packages, nor does it reflect any other back-and-forth communication between admins and reporters. As such, removing any threads that have more than 4 total messages helps remove outliers from the analysis.

Inbound reports come in at any time of day, and can also be automatically generated by reporters. It's common for reports to wait to be handled while we’re asleep. Weekends and holidays often have longer response times as well.

On occasion an inbound report may get overlooked, something we're trying to solve with a new system, more on this later.

A higher response time may indicate that the mostly-volunteer admins missed responding to it the first time around. For the purpose of this analysis, I've excluded 7 total response times that exceed 14 days (20,160 minutes) to remove those outliers.

Using median values over averages helps us account for outliers at either end of the spectrum.

Applying a linear trend line to the collected data shows that response times are generally decreasing over time, which is a good thing.

Here's a distribution of response times for the data collected since March 2023:

This chart informs us that most responses to reports are completed in under ~485 minutes (~8 hours), and almost all are done within a few days of the report, with a long tail.

All said, that doesn't capture the full picture of response times, which is why we've been working on a new system to help us respond faster, and produce better reports.

What's changed as a result?

Part of the PyPI Malware Reporting and Response project is to explore ways to decrease the response times even further, while reducing the toil on maintainers, and increasing visibility to reporters.

As a result of this analysis, one change we've made so far is to leverage a shared inbox system Help Scout to receive inbound emails and allow us to tag, assign, and close out reports. This helps is by not missing reports, and preventing duplicate responses from admins.

Here's how we updated our Google Workspace flow so we can continue to receive inbound emails, as well as copy the conversations from Help Scout for long-term archival into Google Groups.

sequenceDiagram participant External participant googleMail as Google Mail Routing participant googleGroup as Google Group<br/>Archive participant helpScout as Security Inbox<br/>(Help Scout) participant pypiAdmins as PyPI Admins External->>googleMail: sends an email report googleMail->>+helpScout: sends an email report googleMail->>googleGroup: archive email helpScout->>+pypiAdmins: inbound notification pypiAdmins-->>-helpScout: Hi, thanks for your report... helpScout->>googleGroup: archive via auto-bcc helpScout-->>-External: Hi, thanks for your report... title Updated Inbound Flow

There's no change to end users, and we hope our change keeps us on track to continue to respond to reports in a timely fashion.

Here's an initial look at the response times since we started using Help Scout, (2023-09-05) using their reporting. With a total of 31 conversations in the time period since we started using Help Scout, and comparing to the final two weeks of data from the previous chart (59 conversations), we can see that the response times have improved:

Response time bucket	% of total	pre-Help Scout
< 15 min	40%	53%
15-30 min	20%	10%
30-60 min	20%	5%
1-2 hours	10%	7%
2-12 hours	10%	15%
12+ hours	0%	10%

We can now happily report that 80% all reports are responded to within 60 minutes of receipt, with 100% are responded to within 12 hours.

We will continue to monitor our response times and volumes, and make adjustments as needed.

What's next?

We're working on designing a new system to help us respond faster, based in inbound reports, and provide better outcomes. It's still very early, and we're incorporating a lot of ideas in the design based on collective experience of PyPI admins, external researchers and reporters.

We invite you to engage in the conversation on a more machine-readable format for reporting malware in this GitHub Issue, and consider sending pull requests where appropriate.

PyPI has completed its first security audit

Dustin Ingram — Wed, 18 Mar 2026 18:14:42 +0000

This is part one in a three-part series. See part two here, and part three here

We are proud to announce that PyPI has completed its first ever external security audit. This work was funded in partnership with the Open Technology Fund (OTF), a previous supporter of security-related improvements to PyPI.

The Open Technology Fund selected Trail of Bits, an industry-leading cybersecurity firm with significant open-source and Python experience, to perform the audit. Trail of Bits spent a total of 10 engineer-weeks of effort identifying issues, presenting those findings to the PyPI team, and assisting us as we remediated the findings.

Scope

The audit was focused on "Warehouse", the open-source codebase that powers https://pypi.org, and on "cabotage", the custom open-source container orchestration framework we use to deploy Warehouse. It included code review of both codebases, prioritizing areas that accept user input, provide APIs and other public surfaces. The audit also covered the continuous integration / continuous deployment (CI/CD) configurations for both codebases.

Findings

Overall, the auditors determined the Warehouse codebase "was adequately tested and conformed to widely accepted best practices for secure Python and web development," and that while the cabotage codebase lacks the same level of testing, they did not identify any high severity issues in either codebase.

Results & Impact

As a result of the audit, Trail of Bits detailed 29 different advisories discovered across both codebases. When evaluating severity level of each advisory, 14 were categorized as "informational", 6 as "low", 8 as "medium" and zero as "high". At the time of writing, the PyPI team has remediated all advisories that posed a significant risk in both codebases where possible, and has worked with third-party teams to unblock additional remediations where necessary.

More details

In the interest of transparency, today we are publishing the full results of the audit, as prepared by Trail of Bits. You can read more about the audit from their perspective in their accompanying blog post.

Additionally, in two additional blog posts published today, Mike Fiedler (PyPI Security & Safety Engineer) goes into detail about how we remediated these findings in Warehouse and Ee Durbin (Python Software Foundation Director of Infrastructure) similarly details remediation's in cabotage.

Acknowledgements

We would like to thank the Open Technology Fund for their continued support of PyPI and specifically for this significant security milestone for the Python ecosystem. We would also like to thank Trail of Bits for being a dependable, thorough and thoughtful partner throughout the process.

Security Audit Remediation: Warehouse

Mike Fiedler — Wed, 18 Mar 2026 18:14:42 +0000

This is part two in a three-part series. See part one here, and part three here.

This post is a deeper dive into the remediation of the security audit findings for the Warehouse - the main codebase for PyPI.org.

The audit report can be found here. I highly recommend reading that for the fullest context first.

Findings

The audit report identified 18 findings for Warehouse, along with some code quality suggestions. This post will focus on the findings and their remediation. Some of the code quality suggestions were implemented, others deferred.

Here's a table of the items that are relevant to warehouse, and their status:

ID	Title	Severity	Difficulty	Status
TOB-PYPI-1	Unsafe input handling in "Combine PRs" workflow	Informational	High	Remediated
TOB-PYPI-2	Weak signatures used in AWS SNS verification	Medium	Undetermined	Remediated
TOB-PYPI-4	Lack of rate limiting on endpoints that send email	Low	High	Accepted
TOB-PYPI-5	Account status information leak for frozen and disabled accounts	Medium	Low	Remediated
TOB-PYPI-6	Potential race conditions in search locking	Low	High	Remediated
TOB-PYPI-7	Use of multiple distinct URL parsers	Informational	Undetermined	Remediated
TOB-PYPI-8	Overly permissive CSP headers on XML views	Informational	High	Remediated
TOB-PYPI-9	Missing Permissions-Policy	Medium	High	Remediated
TOB-PYPI-10	Domain separation in file digests	Low	Low	Remediated
TOB-PYPI-11	Object storage susceptible to TOC/TOU due to temporary files	Informational	High	Accepted
TOB-PYPI-12	HTTP header is silently trusted if token mismatches	Informational	High	Remediated
TOB-PYPI-13	Bleach library is deprecated	Informational	Undetermined	Remediated
TOB-PYPI-14	Weak hashing in storage backends	Medium	High	Accepted
TOB-PYPI-15	Uncaught exception with crafted README	Informational	Medium	Accepted
TOB-PYPI-16	ReDoS via zxcvbn-python dependency	Informational	High	Accepted
TOB-PYPI-23	Insecure XML processing in XMLRPC server	Low	Low	Remediated
TOB-PYPI-27	Denial-of-service risk on tar.gz uploads	Informational	Medium	Accepted
TOB-PYPI-29	Unescaped values in LIKE SQL queries	Informational	Low	Accepted

IDs are non-consecutive, as the audit report included findings for cabotage as well.

For some of the Remediated entries and all the Accepted ones, I'll go into more detail below.

Details

Now that you've had a chance to read the original audit report, and can see that we've remediated most of the findings, I wanted to take some time to dig into some specifics of particular findings.

PyPI uses AWS SES to send emails to users. The SES configuration is set to use a Message Delivery Status topic, which sends a notification to an AWS SNS topic, which then sends a notification to our application.

This is useful for things like "Accepted/Delivered", but more importantly "Bounced" and "Complaint" notifications, which change the status of user accounts. We don't want to send more emails to a known bad address, and we don't want to send emails to users who have marked us as spam.

Since PyPI receives a webhook from AWS SNS, it needs to verify the signature of the message.

Verifying inbound SNS messages has generally been left up to the user. The AWS SNS docs are clear about that.

We had previously implemented signature verification for version 1, which uses the SHA1 hash algorithm, as that is what existed when we implemented it.

As time evolved, and AWS SNS added support for SHA256, the path to upgrade was still left in the hands of the user. SNS still defaults to SHA1 (SignatureVersion: '1'), and there's no Python SDK function to call to validate the signature for you.

This is also an outstanding request from boto3 users.

In September 2022, AWS SNS added support for SHA256 signatures, and shared the details in this blog post. They also added support for verification in some of the client-side SDKs, but sadly Python is not one of them yet.

While we were already validating SignatureVersion 1, we took this opportunity to add support for SignatureVersion 2, update our settings, and now only accept SHA256 signatures.

As an AWS Hero, I reached out to Farrah Campbell who heads up Modern Compute Community at AWS, and she quickly connected me with the AWS SNS service team for a chat. We discussed some of the challenges, as well as some ideas for the path forward.

I'm hopeful that sometime in the future we will see two big things:

message validation in boto3 for both signature versions This would enable us to remove MessageVerifier we added to warehouse, and benefit from any future enhancements to the validation process.
update AWS SNS to default to SignatureVersion: 2 (SHA256). This could be a breaking change for users who have not updated their settings, but would be a good step forward for security. This make take some time, as the new signature version was only added a year ago. I'll leave that up to the SNS service team.

TOB-PYPI-4: Lack of rate limiting on endpoints that send email

We accepted this finding, as we need to send emails to unverified users as part of the account creation process, and we don't want to block that.

The finding details that PyPI doesn't apply blanket rate limiting, which is correct. The endpoints that send emails to unverified addresses are protected via rate limiting.

PyPI has compensating controls in place to prevent abuse, such as preventing too many password reset emails from being sent to a single user.

Since the risk here was to the cost and reputation of the email service, we decided to accept this finding. At some future point we may revisit the rate limiting strategy for sending email.

TOB-PYPI-6: Potential race conditions in search locking

This is another case of "we implemented something that was good at the time, and as time went on a better solution became available".

We had written a context manager to handle locking the search index when performing updates, to prevent multiple processes from trying to update the search index at the same time. The implementation wasn't tied to the underlying Redis lock expiration, so could lead to the Redis-lock expiring, but Python believing it was still locked.

Here we updated our implementation to use a context manager that redis-py now provides, instead of crafting our own.

A solid reminder to check back on your libraries and services now and then, to see if there's new features that can help you out.

TOB-PYPI-11: Object storage susceptible to TOC/TOU due to temporary files

This is a complex timing attack, which requires a level of access to the system that would allow for a more direct attack. The finding itself details that if an attacker could execute this, they are more likely to do other kinds of damage.

The complexity of navigating between our various storage backends/client APIs does not appear to be worth the resulting defense in depth, given the required access level to exploit.

We have a draft PR with a start of implementation should we decide to pursue this.

TOB-PYPI-14: Weak hashing in storage backends

This is specifically about the Backblaze B2 storage backend, one of PyPI's current object storage providers, which does not currently support SHA-256 checksums. They do support SHA-1 which is useful for detecting data corruption in transit, but is insufficient for non-colliding checksums - we have to use MD5 for that.

During the audit, we reached out to the Backblaze team to discuss and determined it's on their roadmap, and when they implement it, we'll update our usage accordingly.

TOB-PYPI-15: Uncaught exception with crafted README

This finding discovered a bug in docutils, which PyPI uses via the readme_renderer library to render project descriptions from reStructuredText and Markdown to HTML.

The bug is tracked here, and has yet to see a response from the maintainers.

It only applies to client-side behavior when using reStructuredText for a README, so we've accepted this finding. Additionally, any user performing twine check prior to upload will surface this issue.

Once the bug is fixed, we'll update!

TOB-PYPI-16: ReDoS via `zxcvbn-python` dependency

Direct from the audit:

This finding is purely informational. We believe that it has virtually no impact, like many ReDoS vulnerabilities, due to Warehouse’s deployment architecture.

Enough said.

TOB-PYPI-23: Insecure XML processing in XMLRPC server

The audit began when the Warehouse deployment was on Debian 11 bullseye, and as part of normal maintenance we upgraded to Debian 12 bookworm while the audit was in progress.

Python XMLRPC uses expat for XML parsing. The version of expat in Debian bullseye was 2.2.10, which was vulnerable to the specific attack detailed in the audit report.

With bookworm, the version of expat is 2.5.0, which is not vulnerable. (Generally considered fixed as of 2.4.1.)

This was a tricky one to track down, as once the report came in I was unable to reproduce the issue locally, as I had already upgraded.

Using some git bisect magic, I was able to track down the exact commit that fixed the issue (the bookworm upgrade), and then it was a matter of figuring out which library had changed.

After figuring it out, I worked with the auditors to update their recommendations to reflect the upgrade. Until now, the general recommendation was to adopt defusedxml, which might have proven harder as we delegate the majority of our XML parsing to pyramid-rpc, which uses xmlrpc.client from the standard library.

If you want to check your own installation, you can run the following:

python -c "import pyexpat; print(pyexpat.EXPAT_VERSION)"

On bookworm, we get expat_2.5.0, which is not affected by the vulnerability.

This was remediated by underlying OS update to bookworm. Debian distributions pin to a specific version of libraries for the duration of that distribution version.

TOB-PYPI-27: Denial-of-service risk on tar.gz uploads

This is a tricky one, as it's a tradeoff between usability and security.

The audit report details a specific attack vector, where a malicious user could upload a tarball with a highly-compressed file, which would cause the server to spend a lot of time decompressing it.

Since we accept uploads from the general public, we have to take precautions whenever possible to prevent abuse. When it comes to ZIP files (which all .whl or "wheel" files are), we already have a mechanism to detect decompression bombs, and reject them.

However, since .tar.gz files do not advertise file sizes as metadata, in order to detect a decompression bomb we would have to decompress the entire file anyhow.

As the report notes, our deployment architecture compensates for this behavior, where we have a dedicated worker pool for handling uploads.

We may apply additional restrictions at the system level in the future, but for now we've accepted this finding.

TOB-PYPI-29: Unescaped values in `LIKE SQL` queries

The risk here is that a query could "walk the table" and not take advantage of any indexes, leading to higher resource usage.

The majority of the places where we use unescaped LIKE queries is in PyPI admin-only interface, where want to allow admins to search for users, packages, etc.

For the one place where we allow public-facing LIKE queries, there are already rate limits in place to prevent abuse. The table in question is also smaller than 1M rows, so walking an un-indexed column would not be a significant resource usage, and takes a handful of extra milliseconds.

The potential higher resource usage would be limited to malicious internal actors, and if we can't trust each other, we've got bigger problems to deal with.

We've accepted this finding, and will continue to monitor all of relevant resources.

Summary

Working with the folks at Trail of Bits was a pleasure, and I'm thankful for their thoroughness and professionalism.

While the audit was funded through the Open Technology Fund, my work on remediation would not have been as timely if not funded by Amazon Web Services to work as the PyPI Safety and Security Engineer. I am grateful for the continued support of both organizations in making PyPI a safer place for all Python users.

Mike Fiedler is the inaugural PyPI Safety & Security Engineer.

Security Audit Remediation: cabotage

Ee Durbin — Wed, 18 Mar 2026 18:14:42 +0000

This is part three in a three-part series. See part one here, and part two here.

This post is a deeper dive into the remediation of the security audit findings for cabotage - the codebase that deploys PyPI and its supporting services such as conveyor, camo, and inspector.

Relative to the warehouse codebase that is PyPI, cabotage is not as widely known. The goals of cabotage are to provide a seamless and secure way of deploying arbitrary services into a Kubernetes cluster in a "Twelve-Factor" style. There are also a number of firm opinions baked into cabotage that provide end-to-end TLS, protection against recovering secrets through the web UI, and isolation between tenants inside the cluster.

cabotage was initially developed in 2018 as part of the Mozilla Open Source Support Award that enabled the Python Software Foundation (PSF) to fund a team of contracted developers and a project manager to complete the development and deployment of warehouse and sunset the original PyPI codebase.

A primary goal of cabotage is to reduce the PSF Infrastructure's dependence on a specific provider for running PyPI, while providing self-service of configuration for project administrators and fully automated deployments. It is in-effect a "Platform as a Service" that deploys applications into bog-standard Kubernetes clusters, no YAML required.

To date, cabotage has deployed 3,901 releases to PyPI since 2018, and 7,377 releases in total across its current services "fleet".

The audit report can be found here. Reading that before you dive in will provide the fullest context.

Findings

Eleven findings resulted from the audit along with twelve code quality suggestions. This post will focus on the findings and their remediation. Some of the code quality suggestions were implemented, others deferred.

Here's a table of the items that are relevant to cabotage, and their status:

ID	Title	Severity	Difficulty	Status
TOB-PYPI-3	Vulnerable dependencies in cabotage	Undetermined	Low	Remediated
TOB-PYPI-17	Use of shell=True in subprocesses	Medium	Medium	Remediated
TOB-PYPI-18	Use of HMAC with SHA1 for GitHub webhook payload validation	Low	High	Remediated
TOB-PYPI-19	Potential container image manipulation through malicious Procfile	Medium	High	Remediated
TOB-PYPI-20	Repository confusion during image building	Medium	Medium	Remediated
TOB-PYPI-21	Brittle X.509 certificate rewriting	Informational	Undetermined	Accepted
TOB-PYPI-22	Unused dependencies in cabotage	Informational	Undetermined	Remediated
TOB-PYPI-24	Missing resource integrity check of third-party resources	Informational	High	Remediated
TOB-PYPI-25	Brittle secret filtering in logs	Medium	Low	Remediated
TOB-PYPI-26	Routes missing access controls	Low	High	Remediated
TOB-PYPI-28	Deployment hook susceptible to race condition due to temp files	Informational	High	Remediated 1, 2

IDs are non-consecutive, as the audit report included findings for Warehouse as well.

Details

TOB-PYPI-3: Vulnerable dependencies in cabotage

The maintenance of cabotage has been primarily driven by the need for new features or to mitigate issues raised. As a result dependency management and upgrades have often been done as a byproduct of other changes.

During review, there were a number of dependencies with known vulnerabilities found. Of the nine vulnerabilities noted, only GHSA-cg8c-gc2j-2wf7 was determined impact cabotage and was remediated by migrating to the latest release of the maintained fork of flask-security, flask-security-too (diff).

In order to avoid falling behind in this kind of maintenance, automated dependency management was added along with updates to all of known vulnerable dependencies (diff).

TOB-PYPI-17: Use of `shell=True` in subprocesses

An attack vector was identified in the way that cabotage calls out to buildctl when running container builds in development mode. A specifically crafted user-input has the ability to run arbitrary shell commands on the application host.

Ultimately this was not determined to be exploitable in the production instance of cabotage, since the shell commands were only used when building containers in local development mode. The use of shell=True was removed none-the-less as a matter of hygiene (diff).

TOB-PYPI-18: Use of HMAC with SHA1 for GitHub webhook payload validation

Similar to the SNS verification finding in TOB-PYPI-2 for warehouse, the endpoint that received webhook payloads from GitHub for automated deployments was using SHA1 HMAC signatures to validate authenticity when SHA256 HMAC signatures were available.

The remediation of this finding was much more direct than the SNS finding, as GitHub began sending the SHA256 signature in the header, does not require any changes to the configuration of the webhook, and uses standard HMAC signing supported by the Python standard library (diff).

TOB-PYPI-19: Potential container image manipulation through malicious Procfile

Along the same lines as TOB-PYPI-17, some user-supplied values had the ability to alter the cabotage controlled Dockerfile that specifies how release containers are built, which should not be modifiable. Through specifically crafted process names in the Procfile, a user could alter the resulting Dockerfile by injecting newlines.

Remediation was straightforward by adding additional validation of user supplied process names (diff).

TOB-PYPI-20: Repository confusion during image building

Due to a quirk in GitHub's API for fetching references, a given reference may return a concrete SHA/commit that belongs to a repository other than the one specified in the API call. In this case by providing a reference that resolves to a commit on a fork of the configured repository, a user of cabotage had the ability to intentionally (or mistakenly) configure cabotage to deploy code from a repository other than the one defined.

By adding additional validation inspired by Chainguard's clank tool, cabotage now verifies that the resulting SHA for a given reference belongs to the configured repository (diff).

TOB-PYPI-21: Brittle X.509 certificate rewriting

All containers built and deployed by cabotage are done so using short-lived authentication tokens for an internally deployed Docker registry instance. The cabotage application itself provides this authentication and must publish a public key that the registry can use to validate tokens.

In order to avoid handling private-key material in the application, cabotage relies heavily on Hashicorp Vault. The transit backend for vault did not support publishing the required x509 certificate that Docker registry required when cabotage was originally developed in 2018, so some clever use of the cryptography library was employed to create the necessary file ref.

In the audit it was determined that this work around was brittle in the event that an attacker had the ability to alter the length of the signature, resulting in an invalid x509 certificate and broken authentication for registry clients.

In practice, this has not been observed in the five and a half years that it has been in production and the result of a successful attack would only lead to deployments being halted. As such, we have accepted this finding for the time being and will investigate the newly released x509 support in vault 1.15 and adopt it if able (issue).

TOB-PYPI-22: Unused dependencies in cabotage

Similar to TOB-PYPI-3, dependency management for cabotage was lacking. This led to a handful of dependencies being installed that could be additional exposure to vulnerabilities or attacks.

By adopting pip-tools to compile and pin dependencies, only the projects necessary are installed (diff).

TOB-PYPI-24: Missing resource integrity check of third-party resources

When adding support for a new feature, third party JavaScript was added without subresource integrity information being added. This addition guards against malicious replacement of JavaScript an is good practice when loading any third party code.

Remediation was simple, by ensuring that all CDN loaded JavaScript had the correct value set (diff).

TOB-PYPI-25: Brittle secret filtering in logs

There was a brief period where cabotage supported building from private GitHub repositories, which necessitated filtering build logs and removing the plaintext authentication tokens.

This filtering was naive, but also no longer required. Remediation was removal of the filtering code, and a comment directing a future developer to the correct way of providing such authentication for builds in the future, should building from private GitHub repositories be supported (diff).

TOB-PYPI-26: Routes missing access controls

Another vestigial piece of code that allowed for the build context necessary for container builds was identified as allowing for potentially non-public information to be leaked if a release id (UUIDv4) was guessed or surmised.

This route was unauthenticated as a shortcut rather than adding a new authentication method to cabotage itself.

This code was made defunct when cabotage began building from contexts pulled directly from GitHub and supplied via Kubernetes secrets. Remediation was again, a simple removal of the code (diff).

TOB-PYPI-28: Deployment hook susceptible to race condition due to temp files

A final vestigial piece of code was also flagged as part of the audit which was created to fetch and re-package source code from GitHub for deployments. This had the very outside potential of being exploitable if an attacker gained access to the filesystem that the cabotage app uses for temporary files.

This was similarly made defunct when cabotage began building from contexts pulled directly from GitHub. Remediation was a final simple removal of code (diff-0), and a refactor of how temporary files are created and opened (diff-1).

Summary

In addition to the specific findings, the Trail of Bits team also made a number of "Code Quality Recommendations" and analyzed the overall maturity of the codebase. Those sections of the report highlight one of the two themes I see in the report regarding cabotage:

Overall the development experience and continuous integration environment for cabotage is lacking.
There are countless minutiae that one must consider when writing code with security in mind.

In the end, no show stopping or easily exploitable security issues were found, which is a relief! Many of the most interesting security findings were only exploitable by a malicious internal actor who already had configuration permissions in cabotage, was deploying their app there in the first place, or had access to the underlying systems.

The takeaway I have as the sole author and maintainer of cabotage is pretty resounding, and addresses both themes from the report:

Projects with solo maintainers do not benefit from the accountability that comes with collaborative development, are prone to deprioritizing critical improvements to developer experience and testing, and don't have the extra sets of eyes that often assist in spotting small bugs or improper handling of security sensitive software.

So if you're interested in infrastructure and projects that make deploying software securely and reliably more straightforward, I'd love to talk more. Swing by the cabotage repo and consider helping build the software that deploys PyPI, and will soon be deploying more and more of the Python Software Foundation's infrastructure as we migrate from previously gratis PaaS hosting providers.