![\"LiteLLM](\"../../assets/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/litellm_downloads_timeline.png\")
In 2023 PyPI completed its first security audit,\nand I am proud to announce that we have now completed our second external security audit.
\nThis work was funded by the Sovereign Tech Agency,\na supporter of Open Source security-related improvements,\npartnering with Trail of Bits to perform the audit.\nThanks to ongoing support from Alpha-Omega,\nmy role at the PSF enabled me to focus on rapid remediation of the findings.
\nThis time around, there's no three-part series, as the scope was narrower,\nfocused only on PyPI's codebase and behaviors.\nRead on for a summary of issues identified, their resolutions,\nand more details about the audit process.
\n\n\nThe full audit report can be found on the Trail of Bits publication page.\nI highly recommend reading that for the fullest context first.
\nHere's a table of the findings, status,\nand links to the relevant pull requests where applicable:
\n| ID | \nTitle | \nSeverity | \nDifficulty | \nStatus | \n
|---|---|---|---|---|
| TOB-PYPI26-1 | \nOIDC JTI anti-replay lock expires before JWT leeway window closes | \nMedium | \nHigh | \nRemediated | \n
| TOB-PYPI26-2 | \nOIDC token minting is vulnerable to a TOCTOU race in JTI anti-replay | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI26-3 | \nVerification badge bypass on the home page and download URLs | \nLow | \nLow | \nRemediated | \n
| TOB-PYPI26-4 | \nProject-level token deletion audit events silently dropped due to data structure mismatch | \nLow | \nLow | \nRemediated | \n
| TOB-PYPI26-5 | \nPassword reset leaks privileged account status | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI26-6 | \nIP ban bypass via macaroon API token authentication | \nInformational | \nHigh | \nAccepted | \n
| TOB-PYPI26-7 | \nModerators can modify organization applications due to a missing write permission check | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI26-8 | \nOrganization members can invite new owners due to a missing manage permission check | \nHigh | \nMedium | \nRemediated | \n
| TOB-PYPI26-9 | \nTOTP replay prevention bypass via space normalization mismatch between validation and storage | \nInformational | \nHigh | \nRemediated | \n
| TOB-PYPI26-10 | \nWheel METADATA is served to installers without validation against upload metadata | \nLow | \nLow | \nAccepted | \n
| TOB-PYPI26-11 | \nIDOR in API Token Deletion Allows Any Authenticated User to Delete Other Users' Macaroons | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI26-12 | \nGitHub OIDC publisher lookup lacks issuer URL isolation for custom GHES issuers | \nInformational | \nHigh | \nRemediated 1, 2 | \n
| TOB-PYPI26-13 | \nOrganization-scoped project associations persist after project transfer or removal | \nHigh | \nHigh | \nRemediated | \n
| TOB-PYPI26-14 | \nAdmin flag changes lack audit logging | \nInformational | \nHigh | \nRemediated | \n
Of the 14 findings, I used a combination of Severity and Difficulty\nto determine which ones to work on first, and which ones to accept for now.
\nThere were no Critical severity findings, 2 High, 1 Medium, 7 Low,\nand 3 Informational severity findings.
\nAll but 2 findings have been remediated, and the remaining 2 are accepted for now.\nMore details on the accepted findings below, but in general these were accepted\nbecause they require significant effort to remediate, and the risk they pose is relatively low.
\nTo reiterate, the published report PDF goes into deeper detail about each finding,\nso I recommend reading that for the fullest context first.
\nFor some of the Remediated entries and all the Accepted ones, I'll go into more detail below.
\nPyPI's Trusted Publishing flow\nuses OIDC JWTs issued by CI providers to mint short-lived upload tokens.\nEach JWT contains a jti (JWT Token Identifier) claim that should be single-use.\nTo enforce this, we store each jti in cache (Redis) with an expiration of exp + 5 seconds,\nand check whether it already exists before accepting a new token.
The problem: PyJWT is configured with leeway=30,\nmeaning it accepts tokens up to 30 seconds past their exp claim.\nThis created a 25-second window (from exp + 5 to exp + 30)\nwhere the cache key had already been evicted, but the JWT still passed signature verification.\nDuring that window, a replayed token would pass both the signature check and the jti uniqueness check.
The fix was straightforward --\nalign the cache TTL to outlive the full leeway window\nby setting the expiration to exp + leeway + margin.\nI also took the opportunity to centralize these time-window constants\nso they're derived from a shared configuration,\npreventing future drift when one value is updated without the other.
Accepted for now.
\nPyPI administrators can ban IP addresses through the admin dashboard.\nThe session authentication policy enforces this by checking the IP against the ban list\nbefore returning an identity.\nHowever, the macaroon (API token) authentication policy doesn't perform this same check.\nThis means a user with a valid API token could continue uploading packages\nfrom a banned IP address.
\nI've accepted this finding for now. IP bans are a relatively blunt tool that we use sparingly,\nintroduced late last year\nto mitigate a specific wave of abuse.\nThe practical risk here is low - if we've identified a malicious actor,\nwe have other mechanisms to disable their account entirely.\nThat said, it's a gap worth closing, and we'll likely address it as part of broader work\non making security controls consistent across all authentication methods.
\nThis was the highest-severity finding in the audit, and one I prioritized immediately.
\nThe manage_organization_roles view handled both GET (viewing the people page)\nand POST (sending invitations) under a single @view_config decorator\nthat only required OrganizationsRead permission.\nThis meant any organization member could send invitations with any role -\nincluding Owner - to any PyPI user.
The irony is that we already had the correct pattern elsewhere in the codebase.\nViews like resend_organization_invitation and change_organization_role\ncorrectly use separate @view_config decorators for GET and POST\nwith distinct permission requirements.\nThis one was simply missed.
The fix was to split the view configuration:\nGET requires OrganizationsRead, POST requires OrganizationsManage.\nAs part of the audit, Trail of Bits also developed a custom CodeQL query\nto detect this class of issue - views that handle state-changing POST requests\nunder a read-only permission check.\nI'll integrate that into our CI to catch this pattern going forward.
Accepted for now.
\nThis is a nuanced one. When a wheel is uploaded to PyPI,\nwe store two independent sources of metadata:\nthe form-declared metadata from the upload request (which populates the database and the JSON API),\nand the embedded .dist-info/METADATA file extracted from the wheel itself\n(which is served via PEP 658 to pip for dependency resolution).
These two sources are never compared.\nIn theory, an attacker could embed hidden dependencies in the wheel's METADATA\nthat pip would install, but that security tools querying the JSON API would never see.
We've accepted this for now because the fix is non-trivial.\nProperly validating embedded metadata against upload metadata\ntouches a core part of how we handle uploads,\nand requires careful consideration of edge cases across the ecosystem.\nThis is something we want to get right rather than rush,\nand involves a fair amount of database changes, including data backfills.
\nThis was the other High-severity finding, and a subtle one.
\nWhen a project is transferred between organizations,\nthe OrganizationProject junction record is correctly deleted and recreated.\nHowever, the TeamProjectRole records - which grant a team's members\naccess to specific projects - were not cleaned up during the transfer.
This meant that if LexCorp Organization had a \"release-engineers\" team\nwith Owner-level access to a project,\nand that project was transferred to Organization OsCorp,\nthe LexCorp team's members would silently retain full access to the project.\nWorse, the receiving organization had no visibility into these stale associations -\nteam-granted permissions are resolved at ACL evaluation time\nand don't appear as individual collaborator entries in the UI.
\nThe fix in pypi/warehouse#19749\nensures that TeamProjectRole records belonging to the departing organization\nare cleaned up when a project is transferred.\nAuditing database records proved that this has not happened in the past,\nso I am confident there have been no such transfers with dangling permissions.\nI also added defensive filters in the project's ACL computation\nto verify that a team's organization matches the project's current organization\nbefore granting permissions,\nso stale records can't grant access regardless of how they're orphaned.
Working with Trail of Bits was again a pleasure.\nThe team were thorough, communicative, and clearly understood\nthe nuances of a system like PyPI - where the threat model spans everything\nfrom CI/CD token replay to metadata integrity for millions of downstream users.
\nBeyond the 14 findings, the audit also produced proposal reviews\nfor features I'm considering (per-org Trusted Publishers, TOTP hardening, and more),\nas well as custom CodeQL queries to integrate into our CI/CD pipeline.
\nThis audit was funded in partnership with\nthe Sovereign Tech Agency,\nwhich continues to support security improvements across the Open Source ecosystem.
\nMy work at the Python Software Foundation is supported by Alpha-Omega.
", "image": "https://blog.pypi.org/assets/images/social/posts/2026-04-16-pypi-completes-second-audit.png", "date_modified": "2026-04-10T19:38:00+00:00", "date_published": "2026-04-16T19:00:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/", "url": "https://blog.pypi.org/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/", "title": "Incident Report: LiteLLM/Telnyx supply-chain attacks, with guidance", "content_html": "This post will drill deeper into two recent supply chain exploits, targeting users of popular PyPI packages - litellm & telnyx.\nWe also provide Python developers and maintainers with guidance on what they can do to prepare\nand protect themselves from future incidents.
After an API token exposure from an exploited Trivy dependency\nreleases of the packages litellm and telnyx\nwere published to PyPI containing credential harvesting malware.
The malware ran on install, harvesting sensitive credentials and files, and exfiltrating to a remote API. More details published in the advisory for LiteLLM (PYSEC-2026-2), the LiteLLM blog post about the incident,\nthe advisory for telnyx (PYSEC-2026-3) and the telnyx notice.
\nAfter contacting the litellm and telnyx maintainers, Mike and Seth collaborated with each team on steps forward, including token rotation, release removals, and recommendations around further security practices like using Trusted Publishers which both projects have since adopted.
This class of malware is different from most malware published to PyPI, which are mostly published as new packages, either as typosquats or with a plan to share the package with others and hope they install it. This malware is \"injected\" into open source packages that are already in widespread use. The malware injection occurs one of two ways:
\nUsing the API tokens and keys gathered from developer machines, the malware authors are able to further compromise other open source packages if API tokens for PyPI or GitHub are exfiltrated. This cycle continues as long as it is effective in exfiltrating more credentials.
\nWith daily volume of ~700-800 new projects created daily on PyPI, this poses a scaling challenge.\nPyPI partners with security researchers from our community who regularly scan and report malware through elevated channels to facilitate quicker remediation times.
\nBelow see the timeline of events.
\n
During the window of attack, the exploited versions of litellm were downloaded over 119k times.
PyPI received 13 inbound reports from concerned users, leveraging the \"Report project as malware\" feature, added back in 2024, accelerating the review/action.
\nLiteLLM is typically installed ~15-20 million times per week. Averaging this out to an \"installs per minute\" rate nets a value between ~1700 installs per minute.\nThis means between ~40-50% of all installs of LiteLLM were unpinned and fetching the latest version on each install invocation.
\nBecause these systems are pulling latest, this leaves very little time for researchers and PyPI admins to report, triage, and quarantine malware when published to a popular package. Read on for information on \"dependency cooldowns\"
\n![\"Telnyx](\"../../assets/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack/telnyx_downloads_timeline.png\")
PyPI's remediation response to telnyx was autonomously taken thanks to our pool of trusted reporters.\nThese reporters add weight to any given report, triggering an automated quarantine feature.\nSubscribe to the PyPI blog for a future update about our automatic quarantining system.
Below are a few methods to make your usage of Python packages from PyPI more secure and to avoid installing malware.
\nOne method to avoid \"drinking from the firehose\" and allow time for malware to be detected and remediated is using \"Dependency Cooldowns\".\nDependency cooldowns is a strategy for package installers (like pip, uv, etc) to avoid installing packages that have very recently been published to PyPI. By doing so, this allows for human operators like security researchers and PyPI admins a chance to respond to reports of malware.
Dependency cooldowns work best when they are configured \"globally\" on a developer machine and then passively protect developers from compromises on every invocation of pip or uv.\nSetting a relative value like \"3 days\" (\"P3D\" per RFC 3339) means packages that are newer than 3 days will not be installed.
uv already supports setting relative dependency cooldown via --exclude-newer. You can configure the option globally (in ~/.config/uv/uv.toml) or per-project in your pyproject.toml:
[tool.uv]\nexclude-newer = \"P3D\" # \"3 days\" in RFC 3339 formatRelative dependency cooldowns are coming soon to pip v26.1 which should be available in April of this year. Once they are available you can set the option in your pip.conf file (~/.config/pip/pip.conf):
[install]\nuploaded-prior-to = P3DStarting in pip v26.0 you can set absolute dependency cooldowns with pip from the command-line, likely paired with another tool like date to calculate an absolute date from a relative offset like \"3 days\":
python -m pip install \\\n --uploaded-prior-to=$(date -d '-3days' -Idate) \\\n simplepackageApplying dependency cooldowns everywhere isn't a silver bullet, though!\nThere are certain situations where you do want the latest version of a package as soon as possible, like when applying patches for vulnerabilities.\nDependency cooldowns should be paired with a vulnerability scanning strategy so security updates for your application's dependencies aren't waiting to be deployed. For example, Dependabot and Renovate both bypass dependency cooldowns by default for security updates.
\nYou can manually bypass a dependency cooldown in pip and uv by setting a value of \"the current day\" to get the actual latest release:
\npython -m pip install \\\n --uploaded-prior-to=P0D \\\n simplepackage==26.3.31Installing a package from PyPI without a \"lock\" means that it's possible to receive new code every time you run pip install.\nThis leaves the door open for a compromise of a package to immediately get installed and execute malware on the installing systems.
\nIf you're a developer of an application using PyPI packages you should be using lock files both for security and reproducibility of your application.\nSome examples of tools which produce lock files for applications are:
\nuv lockpip-compile --generate-hashespipenvNote that pip freeze doesn't create a lock file: a lock file must include checksums / hashes of the package archives to be secure and reproducible.\npip freeze only records packages and their versions. pip is working on experimental support for the pylock.toml standard through the\npip lock sub-command.
If you are a maintainer of an open source project on PyPI, you can do your part to protect your users from compromises.\nThere are three approaches we recommend:
\nIf you're using a continuous deployment system to publish packages to Python: these workflows are targets for attackers. You can prevent most of the danger by applying a handful of security recommendations:
\npull_request_target from GitHub Actions in particular is difficult to use securely and should be avoided.If you are using GitHub Actions as your continuous deployment provider, we highly recommend the tool \"Zizmor\" for detecting and fixing insecure workflows.
\nIf the platform you use to publish to PyPI supports Trusted Publishers (GitHub, GitLab, Google Cloud Build, ActiveState) then you should use Trusted Publishers instead of API tokens.
\nPyPI API tokens are \"long-lived\", meaning if they are exfiltrated by an attacker that attacker can use the token at a much later date even if you don't detect the initial compromise. Trusted Publishers comparatively uses short-lived tokens, meaning they need to be used immediately and don't require manual \"rotating\" in the case of compromise.
\nTrusted Publishers also provides a valuable signal to downstream users through Digital Attestations. This means users can detect when a release hasn't been published using the typical release workflow, likely drawing more scrutiny.
\nWe may be starting to sound like a broken record, but 2FA should be used for all accounts associated with open source development: not just PyPI.\nThink about accounts like version control / software forges\n(GitHub, GitLab, Codeberg, Forgejo) and your email provider.\nPyPI has required 2FA to be enabled to publish packages since the beginning of 2024,\nbut enabling phishing-resistant 2FA like a hardware key can protect you further.
\nSecurity work isn't free. You can support security work on the Python Package Index by supporting the Python Software Foundation (PSF). If you or your organization is interested in sponsoring or donating to the PSF so we can continue supporting Python, PyPI, and its community, check out the PSF's sponsorship program, donate directly, or contact our team at sponsors@python.org!
\nMike Fiedler and Seth Larson's roles as PyPI Safety & Security Engineer and Security Developer-in-Residence at the Python Software Foundation are supported by Alpha Omega.
", "image": "https://blog.pypi.org/assets/images/social/posts/2026-04-02-incident-report-litellm-telnyx-supply-chain-attack.png", "date_modified": "2026-04-02T15:04:45+00:00", "date_published": "2026-04-02T06:09:00+00:00", "authors": [{"name": "Seth Larson"}, {"name": "Mike Fiedler"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-03-21-welcome-to-the-pypi-blog/", "url": "https://blog.pypi.org/posts/2023-03-21-welcome-to-the-pypi-blog/", "title": "Welcome to the PyPI Blog", "content_html": "Today, we're excited to launch blog.pypi.org,\nthe official blog of the Python Package Index.
\nOne of the most common refrains I hear from Python community members,\nirrespective of if they have been around for days or years,\nis \"I didn't realize that PyPI...\". Followed by something along the lines of:
\n\n\n\nThat is certainly in part due to the fact that the people3 who\nmake PyPI \"happen\" are humble, and do so with the limited time they\nhave to commit to the project among their other responsibilities.\nBut its also because writing is hard. Harder still when there is\nnot a convenient tool at hand.\nWe're hoping that a modern and extensible stack4 for this blog\nwill make writing about and sharing our work more frictionless.
\nThe PyPI team will use this space to communicate with PyPI users,\nprovide announcements about new features and updates,\ninteresting technology, as well as share information and\ncontext around PyPI and related efforts of the\nPython Software Foundation.
\nStay tuned for more, we're very glad you're here! \ud83d\udc9c
\nEe Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
\nWe're thrifty and picky about what we run, but\nFastly is the single most\nimportant factor allowing us to get by with comparatively few\nresources for our backends. ↩
\nThat was until we welcomed Mike Fiedler as an admin in 2023. ↩
\nThe PyPI Administrator team is\nDonald Stufft,\nDustin Ingram,\nMike Fiedler,\nand Ee Durbin.\nThe PyPI Moderator team is\nYeray D\u00edaz,\nCristi\u00e1n Maureira-Fredes,\nJoachim Jablon,\nJason Madden,\nPradyun Gedam,\nand Stargirl Flowers. ↩
\nShout-out: This blog is hosted on\nRead the Docs,\nusing the MkDocs\nstatic site generator,\nand the Material for MkDocs theme.\nMany thanks to the maintainers of those projects for your\ncontributions,\nwe're very excited to have our blog contents live side-by-side\nwith our other documentation and code in\npypi/warehouse! ↩
\nStarting today, PyPI package maintainers can adopt a new, more secure\npublishing method that does not require long-lived passwords or API tokens to\nbe shared with external systems.
\n\n\n\"Trusted publishing\" is our term for using the OpenID Connect (OIDC) standard\nto exchange short-lived identity tokens between a trusted third-party service\nand PyPI. This method can be used in automated environments and eliminates the\nneed to use username/password combinations or manually generated API tokens to\nauthenticate with PyPI when publishing.
\nInstead, PyPI maintainers can configure PyPI to trust an identity provided by a\ngiven OpenID Connect Identity Provider (IdP). This allows allows PyPI to verify\nand delegate trust to that identity, which is then authorized to request\nshort-lived, tightly-scoped API tokens from PyPI. These API tokens never need\nto be stored or shared, rotate automatically by expiring quickly, and provide a\nverifiable link between a published package and its source.
\nPyPI currently supports trusted publishing with GitHub Actions, using their\nsupport for OpenID Connect.
\nAfter configuring PyPI to trust a given GitHub repository and workflow, users\nof the PyPA's 'pypi-publish' GitHub Action can adopt trusted publishing by\nremoving the username and password fields from their workflow\nconfiguration, and adding permissions to generate an identity token:
jobs:\n pypi-publish:\n name: upload release to PyPI\n runs-on: ubuntu-latest\n+ permissions:\n+ # IMPORTANT: this permission is mandatory for trusted publishing\n+ id-token: write\n steps:\n # retrieve your distributions here\n\n - name: Publish package distributions to PyPI\n uses: pypa/gh-action-pypi-publish@release/v1\n- with:\n- username: __token__\n- password: ${{ secrets.PYPI_TOKEN }}Using the PyPA's GitHub action is strongly recommended, but not required. More\ndetails on how to manually exchange tokens are available in our\ndocumentation.
\nPyPI package maintainers can further increase the security of their release\nworkflows by configuring trusted publishers to only release from a specific\nGitHub Actions environment.
\nConfiguring an environment is optional, but strongly recommended: with a GitHub\nenvironment, you can apply additional restrictions to your trusted GitHub\nActions workflow, such as requiring manual approval on each run by a trusted\nsubset of repository maintainers.
\nIn addition to making publishing more secure now, the availability of trusted\npublishers unblocks additional future security improvements for PyPI.
\nConfiguring and using a trusted publisher provides a 'strong link' between a\nproject and its source repository, which can allow PyPI to verify related\nmetadata, like the URL of a source repository for a project1. Additionally,\npublishing with a trusted publisher allows PyPI to correlate more information\nabout where a given file was published from in a verifiable way.
\nFinally, although trusted publishers is currently limited to GitHub Actions,\nmuch of the underlying work that went into making this feature possible is\ngeneralizable and not specific to a single publisher. We're interested in\nsupporting the ability to publish from additional services that provide OpenID\nConnect identities.
\nTo get started with using trusted publishers on PyPI, see our documentation\nhere: https://docs.pypi.org/trusted-publishers/.
\nFunding for this work was provided by the Google Open Source Security Team, and\nmuch of the development work was performed by Trail of Bits, with special\nthanks to contributor William Woodruff.
\nMany thanks as well to Sviatoslav Sydorenko, maintainer of the PyPA's\n'pypi-publish' GitHub Action for his quick and timely work to add support for\ntrusted publishers in the action.
\nFinally, we want to thank all our beta testers, including GitHub staff, for\nworking with us to ensure this feature is intuitive and useful, and for\nproviding valuable feedback to improve this feature along the way.
\nDustin Ingram is a maintainer of the Python Package Index.
\nCurrently, information such as this are provided by the uploader and are not verified as accurate by PyPI. ↩
\nToday, we are rolling out the first step in our plan to build financial\nsupport and long-term sustainability of the Python Packaging Index (PyPI),\nwhile simultaneously giving our users one of our most requested features:\norganization accounts.
\n\n\nOrganizations on PyPI are self-managed teams, with their own exclusive branded\nweb addresses. Our goal is to make PyPI easier to use for large community\nprojects, organizations, or companies who manage multiple sub-teams and multiple\npackages. We\u2019re making organizations available to community projects for free,\nforever, and to corporate projects for a small fee. Additional priority support\nagreements will be available to all paid subscribers, and all revenue will go\nright back into PyPI to continue building better support and infrastructure\nfor all our users.
\nIn the last year PyPI served 235.7 billion downloads for the 448,941 projects hosted\nthere. This means that since the previous period, PyPI saw a 57% annual growth in download counts\nand bandwidth alike. Having more people using and contributing to Python every\nyear is an fantastic problem to have, but it is one we must increase\norganizational capacity to accommodate.
\nIncreased revenue for PyPI allows it to become a\nstaffed platform that can respond to support requests and attend to issues\nin a timeframe that is significantly faster than what our excellent (but thinly\nspread) largely volunteer team could reasonably handle.
\nWe want to be very clear\u2014these new features are completely optional. If\nfeatures for larger projects don't sound like something that would be useful to\nyou as a PyPI maintainer, then there is no obligation to create an organization\nand absolutely nothing about your PyPI experience will change for you.
\nWe look forward to discussing what other features PyPI users would like to see\ntackled next. We know there are lots of ideas out there around safety,\nsecurity and usability and we\u2019re looking forward to hearing about anything that\nyou think would benefit the community. And in the spirit of open source, we\nwelcome your feedback on what's on\noffer so far.
\nBoth community projects (non-profits, NGO\u2019s, hobbyists, etc) and corporate\nteams can sign up to request their organization name starting\ntoday. Submissions will begin\nseeing review and approval in the coming weeks, and corporate teams will be\nable to finalize their signup with billing details in May.
\nOrganization feature development was approved by the\nPackaging Working Group\nand funded through the\nPython Software Foundation's\nsponsorship program -- thanks to our sponsors\nfor making this work possible.
\nWe especially want to thank\nBloomberg\nfor funding our Packaging Project Manager, Shamika Mohanan.
\nWe are also grateful for the many generous PyPI users who shared their\nperspectives with Shamika, which laid the foundation for these new features.
\nAnd thanks as well to our beta\ntesters, including the following \u2728new\u2728 PyPI organizations:
\nEe Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-04-23-introducing-pypi-organizations.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-04-23T06:09:00+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["organizations", "sustainability"]}, {"id": "https://blog.pypi.org/posts/2023-05-09-announcing-pypi-safety-and-security-engr-role/", "url": "https://blog.pypi.org/posts/2023-05-09-announcing-pypi-safety-and-security-engr-role/", "title": "Announcing the PyPI Safety & Security Engineer role", "content_html": "We are pleased to announce\nAmazon Web Services (AWS)\nas the inaugural Security Sponsor for PyPI,\ninvesting $144,000 over one year\nto fund key enhancements to PyPI infrastructure and operations,\nincluding the creation of a new \u201cPyPI Safety & Security Engineer\u201d role.
\n\n\nThis role builds on our existing long term partnership with AWS\nas one of the top sponsors of the Python Software Foundation for the last five years,\nwhich has included in-kind donations of cloud computing infrastructure\nand services to support PyPI.\nThe role will complement our previously announced role for a\nPSF Security Developer in Residence\nand will work closely with the person hired for that role (to be announced soon).
\nThis funding also builds on\npreviously successful project-focused funding efforts,\nsuch as the 2018 full-stack rewrite of PyPI,\nthe introduction of internationalization and localization for PyPI,\nas well as 2FA and WebAuthn support.
\nWe expect this partnership to tangibly improve the experience for all PyPI users,\nfrom consumers downloading packages,\nto package maintainers,\nto large corporate teams.\nSome of the outcomes we are working toward over the next year include\nincreased support for package maintainers including multi-maintainer projects,\nimprovements to reporting infrastructure for malicious projects,\nas well as a reduced response time for malware reports and account recovery requests.
\nThe job posting can be found here,\nand applications for the role are open until June 1st.\nSimilar to existing developer-in-residence roles,\nthe contract for this role will be for a one year period,\nand the PSF will be actively engaging with our sponsors and supporters\nto renew funding for the role in subsequent years.
\nEe Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-05-09-announcing-pypi-safety-and-security-engr-role.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-05-09T06:09:00+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["hiring", "security"]}, {"id": "https://blog.pypi.org/posts/2023-05-23-removing-pgp/", "url": "https://blog.pypi.org/posts/2023-05-23-removing-pgp/", "title": "Removing PGP from PyPI", "content_html": "If you are someone who is currently uploading signatures, your package uploads will\ncontinue to succeed, but any PGP signatures will be silently ignored. If you are\nsomeone who is currently downloading PGP signatures, existing signatures\nSHOULD continue to be available 1, but no new signatures will be made available.\nThe related API fields such as has_sig have all been hardcoded to always be\nFalse.
Historically, PyPI has supported uploading PGP signatures alongside the release\nartifacts in an attempt to provide some level of package signing. However, the\napproach used had long standing,\ndocumented issues\nwhich had previously lead us to deemphasize the support\nfor PGP signatures over time by removing them from the PyPI web user interface.
\nPyPI has continued to support uploading these signatures in the hope that there\nmight be some systems out there that found them useful. Recently though,\nan examination of the signatures on PyPI\nhas revealed to us that the current support for PGP signatures is not proving useful.
\nIn the last 3 years, about 50k signatures had been uploaded to PyPI by 1069\nunique keys. Of those 1069 unique keys, about 30% of them were not discoverable\non major public keyservers, making it difficult or impossible to meaningfully\nverify those signatures. Of the remaining 71%, nearly half of them were unable\nto be meaningfully verified at the time of the audit (2023-05-19) 2.
\nIn other words, out of all of the unique keys that had uploaded signatures to\nPyPI, only 36% of them were capable of being meaningfully verified 3 at the\ntime of audit. Even if all of those signatures uploaded in that 3 year period\nof time were made by one of those 36% of keys that are able to be meaningfully\nverified, that would still represent only 0.3% of all of those files.
\nGiven all of this, the continued support of uploading PGP signatures to PyPI is\nno longer defensible. While it doesn't represent a massive operational burden\nto continue to support it, it does require any new features that touch the\nstorage of files to be made aware of and capable of handling these PGP\nsignatures, which is a non zero cost on the maintainers and contributors of\nPyPI.
\nDonald Stufft is a PyPI administrator and maintainer of the Python Package Index since 2013.
\nFor now, but they may be removed in the future. ↩
\nThese could be because the original signature was made incorrectly and\n never had a binding signature to a associated key identity, or because\n the signature was present but had since expired. ↩
\nWe use meaningfully verified to mean that the signature was valid and the\n key that made it was not expired and had binding identify information that\n could tell us if this key was the correct key. ↩
\nIn March and April 2023, the Python Software Foundation (PSF)\nreceived three (3) subpoenas for PyPI user data.\nAll three subpoenas were issued by the United States Department of Justice.\nThe PSF was not provided with context on the legal circumstances surrounding these subpoenas.\nIn total, user data related to five (5) PyPI usernames were requested.
\n\n\nThe data request was:
\nThe privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators,\nand we are committed to protecting user data from disclosure whenever possible.\nIn this case, however, PSF determined with the advice of counsel that\nour only course of action was to provide the requested data.\nI, as Director of Infrastructure of the Python Software Foundation,\nfulfilled the requests in consultation with PSF's counsel.
\nWe have waited for the string of subpoenas to subside, though we were committed\nfrom the beginning to write and publish this post as a matter of transparency,\nand as allowed by the lack of a non-disclosure order associated with the\nsubpoenas received in March and April 2023.
\nPyPI and the PSF are committed to the freedom, security, and privacy of our users.
\nThis process has offered time to revisit our current data and privacy standards,\nwhich are minimal, to ensure they take into account the varied interests of\nthe Python community.\nThough we collect very little personal data from PyPI users,\nany unnecessarily held data are still subject to these kinds of requests\nin addition to the baseline risk of data compromise via malice or operator error.
\nAs a result we are currently developing new data retention and disclosure policies.\nThese policies will relate to\nour procedures for future government data requests,\nhow and for what duration we store personally identifiable information such as\nuser access records,\nand policies that make these explicit for our users and community.
\nPlease continue to watch this blog for related announcements as policies\nare finalized, published, and implemented.
\nIn order to provide as much transparency as possible,\nthe following will detail the shape of and extent of\nthe data that was contained in the responses to these subpoenas.
\nWe will not be releasing the usernames involved publicly\nor to the users themselves.
\nThese were confirmed via our database records
\nselect id, username, name from users where username = '{USERNAME}';Returning:\n
id | UUID for USERNAME\nusername | PyPI username\nname | Display name for userAnd are also publicly available at https://pypi.org/user/{USERNAME}/.
PyPI allows users to delete their accounts.
\nPyPI does not allow for the username field to be changed without admin intervention,\nand no such intervention has occurred for the users in question.\nIf they had, we would have provided records of those changes.
PyPI does allow for user to update their display names\nand keeps no record of the history of these changes.
\nPyPI only stores email addresses for individual users. No physical addresses are stored.\nOrganization accounts who have signed up for billing (not yet live)\nwill be required to provide a billing address that validates to their payment method.
\nThese were sourced from our database records and is private to PyPI.
\nselect email, user_id from user_emails where user_id ='{UUID for USERNAME}';Returning:\n
email | An email address\nuser_id | UUID for USERNAMEPyPI allows for users to add and remove email addresses without admin intervention.\nRecords of these changes are kept, and no such changes were observed in our records for\nthe usernames in question.\nIf they had, we would have provided records of those changes.
\nPyPI retains records of all changes to projects on the index,\nand has since 2002-11-01 17:11:36 UTC.
These were confirmed via our database records
\nselect * from journals where submitted_by='{USERNAME}' order by submitted_date;Returning:\n
id | An auto incrementing integer representing the \"Serial\"\nname | Name of a PyPI Project\nversion | Version of a PyPI Release if relevant\naction | Description of the action taken against the Project/Release\nsubmitted_date | ISO-8601 datetime in UTC\nsubmitted_by | PyPI Username\nsubmitted_from | IP Addressand with the exception of the submitted_by (PyPI username)\nand submitted_from (IP Address) columns\nare publicly available via our XMLRPC API.
PyPI retains records of critical user events including\naccount creation, emails sent, email address changes, logins, and login failures.\nSee this list\nfor the comprehensive set of events recorded.
\nThese were sourced from our database records
\nselect * from user_events where source_id = '{UUID for USERNAME}' order by time desc;Returning:\n
id | UUID of the event\nsource_id | UUID for USERNAME\ntag | EventTag\ntime | ISO-8601 datetime in UTC\nip_address_string | IP Address\nadditional | JSONB metadata about the event\nip_address_id | UUID of associated IPAddress objectand are private to PyPI.
\nPyPI does not record session durations.
\nSession creation (Login) times were provided as a synopsis of the data in 3b.
\nSessions are not created for uploads, but the associated login events\nfor project uploads were provided as a synopsis of the data in 3a.
\nPyPI retains records of the date that a user account was created,\nas well as the last time it was successfully logged in by any method\n(web UI or command line tool for upload).
\nselect date_joined, last_login from users where username = {USERNAME}Returning:\n
date_joined | ISO-8601 datetime in UTC\nlast_login | ISO-8601 datetime in UTCThese were sourced from our database records and are private to PyPI.
\nTypes of service utilized are \"standard\" to PyPI and include the ability to\ncreate projects, releases, and distribution files for downloads.
\nA synopsis of all IP Addresses for each username from previous records were shared.
\nThese were sourced from our database records and are private to PyPI.
\nPyPI has no cost to use for individual users\nand no such payment records or billing records exist.
\nA list of all past and any current projects associated with each username was provided.
\nThese were sourced from our database records and for past projects, are private to PyPI.
\nPyPI does not retain download logs for packages which include IP addresses.\nDownload logs are processed by a pipeline which includes GeoIP information reported by our CDN only.
\nThese records were sourced from the Google BigQuery Public dataset with the following queries:
\nSELECT * FROM `bigquery-public-data.pypi.file_downloads`\nWHERE project IN ({LIST OF PROJECT NAMES FROM 8})\nAND timestamp > '{START OF PERIOD IN QUESTION}';Ee Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-05-24-pypi-was-subpoenaed.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-05-24T06:09:00+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["compliance", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-05-25-securing-pypi-with-2fa/", "url": "https://blog.pypi.org/posts/2023-05-25-securing-pypi-with-2fa/", "title": "Securing PyPI accounts via Two-Factor Authentication", "content_html": "One of the key security promises that PyPI makes is that when you're downloading\nsomething, that only the people associated with that project are going to be able\nto upload, delete, or otherwise modify a project. That when you look at that\nproject and see that it is owned by someone that you trust, that you can be\nassured that nobody else is making changes to that package on PyPI.
\n\n\nThis promise is predicated on the security of each and every individual account\non PyPI used to create and maintain a Python project. In the past we've taken\nsteps to safeguard these accounts by\nblocking compromised passwords, strong 2FA support using\nTOTP and\nWebAuthN,\nsupport for API tokens with offline attenuation,\nenrolling the most downloaded projects into mandatory 2FA,\nand enabling short lived tokens for upload.
\nToday, as part of that long term effort to secure the Python ecosystem, we are\nannouncing that every account that maintains any project or organization\non PyPI will be required to enable 2FA on their account by the end of 2023.
\nBetween now and the end of the year, PyPI will begin gating access to certain\nsite functionality based on 2FA usage. In addition, we may begin selecting\ncertain users or projects for early enforcement.
\nThe most important things you can do to prepare are to enable 2FA for your\naccount as soon as possible, either with a\nsecurity device (preferred) or an\nauthentication app and to switch to using either\nTrusted Publishers (preferred) or\nAPI tokens to upload to PyPI.
\nAccount takeover attacks typically stem from someone using an insecure password:\nperhaps it was easy to guess, or it was reused and appeared in a breach. With\nthat insecure password, an attacker is able to gain control over a maintainers\naccount and begin to take actions as if they were that user.
\nThis is particularly problematic on a site like PyPI, where the actions that a\nperson can take include releasing software that might be used by people world\nwide, allowing an attacker to install and execute software on unsuspecting\nuser's computers. Account takeover attacks have been previously used to\ncompromise PyPI users in this\nway.
\nTwo-factor authentication immediately neutralizes the risk associated with a\ncompromised password. If an attacker has someone's password, that is no longer\nenough to give them access to that account.
\nThere's two ways to think about this question:
\nNot every account on PyPI has the same value to an attacker. An account with\naccess to the most downloaded project on PyPI can be used to attack far more\npeople than an account with access to the least downloaded project.
\nHowever, it only takes one compromised project in someone's dependency set to\ncompromise their computer. The attacker doesn't care if they get you from a\nwidely used or a niche project, just that they got you. Even worse, once\ncompromised, an attacker can extend that attack to attack other systems,\nincluding other projects on PyPI that the now compromised person maintains.
\nGiven that it only takes one compromised project, no matter how many downloads\nit gets 1, to compromise someone we want to ensure that every project is\nbeing protected by 2FA.
\nOn the flipside, an account without access to any project cannot be used to\nattack anyone 2 so it is a very low value target.
\nWe recognize that enabling 2FA for an account imposes a non zero cost both for\nthe owner of that account and for PyPI 3, so forcing that on accounts that\ncannot affect anyone but themselves is not an effective use of limited\nresources. In addition, from a practical view, the standard 2FA flow that most\npeople are used to and that PyPI implements also involves adding a 2FA token\nto an existing account rather than forcing it to be part of the registration\nflow.
\nOur expectation is that for users who currently are not a project maintainer or\norganization member, the ultimate experience will be whenever they attempt to\ntake some action that would require them to add 2FA (creating a new project,\naccepting an invite, making an organization, etc) they will be prompted to add\n2FA to their account before they can proceed.
\nThe direction of many projects in or serving the Open Source community in the\nlast 5-10 years has been an increasing focus on supply chain security,\npreventing attacks that are being delivered through the \"supply chain\", namely\nthe services and tooling used to create and consume software.
\nIn July of 2022, we announced\na security key giveway in conjunction\nwith a plan to begin mandating 2FA for the top 1% of projects on PyPI by download\ncount.
\nThe initial announcement of that mandate to the top 1%\nof projects and the related giveaway was met with mixed reactions, ranging from\npeople applauding the effort to people deciding to distribute their code on\nplaces other than PyPI. We planned to limit this to the projects in the top 1%\nof downloads because those are likely he highest value targets for an attacker,\nand because we were concerned about the impact of such a mandate on both the\nmaintainers of projects on PyPI, and on the support burden of the PyPI team\nitself.
\nAt that time last year, we did not have any plans or expectations on widening\nour net of users that would fall under that mandate, other than would occur\nnaturally as projects rose in popularity to be part the top 1%.
\nHowever, in the year since then a few things have changed.
\nAll of these together help lead us to the conclusion that we can widen our\nmandate to all project maintainers on PyPI, while minimizing the impact on\nboth project maintainers and PyPI administrators, and to do so in a way that\nimproves the sustainability of PyPI and the wider Python ecosystem.
\nThere are some people who believe that efforts to improve supply chain security\nbenefits only corporate or business users, and that individual developers should\nnot be asked to take on a uncompensated burden for their benefit.
\nWe believe this is shortsighted.
\nA compromise in the supply chain can be used to attack individual developers the\nsame as it able to attack corporate and business users. In fact, we believe\nthat individual developers, are in a more vulnerable position than corporate\nand business users. While businesses are generally able to hire staff and devote\nresources to vetting their dependencies, individual developers generally are\nnot, and must expend their own limited free time to do so 4.
\nTo make matters worse for the individual, in the case of a compromise a business\nis more likely going to have experts available to them to detect and remediate\nthe compromise, while the individual has to do this on their own. At the extreme\nends, businesses often have insurance to compensate them for any losses incurred\nwhile the individual almost always does not.
\nWe recognize that supply chain security effects everyone, no matter how big\nor how small they are, and we are dedicated to protecting all our users.
\nDonald Stufft is a PyPI administrator and maintainer of the Python Package Index since 2013.
\nTechnically a project with 0 downloads is effectively the same as a\n non-existent project, but it's easier to draw the line between\n non-existent and existent than it is to draw the line between 0 and 1\n downloads. This is particularly true on PyPI, where a large network of\n mirrors and scanners mean that no projects truly get downloaded exactly\n 0 times. ↩
\nExcept maybe the account owner themselves, by denying them access to their\n account. ↩
\nFor end users it forces them to purchase some kind of hardware token OR\n to use some sort of TOTP application. In both cases it forces them to keep\n track of something else besides their password and changes the login flow\n from what they are used to. For PyPI it increases the chance that someone\n may get locked out of their account, requiring intervention by administrators. ↩
\nNot for nothing, but PyPI is also an Open Source project, run largely by\n volunteers, and cleaning up after a compromise on PyPI is something that\n affects those volunteers significantly. ↩
\nHi there! I'm Mike, the newest member of the PyPI admin team. Nice to meet you!
\nWe've been working on reducing the amount of IP address data we store,\nand we're making progress.
\n\n\nIf you've read some of the other blogs here, you may have noticed that\nwe've been working on a number of security and privacy improvements.
\nA few months ago we started exploring what it would take\nto remove the concept of IP addresses from our stack,\nand retain the ability to safely manage the platform.
\nSome places of where IP data was historically used:
\nSecurity is a spectrum - where on one extreme, it's the wild west, no security.\nOn the other extreme, everything is locked down, impossible to use.\nWe constantly have to balance the needs and desires of our users\nwith the needs of running a sustainable, trusted platform.
\nA similar mindset can be applied to privacy - where we must strike the balance\nbetween providing a manageable service, and protecting the privacy of our users.
\nThe two main approaches we've pursued in the short term are:
\nI'll provide a couple of examples demonstrating the above.
\nAs we evaluated the different places we stored IP data,\nwe learned that our Journal entries (similar to an append-only transaction log)\nwere never exposed beyond admin users, and even then, used for admin display only.
\nWe never share that via API, or used it for operational purposed.\nSo we audited the code, removed calls to stop writing the column, and dropped it.
\nWoohoo!
\nOther places where we currently still need IP data include rate limiting,\nand fallbacks until we have backfilled the IP data with hashes and geo data.\nOur modern approach has evolved from using the IP data at display time to find\nthe relevant geo data, to storing the geo data directly in the database.
\nAnother use case is for handling abuse - we need to be able to identify\nthe source of abuse, and take action to prevent it.\nWe're thinking about how to manage that without storing IP data,\nbut we're not there yet.
\nFor the other places where we stored IP data,\nwe asked ourselves - could we use an alternative?
\nWe can't store what we don't see, so we explored what we could do to\nreduce the amount of IP data we see.
\nPretty much every request to PyPI is served via our CDN partner, Fastly.\nThey provide a number of features, including the ability to\nadd custom headers.\nWe leverage this ability already for a number of things,\nlike informing the warehouse code of the inbound encoding of the request or language.
\nWe explored whether we could use this to add a hash of the IP address,\nand use that instead of the IP address itself.\nFastly can also provide some basic geographic info about the IP address,\nwhich served our purpose of showing the user where they had connected from.
\nUsing this approach, we have Fastly pass along an already-hashed IP address,\nas well as the geographic data, to our backend, and store those for later use.
\nAnother spot we identified was web access logs.\nWe don't need real IP addresses there,\nas we rarely use them for anything other than low-level debugging,\nso a stable, hashed value serves the purpose of correlating requests.
\nFor the requests that Fastly doesn't serve, we're already hashing the IP address\nourselves prior to storage, so we could \"see\" the IP address briefly,\nbut we try to avoid storing it.\nWe don't get get the geo data for these requests,\nwe're thinking of creative and sustainable solutions already.
\nI tried to think up some questions you might have, and answer them here.\nIf you have more, please feel free to reach out to us!
\nQ: Is a hashed IP address secure?
\nA: It's more secure than a plain IP address, for sure.\nWe apply a salt (known value)\nto the IP address before hashing it.\nIt's not a perfect solution, but it's a step in the right direction.
\nThe hash is non-reversible, but since the known address space is relatively small,\nit's possible to brute force the hash to determine the original IP address.\nBy applying a salt, we require someone to possess both the salt\nand the hashed IP addresses to brute force the value.\nOur salt is not stored in the database while the hashed IP addresses are,\nwe protect against leaks revealing this information.
\nQ: Is this a response to the subpoenas?
\nA: No, we started exploring this back in 2020,\nwith a long term goal to increase end-user security,\nwhile retaining the ability to effectively steer the platform.\nWe picked it up recently as we explored our CDN partner's options for geo IP data.
\nQ: Are we done?
\nA: Nope! Security is an ongoing journey,\nand we're making strong strides to accomplish this goal.\nWe still have some work to do to replace IP data in our models,\nafter we've backfilled our models with the hashed IP data and relevant geo data,\nand clean up some of the code.
\nQ: What's next?
\nA: I can't predict every future step we're likely to take,\nbut some things we're considering:
\nWe believe the steps we're taking are in the right direction,\nand we're excited to share our progress with you.\nHopefully this enriches your understanding of the work we're doing,\nin support of maintaining a secure, trusted platform.
\nThanks for reading!
\nMike Fiedler is a PyPI administrator\nand maintainer of the Python Package Index since 2022.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-05-26-reducing-stored-ip-data.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-05-26T15:00:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-06-01-2fa-enforcement-for-upload/", "url": "https://blog.pypi.org/posts/2023-06-01-2fa-enforcement-for-upload/", "title": "Enforcement of 2FA for upload.pypi.org begins today", "content_html": "Beginning today, all uploads from user accounts with 2FA enabled\nwill be required to use an API Token\nor Trusted Publisher configuration\nin place of their password.
\n\n\nThis change has been planned\nsince 2FA was rolled out in 2019.\nIn February of 2022\nwe began notifying users on upload that this change was coming.
\nIf you have 2FA enabled and have been using only your password to upload,\nthe following email is likely familiar to you:
\n![\"Sample](\"../../assets/2023-06-01-2fa-notice-email.png\")
\n
Initially, we intended for this notice to live for six months before\nwe began enforcement.
\nHowever, some valid concerns were raised regarding\nthe use of user-scoped API tokens for new project creation.
\nWith the introduction of Trusted Publishers\nPyPI now provides a way for users to publish new projects without\nprovisioning a user-scoped token, and to continue publishing without\never provisioning a long lived API token whatsoever.
\nGiven this, and our commitment to further rolling out 2FA across PyPI,\nwe are now enforcing this policy.
\nEe Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-06-01-2fa-enforcement-for-upload.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-06-01T06:09:00+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["2fa", "security"]}, {"id": "https://blog.pypi.org/posts/2023-06-22-malware-detection-project/", "url": "https://blog.pypi.org/posts/2023-06-22-malware-detection-project/", "title": "Announcing the launch of PyPI Malware Reporting and Response project", "content_html": "We are pleased to announce that the PSF has received funding \nfrom the Center for Security and Emerging Technology \n(CSET) to develop and improve the infrastructure for \nmalware reporting and response on PyPI. This project will be executed over the coming year.
\n\n\nCurrently, malware reports are submitted to PyPI admins by email before \nbeing manually triaged and responded to. There is an opportunity for \nimprovement in streamlining the report submission process and the tools \nused to triage and respond to them. The current process cannot scale \neasily or handle duplication of reports. It is not easy to measure \ntime to remediation and is currently impossible to implement \nautomated takedown of threats.
\nThis project has the following aims:
\nAs PyPI is an integral part of the Python ecosystem, this project \nis crucial in ensuring the security of over 450,000 packages that \nare trusted by millions of Python developers. Over the next few \nweeks, we will be working with security reporters to identify \nkey elements that should be supported by the API and useful \nmetrics that would add value to PyPI security reporting. If \nyou or your colleagues are currently performing malware \nanalysis of PyPI uploads, we would love to hear from you \nat https://forms.gle/ixRoNJEPVNekFN7H7.
\nShamika Mohanan is the Packaging Project Manager at the PSF \nsince 2021.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-06-22-malware-detection-project.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-06-22T16:00:00+00:00", "authors": [{"name": "Shamika Monahan"}], "tags": ["security"]}, {"id": "https://blog.pypi.org/posts/2023-06-26-deprecate-egg-uploads/", "url": "https://blog.pypi.org/posts/2023-06-26-deprecate-egg-uploads/", "title": "Deprecation of bdist_egg uploads to PyPI", "content_html": "PEP 715, deprecating bdist_egg/.egg\nuploads to PyPI has been\naccepted.\nWe'll begin the process of implementing this today.
Please note that this does NOT remove any existing uploaded eggs from PyPI.
\n\n\nThe deprecation timeline is as follows:
\nYou can read more detailed rationale in PEP 715.\nThanks to contributor William Woodruff for his\nwork to author and propose PEP 715, as well as support the rollout of the\nimplementation.
\nEe Durbin is the Director of Infrastructure at\nthe Python Software Foundation.\nThey have been contributing to keeping PyPI online, available, and\nsecure since 2013.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-06-26-deprecate-egg-uploads.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-06-26T06:09:00+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["deprecation"]}, {"id": "https://blog.pypi.org/posts/2023-08-04-pypi-hires-safety-engineer/", "url": "https://blog.pypi.org/posts/2023-08-04-pypi-hires-safety-engineer/", "title": "PyPI hires a Safety & Security Engineer", "content_html": "\ud83d\udc4b Hi there! I'm Mike Fiedler (@miketheman)\nI've been a Python Package Index (PyPI) contributor since early 2021, and became a maintainer in 2022.\nNow I'm joining the PSF to work on PyPI full-time as the first PyPI Safety & Security Engineer.
\n\n\nWhat is that, you ask?\nWe had posted about this opening in May,\nand I'm happy to be joining the team to help improve the safety and security of PyPI.
\nWhat does that mean, Mike?\nAs safety and security are pretty broad topics, it boils down to working on any initiatives that we believe will increase the safety and security of the Package Index for all users - end users, package publishers, maintainers, and PyPI moderators and administrators. That's a huge audience!
\nAll of us deserve to have a safe and secure experience when using PyPI, and I'm excited to be able to work on this full-time.
\nMajor thanks to Amazon Web Services (AWS) for their role in funding this position, and to the PSF for their continued support of the global Python community.
\nMike Fiedler is the inaugural PyPI Safety & Security Engineer since 2023.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-08-04-pypi-hires-safety-engineer.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-08-05T06:09:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["hiring"]}, {"id": "https://blog.pypi.org/posts/2023-08-08-2fa-enforcement-for-new-users/", "url": "https://blog.pypi.org/posts/2023-08-08-2fa-enforcement-for-new-users/", "title": "2FA Enforcement for New User Registrations", "content_html": "Starting today, newly registered users must enable 2FA\nbefore they can perform any management actions on PyPI.\nThis change comes after we've also added a rule for accounts to have\na verified, primary email address for the same set of management actions.
\nAs a reminder, PyPI has supported adding 2FA since 2019.
\n\n\nThis change is continuing along the path of enforcing 2FA for all users.\nIn May of this year we announced\nthat by the end of 2023 PyPI will require all users to enable Two-Factor Authentication (2FA).\nThat post has a wealth of information on what enforcement means,\nand how folks can prepare for the change before end of year.
\nManagement actions may include any of the following:
\nThis is not an exhaustive list,\nbut should provide a good idea of the actions we're talking about.
\nIf you only need to browse, download, and install packages from PyPI\nthen a PyPI account isn't needed so this change doesn't affect you.
\nIf you've already enabled 2FA on your PyPI account,\nthis change will not affect you.\nThanks for doing your part to keep the Python ecosystem safe!
\nIf you recently registered a new PyPI account,\nyou are required to enable 2FA before you can perform any management actions.\nWhen attempting to perform a management action,\nyou may see a red banner flash at the top of the page,\nand be redirected to the 2FA setup page for your account.
\nYou will still be able to log in, browse, and download packages without 2FA.\nBut to perform any management actions, you'll need to enable 2FA.
\nAs a reminder, we will enforce the 2FA requirement for all PyPI users\nat the end of 2023.
\nThese changes intend to mitigate scenarios like account takeovers,\nwhere an attacker may be able to gain access to a user's account\nusing only an email and password (either via phishing or credential stuffing).\nIf a user's email account access is compromised,\nand the attacker is able to successfully request a password reset,\nthe attacker would still need to bypass the 2FA requirement.
\nThe more users using a Two Factor Authentication methods available,\nthe safer we all are.\nToday PyPI offers both Time-based One-Time Passwords (TOTP)\nand WebAuthn.
\nSecurity is a spectrum.\nAs long as we continue to make incremental progress,\nwe'll improve the overall posture for all users of PyPI.
\nMike Fiedler is the PyPI Safety & Security Engineer since 2023.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-08-08-2fa-enforcement-for-new-users.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-08-08T06:09:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["2fa", "security"]}, {"id": "https://blog.pypi.org/posts/2023-08-17-github-token-scanning-for-public-repos/", "url": "https://blog.pypi.org/posts/2023-08-17-github-token-scanning-for-public-repos/", "title": "GitHub now scans public issues for PyPI secrets", "content_html": "Back in 2019 we kicked off efforts\nto integrate with GitHub secret scanning.\nDue to the complexity in nature, the completed integration launched in 2021,\nwith the volunteer-led effort by Joachim Jablon (@ewjoachim)\nand the GitHub team.
\n\n\nPyPI didn't have a blog back then, but GitHub did!\nHere's a link their post.
\nThe completed integration increased security for all PyPI users.\nIf a user accidentally made their PyPI token public by committing it\nand pushing it to a GitHub public repository,\nGitHub would notify us and we would automatically revoke the token to prevent any misuse.\nThis process often completes within seconds.
\nCool, Mike, that was two years ago, so what?
\nGitHub announced yesterday that they will now notify users about any secrets exposed in:
\n\n\n... issue's title, description, or comments, including historical revisions ...
\n
This is a great enhancement to their existing secret scanning capabilities,\npreviously only scanning code.
\nAnd even better, we don't have to change anything,\nas we continue to receive the same notifications from our existing integration with GitHub. \ud83c\udf89
\nWhen implementing the integration,\nwe set up some metrics to track the number of tokens revoked.\nHere's a visualization showing the amount of inbound notifications from GitHub,\nand the number of tokens we've revoked (click for larger image):
\n![\"GitHub](\"../../assets/dd-gh-token-scanning_2023-08-16_18.12.26.png\")
\n
We show 15 months of data,\nthanks to our infrastructure sponsor Datadog.
\nThe large discrepancy between \"received\" and \"valid\" can be due to\nGitHub reporting the same token multiple times.\nThis can happen if the token is used in multiple repositories, forks, etc.
\nThe small discrepancy between \"valid\" and \"processed\" is due to that\nwe may raise an exception when attempting to email the owner of a token that has been revoked, after which we don't mark it as \"processed\" when exception is raised.\nThe second time the task runs, the token is now invalid, and doesn't increment the \"valid\" count.
\nRead GitHub's notice on The Changelog\nfor more details.
\nIf your organization performs this kind of secret scanning,\nwe'd love to hear from you and integrate with your tooling.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-08-17-github-token-scanning-for-public-repos.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-08-17T06:09:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["integrations", "security"]}, {"id": "https://blog.pypi.org/posts/2023-09-18-inbound-malware-reporting/", "url": "https://blog.pypi.org/posts/2023-09-18-inbound-malware-reporting/", "title": "Inbound Malware Volume Report", "content_html": "The current PyPI security reporting procedure\ndirects reporters to send an email to security@pypi.org with details.\nsecurity@ was previously an email alias for admin@,\na Google Group that contains all current PyPI Administrators (4 people).
I'll refer to the security@ address as the Security Inbox herein,\ndespite it not being a traditional inbox, and what changes we've made to it.
The inbound reporting workflow was roughly:
\nHere's a rough sequence diagram demonstrating the notification flow:
\n![rejected](\"https://i.kym-cdn.com/photos/images/original/001/402/192/398.jpg\"){kind=link}
sequenceDiagram\n participant External\n participant googleMail as Google Mail Routing\n participant googleGroup as Security Inbox<br/>(Google Group)\n participant pypiAdmins as PyPI Admins\n\n External->>googleMail: sends an email report\n googleMail->>googleGroup: sends an email report\n googleGroup->>+pypiAdmins: sends an email report\n pypiAdmins-->>+External: Hi, thanks for your report...\n pypiAdmins-->>-googleGroup: CC response to group\n\ntitle Previous Inbound FlowI wanted to answer a couple of questions, so as to have some data to work with:
\nAnswering these questions with the email-based system will not be 100% accurate,\nas there are some conditions that lead to inaccuracies,\nbut since we're looking at large volumes of records,\nit's unlikely that the inaccuracies will lead to material differences in the numbers.\nSome examples:
\nThis was largely to establish a baseline of where we are today.\nWith this measurement, we can then observe whether changes to our process\nhave a positive or negative impact on the volume and response times.
\nNote: This analysis is not an accurate measurement of distinct malware packages reported,\nas multiple researchers may report the same package,\nwhich will show up as distinct conversation threads.\nWe could try to normalize the packages, however since the emails are unstructured,\nthat may take more effort than is worthwhile.\nIn any case, admins respond to duplicates, so there's still non-zero effort being done.
\nEarlier this year one of our admins posted some removals stats on Twitter (before we had a blog!):
\n\n\n\nin 2022, the @pypi team removed >12,000 unique projects. each were instances of spam, typosquatting, dependency confusion, exfiltration and/or malware.
— Dustin Ingram (@di_codes) January 4, 2023
2022: ~12K (mostly malware)
2021: ~27K (mostly dep confusion)
2020: ~500
2019: 65
2018: 137
2017: 38
To determine this answer, I elected to use the emails themselves,\nsince that's the feed we have of details.\nWhile this is an imperfect data source, it should be good enough to make some early assertions.\nWe can also leverage any statistics generated thus far to assist in measuring future progress.
\nMy PyPI Google Workspace account was created on 2023-03-02 and began receiving security@pypi.org emails after that.\nGoogle Groups does not surface any APIs I could find\nthat allows an authenticated user to list/read conversations directly from the group.
\nThis approach provides a cleaner signal-to-noise ratio,\nas I often delete the random emails that are sent to the Security Inbox if they are not useful\n(spam, marketing, etc).\nWe can either accept this approach of data collection for the past few months,\nor pursue other methods for collecting longer-term data from older accounts/mailing lists,\nall subject to different data quality issues.
\nUsing the data based on 1,303 email threads sent to the Security Inbox\nby 2023-08-14, we can produce this chart:
\n![\"Inbound](\"../../assets/2023-09-18-inbound-malware-reporting/inbound-by-date.png\")
The same data, grouped by week number:
\n![\"Inbound](\"../../assets/2023-09-18-inbound-malware-reporting/inbound-by-week.png\")
One observation is that post-PyPI Weekend Suspension\nin May (Week 20), the overall volume drops for a while.\nThere's no hard evidence as to why, but it's interesting\nthat a brief disruption reduced some of the toil maintainers currently handle.
\nForm completeness, here's the monthly view:
\n![\"Inbound](\"../../assets/2023-09-18-inbound-malware-reporting/inbound-by-month.png\")
Why is response time interesting?\nThe longer a malicious package is available for end users to install,\nthe more people and systems it may affect.\nThis is further complicated to any package mirrors that capture the malware,\nand may not remove it as quickly as PyPI admins.\nWe have received anecdotal evidence from reporters that PyPI admins are\nalready quite fast at handling inbound reports (#humblebrag),\nbut let's see if we can get data out of the same emails.
Again, since the nature of email isn't 100% accurate in this case,\nwe'll rely on calculating the duration of time (in minutes)\nbetween the first message of a thread and the last message of a thread.\nThis doesn't account for the occasional behavior of a reporter\nre-using the same thread to report more packages,\nnor does it reflect any other back-and-forth communication\nbetween admins and reporters.\nAs such, removing any threads that have more than 4 total messages\nhelps remove outliers from the analysis.
\nInbound reports come in at any time of day,\nand can also be automatically generated by reporters.\nIt's common for reports to wait to be handled while we\u2019re asleep.\nWeekends and holidays often have longer response times as well.
\nOn occasion an inbound report may get overlooked,\nsomething we're trying to solve with a new system, more on this later.
\nA higher response time may indicate that the mostly-volunteer admins\nmissed responding to it the first time around.\nFor the purpose of this analysis, I've excluded 7 total response times\nthat exceed 14 days (20,160 minutes) to remove those outliers.
\n![\"Response](\"../../assets/2023-09-18-inbound-malware-reporting/med-response-times-by-date.png\")
Using median values over averages\nhelps us account for outliers at either end of the spectrum.
\nApplying a linear trend line to the collected data shows\nthat response times are generally decreasing over time, which is a good thing.
\nHere's a distribution of response times for the data collected since March 2023:
\n![\"Response](\"../../assets/2023-09-18-inbound-malware-reporting/hist-resp-time-minutes.png\")
This chart informs us that most responses to reports are completed\nin under ~485 minutes (~8 hours),\nand almost all are done within a few days of the report, with a long tail.
\nAll said, that doesn't capture the full picture of response times,\nwhich is why we've been working on a new system to help us respond faster,\nand produce better reports.
\nPart of the PyPI Malware Reporting and Response project is\nto explore ways to decrease the response times even further,\nwhile reducing the toil on maintainers, and increasing visibility to reporters.
\nAs a result of this analysis, one change we've made so far is to leverage\na shared inbox system Help Scout\nto receive inbound emails and allow us to tag, assign, and close out reports.\nThis helps is by not missing reports, and preventing duplicate responses from admins.
\nHere's how we updated our Google Workspace flow so we can continue\nto receive inbound emails, as well as copy the conversations from Help Scout\nfor long-term archival into Google Groups.
\nsequenceDiagram\n participant External\n participant googleMail as Google Mail Routing\n participant googleGroup as Google Group<br/>Archive\n participant helpScout as Security Inbox<br/>(Help Scout)\n participant pypiAdmins as PyPI Admins\n\n External->>googleMail: sends an email report\n googleMail->>+helpScout: sends an email report\n googleMail->>googleGroup: archive email\n helpScout->>+pypiAdmins: inbound notification\n pypiAdmins-->>-helpScout: Hi, thanks for your report...\n helpScout->>googleGroup: archive via auto-bcc\n helpScout-->>-External: Hi, thanks for your report...\n\ntitle Updated Inbound FlowThere's no change to end users, and we hope our change keeps us on track\nto continue to respond to reports in a timely fashion.
\nHere's an initial look at the response times since we started using Help Scout,\n(2023-09-05) using their reporting.\nWith a total of 31 conversations in the time period since we started using Help Scout,\nand comparing to the final two weeks of data from the previous chart (59 conversations),\nwe can see that the response times have improved:
\n| Response time bucket | \n% of total | \npre-Help Scout | \n
|---|---|---|
| < 15 min | \n40% | \n53% | \n
| 15-30 min | \n20% | \n10% | \n
| 30-60 min | \n20% | \n5% | \n
| 1-2 hours | \n10% | \n7% | \n
| 2-12 hours | \n10% | \n15% | \n
| 12+ hours | \n0% | \n10% | \n
We can now happily report that 80% all reports are responded to\nwithin 60 minutes of receipt,\nwith 100% are responded to within 12 hours.
\nWe will continue to monitor our response times and volumes,\nand make adjustments as needed.
\nWe're working on designing a new system to help us respond faster,\nbased in inbound reports, and provide better outcomes.\nIt's still very early, and we're incorporating a lot of ideas in the design\nbased on collective experience of PyPI admins, external researchers and reporters.
\nWe invite you to engage in the conversation on a more machine-readable\nformat for reporting malware in\nthis GitHub Issue,\nand consider sending pull requests where appropriate.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-09-18-inbound-malware-reporting.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-09-18T06:09:00+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-11-14-1-pypi-completes-first-security-audit/", "url": "https://blog.pypi.org/posts/2023-11-14-1-pypi-completes-first-security-audit/", "title": "PyPI has completed its first security audit", "content_html": "This is part one in a three-part series. See part two here, and part three here
\nWe are proud to announce that PyPI has completed its first ever external security audit.\nThis work was funded in partnership with the Open Technology Fund (OTF), a previous supporter of security-related improvements to PyPI.
\n\n\nThe Open Technology Fund selected Trail of Bits, an industry-leading cybersecurity firm with significant open-source and Python experience, to perform the audit.\nTrail of Bits spent a total of 10 engineer-weeks of effort identifying issues, presenting those findings to the PyPI team, and assisting us as we remediated the findings.
\nThe audit was focused on \"Warehouse\", the open-source codebase that powers https://pypi.org, and on \"cabotage\", the custom open-source container orchestration framework we use to deploy Warehouse.\nIt included code review of both codebases, prioritizing areas that accept user input, provide APIs and other public surfaces.\nThe audit also covered the continuous integration / continuous deployment (CI/CD) configurations for both codebases.
\nOverall, the auditors determined the Warehouse codebase \"was adequately tested and conformed to widely accepted best practices for secure Python and web development,\" and that while the cabotage codebase lacks the same level of testing, they did not identify any high severity issues in either codebase.
\nAs a result of the audit, Trail of Bits detailed 29 different advisories discovered across both codebases.\nWhen evaluating severity level of each advisory, 14 were categorized as \"informational\", 6 as \"low\", 8 as \"medium\" and zero as \"high\".\nAt the time of writing, the PyPI team has remediated all advisories that posed a significant risk in both codebases where possible, and has worked with third-party teams to unblock additional remediations where necessary.
\nIn the interest of transparency, today we are publishing the full results of the audit, as prepared by Trail of Bits.\nYou can read more about the audit from their perspective in their accompanying blog post.
\nAdditionally, in two additional blog posts published today, Mike Fiedler (PyPI Security & Safety Engineer) goes into detail about how we remediated these findings in Warehouse and Ee Durbin (Python Software Foundation Director of Infrastructure) similarly details remediation's in cabotage.
\nWe would like to thank the Open Technology Fund for their continued support of PyPI and specifically for this significant security milestone for the Python ecosystem.\nWe would also like to thank Trail of Bits for being a dependable, thorough and thoughtful partner throughout the process.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-11-14-1-pypi-completes-first-security-audit.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-11-14T00:00:00+00:00", "authors": [{"name": "Dustin Ingram"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-11-14-2-security-audit-remediation-warehouse/", "url": "https://blog.pypi.org/posts/2023-11-14-2-security-audit-remediation-warehouse/", "title": "Security Audit Remediation: Warehouse", "content_html": "This is part two in a three-part series. See part one here, and part three here.
\nThis post is a deeper dive into the remediation of the security audit findings\nfor the Warehouse - the main codebase for PyPI.org.
\nThe audit report can be found here.\nI highly recommend reading that for the fullest context first.
\n\n\nThe audit report identified 18 findings for Warehouse,\nalong with some code quality suggestions.\nThis post will focus on the findings and their remediation.\nSome of the code quality suggestions were implemented, others deferred.
\nHere's a table of the items that are relevant to warehouse, and their status:
\n| ID | \nTitle | \nSeverity | \nDifficulty | \nStatus | \n
|---|---|---|---|---|
| TOB-PYPI-1 | \nUnsafe input handling in \"Combine PRs\" workflow | \nInformational | \nHigh | \nRemediated | \n
| TOB-PYPI-2 | \nWeak signatures used in AWS SNS verification | \nMedium | \nUndetermined | \nRemediated | \n
| TOB-PYPI-4 | \nLack of rate limiting on endpoints that send email | \nLow | \nHigh | \nAccepted | \n
| TOB-PYPI-5 | \nAccount status information leak for frozen and disabled accounts | \nMedium | \nLow | \nRemediated | \n
| TOB-PYPI-6 | \nPotential race conditions in search locking | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI-7 | \nUse of multiple distinct URL parsers | \nInformational | \nUndetermined | \nRemediated | \n
| TOB-PYPI-8 | \nOverly permissive CSP headers on XML views | \nInformational | \nHigh | \nRemediated | \n
| TOB-PYPI-9 | \nMissing Permissions-Policy | \nMedium | \nHigh | \nRemediated | \n
| TOB-PYPI-10 | \nDomain separation in file digests | \nLow | \nLow | \nRemediated | \n
| TOB-PYPI-11 | \nObject storage susceptible to TOC/TOU due to temporary files | \nInformational | \nHigh | \nAccepted | \n
| TOB-PYPI-12 | \nHTTP header is silently trusted if token mismatches | \nInformational | \nHigh | \nRemediated | \n
| TOB-PYPI-13 | \nBleach library is deprecated | \nInformational | \nUndetermined | \nRemediated | \n
| TOB-PYPI-14 | \nWeak hashing in storage backends | \nMedium | \nHigh | \nAccepted | \n
| TOB-PYPI-15 | \nUncaught exception with crafted README | \nInformational | \nMedium | \nAccepted | \n
| TOB-PYPI-16 | \nReDoS via zxcvbn-python dependency | \nInformational | \nHigh | \nAccepted | \n
| TOB-PYPI-23 | \nInsecure XML processing in XMLRPC server | \nLow | \nLow | \nRemediated | \n
| TOB-PYPI-27 | \nDenial-of-service risk on tar.gz uploads | \nInformational | \nMedium | \nAccepted | \n
| TOB-PYPI-29 | \nUnescaped values in LIKE SQL queries | \nInformational | \nLow | \nAccepted | \n
IDs are non-consecutive, as the audit report included findings for cabotage as well.
\nFor some of the Remediated entries and all the Accepted ones, I'll go into more detail below.
\nNow that you've had a chance to read the original audit report,\nand can see that we've remediated most of the findings,\nI wanted to take some time to dig into some specifics of particular findings.
\nPyPI uses AWS SES to send emails to users.\nThe SES configuration is set to use a\nMessage Delivery Status\ntopic, which sends a notification to an AWS SNS topic,\nwhich then sends a notification to our application.
\nThis is useful for things like \"Accepted/Delivered\",\nbut more importantly \"Bounced\" and \"Complaint\" notifications,\nwhich change the status of user accounts.\nWe don't want to send more emails to a known bad address,\nand we don't want to send emails to users who have marked us as spam.
\nSince PyPI receives a webhook from AWS SNS,\nit needs to verify the signature of the message.
\nVerifying inbound SNS messages has generally been left up to the user.\nThe AWS SNS docs\nare clear about that.
\nWe had previously implemented signature verification for version 1,\nwhich uses the SHA1 hash algorithm,\nas that is what existed when we implemented it.
\nAs time evolved, and AWS SNS added support for SHA256,\nthe path to upgrade was still left in the hands of the user.\nSNS still defaults to SHA1 (SignatureVersion: '1'),\nand there's no Python SDK function to call to validate the signature for you.
This is also an outstanding request from boto3 users.
\nIn September 2022, AWS SNS added support for SHA256 signatures,\nand shared the details in this blog post.\nThey also added support for verification in some of the client-side SDKs,\nbut sadly Python is not one of them yet.
\nWhile we were already validating SignatureVersion 1,\nwe took this opportunity to add support for SignatureVersion 2,\nupdate our settings, and now only accept SHA256 signatures.
\nAs an AWS Hero,\nI reached out to Farrah Campbell who heads up Modern Compute Community at AWS,\nand she quickly connected me with the AWS SNS service team for a chat.\nWe discussed some of the challenges, as well as some ideas for the path forward.
\nI'm hopeful that sometime in the future we will see two big things:
\nboto3 for both signature versions\n This would enable us to remove MessageVerifier we added to warehouse, and benefit from any future enhancements to the validation process.SignatureVersion: 2 (SHA256).\n This could be a breaking change for users who have not updated their settings,\n but would be a good step forward for security.\n This make take some time, as the new signature version was only added a year ago.\n I'll leave that up to the SNS service team.We accepted this finding, as we need to send emails to unverified users\nas part of the account creation process, and we don't want to block that.
\nThe finding details that PyPI doesn't apply blanket rate limiting, which is correct.\nThe endpoints that send emails to unverified addresses are protected via rate limiting.
\nPyPI has compensating controls in place to prevent abuse,\nsuch as preventing too many password reset emails from being sent to a single user.
\nSince the risk here was to the cost and reputation of the email service,\nwe decided to accept this finding.\nAt some future point we may revisit the rate limiting strategy for sending email.
\nThis is another case of \"we implemented something that was good at the time,\nand as time went on a better solution became available\".
\nWe had written a context manager to handle locking the search index\nwhen performing updates, to prevent multiple processes from trying to update\nthe search index at the same time.\nThe implementation wasn't tied to the underlying Redis lock expiration,\nso could lead to the Redis-lock expiring, but Python believing it was still locked.
\nHere we updated our implementation to use a context manager that redis-py\nnow provides, instead of crafting our own.
A solid reminder to check back on your libraries and services now and then,\nto see if there's new features that can help you out.
\nThis is a complex timing attack, which requires a level of access to the system\nthat would allow for a more direct attack.\nThe finding itself details that if an attacker could execute this,\nthey are more likely to do other kinds of damage.
\nThe complexity of navigating between our various storage backends/client APIs\ndoes not appear to be worth the resulting defense in depth,\ngiven the required access level to exploit.
\nWe have a draft PR\nwith a start of implementation should we decide to pursue this.
\nThis is specifically about the Backblaze B2 storage backend,\none of PyPI's current object storage providers,\nwhich does not currently support SHA-256 checksums.\nThey do support SHA-1 which is useful for detecting data corruption in transit,\nbut is insufficient for non-colliding checksums - we have to use MD5 for that.
\nDuring the audit, we reached out to the Backblaze team to discuss and determined it's on their roadmap, and when they implement it, we'll update our usage accordingly.
\nThis finding discovered a bug in docutils, which PyPI uses via the readme_renderer library to render project descriptions from reStructuredText and Markdown to HTML.
The bug is tracked here,\nand has yet to see a response from the maintainers.
\nIt only applies to client-side behavior when using reStructuredText for a README,\nso we've accepted this finding.\nAdditionally, any user performing twine check prior to upload\nwill surface this issue.
Once the bug is fixed, we'll update!
\nzxcvbn-python dependencyDirect from the audit:
\n\n\nThis finding is purely informational.\nWe believe that it has virtually no impact, like many ReDoS vulnerabilities,\ndue to Warehouse\u2019s deployment architecture.
\n
Enough said.
\nThe audit began when the Warehouse deployment was on Debian 11 bullseye,\nand as part of normal maintenance we upgraded to\nDebian 12 bookworm\nwhile the audit was in progress.
Python XMLRPC uses expat for XML parsing.\nThe version of expat in Debian bullseye was 2.2.10, which was vulnerable\nto the specific attack detailed in the audit report.
With bookworm, the version of expat is 2.5.0,\nwhich is not vulnerable. (Generally considered fixed as of 2.4.1.)
This was a tricky one to track down, as once the report came in\nI was unable to reproduce the issue locally, as I had already upgraded.
\nUsing some git bisect magic, I was able to track down the exact commit\nthat fixed the issue (the bookworm upgrade),\nand then it was a matter of figuring out which library had changed.
After figuring it out, I worked with the auditors to update their\nrecommendations to reflect the upgrade.\nUntil now, the general recommendation was to adopt defusedxml,\nwhich might have proven harder as we delegate the majority of our XML parsing\nto pyramid-rpc, which uses xmlrpc.client from the standard library.
If you want to check your own installation, you can run the following:
\npython -c \"import pyexpat; print(pyexpat.EXPAT_VERSION)\"On bookworm, we get expat_2.5.0, which is not affected by the vulnerability.
This was remediated by underlying OS update to bookworm.\nDebian distributions pin to a specific version of libraries\nfor the duration of that distribution version.
This is a tricky one, as it's a tradeoff between usability and security.
\nThe audit report details a specific attack vector,\nwhere a malicious user could upload a tarball with a highly-compressed file,\nwhich would cause the server to spend a lot of time decompressing it.
\nSince we accept uploads from the general public,\nwe have to take precautions whenever possible to prevent abuse.\nWhen it comes to ZIP files (which all .whl or \"wheel\" files are), we already have a mechanism to detect\ndecompression bombs, and reject them.
However, since .tar.gz files do not advertise file sizes as metadata,\nin order to detect a decompression bomb we would have to decompress the entire file anyhow.
As the report notes, our deployment architecture compensates for this behavior,\nwhere we have a dedicated worker pool for handling uploads.
\nWe may apply additional restrictions at the system level in the future,\nbut for now we've accepted this finding.
\nLIKE SQL queriesThe risk here is that a query could \"walk the table\"\nand not take advantage of any indexes, leading to higher resource usage.
\nThe majority of the places where we use unescaped LIKE queries\nis in PyPI admin-only interface, where want to allow admins to search for users, packages, etc.
For the one place where we allow public-facing LIKE queries,\nthere are already rate limits in place to prevent abuse.\nThe table in question is also smaller than 1M rows, so walking an un-indexed column\nwould not be a significant resource usage, and takes a handful of extra milliseconds.
The potential higher resource usage would be limited to malicious internal actors,\nand if we can't trust each other, we've got bigger problems to deal with.
\nWe've accepted this finding, and will continue to monitor all of relevant resources.
\nWorking with the folks at Trail of Bits was a pleasure,\nand I'm thankful for their thoroughness and professionalism.
\nWhile the audit was funded through the Open Technology Fund,\nmy work on remediation would not have been as timely\nif not funded by Amazon Web Services\nto work as the PyPI Safety and Security Engineer.\nI am grateful for the continued support of both organizations\nin making PyPI a safer place for all Python users.
\nMike Fiedler is the inaugural PyPI Safety & Security Engineer.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-11-14-2-security-audit-remediation-warehouse.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-11-14T00:00:01+00:00", "authors": [{"name": "Mike Fiedler"}], "tags": ["security", "transparency"]}, {"id": "https://blog.pypi.org/posts/2023-11-14-3-security-audit-remediation-cabotage/", "url": "https://blog.pypi.org/posts/2023-11-14-3-security-audit-remediation-cabotage/", "title": "Security Audit Remediation: cabotage", "content_html": "This is part three in a three-part series.\nSee part one here,\nand part two here.
\nThis post is a deeper dive into the remediation of the security audit findings\nfor cabotage - the codebase that\ndeploys PyPI and its supporting services such as\nconveyor,\ncamo,\nand inspector.
\n\n\nRelative to the warehouse codebase that\nis PyPI, cabotage is not as widely known.\nThe goals of cabotage are to provide a seamless and secure way of deploying\narbitrary services into a Kubernetes cluster\nin a \"Twelve-Factor\" style.\nThere are also a number of firm opinions baked into cabotage that provide\nend-to-end TLS, protection against recovering secrets through the web UI,\nand isolation between tenants inside the cluster.
\ncabotage was initially developed in 2018 as part of the\nMozilla Open Source Support Award\nthat enabled the Python Software Foundation (PSF)\nto fund a team of contracted developers and a project manager to\ncomplete the development and deployment of warehouse and sunset the\noriginal PyPI codebase.
\nA primary goal of cabotage is to reduce the PSF Infrastructure's\ndependence on a specific provider for running PyPI,\nwhile providing self-service of configuration for project administrators\nand fully automated deployments.\nIt is in-effect a \"Platform as a Service\" that deploys applications\ninto bog-standard Kubernetes clusters, no YAML required.
\nTo date, cabotage has deployed 3,901 releases to PyPI since 2018,\nand 7,377 releases in total across its current services \"fleet\".
\nThe audit report can be found here.\nReading that before you dive in will provide the fullest context.
\nEleven findings resulted from the audit along with twelve code quality\nsuggestions.\nThis post will focus on the findings and their remediation.\nSome of the code quality suggestions were implemented, others deferred.
\nHere's a table of the items that are relevant to cabotage, and their status:
\n| ID | \nTitle | \nSeverity | \nDifficulty | \nStatus | \n
|---|---|---|---|---|
| TOB-PYPI-3 | \nVulnerable dependencies in cabotage | \nUndetermined | \nLow | \nRemediated | \n
| TOB-PYPI-17 | \nUse of shell=True in subprocesses | \nMedium | \nMedium | \nRemediated | \n
| TOB-PYPI-18 | \nUse of HMAC with SHA1 for GitHub webhook payload validation | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI-19 | \nPotential container image manipulation through malicious Procfile | \nMedium | \nHigh | \nRemediated | \n
| TOB-PYPI-20 | \nRepository confusion during image building | \nMedium | \nMedium | \nRemediated | \n
| TOB-PYPI-21 | \nBrittle X.509 certificate rewriting | \nInformational | \nUndetermined | \nAccepted | \n
| TOB-PYPI-22 | \nUnused dependencies in cabotage | \nInformational | \nUndetermined | \nRemediated | \n
| TOB-PYPI-24 | \nMissing resource integrity check of third-party resources | \nInformational | \nHigh | \nRemediated | \n
| TOB-PYPI-25 | \nBrittle secret filtering in logs | \nMedium | \nLow | \nRemediated | \n
| TOB-PYPI-26 | \nRoutes missing access controls | \nLow | \nHigh | \nRemediated | \n
| TOB-PYPI-28 | \nDeployment hook susceptible to race condition due to temp files | \nInformational | \nHigh | \nRemediated 1, 2 | \n
IDs are non-consecutive, as the audit report included findings for Warehouse as well.
\nThe maintenance of cabotage has been primarily driven by the need for new\nfeatures or to mitigate issues raised. As a result dependency management and\nupgrades have often been done as a byproduct of other changes.
\nDuring review, there were a number of dependencies with known vulnerabilities\nfound. Of the nine vulnerabilities noted, only\nGHSA-cg8c-gc2j-2wf7\nwas determined impact cabotage and was remediated by migrating to the latest\nrelease of the maintained fork of flask-security, flask-security-too\n(diff).
\nIn order to avoid falling behind in this kind of maintenance, automated\ndependency management was added along with updates to all of known vulnerable\ndependencies\n(diff).
\nshell=True in subprocessesAn attack vector was identified in the way that cabotage calls out to\nbuildctl when\nrunning container builds in development mode. A specifically crafted user-input\nhas the ability to run arbitrary shell commands on the application host.
Ultimately this was not determined to be exploitable in the production instance\nof cabotage, since the shell commands were only used when building containers\nin local development mode. The use of shell=True was removed none-the-less as\na matter of hygiene\n(diff).
Similar to the SNS verification finding in TOB-PYPI-2 for warehouse,\nthe endpoint that received webhook payloads from GitHub for automated\ndeployments was using SHA1 HMAC signatures to validate authenticity when SHA256\nHMAC signatures were available.
\nThe remediation of this finding was much more direct than the SNS finding, as\nGitHub began sending the SHA256 signature in the header, does not require\nany changes to the configuration of the webhook, and uses standard HMAC signing\nsupported by the Python standard library\n(diff).
\nAlong the same lines as TOB-PYPI-17, some user-supplied values had the ability\nto alter the cabotage controlled Dockerfile that specifies how release\ncontainers are built, which should not be modifiable.\nThrough specifically crafted process names in the Procfile,\na user could alter the resulting Dockerfile by injecting newlines.
\nRemediation was straightforward by adding additional validation of user\nsupplied process names\n(diff).
\nDue to a quirk in GitHub's API for fetching references, a given reference may\nreturn a concrete SHA/commit that belongs to a repository other than the one\nspecified in the API call. In this case by providing a reference that resolves\nto a commit on a fork of the configured repository, a user of cabotage had the\nability to intentionally (or mistakenly) configure cabotage to deploy code\nfrom a repository other than the one defined.
\nBy adding additional validation inspired by Chainguard's\nclank\ntool, cabotage now verifies that the resulting SHA for a given reference\nbelongs to the configured repository\n(diff).
\nAll containers built and deployed by cabotage are done so using short-lived\nauthentication tokens for an internally deployed Docker registry\ninstance. The cabotage application itself provides this authentication and must\npublish a public key that the registry can use to validate tokens.
\nIn order to avoid handling private-key material in the application, cabotage\nrelies heavily on Hashicorp Vault. The transit\nbackend for vault\ndid not support publishing the required x509 certificate\nthat Docker registry required when cabotage was originally developed in 2018,\nso some clever use of the cryptography library was employed to create the\nnecessary file ref.
\nIn the audit it was determined that this work around was brittle in the event\nthat an attacker had the ability to alter the length of the signature,\nresulting in an invalid x509 certificate and broken authentication for\nregistry clients.
\nIn practice, this has not been observed in the five and a half years that\nit has been in production and the result of a successful attack would only lead\nto deployments being halted. As such, we have accepted this finding for the\ntime being and will investigate the newly released x509 support in vault 1.15\nand adopt it if able (issue).
\nSimilar to TOB-PYPI-3, dependency management for cabotage was lacking. This\nled to a handful of dependencies being installed that could be additional\nexposure to vulnerabilities or attacks.
\nBy adopting pip-tools to compile and pin dependencies, only the projects\nnecessary are installed\n(diff).
When adding support for a new feature, third party JavaScript was added\nwithout subresource integrity information being added.\nThis addition guards against malicious replacement of JavaScript an is good\npractice when loading any third party code.
\nRemediation was simple, by ensuring that all CDN loaded JavaScript had the\ncorrect value set\n(diff).
\nThere was a brief period where cabotage supported building from private GitHub\nrepositories, which necessitated filtering build logs and removing the\nplaintext authentication tokens.
\nThis filtering was naive, but also no longer required. Remediation was removal\nof the filtering code, and a comment directing a future developer to the\ncorrect way of providing such authentication for builds in the future, should\nbuilding from private GitHub repositories be supported\n(diff).
\nAnother vestigial piece of code that allowed for the build context necessary\nfor container builds was identified as allowing for potentially non-public\ninformation to be leaked if a release id (UUIDv4) was guessed or surmised.
\nThis route was unauthenticated as a shortcut rather than adding a new\nauthentication method to cabotage itself.
\nThis code was made defunct when cabotage began building from contexts pulled\ndirectly from GitHub and supplied via Kubernetes secrets. Remediation was\nagain, a simple removal of the code\n(diff).
\nA final vestigial piece of code was also flagged as part of the audit which\nwas created to fetch and re-package source code from GitHub for deployments.\nThis had the very outside potential of being exploitable if an attacker\ngained access to the filesystem that the cabotage app uses for temporary\nfiles.
\nThis was similarly made defunct when cabotage began building from contexts pulled\ndirectly from GitHub. Remediation was a final simple removal of code\n(diff-0),\nand a refactor of how temporary files are created and opened\n(diff-1).
\nIn addition to the specific findings, the Trail of Bits team also made a number\nof \"Code Quality Recommendations\" and analyzed the overall maturity of the\ncodebase. Those sections of the report highlight one of the two themes I see\nin the report regarding cabotage:
\nOverall the development experience and continuous integration environment for\ncabotage is lacking.
\nThere are countless minutiae that one must consider when writing code with\nsecurity in mind.
\nIn the end, no show stopping or easily exploitable security issues were found,\nwhich is a relief! Many of the most interesting security findings were only\nexploitable by a malicious internal actor who\nalready had configuration permissions in cabotage,\nwas deploying their app there in the first place,\nor had access to the underlying systems.
\nThe takeaway I have as the sole author and maintainer of cabotage is pretty\nresounding, and addresses both themes from the report:
\n\n\nProjects with solo maintainers do not benefit from the accountability\nthat comes with collaborative development, are prone to deprioritizing\ncritical improvements to developer experience and testing, and don't\nhave the extra sets of eyes that often assist in spotting small bugs or\nimproper handling of security sensitive software.
\n
So if you're interested in infrastructure and projects that make deploying\nsoftware securely and reliably more straightforward, I'd love to talk more.\nSwing by the cabotage repo\nand consider helping build the software that deploys PyPI,\nand will soon be deploying more and more of the\nPython Software Foundation's infrastructure\nas we migrate from previously gratis PaaS hosting providers.
", "image": "https://blog.pypi.org/assets/images/social/posts/2023-11-14-3-security-audit-remediation-cabotage.png", "date_modified": "2026-03-18T18:14:42+00:00", "date_published": "2023-11-14T00:00:02+00:00", "authors": [{"name": "Ee Durbin"}], "tags": ["infrastructure", "security", "transparency"]}]}