On July 23, 2022, the Audius governance, staking, and delegation contracts on Ethereum mainnet were compromised due to a bug in the contract initialization code that allowed repeated invocations of the initialize functions. The bug allowed an attacker to maliciously transfer 18MM $AUDIO tokens held by the Audius governance contract (referred to as the “community treasury”) to a wallet of their control and modify dynamics of the voting system to illicitly change their staked $AUDIO amounts in the network. The set of contracts were audited by the OpenZeppelin team [report] August 25, 2020 prior to deployment and some additional changes separate from the affected vulnerable code were audited by Kudelski on October 27, 2021 [report], but unfortunately this vulnerability was not caught in either case.
The Audius governance contracts utilize the OpenZeppelin proxy upgradability pattern with an override to the standard implementation within the AudiusAdminUpgradabilityProxy contract. This permits proxy upgrades to the logic contracts of the Audius system (e.g. Staking, Delegation).
In its implementation, the AudiusAdminUpgradabilityProxy uses storage slot 0 for the address of the proxyAdmin: [code]. The proxyAdmin for the Audius protocol was set to the governance system address of `0x4deca517d6817b6510798b7328f2314d3003abac` which implements various checks and balances to prevent unauthorized use (voting procedures, time delays, a community-run override process, etc.).
This caused a collision with OpenZeppelin's Initializable contract’s initialized and initializing boolean state, which are also stored in slot 0 (the first and second bytes). Because the last byte of the proxyAdmin address is `0xac`, initialized was interpreted as a truthy value. Similarly, because the second byte of the proxyAdmin address is `0xab`, initializing was also interpreted as a truthy value. This caused the initializer() modifier to always succeed:
Furthermore, because `initializing` was already true, the call was not considered to be a `topLevelCall`, which meant that both `initializing` and `initialized` were left unchanged. This allowed for repeated invocations of any function which used the `initializer` modifier. Documentation for this form of attack vector & storage collision can be found [here].
Using this bug, the attacker was able to call the initializer method of deployed Audius contracts that implement Initializable and change storage state that is intended to be set only once in initialization. Specifically, the attacker called initialize on the Governance, Staking & DelegateManagerV2 contracts [source] [source].
With this, the attacker was able to (1) Re-define voting on the Audius protocol and modify the governance contract’s guardian address (2) Set the governance address of both the Staking & DelegateManagerV2 contracts to that of a custom deployment of the Audius governance contract 0xbdbb5945f252bc3466a319cdcc3ee8056bf2e569) and abuse the Audius protocol by
Fortunately, the Audius team was able to develop and apply a patch to quickly regain control of the protocol before the attacker could do more damage.
Note from OpenZeppelin team: We have since validated the source of the issue. It’s important to note this bug exists in the implementation and does NOT affect upgradable contracts in the OpenZeppelin library which are kept safe from storage collisions by following EIP-1967.
The first step of remediation by the Audius team (and assistance of samczsun) was to deploy a patch which used the same vulnerability the attacker used in order to 1) re-gain control of the governance system and 2) block any further action from taking place on the Audius main-net eth contracts. This was achieved by proxy-upgrading each contract to a minimal BlockingContract that did not contain the same bug. This prevented further repeated invocations after relegating proxyAdmin control to a predefined address owned by the team.
Patch to regain control
This contract was deployed at [link] and had the impact of halting the governance system, token transfers, and all other reads & writes from the eth main-net Audius contracts.
After deploying the set of contracts that gave the response team control over the system as well as halted writes, the team was able to one-by-one re-deploy and initialize the proxy contracts for each of the impacted components with a proper implementation change to the OpenZeppelin Initializable contract (as seen below).
Modified OpenZeppelin Initializable contract
The key difference in the patched Initialization contract is the two padding fields that disambiguate the proxyAdmin field used in the proxy contract from the initialized and initializing fields used in the implementation contract. This change prevents repetition of the initialization flow of the deployed contract.
Additionally, to accommodate the changes in layout, the reserved storage space at the end of the file was resized to keep the overall storage size of the contract the same. The original contract reserved 50 slots, and three additional slots were used for the proxyAdmin and two padding fields, so only 47 slots should be reserved after the patch. [source]
Note: the reason that two padding fields are used is because the original intention was to patch the Initializable contract to use storage slot 2 for the boolean state. This would only require one padding field and would have resulted in a layout like that used in the BlockingContract. However, after control was regained over the contracts, it became obvious that the changes to storage that the attacker made would need to be reverted, and even though the BlockingContract allowed for direct changes to storage, an even easier method was to simply re-initialize the contracts one last time. Unfortunately, this was impossible with the original plan as the storage was already set to an initialized state. Therefore, another padding field was added so that the Initializable contract would use storage slot 3 for its state instead. Furthermore, to mitigate any risk of the initialize call being frontran, the Initializable contract was patched so that only the proxyAdmin would be allowed to initialize the contracts.
The resulting updated proxy contracts can be found here:
In order to establish confidence in the scope of damage done by the attacker before upgrading the Staking & DelegateManager contracts, the Audius team ran storage analysis & enumeration of the addresses stored in the system to determine whether there were other modifications to storage that were not clear from the initial transactions. https://link.audius.co/3zvxApA
The decision was made to delay execution of potential fixes to Staking and DelegateManager to allow for external review since they are more involved. More updates to come in this area in the coming week. These proxies remain frozen by the initial patch at the moment so are not at risk of further exploitation.
After re-deploying, upgrading and initializing each contract, the Audius team confirmed that repeated initialization invocations were not permitted on each contract and finally reset the proxyAdmin from a temporary key used to regain control back to the governance system.
There are a few takeaways our team has on how we can help prevent this sort of situation from recurring in future, and if it does, improve our ability to respond:
These learnings will be incorporated into our smart contract review processes and formal incident response process, including the checklist for the on-call engineer to be able to triage potential situations and ring the alarm bells to assemble the full team, day or night.
As noted, the vulnerability was mitigated within a few hours of discovery, and work is continuing to examine the storage modifications made by the attacker and to ensure safe resumption of the remaining Audius smart contract systems (Staking and DelegateManager).
The vast majority of Audius foundation, team, community (eg. via staking) and other funds associated with the ecosystem are safe and were unaffected by this incident. Work is in progress in collaboration with the community on possible remediations for the loss of funds, and we are fortunate that many options are still available. These will be discussed over coming weeks in the Audius governance forum, discord, and other venues before being proposed to the Audius governance process.
First and foremost, we want to thank the venerable samczsun for 1) hopping on to help within just a few minutes of our response team being assembled and 2) pair programming with us and crafting both parts of the remediation noted above over the course of a few hours, including helping our team get up to speed on some of the vagaries of the EVM/Solidity storage layout. We believe our ability to respond quickly and decisively in this situation was key to being able to prevent further damage, especially as others in the crypto community were figuring out what happened here too in real-time.
Additionally, the team at PeckShield reached out to share their findings (which corroborated ours) and collaborated with us on part of the explanation of what happened here.
The team at OpenZeppelin who have been extremely generous with their time to review the recent changes to the code made over the weekend as well as the more involved remediation for Staking and DelegateManager.
Tom Schmidt, Adam Goldberg, Will Wnekowicz, Austin Federa, and countless other Audius community members who reached out to offer their help and support however they could.
And last but certainly not least, we want to thank the broader non-technical Audius community for keeping the faith and remaining patient as this was all worked through.
