On July 23, 2022, the Audius governance, staking, and delegation contracts on Ethereum mainnet were compromised due to a bug in the contract initialization code that allowed repeated invocations of the initialize functions. The bug allowed an attacker to maliciously transfer 18MM $AUDIO tokens held by the Audius governance contract (referred to as the “community treasury”) to a wallet of their control and modify dynamics of the voting system to illicitly change their staked $AUDIO amounts in the network. The set of contracts were audited by the OpenZeppelin team [report] August 25, 2020 prior to deployment and some additional changes separate from the affected vulnerable code were audited by Kudelski on October 27, 2021 [report], but unfortunately this vulnerability was not caught in either case.

‍

‍

Vulnerability

The Audius governance contracts utilize the OpenZeppelin proxy upgradability pattern with an override to the standard implementation within the AudiusAdminUpgradabilityProxy contract. This permits proxy upgrades to the logic contracts of the Audius system (e.g. Staking, Delegation).

‍

In its implementation, the AudiusAdminUpgradabilityProxy uses storage slot 0 for the address of  the proxyAdmin: [code]. The proxyAdmin for the Audius protocol was set to the governance system address of `0x4deca517d6817b6510798b7328f2314d3003abac` which implements various checks and balances to prevent unauthorized use (voting procedures, time delays, a community-run override process, etc.). 

‍

This caused a collision with OpenZeppelin's Initializable contract’s initialized and initializing boolean state, which are also stored in slot 0 (the first and second bytes). Because the last byte of the proxyAdmin address is `0xac`, initialized was interpreted as a truthy value. Similarly, because the second byte of the proxyAdmin address is `0xab`, initializing was also interpreted as a truthy value. This caused the initializer() modifier to always succeed:

require(initializing || isConstructor() || !initialized, "Contract instance has already been initialized"); 

Furthermore, because `initializing` was already true, the call was not considered to be a `topLevelCall`, which meant that both `initializing` and `initialized` were left unchanged. This allowed for repeated invocations of any function which used the `initializer` modifier. Documentation for this form of attack vector & storage collision can be found [here].

‍

Using this bug, the attacker was able to call the initializer method of deployed Audius contracts that implement Initializable and change storage state that is intended to be set only once in initialization. Specifically, the attacker called initialize on the Governance, Staking & DelegateManagerV2 contracts [source] [source].

‍

With this, the attacker was able to (1) Re-define voting on the Audius protocol and modify the governance contract’s guardian address (2) Set the governance address of both the Staking & DelegateManagerV2 contracts to that of a custom deployment of the Audius governance contract 0xbdbb5945f252bc3466a319cdcc3ee8056bf2e569) and abuse the Audius protocol by 

1. Marking an erroneous delegation of 10,000,000,000,000 $AUDIO to themselves in an attempt to pass a governance vote. (No circulating supply impact / confined to storage of Staking & Delegation contracts)
2. Marking a second erroneous delegation of 10,000,000,000,000 $AUDIO to themselves in an attempt to pass a governance vote, which did pass and transferred the funds. (No circulating supply impact / confined to storage of Staking & Delegation contracts)
3. Transferring 18,564,497 $AUDIO tokens from the community treasury: Etherscan

Fortunately, the Audius team was able to develop and apply a patch to quickly regain control of the protocol before the attacker could do more damage.
‍

Note from OpenZeppelin team: We have since validated the source of the issue. It’s important to note this bug exists in the implementation and does NOT affect upgradable contracts in the OpenZeppelin library which are kept safe from storage collisions by following EIP-1967.

Impact

• 18MM $AUDIO was moved from the Audius Community Treasury to a wallet the attacker controlled
• Internally to delegate manager & staking contracts, two transactions were performed that delegated 10T $AUDIO each. These changes were isolated to the internal state of the staking system (no new tokens were minted), and didn’t affect circulating token supply.
• Governance proposal #82 & #83 were executed on the governance system, but failed due to reason `TargetContractAddressChanged`. These two contracts were opened by the Open AUDIO Foundation to update service versions on-chain to 0.3.62.
  https://dashboard.audius.org/#/governance/proposal/82
  https://dashboard.audius.org/#/governance/proposal/83
• Governance proposal #84 was created to transfer the entirety of the Audius community pool to the attacker’s wallet (0xa62c3ced6906b188a4d4a3c981b79f2aabf2107f), but did not pass quorum, so failed during execution.
  https://dashboard.audius.org/#/governance/proposal/84
• Governance proposal #85 was created to transfer the entirety of the Audius community pool to the attacker’s wallet (0xbdbb5945f252bc3466a319cdcc3ee8056bf2e569) and executed successfully. https://dashboard.audius.org/#/governance/proposal/85

‍

‍

Timeline of Key Events

• 22:54:37 UTC - First attempt by attacker to call initialize on Audius contracts, This transaction delegates 10T $AUDIO internally to the staking contract (no token supply change or token impact). This transaction fails because no votes are cast on the proposal.
• 23:10:12 UTC - A second transaction is executed that delegates another 10T $AUDIO. Circulating supply is unaffected again, but the proposal does pass because the 10T $AUDIO is used to cast erroneous votes. [Etherscan]
• 23:11:36 UTC - Suspicious token transfer occurs on-chain as a result [Etherscan]
• 23:35:00 UTC - Audius project team receives report of the transfer, response team assembled
• 00:00:00 UTC - samczsun joins the response team to help pick apart what happened
• 00:26:00 UTC - Statement put out to report potential exploit and response team to community
• 00:30:00 UTC - Response team finds root cause, realizes that exploit is still actively exploitable
• 00:38:00 UTC - Response team begins development on a contract upgrade to mitigate exploit using the same vulnerability
• 01:29:00 UTC - Exploit mitigation contract completed
• 01:50:00 UTC - Exploit mitigation fully tested in local environment
• 01:57:37 UTC - Deployed initial fix (etherscan, blockscout, tenderly) to patch exploit, freezing currently deployed contracts (including token) as a side effect.
• 02:09:00 UTC - Statement notifying community of steps taken to prevent further exploitation
• 04:00:00 UTC - finalized patch of Initializable contract completed
• 04:42:16 UTC - after testing on a local fork, patch successfully deployed to TrustedNotifierManager contract: (0x6f08105c8CEef2BC5653640fcdbBE1e7bb519D39): [Etherscan]
• 04:50:00 UTC - analysis commences to determine breadth of contract storage changes caused by attacker’s exploit
• 07:06:21 UTC Upgraded Contract Patch Deployed to Governance: (0x4DEcA517D6817B6510798b7328F2314d3003AbAC) [Etherscan]
• 08:26:51 UTC Upgraded Contract Patch Deployed to Token, re-enabling transfers and other token-related functionality: (0x18aAA7115705e8be94bfFEBDE57Af9BFc265B998) [Etherscan]
• 09:05:00 UTC - Statement noting that token transfers are re-enabled‍
• Work continues after this point to patch ServiceTypeManager, ServiceProviderFactory, ClaimsManager, EthRewardsManager, WormholeClient, Registry

‍

‍

Remediation Details

The first step of remediation by the Audius team (and assistance of samczsun) was to deploy a patch which used the same vulnerability the attacker used in order to 1) re-gain control of the governance system and 2) block any further action from taking place on the Audius main-net eth contracts. This was achieved by proxy-upgrading each contract to a minimal BlockingContract that did not contain the same bug. This prevented further repeated invocations after relegating proxyAdmin control to a predefined address owned by the team.

‍

Patch to regain control

pragma solidity 0.8.15;

contract AudiusAdminUpgradeabilityProxy {
    address private proxyAdmin;

    function getAudiusProxyAdminAddress() external view returns (address) {
        return proxyAdmin;
    }
}

interface GovernanceLike {
    function initialize(
        address _registryAddress,
        uint256 _votingPeriod,
        uint256 _executionDelay,
        uint256 _votingQuorumPercent,
        uint16 _maxInProgressProposals,
        address _guardianAddress
    ) external;

    function guardianExecuteTransaction(
        bytes32 _targetContractRegistryKey,
        uint256 _callValue,
        string calldata _functionSignature,
        bytes calldata _callData
    ) external;
}

contract MockRegistry {
    address private targetContract;

    constructor(address targetContract_) {
        targetContract = targetContract_;
    }

    function getContract(bytes32) external view returns (address) {
        return targetContract;
    }
}

contract BlockingContract {
    address private immutable deployer = msg.sender;

    address public proxyOwner;

    uint private gap;

    /**
    * @dev Indicates that the contract has been initialized.
    */
    bool private initialized;

    /**
    * @dev Indicates that the contract is in the process of being initialized.
    */
    bool private initializing;

    function fixStorage(address newOwner) external {
        require(msg.sender == deployer, "not deployer");

        proxyOwner = newOwner;

        initialized = true;
        initializing = false;
    }

    function setStorage(uint slot, bytes32 value) external {
        require(msg.sender == deployer || msg.sender == proxyOwner, "not owner");

        assembly {
            sstore(slot, value)
        }
    }

    function exec(address to, bytes calldata data) external {
        require(msg.sender == deployer || msg.sender == proxyOwner, "not owner");

        (bool ok, ) = to.call(data);
        require(ok, "not ok");
    }
}

contract SaveTheFunds {
    constructor(address newOwner) {
        BlockingContract blocker = new BlockingContract();

        GovernanceLike governance = GovernanceLike(0x4DEcA517D6817B6510798b7328F2314d3003AbAC);
        governance.initialize(address(new MockRegistry(address(governance))), 3, 0, 1, 4, address(this));
        governance.guardianExecuteTransaction(bytes32(0x00), 0, "setAudiusProxyAdminAddress(address)", abi.encode(address(governance)));
        governance.guardianExecuteTransaction(bytes32(0x00), 0, "upgradeTo(address)", abi.encode(blocker));
        
        BlockingContract(address(governance)).fixStorage(newOwner);
        require(newOwner == AudiusAdminUpgradeabilityProxy(address(governance)).getAudiusProxyAdminAddress(), "it didn't work");

        fixProxy(governance, blocker, newOwner, 0x18aAA7115705e8be94bfFEBDE57Af9BFc265B998); // token
        fixProxy(governance, blocker, newOwner, 0xe6D97B2099F142513be7A2a068bE040656Ae4591); // staking
        fixProxy(governance, blocker, newOwner, 0x4d7968ebfD390D5E7926Cb3587C39eFf2F9FB225); // delegatemanager
        fixProxy(governance, blocker, newOwner, 0x44617F9dCEd9787C3B06a05B35B4C779a2AA1334); // claimsmanager
        fixProxy(governance, blocker, newOwner, 0xD17A9bc90c582249e211a4f4b16721e7f65156c8); // sp factory
        fixProxy(governance, blocker, newOwner, 0x5aa6B99A2B461bA8E97207740f0A689C5C39C3b0); // eth rewards manager
        fixProxy(governance, blocker, newOwner, 0x6f08105c8CEef2BC5653640fcdbBE1e7bb519D39); // trusted notifier
        fixProxy(governance, blocker, newOwner, 0x9EfB0f4F38aFbb4b0984D00C126E97E21b8417C5); // service type manager
        fixProxy(governance, blocker, newOwner, 0x6E7a1F7339bbB62b23D44797b63e4258d283E095); // wormhole client

        selfdestruct(payable(msg.sender));
    }

    function fixProxy(GovernanceLike governance, BlockingContract blocker, address newOwner, address proxy) private {
        BlockingContract(address(governance)).exec(proxy, abi.encodeWithSignature("upgradeTo(address)", blocker));
        BlockingContract(address(proxy)).fixStorage(newOwner);
        
        require(newOwner == AudiusAdminUpgradeabilityProxy(proxy).getAudiusProxyAdminAddress(), "it didn't work");
    }
}

This contract was deployed at [link] and had the impact of halting the governance system, token transfers, and all other reads & writes from the eth main-net Audius contracts.

‍

After deploying the set of contracts that gave the response team control over the system as well as halted writes, the team was able to one-by-one re-deploy and initialize the proxy contracts for each of the impacted components with a proper implementation change to the OpenZeppelin Initializable contract (as seen below).

‍

Modified OpenZeppelin Initializable contract

// SPDX-License-Identifier: MIT

pragma solidity >=0.4.24 <0.7.0;


/**
 * @title Initializable
 *
 * @dev Helper contract to support initializer functions. To use it, replace
 * the constructor with a function that has the `initializer` modifier.
 * WARNING: Unlike constructors, initializer functions must be manually
 * invoked. This applies both to deploying an Initializable contract, as well
 * as extending an Initializable contract via inheritance.
 * WARNING: When used with inheritance, manual care must be taken to not invoke
 * a parent initializer twice, or ensure that all initializers are idempotent,
 * because this is not dealt with automatically as with constructors.
 */
contract Initializable {
  address private proxyAdmin;
    
  uint256 private filler1;
  uint256 private filler2;

  /**
   * @dev Indicates that the contract has been initialized.
   */
  bool private initialized;

  /**
   * @dev Indicates that the contract is in the process of being initialized.
   */
  bool private initializing;

  /**
   * @dev Modifier to use in the initializer function of a contract.
   */
  modifier initializer() {
    require(msg.sender == proxyAdmin, "Only proxy admin can initialize");
    require(initializing || isConstructor() || !initialized, "Contract instance has already been initialized");

    bool isTopLevelCall = !initializing;
    if (isTopLevelCall) {
      initializing = true;
      initialized = true;
    }

    _;

    if (isTopLevelCall) {
      initializing = false;
    }
  }

  /// @dev Returns true if and only if the function is running in the constructor
  function isConstructor() private view returns (bool) {
    // extcodesize checks the size of the code stored in an address, and
    // address returns the current address. Since the code is still not
    // deployed when running a constructor, any checks on its code size will
    // yield zero, making it an effective way to detect if a contract is
    // under construction or not.
    address self = address(this);
    uint256 cs;
    assembly { cs := extcodesize(self) }
    return cs == 0;
  }

  // Reserved storage space to allow for layout changes in the future.
  uint256[47] private ______gap;
} 

‍

The key difference in the patched Initialization contract is the two padding fields that disambiguate the proxyAdmin field used in the proxy contract from the initialized and initializing fields used in the implementation contract. This change prevents repetition of the initialization flow of the deployed contract.

‍

Additionally, to accommodate the changes in layout, the reserved storage space at the end of the file was resized to keep the overall storage size of the contract the same. The original contract reserved 50 slots, and three additional slots were used for the proxyAdmin and two padding fields, so only 47 slots should be reserved after the patch. [source]

‍

Note: the reason that two padding fields are used is because the original intention was to patch the Initializable contract to use storage slot 2 for the boolean state. This would only require one padding field and would have resulted in a layout like that used in the BlockingContract. However, after control was regained over the contracts, it became obvious that the changes to storage that the attacker made would need to be reverted, and even though the BlockingContract allowed for direct changes to storage, an even easier method was to simply re-initialize the contracts one last time. Unfortunately, this was impossible with the original plan as the storage was already set to an initialized state. Therefore, another padding field was added so that the Initializable contract would use storage slot 3 for its state instead. Furthermore, to mitigate any risk of the initialize call being frontran, the Initializable contract was patched so that only the proxyAdmin would be allowed to initialize the contracts.

‍

The resulting updated proxy contracts can be found here:

• ✅ TrustedNotifierManager: 0x6f08105c8CEef2BC5653640fcdbBE1e7bb519D39
• ✅ Governance: 0x4DEcA517D6817B6510798b7328F2314d3003AbAC
• ✅ Token: 0x18aAA7115705e8be94bfFEBDE57Af9BFc265B998
• ✅ ServiceTypeManager: 0x9EfB0f4F38aFbb4b0984D00C126E97E21b8417C5
• ✅ ClaimsManager: 0x44617F9dCEd9787C3B06a05B35B4C779a2AA1334
• ✅ ServiceProviderFactory: 0xD17A9bc90c582249e211a4f4b16721e7f65156c8
• ✅ EthRewardsManager: 0x5aa6B99A2B461bA8E97207740f0A689C5C39C3b0
• ✅ WormholeClient: 0x6E7a1F7339bbB62b23D44797b63e4258d283E095
• ✅ Registry: 0xd976d3b4f4e22a238c1A736b6612D22f17b6f64C
• ✅ Staking: 0xe6D97B2099F142513be7A2a068bE040656Ae4591
• ✅ DelegateManager: 0x4d7968ebfD390D5E7926Cb3587C39eFf2F9FB225
• ✅ Update 7/26/22 00:05:00 UTC -Per the same mechanisms outlined above, the Staking and DelegateManagerV2 contracts have been updated to their original logical implementations and the altered staking storage has been patched.The code used to patch the contracts and steps followed have been reviewed by the Open Zeppelin team and can be found here.‍

‍

In order to establish confidence in the scope of damage done by the attacker before upgrading the Staking & DelegateManager contracts, the Audius team ran storage analysis & enumeration of the addresses stored in the system to determine whether there were other modifications to storage that were not clear from the initial transactions. https://link.audius.co/3zvxApA

‍

The decision was made to delay execution of potential fixes to Staking and DelegateManager to allow for external review since they are more involved. More updates to come in this area in the coming week. These proxies remain frozen by the initial patch at the moment so are not at risk of further exploitation.

‍

After re-deploying, upgrading and initializing each contract, the Audius team confirmed that repeated initialization invocations were not permitted on each contract and finally reset the proxyAdmin from a temporary key used to regain control back to the governance system.

‍

‍

Learnings

There are a few takeaways our team has on how we can help prevent this sort of situation from recurring in future, and if it does, improve our ability to respond:

• The Audius project team has not worked actively on Solidity/EVM-based code in nearly two years (see the date of the audit above). It took folks time to get back up to speed on all things here. Staying more in-tune with the latest state of the art of dev / debugging tooling here will help us mount more effective responses in the future.
• Audits are not bulletproof, and time spent in the market (and the resulting Lindy effect) can help build confidence but does not rule out opportunities for exploitation. These contracts were deployed in October 2020 and this vulnerability has been live in the wild since that time.
• Speed of incident team assembly is absolutely key - we were fortunate that this occurred during most of our team’s waking hours and we could get a critical mass of team members online within minutes of the initial report. We plan to set up better automated tooling to detect suspicious on-chain activity and page the on-call team members to review and triage.
• Complex storage / proxy patterns are prone to potential issues given the complexity of implementations - these patterns will be avoided in future smart contract work to avoid this class of vulnerabilities.

‍

These learnings will be incorporated into our smart contract review processes and formal incident response process, including the checklist for the on-call engineer to be able to triage potential situations and ring the alarm bells to assemble the full team, day or night.

‍

‍

Conclusion

As noted, the vulnerability was mitigated within a few hours of discovery, and work is continuing to examine the storage modifications made by the attacker and to ensure safe resumption of the remaining Audius smart contract systems (Staking and DelegateManager).

‍

The vast majority of Audius foundation, team, community (eg. via staking) and other funds associated with the ecosystem are safe and were unaffected by this incident. Work is in progress in collaboration with the community on possible remediations for the loss of funds, and we are fortunate that many options are still available. These will be discussed over coming weeks in the Audius governance forum, discord, and other venues before being proposed to the Audius governance process.

‍

Acknowledgements

First and foremost, we want to thank the venerable samczsun for 1) hopping on to help within just a few minutes of our response team being assembled and 2) pair programming with us and crafting both parts of the remediation noted above over the course of a few hours, including helping our team get up to speed on some of the vagaries of the EVM/Solidity storage layout. We believe our ability to respond quickly and decisively in this situation was key to being able to prevent further damage, especially as others in the crypto community were figuring out what happened here too in real-time.

‍

Additionally, the team at PeckShield reached out to share their findings (which corroborated ours) and collaborated with us on part of the explanation of what happened here.

‍

The team at OpenZeppelin who have been extremely generous with their time to review the recent changes to the code made over the weekend as well as the more involved remediation for Staking and DelegateManager.

‍

Tom Schmidt, Adam Goldberg, Will Wnekowicz, Austin Federa, and countless other Audius community members who reached out to offer their help and support however they could.

‍

And last but certainly not least, we want to thank the broader non-technical Audius community for keeping the faith and remaining patient as this was all worked through.