Postmortem Incident 5
Date: 2022-11-09
Authors: markus, ale
Status: complete
Summary: Although market conditions were met on Tuesday, November 8, 2022, 20:30 and some uusd/tez (v3) vaults became open for liquidations, only liquidations were suceeding which were small enough not to have any reward transactions going to the Unified Staking pool. Any liquidations with high enough amounts that would have resulted in reward transfers to the Unified Staking pool failed.
Impact: No funds were at risk at any time during the incident.
On Tuesday, November 8, 2022 around 20:30 UTC, the price of tez reached a low of around USD 1.13 across the observed exchanges. Our target price oracle used by the tez collateralized uusd engine (v3) reported a low of USD 1.131200 / tez in block 2867107 at 20:30 UTC.
As a result, we observed eight uusd/tez vaults becoming undercollateralized (below the 160% emergency price ratio), thus open for liquidation. Of these eight vaults, two had outstanding uusd debt higher than 30 uusd making them attractive to be targeted by liquidators.
Against all expectations, these vaults were not liquidated. Onchain activity reveals that there were liquidations happening against this vaults, but only with minuscule amounts, lower than what should have been possible.
On Wednesday, November 9, 00:33 UTC a user on Discord pointed out that the Unified Staking Pool was missing a default entrypoint to receive tez from the liquidation rewards, which made fail those liquidations with big enough amounts to result in a liquidation reward being sent to the Unified Staking Pool.
Detection:
Youves team (ale, markus) observed around Tuesday, November 8, 2022 - 20:30 UTC that undercollateralized uusd/tez (v3) vaults have not been liquidated for more than minuscule amounts, even though onchain activity indicates that there were users trying to liquidate them
User 'chas' reports on Wednesday, November 9, 00:33 UTC on Discord that the Unified Staking Pool contract is missing a 'default' entry point resulting in failing liquidations
Root causes: As laid out in YIP-007 (A - Changes to the token tracker engine - Point 5), it was defined that the new tracker engines, usually referred to as v3 engines, would be implemented in a way that liquidation rewards will be shared with YOU stakers and the liquidator. At the time of deployment of the v3 engines, the Unified Staking Pool contract was already in place.
The Unified Staking Pool contract KT1UZcNDxTdkn33Xx5HRkqQoZedc3mEs11yV is missing a default entry point which makes it impossible to send tez from another smart contract, in this case the tez collateralized synthetic asset engines (v3): uusd/tez KT1DHndgk8ah1MLfciDnCV2zPJrVbnnAH9fd and ubtc/tez KT1CP1C8afHqdNfBsSE3ggQhzM2iMHd4cRyt.
These engines would have to call the default entry point of the Unified Staking Pool contract during the execution of a liquidation operation group, in order to send tez to it. As the Unified Staking Pool contract is missing a default entry point, such an operation group will always fail.
All v3 engines have been tested thoroughly once they were deployed on mainnet and before they were launched publicly. We follow a vast test protocol to make sure smart contracts and frontend are working as expected and do not show side effects. Liquidations have also been tested and succeded, but during testing only very small amounts of these vaults were being liquidated. These amounts were too small to trigger a reward transaction to the Unified Staking Pool contract, so the reward transaction was never observed during testing.
Reasons why this problem was not caught during testing:
Our testing protocol was not covering the liquidation reward transaction, so the tester didn't specifically check if that transaction could be observed. Only the successful liquidation was observed during testing, not the reward distribution.
As testing took place on mainnet with the engine contracts wired to the real price oracle, the only way to test liquidations within a reasonable time frame, without the oracle price necessarily fluctuating, was the following: borrow the absolute maximum on a vault, then wait until the interest rate would render the vault undercollaterlized, then try to liquidate it. If this liquidation happens shortly after the vault becoming undercollaterlized, the liquidation amount will be extremely small and thus not resulting in a reward being sent to the Unified Staking Pool contract.
We used quite small amounts to test our smart contracts, when we tested them on mainnet, as contract bugs could in extremis result in these funds not being recoverable. This comes with the downside that some effects from bugs will sometimes not be clearly visible, as they would if operating with bigger amounts.
Trigger:
A missing default entry point on the Unified Staking Pool contract when the v3 engine contracts had been deployed.
Resolution:
A proxy contract to the Unified Staking Pool contract is deployed with the address KT1EhQsrb7f5rHWBE9MizBtS5wmAw3F8USxB. The admin of this contract is the unified staking pool. This new proxy has the default entrypoint and will accept the rewards paid in tez. Also any reward sent to in in form of an FA2 token can be forwarded away from this contract to the unified staking pool. Ultimately this proxy enables the unified staking pool to control the rewards it receives in tez.
Action Items:
| Action Item | Type | Owner | State |
|---|---|---|---|
| Identfiy reason for failure | mitigate | ale | DONE |
| Develop and test proxy contract | mitigate | florin | DONE |
| Deploy proxy contract | mitigate | florin | DONE |
| Rewire tez collateral engines to proxy (uUSD, uBTC) | mitigate | multisig signers | DONE |
Timeline: (all times UTC)
- 2022-11-08 20:30 ale and markus observe uusd/tez (v3) vaults not being liquidated
- 2022-11-09 00:33 User 'chas' reports the Unified Staking Pool missing a 'default' entry point
- 2022-11-09 07:30 Solution is defined, development of the proxy contract starts
- 2022-11-09 09:45 Deployment of the proxy contract & definition of multisig lambdas
- 2022-11-09 16:03 Execution of rewiring.
Lessons Learned#
What went well#
No funds were at risk at any time
What went wrong#
- During development of the V3 engines, the problem was not expected
- During testing the problem was not detected
Where we got lucky#
- Only a minor amount of vaults became overcollateralized for a short time period. uUSD depeg couldn't happen on a bigger scale.
- Only tez collateralized engines (v3) are affected (uusd/tez v3 and ubtc/tez)
- Market never dropped below the 100% collateral threshold.
Conclusion#
- Update engine test protocols
- all expected effects must be observed during testing
- After first tests with small amounts, tests with bigger amounts should be considered
- We have an awesome active community members reporting the issues (thank you @chas)