Postmortem Incident 7
Price deviation in the observed gold price for uXAU#
Date: 2023-09-20#
Authors: Markus, Ale, Florin#
Status: ongoing#
Summary:#
On Friday, September 15, 2023 in Block 4234866 (Sept 15, 2023 - 04:15:26 pm), the Generic Oracle from youves KT1KzdDAVkEwLUhemZBp8asMJ9nvfbp6eV9C, had registered a decrease in the price of XAU (Gold). Over the next 952 blocks, the price declined to a minimum of 962.155260 USD per XAU. This was due to an error in one of the APIs which is used to observe the price of gold and due to negligence in the oracle script, during a transition phase (see details below).
This resulted in a price being half of the price what XAU would be. Consequently, the target price in the youves target flat curve slowly decreased, which allowed users and arbitrage bots to buy uXAU cheaper than it was supposed to be. After 3 hours and 45 minutes, the gold markets were paused and no more uXAU could be traded until Monday morning, September 18.
Once gold markets reopened and the oracle reported a recent price, the target flat curve did again allow swapping. Again arbitrage bots and users started to buy cheap uXAU. Some kept it, some swapped it into PAXG and exited via Plenty bridge, essentially causing all pools to be lowered on the uXAU/PAXG side until equilibrium was met again.
Youves contributors from Ubinetic were alerted early Monday morning and started investigating. After finding the issue, measures to fix it were taken. The biggest liquidity providers were contacted in order to make them withdraw as much liquidity as needed, in order to keep the one user, who extracted uXAU below value and still had that amount sitting on his address, from selling it once the price would normalize again. Contact was established to that user in order to convince him to return some of the funds.
Until Sep 18, 2023 03:45:50 PM CET, prices reached normal levels again and most liquidity was withdrawn from all uXAU related pools.
Youves contributors already improved the oracle script in order to avoid similar situations and still work to assess the damage and recover funds. Less than USD 10k was extracted from all uXAU related pools.
More information will be published once available.
Detection:#
On Monday 18 September, 2023 3:43 am CET, youves community member @bitxia contacted @djangobits on telegram to point out that the target price on the uXAU/uUSD pool was off. At 6:22 am CET, @djangobits acknowledged the message, alerted other youves contributors and started investigating. At 7:27 am CET the reason was found.
Root causes:#
The oracle script for youves observes various prices from up to six crypto exchanges and three forex APIs. But of the forex APIs in use, only two are capable of returning a XAU price (ounce of gold in USD). The minimum of valid responses in the script was set to two sources having to response with a valid price. Three processors need to confirm this price in order for it to be ingested in the oracle contract. Forex APIs set in the script were alphavantage, 1forge and currencyapi (House Of APIs). Of these three forex APIs, alphavantage does not return a price for XAU, only for CHF and EUR, which are also fetched.
Trigger:#
On Friday afternoon September 15, 2023, around 16:15 CET (Block 4234866) currencyapi, suddenly returned an inverted price for the same request as usual. Instead of returning around 1920 USD for one ounce of gold, it returned 0.000520, which is the price of USD, expressed in XAU.
Impact:#
The oracle script which was in place (IPFS) handled the returned prices from the APIs in the following way: If the number of returned prices was odd, it returned the middle price as the median. If the number of returned prices was even, it returned the average of the two middle prices as the median. As currencyapi returned around 0.000520 and 1forge returned around 1920, an average of around 960 USD was calculated. All processors followed the same calculation and confirmed the price which was ingested from the oracle contract.
As the oracle contract allows for a maximum price deviation of 6.25% per epoch, the XAUUSD price stored on the oracle contract started to decline until it was on par with the reported median price in block 4235818 (Sep 15, 2023, 08:15:29 PM CET). During that period the target price in the uXAU/uUSD target flat curve also slowly decreased and arbitrage bots started to do arbitrage between uXAU/uUSD on youves and uXAU/PAXG.e and uXAU/uXTZ on Plenty Network, with increasing volumes.
On September 15, 2023, after 22:02 pm CET, the oracle price for XAU went stale, because gold markets were closed and the APIs didn't report updates. Consequently trading was stopped on the uXAU/uUSD dex. Until that time, only one interaction was from a user using 3route and all other transactions came from addresses which identify as arbitrage bots. Low activity and low volume arbitrage trading continued on the Plenty pools.
On September 17, 2023, 23:15 pm CET (Block 4247970), trading opened again, due to new price updates from the APIs. On September 18, 2023, 00:33 am CET, the first user started buying cheap uXAU from the uXAU/uUSD target flat curve again, using 3route.
We are still establishing the total impact and will update this post with more information. A rough estimate puts the maximum of total extracted value lower than 10k USD.
Resolution:#
When the issue with the inverted price was found, the API provider currencyapi (House of APIs) was contacted immediately at 6:47 CET am by email and phone. At 9:04 am CET the provider acknowledged the message and at 10:11 am CET the provider responded to have fixed the issue and the price reported by currencyapi was back to non-inverted:
From block 4251085 (12:15 pm CET) to block 4251914 (15:45 pm CET), the oracle price went back to normal and the target price on the target flat curve was re-established.
In the meantime youves contributors started to work on an improved version of the oracle script which includes a new API provider (twelvedata) and needs a minimum of three price sources to form a valid price. After development and testing, this oracle update was submitted to the youves keyholders and injected on Tuesday September 19, 2023, 11:18 am CET. This new intermediate oracle script (IPFS) will already improve the oracle over the previous script, but work on a completely new script which involves even more price sources have already started.
Exploiting addresses:#
We identified a number of addresses which did exploit the situation. The following is a list of users and arbitrage bots, which did profit from the situation. Note that we will update the list, it might be incomplete in terms of addresses and transactions - it currently also only analyzed the uXAU/uUSD pool and does not yet include any value extracted from the two uXAU pools on Plenty (uXAU/uXTZ and uXAU/PAXG.e):
| Address | Type |
|---|---|
| tz1X2debvxGtmv1px96EHCXZemHFr2kEZner | User |
| tz1UYLQvQAcDGFp1rsvBZaig2yZ2aR8JnNzi | User |
| tz1bkVZNAeQEixqaR5HQGhddsjAvn1CTTzDV | User |
| tz1e52JwM8fUedsr1hEnNgjKHYRBpvV6QVG5 | Bot |
| KT1VQ5TEVgwSPLUPioMRfmtSqA2B515pP3AS | Bot Contract |
| tz1dickuXBAHXxTmy63W5gB9KR6tajWBUNx6 | Bot |
Chain inspection shows that tz1X2debvxGtmv1px96EHCXZemHFr2kEZner is most likely also the one running the arbitrage bot tz1e52JwM8fUedsr1hEnNgjKHYRBpvV6QVG5 and the deployer of the contract KT1VQ5TEVgwSPLUPioMRfmtSqA2B515pP3AS.
Similarly, tz1UYLQvQAcDGFp1rsvBZaig2yZ2aR8JnNzi and tz1bkVZNAeQEixqaR5HQGhddsjAvn1CTTzDV are operated by the same user.
We contacted tz1X2debvxGtmv1px96EHCXZemHFr2kEZner and are in good faith they are willing to cooperate to some extent.
We also urge all other users who have exploited the contract to return at least some of the extracted funds to the Youves Operations Multisig wallet: KT1KAr9hnFEEFXmJ1EUS4ZYMy9G8eK7bHbQM. By doing so you would not only prove your generosity, but also that you care about the Tezos ecosystem.
Action Items:#
What steps were done to fix or mitigate the problem? Where are we at, currently?
| Action Item | Type | Owner | State |
|---|---|---|---|
| Alerting | mitigate | djangobits | DONE |
| Identfiy reason for failure | mitigate | djangobits,ale,florin | DONE |
| Contact API provider | mitigate | djangobits | DONE |
| Alert biggest Liquidity providers to withdraw liquidity | mitigate | djangobits | DONE |
| Contact the one exploiter who didn't sell his uXAU balance, propose resolution | mitigate | djangobits | DONE |
| Analyze impact | mitigate | djangobits | DONE |
| Fix API | mitigate | currencyapi | DONE |
| Improve oracle script | mitigate | florin | DONE |
| Submit new oracle script | mitigate | keyholders | DONE |
| Develop new improved oracle script | improve | florin | DONE |
| Set up monitoring for price swings and 24/7 alerting | improve | florin | DONE |
| Recover as many funds as possible | resolve | djangobits | DONE |
| Redistribute recovered funds | resolve | djangobits | ONGOING |
Timeline:#
What was the exact timeline of the relevant events/actions?
(all times CET)
- 2023-09-15 16:15 currency API suddenly reports an inverted price
- 2023-09-15 20:15 oracle price reaches 962.16 USD
- 2023-09-15 22:02 XAUUSD price becoming stale, trading stops
- 2023-09-17 23:15 XAUUSD price gets reported again, trading opens
- 2023-09-18 03:43 Bitxia reports a low price on the target flat curve
- 2023-09-18 06:22 djangobits acknowledges, alerts team, investigation starts
- 2023-09-18 07:27 inverted price was detected, API provider is alerted
- 2023-09-18 10:11 API provider reports API to be back to normal again
- 2023-09-18 12:08 djangobits contacts one of the exploiters
- 2023-09-18 12:16 Oracle reports the correct price again, price increases
- 2023-09-18 15:45 Oracle price is back at normal levels
Lessons Learned#
What went wrong#
An API suddenly reporting an inverted price was not forseen. Ususally, when APIs do not report a price, they would report an error, which would have led to no price being formed. But blaming the API would be wrong. A minimum of only two sources to observe a price is definitely not enough.
When the uXAU engine was established, it was noted that one of the three forex APIs would not report a price for XAU, which would essentially make the oracle rely on only two APIs for the price of XAU. In the case of CHF, three different APIs would report the price in random order, which would limit an impact of sudden deviation.
As uXAU was about to be shipped it was decided to go with the current solution as we already worked to soon upgrade the oracle to a completely new solution (On Demand Oracles, soon to be shipped) needed for the future trading platform. In hindsight, we should have updated the oracle already to include new APIs and increase the minimum of valid sources to at least three.
Additionally, the oracle monitoring in place was insufficient. While processors and the oracle itself are actively monitored, no monitors have been established to observe and alert on massive price swings.
Both topics are currently being addressed and improved.
What went well#
While the resolution of the problem was done as good and timely as possible, the oracle proved to do exactly what it was meant to do, eg. max. deviation per epoch and market pauses - with the drawback that the script was not good enough to account for such an incident scenario.
Where we got lucky#
First of all, we got lucky that we have loyal and engaged community members, like Bitxia, who really care about the platform. We want to express sincere thanks to him.
In this case fortunately as well was the fact that the relatively new uXAU/uUSD pool had limited liquidity and a limited number of liquidity providers and most of the liquidity was provided by LPs close to youves and some youves contributors as well. We were able to reachout to most of the bigger liquidity providers in time to ask them to remove their liquidity, before the one exploiter which was still holding a uXAU balance was able to start selling again, once the price recovered.
We are currently in contact with that user in order to reach an agreement about the return of the excess of extracted funds.
UPDATE 2023/11/10: After most of the liquidity had been removed from the pools, the user who had extracted most of the uXAU was left with no dexes he could sell the extracted uXAU against. Youves core contributors managed to buy back 5.401599759786 uXAU at a price of 7000 USDT, roughly the amount the user had spent to buy uXAU on the devaluated market. For this transaction, a youves core contributor did put 7000 USDT forward for and received 3.642039542144 of the recovered uXAU in return (which is a valuation of USD 1922 per uXAU). The remaining 1.759560217642 uXAU will be distributed pro rata among those LPs who realized a loss on their liquidity position on the youves target flat curve.
Conclusion#
While this case is not yet fully resolved, we learned the hard lesson that we may not, in any case, make a compromise in central security related parts of the platform, not even for a forseeable limited amount of time. All technical shortcuts come around in some way or another and may impact the mechanisms and security of the platform.
We will continue to analyze the impact on the liquidity providers and will try to recover as many funds as possible in order to compensate the affected liquidity providers.
It's important to note that the uXAU engine and the target flat curve worked as intended and no other engines or contracts had been affected by this incident.
We are aware that incidents like these undermine the trust in the platform. But we also learned a lot and see this experience as a part of battle-testing the platform which will eventually make it more secure and resilient.
We also want to remind users of the possible risks they face when interacting with youves, as laid out in the General Terms Of Use.
We invite you to join our community channels if you have questions, criticism, remarks or improvements for the youves platform.