Postmortem Incident 10
YIP-9 Error on execution#
Date: 2025-04-28
Authors: Markus
Status: DONE
Summary:#
YIP-9, which had been accepted by the Youves community, could not be successfully executed due to an error in the ordering of the submitted lambda. Although the lambda had undergone testing on ghostnet, the issue was not detected before it was submitted as YIP-9.
Youves core contributors will update the lambda, test it again on a 1:1 ghostnet setup and then submit it again as YIP-10.
Detection:#
On friday, April 24, 2025, the youves contributor who submitted YIP-9 tried to execute YIP-9 by calling execute_timelock on the DAO contract. This is the usual and only way to execute an accepted YIP. The transaction to call execute_timelock failed with an error and could not be executed.
Root causes:#
The YIP-9 lambda and the error produced was analysed by members of the youves core contributors. This led to the following conclusions:
- The single operations constructed in the lambda were correct
- The ordering of their calling was reversed, therefore incorrect
With the reversed order, the operation to remove an administrator from the first contract in the lambda was called before the DAO could accept its own administrator proposal, which caused the failure.
Trigger:#
Following the failed YIP-8, which itself also failed due to a wrong partial ordering, one contributor was tasked to correct that lambda. Due to a misunderstanding, the YIP-9 lambda was not tested on the 1:1 copy of the DAOV2 on ghostnet, but on a different contract which also had an execute entrypoint, but would not execute the lambda in the same ordering as the DAOV2 would execute it. The tests on that contract succeeded and the team was confident the lambda would execute correctly when executed from the mainnet DAOV2. It was submitted as YIP-9.
Impact:#
The attempted execution of the successfully voted YIP-9 had to fail, therefor the contracts for stXTZ were not handed over to the DAO as administrator. The core team will only launch the frontend (stacy.fi) once the DAOV2 contract has full and exclusive administrative rights. Therefor the launch will be delayed until a new YIP has passed, which sets the DAO as admin.
Resolution:#
A team of core contributors did set up a complete ghostnet set of contracts, mirroring 1:1 each contract invoved and mirroring the exact state these contracts are pre-vote. Two times a full voting cycle including lambda execution has been conducted. The newest lambda (for YIP-10) was found to be correctly executable, as can be seen on the following operation on ghostnet:
Action Items:#
What steps were done to fix or mitigate the problem? Where are we at, currently?
| Action Item | Type | Owner | State |
|---|---|---|---|
| Trying to execute YIP-9 | - | djangobits | DONE |
| Alert team | mitigate | djangobits | DONE |
| Investigate source issue | mitigate | djangobits | DONE |
| Rewrite lambda | mitigate | isa | DONE |
| Setup complete set of ghostnet contracts | mitigate | isa,djangobits | DONE |
| Test full governance vote on ghostnet | mitigate | isa,djangobits | DONE |
| Submit new lambda as YIP-10 | correct | djangobits | ONGOING |
Timeline:#
What was the exact timeline of the relevant events/actions?
(all times CET)
2025-04-24 18:30 contributor cannot execute YIP-9, alerts co-contributors
2025-04-24 19:15 error in ordering is suspected
2025-04-28 09:00 error in ordering is confirmed
2025-04-28 12:00 exact ghostnet setup is ready, ghostnet vote starts
2025-04-28 14:02 first execution suceeds
2025-04-28 14:30 second lambda is submitted, with slight changes
2025-04-18 15:15 second lambda is successfully executed
Lessons Learned#
What went wrong#
Lambdas for YIPs are written in SmartPy. A former youves contributor had created one builder script in the past, that allowed him to create these lambdas. This script was used in different scenarios, not only in YIPs. As the DAOV2 contract expects a different ordering of operations in Michelson, than other youves contracts, the builder script included an option to reverse that order. A newer youves contributor used that same builder script to create the lambda. When testing the michelson output in a ghostnet contract that was able to execute a lambda, it suceeded, leading the second contributor to the conclusion that the lambda will execute. A third, less technical contributor did submit the lambda for vote. While he checked the correctness of contained target contracts and the submitted parameters, he was not aware of the reversed order and did submit the lambda for vote.
What went well#
In general, it can be said that youves governance works well, but due to the nature of smart contract development, it has also zero tolerance to errors as a lambda operation execution is atomic, independent of how many contract calls are involved. The team was able to find the issues on Friday evening and could resolve and test them on the next working day.
Where we got lucky#
We don't take the two consecutive mistakes lightly and will improve our organisational setup. The only positive effect here is this: Tezos will soon change to the Rio protocol. This brings changes to the cycle length, which has an impact on the main stXTZ pool contract. As the YIP was not executed, the team is now able to set adjustment parameters. Otherwise a specific YIP would have been necessary.
Conclusion#
We learned that we ALWAYS need to test YIPs in a 1:1 setup on ghostnet, always using 1) a fork of the real DAOV2 contract and 2) forks of the involved contracts 3) in an identical state as the ones on mainnet. Additionally we will change our operational setup in order to ensure future YIPs are 100% correct.
We invite you to join our community channels if you have questions, criticism, remarks or improvements for the youves platform.