Incident report: Rootstock peg-in / peg-out service outage on June 24th
FINAL UPDATE — July 24th
- Arrowhead 6.3.1 successfully activated at Mainnet block 6,549,300, fixing the issue in the BitcoinJ library.
- PowPeg operations can now be resumed. However, peg-in operations will initially take a few more hours than they normally do until Rootstock’s bridge contract finishes syncing.
- Peg-ins initiated while the PowPeg was not operational will also be processed in the next few hours, when the bridge contract synchronization is completed.
UPDATE — July 10th
- The fix has been released as part of RSKj Arrowhead 6.3.1 and is now available for download. Details can be found in the following blog post: Introducing Arrowhead 6.3.1
- Rootstock node operators that adhere to these changes must upgrade to this version before the network upgrade is activated at Mainnet block 6,549,300.
UPDATE — July 8th
- The Rootstock Core Development team has successfully concluded testing and will proceed with the release once the open-for-comments window closes. If no significant feedback is received from the community, we plan to make the release available on July 10th, 2024.
- Since the solution involves changes to the network’s consensus rules, this fix will be treated as a network upgrade. The block activation will be announced at the time of release, with activation expected approximately two weeks later.
UPDATE — July 4th
- A patch network upgrade proposal has been published and is open for comments until July, 9th. This network upgrade proposal contains one single RSKIP describing the proposed fix to the PowPeg issue described in this report. The block height activation will be defined once the open for comments window is closed.
- Links to the patch network upgrade RSKIP and discussion forum:
UPDATE — July 2nd
- The team is currently conducting final stage tests in staging environments to ensure everything works as expected. We aim to release the fix with RSKj Arrowhead version v6.3.1 late next week, pending successful completion of all tests. As previously mentioned, the fix involves changes to the protocol consensus rules, necessitating all node operators adhering to this change to upgrade their Rootstock nodes. PowPeg fix activation is expected approximately 2 weeks after release. Specific release date and block activation height will be confirmed upon completion of tests.
UPDATE — June 28th
- Proposed fixes for RSKj, Bitcoinj, and Bitcoinj-thin are now publicly available and open for comments. The team is conducting the necessary tests to ensure these fixes work as expected. After thorough analysis, the team concluded that the solution involves changes to the network’s consensus rules, which will require a network upgrade.
- Links to GitHub pull requests containing the aforementioned fixes:
The following is the incident report for the Rootstock peg-in and peg-out service outage identified on June 24th, 2024. The service has not been resumed at the time this report was written. We understand this service issue has impacted the Rootstock platform users, and we are working to have a solution soon.
Incident Summary
Rootstock Mainnet Network is experiencing an outage in its peg-in and peg-out services, preventing users from converting BTC into RBTC and vice versa through the Rootstock native bridge called PowPeg. The root cause of this service disruption is a bug discovered in Bitcoinj, one of the dependencies used by Rootstock’s PowPeg. The bug prevents Rootstock’s bridge from adequately keeping track of Bitcoin deposits on Bitcoin Mainchain; consequently, the peg-in and peg-out workflows cannot be completed. The Bitcoinj team has already fixed this bug, and we are now working on the following steps to apply this fix to the Rootstock reference client.
It’s important to note that no user funds are at risk, and users can rely on alternative swap methods while the PowPeg service is re-established. Any outstanding current PowPeg pending operation will be processed appropriately after the issue has been resolved. However, users are advised to refrain from conducting peg-in and peg-out transactions to avoid extended waiting periods.
Root Cause Analysis
Users interact with Rootstock’s two-way peg by sending and receiving peg-in and peg-out transactions to the Bridge precompiled smart contract in Rootstock’s blockchain. This Bridge contract maintains an up-to-date view of the Bitcoin blockchain, verifies peg-in requests, and commands peg-outs. To achieve this, the Bridge contract uses a lightweight version of the Bitcoinj library to manage a Bitcoin wallet in SPV (Simple Payment Verification) mode.
On June 24th, the Bitcoinj library encountered a bug that prevents it from syncing past height 849,137 of the Bitcoin blockchain. Due to the usage of a 12-byte signed field, Bitcoinj can no longer fit block 849,138 total chainwork, throwing an “IlegalStateException” with the message “Ran out of space to store chain work!”. As a consequence, Rootstock’s Bridge contract fails to maintain the up-to-date view of the Bitcoin blockchain, and peg-in transactions cannot be processed successfully.
The Bitcoinj team has published a fix that makes the field a 12-byte unsigned field as a short to a mid-term solution while discussions on longer-term solutions continue. The Rootstock Core Development team is currently reviewing this fix and assessing the best way to incorporate it into the RSKj node to roll it out to the network and resume normal peg-in and peg-out operations.
Actions
The Rootstock Core Development team is taking the following actions to address the identified issue:
- Reviewing Bitcoinj fix to determine if it’s appropriately tackling the issue.
- Analyzing any other potential risks.
- Analyzing the following steps to implement and roll out a solution adequately.
At this moment, there is no estimated resolution date. We will publish periodic updates to keep the Rootstock community updated.
We appreciate everyone’s patience until the issue is resolved.
The Rootstock Core Development Team