Post-mortem: Dolphin Testnet V2 dApp Downtime

Dear Manta Community,

On June 16, Manta team upgraded Dolphin testnet runtime to v3.2.0 and Dolphin dApp to v2.1.0. This update introduced an incremental state synchronization RPC API between the dApp and the runtime node. This change was intended to make synchronization between the runtime node and dApp significantly faster.

The upgrade worked perfectly in development. However, when we deployed the upgrade to Dolphin testnet, the new RPC API did not work as expected. After careful examination, the team found two root issues:

  1. The RPC payload needed to be paged to get a reasonable response time.
  2. MantaPay’s UTXO storage layout needed to be optimized.

Only fixing the first issue would have reduced downtime. However, the testnet would not have been able to fulfill its long-term purpose without addressing the second issue. The team therefore chose to fix both issues. After implementing the second fix, we also needed to migrate the existing UTXO storage to the new, optimized layout. This is exactly like fixing an airplane during flight. Manta backend and full-stack teams worked day and night to:

  1. Fix the incremental RPC payload API
  2. Create a new, optimized storage layout for UTXOs
  3. Migrate to the new storage layout so that testnet users could keep existing balances
  4. Run a thorough end-to-end test of the runtime node and dApp

Now, we are releasing new versions of Dolphin runtime (v3.2.0-1) and Dolphin dApp (v2.2.0). You should notice some improvements:

  1. 2x faster synchronization time for users creating a new account
  2. 100x faster incremental synchronization time (this greatly reduces wait time for private transactions)

Enjoy making your Web3 world private on Dolphin.

4 Likes