Fuzzing CKB-VM Into Shape: The Hidden Work Behind Meepo
Do you know that a big part of Meepo’s development time went into fuzzing test, rather than writing new features?
To ensure the reliability and consistency of CKB-VM upgrades on mainnet and testnet, we fuzzed both valid and invalid transactions to catch any subtle compatibility issues early.
The goal was to:
- Ensure stable and consist transaction execution
- Prevent panics
- Verify compatibility in aspects like cycle consumption and error handling across different versions and hardforks
Here’s how fuzzing helped.
1/ Initial Verification: Transaction Replay 🔁
We began by replaying historical on-chain transactions (via `replay`) from mainnet and testnet to check if `cycle` consumption remained consistent in the upgraded CKB-VM.
This caught several mismatches:
As the chain only contains valid transactions, this method verifies past compatibility but not future cases. To broaden coverage, we turned to fuzzing to simulate diverse transaction inputs and assess compatibility across versions, including error handling in invalid transactions.
2/ First Fuzzing Attempt 🧪
We compared the execution results of `data0` and `data1` of the pre- and post-upgrade VM versions:
However, most generated test cases were invalid. The test only compared whether the errors matched, but skipped the cycle consumption for valid cases—not enough to meet our goals.
3/ Improved Fuzzing 🔧
To increase valid transaction input coverage, we refined the strategy:
- Corpus Optimization: Added valid transaction data from CKB-VM tests and CKB debugger binaries to the fuzzing corpus.
- Input Filtering: Modified fuzzing logic to only keep valid transactions in the corpus, further increasing the frequency of valid samples and enhancing `cycle` verification.
4/ Findings 😃
Improved fuzzing uncovered bugs, including:
- Crash caused by an invalid syscall parameter. Fix: github.com/libraries/ckb/commit/38279e118d3fda3c52f1d47d2062f80e19a2d523
- Instruction reordering led to mismatched `cycle` cost and memory out-of-bounds errors. Fix: github.com/libraries/ckb/commit/ea4aea7fa4cd87ce5df6dee6616466458ff5a86e
- Inconsistent error handling due to mismatched `DataPieceId` behavior. Fix: github.com/libraries/ckb/commit/af87dd355a653eaca19a643866300cc5cd907cf5
- Address truncation in x64. Fix: github.com/nervosnetwork/ckb-vm/commit/f6df535bbf8864fd14684c133b1aa8026a0b0868
- Inconsistencies in memory tracking. Fix: github.com/nervosnetwork/ckb-vm/commit/065a6457d06aa17da4f7dfa1954a2601fc7d288b
All issues were reproduced, analyzed, and added to the test corpus and the fuzzing crash directory for regression testing.
5/ Went Deeper: ISA-Level Fuzzing 🦾
In addition to compatibility testing, we fuzzed the instruction set to prevent unexpected VM panics.
See: github.com/nervosnetwork/ckb-vm-fuzzing-test
6/ Fuzzing isn’t flashy, but it pays off. 🛡️
As we know reliability is what gives developers confidence to build.
We’ll gladly keep things safe and steady—and maybe a little boring—so you don’t have to. 😎
8/ Reference Links 🔗
Fuzzing and tools:
- github.com/nervosnetwork/ckb-vm/tree/develop/fuzz
- github.com/libraries/schedfuzz
- github.com/nervosnetwork/ckb-vm-fuzzing-test
On VM 2:
- github.com/nervosnetwork/rfcs/blob/master/rfcs/0049-ckb-vm-version-2/0049-ckb-vm-version-2.md
1/ Initial Verification: Transaction Replay 🔁
We began by replaying historical on-chain transactions (via `replay`) from mainnet and testnet to check if `cycle` consumption remained consistent in the upgraded CKB-VM.
This caught several mismatches:
As the chain only contains valid transactions, this method verifies past compatibility but not future cases. To broaden coverage, we turned to fuzzing to simulate diverse transaction inputs and assess compatibility across versions, including error handling in invalid transactions.
2/ First Fuzzing Attempt 🧪
We compared the execution results of `data0` and `data1` of the pre- and post-upgrade VM versions:
However, most generated test cases were invalid. The test only compared whether the errors matched, but skipped the cycle consumption for valid cases—not enough to meet our goals.
3/ Improved Fuzzing 🔧
To increase valid transaction input coverage, we refined the strategy:
- Corpus Optimization: Added valid transaction data from CKB-VM tests and CKB debugger binaries to the fuzzing corpus.
- Input Filtering: Modified fuzzing logic to only keep valid transactions in the corpus, further increasing the frequency of valid samples and enhancing `cycle` verification.
4/ Findings 😃
Improved fuzzing uncovered bugs, including:
- Crash caused by an invalid syscall parameter. Fix: github.com/libraries/ckb/commit/38279e118d3fda3c52f1d47d2062f80e19a2d523
- Instruction reordering led to mismatched `cycle` cost and memory out-of-bounds errors. Fix: github.com/libraries/ckb/commit/ea4aea7fa4cd87ce5df6dee6616466458ff5a86e
- Inconsistent error handling due to mismatched `DataPieceId` behavior. Fix: github.com/libraries/ckb/commit/af87dd355a653eaca19a643866300cc5cd907cf5
- Address truncation in x64. Fix: github.com/nervosnetwork/ckb-vm/commit/f6df535bbf8864fd14684c133b1aa8026a0b0868
- Inconsistencies in memory tracking. Fix: github.com/nervosnetwork/ckb-vm/commit/065a6457d06aa17da4f7dfa1954a2601fc7d288b
All issues were reproduced, analyzed, and added to the test corpus and the fuzzing crash directory for regression testing.
5/ Went Deeper: ISA-Level Fuzzing 🦾
In addition to compatibility testing, we fuzzed the instruction set to prevent unexpected VM panics.
See: github.com/nervosnetwork/ckb-vm-fuzzing-test
6/ Fuzzing isn’t flashy, but it pays off. 🛡️
As we know reliability is what gives developers confidence to build.
We’ll gladly keep things safe and steady—and maybe a little boring—so you don’t have to. 😎
8/ Reference Links 🔗
Fuzzing and tools:
- github.com/nervosnetwork/ckb-vm/tree/develop/fuzz
- github.com/libraries/schedfuzz
- github.com/nervosnetwork/ckb-vm-fuzzing-test
On VM 2:
- github.com/nervosnetwork/rfcs/blob/master/rfcs/0049-ckb-vm-version-2/0049-ckb-vm-version-2.md