Introduction
As the backbone of numerous financial applications on the blockchain, writing code in Solidity and challenging at the same time. Our goal, as developers, is not just to write code, but to write code that you can trust – confidently.
In recent times, Solidity has become a hotspot for developers, many of whom are just starting with their programming journey. As most smart contracts handle sensitive financial transactions touching user's funds, that doesn't seem good. Well experienced developers usually bring a security-first mindset to their coding practices. For newcomers without any prior experience, understanding the intricacies of secure and reliable coding in Solidity can be quite challenging.
This is where this guide comes in. This handbook has been crafted to be a companion in navigating the fundamentals of testing Solidity smart contracts. I understand that there are multiple guides out there, but the resources are scattered. This handbook doesn't dive deep into specific testing pattern, but this should serve as a very good starting point to understand specific testing patterns and best practices. It doesn't stop at the basics tho. This guide also walks you through advanced strategies like mutation testing and the branching tree technique, helping you understand when and how to apply them.
At the end of the day, Solidity smart contracts are just a piece of software. Many protocols rely only on unit tests, or they just have a one large test suite focused on fork testing. But I feel that's not sufficient. The number of attacks on the ecosystem doesn't seem to reduce. Each testing method serves a unique purpose, helping developers uncover vulnerabilities in specific parts of the code. For example, fuzz tests are great for finding edge cases in simple mathematical functions, while symbolic testing shines in complex calculation scenarios.
It's crucial to recognize that not all testing methods fit every project. For instance, invariant tests might be challenging to implement when building on top of other protocols.
Getting good at testing:
Alright, let's talk about getting good at testing in Solidity, kind of like learning to ride a bike. You cannot master it overnight. And this is not a zero-to-hero guide. You just have to keep riding, falling off, and then getting back on again. Testing is the same. You write a test, it goes all wonky, and then you fix it. It's all part of the game.
Think of your code like a bunch of Lego blocks. Sometimes you think you’ve built the coolest spaceship, but then you notice it's missing a door or a wheel. That's what bugs in code are like. They're those missing pieces that you only spot when you test. And guess what? Everyone misses a piece now and then. Even the best of us!
Heads up!
This guide isn't your typical, super-serious, polished-to-perfection kind of thing. It actually started as a bunch of notes I jotted down for myself. It’s more like a casual chat over coffee, sharing what I've learned and what others have shared over time.
We’re going to take this nice and easy. No rush. Testing in Solidity, or any coding really, should be fun, not a headache.
Ok. Now make yourself a cup of tea and come back. Let's start with the B A S I C S.
Basic Testing
Basic tests are the crucial to have tests for all contracts. These tests should go hand in hand with feature development. By including these tests early, you can identify and address issues promptly, saving time and effort in the long run.
Types:
-
Unit Tests: These are the most fundamental type of tests where you check individual functions or components in isolation. They are quick to run and help in identifying the smallest of issues which might be overlooked otherwise.
-
Integration Tests: These tests check how different parts of your application work together. They are crucial for ensuring that the combination of various components or functions in your codebase interact as expected.
-
Fork Tests: Fork testing involves creating a fork of the network and then deploying your contracts to test in an environment that closely mimics the on-chain network. This helps in understanding how the contracts will behave under real-world conditions.
-
Fuzz Tests: In fuzz testing, you input random, invalid, or unexpected data to your contracts and observe how they handle such inputs. This type of testing is excellent for discovering vulnerabilities and ensuring your contracts can handle unexpected or incorrect inputs gracefully.
Remember, each type of test serves a unique purpose and contributes to building robust and secure core. These tests can help uncover approximately 90% of potential issues in your code if implemented properly.
Note: I'll be using Foundry for demonstrating the testing strategies, but you can apply them irrespective of the framework.
Unit tests
Unit testing is the simplest form of testing. As the name suggests, each unit test should just test one thing at a time. It involves testing the smallest parts of your code – often individual functions – to ensure they work as expected.
Key Characteristics:
- Isolation: Should focus on a single functionality.
- Speed: Should run quickly to facilitate rapid iterations.
- Independence: Must not rely on external systems or states.
For example, let's implement unit tests for a simple SetterGetter contract.
contract SetterGetter {
uint256 public number;
function setNumber(uint256 newNumber) public {
number = newNumber;
}
function getNumber() public view returns (uint256 _number) {
_number = number;
}
}
You can see that there are only 2 key methods available in the above contract.
- Setting value to the
number. - Retrieving the value stored.
Unit testing the setNumber() method:
function test_setNumber() public {
getterSetter.setNumber(10);
assertEq(getterSetter.number(), 10);
}
As mentioned earlier, the above function tests only one functionality: setNumber(). Note that in the assertion getterSetter.number() is used for validation and not getterSetter.getNumber(). Even though it doesn't make a big difference, we are avoiding the assumption that the user defined getNumber() method returns the actual value stored in the state number. Fewer assumptions help us implement more reliable tests!!
💡 Random Tip:
Solidity compiler includes a getter method for all the public variables (for simple types like
uint,bytes32,address, etc.). So if you need to reduce your contract's bytecode size, you can change the variables' scope tointernalorprivateand expense only the required values via a getter. You can read more about this here.
So it's always a good practice to test the actual state change by reading it directly. By doing so, we are trusting the Solidity's auto-generated getter method rather than the user-defined one. When writing tests, the developer should think like an attacker to figure out what could go wrong with the given function. It's the most difficult part in writing tests: identifying edge cases. This is where some techniques like BTT comes into picture, which we'll cover as a separate chapter.
If possible, protocols should avoid asking the developer(s) responsible for developing the feature to test it.
Do not over test!
When writing tests, it's easy to go beyond the boundaries and start over testing the functions. By over-testing, I mean, writing tests that adds very little to no value at all. Tests should be meaningful.
One example would be to pass a value greater than what uint256 can hold and make sure it fails:
- Passing an invalid type as input (string, address, etc.) to make sure it fails.
function testFail_setNumber() public {
cut.setNumber(type(uint256).max + 1);
}
We already know that Solidity provides overflow protection by default. The goal is to test the user logic, not the compiler. Therefore, it's better to avoid these kinds of tests.
Okay, now let's get back to our setNumber() unit test:
function test_setNumber() public {
getterSetter.setNumber(10);
assertEq(getterSetter.number(), 10);
}
Even though, this test works fine in our case, we're making another assumption here that the setNumber() actually updates the value. Consider the implementation of the setNumber() method as follows:
uint256 public number = 10
function setNumber(uint256 value) public {}
The previous test works for this too. But is this a valid implementation? No.
So what do we do about this?
Good question. In order to avoid such scenarios, we need to make sure that the state change actually happens. To test a state change, the best way is to validate the before and after value of the state. So the test would become something like:
function test_setNumber() public {
uint256 numberBefore = getterSetter.number();
getterSetter.setNumber(10);
uint256 numberAfter = getterSetter.number();
assertEq(numberBefore, 0);
assertEq(numberAfter, 10);
}
The scenario explained here is quite simple, but it could be more useful if you apply such testing techniques in real-world applications, for example, transfer() method of the ERC20 spec, should reduce the sender's balance while increasing the recipient's balance. But most protocols don't make this explicit check in their deposit() method where token transfer takes place. They only check for the recipient's balance after transfer. The more robust check would be to check before and after balances of both the sender and the recipient to avoid the assumption that the underlying token actually follows the ERC20 spec and is not malicious.
Implementing test for getNumber() method:
For the getter method, the test would be straightforward.
Simpler version (more assumptions):
function test_getNumber_Simple() public {
getterSetter.setNumber(10);
assertEq(getterSetter.getNumber(), 10);
}
Robust version (less assumptions):
function test_getNumber_Robust() public {
getterSetter.setNumber(322e26);
assertEq(getterSetter.getNumber(), 322e26);
assertEq(getterSetter.getNumber(), getterSetter.number());
getterSetter.setNumber(0);
assertEq(getterSetter.getNumber(), 0);
assertEq(getterSetter.getNumber(), getterSetter.number());
}
I'll leave it to the readers to examine how the latter test is quite stronger than the former.
All the code snippets in this guide are available on the GitHub for your reference.
Mocking:
In some cases, you might need to mock certain calls to unit test the functions. For ex, consider a deposit() function in which some ERC20 tokens are transferred to a Vault contract. Instead of deploying a mock erc20 contract and trying to perform an actual transferFrom() call, you can use vm.mockCall() cheatcode (from Foundry) and make the transferFrom() call to return true so that you can go ahead and test the actual logic ignoring the nuances of setting up a token contract. This facilitates the testing of the contract's logic in isolation, bypassing the complexities associated with setting up and interacting with other contracts.
deposit() method:
function deposit(uint256 _amount) external {
require(token.transferFrom(msg.sender, address(this), _amount), "Transfer failed");
balances[msg.sender] += _amount;
}
Unit test:
// Vault.t.sol
contract VaultTest is Test {
...
address tokenA = makeAddr("TokenA");
...
function test_deposit() external {
vm.mockCall(address(tokenA), abi.encodeWithSelector(IERC20.transferFrom.selector), abi.encode(true));
vault.deposit(10);
assert(vault.balances(address(this))== 10);
}
}
This approach enables focused testing on the contract in question, allowing for a more efficient and targeted validation of its logic and behavior. For comprehensive testing that involves the entire transaction flow and interaction between multiple contracts, integration tests should be implemented.
Integration tests
Unit testing is a vital step in ensuring each individual contract works as expected. However, protocols often involve several contracts working together. It's crucial to check that these contracts interact correctly, which is where integration testing becomes essential. The goal of the integration test should be to ensure that our contracts work together as expected, without focusing on the behavior of external contracts.
Points to note:
- It is essential to simulate the actual deployment environment as closely as possible, which means using real contract deployments instead of mocks. This ensures the tests reflect real-world operation and interactions.
- Integration tests should concentrate on the interaction between contracts rather than repeating validations of internal logic covered by unit tests. This approach keeps tests focused and avoids redundancy.
- Typically, integration tests follow unit tests in the development cycle. Once individual components are verified to work as expected, integration tests assess the system's overall functionality.
Consider a "Governance" contract that manages a Vault contract that manages deposits and withdrawals of ERC20 tokens. To ensure the governance and vault contracts operate without breaking, proper integration tests should be implemented. This confirms that the protocol functions properly as a whole, not just in isolation.
Example:
Below is a simple example for illustrating the integration test. There is a Governance contract that sets a value in the Vault contract.
contract Governance {
address public owner;
mapping(address vault => uint256 rewardRate) rewardRates;
constructor() {
owner = msg.sender;
}
modifier onlyOwner() {
require(msg.sender == owner, "Only the owner can perform this action");
_;
}
function setRewardRate(address _vaultAddress, uint256 _rewardRate) public onlyOwner {
rewardRates[_vaultAddress] = _rewardRate;
IVault(_vaultAddress).setRewardRate(_rewardRate);
}
}
Vault contract:
contract Vault {
...
function setRewardRate(uint256 _rewardRate) public onlyGovernance {
rewardRate = _rewardRate;
}
...
}
To ensure integration tests are effective and reflect real-world scenarios, it's important to set up the testing environment accurately. This means using actual contract deployments rather than mock addresses or simplified versions. The goal is to closely mimic how these contracts would interact in a live setting rather than using mocks.
So the integration test for the above contract would look something like:
contract GovernanceIntegrationTest is Test {
Vault vault;
Governance governance;
function setUp() public {
governance = new Governance();
vault = new Vault(address(governance));
}
function testGovernanceUpdatesRewardRate() public {
uint256 newRewardRate = 100;
governance.setRewardRate(address(vault), newRewardRate);
assertEq(vault.rewardRate(), newRewardRate, "Vault's rewardRate should be updated to 100");
}
}
The above test validates that the reward rate in the vault contract has been successfully updated by the governance contract. You can also notice that we're not validating if the rewardRates mapping is updated with the reward rate as it should be a unit test.
💡 Random Tip:
To test the functions with external call to other contracts, you can follow the mocking technique discussed in the Unit test chapter.
Key takeaways:
- Integration test should come after unit tests.
- All contracts should be properly setup, avoiding mock contracts.
- Should not repeat the validations performed in the unit tests.
Fork tests
Fork tests are very similar to Integration tests. Fork tests ensure that our contracts works together as expected but in a live environment without or less mocking. This helps us mimic the behavior of the smart contracts post deployment, helping us catch any unexpected behavior.
While mocks can help you test basic interactions quickly, they often don’t capture real-world behavior, meaning critical bugs can slip through unnoticed. This is where fork tests come in.
In this chapter, we'll walk through a simple example of how fork tests can be helpful. For this example, we can consider a simple LiquidityAdder contract that has a function to add liquidity.
contract LiquidityAdder {
IUniswapV2Router02 public uniswapRouter;
constructor(address _uniswapRouter) {
uniswapRouter = IUniswapV2Router02(_uniswapRouter);
}
function addLiquidity(
address tokenA,
address tokenB,
uint amountADesired,
uint amountBDesired
) external returns (uint amountA, uint amountB, uint liquidity) {
IERC20(tokenA).transferFrom(msg.sender, address(this), amountADesired);
IERC20(tokenB).transferFrom(msg.sender, address(this), amountBDesired);
IERC20(tokenA).approve(address(uniswapRouter), amountADesired);
IERC20(tokenB).approve(address(uniswapRouter), amountBDesired);
return uniswapRouter.addLiquidity(
tokenA,
tokenB,
amountADesired,
amountBDesired,
0,
0,
msg.sender,
block.timestamp
);
}
}
The function addLiquidity() in the above contract just pulls the tokens from the user and adds liquidity to the uniswap v2 pool. Let's add a unit test for the above method.
contract LiquidityAdderTest is Test {
LiquidityAdder liquidityAdder;
address constant UNISWAP_ROUTER = address(0xdeadbeef);
function setUp() public {
liquidityAdder = new LiquidityAdder(UNISWAP_ROUTER);
}
function testAddLiquidityMock() public {
address tokenA = address(0x1);
address tokenB = address(0x2);
// Mock token transfers
vm.mockCall(
tokenA,
abi.encodeWithSelector(IERC20.transferFrom.selector),
abi.encode(true)
);
vm.mockCall(
tokenB,
abi.encodeWithSelector(IERC20.transferFrom.selector),
abi.encode(true)
);
// Mock the addLiquidity function call
vm.mockCall(
UNISWAP_ROUTER,
abi.encodeWithSelector(IUniswapV2Router02.addLiquidity.selector),
abi.encode(1000, 1000, 1000)
);
(uint amountA, uint amountB, uint liquidity) = liquidityAdder.addLiquidity(tokenA, tokenB, 1000, 1000);
assertEq(amountA, 1000);
assertEq(amountB, 1000);
assertEq(liquidity, 1000);
}
}
When you run the above test it passes. Voila! But does it actually validate that the logic works onchain after deploying the contracts? To make sure it works, let's implement a fork test with real mainnet address without any mocks.
contract LiquidityAdderForkTest is Test {
LiquidityAdder liquidityAdder;
address constant UNISWAP_ROUTER = 0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D;
address constant USDC = 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48;
address constant WETH = 0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2;
address constant TOKEN_WHALE = 0x8EB8a3b98659Cce290402893d0123abb75E3ab28;
function setUp() public {
// Fork Ethereum mainnet
vm.createSelectFork("https://rpc.flashbots.net");
liquidityAdder = new LiquidityAdder(UNISWAP_ROUTER);
}
function testAddLiquidityFork() public {
vm.startPrank(TOKEN_WHALE);
IERC20(USDC).approve(address(liquidityAdder), 1000e6);
IERC20(WETH).approve(address(liquidityAdder), 1 ether);
(uint amountA, uint amountB, uint liquidity) = liquidityAdder.addLiquidity(USDC, WETH, 1000e6, 1 ether);
}
When you run the above test, you can see it fails with the following error:
│ │ ├─ [8384] 0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48::transferFrom(LiquidityAdder: [0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f], 0xB4e16d0168e52d35CaCD2c6185b44281Ec28C9Dc, 1000000000 [1e9])
│ │ │ ├─ [7573] 0x43506849D7C04F9138D1A2050bbF3A0c054402dd::transferFrom(LiquidityAdder: [0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f], 0xB4e16d0168e52d35CaCD2c6185b44281Ec28C9Dc, 1000000000 [1e9]) [delegatecall]
│ │ │ │ └─ ← [Revert] revert: ERC20: transfer amount exceeds allowance
│ │ │ └─ ← [Revert] revert: ERC20: transfer amount exceeds allowance
│ │ └─ ← [Revert] revert: TransferHelper: TRANSFER_FROM_FAILED
│ └─ ← [Revert] revert: TransferHelper: TRANSFER_FROM_FAILED
└─ ← [Revert] revert: TransferHelper: TRANSFER_FROM_FAILED
You might be surprised to find that it fails! The error message indicates that the Uniswap router reverted the transaction. What happened?
The issue is in our LiquidityAdder contract. We're transferring tokens to the contract itself, but we never approved the Uniswap router to spend these tokens on behalf of the contract. The mock test didn't catch this because we mocked all the calls, but the fork test revealed the bug. We can see how fork tests can be useful even if we have unit/integration tests in place.
Do we even need integration tests?
Now you might ask since mocking is dangerous and error-prone do we even need integration tests? The answer is it depends. The example we saw is quite basic. For complex protocol, single function might interact with multiple different contracts (both internal and external). In such cases, integration tests help us carefully curate tests to identify edge cases in different interactions. Also integration tests are fast. So yes, in most cases integration tests provides value.
For some protocols, integration tests might not be the most effective approach, especially when dealing with a single contract that primarily interacts with external protocols. On the other hand, for protocols that don't rely on external contract calls, fork tests may not add much value.
Therefore, it's important to tailor the test suite to the specific needs of the protocol, by focusing on what makes sense for each scenario. It provides a much higher level of confidence before deploying your contracts to mainnet.