Written by: William Entriken, Tadej Vengust
This article introduces a technique that you can use to test any deployed contract in arbitrary ways, but without spending any gas. It works on Ethereum Mainnet, Wanchain, Ethereum Ropsten, Hyperledger Burrow, Proof of Authority Network, private chains, and every other network based on Ethereum Virtual Machine.
This new technique was first demonstrated with the ERC-721 Validator, and it doesn’t work with Truffle.
This technique can help you out in cases when:
- Your contract deployment is too complicated to use Truffle
- You are testing somebody else’s contract, possibly with no access to its source code
- You want to batch test deployed contracts, regardless of how they were deployed
- You are into unusual stuff and errors in the Ethereum Yellow Paper
Note: we have written all Solidity examples in this article in a way they can be copy-pasted and compiled with Remix. Page example uses Web3 and requires MetaMask.
Clean Room Testing vs. Live Testing
The clean room testing methodology is preferred for the majority of professional smart contract developers. It is cheap (doesn’t spend gas), fast (doesn’t wait for transactions), and it provides reproducible results. Combined with Ganache (personal blockchain), Truffle is the best-known tool for this approach:
Several situations could impede your ability to use this technique. For example, your contract may require live test data (not reproducible) to confirm that specific code paths work correctly. It could be that your deployment process is too complicated to use Truffle. Maybe, you want to test compatibility with a contract that has no source code available. Or, perhaps, your CEO had you deploy a smart contract before test cases were finished.
Live testing is a new methodology that allows you to perform validation tests on smart contracts on Mainnet while having access to all Mainnet data.
Of course, you can also deploy your stubs to Mainnet. However, this could turn out as unnecessarily expensive and slow. Also note that in this setup, the validation process has full access to all Mainnet contracts and data.
Testing With a Deployed Contract
The live testing technique can be performed by deploying a smart contract written explicitly for testing to Mainnet and then calling that contract.
In our first example, we will test whether or not Su Squares (an ERC-721 contract) provides a total supply of 10,000 tokens. (Su Squares was deployed with, and is intended to always comprise, precisely 10,000 non-fungible tokens).
Please note that we picked the 10,000 tokens supply test because it allows for a simple demonstration of the technique. The same technique can be applied in much more complex tests.
pragma solidity 0.5.6;
import "https://github.com/0xcert/ethereum-erc721/src/contracts/tokens/erc721-enumerable.sol";
contract SuSquaresTests
{
ERC721Enumerable testSubject;
/**
* @notice Deploy test contract with a subject to be used for all tests.
*/
constructor(
address _testSubject
)
public
{
testSubject = ERC721Enumerable(_testSubject);
}
/**
* @notice Test token supply invariant.
*/
function testIsTotalSupply10000()
external
view
returns (bool testResult)
{
testResult = testSubject.totalSupply() == 10000;
}
}
This contract has been deployed on Mainnet already, and you can play with it at 0x37d3bffed6f784d2cb5542bb9d9007c16e5938df.
For some good reason, this test performs actions that require changing the Mainnet state. This makes the test expensive to deploy and costly to run. Also, the test takes time to execute (due to one-block confirmation).
In the following chapter, we will remove one of these expenses.
Tests With ESTIMATEGAS
We can eliminate the cost of executing this test function by using a built-in feature of the standard Ethereum client JSON-RPC (and by extension Web3.js), namely the ESTIMATEGAS.
In the diagram below, “Client” refers to a Geth/Parity console session, Truffle or a web browser using web3.js.
The parameters to estimateGas are equivalent to making a normal transaction except that you do not have to sign it and it delivers the amount of gas used. (It is also possible to have a false/failed test result if all gas is used). Simply put, estimateGas runs everything the same way as when making a transaction, but instead of committing the transaction it provides the information on the cost of the transaction.
You should also note that in the case of running estimateGas on a transaction that reverts (fails), the returned gas amount will equal the total gas available in a block.
With this in mind, we can alter the above test case so that the test result is indicated by the amount of gas used:
/**
* @notice Test token supply invariant.
*/
function testIsTotalSupply10000()
external
{
require(testSubject.totalSupply() == 10000);
}
In this new version, if the test fails, the gas spent will represent the total gas available.
Now we have established a way to run arbitrary test cases without the need to pay each time we use them. In the next part, we will also eliminate the expense of deploying the contract. But before proceeding, we must violate a statement from the Ethereum Yellow Paper.
The Singleton Transaction Pattern
The current version of the Ethereum Yellow Paper (the specification which all Ethereum clients should follow) states:
There are [only] two types of transactions: those which result in message calls and those which result in the creation of new accounts with associated code (known informally as “contract creation”).
This is actually inaccurate since the Ethereum reference implementations we have checked — Geth, Parity and all other implementations we know — all allow for a third type of transaction. We will abuse this fact so that we can refactor the test contract, immediately perform a test case and return results — without the need to actually deploy the contract.
How can we do that? By moving the test case into the constructor.
pragma solidity 0.5.6;
import "https://github.com/0xcert/ethereum-erc721/src/contracts/tokens/erc721-enumerable.sol";
contract SuSquaresTests
{
/**
* @notice Deploy test contract with a subject to be used for all tests.
*/
constructor(
ERC721Enumerable _testSubject
)
public
{
testIsTotalSupply10000(_testSubject);
}
/**
* @notice Test token supply invariant.
*/
function testIsTotalSupply10000(
ERC721Enumerable _testSubject
)
public
{
require(_testSubject.totalSupply() == 10000);
}
}
This example has violated the Ethereum Yellow Paper specification since the transaction that created a contract has also performed a message call into that contract. One call can even create multiple contracts and perform multiple message calls to those contracts, as we will show in the stubs example later.
You must compile this contract to bytecode, first. Here is how you can achieve this with Remix IDE:
<html>
<head>
<title>Estimate gas tests</title>
</head>
<body>
<script src="https://cdn.jsdelivr.net/gh/ethereum/[email protected]/dist/web3.min.js" type="text/javascript"></script>
<script>
if (typeof window.web3 !== "undefined" && typeof window.web3.currentProvider !== "undefined") {
var web3 = new Web3(window.web3.currentProvider);
} else {
alert("No web3");
}
const abi = [
{
"constant": false,
"inputs": [
{
"name": "_testSubject",
"type": "address"
}
],
"name": "testIsTotalSupply10000",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"inputs": [
{
"name": "_testSubject",
"type": "address"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "constructor"
}
];
const bytecode = "0x608060405234801561001057600080fd5b5060405160208061021d8339810180604052602081101561003057600080fd5b81019080805190602001909291905050506100508161005660201b60201c565b506100e7565b6127108173ffffffffffffffffffffffffffffffffffffffff166318160ddd6040518163ffffffff1660e01b815260040160206040518083038186803b15801561009f57600080fd5b505afa1580156100b3573d6000803e3d6000fd5b505050506040513d60208110156100c957600080fd5b8101908080519060200190929190505050146100e457600080fd5b50565b610127806100f66000396000f3fe6080604052348015600f57600080fd5b506004361060285760003560e01c806347d54a4514602d575b600080fd5b606c60048036036020811015604157600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050606e565b005b6127108173ffffffffffffffffffffffffffffffffffffffff166318160ddd6040518163ffffffff1660e01b815260040160206040518083038186803b15801560b657600080fd5b505afa15801560c9573d6000803e3d6000fd5b505050506040513d602081101560de57600080fd5b81019080805190602001909291905050501460f857600080fd5b5056fea165627a7a723058201eaa86ca0d93585836b0d06315ab78c83079efff596c1a0417c5195ad0fc9e2d0029";
async function test(){
const contract = new web3.eth.Contract(abi);
const contractAddress = document.getElementById("contractAddress").value;
// First we check max gas limit.
let gasLimit = 8000029;
await web3.eth.getBlock("latest", false, (error, result) => {
gasLimit = result.gasLimit;
});
contract.deploy({
data: bytecode,
arguments: [contractAddress]
}).estimateGas({
from: '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2'
}, (err, gas) => {
// If there is no error and gas is bellow has limit the test succedded.
if (err === null && gas < gasLimit) {
document.getElementById("console").innerHTML = "Contract has 10000 tokens."
} else {
document.getElementById("console").innerHTML = "Contract does NOT have 10000 tokens."
}
});
}
</script>
<h3>Testing if ERC-721 enumerable contract has a total supply of 10.000.</h3>
<h4>Quick copy contract addresses for tests:</h4>
<b>Su Squares (passing test): </b>0xE9e3F9cfc1A64DFca53614a0182CFAD56c10624F <br/>
<b>Axie Infinity (failing test): </b>0xf5b0a3efb8e8e4c201e2a935f110eaaf3ffecb8d <br/><br/>
<input id="contractAddress" type="text" placeholder="Input contract address" />
<button onclick="test()">Test</button>
<p id="console"></p>
</body>
</html>
This is an arbitrary test case written in Solidity. It can be called against arbitrary contracts, it can use arbitrary data and make arbitrary calls to other contracts.
Oh, and P.S.:
This same test case can be run against any EVM network, such as Ethereum Mainnet, Wanchain, Ethereum Ropsten, Hyperledger Burrow, Proof of Authority Network, as well as private chains.
Stubs and Artifacts
A stub is a specific, additional contract that is required just to validate a test subject contract. For example, if you want to validate the ability of an ERC-721 contract to transfer tokens, you will need a contract that can receive tokens. You can check a fully developed example at 0xcert.
Below is a basic example for an ERC-721 receiver stub. You should note that this example assumes that the contract owns a token. You can achieve this in several ways that we describe in the following section. The “giver” technique has also been implemented in the 0xcert example linked here.
pragma solidity 0.5.6;
import "https://github.com/0xcert/ethereum-erc721/src/contracts/tokens/erc721.sol";
import "https://github.com/0xcert/ethereum-erc721/src/contracts/tokens/erc721-token-receiver.sol";
contract StubsAndArtifacts
{
/**
* @notice Deploy test contract with subject to be used for all tests.
*/
constructor(
ERC721 _testSubject,
uint256 _tokenId
)
public
{
// This test assumes that this contract owns that particular _tokenId. You first
// need to get a token. There are multiple ways to achieve this like "buying" it
// before running test or using "the giver" technique. All of this is explained
// below in the article.
testDoesCorrectlyTransferData(_testSubject, _tokenId);
}
function testDoesCorrectlyTransferData(
ERC721 _testSubject,
uint256 _tokenId
)
public
{
StubTokenReceiver stub = new StubTokenReceiver();
// we expect failure
_testSubject.safeTransferFrom(address(this), address(stub), _tokenId, "ffff");
}
}
contract StubTokenReceiver is
ERC721TokenReceiver
{
bytes4 constant FAKE_MAGIC_ON_ERC721_RECEIVED = 0xb5eb7a03;
/**
* @dev Receive token and map id to contract address (which is parsed from data).
*/
function onERC721Received(
address _operator,
address _from,
uint256 _tokenId,
bytes memory _data
)
public
returns(bytes4)
{
return FAKE_MAGIC_ON_ERC721_RECEIVED;
}
}
In the same way you can deploy stubs employing the live testing technique, you can also test contract deployments.
Limitations
The shown technique provides a much-needed method of testing deployed contracts but does not come without limitations. We need to acknowledge the limitations and offer solutions to some.
Smart contract events are not accessible from within contracts, therefore, there is no way of testing them without actually performing a transaction. This means that testing whether an event is emitted is not possible with estimateGas.
What if you are testing something like token transfer? To be able to transfer tokens, you need to be the owner of tokens or an approved party. This is not a problem for estimateGas since it does not need to be signed by a private key and therefore can be run from any address. This way, you can run the transaction as the address that actually owns the tokens even though you do not have its private key.
What about testing the safeTransferFrom method in an ERC-721 contract with no input from any of the token owners? In this case, you need a receiver similar to the one described in the Stubs and Artifacts section and with the ability to receive the token. Preferably, you would want to add multiple receivers with different types of passing/failing tests. You can solve this with minimum costs. All you have to do is create and deploy a few contracts that will serve as receiver tests. Once they are deployed, you can test any ERC-721 contracts safeTransferFrom, as long as you know an existing ID of that contract, and if the contract supports the enumerable extension, this method can produce immediate results.
In practice, it would work like this: you find the token owner by token ID, and create an estimateGas on a safeTransferFrom with the destination of the test contract you already deployed. To show you how simple it is, we’ve compiled a working example of this method that you can check here.
What about a case that involves multiple parties that need to provide approval/ownership? Well, this is a limitation that cannot always be solved completely gas-free. Still, here are two examples that show how you could do it:
- If you would like to test an ERC-721 asset that you don’t own or that has not yet been minted, but the asset is for sale on OpenSea, your test case could run as the WETH account (with 2 million Ether available). After you purchase the token on OpenSea, you can use that token as you please.
- Another way is the giver contract approach. This method is not completely free but makes the token completely safe. You can check the ERC-721 Validator to see this approach in action.
Conclusion
Live testing technique provides a complete alternative to testing with clean room tools such as Truffle. This new approach makes the entire state of the network available for your test cases. And since the way in which the test subjects were deployed is not important in your case, you can run a single test suite very easily against many deployed contracts.
Reading circle questions
- Which test situations would work better in the clean room approach and which in the live test approach?
- Does the introduction of live testing limit the usefulness of Truffle and similar tools?
- What are the benefits of using a live test approach for research on security vulnerabilities?
- What changes could be made to the (Ethereum) network client to make live testing simpler?
- Was the Ethereum Yellow Paper literally wrong, or is this a contrived interpretation?
The present article along with the method described and the code snippets is the work of William Entriken, lead author of ERC-721 standard, and Tadej Vengust, lead blockchain engineer at 0xcert.
Check how ERC-721 Validator worksCheck the complete code of live testing with estimateGas on GitHubCheck the validator example
Visit and star our GitHub repository.
<a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a>