Eternal Storage : Building Dapps For the Future

TLDR: Eternal Storage provides the stable and flexible base needed to build and maintain long-lived decentralized applications. Once deployed to a network, smart contract logic cannot be altered. However, developers are not perfect. Bugs are bound to loom in our logic no matter how carefully we plan our logic, test it, and have it externally audited. Iteration is inevitable in any complicated application, decentralized or not. Smart contracts will need to be updated and redeployed during the lifetime of a decentralized app (dapp). In this post we will learn about the Eternal Storage pattern and why it's an elegant answer to our need for mutability in an immutable world. This article makes the assumption that the reader has some basic understanding of writing smart contracts in and deploying and testing them locally (we like ). Some patterns are based off examples from , , and . Solidity Truffle EIP#930 fravoll RocketPool To illustrate the typical evolution of a dapp, let's consider a potential implementation of Decentralized DNS. The most basic functionality required in a first iteration might include: Claiming a domain Only allowed if domain is not already claimed Releasing a domain Only allowed by the owner Updating a domain's records Type (A, CNAME, MX, etc) Value TTL Listing all domains owned by a particular person How might the logic for this feature set look like? Initial Implementation solidity >= .4 Domain) domains; claim( memory name) { address domainOwner = domains[name].owner; (domainOwner == address( )) { domains[name].owner = msg.sender; ; } require(domainOwner == msg.sender, ); } updateRecords( memory name, DomainRecord[] memory records) { claim(name); domains[name].records; size = records.length; ( i = ; i < size; i++) { domains[name].records.push(records[i]); } } release( memory name) { require(domains[name].owner == msg.sender, ); domains[name].owner = address( ); } recordsForName( memory name) view returns (DomainRecord[] memory recordsForName_) { recordsForName_ = domains[name].records; } } pragma 0.6 0.7 pragma struct string // "A", "CNAME", "MX", etc... string uint struct string public function string public if 0 return "Domain has already been claimed" function string public delete uint for uint 0 function string public "Sender does not own this domain" 0 function string public ] [View DNS_v1.sol This contract allows users to perform basic actions we associate with domain names: claim them, modify their records, and release them. The data associated with these actions are stored directly in the contract itself. Its deployed address is . We register with our wallet at address and confirm we do own it: 0x00000001 test.com 0x99999999 truffle> let dns = await DNS.deployed() undefined truffle> dns.claim( ) {...} truffle> dns.domains( ) ' ' "test.com" "test.com" 0x99999999 Great start. Users start registering domain names in our dapp, but a few weeks later someone points out that doesn’t clear the records on a domain before giving up ownership. We decide that's a bug, so we have to update the contract by updating the method: release release function release(string memory name) public { require(domains[name].owner == msg.sender, "Sender does not own this domain"); domains[name].owner = address(0); } + delete domains[name].records; // now it clears records during release ] [View DNS_v2.sol We deploy this version of the contract with release clearing the records. Its deployed address is now . The address has changed, that's bad. Let's test our new release logic against anyway: 0x00000002 test.com truffle> let dns = await DNS.deployed() undefined truffle> dns.release( ) Uncaught: Error: Returned error: VM Exception while processing transaction: revert Sender does not own this domain -- Reason given: Sender does not own this domain. truffle> dns.domains( ) ' ' "test.com" "test.com" 0x00000000 What happened? Didn't we claim already? test.com Unfortunately data stored in a contract doesn't automatically migrate over on subsequent deploys of new versions of the contract. Two issues are now apparent: of the contract is missing all the data saved from user interaction with of the contract. Version 2 Version 1 Users need to know that of the contract is now deprecated and that the newest version now lives at . Version 1 0x00000002 Couldn’t we just migrate the data from to ourselves? Yes, but writes are expensive and paying the cost to copy data becomes prohibitively expensive as users create more data in our contracts. Not to mention we now must be careful not to introduce any errors as part of the data migration process. Version 1 Version 2 What if we just let users know a new version has been deployed at via our website or social media? It’s highly likely more than a few users will miss these announcements which will lead to very poor user experience and distrust in the dapp/developers. 0x00000002 Maybe we can get around this with some cleverness? Proxy/Delegate First, let's solve the problem where users of the old contract don't know about the newly deployed contract. We create a proxy contract, that points to another contract (our evolving dapp), which allows us to update the underlying contract at any point in the future. pragma solidity >= < ; contract Proxy { address delegate; address owner = msg.sender; { (msg.sender == owner, ); delegate = newDelegateAddress; } { assembly { _target := sload( ) calldatacopy( , , calldatasize) result := delegatecall(gas(), _target, , calldatasize, , ) returndatacopy( , , returndatasize) result {revert( , )} { ( , returndatasize)} } } } 0.5 .14 0.6 .0 private private // allows us to update where the proxy contract points to function upgradeDelegate (address newDelegateAddress) public require "Sender is not owner" // fallback function function () external payable let 0 0x0 0x0 let 0x0 0x0 0 0x0 0x0 switch case 0 0 0 default return 0 [View Proxy.sol] The contract leverages Solidity's unnamed fallback to forward calls it receives to a contract at any delegate address set by us via . It gets deployed and lives at . We set the delegate address to . We let users know they can use our dapp via . By updating the instance to point to the newest version of the contract every time we redeploy it, we now have a reliable static address that users can trust. Proxy function() upgradeDelegate() 0x11111111 0x00000002 0x11111111 Proxy DNS What about handling data across updates? One interesting property of is that the contract becomes the context for storage and all read/writes. Storage operations that normally would happen on the contract if we called it directly instead happen on the contract itself. This is a nice side effect because we no longer have to worry about migrating data each time we redeploy the contract. However, there’s an important caveat to this. Data structure declarations cannot be removed, or even reordered, because existing stored data structure will not match and unintended behavior will likely occur ( ). delegatecall Proxy DNS Proxy DNS example exploit What if we tried completely separating our data storage and business logic? Eternal Storage Eternal Storage contracts consist of mappings from type to other usable base types along with their respective getters and setters. This forms a flat key-value store. Mappings give us full flexibility in how we label and organize our data. bytes32 pragma solidity >= < ; contract EternalStorage { mapping( string) private stringStorage; mapping( bool) private boolStorage; ... modifier onlyNetworkContract() { (boolStorage[keccak256(abi.encodePacked( ))] == ) { (boolStorage[keccak256(abi.encodePacked( , , msg.sender))] == , ); } _; } () public { boolStorage[keccak256(abi.encodePacked( , , msg.sender))] = ; } { stringStorage[_key]; } { boolStorage[_key]; } ... { stringStorage[_key] = _value; } { boolStorage[_key] = _value; } ... { stringStorage[_key]; } { boolStorage[_key]; } ... } 0.5 .14 0.6 .0 => bytes32 => bytes32 // this will help us lock down access to only authorized addresses. ignore it for now if "contract.storage.initialized" true require "access.role" "contract" true "Sender is not a contract in the network" constructor "access.role" "owner" true // Get ( ) ( ) function getString bytes32 _key external view returns string memory return ( ) ( ) function getBool bytes32 _key external view returns bool return // Set ( ) function setString bytes32 _key, string memory _value public onlyNetworkContract ( ) function setBool bytes32 _key, bool _value external onlyNetworkContract // Delete ( ) function deleteString bytes32 _key external onlyNetworkContract delete ( ) function deleteBool bytes32 _key external onlyNetworkContract delete // And similar get/set/delete declarations and functions for the types: // "address", "bytes", "int", "bytes32" [View EternalStorage.sol] To keep things sane and organized, we will use dot notation to address our pseudo objects and their properties. Interacting with our Eternal Storage contract may look something like this: eternalStorage.set , msg.sender)) eternalStorage.set , “Hello World”)) Address( ( . ( ) keccak256 abi encodePacked "foo.bar.owner" String( ( . ( ) keccak256 abi encodePacked "foo.bar.title" Keccak is the algorithm that became SHA-3, but NIST changed an implementation detail in the standardization process. Chances of collision are extremely low, giving us 64^36 addressable keys for each mapping. The function expects array and returns . The function expects any type of data and returns it encoded in the smallest byte representation possible via a array. Hashing our keys like this allows us to stay at 32 bytes regardless of key length, helping save on gas costs. keccak256() bytes bytes32 abi.encodePacked() bytes Setters are locked down to allow access only from registered contracts in the network once admins mark the boolean map with key to be true. “contract.storage.intialized” Unfortunately adding Eternal Storage increases the complexity in our contracts. See how our original contract logic changes: DNS update public { claim(name); delete domains .records; uint size = records.length; for (uint i = ; i < size; i++) { domains .records.push(records ); } } EternalStorage eternalStorage; update public { uint length = records.length; eternalStorage.set ), length); for (uint i = ; i < length; i++) { eternalStorage.set ), records .recordType); eternalStorage.set ), records .value); eternalStorage.set ), records .ttl); } // Old function Records( , DomainRecord[] ) string memory name memory records [ ] name 0 [ ] name [ ] i // New ... function Records( , DomainRecord[] ) string memory name memory records Uint( ( . ( , , , ) keccak256 abi encodePacked "domains" name "records" "length" 0 String( ( . ( , , , , ) keccak256 abi encodePacked "domains" name "records" i "type" [ ] i String( ( . ( , , , , ) keccak256 abi encodePacked "domains" name "records" i "value" [ ] i Uint( ( . ( , , , , ) keccak256 abi encodePacked "domains" name "records" i "ttl" [ ] i Luckily, we can utilize helper functions to keep our business logic free of the complexity introduced above by using Eternal Storage. Storage Operations One common data type we will have to reintroduce is lists/arrays. highlights the many steps involved in implementing a string array on top of Eternal Storage and why it is important to separate this into a helper. StringListStorage pragma solidity >= . < . ; contract StringListStorage { EternalStorage es; constructor(address _es) public { es = ; } get external view returns (uint) { return es.get )); } get external view returns ( memory) { return es.get )); } push public { uint length = es.get )); es.set ), _value); es.set ), length + ); } remove external { require(_index < es.get )), ); uint length = es.get )); (_index < length - ) { memory lastItem = es.get )); es.set ), lastItem); } es.set ), length - ); } } 0.5 14 0.6 0 EternalStorage( ) _es // Get function ListLength( ) bytes32 _key Uint( ( . ( , ) keccak256 abi encodePacked _key "length" function ListItem( , ) bytes32 _key uint _index string String( ( . ( , , ) keccak256 abi encodePacked _key "item" _index ... // Set function ListItem( , ) bytes32 _key string memory _value Uint( ( . ( , ) keccak256 abi encodePacked _key "length" String( ( . ( , , ) keccak256 abi encodePacked _key "item" length Uint( ( . ( , ) keccak256 abi encodePacked _key "length" 1 function UnorderedListItem( , ) bytes32 _key uint _index Uint( ( . ( , ) keccak256 abi encodePacked _key "length" "List index out of bounds" Uint( ( . ( , ) keccak256 abi encodePacked _key "length" if 1 string String( ( . ( , , - 1) keccak256 abi encodePacked _key "item" length String( ( . ( , , ) keccak256 abi encodePacked _key "item" _index Uint( ( . ( , ) keccak256 abi encodePacked _key "length" 1 ... // See complete contract linked below [View StringListStorage_v1.sol] Now that we have a nice wrapper for string arrays, we can create a helper around our previous idea of a struct. shows usage of the helper and how we are able to maintain friendly method interfaces through structs despite the underlying storage mechanisms having no notion of them. Domain DomainModel StringListStorage pragma solidity >= . < . ; pragma experimental ABIEncoderV2; contract DomainModel { DomainRecord { recordType; value; uint ttl; } EternalStorage es; StringListStorage sls; constructor(address _es) public { es = ; sls = ; } get public view returns (address owner_) { address domainOwner = es.get )); owner_ = domainOwner; } set public { (owner address( )) { return; } es.set ), owner); sls.push ), name); } } 0.5 14 0.6 0 struct string // "A", "CNAME", "MX", etc... string EternalStorage( ) _es new StringListStorage( ) _es function Owner( ) string memory name Address( ( . ( , , ) keccak256 abi encodePacked "domains" name "owner" function Owner( , ) string memory name address owner if == 0 Address( ( . ( , , ) keccak256 abi encodePacked "domains" name "owner" ListItem( ( . ( , , ) keccak256 abi encodePacked "owners" owner "domains" ... // See complete contract linked below [View DomainModel_v1.sol] With , we now have an idea of how we can organize larger data structures around Eternal Storage. Finally, let's use inside our contract. DomainModel DomainModel DNS pragma solidity >= . < . ; pragma experimental ABIEncoderV2; contract DNS { DomainsModel _domains; constructor(address _es) public { _domains = ; } claim( memory name) public { address domainOwner = get ; (domainOwner address( )) { set ; return; } require(domainOwner msg.sender, ); } update public { claim(name); set ; } release( memory name) public { address domainOwner = get ; require(domainOwner msg.sender, ); set ; delete ; } domains( memory name) public view returns (address owner_) { address domainOwner = get ; owner_ = domainOwner; } records public view returns (DomainsModel.DomainRecord memory recordsForName_) { DomainsModel.DomainRecord memory records = get ; recordsForName_ = records; } domain public view returns ( memory domainNamesForOwner_) { memory domainNames = get ; domainNamesForOwner_ = domainNames; } domain public view returns ( memory domainNamesForSender_) { memory domainNames = get ; domainNamesForSender_ = domainNames; } } 0.5 14 0.6 0 new DomainsModel( ) _es function string . _domains Owner( ) name if == 0 . _domains Owner( , . ) name msg sender == "Domain has already been claimed" function Records( , DomainsModel.DomainRecord[] ) string memory name memory records . _domains Records( , ) name records function string . _domains Owner( ) name == "Sender does not own this domain" . _domains Records( , []) name . _domains Owner( ) name function string . _domains Owner( ) name function ForName( ) string memory name [] [] . _domains Records( ) name function NamesForOwner( ) address owner string [] string [] . _domains DomainNamesForOwner( ) owner function NamesForSender() string [] string [] . _domains DomainNamesForOwner( . ) msg sender [View DNS_v3.sol] This isn't so bad. In fact it reads much more nicely than before. Now that we are happy with our storage, let's wrap it all up and secure our new network of contracts. Authorization & Upgradeability We cannot talk about authorization and upgradeability separately as each implementation depends on the other. Authorization Our contracts must be explicit about who can and cannot access its methods. Eternal Storage must only allow storage access from certain contracts/senders for write operations. Upgradeability Without having unrestricted access to all data in the Eternal Storage contract, Admins must still retain the ability to add, upgrade, and authorize new contracts to the network. Our contracts must be able to retrieve the most up-to-date address for any contract it wishes to make a call to. Once we set , only contracts registered at the boolean map with key will be able to modify storage. Unfortunately, admins will locked out and now unable to add/upgrade/authorize contracts. Additionally with our current logic, any sender can still bypass domain ownership checks in the contract by calling the methods in and directly. "contract.storage.intialized" “access.role”, “contract”, CONTRACT_ADDRESS DNS DomainModel StringListStorage Upgradeability, and any other actions admins may need to perform can be implemented as... contracts! Admins should never need fully unrestricted access to Eternal Storage. Limiting the power admins hold also reduces risk and improves overall trust in the system. We need the following for authorization/upgradeability: A way to lookup if a sender is a contract in the network or is an admin so we can limit access by role. (boolean) “access.role”, “contract”, _SenderAddress (boolean) “access.role”, “admin”, _SenderAddress A way to lookup the address for the most recent version of any contract in the network so we can limit access to certain methods to a specific contract(s) and so we can make sure we are always calling the most up-to-date version of a contract. (address) “contract”, _contractName, “address” Let’s jump ahead a little bit and add a contract that implements these checks. Base pragma solidity >= . . < . . contract { EternalStorage internal es modifier onlyNetworkContracts() { require( hasRole( ) == true, ) _ } modifier onlyNetworkContract(string memory _contractName) { contractAddress = getContractAddress(_contractName) require(contractAddress == msg.sender, ) _ } modifier onlyRole(string memory _role) { require( hasRole(_role) == true, ) _ } constructor( _es) public { es = EternalStorage(_es) } function getContractAddress(string memory _contractName) public view returns( { contractAddress = es.getAddress( abi.encodePacked(keccak256(abi.encodePacked( , _contractName, ))) ) require( != ) return contractAddress } function hasRole(string memory _role) public view returns( { return es.getBool( abi.encodePacked(keccak256(abi.encodePacked( , _role, msg.sender))) ) } } 0 5 14 0 6 0 ; Base ; "contract" "Sender is not a contract in the network" ; ; address ; "Incorrect or outdated contract access used" ; ; "Sender doesn't have matching role" ; ; address ; address) address "contracts" "address" ; address(contractAddress) address(0x0), "Contract not found" ; ; bool) "access.role" ; [View Base.sol] The contract gives us: Base constructor that initializes a reference to Eternal Storage. modifiers that allow us to guard methods to any or only specific contracts on the network or admins. a method to get the address for the most recent version of any contract type deployed on the network. pragma solidity >= < ; contract ContractAdmin is Base { (address _es) Base(_es) public {} { (_contractAddress != address( ), ); address existingContractName = es.getAddress(keccak256(abi.encodePacked( , _name, ))); (existingContractName == address( ), ); bool existingContractAddress = es.getBool(keccak256(abi.encodePacked( , , _contractAddress))); (!existingContractAddress, ); es.setAddress(keccak256(abi.encodePacked( , _name, )), _contractAddress); es.setString(keccak256(abi.encodePacked( , _name, )), _contractAbi); es.setBool(keccak256(abi.encodePacked( , , _contractAddress)), ); } ... } 0.5 .14 0.6 .0 constructor ( ) ( ) function addContract string memory _name, address _contractAddress, string memory _contractAbi public onlyRole "owner" require 0x0 "Invalid contract address" "contracts" "address" require 0x0 "Contract name is already in use" "access.role" "contract" require "Contract address is already in use" "contracts" "address" "contracts" "abi" "access.role" "contract" true // See complete contract linked below [View ContractAdmin.sol] The contract gives admins (via ) the ability to add and upgrade contracts with required authorization-related entries being updated automatically. ContractAdmin onlyRole("owner") Now how will the rest of our contracts look after they inherit the contract and lock down access to their methods? Base pragma solidity >= . . < . . ; contract { constructor(address _es) (_es) public {} // remain the same function getListLength(bytes32 _key) external view returns ( ) function getListItem(bytes32 _key, _index) external view returns ( memory) // have the onlyNetworkContracts modifier applied to them function pushListItem(bytes32 _key, memory _value) public onlyNetworkContracts function removeUnorderedListItem(bytes32 _key, _index) external onlyNetworkContracts ... // complete contract linked below } 0 5 14 0 6 0 StringListStorage is Base Base Getters uint {...} uint string {...} Setters string {...} uint {...} See [View StringListStorage_v2.sol] The contract locks down storage-modifying methods to only contracts in the network. StringListStorage pragma solidity >= . < . ; pragma experimental ABIEncoderV2; contract DomainModel is Base { DomainRecord { } constructor(address _es) public {} set only public { StringListStorage sls = ); sls.push ), name); } } 0.5 14 0.6 0 struct ... Base( ) _es ... // this is method and the other setters are now only accessible by the most recently deployed DNS contract function Owner( , ) string memory name address owner NetworkContract( ) "DNS" ... // notice how we are using getContractAddress to discover the address of the StringListStorage contract StringListStorage( ( ) getContractAddress "StringListStorage" ... ListItem( ( . ( , , ) keccak256 abi encodePacked "owners" owner "domains" ... // See complete contract linked below [View DomainModel_v2.sol] The contract locks down storage modifying methods to . Additionally, it now fetches the most current address of from Eternal Storage rather than having the address passed to its constructor. DomainModel only the currently deployed DNS contract StringListStorage pragma solidity >= . < . ; pragma experimental ABIEncoderV2; contract DNS is Base { constructor(address _es) public {} claim( memory name) public { DomainModel dm = ); address domainOwner = dm.get ; (domainOwner address( )) { dm.set ; return; } require(domainOwner msg.sender, ); } update public { claim(name); DomainModel dm = ); dm.set ; } release( memory name) public { DomainModel dm = ); address domainOwner = dm.get ; require(domainOwner msg.sender, ); dm.delete ; } domains( memory name) public view returns (address owner_) { DomainModel dm = ); address domainOwner = dm.get ; owner_ = domainOwner; } records public view returns (DomainModel.DomainRecord memory recordsForName_) { DomainModel dm = ); DomainModel.DomainRecord memory records = dm.get ; recordsForName_ = records; } domain public view returns ( memory domainNamesForOwner_) { DomainModel dm = ); memory domainNames = dm.get ; domainNamesForOwner_ = domainNames; } domain public view returns ( memory domainNamesForSender_) { DomainModel dm = ); memory domainNames = dm.get ; domainNamesForSender_ = domainNames; } } 0.5 14 0.6 0 Base( ) _es function string // notice how we are using getContractAddress to discover the address of the DomainModel contract DomainModel( ( ) getContractAddress "DomainModel" Owner( ) name if == 0 Owner( , . ) name msg sender == "Domain has already been claimed" function Records( , DomainModel.DomainRecord[] ) string memory name memory records // another example of using getContractAddress DomainModel( ( ) getContractAddress "DomainModel" Records( , ) name records function string DomainModel( ( ) getContractAddress "DomainModel" Owner( ) name == "Sender does not own this domain" Owner( ) name function string DomainModel( ( ) getContractAddress "DomainModel" Owner( ) name function ForName( ) string memory name [] DomainModel( ( ) getContractAddress "DomainModel" [] Records( ) name function NamesForOwner( ) address owner string [] DomainModel( ( ) getContractAddress "DomainModel" string [] DomainNamesForOwner( ) owner function NamesForSender() string [] DomainModel( ( ) getContractAddress "DomainModel" string [] DomainNamesForOwner( . ) msg sender [View DNS_v4.sol] The DNS contract does not lock down access to any of its methods since it’s directly user facing. It now fetches the most current address of the contract from Eternal Storage rather than deploying an instance and storing that address in its constructor. DomainModel We now have a dapp with the following properties: contracts can be upgraded over time. existing contracts will be aware of newly deployed contracts. data remains intact regardless of any contract upgrades. old version of contracts are locked out of the network. contract methods can be restricted by role/contract/address. You can find all of the contracts and their revisions . here Conclusion Finally, we have a strong base for building and maintaining long-lived decentralized applications. This system isolates the trust to admins who can't directly access the eternal storage, and can only modify it by contract upgrades. Furthermore it allows for the implementation of clear and appropriate governance policies to further control the actions of the admins. Hopefully this exercise has demonstrated why it's a good idea to use Eternal Storage in your future dapp.