Testing
Testing an ape project is important and easy.
Pytest
Before learning how testing works in Ape, you should have an understanding of the pytest framework and its concepts such as fixtures, mark-decorators, and pytest plugins such as x-dist, pytest-mock, and pytest-cov.
Once you have learned about pytest, Ape testing becomes intuitive because it is built on top of pytest.
In fact, ape-test is itself a pytest plugin!
You write your smart-contracts much like you write regular Python tests.
Test Structure
Tests must be located in a project’s tests/ directory. Each test file must start with test_ and have the .py extension, such as test_my_contract.py.
Each test method within the file must also start with test_.
The following is an example test:
def test_add():
assert 1 + 1 == 2
NOTE: pytest assumes the actual value is on the left and the expected value is on the right.
Test Pattern
Tests are generally divisible into three parts:
Set-up
Invocation
Assertion
An example of the setup-phase would be creating a pytest.fixture that deploys our smart contract.
(To learn more about pytest fixtures in Ape, see the Fixtures section below!)
For now, what you need to know is that it’s a piece of code that executes before the test runs, and it is decorated with a @pytest.fixture.
The second phase is Invocation, which encompasses invoking the function we are testing.
The last phase, Assertion, requires enacting on the expectation about how the code should behave.
Let’s assume there is an authorized_method() that requires the owner of the contract to make the transaction.
If the sender of the transaction is not the owner, the transaction will fail to complete and will revert.
We use assert statements in Ape (and pytest) to check that our expectations are correct.
A test passes if all the assert statements are True and it fails if any are False.
This is an example of how that test may look:
import ape
import pytest
# SETUP PHASE
# NOTE: More on fixtures is discussed in later sections of this guide!
@pytest.fixture
def owner(accounts):
return accounts[0]
@pytest.fixture
def my_contract(owner, project):
return owner.deploy(project.MyContract)
def test_authorization(my_contract, owner, not_owner):
# INVOCATION PHASE
my_contract.set_owner(sender=owner)
assert owner == my_contract.owner()
# ASSERTION PHASE
with ape.reverts("!authorized"):
my_contract.authorized_method(sender=not_owner)
Note
Ape has built-in test and fixture isolation for all pytest scopes.
To disable isolation add the --disable-isolation flag when running ape test
Fixtures
Now that we have discussed the full flow of a test, let’s dive deeper into the specific parts, starting with pytest.fixtures.
You can define and use pytest fixtures in your Ape tests.
Learn more about fixtures from this guide.
The syntax and functionalities of fixtures work exactly the same in Ape as it does with pytest.
The ape-test plugin comes with fixtures you will likely want to use.
The following guide explains each fixture that comes with Ape.
accounts fixture
You have access to test accounts.
These accounts are automatically funded, and you can use them to transact in your tests.
Access each test account by index from the accounts fixture:
def test_my_method(accounts):
owner = accounts[0]
receiver = accounts[1]
For code readability and sustainability, create your own fixtures using the accounts fixture:
import pytest
@pytest.fixture
def owner(accounts):
return accounts[0]
@pytest.fixture
def receiver(accounts):
return accounts[1]
def test_my_method(owner, receiver):
...
You can configure your accounts by changing the mnemonic or number_of_accounts settings in the test section of your ape-config.yaml file:
test:
mnemonic: test test test test test test test test test test test junk
number_of_accounts: 5
If you are running tests against anvil, your generated test accounts may not correspond to the anvil’s default generated accounts despite using the same mnemonic. In such a case, you are able to specify a custom derivation path in ape-config.yaml:
test:
mnemonic: test test test test test test test test test test test junk
number_of_accounts: 5
hd_path: "m/44'/60'/0'/0/{}"
If you are using a fork-provider, such as Hardhat, you can use impersonated accounts by accessing random addresses off the fixture:
@pytest.fixture
def vitalik(accounts):
return accounts["0xab5801a7d398351b8be11c439e05c5b3259aec9b"]
Using a fork-provider such as Hardhat, when using a contract instance as the sender in a transaction, it will be automatically impersonated:
def test_my_method(project, accounts):
contract = project.MyContract.deploy(sender=accounts[0])
other_contract = project.OtherContract.deploy(sender=accounts[0])
contract.my_method(sender=other_contract)
It has the same interface as the TestAccountManager, (same as doing accounts.test_accounts in a script or the console).
chain fixture
Use the chain fixture to access the connected provider or adjust blockchain settings.
For example, increase the pending timestamp:
def test_in_future(chain):
chain.pending_timestamp += 86000
assert "Something"
chain.pending_timestamp += 86000
assert "Something else"
It has the same interface as the ChainManager.
networks fixture
Use the networks fixture to change the active provider in tests.
def test_multi_chain(networks):
assert "Something" # Make assertion in root network
# NOTE: Assume have ecosystem named "foo" with network "local" and provider "bar"
with networks.foo.local.use_provider("bar"):
assert "Something else"
It has the same interface as the NetworkManager.
project fixture
You also have access to the project you are testing. You will need this to deploy your contracts in your tests.
import pytest
@pytest.fixture
def owner(accounts):
return accounts[0]
@pytest.fixture
def my_contract(project, owner):
# ^ use the 'project' fixture from the 'ape-test' plugin
return owner.deploy(project.MyContract)
It has the same interface as the ProjectManager.
Contract fixture
Use the Contract fixture to create contract instances:
@pytest.fixture
def my_contract(Contract):
return Contract(<address>)
It has the same interface as the ChainManager.
Ape testing commands
ape test
To run a particular test:
ape test test_my_contract
Use ape test -I to open the interactive mode at the point of exception. This allows the user to inspect the point of failure in your tests.
ape test test_my_contract -I -s
Test Providers
Out-of-the-box, your tests run using the eth-tester provider, which comes bundled with ape. If you have geth installed, you can use the ape-geth plugin that also comes with ape.
ape test --network ethereum:local:geth
Each testing plugin should work the same way. You will have access to the same test accounts.
Another option for testing providers is the ape-hardhat plugin, which does not come with ape but can be installed by including it in the plugins list in your ape-config.yaml file or manually installing it using the command:
ape plugins install hardhat
Advanced Testing Tips
If you want to use sample projects, follow this link to Ape Academy.
project # The root project directory
└── tests/ # Project tests folder, ran using the 'ape test' command to run all tests within the folder.
└── conftest.py # A file to define global variable for testing
└── test_accounts.py # A test file, if you want to ONLY run one test file you can use 'ape test test_accounts.py' command
└── test_mint.py # A test file
Here is an example of a test function from a sample NFT project
def test_account_balance(project, owner, receiver, nft):
quantity = 1
nft.mint(receiver, quantity, ["0"], value=nft.PRICE() * quantity, sender=owner)
actual = project.balanceOf(receiver)
expect = quantity
assert actual == expect
Testing Transaction Failures
Similar to pytest.raises(), you can use ape.reverts() to assert that contract transactions fail and revert.
From our earlier example we can see this in action:
def test_authorization(my_contract, owner, not_owner):
my_contract.set_owner(sender=owner)
assert owner == my_contract.owner()
with ape.reverts("!authorized"):
my_contract.authorized_method(sender=not_owner)
reverts() takes two optional parameters:
expected_message
This is the expected revert reason given when the transaction fails.
If the message in the ContractLogicError raised by the transaction failure is empty or does not match the expected_message, then ape.reverts() will raise an AssertionError.
You may also supply an re.Pattern object to assert on a message pattern, rather than on an exact match.
# Matches explicitly "foo" or "bar"
with ape.reverts(re.compile(r"^(foo|bar)$")):
...
dev_message
This is the expected dev message corresponding to the line in the contract’s source code where the error occurred. These can be helpful in optimizing for gas usage and keeping revert reason strings shorter.
Dev messages take the form of a comment in Vyper, and should be placed on the line that may cause a transaction revert:
assert x != 0 # dev: invalid value
Take for example:
# @version 0.3.7
@external
def check_value(_value: uint256) -> bool:
assert _value != 0 # dev: invalid value
return True
We can explicitly cause a transaction revert and check the failed line by supplying an expected dev_message:
def test_authorization(my_contract, owner):
with ape.reverts(dev_message="dev: invalid value"):
my_contract.check_value(sender=owner)
When the transaction reverts and ContractLogicError is raised, ape.reverts() will check the source contract to see if the failed line contains a message.
There are a few scenarios where AssertionError will be raised when using dev_message:
If the line in the source contract has a different dev message or no dev message
If the contract source cannot be obtained
If the transaction trace cannot be obtained
Because dev_message relies on transaction tracing to function, you must use a provider like ape-hardhat when testing with dev_message.
You may also supply an re.Pattern object to assert on a dev message pattern, rather than on an exact match.
# Matches explictly "dev: foo" or "dev: bar"
with ape.reverts(dev_message=re.compile(r"^dev: (foo|bar)$")):
...
Caveats
Language Support
As of ape version 0.5.6, dev_messages assertions are available for contracts compiled with ape-vyper, but not for those compiled with ape-solidity or ape-cairo.
Inlining
Due to function inlining, the position of the # dev: ... message may sometimes be one line higher than expected:
@external
def foo(_x: decimal) -> decimal: # dev: correct location
return sqrt(_x) # dev: incorrect location
This typically only applies when trying to add dev messages to statements containing built-in function calls.
Non-reentrant Functions
Similarly, if you require dev assertions for non-reentrant functions you must be sure to leave the comment on the function that should not have reentry:
@internal
@nonreentrant('lock')
def _foo_internal(): # dev: correct location
pass
@external
@nonreentrant('lock')
def foo():
self._foo_internal() # dev: incorrect location
Custom Errors
As of Solidity 0.8.4, custom errors have been introduced to the ABI. To make assertions on custom errors, you can use the types defined on your contracts.
For example, if I have a contract called MyContract.sol:
// SPDX-License-Identifier: GPL-3.0
pragma solidity ^0.8.4;
error Unauthorized(address unauth_address);
contract MyContract {
address payable owner = payable(msg.sender);
function withdraw() public {
if (msg.sender != owner)
revert Unauthorized(msg.sender);
owner.transfer(address(this).balance);
}
}
I can ensure unauthorized withdraws are disallowed by writing the following test:
import ape
import pytest
@pytest.fixture
def owner(accounts):
return accounts[0]
@pytest.fixture
def hacker(accounts):
return accounts[1]
@pytest.fixture
def contract(owner, project):
return owner.deploy(project.MyContract)
def test_unauthorized_withdraw(contract, hacker):
with ape.reverts(contract.Unauthorized, unauth_address=hacker.address):
contract.withdraw(sender=hacker)
You can also use custom error types from the contract container (from ape.project or the project fixture):
import ape
def test_unauthorized(contract, hacker, project):
with ape.reverts(project.MyContract.Unauthorized, unauth_address=hacker.address):
contract.withdraw(sender=hacker)
You may need to use the container approach for asserting on custom errors that occur during failing deploy transactions because you won’t have access to the contract instance yet.
Here is an example of what that may look like:
import ape
def test_error_on_deploy(account, project):
with ape.reverts(project.Token.MyCustomError):
ape.project.HasError.deploy(sender=account)
Alternatively, you can attempt to use the address from the revert error to find the error type. NOTE: The address will only exist for transactions that were published (e.g. not for failures during estimating gas), and this may only work on certain providers.
import ape
def test_error_on_deploy(account):
# NOTE: We are using `as rev` here to capture the revert info
# so we can attempt to lookup the contract later.
with ape.reverts() as rev:
ape.project.HasError.deploy(sender=account)
assert rev.value.address is not None, "Receipt never found, contract never cached"
# Grab the cached instance using the error's address
# and assert the custom error this way.
contract = ape.Contract(rev.value.address)
assert isinstance(rev.value, contract.MyError)
Multi-chain Testing
The Ape framework supports connecting to alternative networks / providers in tests.
To run an entire test using a specific network / provider combination, use the use_network pytest marker:
import pytest
@pytest.mark.use_network("fantom:local:test")
def test_my_fantom_test(chain):
assert chain.provider.network.ecosystem.name == "fantom"
@pytest.mark.use_network("ethereum:local:test")
def test_my_ethereum_test(chain):
assert chain.provider.network.ecosystem.name == "ethereum"
To switch networks mid-test, use the networks context-manager:
# Switch to Fantom mid test
def test_my_multichain_test(networks):
# The test starts in 1 ecosystem but switches to another
assert networks.provider.network.ecosystem.name == "ethereum"
with networks.fantom.local.use_provider("test") as provider:
assert provider.network.ecosystem.name == "fantom"
# You can also use the context manager like this:
with networks.parse_network_choice("fantom:local:test") as provider:
assert provider.network.ecosystem.name == "fantom"
You can also set the network context in a pytest fixture. This is useful if certain fixtures must run in certain networks.
import pytest
@pytest.fixture
def stark_contract(networks, project):
with networks.parse_network_choice("starknet:local"):
yield project.MyStarknetContract.deploy()
def test_starknet_thing(stark_contract, stark_account):
# Uses the starknet connection via the stark_contract fixture
receipt = stark_contract.my_method(sender=stark_account)
assert not receipt.failed
When you exit a provider’s context, Ape does not disconnect the provider. When you re-enter that provider’s context, Ape uses the previously-connected provider. At the end of the tests, Ape disconnects all the providers. Thus, you can enter and exit a provider’s context as much as you need in tests.
Gas Reporting
To include a gas report at the end of your tests, you can use the --gas flag.
NOTE: This feature requires using a provider with tracing support, such as ape-hardhat.
ape test --network ethereum:local:hardhat --gas
At the end of test suite, you will see tables such as:
FundMe Gas
Method Times called Min. Max. Mean Median
────────────────────────────────────────────────────────────────
fund 8 57198 91398 82848 91398
withdraw 2 28307 38679 33493 33493
changeOnStatus 2 23827 45739 34783 34783
getSecret 1 24564 24564 24564 24564
Transferring ETH Gas
Method Times called Min. Max. Mean Median
───────────────────────────────────────────────────────
to:test0 2 2400 9100 5750 5750
TestContract Gas
Method Times called Min. Max. Mean Median
───────────────────────────────────────────────────────────
setNumber 1 51021 51021 51021 51021
The following demonstrates how to use the ape-config.yaml file to exclude contracts and / or methods from the gas report:
test:
gas:
exclude:
- method_name: DEBUG_* # Exclude all methods starting with `DEBUG_`.
- contract_name: MockToken # Exclude all methods in contract named `MockToken`.
- contract_name: PoolContract # Exclude methods starting with `reset_` in `PoolContract`.
method_name: reset_*
Similarly, you can exclude sources via the CLI option --gas-exclude.
The value --gas-exclude takes is a comma-separated list of colon-separated values representing the structure similar as above, except you must explicitly use * where meaning “all”.
For example to exclude all methods starting with DEBUG_, you would do:
ape test --gas --gas-exclude "*:DEBUG_*".
To exclude all methods in the MockToken contract, do:
ape test --gas --gas-exclude MockToken
And finally, to exclude all methods starting with reset_ in PoolContract, do:
ape test --gas --gas-exclude "PoolContract:reset_*"
Iterative Testing
Ape has a set of flags that controls running your test suite locally in a “watch” mode, which means watching for updates to files in your project and re-triggering the test suite.
To enable this mode, run ape test --watch to set up this mode using the default settings.
While in this mode, any time a .py file (i.e. your tests) or smart contract source file
(i.e. any files that get compiled using your installed compiler plugins) is added, removed,
or changed, then the ape test task will be re-triggered, based on a polling interval.
To exit this mode, press Ctrl+D (on Linux or macOS) to stop the execution and undo it.
Contract Coverage
To get contract coverage, use the --coverage flag when running ape test:
ape test --coverage
NOTE: Some types of coverage require using a provider that supports transaction tracing, such as ape-hardhat or ape-foundry.
Afterwards, you should see a coverage report looking something like:
============================================= Coverage Profile =============================================
Contract Coverage
Name Stmts Miss Cover Funcs
─────────────────────────────────────────────
Contract.vy 7 1 85.71% 80.0%
To generate other coverage reports such as XML or HTML, configure it like so:
test:
coverage:
reports:
terminal: False # Disable the terminal table (True by default)
xml: True # Enable XML report (.build/coverage.xml)
html: True # Enable HTML report (.build/htmlcov)
To see a much more verbose coverage report, set the terminal field to a dictionary that includes "verbose": true:
test:
coverage:
reports:
terminal:
verbose: true # Show verbose coverage information in the terminal.
Then, you will see table outputs like this:
===================================== Coverage Profile ========================================
MyContract Coverage
Func Stmts Miss Cover
─────────────────────────────────────────────────────
__builtin__ 2 0 100.0%
_immutable_number 0 0 100.0%
_number 0 0 100.0%
foo_method() 1 0 100.0%
foo_method(uint256) 1 0 100.0%
foo_method(uint256,uint256) 3 0 100.0%
view_method 1 0 100.0%
line=0.0%, func=0.0%
This is useful when trying to find the missing areas to cover.
The HTML report also supports verbose: true and it will show similar tables.
NOTE: You may notice methods with zero statements. One example of a method with zero statements may be from an auto-generated getter method for a public variable; certain versions of Vyper do not contain source mappings for these methods. However, Ape will still check to see if this method has been called in your tests. To get 100% coverage, you must call these methods in your tests.
NOTE: Notice some methods use the full selector while others don’t. Methods that use the selector mean that their short name is shared with other methods. This happens in Vyper from auto-generated kwarg-based methods. Thus, the full selector is used to distinguish the methods in the coverage (and gas) reports.
Much like gas reporting, you can also exclude contracts and methods from tracking coverage using your ape-config.yaml file.
The following demonstrates how to do this:
test:
coverage:
exclude:
- method_name: DEBUG_* # Exclude all methods starting with `DEBUG_`.
- contract_name: MockToken # Exclude all methods in contract named `MockToken`.
- contract_name: PoolContract # Exclude methods starting with `reset_` in `PoolContract`.
method_name: reset_*