jon atack - How to Compile Bitcoin Core and Run the Tests

How to compile Bitcoin Core and run the unit and functional tests

This is a summary of the documentation in the Bitcoin Core repository. Don't hesitate to read it for more information.

All steps are to be run from your terminal emulator, i.e. the command line.

Important change starting from September 2024: The master branch of Bitcoin Core has migrated its build system from Autotools to CMake. Bitcoin Core v28, expected to be released in October 2024, will be the last release that uses Autotools as covered by this article. The upcoming v29 release in early 2025 will use CMake for compiling and running tests, and a new version of this article will cover that.

Ensure the dependencies are installed. The following lists include some optional dependencies. Don't hesitate to consult the links below, adapt the dependencies to your needs, and let me know if any are out of date here. Note that ccache isn't strictly required, but you'll probably want to install it (see below). You'll also need a compiler like GCC and/or Clang installed. Refer to doc/dependencies.md for more information.
- Linux (see doc/build-unix.md for details): sudo apt-get install automake autotools-dev bsdmainutils build-essential ccache clang gcc git libboost-dev libboost-filesystem-dev libboost-system-dev libboost-test-dev libevent-dev libminiupnpc-dev libnatpmp-dev libsqlite3-dev libtool libzmq3-dev pkg-config python3 qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 systemtap-sdt-dev
- macOS (with command line tools and Homebrew already installed, see doc/build-osx.md for details): brew install automake boost ccache git libevent libnatpmp libtool llvm miniupnpc pkg-config python qrencode qt@5 sqlite zeromq
Download the Bitcoin source files by git cloning the repository (after forking it, if you plan to push any changes later on), then cd (change the current directory) to it.
- git clone https://github.com/bitcoin/bitcoin.git
- cd bitcoin
Berkeley DB (BDB) is only needed if you want legacy wallet compatibility, i.e. before the current descriptor wallets that use SQLite3. If not, you can skip this step.
- To install BDB 4.8, a backward-compatible legacy version, run brew install berkeley-db@4 for macOS, and for Linux see the Bitcoin Core build-*.md documentation for your operating system (the legacy install_db4.sh script was removed in Bitcoin Core v25 with PR 26834).
- If you have another version of BDB already installed that you wish to use, simply add --with-incompatible-bdb to your ./configure flags below instead. This option is recommended unless you specifically need BDB 4.8.
Compile from a tagged release branch, unless you wish to test a specific branch or PR.
- git tag -n | sort -V to see tags and descriptions ordered by most recent last
- git checkout <TAG> to use a tagged release, for example: git checkout v0.21.0
- If you want to pull down a PR or specific branch from the remote repository to build and test locally, here is how.
Compile Bitcoin from source.
- ./autogen.sh
- ./configure
  or, for legacy wallet support with a recent BDB
  
  ./configure --with-incompatible-bdb
  
  or, for legacy wallet support with BDB 4.8
```
  export BDB_PREFIX='<PATH-TO>/db4'
  ./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include"
```
- make, or if you have multiple CPU cores, which is the usual case nowadays, you can tell make to use all of them and reduce compile time significantly with
  make -j "$(($(nproc) + 1))" on Linux, or
  
  make -j "$(($(sysctl -n hw.physicalcpu) + 1))" on macOS
Compiling with Clang for Linux users (macOS uses Clang by default).
To optionally compile with Clang instead of GCC (e.g. for fuzzing, sanitizers, better warnings/errors, or to use less resources), add CC=clang CXX=clang++ to your configure flags:

./configure CC=clang CXX=clang++
Pro tips.
If you're re-compiling frequently (e.g. for testing small changes), as long as you're not changing the build configuration you can skip directly to the make -j <n> step for subsequent builds.

On the other hand, when you change the build configuration (e.g. for a fuzz build), or you are building a branch containing substantial changes to the autoconf/automake scripts, or when the build isn't working, it's often best to start with a clean slate using make clean or make distclean. Here's a complete example:

./autogen.sh && ./configure <flags> && make clean && make -j <n>

Be sure to use ccache, the fast C/C++ compiler cache, to speed up your builds. It should already be installed as part of the dependencies described at the top of this article. Run man ccache or ccache -h for help.

You can also gain time by building only what you need. See the Bitcoin Core productivity notes for more.

You can run ./configure --help to see all the various configuration options. It's a long list, so it may be more practical to search for what you want with grep: ./configure --help | grep -A1 "your-search-term". The options I use the most often are --with-incompatible-bdb as mentioned above, and --enable-debug for debug builds for reviewing and testing pull requests.

If you build often, bash aliases may be helpful for abstracting the repetitive details down to short commands.

Fuzz Testing.

To compile for fuzz testing, build with Clang using the following:

  ./autogen.sh
  ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined CC=clang CXX=clang++
  make clean
  make -j <n>

The steps for fuzz builds with macOS are different (see this link for more details):

  ./autogen.sh
  ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++
  make clean
  make -j <n>

You can test running a single fuzzer with, for example: FUZZ=process_message ./src/test/fuzz/fuzz

Sanitizers.
To compile with sanitizers (credit to Marco Falke for this section), build with Clang using the following:
- MSan: Compiling with MSan is quite involved. For running tests with MSan, it's probably easiest to just compile normally and then invoke valgrind ./src/test/test_bitcoin or ./test/functional/test_runner.py --valgrind instead. Alternatively, you can use the ./ci/ system to run any sanitizer task.
- Other sanitizers: Simply append --with-sanitizers=undefined,integer CC=clang CXX=clang++ to your ./configure call (or --with-sanitizers=address, ...). Then, run the tests normally, using the env vars as needed:
```
  export LSAN_OPTIONS="suppressions=$(pwd)/test/sanitizer_suppressions/lsan"
  export TSAN_OPTIONS="suppressions=$(pwd)/test/sanitizer_suppressions/tsan:halt_on_error=1:second_deadlock_stack=1"
  export UBSAN_OPTIONS="suppressions=$(pwd)/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
```
See the Sanitizers section of the Bitcoin Core developer notes for more information.
Troubleshooting.
If you're seeing any odd issues while compiling, try using make clean or make distclean and rebuild from scratch as described above.

If that doesn't work, or if you are seeing linker issues, the header and compilation might be out of sync and you may want to ensure that you are building from a clean tree. If you are using ccache, try clearing the cache by running ccache -C (see man ccache or ccache -h for help) and then rebuilding.

If that doesn't solve it, you can also run git clean -f -x -d and rebuild. This wipes everything in the tree that doesn't belong to git, so be careful that you don't have anything else in the directory that you don't want to lose and move it out first.
Run the unit tests.
- make check, or
- make -j "$(($(nproc)+1))" check to use multithreading on Linux, or
- make -j "$(($(sysctl -n hw.physicalcpu)+1))" check to use multithreading on macOS
- See the Bitcoin Core unit tests documentation for more info.
Run the functional tests. From the repository root:
- test/functional/test_runner.py to run the standard test suite (try test/functional/test_runner.py -j 60 or a similar high number to run the tests more quickly in parallel)
- test/functional/.py to run an individual test file
- test/functional/test_runner.py --extended to run the extended test suite
- test/functional/test_runner.py --help to see the various options for running tests
- See the Bitcoin Core functional tests documentation for more info.

Cheers,

Jon Atack