| | ============== |
| | Testing libc++ |
| | ============== |
| |
|
| | .. contents:: |
| | :local: |
| |
|
| | Getting Started |
| | =============== |
| |
|
| | libc++ uses LIT to configure and run its tests. |
| |
|
| | The primary way to run the libc++ tests is by using `make check-libcxx`. |
| |
|
| | However since libc++ can be used in any number of possible |
| | configurations it is important to customize the way LIT builds and runs |
| | the tests. This guide provides information on how to use LIT directly to |
| | test libc++. |
| |
|
| | Please see the `Lit Command Guide`_ for more information about LIT. |
| |
|
| | .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html |
| |
|
| | Setting up the Environment |
| | -------------------------- |
| |
|
| | After building libc++ you must setup your environment to test libc++ using |
| | LIT. |
| |
|
| | #. Create a shortcut to the actual lit executable so that you can invoke it |
| | easily from the command line. |
| |
|
| | .. code-block:: bash |
| |
|
| | $ alias lit='python path/to/llvm/utils/lit/lit.py' |
| |
|
| | #. Tell LIT where to find your build configuration. |
| |
|
| | .. code-block:: bash |
| |
|
| | $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg |
| |
|
| | Example Usage |
| | ------------- |
| |
|
| | Once you have your environment set up and you have built libc++ you can run |
| | parts of the libc++ test suite by simply running `lit` on a specified test or |
| | directory. For example: |
| |
|
| | .. code-block:: bash |
| |
|
| | $ cd path/to/src/libcxx |
| | $ lit -sv test/std/re # Run all of the std::regex tests |
| | $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test |
| | $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic |
| |
|
| | Sometimes you'll want to change the way LIT is running the tests. Custom options |
| | can be specified using the `--param=<name>=<val>` flag. The most common option |
| | you'll want to change is the standard dialect (ie -std=c++XX). By default the |
| | test suite will select the newest C++ dialect supported by the compiler and use |
| | that. However if you want to manually specify the option like so: |
| |
|
| | .. code-block:: bash |
| |
|
| | $ lit -sv test/std/containers # Run the tests with the newest -std |
| | $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 |
| |
|
| | Occasionally you'll want to add extra compile or link flags when testing. |
| | You can do this as follows: |
| |
|
| | .. code-block:: bash |
| |
|
| | $ lit -sv --param=compile_flags='-Wcustom-warning' |
| | $ lit -sv --param=link_flags='-L/custom/library/path' |
| |
|
| | Some other common examples include: |
| |
|
| | .. code-block:: bash |
| |
|
| | # Specify a custom compiler. |
| | $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std |
| |
|
| | # Enable warnings in the test suite |
| | $ lit -sv --param=enable_warnings=true test/std |
| |
|
| | # Use UBSAN when running the tests. |
| | $ lit -sv --param=use_sanitizer=Undefined |
| |
|
| |
|
| | LIT Options |
| | =========== |
| |
|
| | :program:`lit` [*options*...] [*filenames*...] |
| |
|
| | Command Line Options |
| | -------------------- |
| |
|
| | To use these options you pass them on the LIT command line as --param NAME or |
| | --param NAME=VALUE. Some options have default values specified during CMake's |
| | configuration. Passing the option on the command line will override the default. |
| |
|
| | .. program:: lit |
| |
|
| | .. option:: cxx_under_test=<path/to/compiler> |
| |
|
| | Specify the compiler used to build the tests. |
| |
|
| | .. option:: cxx_stdlib_under_test=<stdlib name> |
| |
|
| | **Values**: libc++, libstdc++ |
| |
|
| | Specify the C++ standard library being tested. Unless otherwise specified |
| | libc++ is used. This option is intended to allow running the libc++ test |
| | suite against other standard library implementations. |
| |
|
| | .. option:: std=<standard version> |
| |
|
| | **Values**: c++98, c++03, c++11, c++14, c++17, c++2a |
| |
|
| | Change the standard version used when building the tests. |
| |
|
| | .. option:: libcxx_site_config=<path/to/lit.site.cfg> |
| |
|
| | Specify the site configuration to use when running the tests. This option |
| | overrides the environment variable LIBCXX_SITE_CONFIG. |
| |
|
| | .. option:: cxx_headers=<path/to/headers> |
| |
|
| | Specify the c++ standard library headers that are tested. By default the |
| | headers in the source tree are used. |
| |
|
| | .. option:: cxx_library_root=<path/to/lib/> |
| |
|
| | Specify the directory of the libc++ library to be tested. By default the |
| | library folder of the build directory is used. This option cannot be used |
| | when use_system_cxx_lib is provided. |
| |
|
| |
|
| | .. option:: cxx_runtime_root=<path/to/lib/> |
| |
|
| | Specify the directory of the libc++ library to use at runtime. This directory |
| | is not added to the linkers search path. This can be used to compile tests |
| | against one version of libc++ and run them using another. The default value |
| | for this option is `cxx_library_root`. |
| |
|
| | .. option:: use_system_cxx_lib=<bool> |
| |
|
| | **Default**: False |
| |
|
| | Enable or disable testing against the installed version of libc++ library. |
| | Note: This does not use the installed headers. |
| |
|
| | .. option:: use_lit_shell=<bool> |
| |
|
| | Enable or disable the use of LIT's internal shell in ShTests. If the |
| | environment variable LIT_USE_INTERNAL_SHELL is present then that is used as |
| | the default value. Otherwise the default value is True on Windows and False |
| | on every other platform. |
| |
|
| | .. option:: compile_flags="<list-of-args>" |
| |
|
| | Specify additional compile flags as a space delimited string. |
| | Note: This options should not be used to change the standard version used. |
| |
|
| | .. option:: link_flags="<list-of-args>" |
| |
|
| | Specify additional link flags as a space delimited string. |
| |
|
| | .. option:: debug_level=<level> |
| |
|
| | **Values**: 0, 1 |
| |
|
| | Enable the use of debug mode. Level 0 enables assertions and level 1 enables |
| | assertions and debugging of iterator misuse. |
| |
|
| | .. option:: use_sanitizer=<sanitizer name> |
| |
|
| | **Values**: Memory, MemoryWithOrigins, Address, Undefined |
| |
|
| | Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when |
| | building libc++ then that sanitizer will be used by default. |
| |
|
| | .. option:: color_diagnostics |
| |
|
| | Enable the use of colorized compile diagnostics. If the color_diagnostics |
| | option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is |
| | present then color diagnostics will be enabled. |
| |
|
| | .. option:: llvm_unwinder |
| |
|
| | Enable the use of LLVM unwinder instead of libgcc. |
| |
|
| | .. option:: builtins_library |
| |
|
| | Path to the builtins library to use instead of libgcc. |
| |
|
| |
|
| | Environment Variables |
| | --------------------- |
| |
|
| | .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> |
| |
|
| | Specify the site configuration to use when running the tests. |
| | Also see `libcxx_site_config`. |
| |
|
| | .. envvar:: LIBCXX_COLOR_DIAGNOSTICS |
| |
|
| | If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt |
| | to use color diagnostic outputs from the compiler. |
| | Also see `color_diagnostics`. |
| |
|
| | Benchmarks |
| | ========== |
| |
|
| | Libc++ contains benchmark tests separately from the test of the test suite. |
| | The benchmarks are written using the `Google Benchmark`_ library, a copy of which |
| | is stored in the libc++ repository. |
| |
|
| | For more information about using the Google Benchmark library see the |
| | `official documentation <https://github.com/google/benchmark>`_. |
| |
|
| | .. _`Google Benchmark`: https://github.com/google/benchmark |
| |
|
| | Building Benchmarks |
| | ------------------- |
| |
|
| | The benchmark tests are not built by default. The benchmarks can be built using |
| | the ``cxx-benchmarks`` target. |
| |
|
| | An example build would look like: |
| |
|
| | .. code-block:: bash |
| |
|
| | $ cd build |
| | $ cmake [options] <path to libcxx sources> |
| | $ make cxx-benchmarks |
| |
|
| | This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be |
| | built against the just-built libc++. The compiled tests are output into |
| | ``build/benchmarks``. |
| |
|
| | The benchmarks can also be built against the platforms native standard library |
| | using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This |
| | is useful for comparing the performance of libc++ to other standard libraries. |
| | The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and |
| | ``<test>.native.out`` otherwise. |
| |
|
| | Also See: |
| |
|
| | * :ref:`Building Libc++ <build instructions>` |
| | * :ref:`CMake Options` |
| |
|
| | Running Benchmarks |
| | ------------------ |
| |
|
| | The benchmarks must be run manually by the user. Currently there is no way |
| | to run them as part of the build. |
| |
|
| | For example: |
| |
|
| | .. code-block:: bash |
| |
|
| | $ cd build/benchmarks |
| | $ make cxx-benchmarks |
| | $ ./algorithms.libcxx.out # Runs all the benchmarks |
| | $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks |
| |
|
| | For more information about running benchmarks see `Google Benchmark`_. |
| |
|
