Spaces:
Sleeping
Sleeping
| <html> | |
| <!-- Created by , GNU Texinfo 7.1 --> | |
| <head> | |
| <meta charset="utf-8"> | |
| <title> | |
| FFmpeg Automated Testing Environment | |
| </title> | |
| <meta name="viewport" content="width=device-width,initial-scale=1.0"> | |
| <link rel="stylesheet" type="text/css" href="bootstrap.min.css"> | |
| <link rel="stylesheet" type="text/css" href="style.min.css"> | |
| </head> | |
| <body> | |
| <div class="container"> | |
| <h1> | |
| FFmpeg Automated Testing Environment | |
| </h1> | |
| <a name="Top"></a> | |
| <a name="SEC_Top"></a> | |
| <div class="element-contents" id="SEC_Contents"> | |
| <h2 class="contents-heading">Table of Contents</h2> | |
| <div class="contents"> | |
| <ul class="toc-numbered-mark"> | |
| <li><a id="toc-Introduction" href="#Introduction">1 Introduction</a></li> | |
| <li><a id="toc-Using-FATE-from-your-FFmpeg-source-directory" href="#Using-FATE-from-your-FFmpeg-source-directory">2 Using FATE from your FFmpeg source directory</a></li> | |
| <li><a id="toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server">3 Submitting the results to the FFmpeg result aggregation server</a></li> | |
| <li><a id="toc-Uploading-new-samples-to-the-fate-suite" href="#Uploading-new-samples-to-the-fate-suite">4 Uploading new samples to the fate suite</a></li> | |
| <li><a id="toc-FATE-makefile-targets-and-variables" href="#FATE-makefile-targets-and-variables">5 FATE makefile targets and variables</a> | |
| <ul class="toc-numbered-mark"> | |
| <li><a id="toc-Makefile-targets" href="#Makefile-targets">5.1 Makefile targets</a></li> | |
| <li><a id="toc-Makefile-variables" href="#Makefile-variables">5.2 Makefile variables</a></li> | |
| <li><a id="toc-Examples" href="#Examples">5.3 Examples</a></li> | |
| </ul></li> | |
| </ul> | |
| </div> | |
| </div> | |
| <a name="Introduction"></a> | |
| <h2 class="chapter">1 Introduction<span class="pull-right"><a class="anchor hidden-xs" href="#Introduction" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Introduction" aria-hidden="true">TOC</a></span></h2> | |
| <p>FATE is an extended regression suite on the client-side and a means | |
| for results aggregation and presentation on the server-side. | |
| </p> | |
| <p>The first part of this document explains how you can use FATE from | |
| your FFmpeg source directory to test your ffmpeg binary. The second | |
| part describes how you can run FATE to submit the results to FFmpeg’s | |
| FATE server. | |
| </p> | |
| <p>In any way you can have a look at the publicly viewable FATE results | |
| by visiting this website: | |
| </p> | |
| <p><a class="url" href="http://fate.ffmpeg.org/">http://fate.ffmpeg.org/</a> | |
| </p> | |
| <p>This is especially recommended for all people contributing source | |
| code to FFmpeg, as it can be seen if some test on some platform broke | |
| with their recent contribution. This usually happens on the platforms | |
| the developers could not test on. | |
| </p> | |
| <p>The second part of this document describes how you can run FATE to | |
| submit your results to FFmpeg’s FATE server. If you want to submit your | |
| results be sure to check that your combination of CPU, OS and compiler | |
| is not already listed on the above mentioned website. | |
| </p> | |
| <p>In the third part you can find a comprehensive listing of FATE makefile | |
| targets and variables. | |
| </p> | |
| <a name="Using-FATE-from-your-FFmpeg-source-directory"></a> | |
| <h2 class="chapter">2 Using FATE from your FFmpeg source directory<span class="pull-right"><a class="anchor hidden-xs" href="#Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Using-FATE-from-your-FFmpeg-source-directory" aria-hidden="true">TOC</a></span></h2> | |
| <p>If you want to run FATE on your machine you need to have the samples | |
| in place. You can get the samples via the build target fate-rsync. | |
| Use this command from the top-level source directory: | |
| </p> | |
| <div class="example"> | |
| <pre class="example-preformatted">make fate-rsync SAMPLES=fate-suite/ | |
| make fate SAMPLES=fate-suite/ | |
| </pre></div> | |
| <p>The above commands set the samples location by passing a makefile | |
| variable via command line. It is also possible to set the samples | |
| location at source configuration time by invoking configure with | |
| <samp class="option">--samples=<path to the samples directory></samp>. Afterwards you can | |
| invoke the makefile targets without setting the <var class="var">SAMPLES</var> makefile | |
| variable. This is illustrated by the following commands: | |
| </p> | |
| <div class="example"> | |
| <pre class="example-preformatted">./configure --samples=fate-suite/ | |
| make fate-rsync | |
| make fate | |
| </pre></div> | |
| <p>Yet another way to tell FATE about the location of the sample | |
| directory is by making sure the environment variable FATE_SAMPLES | |
| contains the path to your samples directory. This can be achieved | |
| by e.g. putting that variable in your shell profile or by setting | |
| it in your interactive session. | |
| </p> | |
| <div class="example"> | |
| <pre class="example-preformatted">FATE_SAMPLES=fate-suite/ make fate | |
| </pre></div> | |
| <div class="info"> | |
| <p>Do not put a ’~’ character in the samples path to indicate a home | |
| directory. Because of shell nuances, this will cause FATE to fail. | |
| </p></div> | |
| <p>Beware that some assertions are disabled by default, so mind setting | |
| <samp class="option">--assert-level=<level></samp> at configuration time, e.g. when seeking | |
| the highest possible test coverage: | |
| </p><div class="example"> | |
| <pre class="example-preformatted">./configure --assert-level=2 | |
| </pre></div> | |
| <p>Note that raising the assert level could have a performance impact. | |
| </p> | |
| <p>To get the complete list of tests, run the command: | |
| </p><div class="example"> | |
| <pre class="example-preformatted">make fate-list | |
| </pre></div> | |
| <p>You can specify a subset of tests to run by specifying the | |
| corresponding elements from the list with the <code class="code">fate-</code> prefix, | |
| e.g. as in: | |
| </p><div class="example"> | |
| <pre class="example-preformatted">make fate-ffprobe_compact fate-ffprobe_xml | |
| </pre></div> | |
| <p>This makes it easier to run a few tests in case of failure without | |
| running the complete test suite. | |
| </p> | |
| <p>To use a custom wrapper to run the test, pass <samp class="option">--target-exec</samp> to | |
| <code class="command">configure</code> or set the <var class="var">TARGET_EXEC</var> Make variable. | |
| </p> | |
| <a name="Submitting-the-results-to-the-FFmpeg-result-aggregation-server"></a> | |
| <h2 class="chapter">3 Submitting the results to the FFmpeg result aggregation server<span class="pull-right"><a class="anchor hidden-xs" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" aria-hidden="true">TOC</a></span></h2> | |
| <p>To submit your results to the server you should run fate through the | |
| shell script <samp class="file">tests/fate.sh</samp> from the FFmpeg sources. This script needs | |
| to be invoked with a configuration file as its first argument. | |
| </p> | |
| <div class="example"> | |
| <pre class="example-preformatted">tests/fate.sh /path/to/fate_config | |
| </pre></div> | |
| <p>A configuration file template with comments describing the individual | |
| configuration variables can be found at <samp class="file">doc/fate_config.sh.template</samp>. | |
| </p> | |
| <p>The mentioned configuration template is also available here: | |
| </p><pre class="verbatim">slot= # some unique identifier | |
| repo=https://git.ffmpeg.org/ffmpeg.git # the source repository | |
| #branch=release/2.6 # the branch to test | |
| samples= # path to samples directory | |
| workdir= # directory in which to do all the work | |
| #fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report | |
| comment= # optional description | |
| build_only= # set to "yes" for a compile-only instance that skips tests | |
| ignore_tests= | |
| # the following are optional and map to configure options | |
| arch= | |
| cpu= | |
| cross_prefix= | |
| as= | |
| cc= | |
| ld= | |
| target_os= | |
| sysroot= | |
| target_exec= | |
| target_path= | |
| target_samples= | |
| extra_cflags= | |
| extra_ldflags= | |
| extra_libs= | |
| extra_conf= # extra configure options not covered above | |
| #make= # name of GNU make if not 'make' | |
| makeopts= # extra options passed to 'make' | |
| #makeopts_fate= # extra options passed to 'make' when running tests, | |
| # defaulting to makeopts above if this is not set | |
| #tar= # command to create a tar archive from its arguments on stdout, | |
| # defaults to 'tar c' | |
| #fate_targets= # targets to make when running fate; defaults to "fate", | |
| # can be set to run a subset of tests, e.g. "fate-checkasm". | |
| #fate_environments= # a list of names of configurations to run tests for; | |
| # each round is run with variables from ${${name}_env} set. | |
| # One example of using fate_environments: | |
| # target_exec="qemu-aarch64-static" | |
| # fate_targets="fate-checkasm fate-cpu" | |
| # fate_environments="sve128 sve256" | |
| # sve128_env="QEMU_CPU=max,sve128=on" | |
| # sve256_env="QEMU_CPU=max,sve256=on" | |
| # The variables set by fate_environments can also be used explicitly | |
| # by target_exec, e.g. like this: | |
| # target_exec="qemu-aarch64-static -cpu \$(MY_CPU)" | |
| # fate_targets="fate-checkasm fate-cpu" | |
| # fate_environments="sve128 sve256" | |
| # sve128_env="MY_CPU=max,sve128=on" | |
| # sve256_env="MY_CPU=max,sve256=on" | |
| </pre> | |
| <p>Create a configuration that suits your needs, based on the configuration | |
| template. The <code class="env">slot</code> configuration variable can be any string that is not | |
| yet used, but it is suggested that you name it adhering to the following | |
| pattern ‘<samp class="samp"><var class="var">arch</var>-<var class="var">os</var>-<var class="var">compiler</var>-<var class="var">compiler version</var></samp>’. The | |
| configuration file itself will be sourced in a shell script, therefore all | |
| shell features may be used. This enables you to setup the environment as you | |
| need it for your build. | |
| </p> | |
| <p>For your first test runs the <code class="env">fate_recv</code> variable should be empty or | |
| commented out. This will run everything as normal except that it will omit | |
| the submission of the results to the server. The following files should be | |
| present in $workdir as specified in the configuration file: | |
| </p> | |
| <ul class="itemize mark-bullet"> | |
| <li>configure.log | |
| </li><li>compile.log | |
| </li><li>test.log | |
| </li><li>report | |
| </li><li>version | |
| </li></ul> | |
| <p>When you have everything working properly you can create an SSH key pair | |
| and send the public key to the FATE server administrator who can be contacted | |
| at the email address <a class="email" href="mailto:fate-admin@ffmpeg.org">fate-admin@ffmpeg.org</a>. | |
| </p> | |
| <p>Configure your SSH client to use public key authentication with that key | |
| when connecting to the FATE server. Also do not forget to check the identity | |
| of the server and to accept its host key. This can usually be achieved by | |
| running your SSH client manually and killing it after you accepted the key. | |
| The FATE server’s fingerprint is: | |
| </p> | |
| <dl class="table"> | |
| <dt>‘<samp class="samp">RSA</samp>’</dt> | |
| <dd><p>d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 | |
| </p></dd> | |
| <dt>‘<samp class="samp">ECDSA</samp>’</dt> | |
| <dd><p>76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 | |
| </p></dd> | |
| </dl> | |
| <p>If you have problems connecting to the FATE server, it may help to try out | |
| the <code class="command">ssh</code> command with one or more <samp class="option">-v</samp> options. You should | |
| get detailed output concerning your SSH configuration and the authentication | |
| process. | |
| </p> | |
| <p>The only thing left is to automate the execution of the fate.sh script and | |
| the synchronisation of the samples directory. | |
| </p> | |
| <a name="Uploading-new-samples-to-the-fate-suite"></a> | |
| <h2 class="chapter">4 Uploading new samples to the fate suite<span class="pull-right"><a class="anchor hidden-xs" href="#Uploading-new-samples-to-the-fate-suite" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Uploading-new-samples-to-the-fate-suite" aria-hidden="true">TOC</a></span></h2> | |
| <p>If you need a sample uploaded send a mail to samples-request. | |
| </p> | |
| <p>This is for developers who have an account on the fate suite server. | |
| If you upload new samples, please make sure they are as small as possible, | |
| space on each client, network bandwidth and so on benefit from smaller test cases. | |
| Also keep in mind older checkouts use existing sample files, that means in | |
| practice generally do not replace, remove or overwrite files as it likely would | |
| break older checkouts or releases. | |
| Also all needed samples for a commit should be uploaded, ideally 24 | |
| hours, before the push. | |
| If you need an account for frequently uploading samples or you wish to help | |
| others by doing that send a mail to ffmpeg-devel. | |
| </p> | |
| <div class="example"> | |
| <pre class="example-preformatted">#First update your local samples copy: | |
| rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite | |
| #Then do a dry run checking what would be uploaded: | |
| rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite | |
| #Upload the files: | |
| rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite | |
| </pre></div> | |
| <a name="FATE-makefile-targets-and-variables"></a> | |
| <h2 class="chapter">5 FATE makefile targets and variables<span class="pull-right"><a class="anchor hidden-xs" href="#FATE-makefile-targets-and-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-FATE-makefile-targets-and-variables" aria-hidden="true">TOC</a></span></h2> | |
| <a name="Makefile-targets"></a> | |
| <h3 class="section">5.1 Makefile targets<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-targets" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-targets" aria-hidden="true">TOC</a></span></h3> | |
| <dl class="table"> | |
| <dt><samp class="option">fate-rsync</samp></dt> | |
| <dd><p>Download/synchronize sample files to the configured samples directory. | |
| </p> | |
| </dd> | |
| <dt><samp class="option">fate-list</samp></dt> | |
| <dd><p>Will list all fate/regression test targets. | |
| </p> | |
| </dd> | |
| <dt><samp class="option">fate-list-failing</samp></dt> | |
| <dd><p>List the fate tests that failed the last time they were executed. | |
| </p> | |
| </dd> | |
| <dt><samp class="option">fate-clear-reports</samp></dt> | |
| <dd><p>Remove the test reports from previous test executions (getting rid of | |
| potentially stale results from fate-list-failing). | |
| </p> | |
| </dd> | |
| <dt><samp class="option">fate</samp></dt> | |
| <dd><p>Run the FATE test suite (requires the fate-suite dataset). | |
| </p></dd> | |
| </dl> | |
| <a name="Makefile-variables"></a> | |
| <h3 class="section">5.2 Makefile variables<span class="pull-right"><a class="anchor hidden-xs" href="#Makefile-variables" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Makefile-variables" aria-hidden="true">TOC</a></span></h3> | |
| <dl class="table"> | |
| <dt><code class="env">V</code></dt> | |
| <dd><p>Verbosity level, can be set to 0, 1 or 2. | |
| </p><ul class="itemize mark-bullet"> | |
| <li>0: show just the test arguments | |
| </li><li>1: show just the command used in the test | |
| </li><li>2: show everything | |
| </li></ul> | |
| </dd> | |
| <dt><code class="env">SAMPLES</code></dt> | |
| <dd><p>Specify or override the path to the FATE samples at make time, it has a | |
| meaning only while running the regression tests. | |
| </p> | |
| </dd> | |
| <dt><code class="env">THREADS</code></dt> | |
| <dd><p>Specify how many threads to use while running regression tests, it is | |
| quite useful to detect thread-related regressions. | |
| </p> | |
| <p>This variable may be set to the string "random", optionally followed by a | |
| number, like "random99", This will cause each test to use a random number of | |
| threads. If a number is specified, it is used as a maximum number of threads, | |
| otherwise 16 is the maximum. | |
| </p> | |
| <p>In case a test fails, the thread count used for it will be written into the | |
| errfile. | |
| </p> | |
| </dd> | |
| <dt><code class="env">THREAD_TYPE</code></dt> | |
| <dd><p>Specify which threading strategy test, either ‘<samp class="samp">slice</samp>’ or ‘<samp class="samp">frame</samp>’, | |
| by default ‘<samp class="samp">slice+frame</samp>’ | |
| </p> | |
| </dd> | |
| <dt><code class="env">CPUFLAGS</code></dt> | |
| <dd><p>Specify CPU flags. | |
| </p> | |
| </dd> | |
| <dt><code class="env">TARGET_EXEC</code></dt> | |
| <dd><p>Specify or override the wrapper used to run the tests. | |
| The <code class="env">TARGET_EXEC</code> option provides a way to run FATE wrapped in | |
| <code class="command">valgrind</code>, <code class="command">qemu-user</code> or <code class="command">wine</code> or on remote targets | |
| through <code class="command">ssh</code>. | |
| </p> | |
| </dd> | |
| <dt><code class="env">GEN</code></dt> | |
| <dd><p>Set to ‘<samp class="samp">1</samp>’ to generate the missing or mismatched references. | |
| </p> | |
| </dd> | |
| <dt><code class="env">HWACCEL</code></dt> | |
| <dd><p>Specify which hardware acceleration to use while running regression tests, | |
| by default ‘<samp class="samp">none</samp>’ is used. | |
| </p> | |
| </dd> | |
| <dt><code class="env">KEEP</code></dt> | |
| <dd><p>Set to ‘<samp class="samp">1</samp>’ to keep temp files generated by fate test(s) when test is successful. | |
| Default is ‘<samp class="samp">0</samp>’, which removes these files. Files are always kept when a test | |
| fails. | |
| </p> | |
| </dd> | |
| </dl> | |
| <a name="Examples"></a> | |
| <h3 class="section">5.3 Examples<span class="pull-right"><a class="anchor hidden-xs" href="#Examples" aria-hidden="true">#</a> <a class="anchor hidden-xs"href="#toc-Examples" aria-hidden="true">TOC</a></span></h3> | |
| <div class="example"> | |
| <pre class="example-preformatted">make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate | |
| </pre></div> | |
| <p style="font-size: small;"> | |
| This document was generated using <a class="uref" href="https://www.gnu.org/software/texinfo/"><em class="emph">makeinfo</em></a>. | |
| </p> | |
| </div> | |
| </body> | |
| </html> | |