Add MVP model for the WIC and add new configuration arguments, along with new documentation

.codecov.yml

@@ -5,6 +5,7 @@ ignore:
   - "app.py"
   - "app2.py"
   - "scoutbot/*/convert.py"
+  - "scoutbot/*/convert.mvp.py"
   - "scoutbot/scoutbot.py"
   - "scoutbot/loc/transforms"
 

README.rst

@@ -49,6 +49,47 @@ or, you can run the image-base Gradio demo with:
 Docker
 ------
 
+To run with Docker:
+
+.. code-block:: console
+
+    docker run \
+       -it \
+       --rm \
+       -p 7860:7860 \
+       -e CONFIG=phase1 \
+       -e WIC_BATCH_SIZE=512 \
+       --gpus all \
+       --name scoutbot \
+       wildme/scoutbot:main \
+       python3 app2.py
+
+To run with Docker Compose:
+
+.. code-block:: yaml
+
+    version: "3"
+
+    services:
+      scoutbot:
+        image: wildme/scoutbot:main
+        command: python3 app2.py
+        ports:
+          - "7860:7860"
+        environment:
+          CONFIG: phase1
+          WIC_BATCH_SIZE: 512
+        restart: unless-stopped
+        deploy:
+          resources:
+            reservations:
+              devices:
+                - driver: nvidia
+                  device_ids: ["all"]
+                  capabilities: [gpu]
+
+and run ``docker compose up -d``.
+
 The application can also be built into a Docker image and is hosted on Docker Hub as ``wildme/scoutbot:latest``.
 
 .. code-block:: console
@@ -65,17 +106,6 @@ The application can also be built into a Docker image and is hosted on Docker Hu
         --push \
         .
 
-To run with Docker:
-
-.. code-block:: console
-
-    docker run \
-       -it \
-       --rm \
-       -p 7860:7860 \
-       --name scoutbot \
-       wildme/scoutbot:latest
-
 Tests and Coverage
 ------------------
 

app2.py

@@ -25,7 +25,7 @@ def predict(filepath, wic_thresh, loc_thresh, agg_thresh, loc_nms_thresh, agg_nm
     pixels = h * w
     megapixels = pixels / 1e6
 
-    detects = scoutbot.pipeline(
+    wic_, detects = scoutbot.pipeline(
         filepath, wic_thresh, loc_thresh, loc_nms_thresh, agg_thresh, agg_nms_thresh
     )
 

docs/cli.rst

@@ -1,11 +1,16 @@
 ScoutBot CLI
 ============
 
+ScoutBot is the machine learning interface for the Wild Me Scout project.  This page specifies
+the Command Line Interface (CLI) to interact with all of the algorithms and machine learning
+models that have been pretrained for inference in a production environment.
+
 .. toctree::
    :maxdepth: 3
    :caption: Contents:
 
-
 .. click:: scoutbot.scoutbot:cli
    :prog: scoutbot
    :nested: full
+
+.. include:: environment.rst

docs/environment.rst

@@ -0,0 +1,15 @@
+Environment Variables
+---------------------
+
+The Scoutbot API and CLI have two environment variables (envars) that allow you to configure global settings
+and configurations.
+
+   - ``CONFIG`` (default: phase1)
+      The configuration setting for which machine lerning models to use.
+      Must be one of ``phase1`` or ``mvp``.
+   - ``WIC_BATCH_SIZE`` (default: 256)
+      The configuration setting for how many tiles to send to the GPU in a single batch during the WIC
+      prediction (forward inference).  The LOC model has a fixed batch size (16 for ``phase1`` and
+      32 for ``mvp``) and cannot be adjusted.  This setting can be used to control how fast the pipeline
+      runs, as a trade-off of faster compute for more memory usage.  It is highly suggested to set this
+      value as high as possible to fit into the GPU.

docs/index.rst

@@ -1,12 +1,5 @@
 .. include:: ../README.rst
 
-.. note::
-
-   This project is under active development.
-
-Contents
---------
-
 .. toctree::
 
    Home <self>

docs/onnx.rst

@@ -0,0 +1,27 @@
+CDN Model Download (ONNX)
+-------------------------
+
+All of the machine learning models are hosted on GitHub as LFS files.  The two modules (``WIC`` and ``LOC``)
+however need those files downloaded to the local machine prior to running inference.  These models are
+hosted on a separate CDN for convenient access and can be fetched by running the following functions:
+
+   - :meth:`scoutbot.wic.fetch`
+   - :meth:`scoutbot.loc.fetch`
+
+To pre-download the models for a specific config (e.g., ``mvp``), you can specify an optional config:
+
+   - :obj:`scoutbot.wic.fetch(config="mvp")`
+   - :obj:`scoutbot.loc.fetch(config="mvp")`
+
+These functions will download the following files and will store them in your Operating System's default
+cache folder:
+
+   - Phase 1
+      - ``WIC``: ``https://wildbookiarepository.azureedge.net/models/scout.wic.5fbfff26.3.0.onnx`` (81MB)
+         SHA256 checksum: ``cbc7f381fa58504e03b6510245b6b2742d63049429337465d95663a6468df4c1``
+      - ``LOC``: ``https://wildbookiarepository.azureedge.net/models/scout.loc.5fbfff26.0.onnx`` (209MB)
+         SHA256 checksum: ``85a9378311d42b5143f74570136f32f50bf97c548135921b178b46ba7612b216``
+
+   - MVP
+      - ``WIC``: ``https://wildbookiarepository.azureedge.net/models/scout.wic.mvp.2.0.onnx`` (97MB)
+         SHA256 checksum: ``3ff3a192803e53758af5e112526ba9622f1dedc55e2fa88850db6f32af160f32``

docs/overview.rst

@@ -0,0 +1,38 @@
+Overview
+--------
+
+In general, the structure of this API is to expose four main processing components for the Scout project.
+These components are, in order: ``TILE``, ``WIC``, ``LOC``, and ``AGG``.
+
+   1. ``TILE``: A module to convert images to tiles
+   2. ``WIC``: A module to classify tiles as relevant for further processing (i.e., does it likely have an elephant?)
+   3. ``LOC``: A module to detect elephants in tiles
+   4. ``AGG``: A module to aggregate the tile-level detections back onto the original image
+
+The ``TILE`` step and ``AGG`` steps are heuristic-based algorithms and do not need to use any
+machine learning (ML) models or GPU offload.  In contrast, the ``WIC`` and ``LOC`` steps both require
+their own ML models and can be computed on CPU or GPU (if available).
+
+The non-ML components (``TILE`` and ``AGG``) both expose :func:`compute` functions, which is the single
+point of interaction as the developer:
+
+   - :meth:`scoutbot.tile.compute`
+   - :meth:`scoutbot.agg.compute`
+
+The ML components (``WIC`` and ``LOC``), in contrast, is a bit more complex and exposes three functions:
+
+   - :func:`pre` (preprocessing)
+   - :func:`predict` (inference)
+   - :func:`post` (postprocessing)
+
+For the WIC, these functions are:
+
+   - :meth:`scoutbot.wic.pre`
+   - :meth:`scoutbot.wic.predict`
+   - :meth:`scoutbot.wic.post`
+
+and for the LOC, these functions are:
+
+   - :meth:`scoutbot.loc.pre`
+   - :meth:`scoutbot.loc.predict`
+   - :meth:`scoutbot.loc.post`

docs/scoutbot.rst

@@ -1,70 +1,19 @@
 ScoutBot API
 ============
 
-.. toctree::
-   :maxdepth: 3
-   :caption: Contents:
-
 ScoutBot is the machine learning interface for the Wild Me Scout project.  This page specifies
 the Python API to interact with all of the algorithms and machine learning models that have been
 pretrained for inference in a production environment.
 
-Overview
---------
-
-In general, the structure of this API is to expose four main processing components for the Scout project.
-These components are, in order: ``TILE``, ``WIC``, ``LOC``, and ``AGG``.
-
-   1. ``TILE``: A module to convert images to tiles
-   2. ``WIC``: A module to classify tiles as relevant for further processing (i.e., does it likely have an elephant?)
-   3. ``LOC``: A module to detect elephants in tiles
-   4. ``AGG``: A module to aggregate the tile-level detections back onto the original image
-
-The ``TILE`` step and ``AGG`` steps are heuristic-based algorithms and do not need to use any
-machine learning (ML) models or GPU offload.  In contrast, the ``WIC`` and ``LOC`` steps both require
-their own ML models and can be computed on CPU or GPU (if available).
-
-The non-ML components (``TILE`` and ``AGG``) both expose :func:`compute` functions, which is the single
-point of interaction as the developer:
-
-   - :meth:`scoutbot.tile.compute`
-   - :meth:`scoutbot.agg.compute`
-
-The ML components (``WIC`` and ``LOC``), in contrast, is a bit more complex and exposes three functions:
-
-   - :func:`pre` (preprocessing)
-   - :func:`predict` (inference)
-   - :func:`post` (postprocessing)
-
-For the WIC, these functions are:
-
-   - :meth:`scoutbot.wic.pre`
-   - :meth:`scoutbot.wic.predict`
-   - :meth:`scoutbot.wic.post`
-
-and for the LOC, these functions are:
-
-   - :meth:`scoutbot.loc.pre`
-   - :meth:`scoutbot.loc.predict`
-   - :meth:`scoutbot.loc.post`
-
-CDN Model Download (ONNX)
--------------------------
-
-All of the machine learning models are hosted on GitHub as LFS files.  The two modules (``WIC`` and ``LOC``)
-however need those files downloaded to the local machine prior to running inference.  These models are
-hosted on a separate CDN for convenient access and can be fetched by running the following functions:
+.. toctree::
+   :maxdepth: 3
+   :caption: Contents:
 
-   - :meth:`scoutbot.wic.fetch`
-   - :meth:`scoutbot.loc.fetch`
+.. include:: overview.rst
 
-These functions will download the following files and will store them in your Operating System's default
-cache folder:
+.. include:: environment.rst
 
-   - ``WIC``: ``https://wildbookiarepository.azureedge.net/models/scout.wic.5fbfff26.3.0.onnx`` (81MB)
-      SHA256 checksum: ``cbc7f381fa58504e03b6510245b6b2742d63049429337465d95663a6468df4c1``
-   - ``LOC``: ``https://wildbookiarepository.azureedge.net/models/scout.loc.5fbfff26.0.onnx`` (209MB)
-      SHA256 checksum: ``85a9378311d42b5143f74570136f32f50bf97c548135921b178b46ba7612b216``
+.. include:: onnx.rst
 
 Tiles (TILE)
 ------------
@@ -74,7 +23,6 @@ Tiles (TILE)
    :undoc-members:
    :show-inheritance:
 
-
 Whole-Image Classifier (WIC)
 ----------------------------
 

requirements.txt

@@ -1,4 +1,6 @@
 click
+codecov
+coverage
 cryptography
 gradio
 imgaug

scoutbot/__init__.py

@@ -13,12 +13,13 @@ how the entire pipeline can be run on tiles or images, respectively.
 
     # Get image filepath
     filepath = '/path/to/image.ext'
+    config = 'mvp'
 
     # Run tiling
     img_shape, tile_grids, tile_filepaths = tile.compute(filepath)
 
     # Run WIC
-    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths)))
+    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths, config=config)))
 
     # Threshold for WIC
     flags = [wic_output.get('positive') >= wic_thresh for wic_output in wic_outputs]
@@ -28,7 +29,7 @@ how the entire pipeline can be run on tiles or images, respectively.
     # Run localizer
     loc_outputs = loc.post(
         loc.predict(
-            loc.pre(loc_tile_filepaths)
+            loc.pre(loc_tile_filepaths, config=config)
         ),
         loc_thresh=loc_thresh,
         nms_thresh=loc_nms_thresh
@@ -39,6 +40,7 @@ how the entire pipeline can be run on tiles or images, respectively.
         img_shape,
         loc_tile_grids,
         loc_outputs,
+        config=config,
         agg_thresh=agg_thresh,
         nms_thresh=agg_nms_thresh,
     )
@@ -55,12 +57,12 @@ log = utils.init_logging()
 
 from scoutbot import agg, loc, tile, wic  # NOQA
 
-VERSION = '0.1.14'
+VERSION = '0.1.15'
 version = VERSION
 __version__ = VERSION
 
 
-def fetch(pull=False):
+def fetch(pull=False, config=None):
     """
     Fetch the WIC and Localizer ONNX model files from a CDN if they do not exist locally.
 
@@ -68,8 +70,10 @@ def fetch(pull=False):
     files otherwise do not exist locally on disk.
 
     Args:
-        pull (bool, optional): If :obj:`True`, use the downloaded versions stored in
-            the local system's cache.  Defaults to :obj:`False`.
+        pull (bool, optional): If :obj:`True`, force using the downloaded versions
+            stored in the local system's cache.  Defaults to :obj:`False`.
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
 
     Returns:
         None
@@ -77,17 +81,18 @@ def fetch(pull=False):
     Raises:
         AssertionError: If any model cannot be fetched.
     """
-    wic.fetch(pull=pull)
-    loc.fetch(pull=pull)
+    wic.fetch(pull=pull, config=None)
+    loc.fetch(pull=pull, config=None)
 
 
 def pipeline(
     filepath,
-    wic_thresh=wic.WIC_THRESH,
-    loc_thresh=loc.LOC_THRESH,
-    loc_nms_thresh=loc.NMS_THRESH,
-    agg_thresh=agg.AGG_THRESH,
-    agg_nms_thresh=agg.NMS_THRESH,
+    config=None,
+    wic_thresh=wic.CONFIGS[None]['thresh'],
+    loc_thresh=loc.CONFIGS[None]['thresh'],
+    loc_nms_thresh=loc.CONFIGS[None]['nms'],
+    agg_thresh=agg.CONFIGS[None]['thresh'],
+    agg_nms_thresh=agg.CONFIGS[None]['nms'],
     clean=True,
 ):
     """
@@ -109,6 +114,21 @@ def pipeline(
 
     Args:
         filepath (str): image filepath (relative or absolute)
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
+        wic_thresh (float or None, optional): the confidence threshold for the WIC's
+            predictions.  Defaults to the ``phase1`` configuration setting.
+        loc_thresh (float or None, optional): the confidence threshold for the localizer's
+            predictions.  Defaults to the ``phase1`` configuration setting.
+        nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the localizer's predictions.  Defaults to the ``phase1`` configuration setting.
+        agg_thresh (float or None, optional): the confidence threshold for the aggregated
+            localizer predictions.  Defaults to the ``phase1`` configuration setting.
+        agg_nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the aggregated localizer's predictions.  Defaults to the ``phase1``
+            configuration setting.
+        clean (bool, optional): a flag to clean up any on-disk tiles that were generated.
+            Defaults to :obj:`True`.
 
     Returns:
         tuple ( float, list ( dict ) ): wic score, list of predictions
@@ -119,7 +139,7 @@ def pipeline(
     img_shape, tile_grids, tile_filepaths = tile.compute(filepath)
 
     # Run WIC
-    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths)))
+    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths, config=config)))
 
     # Threshold for WIC
     wic_ = max(wic_output.get('positive') for wic_output in wic_outputs)
@@ -131,7 +151,7 @@ def pipeline(
 
     # Run localizer
     loc_outputs = loc.post(
-        loc.predict(loc.pre(loc_tile_filepaths)),
+        loc.predict(loc.pre(loc_tile_filepaths, config=config)),
         loc_thresh=loc_thresh,
         nms_thresh=loc_nms_thresh,
     )
@@ -142,6 +162,7 @@ def pipeline(
         img_shape,
         loc_tile_grids,
         loc_outputs,
+        config=config,
         agg_thresh=agg_thresh,
         nms_thresh=agg_nms_thresh,
     )
@@ -156,11 +177,12 @@ def pipeline(
 
 def batch(
     filepaths,
-    wic_thresh=wic.WIC_THRESH,
-    loc_thresh=loc.LOC_THRESH,
-    loc_nms_thresh=loc.NMS_THRESH,
-    agg_thresh=agg.AGG_THRESH,
-    agg_nms_thresh=agg.NMS_THRESH,
+    config=None,
+    wic_thresh=wic.CONFIGS[None]['thresh'],
+    loc_thresh=loc.CONFIGS[None]['thresh'],
+    loc_nms_thresh=loc.CONFIGS[None]['nms'],
+    agg_thresh=agg.CONFIGS[None]['thresh'],
+    agg_nms_thresh=agg.CONFIGS[None]['nms'],
     clean=True,
 ):
     """
@@ -184,6 +206,21 @@ def batch(
 
     Args:
         filepaths (list): list of str image filepath (relative or absolute)
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
+        wic_thresh (float or None, optional): the confidence threshold for the WIC's
+            predictions.  Defaults to the ``phase1`` configuration setting.
+        loc_thresh (float or None, optional): the confidence threshold for the localizer's
+            predictions.  Defaults to the ``phase1`` configuration setting.
+        nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the localizer's predictions.  Defaults to the ``phase1`` configuration setting.
+        agg_thresh (float or None, optional): the confidence threshold for the aggregated
+            localizer predictions.  Defaults to the ``phase1`` configuration setting.
+        agg_nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the aggregated localizer's predictions.  Defaults to the ``phase1``
+            configuration setting.
+        clean (bool, optional): a flag to clean up any on-disk tiles that were generated.
+            Defaults to :obj:`True`.
 
     Returns:
         tuple ( list ( float ), list ( list ( dict ) ) : corresponding list of wic scores, corresponding list of lists of predictions
@@ -218,7 +255,7 @@ def batch(
         tile_grids += batch_grids
         tile_filepaths += batch_filepaths
 
-    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths)))
+    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths, config=config)))
 
     wic_dict = {}
     for tile_img_filepath, wic_output in zip(tile_img_filepaths, wic_outputs):
@@ -238,7 +275,7 @@ def batch(
 
     # Run localizer
     loc_outputs = loc.post(
-        loc.predict(loc.pre(loc_tile_filepaths)),
+        loc.predict(loc.pre(loc_tile_filepaths, config=config)),
         loc_thresh=loc_thresh,
         nms_thresh=loc_nms_thresh,
     )
@@ -266,6 +303,7 @@ def batch(
             img_shape,
             loc_tile_grids,
             loc_outputs,
+            config=config,
             agg_thresh=agg_thresh,
             nms_thresh=agg_nms_thresh,
         )
@@ -283,7 +321,7 @@ def batch(
 
 def example():
     """
-    Run the pipeline on an example image
+    Run the pipeline on an example image with the Phase 1 models
     """
     TEST_IMAGE = 'scout.example.jpg'
     TEST_IMAGE_HASH = (

scoutbot/agg/__init__.py

@@ -6,14 +6,28 @@ at the image level.  This includes the ability to weight the importance of detec
 along the border of each tile within an image, and performing non-maximum suppression (NMS)
 on the combined results.
 """
+import os
+
 import numpy as np
 import utool as ut
 
 from scoutbot import log
 
 MARGIN = 32.0
-AGG_THRESH = 0.4
-NMS_THRESH = 0.2
+
+DEFAULT_CONFIG = os.getenv('CONFIG', 'phase1').strip().lower()
+CONFIGS = {
+    'phase1': {
+        'thresh': 0.4,
+        'nms': 0.2,
+    },
+    'mvp': {
+        'thresh': 0.4,
+        'nms': 0.2,
+    },
+}
+CONFIGS[None] = CONFIGS[DEFAULT_CONFIG]
+assert DEFAULT_CONFIG in CONFIGS
 
 
 def iou(box1, box2):
@@ -76,6 +90,16 @@ def demosaic(img_shape, tile_grids, loc_outputs, margin=MARGIN):
     """
     Demosaics a list of tiles and their respective detections back into the original
     image's coordinate system.
+
+    Args:
+        img_shape (tuple): a tuple of the image shape as ``h, w, c`` or ``h, w``
+        tile_grids (list of dict): a list of tile coordinates
+        loc_output (list of list of dict): the output predictions from the Localizer.
+        margin (float, optional): the margin of the image to weight predictions.
+            Defaults to 32.0
+
+    Returns:
+        list ( dict ): list of Localizer predictions
     """
     assert len(tile_grids) == len(loc_outputs)
 
@@ -165,15 +189,36 @@ def demosaic(img_shape, tile_grids, loc_outputs, margin=MARGIN):
 
 
 def compute(
-    img_shape, tile_grids, loc_outputs, agg_thresh=AGG_THRESH, nms_thresh=NMS_THRESH
+    img_shape, tile_grids, loc_outputs, config=None, agg_thresh=None, nms_thresh=None
 ):
     """
     Compute the aggregated image-level detection results for a given list of tile-level detections.
+
+    Args:
+        img_shape (tuple): a tuple of the image shape as ``h, w, c`` or ``h, w``
+        tile_grids (list of dict): a list of tile coordinates
+        loc_output (list of list of dict): the output predictions from the Localizer.
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
+        agg_thresh (float or None, optional): the confidence threshold for the aggregated
+            localizer predictions.  Defaults to None.  Defaults to :obj:`None`
+            (the ``phase1`` model's settings).
+        nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the aggregated localizer's predictions.  Defaults to :obj:`None`
+            (the ``phase1`` model's settings).
+
+    Returns:
+        list ( dict ): list of Localizer predictions
     """
     from scoutbot.agg.py_cpu_nms import py_cpu_nms
 
     assert len(tile_grids) == len(loc_outputs)
 
+    if agg_thresh is None:
+        agg_thresh = CONFIGS[config]['thresh']
+    if nms_thresh is None:
+        nms_thresh = CONFIGS[config]['nms']
+
     log.info(f'Aggregating {len(tile_grids)} tiles onto {img_shape} canvas')
 
     if len(tile_grids) == 0:

scoutbot/loc/__init__.py

@@ -7,6 +7,7 @@ Localization ONNX model on this input, and finally how to convert this raw CNN
 output into usable detection bounding boxes with class labels and confidence
 scores.
 '''
+import os
 from os.path import exists, join
 from pathlib import Path
 
@@ -31,53 +32,90 @@ from scoutbot.loc.transforms import (
 
 PWD = Path(__file__).absolute().parent
 
-PHASE1 = True
-
-if PHASE1:
-    BATCH_SIZE = 16
-    INPUT_SIZE = (416, 416)
-    INPUT_SIZE_H, INPUT_SIZE_W = INPUT_SIZE
-    NETWORK_SIZE = (INPUT_SIZE_H, INPUT_SIZE_W, 3)
-
-    NUM_CLASSES = 1
-    ANCHORS = [
-        (1.3221, 1.73145),
-        (3.19275, 4.00944),
-        (5.05587, 8.09892),
-        (9.47112, 4.84053),
-        (11.2364, 10.0071),
-    ]
-    CLASS_LABEL_MAP = ['elephant_savanna']
-    LOC_THRESH = 0.4
-    NMS_THRESH = 0.8
-
-    ONNX_MODEL = 'scout.loc.5fbfff26.0.onnx'
-    ONNX_MODEL_PATH = join(PWD, 'models', 'onnx', ONNX_MODEL)
-    ONNX_MODEL_HASH = '85a9378311d42b5143f74570136f32f50bf97c548135921b178b46ba7612b216'
-else:
-    BATCH_SIZE = 16
-    INPUT_SIZE = (416, 416)
-    INPUT_SIZE_H, INPUT_SIZE_W = INPUT_SIZE
-    NETWORK_SIZE = (INPUT_SIZE_H, INPUT_SIZE_W, 3)
-
-    NUM_CLASSES = 1
-    ANCHORS = [
-        (1.3221, 1.73145),
-        (3.19275, 4.00944),
-        (5.05587, 8.09892),
-        (9.47112, 4.84053),
-        (11.2364, 10.0071),
-    ]
-    CLASS_LABEL_MAP = ['elephant_savanna']
-    LOC_THRESH = 0.4
-    NMS_THRESH = 0.8
-
-    ONNX_MODEL = 'scout.loc.5fbfff26.0.onnx'
-    ONNX_MODEL_PATH = join(PWD, 'models', 'onnx', ONNX_MODEL)
-    ONNX_MODEL_HASH = '85a9378311d42b5143f74570136f32f50bf97c548135921b178b46ba7612b216'
-
-
-def fetch(pull=False):
+INPUT_SIZE = (416, 416)
+INPUT_SIZE_H, INPUT_SIZE_W = INPUT_SIZE
+NETWORK_SIZE = (INPUT_SIZE_H, INPUT_SIZE_W, 3)
+
+DEFAULT_CONFIG = os.getenv('CONFIG', 'phase1').strip().lower()
+CONFIGS = {
+    'phase1': {
+        'batch': 16,
+        'name': 'scout.loc.5fbfff26.0.onnx',
+        'path': join(PWD, 'models', 'onnx', 'scout.loc.5fbfff26.0.onnx'),
+        'hash': '85a9378311d42b5143f74570136f32f50bf97c548135921b178b46ba7612b216',
+        'classes': ['elephant_savanna'],
+        'thresh': 0.4,
+        'nms': 0.8,
+        'anchors': [
+            (1.3221, 1.73145),
+            (3.19275, 4.00944),
+            (5.05587, 8.09892),
+            (9.47112, 4.84053),
+            (11.2364, 10.0071),
+        ],
+    },
+    'mvp': {
+        'batch': 32,
+        'name': 'scout.loc.mvp.0.onnx',
+        'path': join(PWD, 'models', 'onnx', 'scout.loc.mvp.0.onnx'),
+        'hash': 'AAA',
+        'classes': [
+            'buffalo',
+            'camel',
+            'canoe',
+            'car',
+            'cow',
+            'crocodile',
+            'dead_animalwhite_bones',
+            'deadbones',
+            'eland',
+            'elecarcass_old',
+            'elephant',
+            'gazelle_gr',
+            'gazelle_grants',
+            'gazelle_th',
+            'gazelle_thomsons',
+            'gerenuk',
+            'giant_forest_hog',
+            'giraffe',
+            'goat',
+            'hartebeest',
+            'hippo',
+            'impala',
+            'kob',
+            'kudu',
+            'motorcycle',
+            'oribi',
+            'oryx',
+            'ostrich',
+            'roof_grass',
+            'roof_mabati',
+            'sheep',
+            'test',
+            'topi',
+            'vehicle',
+            'warthog',
+            'waterbuck',
+            'white_bones',
+            'wildebeest',
+            'zebra',
+        ],
+        'thresh': 0.4,
+        'nms': 0.8,
+        'anchors': [
+            (1.3221, 1.73145),
+            (3.19275, 4.00944),
+            (5.05587, 8.09892),
+            (9.47112, 4.84053),
+            (11.2364, 10.0071),
+        ],
+    },
+}
+CONFIGS[None] = CONFIGS[DEFAULT_CONFIG]
+assert DEFAULT_CONFIG in CONFIGS
+
+
+def fetch(pull=False, config=DEFAULT_CONFIG):
     """
     Fetch the Localizer ONNX model file from a CDN if it does not exist locally.
 
@@ -85,8 +123,10 @@ def fetch(pull=False):
     file otherwise does not exists locally on disk.
 
     Args:
-        pull (bool, optional): If :obj:`True`, use a downloaded version stored in
-            the local system's cache.  Defaults to :obj:`False`.
+        pull (bool, optional): If :obj:`True`, force using the downloaded versions
+            stored in the local system's cache.  Defaults to :obj:`False`.
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
 
     Returns:
         str: local ONNX model file path.
@@ -94,21 +134,26 @@ def fetch(pull=False):
     Raises:
         AssertionError: If the model cannot be fetched.
     """
-    if not pull and exists(ONNX_MODEL_PATH):
-        onnx_model = ONNX_MODEL_PATH
+    model_name = CONFIGS[config]['name']
+    model_path = CONFIGS[config]['path']
+    model_hash = CONFIGS[config]['hash']
+
+    if not pull and exists(model_path):
+        onnx_model = model_path
     else:
         onnx_model = pooch.retrieve(
-            url=f'https://wildbookiarepository.azureedge.net/models/{ONNX_MODEL}',
-            known_hash=ONNX_MODEL_HASH,
+            url=f'https://wildbookiarepository.azureedge.net/models/{model_name}',
+            known_hash=model_hash,
             progressbar=True,
         )
         assert exists(onnx_model)
+
     log.info(f'LOC Model: {onnx_model}')
 
     return onnx_model
 
 
-def pre(inputs):
+def pre(inputs, config=DEFAULT_CONFIG):
     """
     Load a list of filepaths and return a corresponding list of the image
     data as a 4-D list of floats.  The image data is loaded from disk, transformed
@@ -119,22 +164,27 @@ def pre(inputs):
 
     Args:
         inputs (list(str)): list of tile image filepaths (relative or absolute)
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
 
     Returns:
-        generator ( tuple ( list ( list ( list ( list ( float ) ) ) ), list ( tuple ( int ) ) ) ):
+        generator ( np.ndarray<np.float32>, list ( tuple ( int ) ), int, str ):
             - generator ->
-            - - list of transformed image data.
-            - - list of each tile's original size.
+            - - list of transformed image data with shape ``(b, c, w, h)``
+            - - list of each tile's original size
+            - - trim index
+            - - model configuration
     """
     if len(inputs) == 0:
-        return []
+        return [], config
 
-    log.info(f'Preprocessing {len(inputs)} LOC inputs in batches of {BATCH_SIZE}')
+    batch_size = CONFIGS[config]['batch']
+    log.info(f'Preprocessing {len(inputs)} LOC inputs in batches of {batch_size}')
 
     transform = torchvision.transforms.ToTensor()
 
-    for filepaths in ut.ichunks(inputs, BATCH_SIZE):
-        data = np.zeros((BATCH_SIZE, 3, INPUT_SIZE_H, INPUT_SIZE_W), dtype=np.float32)
+    for filepaths in ut.ichunks(inputs, batch_size):
+        data = np.zeros((batch_size, 3, INPUT_SIZE_H, INPUT_SIZE_W), dtype=np.float32)
         sizes = []
         trim = len(filepaths)
 
@@ -150,10 +200,10 @@ def pre(inputs):
             data[index] = img
             sizes.append(size)
 
-        while len(sizes) < BATCH_SIZE:
+        while len(sizes) < batch_size:
             sizes.append((0, 0))
 
-        yield data, sizes, trim
+        yield data, sizes, trim, config
 
 
 def predict(gen):
@@ -165,26 +215,33 @@ def predict(gen):
             :meth:`scoutbot.loc.pre`
 
     Returns:
-        generator ( list ( list ( float ) ), list ( tuple ( int ) ) ) ):
+        generator ( np.ndarray<np.float32>, list ( tuple ( int ) ), str ):
             - generator ->
-            - - list of raw ONNX model outputs.
-            - - list of each tile's original size.
+            - - list of raw ONNX model outputs as shape ``(b, n)``
+            - - list of each tile's original size
+            - - model configuration
     """
-    onnx_model = fetch()
-
     log.info('Running LOC inference')
 
-    ort_session = ort.InferenceSession(
-        onnx_model, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
-    )
+    ort_sessions = {}
 
-    for chunk, sizes, trim in tqdm.tqdm(gen):
+    for chunk, sizes, trim, config in tqdm.tqdm(gen):
         assert len(chunk) == len(sizes)
 
         if len(chunk) == 0:
             preds = []
             sizes = []
         else:
+            ort_session = ort_sessions.get(config)
+            if ort_session is None:
+                onnx_model = fetch(config=config)
+
+                ort_session = ort.InferenceSession(
+                    onnx_model,
+                    providers=['CUDAExecutionProvider', 'CPUExecutionProvider'],
+                )
+                ort_sessions[config] = ort_session
+
             assert trim <= len(chunk)
 
             pred = ort_session.run(
@@ -196,10 +253,10 @@ def predict(gen):
             preds = preds[:trim]
             sizes = sizes[:trim]
 
-        yield preds, sizes
+        yield preds, sizes, config
 
 
-def post(gen, loc_thresh=LOC_THRESH, nms_thresh=NMS_THRESH):
+def post(gen, loc_thresh=None, nms_thresh=None):
     """
     Apply a post-processing normalization of the raw ONNX network outputs.
 
@@ -228,27 +285,40 @@ def post(gen, loc_thresh=LOC_THRESH, nms_thresh=NMS_THRESH):
     Args:
         gen (generator): generator of batches of raw ONNX model outputs and sizes,
             the return of :meth:`scoutbot.loc.predict`
+        loc_thresh (float or None, optional): the confidence threshold for the localizer's
+            predictions.  Defaults to None.  Defaults to :obj:`None`
+            (the ``phase1`` model).
+        nms_thresh (float or None, optional): the non-maximum suppression (NMS) threshold
+            for the localizer's predictions.  Defaults to :obj:`None`
+            (the ``phase1`` model).
 
     Returns:
         list ( list ( dict ) ): nested list of Localizer predictions
     """
     log.info('Postprocessing LOC outputs')
 
-    postprocess = Compose(
-        [
-            GetBoundingBoxes(NUM_CLASSES, ANCHORS, loc_thresh),
-            NonMaxSupression(nms_thresh),
-            TensorToBrambox(NETWORK_SIZE, CLASS_LABEL_MAP),
-        ]
-    )
-
     # Exhaust generator and format output
     outputs = []
-    for preds, sizes in gen:
+    for preds, sizes, config in gen:
         assert len(preds) == len(sizes)
         if len(preds) == 0:
             continue
 
+        anchors = CONFIGS[config]['anchors']
+        classes = CONFIGS[config]['classes']
+        if loc_thresh is None:
+            loc_thresh = CONFIGS[config]['thresh']
+        if nms_thresh is None:
+            nms_thresh = CONFIGS[config]['nms']
+
+        postprocess = Compose(
+            [
+                GetBoundingBoxes(len(classes), anchors, loc_thresh),
+                NonMaxSupression(nms_thresh),
+                TensorToBrambox(NETWORK_SIZE, classes),
+            ]
+        )
+
         preds = postprocess(torch.tensor(preds))
 
         for pred, size in zip(preds, sizes):

scoutbot/loc/convert.py

@@ -20,7 +20,7 @@ import vtool as vt
 import wbia
 
 WITH_GPU = False
-BATCH_SIZE = 16
+BATCH_SIZE = 32
 
 
 ibs = wbia.opendb(dbdir='/data/db')

scoutbot/scoutbot.py

@@ -21,11 +21,17 @@ def pipeline_filepath_validator(ctx, param, value):
 
 
 @click.command('fetch')
-def fetch():
+@click.option(
+    '--config',
+    help='Which ML models to use for inference',
+    default=None,
+    type=click.Choice(['phase1', 'mvp']),
+)
+def fetch(config):
     """
     Fetch the required machine learning ONNX models for the WIC and LOC
     """
-    scoutbot.fetch()
+    scoutbot.fetch(config=config)
 
 
 @click.command('pipeline')
@@ -35,6 +41,12 @@ def fetch():
     type=str,
     callback=pipeline_filepath_validator,
 )
+@click.option(
+    '--config',
+    help='Which ML models to use for inference',
+    default=None,
+    type=click.Choice(['phase1', 'mvp']),
+)
 @click.option(
     '--output',
     help='Path to output JSON (if unspecified, results are printed to screen)',
@@ -44,39 +56,47 @@ def fetch():
 @click.option(
     '--wic_thresh',
     help='Whole Image Classifier (WIC) confidence threshold',
-    default=int(wic.WIC_THRESH * 100),
+    default=int(wic.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--loc_thresh',
     help='Localizer (LOC) confidence threshold',
-    default=int(loc.LOC_THRESH * 100),
+    default=int(loc.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--loc_nms_thresh',
     help='Localizer (LOC) non-maximum suppression (NMS) threshold',
-    default=int(loc.NMS_THRESH * 100),
+    default=int(loc.CONFIGS[None]['nms'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--agg_thresh',
     help='Aggregation (AGG) confidence threshold',
-    default=int(agg.AGG_THRESH * 100),
+    default=int(agg.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--agg_nms_thresh',
     help='Aggregation (AGG) non-maximum suppression (NMS) threshold',
-    default=int(agg.NMS_THRESH * 100),
+    default=int(agg.CONFIGS[None]['nms'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 def pipeline(
-    filepath, output, wic_thresh, loc_thresh, loc_nms_thresh, agg_thresh, agg_nms_thresh
+    filepath,
+    config,
+    output,
+    wic_thresh,
+    loc_thresh,
+    loc_nms_thresh,
+    agg_thresh,
+    agg_nms_thresh,
 ):
     """
     Run the ScoutBot pipeline on an input image filepath
     """
+    config = config.strip().lower()
     wic_thresh /= 100.0
     loc_thresh /= 100.0
     loc_nms_thresh /= 100.0
@@ -85,6 +105,7 @@ def pipeline(
 
     wic_, detects = scoutbot.pipeline(
         filepath,
+        config=config,
         wic_thresh=wic_thresh,
         loc_thresh=loc_thresh,
         loc_nms_thresh=loc_nms_thresh,
@@ -113,6 +134,12 @@ def pipeline(
     nargs=-1,
     type=str,
 )
+@click.option(
+    '--config',
+    help='Which ML models to use for inference',
+    default=None,
+    type=click.Choice(['phase1', 'mvp']),
+)
 @click.option(
     '--output',
     help='Path to output JSON (if unspecified, results are printed to screen)',
@@ -122,39 +149,47 @@ def pipeline(
 @click.option(
     '--wic_thresh',
     help='Whole Image Classifier (WIC) confidence threshold',
-    default=int(wic.WIC_THRESH * 100),
+    default=int(wic.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--loc_thresh',
     help='Localizer (LOC) confidence threshold',
-    default=int(loc.LOC_THRESH * 100),
+    default=int(loc.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--loc_nms_thresh',
     help='Localizer (LOC) non-maximum suppression (NMS) threshold',
-    default=int(loc.NMS_THRESH * 100),
+    default=int(loc.CONFIGS[None]['nms'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--agg_thresh',
     help='Aggregation (AGG) confidence threshold',
-    default=int(agg.AGG_THRESH * 100),
+    default=int(agg.CONFIGS[None]['thresh'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 @click.option(
     '--agg_nms_thresh',
     help='Aggregation (AGG) non-maximum suppression (NMS) threshold',
-    default=int(agg.NMS_THRESH * 100),
+    default=int(agg.CONFIGS[None]['nms'] * 100),
     type=click.IntRange(0, 100, clamp=True),
 )
 def batch(
-    filepaths, output, wic_thresh, loc_thresh, loc_nms_thresh, agg_thresh, agg_nms_thresh
+    filepaths,
+    config,
+    output,
+    wic_thresh,
+    loc_thresh,
+    loc_nms_thresh,
+    agg_thresh,
+    agg_nms_thresh,
 ):
     """
     Run the ScoutBot pipeline in batch on a list of input image filepaths
     """
+    config = config.strip().lower()
     wic_thresh /= 100.0
     loc_thresh /= 100.0
     loc_nms_thresh /= 100.0
@@ -165,6 +200,7 @@ def batch(
 
     wic_list, detects_list = scoutbot.batch(
         filepaths,
+        config=config,
         wic_thresh=wic_thresh,
         loc_thresh=loc_thresh,
         loc_nms_thresh=loc_nms_thresh,
@@ -192,7 +228,7 @@ def batch(
 @click.command('example')
 def example():
     """
-    Run a test of the pipeline on an example image
+    Run a test of the pipeline on an example image with the Phase 1 models
     """
     scoutbot.example()
 

scoutbot/tile/__init__.py

@@ -147,11 +147,12 @@ def tile_grid(
 
     Args:
         shape (tuple): the image's shape as ``(h, w, c)`` or ``(h, w)``
-        size (tuple): the tile's shape as ``(w, h)``
-        overlap (int): The amount of pixel overlap between each tile, for both the x-axis
-            and the y-axis.
-        offset (int): The amount of pixel offset for the entire grid
-        borders (bool): If :obj:`True`, include a set of border-only tiles.  Defaults to :obj:`True`.
+        size (tuple, optional): the tile's shape as ``(w, h)``
+        overlap (int, optional): The amount of pixel overlap between each tile, for
+            both the x-axis and the y-axis.
+        offset (int, optional): The amount of pixel offset for the entire grid
+        borders (bool, optional): If :obj:`True`, include a set of border-only tiles.
+            Defaults to :obj:`True`.
 
     Returns:
         list ( dict ): a list of grid coordinate dictionaries

scoutbot/wic/__init__.py

@@ -6,6 +6,7 @@ how to load an image and prepare it for inference, demonstrates how to run the
 WIC ONNX model on this input, and finally how to convert this raw CNN output
 into usable confidence scores.
 '''
+import os
 from os.path import exists, join
 from pathlib import Path
 
@@ -14,7 +15,6 @@ import onnxruntime as ort
 import pooch
 import torch
 import tqdm
-import utool as ut
 
 from scoutbot import log
 from scoutbot.wic.dataloader import (  # NOQA
@@ -26,24 +26,29 @@ from scoutbot.wic.dataloader import (  # NOQA
 
 PWD = Path(__file__).absolute().parent
 
-PHASE1 = True
 
-
-if PHASE1:
-    ONNX_MODEL = 'scout.wic.5fbfff26.3.0.onnx'
-    ONNX_MODEL_PATH = join(PWD, 'models', 'onnx', ONNX_MODEL)
-    ONNX_MODEL_HASH = 'cbc7f381fa58504e03b6510245b6b2742d63049429337465d95663a6468df4c1'
-    ONNX_CLASSES = ['negative', 'positive']
-    WIC_THRESH = 0.2
-else:
-    ONNX_MODEL = 'scout.wic.5fbfff26.3.0.onnx'
-    ONNX_MODEL_PATH = join(PWD, 'models', 'onnx', ONNX_MODEL)
-    ONNX_MODEL_HASH = 'cbc7f381fa58504e03b6510245b6b2742d63049429337465d95663a6468df4c1'
-    ONNX_CLASSES = ['negative', 'positive']
-    WIC_THRESH = 0.2
-
-
-def fetch(pull=False):
+DEFAULT_CONFIG = os.getenv('CONFIG', 'phase1').strip().lower()
+CONFIGS = {
+    'phase1': {
+        'name': 'scout.wic.5fbfff26.3.0.onnx',
+        'path': join(PWD, 'models', 'onnx', 'scout.wic.5fbfff26.3.0.onnx'),
+        'hash': 'cbc7f381fa58504e03b6510245b6b2742d63049429337465d95663a6468df4c1',
+        'classes': ['negative', 'positive'],
+        'thresh': 0.2,
+    },
+    'mvp': {
+        'name': 'scout.wic.mvp.2.0.onnx',
+        'path': join(PWD, 'models', 'onnx', 'scout.wic.mvp.2.0.onnx'),
+        'hash': '3ff3a192803e53758af5e112526ba9622f1dedc55e2fa88850db6f32af160f32',
+        'classes': ['negative', 'positive'],
+        'thresh': 0.07,
+    },
+}
+CONFIGS[None] = CONFIGS[DEFAULT_CONFIG]
+assert DEFAULT_CONFIG in CONFIGS
+
+
+def fetch(pull=False, config=DEFAULT_CONFIG):
     """
     Fetch the WIC ONNX model file from a CDN if it does not exist locally.
 
@@ -51,8 +56,10 @@ def fetch(pull=False):
     file otherwise does not exists locally on disk.
 
     Args:
-        pull (bool, optional): If :obj:`True`, use a downloaded version stored in
-            sthe local system's cache.  Defaults to :obj:`False`.
+        pull (bool, optional): If :obj:`True`, force using the downloaded versions
+            stored in the local system's cache.  Defaults to :obj:`False`.
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
 
     Returns:
         str: local ONNX model file path.
@@ -60,12 +67,16 @@ def fetch(pull=False):
     Raises:
         AssertionError: If the model cannot be fetched.
     """
-    if not pull and exists(ONNX_MODEL_PATH):
-        onnx_model = ONNX_MODEL_PATH
+    model_name = CONFIGS[config]['name']
+    model_path = CONFIGS[config]['path']
+    model_hash = CONFIGS[config]['hash']
+
+    if not pull and exists(model_path):
+        onnx_model = model_path
     else:
         onnx_model = pooch.retrieve(
-            url=f'https://wildbookiarepository.azureedge.net/models/{ONNX_MODEL}',
-            known_hash=ONNX_MODEL_HASH,
+            url=f'https://wildbookiarepository.azureedge.net/models/{model_name}',
+            known_hash=model_hash,
             progressbar=True,
         )
         assert exists(onnx_model)
@@ -75,7 +86,7 @@ def fetch(pull=False):
     return onnx_model
 
 
-def pre(inputs, batch_size=BATCH_SIZE):
+def pre(inputs, batch_size=BATCH_SIZE, config=DEFAULT_CONFIG):
     """
     Load a list of filepaths and return a corresponding list of the image
     data as a 4-D list of floats.  The image data is loaded from disk, transformed
@@ -86,13 +97,19 @@ def pre(inputs, batch_size=BATCH_SIZE):
 
     Args:
         inputs (list(str)): list of tile image filepaths (relative or absolute)
+        batch_size (int, optional): the maximum number of images to load in a
+            single batch.  Defaults to the environment variable ``WIC_BATCH_SIZE``.
+        config (str or None, optional): the configuration to use, one of ``phase1``
+            or ``mvp``.  Defaults to :obj:`None` (the ``phase1`` model).
 
     Returns:
-        generator ( list ( list ( list ( list ( float ) ) ) ) ) : generator ->
-        list of transformed image data
+        generator ( np.ndarray<np.float32>, str ):
+            - generator ->
+            - - list of transformed image data with shape ``(b, c, w, h)``
+            - - model configuration
     """
     if len(inputs) == 0:
-        return []
+        return [], config
 
     log.info(f'Preprocessing {len(inputs)} WIC inputs in batches of {batch_size}')
 
@@ -103,7 +120,7 @@ def pre(inputs, batch_size=BATCH_SIZE):
     )
 
     for (data,) in dataloader:
-        yield data.numpy().astype(np.float32)
+        yield data.numpy().astype(np.float32), config
 
 
 def predict(gen):
@@ -115,18 +132,26 @@ def predict(gen):
             return of :meth:`scoutbot.wic.pre`
 
     Returns:
-        generator ( list ( list ( float ) ) ): generator -> list of raw ONNX
-        model outputs
+        generator ( np.ndarray<np.float32>, str ):
+            - generator ->
+            - - list of raw ONNX model outputs as shape ``(b, n)``
+            - - model configuration
     """
-    onnx_model = fetch()
-
     log.info('Running WIC inference')
 
-    ort_session = ort.InferenceSession(
-        onnx_model, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
-    )
+    ort_sessions = {}
+
+    for chunk, config in tqdm.tqdm(gen):
+
+        ort_session = ort_sessions.get(config)
+        if ort_session is None:
+            onnx_model = fetch(config=config)
+
+            ort_session = ort.InferenceSession(
+                onnx_model, providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
+            )
+            ort_sessions[config] = ort_session
 
-    for chunk in tqdm.tqdm(gen):
         if len(chunk) == 0:
             preds = []
         else:
@@ -135,7 +160,7 @@ def predict(gen):
                 {'input': chunk},
             )
             preds = pred[0]
-        yield preds
+        yield preds, config
 
 
 def post(gen):
@@ -155,5 +180,11 @@ def post(gen):
     # Exhaust generator and format output
     log.info('Postprocessing WIC outputs')
 
-    outputs = [dict(zip(ONNX_CLASSES, pred.tolist())) for pred in ut.flatten(gen)]
+    outputs = []
+    for preds, config in gen:
+        classes = CONFIGS[config]['classes']
+        for pred in preds:
+            output = dict(zip(classes, pred.tolist()))
+            outputs.append(output)
+
     return outputs

scoutbot/wic/convert.mvp.py

@@ -0,0 +1,276 @@
+# -*- coding: utf-8 -*-
+"""
+
+pip install torch torchvision onnx onnxruntime-gpu tqdm wbia-utool scikit-learn numpy
+
+"""
+import random
+import time
+from collections import OrderedDict
+from os.path import exists, join, split, splitext
+
+import numpy as np
+import onnx
+import onnxruntime as ort
+import sklearn
+import torch
+import torch.nn as nn
+import torchvision
+import tqdm
+import utool as ut
+import wbia
+from wbia.algo.detect.densenet import INPUT_SIZE, ImageFilePathList, _init_transforms
+
+WITH_GPU = False
+BATCH_SIZE = 128
+
+
+ibs = wbia.opendb(dbdir='/data/db')
+
+
+pkl_path = 'scout.pkl'
+if not exists(pkl_path):
+    if False:
+        tids = ibs.get_valid_gids(is_tile=True)
+    else:
+        imageset_text_list = ['TEST_SET']
+        imageset_rowid_list = ibs.get_imageset_imgsetids_from_text(imageset_text_list)
+        gids_list = ibs.get_imageset_gids(imageset_rowid_list)
+        gids = ut.flatten(gids_list)
+        flags = ibs.get_tile_flags(gids)
+        test_gids = ut.filterfalse_items(gids, flags)
+        assert sum(ibs.get_tile_flags(test_gids)) == 0
+        tids = ibs.scout_get_valid_tile_rowids(gid_list=test_gids)
+
+    random.shuffle(tids)
+    positive, negative = [], []
+    for chunk_tids in tqdm.tqdm(ut.ichunks(tids, 1000)):
+        _, _, chunk_flags = ibs.scout_tile_positive_cumulative_area(chunk_tids)
+        chunk_filepaths = ibs.get_image_paths(chunk_tids)
+        for index, (tid, flag, filepath) in enumerate(
+            zip(chunk_tids, chunk_flags, chunk_filepaths)
+        ):
+            if not exists(filepath):
+                continue
+            if flag:
+                positive.append(tid)
+            else:
+                negative.append(tid)
+        if len(positive) >= 100 and len(negative) >= 100:
+            break
+        print(len(positive), len(negative))
+
+    random.shuffle(positive)
+    random.shuffle(negative)
+    positive = positive[:100]
+    negative = negative[:100]
+    data = positive + negative
+    filepaths = ibs.get_image_paths(data)
+    labels = [True] * len(positive) + [False] * len(negative)
+    ut.save_cPkl(pkl_path, (data, labels))
+
+    OUTPUT_PATH = '/data/db/checks'
+    ut.delete(OUTPUT_PATH)
+    ut.ensuredir(OUTPUT_PATH)
+    for filepath, label in zip(filepaths, labels):
+        path, filename = split(filepath)
+        name, ext = splitext(filename)
+        tag = 'true' if label else 'false'
+        filename_ = f'{name}.{tag}{ext}'
+        filepath_ = join(OUTPUT_PATH, filename_)
+        if not exists(filepath_):
+            ut.copy(filepath, filepath_)
+
+assert exists(pkl_path)
+data, labels = ut.load_cPkl(pkl_path)
+
+filepaths = ibs.get_image_paths(data)
+
+assert len(data) == len(set(data))
+assert set(ibs.get_image_sizes(data)) == {(256, 256)}
+assert sum(map(exists, filepaths)) == len(filepaths)
+
+##########
+
+INDEX = 0
+
+weights_path = f'/cache/wbia/classifier2.scout.mvp.2/classifier.{INDEX}.weights'
+
+assert exists(weights_path)
+weights = torch.load(weights_path, map_location='cpu')
+state = weights['state']
+classes = weights['classes']
+
+# Initialize the model for this run
+model = torchvision.models.resnet50()
+num_ftrs = model.fc.in_features
+model.fc = nn.Linear(num_ftrs, len(classes))
+
+# Convert any weights to non-parallel version
+new_state = OrderedDict()
+for k, v in state.items():
+    k = k.replace('module.', '')
+    new_state[k] = v
+
+# Load state without parallel
+model.load_state_dict(new_state)
+
+# Add softmax
+model.fc = nn.Sequential(model.fc, nn.LogSoftmax(), nn.Softmax())
+if WITH_GPU:
+    model = model.cuda()
+model.eval()
+
+#############
+
+transforms = _init_transforms()
+transform = transforms['test']
+dataset = ImageFilePathList(filepaths, labels, transform=transform)
+dataloader = torch.utils.data.DataLoader(
+    dataset, batch_size=BATCH_SIZE, num_workers=0, pin_memory=False
+)
+
+time_pytorch = 0.0
+inputs = []
+outputs = []
+targets = []
+for (inputs_, targets_) in tqdm.tqdm(dataloader, desc='test'):
+    if WITH_GPU:
+        inputs_ = inputs_.cuda()
+
+    time_start = time.time()
+    with torch.set_grad_enabled(False):
+        output_ = model(inputs_)
+    time_end = time.time()
+    time_pytorch += time_end - time_start
+
+    inputs += inputs_.tolist()
+    outputs += output_.tolist()
+    targets += targets_.tolist()
+
+inputs = np.array(inputs, dtype=np.float32)
+globals().update(locals())
+predictions_pytorch = [dict(zip(classes, output)) for output in outputs]
+
+#############
+
+threshs = list(np.arange(0.0, 1.01, 0.01))
+best_thresh = None
+best_accuracy = 0.0
+best_confusion = None
+for thresh in tqdm.tqdm(threshs):
+    globals().update(locals())
+    values = [prediction['positive'] >= thresh for prediction in predictions_pytorch]
+    accuracy = sklearn.metrics.accuracy_score(targets, values)
+    confusion = sklearn.metrics.confusion_matrix(targets, values)
+    if accuracy > best_accuracy:
+        best_thresh = thresh
+        best_accuracy = accuracy
+        best_confusion = confusion
+
+tn, fp, fn, tp = best_confusion.ravel()
+print(f'Thresh:    {best_thresh}')
+print(f'Accuracy:  {best_accuracy}')
+print(f'TP:        {tp}')
+print(f'TN:        {tn}')
+print(f'FP:        {fp}')
+print(f'FN:        {fn}')
+
+# Thresh:    0.17                                                                                                                                                                       │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# cd ^C
+# Accuracy:  0.885                                                                                                                                                                      │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# cd classifier.0.weights
+# TP:        83                                                                                                                                                                         │bash: cd: classifier.0.weights: Not a directory
+# TN:        94                                                                                                                                                                         │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# ls
+# FP:        6                                                                                                                                                                          │classifier.0.weights
+# FN:        17
+
+#############
+
+dummy_input = torch.randn(BATCH_SIZE, 3, INPUT_SIZE, INPUT_SIZE, device='cpu')
+input_names = ['input']
+output_names = ['output']
+
+onnx_filename = f'scout.wic.mvp.2.{INDEX}.onnx'
+output = torch.onnx.export(
+    model,
+    dummy_input,
+    onnx_filename,
+    verbose=True,
+    input_names=input_names,
+    output_names=output_names,
+    dynamic_axes={
+        'input': {0: 'batch_size'},  # variable length axes
+        'output': {0: 'batch_size'},
+    },
+)
+
+###########
+
+model = onnx.load(onnx_filename)
+onnx.checker.check_model(model)
+print(onnx.helper.printable_graph(model.graph))
+
+###########
+
+ort_session = ort.InferenceSession(onnx_filename, providers=['CPUExecutionProvider'])
+
+time_onnx = 0.0
+outputs = []
+for chunk in ut.ichunks(inputs, BATCH_SIZE):
+    trim = len(chunk)
+    while (len(chunk)) < BATCH_SIZE:
+        chunk.append(np.random.randn(3, INPUT_SIZE, INPUT_SIZE).astype(np.float32))
+    input_ = np.array(chunk, dtype=np.float32)
+
+    time_start = time.time()
+    output_ = ort_session.run(
+        None,
+        {'input': input_},
+    )
+    time_end = time.time()
+    time_onnx += time_end - time_start
+
+    outputs += output_[0].tolist()[:trim]
+
+predictions_onnx = [dict(zip(classes, output)) for output in outputs]
+
+###########
+
+values_pytorch = [
+    prediction_pytorch['positive'] for prediction_pytorch in predictions_pytorch
+]
+values_onnx = [prediction_onnx['positive'] for prediction_onnx in predictions_onnx]
+deviations = [
+    abs(value_pytorch - value_onnx)
+    for value_pytorch, value_onnx in zip(values_pytorch, values_onnx)
+]
+
+print(f'Min:  {np.min(deviations):0.08f}')
+print(f'Max:  {np.max(deviations):0.08f}')
+print(f'Mean: {np.mean(deviations):0.08f} +/- {np.std(deviations):0.08f}')
+print(f'Time Pytorch: {time_pytorch:0.02f} sec.')
+print(f'Time ONNX:    {time_onnx:0.02f} sec.')
+
+globals().update(locals())
+values = [prediction['positive'] >= best_thresh for prediction in predictions_onnx]
+accuracy = sklearn.metrics.accuracy_score(targets, values)
+confusion = sklearn.metrics.confusion_matrix(targets, values)
+tn, fp, fn, tp = best_confusion.ravel()
+
+print(f'Thresh:    {best_thresh}')
+print(f'Accuracy:  {best_accuracy}')
+print(f'TP:        {tp}')
+print(f'TN:        {tn}')
+print(f'FP:        {fp}')
+print(f'FN:        {fn}')
+
+# Min:  0.00000000                                                                                                                                                                      │labeler.fins.v1.1.zip                 labeler.lynx.v3             labeler.spotted_eagle_ray.v0.zip.md5  vsone.zebra_mountain.match_state.RF.131.lciwhwikfycthvva.cPkl.meta.json
+# Max:  0.00000215                                                                                                                                                                      │labeler.fins.v1.1.zip.md5             labeler.lynx.v3.zip         labeler.wild_dog.v1
+# Mean: 0.00000010 +/- 0.00000031                                                                                                                                                       │root@25a43ccd71e0:/cache/wbia# cd classifier2.scout.mvp.2
+# Time Pytorch: 6.34 sec.                                                                                                                                                               │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# ls
+# Time ONNX:    1.33 sec.                                                                                                                                                               │classifier.0.weights
+# Thresh:    0.17                                                                                                                                                                       │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# cd ^C
+# Accuracy:  0.885                                                                                                                                                                      │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# cd classifier.0.weights
+# TP:        83                                                                                                                                                                         │bash: cd: classifier.0.weights: Not a directory
+# TN:        94                                                                                                                                                                         │root@25a43ccd71e0:/cache/wbia/classifier2.scout.mvp.2# ls
+# FP:        6                                                                                                                                                                          │classifier.0.weights
+# FN:        17

scoutbot/wic/dataloader.py

@@ -20,7 +20,7 @@ class ImageFilePathList(torch.utils.data.Dataset):
         args = (filepaths, targets) if self.targets else (filepaths,)
         self.samples = list(zip(*args))
 
-        if self.targets:
+        if self.targets:  # nocov
             self.classes = sorted(set(ut.take_column(self.samples, 1)))
             self.class_to_idx = {self.classes[i]: i for i in range(len(self.classes))}
         else:
@@ -60,19 +60,6 @@ class ImageFilePathList(torch.utils.data.Dataset):
     def __len__(self):
         return len(self.samples)
 
-    def __repr__(self):
-        fmt_str = 'Dataset ' + self.__class__.__name__ + '\n'
-        fmt_str += '    Number of samples: {}\n'.format(self.__len__())
-        tmp = '    Transforms (if any): '
-        fmt_str += '{}{}\n'.format(
-            tmp, self.transform.__repr__().replace('\n', '\n' + ' ' * len(tmp))
-        )
-        tmp = '    Target Transforms (if any): '
-        fmt_str += '{}{}'.format(
-            tmp, self.target_transform.__repr__().replace('\n', '\n' + ' ' * len(tmp))
-        )
-        return fmt_str
-
 
 class Augmentations(object):
     def __call__(self, img):

scoutbot/wic/models/onnx/scout.wic.mvp.2.0.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ff3a192803e53758af5e112526ba9622f1dedc55e2fa88850db6f32af160f32
+size 94359210

scoutbot/wic/models/pytorch/classifier2.scout.mvp.2/classifier.0.weights

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf8634426ac451acfbf4211eaf80f880c3c3220380883c62e1d5dff429c85032
+size 94369625

setup.cfg

@@ -19,6 +19,8 @@ platforms = any
 include_package_data = True
 install_requires =
     click
+    codecov
+    coverage
     cryptography
     gradio
     imgaug

tests/conftest.py

@@ -1,33 +0,0 @@
-# -*- coding: utf-8 -*-
-import logging
-
-log = logging.getLogger('pytest.conftest')  # pylint: disable=invalid-name
-
-
-# @pytest.fixture()
-# def cfg(config):
-#     from scoutbot import utils
-
-#     log = utils.init_logging()
-#     cfg = utils.init_config(config, log)
-
-#     cfg['output'] = 'scoutbot/{}'.format(cfg['output'])
-
-#     return cfg
-
-
-# @pytest.fixture()
-# def device(cfg):
-#     device = cfg.get('device')
-
-#     return device
-
-
-# @pytest.fixture()
-# def net(cfg):
-#     from scoutbot import model
-
-#     net, _, _ = model.load(cfg)
-#     net.eval()
-
-#     return net

tests/test_agg.py

@@ -6,7 +6,7 @@ import utool as ut
 from scoutbot import agg, loc, tile, wic
 
 
-def test_agg_compute():
+def test_agg_compute_phase1():
     img_filepath = abspath(join('examples', '1be4d40a-6fd0-42ce-da6c-294e45781f41.jpg'))
 
     # Run tiling
@@ -14,31 +14,24 @@ def test_agg_compute():
     assert len(tile_filepaths) == 1252
 
     # Run WIC
-    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths)))
+    wic_outputs = wic.post(wic.predict(wic.pre(tile_filepaths, config='phase1')))
     assert len(wic_outputs) == len(tile_filepaths)
 
     # Threshold for WIC
-    flags = [wic_output.get('positive') >= wic.WIC_THRESH for wic_output in wic_outputs]
+    flags = [
+        wic_output.get('positive') >= wic.CONFIGS[None]['thresh']
+        for wic_output in wic_outputs
+    ]
     loc_tile_grids = ut.compress(tile_grids, flags)
     loc_tile_filepaths = ut.compress(tile_filepaths, flags)
     assert sum(flags) == 15
 
     # Run localizer
-    loc_outputs = loc.post(
-        loc.predict(loc.pre(loc_tile_filepaths)),
-        loc_thresh=loc.LOC_THRESH,
-        nms_thresh=loc.NMS_THRESH,
-    )
+    loc_outputs = loc.post(loc.predict(loc.pre(loc_tile_filepaths, config='phase1')))
     assert len(loc_tile_grids) == len(loc_outputs)
 
     # Aggregate
-    detects = agg.compute(
-        img_shape,
-        loc_tile_grids,
-        loc_outputs,
-        agg_thresh=agg.AGG_THRESH,
-        nms_thresh=agg.NMS_THRESH,
-    )
+    detects = agg.compute(img_shape, loc_tile_grids, loc_outputs, config='phase1')
 
     assert len(detects) == 3
 

tests/test_loc.py

@@ -4,10 +4,10 @@ from os.path import abspath, exists, join
 import onnx
 
 
-def test_loc_onnx_load():
+def test_loc_onnx_load_phase1():
     from scoutbot.loc import fetch
 
-    onnx_model = fetch()
+    onnx_model = fetch(config='phase1')
     model = onnx.load(onnx_model)
     assert exists(onnx_model)
 
@@ -17,8 +17,8 @@ def test_loc_onnx_load():
     assert graph.count('\n') == 107
 
 
-def test_loc_onnx_pipeline():
-    from scoutbot.loc import BATCH_SIZE, INPUT_SIZE, post, pre, predict
+def test_loc_onnx_pipeline_phase1():
+    from scoutbot.loc import CONFIGS, INPUT_SIZE, post, pre, predict
 
     inputs = [
         abspath(join('examples', '0d01a14e-311d-e153-356f-8431b6996b84.true.jpg')),
@@ -26,23 +26,26 @@ def test_loc_onnx_pipeline():
 
     assert exists(inputs[0])
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
+    batch_size = CONFIGS[None]['batch']
 
-    temp, sizes, trim = next(data)
-    assert temp.shape == (BATCH_SIZE, 3, INPUT_SIZE[0], INPUT_SIZE[1])
+    temp, sizes, trim, config = next(data)
+    assert temp.shape == (batch_size, 3, INPUT_SIZE[0], INPUT_SIZE[1])
     assert len(temp) == len(sizes)
     assert sizes[0] == (256, 256)
     assert set(sizes[1:]) == {(0, 0)}
+    assert config == 'phase1'
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
     preds = predict(data)
 
-    temp, sizes = next(preds)
+    temp, sizes, config = next(preds)
     assert temp.shape == (1, 30, 13, 13)
     assert len(temp) == len(sizes)
     assert sizes == [(256, 256)]
+    assert config == 'phase1'
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
     preds = predict(data)
     outputs = post(preds)
 
@@ -103,6 +106,10 @@ def test_loc_onnx_pipeline():
             else:
                 assert abs(output.get(key) - target.get(key)) < 3
 
+
+def test_loc_onnx_pipeline_empty():
+    from scoutbot.loc import post, pre, predict
+
     data = pre([])
     preds = predict(data)
     outputs = post(preds)

tests/test_scoutbot.py

@@ -8,11 +8,19 @@ def test_fetch():
     scoutbot.fetch(pull=False)
     scoutbot.fetch(pull=True)
 
+    scoutbot.fetch(pull=False, config='phase1')
+    scoutbot.fetch(pull=True, config='phase1')
 
-def test_pipeline():
+    scoutbot.fetch(pull=False, config='mvp')
+    scoutbot.fetch(pull=True, config='mvp')
+
+
+def test_pipeline_phase1():
     img_filepath = abspath(join('examples', '1be4d40a-6fd0-42ce-da6c-294e45781f41.jpg'))
 
-    wic_, detects = scoutbot.pipeline(img_filepath)
+    wic_, detects = scoutbot.pipeline(img_filepath, config='phase1')
+
+    assert abs(wic_ - 1.0) < 1e-2
     assert len(detects) == 3
 
     targets = [
@@ -29,3 +37,37 @@ def test_pipeline():
                 assert abs(output.get(key) - target.get(key)) < 1e-2
             else:
                 assert abs(output.get(key) - target.get(key)) < 3
+
+
+def test_batch_phase1():
+    img_filepath = abspath(join('examples', '1be4d40a-6fd0-42ce-da6c-294e45781f41.jpg'))
+
+    img_filepaths = [img_filepath]
+    wic_list, detects_list = scoutbot.batch(img_filepaths, config='phase1')
+    assert len(wic_list) == 1
+    assert len(detects_list) == 1
+
+    wic_ = wic_list[0]
+    detects = detects_list[0]
+
+    assert abs(wic_ - 1.0) < 1e-2
+    assert len(detects) == 3
+
+    targets = [
+        {'l': 'elephant_savanna', 'c': 0.9299, 'x': 4597, 'y': 2322, 'w': 72, 'h': 149},
+        {'l': 'elephant_savanna', 'c': 0.8739, 'x': 4865, 'y': 2422, 'w': 97, 'h': 109},
+        {'l': 'elephant_savanna', 'c': 0.7115, 'x': 4806, 'y': 2476, 'w': 66, 'h': 119},
+    ]
+
+    for output, target in zip(detects, targets):
+        for key in target.keys():
+            if key == 'l':
+                assert output.get(key) == target.get(key)
+            elif key == 'c':
+                assert abs(output.get(key) - target.get(key)) < 1e-2
+            else:
+                assert abs(output.get(key) - target.get(key)) < 3
+
+
+def test_example():
+    scoutbot.example()

tests/test_wic.py

@@ -4,10 +4,10 @@ from os.path import abspath, exists, join
 import onnx
 
 
-def test_wic_onnx_load():
+def test_wic_onnx_load_phase1():
     from scoutbot.wic import fetch
 
-    onnx_model = fetch()
+    onnx_model = fetch(config='phase1')
     model = onnx.load(onnx_model)
     assert exists(onnx_model)
 
@@ -17,8 +17,21 @@ def test_wic_onnx_load():
     assert graph.count('\n') == 1334
 
 
-def test_wic_onnx_pipeline():
-    from scoutbot.wic import INPUT_SIZE, ONNX_CLASSES, post, pre, predict
+def test_wic_onnx_load_mvp():
+    from scoutbot.wic import fetch
+
+    onnx_model = fetch(config='mvp')
+    model = onnx.load(onnx_model)
+    assert exists(onnx_model)
+
+    onnx.checker.check_model(model)
+
+    graph = onnx.helper.printable_graph(model.graph)
+    assert graph.count('\n') == 237
+
+
+def test_wic_onnx_pipeline_phase1():
+    from scoutbot.wic import CONFIGS, INPUT_SIZE, post, pre, predict
 
     inputs = [
         abspath(join('examples', '1e8372e4-357d-26e6-d7fd-0e0ae402463a.true.jpg')),
@@ -26,33 +39,80 @@ def test_wic_onnx_pipeline():
 
     assert exists(inputs[0])
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
 
-    temp = next(data)
+    temp, config = next(data)
     assert temp.shape == (1, 3, INPUT_SIZE, INPUT_SIZE)
+    assert config == 'phase1'
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
     preds = predict(data)
 
-    temp = next(preds)
+    temp, config = next(preds)
     assert temp.shape == (1, 2)
     assert temp[0][1] > temp[0][0]
     assert abs(temp[0][0] - 0.00001503) < 1e-4
     assert abs(temp[0][1] - 0.99998497) < 1e-4
+    assert config == 'phase1'
 
-    data = pre(inputs)
+    data = pre(inputs, config='phase1')
     preds = predict(data)
     outputs = post(preds)
 
     assert len(outputs) == 1
     output = outputs[0]
-    assert output.keys() == set(ONNX_CLASSES)
+    classes = CONFIGS[None]['classes']
+    assert output.keys() == set(classes)
     assert output['positive'] > output['negative']
     assert abs(output['negative'] - 0.00001503) < 1e-4
     assert abs(output['positive'] - 0.99998497) < 1e-4
     assert isinstance(output['negative'], float)
     assert isinstance(output['positive'], float)
 
+
+def test_wic_onnx_pipeline_mvp():
+    from scoutbot.wic import CONFIGS, INPUT_SIZE, post, pre, predict
+
+    inputs = [
+        abspath(join('examples', '1e8372e4-357d-26e6-d7fd-0e0ae402463a.true.jpg')),
+    ]
+
+    assert exists(inputs[0])
+
+    data = pre(inputs, config='mvp')
+
+    temp, config = next(data)
+    assert temp.shape == (1, 3, INPUT_SIZE, INPUT_SIZE)
+    assert config == 'mvp'
+
+    data = pre(inputs, config='mvp')
+    preds = predict(data)
+
+    temp, config = next(preds)
+    assert temp.shape == (1, 2)
+    assert temp[0][1] > temp[0][0]
+    assert abs(temp[0][0] - 0.00000000) < 1e-4
+    assert abs(temp[0][1] - 1.00000000) < 1e-4
+    assert config == 'mvp'
+
+    data = pre(inputs, config='mvp')
+    preds = predict(data)
+    outputs = post(preds)
+
+    assert len(outputs) == 1
+    output = outputs[0]
+    classes = CONFIGS[None]['classes']
+    assert output.keys() == set(classes)
+    assert output['positive'] > output['negative']
+    assert abs(output['negative'] - 0.00000000) < 1e-4
+    assert abs(output['positive'] - 1.00000000) < 1e-4
+    assert isinstance(output['negative'], float)
+    assert isinstance(output['positive'], float)
+
+
+def test_wic_onnx_pipeline_empty():
+    from scoutbot.wic import post, pre, predict
+
     data = pre([])
     preds = predict(data)
     outputs = post(preds)