Added preliminary WIC and Localizer models

.gitignore

@@ -7,5 +7,6 @@ output.*.jpg
 .coverage
 coverage/
 
+gradio_cached_examples/
 __pycache__/
 docs/build/

Dockerfile

@@ -0,0 +1,14 @@
+FROM continuumio/anaconda3:latest
+
+ENV GRADIO_SERVER_NAME=0.0.0.0
+
+ENV GRADIO_SERVER_PORT=7860
+
+WORKDIR /code
+
+COPY ./ /code
+
+RUN conda install pip \
+ && pip install --no-cache-dir -r requirements.txt
+
+CMD python app.py

README.md

@@ -1,5 +1,5 @@
 ---
-title: Wild Me Scout
+title: Wild Me ScoutBot
 metaTitle: "The computer vision for Wild Me's Scout project"
 emoji: 🌎
 colorFrom: blue
@@ -9,137 +9,4 @@ sdk_version: 3.1.4
 app_file: app.py
 pinned: true
 python_version: 3.10.5
----
-
-
-Wild Me Scout
-=============
-
-[![GitHub CI](https://github.com/WildMeOrg/scoutbot/actions/workflows/testing.yml/badge.svg?branch=main)](https://github.com/WildMeOrg/scoutbot/actions/workflows/testing.yml)
-[![Python Wheel](https://github.com/WildMeOrg/scoutbot/actions/workflows/python-publish.yml/badge.svg)](https://github.com/WildMeOrg/scoutbot/actions/workflows/python-publish.yml)
-[![ReadTheDocs](https://readthedocs.org/projects/scoutbot/badge/?version=latest)](https://scoutbot.readthedocs.io/en/latest/?badge=latest)
-[![Huggingface](https://img.shields.io/badge/HuggingFace-Running-yellow)](https://huggingface.co/spaces/WildMeOrg/scoutbot)
-
-::: {.contents backlinks="none"}
-Quick Links
-:::
-
-::: {.sectnum}
-:::
-
-How to Install
---------------
-
-You need to first install Anaconda on your machine. Below are the
-instructions on how to install Anaconda on an Apple macOS machine, but
-it is possible to install on a Windows and Linux machine as well.
-Consult the [official Anaconda page](https://www.anaconda.com) to
-download and install on other systems. For Windows computers, it is
-highly recommended that you intall the [Windows Subsystem for
-Linux](https://docs.microsoft.com/en-us/windows/wsl/install).
-
-``` {.bash}
-# Install Homebrew
-/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
-
-# Install Anaconda and expose conda to the terminal
-brew install anaconda
-export PATH="/opt/homebrew/anaconda3/bin:$PATH"
-conda init zsh
-conda update conda
-```
-
-Once Anaconda is installed, you will need an environment and the
-following packages installed
-
-``` {.bash}
-# Create Environment
-conda create --name scout
-conda activate scout
-
-# Install Python dependencies
-conda install pip
-
-conda install -r requirements.txt
-conda install pytorch torchvision -c pytorch-nightly
-```
-
-How to Run
-----------
-
-It is recommended to use [ipython]{.title-ref} and to copy sections of code
-into and inspecting the
-
-``` {.bash}
-# Run the training script
-cd scoutbot/
-python train.py
-
-# Run the live demo
-python app.py
-```
-
-Unit Tests
-----------
-
-You can run the automated tests in the [tests/]{.title-ref} folder by
-running [pytest]{.title-ref}. This will give an output of which tests
-have failed. You may also get a coverage percentage by running [coverage
-html]{.title-ref} and loading the [coverage/html/index.html]{.title-ref}
-file in your browser. pytest
-
-Building Documentation
-----------------------
-
-There is Sphinx documentation in the [docs/]{.title-ref} folder, which
-can be built with the code below:
-
-``` {.bash}
-cd docs/
-sphinx-build -M html . build/
-```
-
-Logging
--------
-
-The script uses Python\'s built-in logging functionality called
-[logging]{.title-ref}. All print functions are replaced with
-[log.info]{.title-ref} within this script, which sends the output to two
-places: 1) the terminal window, 2) the file [scout.log]{.title-ref}.
-Get into the habit of writing text logs and keeping date-specific
-versions for comparison and debugging.
-
-Code Formatting
----------------
-
-It\'s recommended that you use `pre-commit` to ensure linting procedures
-are run on any code you write. (See also
-[pre-commit.com](https://pre-commit.com/))
-
-Reference [pre-commit\'s installation
-instructions](https://pre-commit.com/#install) for software installation
-on your OS/platform. After you have the software installed, run
-`pre-commit install` on the command line. Now every time you commit to
-this project\'s code base the linter procedures will automatically run
-over the changed files. To run pre-commit on files preemtively from the
-command line use:
-
-``` {.bash}
-git add .
-pre-commit run
-
-# or
-
-pre-commit run --all-files
-```
-
-The code base has been formatted by Brunette, which is a fork and more
-configurable version of Black
-(<https://black.readthedocs.io/en/stable/>). Furthermore, try to conform
-to PEP8. You should set up your preferred editor to use flake8 as its
-Python linter, but pre-commit will ensure compliance before a git commit
-is completed. This will use the flake8 configuration within `setup.cfg`,
-which ignores several errors and stylistic considerations. See the
-`setup.cfg` file for a full and accurate listing of stylistic codes to
-ignore.
-
+---

README.rst

@@ -0,0 +1,132 @@
+================
+Wild Me ScoutBot
+================
+
+|Tests| |Wheel| |Docker| |ReadTheDocs| |Huggingface|
+
+.. contents:: Quick Links
+    :backlinks: none
+
+.. sectnum::
+
+How to Install
+--------------
+
+You need to first install Anaconda on your machine.  Below are the instructions on how to install Anaconda on an Apple macOS machine, but it is possible to install on a Windows and Linux machine as well.  Consult the `official Anaconda page <https://www.anaconda.com>`_ to download and install on other systems.
+
+.. code:: bash
+
+   # Install Homebrew
+   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
+
+   # Install Anaconda and expose conda to the terminal
+   brew install anaconda
+   export PATH="/opt/homebrew/anaconda3/bin:$PATH"
+   conda init zsh
+   conda update conda
+
+Once Anaconda is installed, you will need an environment and the following packages installed
+
+.. code:: bash
+
+   # Create Environment
+   conda create --name scoutbot
+   conda activate scoutbot
+
+   # Install Python dependencies
+   conda install pip
+
+   pip install -r requirements.txt
+   conda install pytorch torchvision -c pytorch-nightly
+
+How to Run
+----------
+
+It is recommended to use `ipython` and to copy sections of code into and inspecting the
+
+.. code:: bash
+
+   # Run the live demo
+   python app.py
+
+Docker
+------
+
+The application can also be built into a Docker image and hosted on Docker Hub.
+
+.. code:: bash
+
+    docker build . -t wildme/scoutbot:latest
+    docker push wildme/scoutbot:latest
+
+To run:
+
+.. code:: bash
+
+    docker run \
+       -it \
+       --rm \
+       -p 7860:7860 \
+       --name scoutbot \
+       wildme/scoutbot:latest
+
+Unit Tests
+----------
+
+You can run the automated tests in the `tests/` folder by running `pytest`.  This will give an output of which tests have failed.  You may also get a coverage percentage by running `coverage html` and loading the `coverage/html/index.html` file in your browser.
+pytest
+
+Building Documentation
+----------------------
+
+There is Sphinx documentation in the `docs/` folder, which can be built with the code below:
+
+.. code:: bash
+
+    cd docs/
+    sphinx-build -M html . build/
+
+Logging
+-------
+
+The script uses Python's built-in logging functionality called `logging`.  All print functions are replaced with `log.info` within this script, which sends the output to two places: 1) the terminal window, 2) the file `scoutbot.log`.  Get into the habit of writing text logs and keeping date-specific versions for comparison and debugging.
+
+Code Formatting
+---------------
+
+It's recommended that you use ``pre-commit`` to ensure linting procedures are run
+on any code you write. (See also `pre-commit.com <https://pre-commit.com/>`_)
+
+Reference `pre-commit's installation instructions <https://pre-commit.com/#install>`_ for software installation on your OS/platform. After you have the software installed, run ``pre-commit install`` on the command line. Now every time you commit to this project's code base the linter procedures will automatically run over the changed files.  To run pre-commit on files preemtively from the command line use:
+
+.. code:: bash
+
+    git add .
+    pre-commit run
+
+    # or
+
+    pre-commit run --all-files
+
+The code base has been formatted by Brunette, which is a fork and more configurable version of Black (https://black.readthedocs.io/en/stable/).  Furthermore, try to conform to PEP8.  You should set up your preferred editor to use flake8 as its Python linter, but pre-commit will ensure compliance before a git commit is completed.  This will use the flake8 configuration within ``setup.cfg``, which ignores several errors and stylistic considerations.  See the ``setup.cfg`` file for a full and accurate listing of stylistic codes to ignore.
+
+
+.. |Tests| image:: https://github.com/WildMeOrg/scoutbot/actions/workflows/testing.yml/badge.svg?branch=main
+    :target: https://github.com/WildMeOrg/scoutbot/actions/workflows/testing.yml
+    :alt: GitHub CI
+
+.. |Wheel| image:: https://github.com/WildMeOrg/scoutbot/actions/workflows/python-publish.yml/badge.svg
+    :target: https://github.com/WildMeOrg/scoutbot/actions/workflows/python-publish.yml
+    :alt: Python Wheel
+
+.. |Docker| image:: https://img.shields.io/docker/image-size/wildme/scoutbot/latest
+    :target: https://hub.docker.com/r/wildme/scoutbot
+    :alt: Docker
+
+.. |ReadTheDocs| image:: https://readthedocs.org/projects/scoutbot/badge/?version=latest
+    :target: https://scoutbot.readthedocs.io/en/latest/?badge=latest
+    :alt: ReadTheDocs
+
+.. |Huggingface| image:: https://img.shields.io/badge/HuggingFace-Running-yellow
+    :target: https://huggingface.co/spaces/WildMeOrg/scoutbot
+    :alt: Huggingface

app.py

@@ -1,53 +1,85 @@
 # -*- coding: utf-8 -*-
 import gradio as gr
-import numpy as np  # NOQA
-import torch
-from PIL import Image, ImageOps  # NOQA
-from torchvision.transforms import Compose, Resize, ToTensor
+import numpy as np
+import cv2
 
-from scoutbot import model, utils
+from scoutbot import wic, loc
 
-config = 'scoutbot/configs/mnist_resnet18.yaml'
 
-log = utils.init_logging()
-cfg = utils.init_config(config, log)
-device = cfg.get('device')
+def predict(filepath, wic_thresh, loc_thresh, nms_thresh):
+    # Load data
+    img = cv2.imread(filepath)
+    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+    inputs = [filepath]
 
-cfg['output'] = 'scoutbot/{}'.format(cfg['output'])
+    wic_thresh /= 100.0
+    loc_thresh /= 100.0
+    nms_thresh /= 100.0
 
-net, _, _ = model.load(cfg)
-net.eval()
+    # Run WIC
+    outputs = wic.post(wic.predict(wic.pre(inputs)))
+    output = outputs[0]
 
+    # Get WIC confidence
+    wic_confidence = output.get('positive')
 
-def predict(inp):
-    inp = ImageOps.grayscale(inp)
+    # Run Localizer
 
-    transforms = Compose([Resize(cfg['image_size']), ToTensor()])
-    inp = transforms(inp).unsqueeze(0)
-    data = inp.to(device)
+    loc_detections = []
+    if wic_confidence > wic_thresh:
+        data, sizes = loc.pre(inputs)
+        preds = loc.predict(data)
+        outputs = loc.post(preds, sizes, loc_thresh=loc_thresh, nms_thresh=nms_thresh)
+        detects = outputs[0]
 
-    with torch.no_grad():
-        prediction = net(data)
+        for detect in detects:
+            if detect.confidence >= loc_thresh:
+                point1 = (
+                    int(np.around(detect.x_top_left)),
+                    int(np.around(detect.y_top_left)),
+                )
+                point2 = (
+                    int(np.around(detect.x_top_left + detect.width)),
+                    int(np.around(detect.y_top_left + detect.height)),
+                )
+                color = (255, 0, 0)
+                img = cv2.rectangle(img, point1, point2, color, 2)
+                loc_detections.append(
+                    f'{detect.class_label}: {detect.confidence:0.05f}'
+                )
+    loc_detections = '\n'.join(loc_detections)
 
-    confidences = torch.softmax(prediction[0], dim=0).cpu().numpy()
-    confidences = list(enumerate(confidences))
-    confidences = [
-        (
-            str(label),
-            float(conf),
-        )
-        for label, conf in confidences
-    ]
-    confidences = dict(confidences)
-
-    return confidences
+    return img, wic_confidence, loc_detections
 
 
 interface = gr.Interface(
     fn=predict,
-    inputs=gr.Image(type='pil'),
-    outputs=gr.Label(num_top_classes=3),
-    examples=[f'examples/example_{index}.jpg' for index in range(1, 31)],
+    title='Scout Demo',
+    inputs=[
+        gr.Image(type='filepath'),
+        gr.Slider(label='WIC Confidence Threshold', value=20),
+        gr.Slider(label='Localizer Confidence Threshold', value=48),
+        gr.Slider(label='Localizer NMS Threshold', value=20),
+    ],
+    outputs=[
+        gr.Image(type='numpy'),
+        gr.Number(label='Predicted WIC Confidence', precision=5, interactive=False),
+        gr.Textbox(label='Predicted Localizer Detections', interactive=False),
+    ],
+    examples=[
+        ['examples/07a4b8db-f31c-261d-4580-e9402768fd45.true.jpg', 20, 48, 20],
+        ['examples/15e815d9-5aad-fa53-d1ed-33429020e15e.true.jpg', 10, 48, 20],
+        ['examples/1bb79811-3149-7a60-2d88-613dc3eeb261.true.jpg', 20, 48, 20],
+        ['examples/1e8372e4-357d-26e6-d7fd-0e0ae402463a.true.jpg', 20, 48, 20],
+        ['examples/201bc65e-d64e-80d3-2610-5865a22d04b4.false.jpg', 20, 48, 20],
+        ['examples/3affd8b6-9722-f2d5-9171-639615b4c38f.true.jpg', 20, 48, 20],
+        ['examples/4aedb818-f2f4-e462-8b75-5c8e34a01a59.false.jpg', 20, 48, 20],
+        ['examples/474bc2b6-dc51-c1b5-4612-efe810bbe091.true.jpg', 20, 48, 20],
+        ['examples/c3014107-3464-60b5-e04a-e4bfafdf8809.false.jpg', 20, 48, 20],
+        ['examples/f835ce33-292a-9116-794e-f8859b5956ec.true.jpg', 20, 48, 20],
+    ],
+    cache_examples=True,
+    allow_flagging='never',
 )
 
 interface.launch(server_name='0.0.0.0')

docs/cli.rst

@@ -0,0 +1,12 @@
+ScoutBot CLI
+============
+
+.. toctree::
+   :maxdepth: 3
+   :caption: Contents:
+
+
+.. automodule:: scoutbot.scoutbot
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/index.rst

@@ -11,4 +11,5 @@ Contents
 
    Home <self>
    usage
-   package
+   scoutbot
+   cli

docs/{package.rst → scoutbot.rst}

@@ -1,37 +1,38 @@
-Package
-=======
+ScoutBot API
+============
 
 .. toctree::
    :maxdepth: 3
    :caption: Contents:
 
 
-dataset.py
-----------
+Tiles
+-----
 
-.. automodule:: scoutbot.dataset
+.. automodule:: scoutbot.tile
    :members:
    :undoc-members:
    :show-inheritance:
 
-model.py
-----------
 
-.. automodule:: scoutbot.model
+Whole-Image Classifier (WIC)
+----------------------------
+
+.. automodule:: scoutbot.wic
    :members:
    :undoc-members:
    :show-inheritance:
 
-train.py
---------
+Localizer (LOC)
+---------------
 
-.. automodule:: scoutbot.train
+.. automodule:: scoutbot.loc
    :members:
    :undoc-members:
    :show-inheritance:
 
-utils.py
---------
+Utilities
+---------
 
 .. automodule:: scoutbot.utils
    :members:

requirements.optional.txt

@@ -0,0 +1,17 @@
+brunette
+codecov
+coverage
+flake8
+ipython
+onnx
+pre-commit
+pytest
+pytest-benchmark[histogram]
+pytest-cov
+pytest-profiling
+pytest-random-order
+pytest-sugar
+pytest-xdist
+Sphinx>=5,<6
+sphinx_rtd_theme
+xdoctest

requirements.txt

@@ -1,30 +1,13 @@
-argparse
-brunette
-click
-codecov
-coverage
-cryptography
-flake8
-gradio
-ipython
-numpy
-onnx
 onnxruntime
-Pillow
-pre-commit
-pytest
-pytest-cov
-pytest-random-order
-pytest-sugar
-PyYAML
-rich
-Sphinx>=5,<6
-sphinx_rtd_theme
+numpy
+wbia-utool
 torch
 torchvision
+opencv-python-headless
+Pillow
+imgaug
+rich
 tqdm
-wbia-utool
-wbia-vtool
-python-opencv-headless
-lightnet
-scikit-learn
+gradio
+cryptography
+click

scoutbot/__init__.py

@@ -2,6 +2,11 @@
 '''
 2022 Wild Me
 '''
+from scoutbot import utils
 
-version = '0.1.0'
-__version__ = version
+VERSION = '0.1.0'
+version = VERSION
+__version__ = VERSION
+
+
+log = utils.init_logging()

scoutbot/loc/__init__.py

@@ -2,6 +2,108 @@
 '''
 2022 Wild Me
 '''
+from os.path import join
+import onnxruntime as ort
+from pathlib import Path
+import torchvision
+import numpy as np
+import utool as ut
+import torch
+import cv2
+from scoutbot.loc.transforms import (
+    Letterbox,
+    Compose,
+    GetBoundingBoxes,
+    NonMaxSupression,
+    TensorToBrambox,
+    ReverseLetterbox,
+)
 
-version = '0.1.0'
-__version__ = version
+
+PWD = Path(__file__).absolute().parent
+
+BATCH_SIZE = 128
+INPUT_SIZE = (416, 416)
+INPUT_SIZE_H, INPUT_SIZE_W = INPUT_SIZE
+NETWORK_SIZE = (INPUT_SIZE_H, INPUT_SIZE_W, 3)
+
+NUM_CLASSES = 1
+ANCHORS = [
+    (1.3221, 1.73145),
+    (3.19275, 4.00944),
+    (5.05587, 8.09892),
+    (9.47112, 4.84053),
+    (11.2364, 10.0071),
+]
+CLASS_LABEL_MAP = ['elephant_savanna']
+CONF_THRESH = 0.4
+NMS_THRESH = 0.8
+
+ONNX_MODEL = join(PWD, 'models', 'onnx', 'scout.loc.5fbfff26.0.onnx')
+
+
+def pre(inputs):
+    transform = torchvision.transforms.ToTensor()
+
+    data = []
+    sizes = []
+    for filepath in inputs:
+        img = cv2.imread(filepath)
+        size = img.shape[:2][::-1]
+
+        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+        img = Letterbox.apply(
+            img,
+            dimension=INPUT_SIZE
+        )
+        img = transform(img)
+
+        data.append(img.tolist())
+        sizes.append(size)
+
+    return data, sizes
+
+
+def predict(data):
+    ort_session = ort.InferenceSession(
+        ONNX_MODEL,
+        providers=['CPUExecutionProvider']
+    )
+
+    preds = []
+    for chunk in ut.ichunks(data, BATCH_SIZE):
+        trim = len(chunk)
+        while(len(chunk)) < BATCH_SIZE:
+            chunk.append(np.random.randn(3, INPUT_SIZE_H, INPUT_SIZE_W).astype(np.float32))
+        input_ = np.array(chunk, dtype=np.float32)
+
+        pred_ = ort_session.run(
+            None,
+            {'input': input_},
+        )
+        preds += pred_[0].tolist()[:trim]
+
+    return preds
+
+
+def post(preds, sizes, loc_thresh=CONF_THRESH, nms_thresh=NMS_THRESH):
+    postprocess = Compose(
+        [
+            GetBoundingBoxes(
+                NUM_CLASSES, ANCHORS, loc_thresh
+            ),
+            NonMaxSupression(nms_thresh),
+            TensorToBrambox(NETWORK_SIZE, CLASS_LABEL_MAP),
+        ]
+    )
+
+    preds = postprocess(torch.tensor(preds))
+
+    outputs = []
+    for pred, size in zip(preds, sizes):
+        output = ReverseLetterbox.apply(
+            [pred], INPUT_SIZE, size
+        )
+        outputs.append(output[0])
+
+    return outputs

scoutbot/loc/transforms/__init__.py

@@ -0,0 +1,9 @@
+# -*- coding: utf-8 -*-
+#
+#   Lightnet data transforms
+#   Copyright EAVISE
+#
+
+from ._preprocess import *
+from ._postprocess import *
+from .util import *

scoutbot/loc/transforms/_postprocess.py

@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+#
+#   Lightnet related postprocessing
+#   Thers are functions to transform the output of the network to brambox detection objects
+#   Copyright EAVISE
+#
+
+import logging
+import torch
+# from torch.autograd import Variable
+from scoutbot.loc.transforms.detections.detection import Detection
+from .util import BaseTransform
+
+__all__ = [
+    'GetBoundingBoxes',
+    'NonMaxSupression',
+    'TensorToBrambox',
+    'ReverseLetterbox',
+]
+log = logging.getLogger(__name__)
+
+
+class GetBoundingBoxes(BaseTransform):
+    """ Convert output from darknet networks to bounding box tensor.
+
+    Args:
+        num_classes (int): number of categories
+        anchors (list): 2D list representing anchor boxes (see :class:`lightnet.network.Darknet`)
+        conf_thresh (Number [0-1]): Confidence threshold to filter detections
+
+    Returns:
+        (list [Batch x Tensor [Boxes x 6]]): **[x_center, y_center, width, height, confidence, class_id]** for every bounding box
+
+    Note:
+        The output tensor uses relative values for its coordinates.
+    """
+
+    def __init__(self, num_classes, anchors, conf_thresh):
+        super().__init__(
+            num_classes=num_classes, anchors=anchors, conf_thresh=conf_thresh
+        )
+
+    @classmethod
+    def apply(cls, network_output, num_classes, anchors, conf_thresh):
+        # Check dimensions
+        if network_output.dim() == 3:
+            network_output.unsqueeze_(0)
+
+        # Variables
+        num_anchors = len(anchors)
+        # anchor_step = len(anchors[0])
+        anchors = torch.Tensor(anchors)
+        device = network_output.device
+        batch = network_output.size(0)
+        h = network_output.size(2)
+        w = network_output.size(3)
+
+        # Compute xc,yc, w,h, box_score on Tensor
+        lin_x = torch.linspace(0, w - 1, w).repeat(h, 1).view(h * w).to(device)
+        lin_y = torch.linspace(0, h - 1, h).view(h, 1).repeat(1, w).view(h * w).to(device)
+        anchor_w = anchors[:, 0].contiguous().view(1, num_anchors, 1).to(device)
+        anchor_h = anchors[:, 1].contiguous().view(1, num_anchors, 1).to(device)
+
+        network_output = network_output.view(
+            batch, num_anchors, -1, h * w
+        )  # -1 == 5+num_classes (we can drop feature maps if 1 class)
+        network_output[:, :, 0, :].sigmoid_().add_(lin_x).div_(w)  # X center
+        network_output[:, :, 1, :].sigmoid_().add_(lin_y).div_(h)  # Y center
+        network_output[:, :, 2, :].exp_().mul_(anchor_w).div_(w)  # Width
+        network_output[:, :, 3, :].exp_().mul_(anchor_h).div_(h)  # Height
+        network_output[:, :, 4, :].sigmoid_()  # Box score
+
+        # Compute class_score
+        if num_classes > 1:
+            with torch.no_grad():
+                cls_scores = torch.nn.functional.softmax(network_output[:, :, 5:, :], 2)
+            cls_max, cls_max_idx = torch.max(cls_scores, 2)
+            cls_max_idx = cls_max_idx.float()
+            cls_max.mul_(network_output[:, :, 4, :])
+        else:
+            cls_max = network_output[:, :, 4, :]
+            cls_max_idx = torch.zeros_like(cls_max)
+
+        score_thresh = cls_max > conf_thresh
+        score_thresh_flat = score_thresh.view(-1)
+
+        if score_thresh.sum() == 0:
+            boxes = []
+            for i in range(batch):
+                boxes.append(torch.tensor([]))
+            return boxes
+
+        # Mask select boxes > conf_thresh
+        coords = network_output.transpose(2, 3)[..., 0:4]
+        coords = coords[score_thresh[..., None].expand_as(coords)].view(-1, 4)
+        scores = cls_max[score_thresh]
+        idx = cls_max_idx[score_thresh]
+        detections = torch.cat([coords, scores[:, None], idx[:, None]], dim=1)
+
+        # Get indexes of splits between images of batch
+        max_det_per_batch = num_anchors * h * w
+        slices = [
+            slice(max_det_per_batch * i, max_det_per_batch * (i + 1))
+            for i in range(batch)
+        ]
+        det_per_batch = torch.IntTensor(
+            [score_thresh_flat[s].int().sum() for s in slices]
+        )
+        split_idx = torch.cumsum(det_per_batch, dim=0)
+
+        # Group detections per image of batch
+        boxes = []
+        start = 0
+        for end in split_idx:
+            boxes.append(detections[start:end])
+            start = end
+
+        return boxes
+
+
+class NonMaxSupression(BaseTransform):
+    """ Performs nms on the bounding boxes, filtering boxes with a high overlap.
+
+    Args:
+        nms_thresh (Number [0-1]): Overlapping threshold to filter detections with non-maxima suppresion
+        class_nms (Boolean, optional): Whether to perform nms per class; Default **True**
+
+    Returns:
+        (list [Batch x Tensor [Boxes x 6]]): **[x_center, y_center, width, height, confidence, class_id]** for every bounding box
+
+    Note:
+        This post-processing function expects the input to be bounding boxes,
+        like the ones created by :class:`lightnet.data.GetBoundingBoxes` and outputs exactly the same format.
+    """
+
+    def __init__(self, nms_thresh, class_nms=True):
+        super().__init__(nms_thresh=nms_thresh, class_nms=class_nms)
+
+    @classmethod
+    def apply(cls, boxes, nms_thresh, class_nms=True):
+        return [cls._nms(box, nms_thresh, class_nms) for box in boxes]
+
+    @staticmethod
+    def _nms(boxes, nms_thresh, class_nms):
+        """ Non maximum suppression.
+
+        Args:
+          boxes (tensor): Bounding boxes of one image
+
+        Return:
+          (tensor): Pruned boxes
+        """
+        if boxes.numel() == 0:
+            return boxes
+
+        a = boxes[:, :2]
+        b = boxes[:, 2:4]
+        bboxes = torch.cat([a - b / 2, a + b / 2], 1)
+        scores = boxes[:, 4]
+        classes = boxes[:, 5]
+
+        # Sort coordinates by descending score
+        scores, order = scores.sort(0, descending=True)
+        x1, y1, x2, y2 = bboxes[order].split(1, 1)
+
+        # Compute dx and dy between each pair of boxes (these mat contain every pair twice...)
+        dx = (x2.min(x2.t()) - x1.max(x1.t())).clamp(min=0)
+        dy = (y2.min(y2.t()) - y1.max(y1.t())).clamp(min=0)
+
+        # Compute iou
+        intersections = dx * dy
+        areas = (x2 - x1) * (y2 - y1)
+        unions = (areas + areas.t()) - intersections
+        ious = intersections / unions
+
+        # Filter based on iou (and class)
+        conflicting = (ious > nms_thresh).triu(1)
+
+        if class_nms:
+            classes = classes[order]
+            same_class = classes.unsqueeze(0) == classes.unsqueeze(1)
+            conflicting = conflicting & same_class
+
+        conflicting = conflicting.cpu()
+        keep = torch.zeros(len(conflicting), dtype=torch.uint8)
+        supress = torch.zeros(len(conflicting), dtype=torch.float)
+        for i, row in enumerate(conflicting):
+            if not supress[i]:
+                keep[i] = 1
+                supress[row] = 1
+
+        return boxes[order][keep[:, None].expand_as(boxes)].view(-1, 6).contiguous()
+
+
+class TensorToBrambox(BaseTransform):
+    """ Converts a tensor to a list of brambox objects.
+
+    Args:
+        network_size (tuple): Tuple containing the width and height of the images going in the network
+        class_label_map (list, optional): List of class labels to transform the class id's in actual names; Default **None**
+
+    Returns:
+        (list [list [brambox.boxes.Detection]]): list of brambox detections per image
+
+    Note:
+        If no `class_label_map` is given, this transform will simply convert the class id's in a string.
+
+    Note:
+        Just like everything in PyTorch, this transform only works on batches of images.
+        This means you need to wrap your tensor of detections in a list if you want to run this transform on a single image.
+    """
+
+    def __init__(self, network_size, class_label_map=None):
+        super().__init__(network_size=network_size, class_label_map=class_label_map)
+        if self.class_label_map is None:
+            log.warn(
+                'No class_label_map given. The indexes will be used as class_labels.'
+            )
+
+    @classmethod
+    def apply(cls, boxes, network_size, class_label_map=None):
+        converted_boxes = []
+        for box in boxes:
+            if box.numel() == 0:
+                converted_boxes.append([])
+            else:
+                converted_boxes.append(
+                    cls._convert(box, network_size[0], network_size[1], class_label_map)
+                )
+        return converted_boxes
+
+    @staticmethod
+    def _convert(boxes, width, height, class_label_map):
+        boxes[:, 0:3:2].mul_(width)
+        boxes[:, 0] -= boxes[:, 2] / 2
+        boxes[:, 1:4:2].mul_(height)
+        boxes[:, 1] -= boxes[:, 3] / 2
+
+        brambox = []
+        for box in boxes:
+            det = Detection()
+            det.x_top_left = box[0].item()
+            det.y_top_left = box[1].item()
+            det.width = box[2].item()
+            det.height = box[3].item()
+            det.confidence = box[4].item()
+            if class_label_map is not None:
+                det.class_label = class_label_map[int(box[5].item())]
+            else:
+                det.class_label = str(int(box[5].item()))
+
+            brambox.append(det)
+
+        return brambox
+
+
+class ReverseLetterbox(BaseTransform):
+    """ Performs a reverse letterbox operation on the bounding boxes, so they can be visualised on the original image.
+
+    Args:
+        network_size (tuple): Tuple containing the width and height of the images going in the network
+        image_size (tuple): Tuple containing the width and height of the original images
+
+    Returns:
+        (list [list [brambox.boxes.Detection]]): list of brambox detections per image
+
+    Note:
+        This transform works on :class:`brambox.boxes.Detection` objects,
+        so you need to apply the :class:`~lightnet.data.TensorToBrambox` transform first.
+
+    Note:
+        Just like everything in PyTorch, this transform only works on batches of images.
+        This means you need to wrap your tensor of detections in a list if you want to run this transform on a single image.
+    """
+
+    def __init__(self, network_size, image_size):
+        super().__init__(network_size=network_size, image_size=image_size)
+
+    @classmethod
+    def apply(cls, boxes, network_size, image_size):
+        im_w, im_h = image_size[:2]
+        net_w, net_h = network_size[:2]
+
+        if im_w == net_w and im_h == net_h:
+            scale = 1
+        elif im_w / net_w >= im_h / net_h:
+            scale = im_w / net_w
+        else:
+            scale = im_h / net_h
+        pad = int((net_w - im_w / scale) / 2), int((net_h - im_h / scale) / 2)
+
+        converted_boxes = []
+        for b in boxes:
+            converted_boxes.append(cls._transform(b, scale, pad))
+        return converted_boxes
+
+    @staticmethod
+    def _transform(boxes, scale, pad):
+        for box in boxes:
+            box.x_top_left -= pad[0]
+            box.y_top_left -= pad[1]
+
+            box.x_top_left *= scale
+            box.y_top_left *= scale
+            box.width *= scale
+            box.height *= scale
+        return boxes

scoutbot/loc/transforms/_preprocess.py

@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+#
+#   Image and annotations preprocessing for lightnet networks
+#   The image transformations work with both Pillow and OpenCV images
+#   The annotation transformations work with brambox.annotations.Annotation objects
+#   Copyright EAVISE
+#
+import collections
+import logging
+import numpy as np
+from PIL import Image, ImageOps
+from .util import BaseMultiTransform
+
+log = logging.getLogger(__name__)
+
+try:
+    import cv2
+except ImportError:
+    log.warn('OpenCV is not installed and cannot be used')
+    cv2 = None
+
+__all__ = ['Letterbox']
+
+
+class Letterbox(BaseMultiTransform):
+    """ Transform images and annotations to the right network dimensions.
+
+    Args:
+        dimension (tuple, optional): Default size for the letterboxing, expressed as a (width, height) tuple; Default **None**
+        dataset (lightnet.data.Dataset, optional): Dataset that uses this transform; Default **None**
+
+    Note:
+        Create 1 Letterbox object and use it for both image and annotation transforms.
+        This object will save data from the image transform and use that on the annotation transform.
+    """
+
+    def __init__(self, dimension=None, dataset=None):
+        super().__init__(dimension=dimension, dataset=dataset)
+        if self.dimension is None and self.dataset is None:
+            raise ValueError(
+                'This transform either requires a dimension or a dataset to infer the dimension'
+            )
+
+        self.pad = None
+        self.scale = None
+        self.fill_color = 127
+
+    def __call__(self, data):
+        if data is None:
+            return None
+        elif isinstance(data, collections.abc.Sequence):
+            return self._tf_anno(data)
+        elif isinstance(data, Image.Image):
+            return self._tf_pil(data)
+        elif isinstance(data, np.ndarray):
+            return self._tf_cv(data)
+        else:
+            log.error(
+                f'Letterbox only works with <brambox annotation lists>, <PIL images> or <OpenCV images> [{type(data)}]'
+            )
+            return data
+
+    def _tf_pil(self, img):
+        """ Letterbox an image to fit in the network """
+        if self.dataset is not None:
+            net_w, net_h = self.dataset.input_dim
+        else:
+            net_w, net_h = self.dimension
+        im_w, im_h = img.size
+
+        if im_w == net_w and im_h == net_h:
+            self.scale = None
+            self.pad = None
+            return img
+
+        # Rescaling
+        if im_w / net_w >= im_h / net_h:
+            self.scale = net_w / im_w
+        else:
+            self.scale = net_h / im_h
+        if self.scale != 1:
+            bands = img.split()
+            bands = [
+                b.resize((int(self.scale * im_w), int(self.scale * im_h))) for b in bands
+            ]
+            img = Image.merge(img.mode, bands)
+            im_w, im_h = img.size
+
+        if im_w == net_w and im_h == net_h:
+            self.pad = None
+            return img
+
+        # Padding
+        img_np = np.array(img)
+        channels = img_np.shape[2] if len(img_np.shape) > 2 else 1
+        pad_w = (net_w - im_w) / 2
+        pad_h = (net_h - im_h) / 2
+        self.pad = (int(pad_w), int(pad_h), int(pad_w + 0.5), int(pad_h + 0.5))
+        img = ImageOps.expand(img, border=self.pad, fill=(self.fill_color,) * channels)
+        return img
+
+    def _tf_cv(self, img):
+        """ Letterbox and image to fit in the network """
+        if self.dataset is not None:
+            net_w, net_h = self.dataset.input_dim
+        else:
+            net_w, net_h = self.dimension
+        im_h, im_w = img.shape[:2]
+
+        if im_w == net_w and im_h == net_h:
+            self.scale = None
+            self.pad = None
+            return img
+
+        # Rescaling
+        if im_w / net_w >= im_h / net_h:
+            self.scale = net_w / im_w
+        else:
+            self.scale = net_h / im_h
+        if self.scale != 1:
+            img = cv2.resize(
+                img, None, fx=self.scale, fy=self.scale, interpolation=cv2.INTER_CUBIC
+            )
+            im_h, im_w = img.shape[:2]
+
+        if im_w == net_w and im_h == net_h:
+            self.pad = None
+            return img
+
+        # Padding
+        # channels = img.shape[2] if len(img.shape) > 2 else 1
+        pad_w = (net_w - im_w) / 2
+        pad_h = (net_h - im_h) / 2
+        self.pad = (int(pad_w), int(pad_h), int(pad_w + 0.5), int(pad_h + 0.5))
+        img = cv2.copyMakeBorder(
+            img,
+            self.pad[1],
+            self.pad[3],
+            self.pad[0],
+            self.pad[2],
+            cv2.BORDER_CONSTANT,
+            value=self.fill_color,
+        )
+        return img
+
+    def _tf_anno(self, annos):
+        """ Change coordinates of an annotation, according to the previous letterboxing """
+        for anno in annos:
+            if self.scale is not None:
+                anno.x_top_left *= self.scale
+                anno.y_top_left *= self.scale
+                anno.width *= self.scale
+                anno.height *= self.scale
+            if self.pad is not None:
+                anno.x_top_left += self.pad[0]
+                anno.y_top_left += self.pad[1]
+        return annos

scoutbot/loc/transforms/annotations/annotation.py

@@ -0,0 +1,165 @@
+# -*- coding: utf-8 -*-
+#
+#   Copyright EAVISE
+#
+
+# from enum import Enum
+
+from scoutbot.loc.transforms import box as b
+from scoutbot.loc.transforms.detections import detection as det
+
+__all__ = ['Annotation', 'ParserType', 'Parser']
+
+
+class Annotation(b.Box):
+    """ This is a generic annotation class that provides some common functionality all annotations need.
+    It builds upon :class:`~brambox.boxes.box.Box`.
+
+    Attributes:
+        lost (Boolean): Flag indicating whether the annotation is visible in the image; Default **False**
+        difficult (Boolean): Flag indicating whether the annotation is considered difficult; Default **False**
+        interest (Boolean): Flag indicating whether the annotation is an Annotation of Interest (AoI); Default **False**
+        occluded (Boolean): Flag indicating whether the annotation is occluded; Default **False**
+        ignore (Boolean): Flag that is used to ignore a bounding box during statistics processing; Default **False**
+        occluded_fraction (Number): value between 0 and 1 that indicates the amount of occlusion (1 = completely occluded); Default **0.0**
+        truncated_fraction (Number): value between 0 and 1 that indicates the amount of truncation (1 = completely truncated); Default **0.0**
+        visible_x_top_left (Number): X pixel coordinate of the top left corner of the bounding box that frames the visible part of the object; Default **0.0**
+        visible_y_top_left (Number): Y pixel coordinate of the top left corner of the bounding box that frames the visible part of the object; Default **0.0**
+        visible_width (Number): Width of the visible bounding box in pixels; Default **0.0**
+        visible_height (Number): Height of the visible bounding box in pixels; Default **0.0**
+
+    Note:
+        The ``visible_x_top_left``, ``visible_y_top_left``, ``visible_width`` and ``visible_height`` attributes
+        are only valid when the ``occluded`` flag is set to **True**.
+    Note:
+        The ``occluded`` flag is actually a property that returns **True** if the ``occluded_fraction`` > **0.0** and **False** if
+        the occluded_fraction equals **0.0**. Thus modifying the ``occluded_fraction`` will affect the ``occluded`` flag and visa versa.
+    """
+
+    def __init__(self):
+        """ x_top_left,y_top_left,width,height are in pixel coordinates """
+        super(Annotation, self).__init__()
+        self.lost = False  # if object is not seen in the image, if true one must ignore this annotation
+        self.difficult = False  # if the object is considered difficult
+        self.interest = False  # if the object is an Annotation of Interest (AoI)
+        self.ignore = False  # if true, this bounding box will not be considered in statistics processing
+        self.occluded_fraction = (
+            0.0  # value between 0 and 1 that indicates how much an object is occluded
+        )
+        self.truncated_fraction = (
+            0.0  # value between 0 and 1 that indicates how much an object is truncated
+        )
+
+        # variables below are only valid if the 'occluded' property is True (occluded_fraction > 0) and
+        # represent a bounding box that indicates the visible area inside the normal bounding box
+        self.visible_x_top_left = 0.0  # x position top left in pixels
+        self.visible_y_top_left = 0.0  # y position top left in pixels
+        self.visible_width = 0.0  # width in pixels
+        self.visible_height = 0.0  # height in pixels
+
+    @property
+    def occluded(self):
+        return self.occluded_fraction > 0.0
+
+    @occluded.setter
+    def occluded(self, val):
+        self.occluded_fraction = float(val)
+
+    @property
+    def truncated(self):
+        return self.truncated_fraction > 0.0
+
+    @truncated.setter
+    def truncated(self, val):
+        self.truncated_fraction = float(val)
+
+    @classmethod
+    def create(cls, obj=None):
+        """ Create an annotation from a string or other box object.
+
+        Args:
+            obj (Box or string, optional): Bounding box object to copy attributes from or string to deserialize
+
+        Note:
+            The obj can be both an :class:`~brambox.boxes.annotations.Annotation` or a :class:`~brambox.boxes.detections.Detection`.
+            For Annotations every attribute is copied over, for Detections the flags are all set to **False**.
+        """
+        instance = super(Annotation, cls).create(obj)
+
+        if obj is None:
+            return instance
+
+        if isinstance(obj, Annotation):
+            instance.lost = obj.lost
+            instance.difficult = obj.difficult
+            instance.interest = obj.interest
+            instance.ignore = obj.ignore
+            instance.truncated_fraction = obj.truncated_fraction
+            instance.occluded_fraction = obj.occluded_fraction
+            instance.visible_x_top_left = obj.visible_x_top_left
+            instance.visible_y_top_left = obj.visible_y_top_left
+            instance.visible_width = obj.visible_width
+            instance.visible_height = obj.visible_height
+        elif isinstance(obj, det.Detection):
+            instance.lost = False
+            instance.difficult = False
+            instance.interest = False
+            instance.occluded = False
+            instance.visible_x_top_left = 0.0
+            instance.visible_y_top_left = 0.0
+            instance.visible_width = 0.0
+            instance.visible_height = 0.0
+
+        return instance
+
+    def __repr__(self):
+        """ Unambiguous representation """
+        string = f'{self.__class__.__name__} ' + '{'
+        string += f"class_label = '{self.class_label}', "
+        string += f'object_id = {self.object_id}, '
+        string += f'x = {self.x_top_left}, '
+        string += f'y = {self.y_top_left}, '
+        string += f'w = {self.width}, '
+        string += f'h = {self.height}, '
+        string += f'ignore = {self.ignore}, '
+        string += f'lost = {self.lost}, '
+        string += f'difficult = {self.difficult}, '
+        string += f'interest = {self.interest}, '
+        string += f'truncated_fraction = {self.truncated_fraction}, '
+        string += f'occluded_fraction = {self.occluded_fraction}, '
+        string += f'visible_x = {self.visible_x_top_left}, '
+        string += f'visible_y = {self.visible_y_top_left}, '
+        string += f'visible_w = {self.visible_width}, '
+        string += f'visible_h = {self.visible_height}'
+        return string + '}'
+
+    def __str__(self):
+        """ Pretty print """
+        string = 'Annotation {'
+        string += f'\'{self.class_label}\'{"" if self.object_id is None else " "+str(self.object_id)}, '
+        string += f'[{int(self.x_top_left)}, {int(self.y_top_left)}, {int(self.width)}, {int(self.height)}]'
+        if self.difficult:
+            string += ', difficult'
+        if self.interest:
+            string += ', interest'
+        if self.lost:
+            string += ', lost'
+        if self.ignore:
+            string += ', ignore'
+        if self.truncated:
+            string += f', truncated {self.truncated_fraction*100}%'
+        if self.occluded:
+            if self.occluded_fraction == 1.0:
+                string += f', occluded [{int(self.visible_x_top_left)}, {int(self.visible_y_top_left)}, {int(self.visible_width)}, {int(self.visible_height)}]'
+            else:
+                string += f', occluded {self.occluded_fraction*100}%'
+        return string + '}'
+
+
+ParserType = b.ParserType
+
+
+class Parser(b.Parser):
+    """ Generic parser class """
+
+    box_type = Annotation  # Derived classes should set the correct box_type

scoutbot/loc/transforms/box.py

@@ -0,0 +1,150 @@
+# -*- coding: utf-8 -*-
+#
+#   Copyright EAVISE
+#
+
+from enum import Enum
+
+__all__ = ['Box', 'ParserType', 'Parser']
+
+
+class Box:
+    """ This is a generic bounding box representation.
+    This class provides some base functionality to both annotations and detections.
+
+    Attributes:
+        class_label (string): class string label; Default **''**
+        object_id (int): Object identifier for reid purposes; Default **None**
+        x_top_left (Number): X pixel coordinate of the top left corner of the bounding box; Default **0.0**
+        y_top_left (Number): Y pixel coordinate of the top left corner of the bounding box; Default **0.0**
+        width (Number): Width of the bounding box in pixels; Default **0.0**
+        height (Number): Height of the bounding box in pixels; Default **0.0**
+    """
+
+    def __init__(self):
+        self.class_label = ''  # class string label
+        self.object_id = None  # object identifier
+        self.x_top_left = 0.0  # x pixel coordinate top left of the box
+        self.y_top_left = 0.0  # y pixel coordinate top left of the box
+        self.width = 0.0  # width of the box in pixels
+        self.height = 0.0  # height of the box in pixels
+
+    @classmethod
+    def create(cls, obj=None):
+        """ Create a bounding box from a string or other detection object.
+
+        Args:
+            obj (Box or string, optional): Bounding box object to copy attributes from or string to deserialize
+        """
+        instance = cls()
+
+        if obj is None:
+            return instance
+
+        if isinstance(obj, str):
+            instance.deserialize(obj)
+        elif isinstance(obj, Box):
+            instance.class_label = obj.class_label
+            instance.object_id = obj.object_id
+            instance.x_top_left = obj.x_top_left
+            instance.y_top_left = obj.y_top_left
+            instance.width = obj.width
+            instance.height = obj.height
+        else:
+            raise TypeError(
+                'Object is not of type Box or not a string [obj.__class__.__name__]'
+            )
+
+        return instance
+
+    def __eq__(self, other):
+        # TODO: refactor -> use almost equal for floats
+        return self.__dict__ == other.__dict__
+
+    def serialize(self):
+        """ abstract serializer, implement in derived classes. """
+        raise NotImplementedError
+
+    def deserialize(self, string):
+        """ abstract parser, implement in derived classes. """
+        raise NotImplementedError
+
+
+class ParserType(Enum):
+    """ Enum for differentiating between different parser types. """
+
+    UNDEFINED = 0  #: Undefined parsertype. Do not use this!
+    SINGLE_FILE = 1  #: One single file contains all annotations
+    MULTI_FILE = 2  #: One annotation file per image
+
+
+class Parser:
+    """ This is a Generic parser class.
+
+    Args:
+        kwargs (optional): Derived parsers should use keyword arguments to get any information they need upon initialisation.
+    """
+
+    parser_type = (
+        ParserType.UNDEFINED
+    )  #: Type of parser. Derived classes should set the correct value.
+    box_type = Box  #: Type of bounding box this parser parses or generates. Derived classes should set the correct type.
+    extension = '.txt'  #: Extension of the files this parser parses or creates. Derived classes should set the correct extension.
+    read_mode = 'r'  #: Reading mode this parser uses when it parses a file. Derived classes should set the correct mode.
+    write_mode = 'w'  #: Writing mode this parser uses when it generates a file. Derived classes should set the correct mode.
+
+    def __init__(self, **kwargs):
+        pass
+
+    def serialize(self, box):
+        """ Serialization function that can be overloaded in the derived class.
+        The default serializer will call the serialize function of the bounding boxes and join them with a newline.
+
+        Args:
+            box: Bounding box objects
+
+        Returns:
+            string: Serialized bounding boxes
+
+        Note:
+            The format of the box parameter depends on the type of parser. |br|
+            If it is a :any:`brambox.boxes.ParserType.SINGLE_FILE`, the box parameter should be a dictionary ``{"image_id": [box, box, ...], ...}``. |br|
+            If it is a :any:`brambox.boxes.ParserType.MULTI_FILE`, the box parameter should be a list ``[box, box, ...]``.
+        """
+        if self.parser_type != ParserType.MULTI_FILE:
+            raise TypeError(
+                'The default implementation of serialize only works with MULTI_FILE'
+            )
+
+        result = ''
+        for b in box:
+            new_box = self.box_type.create(b)
+            result += new_box.serialize() + '\n'
+
+        return result
+
+    def deserialize(self, string):
+        """ Deserialization function that can be overloaded in the derived class.
+        The default deserialize will create new ``box_type`` objects and call the deserialize function of these objects with every line of the input string.
+
+        Args:
+            string (string): Input string to deserialize
+
+        Returns:
+            box: Bounding box objects
+
+        Note:
+            The format of the box return value depends on the type of parser. |br|
+            If it is a :any:`brambox.boxes.ParserType.SINGLE_FILE`, the return value should be a dictionary ``{"image_id": [box, box, ...], ...}``. |br|
+            If it is a :any:`brambox.boxes.ParserType.MULTI_FILE`, the return value should be a list ``[box, box, ...]``.
+        """
+        if self.parser_type != ParserType.MULTI_FILE:
+            raise TypeError(
+                'The default implementation of deserialize only works with MULTI_FILE'
+            )
+
+        result = []
+        for line in string.splitlines():
+            result += [self.box_type.create(line)]
+
+        return result

scoutbot/loc/transforms/detections/detection.py

@@ -0,0 +1,112 @@
+# -*- coding: utf-8 -*-
+#
+#   Copyright EAVISE
+#
+
+# from enum import Enum
+
+from scoutbot.loc.transforms import box as b
+from scoutbot.loc.transforms.annotations import annotation as anno
+
+__all__ = ['Detection', 'ParserType', 'Parser']
+
+
+class Detection(b.Box):
+    """ This is a generic detection class that provides some base functionality all detections need.
+    It builds upon :class:`~brambox.boxes.box.Box`.
+
+    Attributes:
+        confidence (Number): confidence score between 0-1 for that detection; Default **0.0**
+    """
+
+    def __init__(self):
+        """ x_top_left,y_top_left,width,height are in pixel coordinates """
+        super(Detection, self).__init__()
+        self.confidence = 0.0  # Confidence score between 0-1
+
+    @classmethod
+    def create(cls, obj=None):
+        """ Create a detection from a string or other box object.
+
+        Args:
+            obj (Box or string, optional): Bounding box object to copy attributes from or string to deserialize
+
+        Note:
+            The obj can be both an :class:`~brambox.boxes.annotations.Annotation` or a :class:`~brambox.boxes.detections.Detection`.
+            For Detections the confidence score is copied over, for Annotations it is set to 1.
+        """
+        instance = super(Detection, cls).create(obj)
+
+        if obj is None:
+            return instance
+
+        if isinstance(obj, Detection):
+            instance.confidence = obj.confidence
+        elif isinstance(obj, anno.Annotation):
+            instance.confidence = 1.0
+
+        return instance
+
+    def __repr__(self):
+        """ Unambiguous representation """
+        string = f'{self.__class__.__name__} ' + '{'
+        string += f'class_label = {self.class_label}, '
+        string += f'object_id = {self.object_id}, '
+        string += f'x = {self.x_top_left}, '
+        string += f'y = {self.y_top_left}, '
+        string += f'w = {self.width}, '
+        string += f'h = {self.height}, '
+        string += f'confidence = {self.confidence}'
+        return string + '}'
+
+    def __str__(self):
+        """ Pretty print """
+        string = 'Detection {'
+        string += f'\'{self.class_label}\'{"" if self.object_id is None else " "+str(self.object_id)}, '
+        string += f'[{int(self.x_top_left)}, {int(self.y_top_left)}, {int(self.width)}, {int(self.height)}]'
+        string += f', {round(self.confidence*100, 2)}%'
+        return string + '}'
+
+    def serialize(self, return_dict=False):
+        import json
+
+        serialize_list = [
+            self.class_label,
+            self.object_id,
+            self.x_top_left,
+            self.y_top_left,
+            self.width,
+            self.height,
+            self.confidence,
+        ]
+        if return_dict:
+            return serialize_list
+        else:
+            serialize_str = json.dumps(serialize_list)
+            return serialize_str
+
+    def deserialize(self, serialize_str, input_dict=False):
+        import json
+
+        if input_dict:
+            assert isinstance(serialize_str, dict)
+            serialize_list = serialize_str
+        else:
+            serialize_list = json.loads(serialize_str)
+        self.class_label = serialize_list[0]
+        self.object_id = serialize_list[1]
+        self.x_top_left = serialize_list[2]
+        self.y_top_left = serialize_list[3]
+        self.width = serialize_list[4]
+        self.height = serialize_list[5]
+        self.confidence = serialize_list[6]
+        return True
+
+
+ParserType = b.ParserType
+
+
+class Parser(b.Parser):
+    """ Generic parser class """
+
+    box_type = Detection  # Derived classes should set the correct box_type

scoutbot/loc/transforms/util.py

@@ -0,0 +1,112 @@
+# -*- coding: utf-8 -*-
+#
+#   Lightnet related data processing
+#   Utilitary classes and functions for the data subpackage
+#   Copyright EAVISE
+#
+
+from abc import ABC, abstractmethod
+
+__all__ = ['Compose']
+
+
+class Compose(list):
+    """ This is lightnet's own version of :class:`torchvision.transforms.Compose`.
+
+    Note:
+        The reason we have our own version is because this one offers more freedom to the user.
+        For all intends and purposes this class is just a list.
+        This `Compose` version allows the user to access elements through index, append items, extend it with another list, etc.
+        When calling instances of this class, it behaves just like :class:`torchvision.transforms.Compose`.
+
+    Note:
+        I proposed to change :class:`torchvision.transforms.Compose` to something similar to this version,
+        which would render this class useless. In the meanwhile, we use our own version
+        and you can track `the issue`_ to see if and when this comes to torchvision.
+
+    Ignore:
+        >>> tf = ln.data.transform.Compose([lambda n: n+1])
+        >>> tf(10)  # 10+1
+        11
+        >>> tf.append(lambda n: n*2)
+        >>> tf(10)  # (10+1)*2
+        22
+        >>> tf.insert(0, lambda n: n//2)
+        >>> tf(10)  # ((10//2)+1)*2
+        12
+        >>> del tf[2]
+        >>> tf(10)  # (10//2)+1
+        6
+
+    .. _the issue: https://github.com/pytorch/vision/issues/456
+    """
+
+    def __call__(self, data):
+        for tf in self:
+            data = tf(data)
+        return data
+
+    def __repr__(self):
+        format_string = self.__class__.__name__ + ' ['
+        for tf in self:
+            format_string += '\n  {tf}'
+        format_string += '\n]'
+        return format_string
+
+
+class BaseTransform(ABC):
+    """ Base transform class for the pre- and post-processing functions.
+    This class allows to create an object with some case specific settings, and then call it with the data to perform the transformation.
+    It also allows to call the static method ``apply`` with the data and settings. This is usefull if you want to transform a single data object.
+    """
+
+    def __init__(self, **kwargs):
+        for key in kwargs:
+            setattr(self, key, kwargs[key])
+
+    def __call__(self, data):
+        return self.apply(data, **self.__dict__)
+
+    @classmethod
+    @abstractmethod
+    def apply(cls, data, **kwargs):
+        """ Classmethod that applies the transformation once.
+
+        Args:
+            data: Data to transform (eg. image)
+            **kwargs: Same arguments that are passed to the ``__init__`` function
+        """
+        return data
+
+
+class BaseMultiTransform(ABC):
+    """ Base multiple transform class that is mainly used in pre-processing functions.
+    This class exists for transforms that affect both images and annotations.
+    It provides a classmethod ``apply``, that will perform the transormation on one (data, target) pair.
+    """
+
+    def __init__(self, **kwargs):
+        for key in kwargs:
+            setattr(self, key, kwargs[key])
+
+    @abstractmethod
+    def __call__(self, data):
+        return data
+
+    @classmethod
+    def apply(cls, data, target=None, **kwargs):
+        """ Classmethod that applies the transformation once.
+
+        Args:
+            data: Data to transform (eg. image)
+            target (optional): ground truth for that data; Default **None**
+            **kwargs: Same arguments that are passed to the ``__init__`` function
+        """
+        obj = cls(**kwargs)
+        res_data = obj(data)
+
+        if target is None:
+            return res_data
+
+        res_target = obj(target)
+        return res_data, res_target

scoutbot/scoutbot.py

@@ -0,0 +1,32 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+The lecture materials for Lecture 1: Dataset Prototyping and Visualization
+"""
+import click
+
+
+@click.command()
+@click.option(
+    '--config', help='Path to config file', default='configs/mnist_resnet18.yaml'
+)
+def wic(config):
+    """
+    
+    """
+    pass
+
+
+@click.command()
+@click.option(
+    '--config', help='Path to config file', default='configs/mnist_resnet18.yaml'
+)
+def main(config):
+    """
+    
+    """
+    pass
+
+
+if __name__ == '__main__':
+    main()

scoutbot/utils.py

@@ -5,8 +5,6 @@
 import logging
 from logging.handlers import TimedRotatingFileHandler
 
-import torch
-import yaml
 
 DAYS = 21
 
@@ -71,29 +69,3 @@ def init_logging():
     log = logging.getLogger(name)
 
     return log
-
-
-def init_config(config, log):
-    # load config
-    log.info(f'Using config "{config}"')
-    cfg = yaml.safe_load(open(config, 'r'))
-
-    cfg['log'] = log
-
-    # check if GPU is available
-    device = cfg.get('device')
-    if device not in ['cpu']:
-        if torch.cuda.is_available():
-            cfg['device'] = 'cuda'
-        elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
-            cfg['device'] = 'mps'
-        else:
-            log.warning(
-                f'WARNING: device set to "{device}" but not available; falling back to CPU...'
-            )
-            cfg['device'] = 'cpu'
-
-    device = cfg.get('device')
-    log.info(f'Using device "{device}"')
-
-    return cfg

scoutbot/wic/__init__.py

@@ -2,12 +2,60 @@
 '''
 2022 Wild Me
 '''
-from os.path import abspath
-
+from os.path import join
+import onnxruntime as ort
+from pathlib import Path
+from scoutbot.wic.dataloader import _init_transforms, ImageFilePathList, BATCH_SIZE, INPUT_SIZE
+import numpy as np
+import utool as ut
 import torch
-from torchvision import datasets
-from torchvision.transforms import Compose, Resize, ToTensor
 
 
-def pre(filepath):
-    pass
+PWD = Path(__file__).absolute().parent
+
+ONNX_MODEL = join(PWD, 'models', 'onnx', 'scout.wic.5fbfff26.3.0.onnx')
+ONNX_CLASSES = ['negative', 'positive']
+
+
+def pre(inputs):
+    transform = _init_transforms()
+    dataset = ImageFilePathList(inputs, transform=transform)
+    dataloader = torch.utils.data.DataLoader(
+        dataset, batch_size=BATCH_SIZE, num_workers=0, pin_memory=False
+    )
+
+    data = []
+    for data_, in dataloader:
+        data += data_.tolist()
+
+    return data
+
+
+def predict(data):
+    ort_session = ort.InferenceSession(
+        ONNX_MODEL, 
+        providers=['CPUExecutionProvider']
+    )
+
+    preds = []
+    for chunk in ut.ichunks(data, BATCH_SIZE):
+        trim = len(chunk)
+        while(len(chunk)) < BATCH_SIZE:
+            chunk.append(np.random.randn(3, INPUT_SIZE, INPUT_SIZE).astype(np.float32))
+        input_ = np.array(chunk, dtype=np.float32)
+
+        pred_ = ort_session.run(
+            None,
+            {'input': input_},
+        )
+        preds += pred_[0].tolist()[:trim]
+
+    return preds
+
+
+def post(preds):
+    outputs = [
+        dict(zip(ONNX_CLASSES, pred)) 
+        for pred in preds
+    ]
+    return outputs

scoutbot/wic/dataloader.py

@@ -0,0 +1,99 @@
+import torch
+import torchvision
+import utool as ut
+import numpy as np
+import PIL
+
+
+BATCH_SIZE = 128
+INPUT_SIZE = 224
+
+
+class ImageFilePathList(torch.utils.data.Dataset):
+    def __init__(self, filepaths, targets=None, transform=None, target_transform=None):
+        from torchvision.datasets.folder import default_loader
+
+        self.targets = targets is not None
+
+        args = (filepaths, targets) if self.targets else (filepaths,)
+        self.samples = list(zip(*args))
+
+        if self.targets:
+            self.classes = sorted(set(ut.take_column(self.samples, 1)))
+            self.class_to_idx = {self.classes[i]: i for i in range(len(self.classes))}
+        else:
+            self.classes, self.class_to_idx = None, None
+
+        self.loader = default_loader
+        self.transform = transform
+        self.target_transform = target_transform
+
+    def __getitem__(self, index):
+        """
+        Args:
+            index (int): Index
+        Returns:
+            tuple: (sample, target) where target is class_index of the target class.
+        """
+        sample = self.samples[index]
+
+        if self.targets:
+            path, target = sample
+        else:
+            path = sample[0]
+            target = None
+
+        sample = self.loader(path)
+
+        if self.transform is not None:
+            sample = self.transform(sample)
+
+        if self.target_transform is not None:
+            target = self.target_transform(target)
+
+        result = (sample, target) if self.targets else (sample,)
+
+        return result
+
+    def __len__(self):
+        return len(self.samples)
+
+    def __repr__(self):
+        fmt_str = 'Dataset ' + self.__class__.__name__ + '\n'
+        fmt_str += '    Number of samples: {}\n'.format(self.__len__())
+        tmp = '    Transforms (if any): '
+        fmt_str += '{}{}\n'.format(
+            tmp, self.transform.__repr__().replace('\n', '\n' + ' ' * len(tmp))
+        )
+        tmp = '    Target Transforms (if any): '
+        fmt_str += '{}{}'.format(
+            tmp, self.target_transform.__repr__().replace('\n', '\n' + ' ' * len(tmp))
+        )
+        return fmt_str
+
+
+class Augmentations(object):
+    def __call__(self, img):
+        img = np.array(img)
+        return self.aug.augment_image(img)
+
+
+class TestAugmentations(Augmentations):
+    def __init__(self, **kwargs):
+        from imgaug import augmenters as iaa
+
+        self.aug = iaa.Sequential([iaa.Scale((INPUT_SIZE, INPUT_SIZE))])
+
+
+def _init_transforms(**kwargs):
+    transform = torchvision.transforms.Compose(
+        [
+            TestAugmentations(**kwargs),
+            torchvision.transforms.Lambda(PIL.Image.fromarray),
+            torchvision.transforms.ToTensor(),
+            torchvision.transforms.Normalize(
+                [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
+            ),
+        ]
+    )
+    return transform

setup.cfg

@@ -1,7 +1,8 @@
 [metadata]
 name = scoutbot
 description = The computer vision for Wild Me's Scout project
-long_description = file: README.md
+version = attr: scoutbot.VERSION
+long_description = file: README.rst
 long_description_content_type = text/restructured; charset=UTF-8
 url = https://github.com/WildMeOrg
 author = Wild Me
@@ -17,21 +18,38 @@ packages = find:
 platforms = any
 include_package_data = True
 install_requires =
+    onnxruntime
+    numpy
+    wbia-utool
     torch
-	torchvision
-	Pillow
-	numpy
-	cryptography
-	argparse
-	gradio
+    torchvision
+    opencv-python-headless
+    Pillow
+    imgaug
+    rich
+    tqdm
+    gradio
+    cryptography
+    click
 python_requires = >=3.7
 
+[options.entry_points]
+console_scripts =
+    scoutbot = scoutbot.scoutbot:cli
+    
 [bdist_wheel]
 universal = 1
 
 [aliases]
 test=pytest
 
+[tool:pytest]
+minversion = 5.4
+addopts = -v -p no:doctest --xdoctest --xdoctest-style=google --random-order --random-order-bucket=global --cov=./ --cov-report html -m "not separate" --durations=0 --durations-min=3.0 --color=yes --code-highlight=yes --show-capture=log -ra
+testpaths =
+    scoutbot
+    tests
+
 [options.extras_require]
 test =
     pytest >= 6.2.2

tests/conftest.py

@@ -6,35 +6,30 @@ import pytest
 log = logging.getLogger('pytest.conftest')  # pylint: disable=invalid-name
 
 
-@pytest.fixture()
-def config():
-    return 'scoutbot/configs/mnist_resnet18.yaml'
+# @pytest.fixture()
+# def cfg(config):
+#     from scoutbot import utils
 
+#     log = utils.init_logging()
+#     cfg = utils.init_config(config, log)
 
-@pytest.fixture()
-def cfg(config):
-    from scoutbot import utils
+#     cfg['output'] = 'scoutbot/{}'.format(cfg['output'])
 
-    log = utils.init_logging()
-    cfg = utils.init_config(config, log)
+#     return cfg
 
-    cfg['output'] = 'scoutbot/{}'.format(cfg['output'])
 
-    return cfg
+# @pytest.fixture()
+# def device(cfg):
+#     device = cfg.get('device')
 
+#     return device
 
-@pytest.fixture()
-def device(cfg):
-    device = cfg.get('device')
 
-    return device
+# @pytest.fixture()
+# def net(cfg):
+#     from scoutbot import model
 
+#     net, _, _ = model.load(cfg)
+#     net.eval()
 
-@pytest.fixture()
-def net(cfg):
-    from scoutbot import model
-
-    net, _, _ = model.load(cfg)
-    net.eval()
-
-    return net
+#     return net

tests/test_loc.py

@@ -0,0 +1,95 @@
+# -*- coding: utf-8 -*-
+import onnx
+from os.path import exists, join, abspath
+
+
+def test_loc_onnx_load():
+    from scoutbot.loc import ONNX_MODEL
+
+    model = onnx.load(ONNX_MODEL)
+    assert exists(ONNX_MODEL)
+
+    onnx.checker.check_model(model)
+
+    graph = onnx.helper.printable_graph(model.graph)
+    assert graph.count('\n') == 107
+
+
+def test_loc_onnx_pipeline():
+    from scoutbot.loc import pre, predict, post, INPUT_SIZE
+
+    inputs = [
+        abspath(join('examples', '0d01a14e-311d-e153-356f-8431b6996b84.true.jpg')),
+    ]
+
+    assert exists(inputs[0])
+
+    data, sizes = pre(inputs)
+
+    assert len(data) == 1
+    assert len(data[0]) == 3
+    assert len(data[0][0]) == INPUT_SIZE[0]
+    assert len(data[0][0][0]) == INPUT_SIZE[1]
+    assert sizes == [(256, 256)]
+
+    preds = predict(data)
+
+    assert len(preds) == 1
+    assert len(preds[0]) == 30
+
+    outputs = post(preds, sizes)
+
+    assert len(outputs) == 1
+    assert len(outputs[0]) == 5
+
+    # fmt: off
+    targets = [
+        {
+            'class_label': 'elephant_savanna',
+            'x_top_left': 206.00893930,
+            'y_top_left': 189.09138371,
+            'width'     :  53.78145658,
+            'height'    :  66.46106896,
+            'confidence':   0.77065581,
+        },
+        {
+            'class_label': 'elephant_savanna',
+            'x_top_left': 216.61065204,
+            'y_top_left': 193.30525090,
+            'width'     :  42.83404541,
+            'height'    :  62.44728440,
+            'confidence':   0.61152166,
+        },
+        {
+            'class_label': 'elephant_savanna',
+            'x_top_left':  51.61210749,
+            'y_top_left': 235.37819260,
+            'width'     :  79.69709660,
+            'height'    :  17.41258826,
+            'confidence':   0.50862342,
+        },
+        {
+            'class_label': 'elephant_savanna',
+            'x_top_left':  57.47630427,
+            'y_top_left': 236.92587515,
+            'width'     :  94.69935960,
+            'height'    :  16.03246718,
+            'confidence':   0.44841822,
+        },
+        {
+            'class_label': 'elephant_savanna',
+            'x_top_left':  37.07233605,
+            'y_top_left': 230.39122596,
+            'width'     : 105.40560208,
+            'height'    :  24.81017362,
+            'confidence':   0.44012001,
+        },
+    ]
+    # fmt: on
+
+    for output, target in zip(outputs[0], targets):
+        for key in target.keys():
+            if key == 'class_label':
+                assert getattr(output, key) == target.get(key)
+            else:
+                assert abs(getattr(output, key) - target.get(key)) < 1e-6

tests/test_model.py

@@ -1,25 +0,0 @@
-# -*- coding: utf-8 -*-
-import torch
-from PIL import Image, ImageOps
-from torchvision.transforms import Compose, Resize, ToTensor
-
-
-def test_architecture_params(net):
-    total_params = sum(params.numel() for params in net.parameters())
-    assert total_params == 133578
-
-
-def test_model_prediction(cfg, device, net):
-    image = Image.open('examples/example_1.jpg')
-
-    image = ImageOps.grayscale(image)
-
-    transforms = Compose([Resize(cfg['image_size']), ToTensor()])
-    image = transforms(image).unsqueeze(0)
-    data = image.to(device)
-
-    with torch.no_grad():
-        prediction = net(data)
-
-    prediction = torch.argmax(prediction[0], dim=0).item()
-    assert prediction == 5

tests/test_wic.py

@@ -0,0 +1,49 @@
+# -*- coding: utf-8 -*-
+import onnx
+from os.path import exists, join, abspath
+
+
+def test_wic_onnx_load():
+    from scoutbot.wic import ONNX_MODEL
+
+    model = onnx.load(ONNX_MODEL)
+    assert exists(ONNX_MODEL)
+
+    onnx.checker.check_model(model)
+
+    graph = onnx.helper.printable_graph(model.graph)
+    assert graph.count('\n') == 1334
+
+
+def test_wic_onnx_pipeline():
+    from scoutbot.wic import pre, predict, post, ONNX_CLASSES, INPUT_SIZE
+
+    inputs = [
+        abspath(join('examples', '1e8372e4-357d-26e6-d7fd-0e0ae402463a.true.jpg')),
+    ]
+
+    assert exists(inputs[0])
+
+    data = pre(inputs)
+
+    assert len(data) == 1
+    assert len(data[0]) == 3
+    assert len(data[0][0]) == INPUT_SIZE
+    assert len(data[0][0][0]) == INPUT_SIZE
+
+    preds = predict(data)
+
+    assert len(preds) == 1
+    assert len(preds[0]) == 2
+    assert preds[0][1] > preds[0][0]
+    assert abs(preds[0][0] - 0.00001503) < 1e-6, str(preds)
+    assert abs(preds[0][1] - 0.99998497) < 1e-6
+
+    outputs = post(preds)
+
+    assert len(outputs) == 1
+    output = outputs[0]
+    assert output.keys() == set(ONNX_CLASSES)
+    assert output['positive'] > output['negative']
+    assert abs(output['negative'] - 0.00001503) < 1e-6
+    assert abs(output['positive'] - 0.99998497) < 1e-6