Adding the README info

README.md

@@ -11,3 +11,398 @@ short_description: Helps train SR Models
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+
+* Introduction
+
+  The SyNet repository is a library for developing networks for
+  Synaptics vision chips.  It consists of PyTorch model components
+  which can be exported to TensorFlow and tflite without an
+  intermediate export backend like ONNX.  The resulting tflite files
+  are clean, and respect chip memory constraints.  The aim of SyNet is
+  to streamline the process of generating trained models to deploy on
+  Synaptics chips, for internal and external use.
+
+  In addition to model definitions, analysis, data manipulation, and
+  data analysis tools, SyNet also aims to support several "backends".
+  For now, the only backend supported in Ultralytics.  Ultralytics
+  provides a suite of tools for training vision models and visualizing
+  those models.  Using Ultralytics as a backend, you can generate
+  trained vision models optimized for our chips.
+
+  This code is GPL, and the Ultralytics backend is AGPL, but the
+  output of this code are tflite files.  These output tflite files are
+  data that is not covered by the copyright on the code of either code
+  base.  Consequently, the respective licenses of the code do not
+  apply to the output tflite files.
+
+* Performance
+
+  The following table summarizes model performance on the person class
+  of the COCO dataset for four major computer vision tasks.  All of
+  these models run on VGA (640x480) resolution at about 10fps on the
+  Sabre A0 chip.
+
+  | Task                  | Score | Metric        | Data                                                 |
+  |-----------------------+-------+---------------+------------------------------------------------------|
+  | Classification        | 0.945 | Top1 accuracy | Person Visual Wake Words with standard minival split |
+  | Object Detection      | 0.730 | Box AP50      | COCO detection subset used by Ultralytics     |
+  | Pose Estimation       | 0.729 | Pose AP50     | COCO keypoint subset used by Ultralytics       |
+  | Instance Segmentation | 0.631 | Mask AP50     | COCO segmentation subset used by Ultralytics   |
+
+* Roadmap
+
+** Current Features
+
+   - Models optimized for Sabre
+     - Memory and compute efficient model components
+   - Usable with PyTorch and TensorFlow training libraries
+   - Usable from the command line or python environment
+   - In-model demosaicing export option
+     - faster than demosaic hardware block
+     - increases available weight memory by >2x
+   - Includes slim tflite runtime utilities (for demos)
+   - Has the ability to support training backends
+     - Main backend is Ultralytics
+     - Currently supports all core visions tasks from Ultralytics
+       - Object Detection
+       - Pose Estimation
+       - Instance Segmintation
+       - Classification
+     - Supports evaluating tflites through Ultralytics
+     - Allows for easy setup of laptop demos
+       - quickly run and view model running on webcam
+   - Includes more advanced custom tflite evaluation (not through
+     Ultralytics).
+     - Computes combined and per-dataset statistics of model
+       performance.
+
+** Planned for later releases
+
+   - Models optimized for VS680
+   - Automatic model selection from zoo
+     - Select training resolution, inference resolution, and heads.
+   - Dataset manipulation tools
+     - subsample and combine classes for embedded application
+     - camera augmentations
+   - Enable arbitrary addition of box attributes to regress
+     - age, orientation (pitch, yaw, roll)
+   - Complete miscelaneous tasks (see corresponding Gitlab milestone)
+   - Mixed precision support
+
+** Future Research
+
+   - Hugging Face backend integration
+
+* Installation
+
+  In more complex setups, you should create a virtual environment:
+  https://docs.python.org/3/library/venv.html.  In the following
+  examples, we include the Ultralytics backend by adding '[ultra]'.
+  To install via pip:
+
+  #+begin_src shell
+    pip install "synet[ultra] @ git+ssh://git@gitlab.synaptics.com/wssd-ai-algorithms/synet-fork.git"
+  #+end_src
+
+  or if you have cloned to a local copy of the repository:
+
+  #+begin_src shell
+    pip install [-e] "/PATH/TO/LOCAL/SYNET[ultra]"
+  #+end_src
+
+  where '-e' will allow you to make edits to your local clone after
+  install.  If the install fails in any way, please report this to us.
+  In this case, you can use the exact library versions used to produce
+  the results in the Performance section above using the following
+  (requires CUDA 11.8)
+
+  #+begin_src shell
+    pip install -r /PATH/TO/LOCAL/SYNET/requirements-11.8.txt "/PATH/TO/LOCAL/SYNET[ultra]"
+  #+end_src
+
+* Quickstart
+
+  In the following, I give a simple example of how one can train a model on COCO, quantize to tflite, and benchmark that tflite on a custom dataset specified by a user's CUSTOM_DATA.yaml.  First, train the model:
+  #+begin_src shell
+    synet ultralytics train model=sabre-detect-vga.yaml data=coco.yaml
+  #+end_src
+  Quantize the trained model.
+  #+begin_src shell
+    synet quantize --backend ultralytics --tflite runs/train/detect/weights/best.pt --data /path/to/coco.yaml
+  #+end_src
+  Evaluate that trained and quantized model.
+  #+begin_src shell
+    synet ultralytics val model=runs/train/detect/weights/best.tflite task=detect data=coco.yaml
+  #+end_src
+  If you have a custom evaluation dataset, you can evaluate on that (e.g. test split) as well
+  #+begin_src shell
+    synet ultralytics val model=runs/train/detect/weights/best.tflite split=test task=detect save_txt=True save_conf=True data=CUSTOM_DATA.yaml
+  #+end_src
+  And finally generate metrics for the model performance, especially at the .95 precision operating point.
+  #+begin_src shell
+    synet metrics CUSTOM_DATA.YAML --out-dirs runs/detect/val --project runs/detect/val --precision .95
+  #+end_src
+
+* Core Shell API
+
+  The basic syntax for running SyNet from a shell is:
+
+  #+begin_src shell
+    synet [entrypoint] [entrypoint specific args]
+  #+end_src
+
+  Where entrypoint can be a native SyNet module, or a backend like
+  ultralytics.  For instance:
+
+  #+begin_src shell
+    synet ultralytics train ...
+    synet quantize --backend ultralytics ...
+  #+end_src
+
+  Notice that while some backends are callable this way, the backend
+  may also need to be specified for other modules.  For instance,
+  synet.quantize needs to know with which backend to load the model.
+
+  For information on training/visualizing models, see the section on
+  backends below.
+  
+** Quantize
+
+   The SyNet repository includes the ability to quantize models
+
+   #+begin_src shell
+     synet quantize --backend BACKEND --weights MODEL_PT_SAVE --data REP_DATA
+   #+end_src
+
+   For instance, running:
+
+   #+begin_src shell
+     synet quantize --backend ultralytics --weights ./exp/weights/best.pt --data /PATH/TO/CUSTOM_DATASET.YAML --image-shape 480 640
+   #+end_src
+
+   will create a tflite at ./exp/weights/best.tflite with input shape
+   [480, 640].  The image shape will default to whatever the model is
+   designed to take, but can be overrided in this way.  You may also
+   specify a model yaml like so:
+
+   #+begin_src shell
+     synet quantize --backend ultralytics --cfg sabre-detect-qvga.yaml
+   #+end_src
+
+   This will place a quantized model at ./sabre-detect-qvga.tflite.
+   This will let you inspect the architecture, though it will not be a
+   trained model, so the model output will be useless.  For more
+   information see:
+
+   #+begin_src shell
+     synet quantize --help
+   #+end_src
+
+** Metrics
+
+   SyNet's metrics code is an advanced model benchmarking tool which
+   allows the user to simultaneously score object detection on
+   multiple datasets.  The benefit of doing multiple datasets is that
+   it can find a confidence threshold by applying a precision
+   threshold to the combined data.  This global operating point is
+   then applied to each dataset individually.  Plots are generated
+   showing the mAP curves for each class, each dataset, the combined
+   dataset, and combined classes.  Additionally, on each curve, the
+   global precision point, the dataset precision point, and the .5
+   confidence point are plotted.  The exact coordinates and
+   confidences of each point are printed.  The basic usage is:
+
+   #+begin_src shell
+     synet metrics DATA1.YAML DATA2.YAML... --out-dirs OUT_AIR1 OUT_DIR2... --project PLOT_DIR  --precisions PRECISION...
+   #+end_src
+
+   There must be one data yaml for each dataset, and they are expected
+   to be in Ultralytics format:
+   https://docs.ultralytics.com/datasets/?h=data#steps-to-contribute-a-new-dataset
+
+   If present, the 'test' data split is used.  Otherwise, the 'val'
+   split is used for each dataset.  The metrics code does not actually
+   run the model, but instead uses the output from running the model
+   via a different code, hence the "OUT_DIR" is the output directory
+   of that other code.  This may be changed in the future, but
+   currently you should populate the out dir with the only supported
+   backend:
+
+   #+begin_src shell
+     synet ultralytics val model=/PATH/TO/BEST.TFLITE split=test imgsz=HEIGHT,WIDTH data=DATA1.YAML task=detect save_txt=True save_conf=True
+   #+end_src
+
+   See notes on validation in the ultralytics backend section below.
+   For more information on the metrics code see:
+
+   #+begin_src shell
+     synet metrics --help
+   #+end_src
+
+* Core Python API
+
+** Base Layers
+
+*** Converting to Keras/TensorFlow
+
+    SyNet exists to be the glue between State of the Art training, and
+    our chips.  Each model component knows how to "export itself" to a
+    Keras/TensorFlow model.  This done approximately like so:
+
+    #+begin_src python
+      from keras import Input, Model
+      from synet.base import askeras
+      model = ...
+      inp = Input(...)
+      with askeras:
+          kmodel = Model(inp, model(inp))
+    #+end_src
+
+    This method works so long as only SyNet blocks operate directly on
+    the input.  For a more complex example, see quantize.py.
+
+* Backends
+
+  For now, the only backend supported is Ultralytics.
+
+** Ultralytics
+
+   Any Ultralytics function (train, predict, val, etc.) will run
+   through SyNet with SyNet modules.  The basic shell syntax is:
+
+   #+begin_src shell
+     synet ultralytics [ultralytics ARGS]...
+   #+end_src
+
+   This performs 3 SyNet-specific operations, then passes off
+   execution to the normal Ultralytics code entrypoint:
+   - Copy the model config from the synet zoo (synet/zoo/ultralytics) if necessary.
+   - Set the imgsz (image size) ultralytics parameter according to the
+     model specification.
+   - Apply patches to the Ultralytics modules where necessary to
+     enable proper SyNet model loading within Ultralytics.
+   If you need to use this backend through python (instead of a
+   shell), then the only necessary step is to apply the patches as in
+   the following snippet:
+
+   #+begin_src python
+     from synet.backends import get_backend
+     get_backend('ultralytics').patch()
+   #+end_src
+
+   After this point, you are free to use SyNet models and tflites
+   using the normal Ultralytics API, but do not try to use
+   Ultralytics' "export" functionality to deploy to Sabre.  Use
+   SyNet's quantize instead.  The resulting models will not be
+   properly optimized and are not expected to run on our chips.
+
+   We give some examples/explanations for basic Ultralytics usage
+   here, but for any further questions about Ultralytics, you should
+   consult the Ultralytics github page and documentation:
+   - [[https://github.com/ultralytics/ultralytics]]
+   - https://docs.ultralytics.com/
+
+*** Train
+
+    The SyNet repository provides a thin wrapper around Ultralytics
+    training for simple training situations.  The basic usage is
+
+    #+begin_src shell
+      synet ultralytics [OTHER ULTRALYTICS ARGS]
+    #+end_src
+
+    For instance, if you want to train a person detect model, you
+    can train a VGA (640x480) model for the sabre chip with.
+
+    #+begin_src shell
+      synet ultralytics train model=sabre-detect-vga.yaml data=coco.yaml
+    #+end_src
+
+    This will put all output at ./runs/train/exp.  See "name",
+    "project" and "exists-ok" in the Ultralytics docs for changing
+    this.  The above command also tries to download the coco dataset
+    to ../datasets.
+
+    For any further information, see the ultralytics documentation for
+    training: https://docs.ultralytics.com/modes/train
+
+*** Validation
+
+    Validation will be performed during training, but only on the
+    validation set, and only with the floating point (non-quantized)
+    model.  In order to use ultralytics to run validation on your
+    quantized (.tflite) model, you will need to specify the model, the
+    task, the dataset split, and the canvas size.  Additionally, if
+    you want to use SyNet's advanced metrics tools, you should be sure
+    to cache the results of model evaluation by passing 'save_txt' and
+    'save_conf' like so:
+
+    #+begin_src shell
+      synet ultralytics val model=runs/train/detect/weights/best.tflite split=val task=detect save_txt=True save_conf=True imgsz=640,480 data=coco.yaml
+    #+end_src
+
+    This should place the results of model evaluation in
+    runs/val/detect, which you can point to when calling "synet
+    metrcis" (see above).  For more information, see the ultralytics
+    documentation for validation:
+    https://docs.ultralytics.com/modes/val
+
+*** Predict (for demos)
+
+    You can use Ultralytics' Predict to infer the model on an input
+    and optionally generate visualizations.  For example, you can see
+    the results of the model on your webcam stream with:
+
+    #+begin_src shell
+      synet ultralytics predict model=vga/detect/finetuned.tflite source=0 imgsz='[480,640]' show=True iou=.3 conf=.5
+    #+end_src
+
+    Breaking this apart: You are calling SyNet with the ultralytics
+    backend in predict mode.  You are passing predict the path to your
+    model (tflite in this case), telling it to run from a webcam
+    (undocumented in Ultralytics, but this is source=0), setting the
+    image shape (ultralytics cannot infer image shape from tflite),
+    telling it to generate a graphical display, and specifying iou and
+    confidence thresholds.  For more information, see the ultralytics
+    documentation: https://docs.ultralytics.com/modes/predict
+
+* Contributing
+
+** Test Suite
+
+   Please run the test suite before pushing ANY changes upstream.  To
+   do so, ensure that you have the development dependencies by
+   installing synet with the [dev] set of optional dependencies.
+
+   #+begin_src shell
+     pip install -e ...synet[dev]
+   #+end_src
+
+   Then run the following in the synet root folder (the directory
+   containing the "synet" folder):
+
+   #+begin_src shell
+     pytest -v
+   #+end_src
+
+   If you notice that a bug is present despite the tests passing,
+   please consider adding an appropriate test case in the 'tests'
+   folder: https://docs.pytest.org/en/latest/getting-started.html
+
+** Docstring Style
+
+   Docstrings conform to numpy, scipy, and scikits docstring
+   conventions: https://numpydoc.readthedocs.io/en/latest/format.html
+
+** Imports
+
+   Only quantize.py and tflite_utils.py should import TensorFlow at
+   the top of the file.  Otherwise, TensorFlow modules should be
+   imported at the beginning of functions where they are used.  This
+   ensures TensorFlow is only loaded when strictly necessary.
+
+   Only backends/ultralytics.py should directly import anything from
+   ultralytics, and backends.ultralytics should only be accessed by
+   obtaining the ultralytics backend from backends.get_backend().:w
+