Wildbook IA Overview
Wildbook Image Analysis (WBIA or Wildbook IA) is a computer vision program for analyzing photos of animals and answering important biological questions. It aims to detect animals in photographs, label their species, and identify which individual animals they are; answering who, what, and where in wildlife images. WBIA contains tools to answer these questions as well as a structured database to store images and their derived data.
This project is the machine learning and computer vision component of the Wildbook project. WBIA began as a fork of the IBEIS (Image Based Ecological Information System) software suite for wildlife conservation, which is maintained by Jon Crall (@Erotemic) at https://github.com/Erotemic/ibeis. The IBEIS toolkit was originally a wrapper around HotSpotter, an individual identification algorithm authored by Jon Crall that we still use to identify species such as zebras and humpback whales.
WBIA is build around an SQLite database, a web GUI, and matplotlib visualizations, and it communicates with Wildbook via http requests. Algorithms employed include convolutional neural networks for detection, localization, and classification; hessian-affine keypoint detection; SIFT keypoint description; LNBNN identification using approximate nearest neighbors; and even simple decision trees. Optional plugins provide features such as species-specific identification algorithms; their design is described on our WBIA Plugin Integration page.
Requirements#
- Python 3.5+
- OpenCV 3.4.10
- Python dependencies listed in requirements.txt in https://github.com/WildMeOrg/wildbook-ia
Installation Instructions#
PyPI#
The WBIA software is now available on pypi for Linux systems. This means if you have Python installed. You can simply run:
to install the software. Then the command to run the GUI is:
We highly recommend using a Python virtual environment: https://docs.python-guide.org/dev/virtualenvs/#lower-level-virtualenv
Docker#
The WBIA software is built and deployed as a Docker image wildme/wbia. You can download and run the pre-configured instance from the command line using:
This image is built using the multi-stage Dockerfiles in devops/.
Source#
This project depends on an array of other repositories for functionality.
First Party Toolkits (Required)
First Party Dependencies for Third Party Libraries (Required)
First Party Plugins (Optional)
Plugin Templates (Reference--see also the WBIA Plugin Integration page)
Miscellaneous (Reference)
Documentation#
The package documentation is built and available online at wildmeorg.github.io/wildbook-ia/. To build the docs locally,
The WBIA documentation style uses a modified version of Sphinx doctests for all documentation and testing. The ability to write good documentation directly in the header of the function is of high value to any contributor to the WBIA code base and any plugin maintainer; please document your contributions!
Code Style and Development Guidelines#
Contributing#
It's recommended that you use pre-commit to ensure linting procedures are run on any commit you make. (See also pre-commit.com)
Reference pre-commit's installation instructions for software installation on your OS/platform. After you have the software installed, run pre-commit install on the command line. Now every time you commit to this project's code base the linter procedures will automatically run over the changed files. To run pre-commit on files preemtively from the command line use:
Brunette#
Our code base has been formatted by Brunette, which is a fork and more configurable version of Black (https://black.readthedocs.io/en/stable/).
Flake8#
Try to conform to PEP8. You should set up your preferred editor to use flake8 as its Python linter, but pre-commit will ensure compliance before a git commit is completed.
To run flake8 from the command line use:
This will use the flake8 configuration within setup.cfg,
which ignores several errors and stylistic considerations.
See the setup.cfg file for a full and accurate listing of stylistic codes to ignore.
All WBIA Python code requires a Linter to be run on any code contributions for the main code base. While we do not explicitly require this for WBIA plugins, we STRONGLY suggest using one for all Python code. We do acknowledge, however, that the full Python Flake8 (http://flake8.pycqa.org/en/latest/) specification is quire restrictive. We have a set of allowed errors and warnings to ignore, below:
Writing good Python code is subjective but a linter will help to make all Python contributors follow a consistent set of cleanliness standards.
PyTest#
Our code uses Google-style documentation tests (doctests) that uses pytest and xdoctest to enable full support. To run the tests from the command line use:
To run doctests with +REQUIRES(--web-tests) do:
The immediate benefit of this structure is that documentation can live alongside parameter specifications, REST API endpoints, and unit testing code block(s). To run a given test code block, one must simply tell python to execute the module and pass in the function name and the test index. For example, if you want to run the first code test for the function ibs.wbia_plugin_id_hello_world() in this file, you can call
Note the :0 index specifier at the end of this Command Line call. To run all of the tests for a specified function, you must remove any post-fix. For example,
will run all of the tests for that function. To run all tests for an entire file, you can simply call:
We also provide a handy script at the top level path for this repository called run_tests.py that will execute all of the tests for all files. A summary report timeings.txt and failed_doctests.txt will be created for each run.
When run in bulk, you may with to disable a specific test from the set of all tests you end up writing. You may do this to improve the speed of the batch test call or to exclude tests that are currently a work-in-progress or are intended for storing notes and helpful code snippets. You can add the first-line macro ENABLE_DOCTEST or DISABLE_DOCTEST to enable or disable the test.
Python Code Profiling (ut.inject2)#
On top of a large suite of tools, utool (referred to as ut) also offers profiling functions for code segments. To see this in action, run any Python CLI command (e.g. a test) with the extra CLI parameter --profile at the end. Any function that has the @profile decorator on the function will be profiled for run-time efficiency. For example, running from the CLI:
Will run the test of downloading an image from a remote server, check a local copy, delete it, then re-download the image. The output of this call will look something like this:
This output is indicating that the function tool 0.14 seconds to finish. Inspecting the file profile_output.txt shows:
This output tells us that we spent 0.1369 seconds and 99.9% of our total run-time in this function downloading the file on line 587. This output is very helpful for any developer wishing to optimize the run-time performance of their code.
WBIA Controller and Database#
WBIA supports dynamically-injected methods to the controller object. Any function that you wrap with the @register_ibs_method decorator is automatically added to the controller as ibs.<function_name>() method. The function then becomes accessible anywhere in the codebase if one has access to the WBIA controller.
For example,
will be accessible at ibs.new_plugin_function('test') and will return the value test for testdb_identification when executed if the database's containing folder is named testdb_identification.
Database Structure#
A WBIA database, at its core, is simply a folder on the local file system. WBIA uses SQLite3 and static folders for all of its database and asset storage and has the following structure:
The volatile (i.e. cannot be re-computed) database files and folders are:
| Path | Description |
|---|---|
_wbia_database.sqlite3 | The primary WBIA database. This database contains many tables to store image, annotation, name, asset properties and store various decisions and computed results. |
_wbia_staging.sqlite3 | A staging database for semi-temporary results, content in this database is intended to eventually be committed or resolved into the primary database |
images/ | A folder containing all of the localized images for the datbase. Images are renamed when copied to this folder to be <UUID>.<EXT>. The image's UUID is computed deterministically based on the pixel content of the image and we inherit any original file extensions to prevent loosing any metadata. |
We also have extensive caching processes for different computer vision and machine learning results along with modified versions of the original assets. All cached data is stored in the _ibsdb/_wbia_cache/ folder. This folder can be deleted freely at any point to delete any pre-computed results. Conversely, we also take daily snapshots of the primary and staging database whenever the WBIA controller is first started. These backups are stored in _ibsdb/_wbia_backups/. The WBIA controller also supports incremental updates, so a database backup is also performed before any database update.
We provide an example below on how to write a customized database controller along with example incremental update functions.
First-order Data Objects#
There are 5 main data constructs in WBIA:
| Object | Prefix | Description |
|---|---|---|
| Image | GID | Original images provided by the user or contributor. ImageSets and Images have a many-to-many relationship as more than one image can be in an ImageSet and an image can be a member of multiple ImageSets. |
| Annotation | AID | Pixel regions within an Image to designate a single animal. Annotations are, in their most basic form, a bounding box. Bounding boxes in WBIA are parameterized by (xtl, ytl, w, h) where xtl designates "x pixel coordinate for the top-left corner", ytl designates "y pixel coordinate for the top-left corner", w designates "the width of the box in pixel coordinates", and h designates "the height of the box in pixel coordinates". Annotations can also have a theta parameter describing rotation of the bounding box in cases where the region of interest is not parallel to the image boundaries, but annotations are always rectangular. Images and Annotations have a one-to-many relationship. |
| Part | PID | A sub-region of an Annotation that designates a part of that animal (e.g. head, tail, fluke). The Part object has almost 100% feature parity with Annotations except it has an additional parent Annotation attribute. Parts are not the same as Annotations, and most algorithms do not accept Parts natively. |
| Name | NID | A ID label for an annotation. A one-to-many relationship between Names and Annotations is usually the end-result of using the WBIA system. |
| ImageSet | IMGSETID | A collection of images into a single group. This grouping is used as a very general association and can indicate, for example, the set of images taken at the same time and place or the images that all contain a target species or the images that are to be used by a machine learning algorithm for training or testing. |
In general, a single instance of the WBIA code base only has one WBIA controller (commonly referred to by the variable name ibs) and is the primary development handle for any new features. The controller is packed with very handy functions, including:
These lists of internal rowids allow for a wide range of adders, getters, setters, deleters and algorithm calls. For example,
In general, if you think a function should exist to associate two object types or compute some value, it probably already exists.
WBIA API#
Controller API Injection#
Any function that is decorated by @register_ibs_method should accept the WBIA ibs controller object as the first parameter.
IMPORTANT: To be enabled, a plugin must be manually registered with the WBIA controller. The registration process involves adding the module import name for this _plugin.py file to a list of plugins. This list lives in the main WBIA repository at:
The plugin must be added to the list with variable name AUTOLOAD_PLUGIN_MODNAMES at the top of the file. On load, the WBIA controller will add its own injected controller functions and will then proceed to add any external plugin functions. The injection code will look primarily for any functions with the @register_ibs_method decorator, but will also add any of the decorators described below for web requests.
WBIA supports a web-based REST API that allows for local functions to be called from public end-points. The POST and GET arguments that are passed to the web server by the client are automatically parsed into Python-valid objects using a customized JSON converter. Any responses from API calls are also wrapped by a JSON encoding and thus also need to be serialize-able.
Any function that you wish to expose to the web simply needs a @register_api decorator added above the function's definition (after any @register_ibs_method) decorators. This decorator takes in the endpoint and the allowed HTTP method(s) (e.g. GET, POST, PUT, DELETE) that are allowed. The same REST endpoint can point to different functions through the specifications of different methods.
WBIA REST API#
There already exists an extensive REST API that mirrors the existing Python API, but is a curated subset of useful functions. For example, the Python API function
has a mirrored REST API interface at the endpoint
and returns a JSON-formatted response with the same contents for gps_list.
Alternatively, a user can opt to expose an endpoint that does not apply any JSON response serialization. Specific examples of this include serving HTML pages, raw image bytes, CSV or XML downloads, redirects, HTTP errors or any other non-JSON response. Functions that are decorated with @register_route also benefit from the same parameter JSON serialization as @register_api.
WBIA Web Interface#
WBIA also supports a basic web interface with viewing and curation tools. For example, the routes /view/imagesets/, /view/images/, /view/annotations/, and /view/parts/ allow for a user to quickly see the state of the database. There is also a batch uploading function to easily add a handful of images to a database without needing to use the full Python or REST APIs for larger imports.
The web tools for turking (borrowing the phrase from Amazon Mechanical Turk) support a wide range of helpful functions. The most helpful interface is arguably the annotation bounding box and metadata interface. This interface can be viewed at /turk/detection/?gid=X and shows the current annotations for Image RowID=X, where they are located, what metadata is set on them, what parts (if any) have been added to the annotations, and what metadata is on the parts.
Background API Job Engine#
The REST API has a background job engine. The purpose of the job engine is to preserve the special properties of a responsive and state-less web paradigm. In short, the WBIA web controller will automatically serialize concurrent requests but some API calls involve a very long processing delay. If a long API call is called in the main thread, it will deny any and all simultaneous web calls from resolving. The solution is to mode any long API calls that are exposed by the web interface to the job engine to promote responsiveness and reduce or eliminate client-side web time outs. Any API that uses the job engine (as shown in an example below) should return a job UUID and should be prefixed with /api/engine/plugin/<Plugin Name>/.../.../ to differentiate it from an instantaneous call. We also recommend having any engine calls accept the POST HTTP method by default.
A user or application can query on the status of a background job, get its original request metadata, and retrieve any results. The job engine supports automatic callbacks with a specified URL and HTTP method when the job is complete.
WBIA Dependency Graph#
WBIA supports a dependency-graph cache computation structure for easy pipelining. This is similar to other frameworks (e.g. Luigi) but has been customized and implemented by hand to resolve assets consistent with the WBIA database.
In general, a dependency cache (referred to by depc in the code base) is a coding tool to eliminate the complexity of managing state and staged pipeline configurations. For example, say we want to compute the sum and product of a cryptographically secure hash of an image's pixel data. We may want to build a dependency graph that looks like this
To compute the HashSum on an Image, we first need to compute a Hash on the Image. The dependency cache will preserve this ordering by computing and storing intermediate results for use by the depc node you want to retrieve. Therefore, we would expect that any call to HashSum for Image RowID=10 would also compute the Hash for Image RowID=10. Subsequently calling the call for HashProd on Image RowID=10 (with the same configuration) will simply retrieve the pre-computed Hash for RowID=10.
The WBIA dependency cache infrastructure designates three ROOT object types for this type of computation: Images, Annotations, and Parts. There exists three parallel decorators that allow one to make a new function for the appropriate dependency graph tree. Adding a new node to the graph requires writing a function with the correct inputs and a specific configured decorator. We provide a working example below on how to write the decorator for Hash and HashSum.
We provide an example below on how to create your own custom depc and decorator function
Miscellaneous notes#
Note 1: The WBIA code base has a constants file for a lot of convenient conversions and names of constructs. This constants module also keeps track of very convenient environment variables:
When PRODUCTION is set to True, please observe a restraint in resource utilization for system memory, number of concurrent threads, and GPU memory.
Note 2: We suggest to use interactive embedding with utool.embed() whenever and wherever possible. The use of ut.embed() (we commonly import utool with the shorthand namespace of ut) is used throughout the WBIA code base and is supremely helpful when debugging troublesome code. We have set an example below that uses ut.embed() in the ibs.wbia_plugin_id_hello_world() documentation. We highly recommend calling this function's example test and play around with the ibs controller object. The ibs controller supports TAB completion for all method functions. For example, when in the embedded iPython terminal, you can input:
This embedded terminal shows all of the WBIA functions that start with the prefix ibs.get_image_p, for example ibs.get_image_paths(). If you are unsure about the API specification for this function, you can ask help from Python directly in the embedded session.
The embedded session will dump out the doctest (hopefully with parameter and example usage documentation) for that controller function.
Citation#
If you use this code or its models in your research, please cite:
Conclusion#
Support for WBIA is maintained by Wild Me.
Wild Me is a non-profit located in Portland, OR, USA.
Please refer any questions to dev@wildme.org or https://github.com/WildMeOrg/wildbook-ia .