Wildbook IA (WBIA) allows easy extensibility with plugins, python packages built to add specific functionality to the WBIA software. This document is a guide to plugin development & integration and assumes the reader is already familiar with the overview of the WBIA software and python packages in general.
As a refresher, WBIA contains computer vision tools to detect and identify animals in photographs, as well as a structured database to store those photos and their derived data. WBIA contains core algorithms to answer "what species is this" and "what individual is this," but is most often used with additional plugins that are specialized for particular species or research questions. Plugins might integrate new algorithms into the established WBIA pipeline, such as wbia-plugin-finfindr which is an individual identification algorithm optimized for whale and dolphin dorsal fins; or add entirely new steps to that pipeline, such as wbia-plugin-2d-orientation which rotates annotations so they fit animal bodies more closely as part of the detection process.
Much can be gleaned from our wbia-plugin-id-example. wbia-plugin-finfindr is also a fairly straightforward yet full-featured ID algorithm integration and will be used as an example throughout this document.
For resource usage and maintenance reasons, the ideal plugin runs natively in python on the wbia Docker container. When this is not feasible however, an easy route for integrating other software is to containerize your software in its own separate container and write a wbia plugin that calls that container via http requests. This is the architecture of the wbia-plugin-deepsense and finFindR plugins.
WBIA plugin names look like
wbia-plugin-X. The plugin should have its own git repo with that name which follows python package guidelines. In the
wbia-plugin-X folder, plugin code belongs in a subfolder
wbia_X (note the switch from hyphens to underscores). The core plugin logic and api should be defined in that subfolder in
Example plugin architecture:
While not required, we STRONGLY suggest all API endpoints (see the web API section in the WBIA overview) in a plugin in to be prefixed with
/api/plugin/<Plugin Name>/.../.../ to keep the registered APIs from conflicting. We also suggest doing the same with your function names as well that are injected into the WBIA controller on load, for example using a function name of
Adding new individual ID algorithms to WBIA requires only one key method be defined in your _plugin.py, of course being the method that performs identification. Our convention is that this method has the same name as the plugin itself. Let's look at
wbia-plugin-finfindr as an example (doctests and comments removed for brevity):
Here are some key points we can see above:
- The method is integrated with our dependency cache. The depc decorator tells us that for any two annotations, we will cache a float
scorewhich is the finFindR similarity of those annotations. This score will never be recomputed unless the parent annotations change or the cache is deleted.
- The input of the ID call is two lists of annotations, the query annots and the database annots. The database annots are the "match-against set", the current catalog of annots (and the attached names) against which results are computed. In practice the query list is often a single annotation, and some of our plugins require that to be the case.
- The ID method returns similarity scores, not distances. In other words a higher score must represent a stronger match. Ideally these scores range from 0 to 1. Since many algorithms return distances in an embedding space, we transform these distances using various methods that are up to your discretion. For finFindR:
As is the case across ML, providing explainability to the end-user greatly increases the value of an ID algorithm. We do this by, for a query-database annot pair, rendering both annotations side-by-side and if possible illustrating the features that influenced the match score. For methods where these determining features might be obscure or hard to illustrate, simply showing the annots side-by-side is helpful to the users.
Looking again to finFindR, here are the illustration methods that once defined, integrate an algorithm's illustrations with the rest of WBIA. WBIA will save these illustrations to disk and serve them to the users as necessary, and also determines how many illustrations to render for a given match (since the daid_list might be many thousands of images, we only need to illustrate e.g. the top 16 match results).
- The FinfindrRequest class extends the VsOneSimilarityRequest base class defined in the wbia database module dtool.
render_single_resultmakes the illustration using the finFindR-specific method
FinfindrPassport. These passport photos highlight the finFindR feature on a cropped annotation image.
render_single_resultsimply places the query and database annots' passport photos next to each other.
The WBIA package itself must be made aware of your ID plugin in a few places. Modifying these files in a way consistent with other plugins like finFindR will integrate your package with WBIA. Within the
web/apis_engine.py:review_graph_match_htmllists the available ID plugins
viz/viz_matches.py:get_query_annot_pair_infolists the ID algorithms with custom visualizations
control/IBEISControl.pydefines the ibeis/WBIA controller referred to as
ibs; the top of this file has logic which loads plugins depending on flags at startup.