This API proposal covers the API exposed to the users of the mechanical turk annotation system. To main goal of the Mech Turk stack is to provide high level interfaces for image annotation.

Review parts

ROS API

ROS API defines ROS messages for image annotations and 2 ways of annotating images: streaming and action.

ROS messages are defined in mech_turk_ros_msgs: mech_turk_ros_msgs/ImageReference, mech_turk_ros_msgs/BoundingBox, mech_turk_ros_msgs/ObjectAnnotation and mech_turk_ros_msgs/ExternalAnnotation. mech_turk_ros_msgs/ExternalAnnotation contains an image reference and a list of objects. Each object has a unique ID, name, bounding box and an optional outline.

Streaming annotation forwards all images on a topic to the annotation server. This mode is implemented in the snapper node. The annotations are published on a topic by the annotation server. To cross the firewall boundary, link_topic node polls the annotation server and registers the remote publisher in the local core.

Action-based annotation is implemented in AnnotateImageAction. The action takes an image and produces mech_turk_ros_msgs/ExternalAnnotation. The action server allows multiple concurrent goals to be active. The goals succeed when the annotation is received from the server. The action submits the images directly to the server, but relies on link_topic and annotation streaming to receive the annotations from the server.

Command line tools

Command line tools allow to send images and raw annotation tasks to the server, retrieve results and convert outlines to segmentation masks.

To send images to the server, we provide cv_mech_turk2/submit_image.py. It sends images to the annotation session on the server.

To receive annotated images from the server we provide cv_mech_turk2.session_results.py. It downloads the annotations and the images from the server. For convenience, we provide conversion tool cv_mech_turk2.convert_outlines_to_masks to get object masks from the outlines.

To send raw task data to the server (e.g. for attributes, grouping, etc) we provide low-level mech_turk/submit_work_unit.py.

To download the raw submission data from the server, we provide mech_turk.get_raw_session_results.py. The downloaded data contains the work units and the submissions.

Server access API

The server presents an XML-RPC and JSON-RPC interface for direct operations on the task and session data (Live auto-generated docs, doxygen).

All data structures are communicated as JSON objects. The server defines the following data structures:

Session representation:
   id               (string)
   task             (name)
   hit_limit        (number)
   owner            (username)
   funding          (funding account name)
   qualifications   (list of qualification names)
   work_engine      (processing engine name)


Work Unit data structure:
   id           - Unique ID
   session      - The session ID
   state        - State as defined above
   assignments  - reserved
   submissions  - submission IDs
   data         - raw work unit parameters


Submission data structure:
   id           - Unique submission ID
   session      - Session ID 
   work_unit    - Work unit ID
   worker       - Worker ID
   data         - Raw submission data
   start_time   - When the work started
   end_time     - When the work ended
   grades       - List of grades (inline structures,  not IDs)

Grade data structure:
   id           - Unique grade ID
   quality      - Quality number (10 - good, 7 - approve/redo, 3 -reject)
   feedback     - (ptional) feedback
   worker       - The grader
   reference    - Grading reference (e.g. source algorithm or system tags)

To establish an RPC connection to the server, the client uses mech_turk.rpc_communication.connect function. The connect function is in charge of resolving the server name to the user and password. It uses auth.txt file to resolve the server name.

The server allows to submit images for annotation via POST request. The HTTP communication object is obtained with mech_turk.http_communication.connect. It resolves the server name, but the authentication is optional.

Web-based UI

To setup the annotation task on the server, we use web-based interface: django admin interface(e.g. create tasks and sessions); session dashboard (e.g. monitoring the process, approving/rejecting work); grading interface (reviewing the work).

Django admin interface allows to create users, create annotation tasks, create new annotation sessions, set limits on work units per session, create required qualifications.

Session dashboard shows the status of tasks within the session, allows to expire/resubmit them, perform grading, aprove/reject work, block workers.

Basic annotation tutorial shows how to create tasks, sessions and submit images through the web interface.