Queue Schema

Every queue has an associated schema that specifies which fields will be extracted from documents as well as the structure of the data sent to connector and exported from the platform.

Rossum schema supports data fields with single values (datapoint), fields with multiple values (multivalue) or tuples of fields (tuple). At the topmost level, each schema consists of sections, which may either directly contain actual data fields (datapoints) or use nested multivalues and tuples as containers for single datapoints.

But while schema may theoretically consist of an arbitrary number of nested containers, the Rossum UI supports only certain particular combinations of datapoint types. The supported shapes are:

• simple: atomic datapoints of type number, string, date or enum

• list: simple datapoint within a multivalue

• tabular: simple datapoint within a "multivalue tuple" (a multivalue list containing a tuple for every row)

Schema content

Schema content consists of a list of section objects.

Common attributes

The following attributes are common for all schema objects:

Section

Example section object:

{
  "category": "section",
  "id": "amounts_section",
  "label": "Amounts",
  "children": [...],
  "icon": ""
}

Section represents a logical part of the document, such as amounts or vendor info. It is allowed only at the top level. Schema allows multiple sections, and there should be at least one section in the schema.

Datapoint

A datapoint represents a single value, typically a field of a document or some global document information. Fields common to all datapoint types:

rir_field_names attribute allows to specify source of initial value of the object. List items may be:

• one of extracted field types to use the AI engine prediction value
• upload:id to identify a value specified while uploading the document
• edit:id to identify a value specified in edit_pages endpoint
• email_header:<id> to use a value extracted from email headers. Supported email headers: from, to, reply-to, subject, message-id, date.
• email_body:<id> to select email body. Supported values are text_html for HTML body or text_plain for plain text body.
• email:<id> to identify a value specified in email.received hook response
• emails_import:<id> to identify a value specified in the values parameter when importing an email.

If more list items in rir_field_names are specified, the first available value will be used.

String type

Example string type datapoint with constraints:

{
  "category": "datapoint",
  "id": "document_id",
  "label": "Invoice ID",
  "type": "string",
  "default_value": null,
  "rir_field_names": ["document_id"],
  "constraints": {
    "length": {
      "exact": null,
      "max": 16,
      "min": null
    },
    "regexp": {
      "pattern": "^INV[0-9]+$"
    },
    "required": false
  }
}

String datapoint does not have any special attribute.

Date type

Example date type datapoint:

{
  "id": "item_delivered",
  "type": "date",
  "label": "Item Delivered",
  "format": "MM/DD/YYYY",
  "category": "datapoint"
}

Attributes specific to Date datapoint:

Date format supported: available tokens

Example date formats:

• D/M/YYYY: e.g. 23/1/2019
• MM/DD/YYYY: e.g. 01/23/2019
• YYYY-MM-DD: e.g. 2019-01-23 (ISO date format)

Number type

Example number type datapoint:

{
  "id": "item_quantity",
  "type": "number",
  "label": "Quantity",
  "format": "#,##0.#",
  "category": "datapoint"
}

Attributes specific to Number datapoint:

The following table shows numeric formats with their examples.

Aggregations

Example number type datapoint with sum aggregation:

{
  "id": "quantity",
  "type": "number",
  "label": "Quantity",
  "category": "datapoint",
  "aggregations": {
    "sum": {
      "label": "Total"
    }
  },
  "default_value": null,
  "rir_field_names": []
}

Aggregations allow computation of some informative values, e.g. a sum of a table column with numeric values. These are returned among messages when the validate endpoint is called. Aggregations can be computed only for tables (multivalues of tuples).

All aggregation objects can have an attribute label that will be shown in the UI.

Enum type

Example enum type datapoint with options:

{
  "id": "document_type",
  "type": "enum",
  "label": "Document type",
  "hidden": false,
  "category": "datapoint",
  "options": [
    {
      "label": "Invoice Received",
      "value": "21"
    },
    {
      "label": "Invoice Sent",
      "value": "22"
    },
    {
      "label": "Receipt",
      "value": "23"
    }
  ],
  "default_value": "21",
  "rir_field_names": [],
  "enum_value_type": "number"
}

Attributes specific to Enum datapoint:

Every option consists of an object with keys:

Enum datapoint value is matched in a case insensitive mode, e.g. EUR currency value returned by the AI Core Engine is matched successfully against {"value": "eur", "label": "Euro"} option.

Button type

Specifies a button shown in Rossum UI. For more details please refer to custom UI extension.

Example button type datapoint:

{
  "id": "show_email",
  "type": "button",
  "category": "datapoint",
  "popup_url": "http://example.com/show_customer_data",
  "can_obtain_token": true
}

Buttons cannot be direct children of multivalues (simple multivalues with buttons are not allowed. In tables, buttons are children of tuples). Despite being a datapoint object, button currently cannot hold any value. Therefore, the set of available Button datapoint attributes is limited to:

Value constraints

Example datapoint with value constraints:

{
  "id": "document_id",
  "type": "string",
  "label": "Invoice ID",
  "category": "datapoint",
  "constraints": {
    "length": {
      "max": 32,
      "min": 5
    },
    "required": false
  },
  "default_value": null,
  "rir_field_names": [
    "document_id"
  ]
}

Constraints limit allowed values. When constraints is not satisfied, annotation is considered invalid and cannot be exported.

UI configuration

Example datapoint with UI configuration:

{
  "id": "document_id",
  "type": "string",
  "label": "Invoice ID",
  "category": "datapoint",
  "ui_configuration": {
    "type":  "captured",
    "edit": "disabled"
  },
  "default_value": null,
  "rir_field_names": [
    "document_id"
  ]
}

UI configuration provides a group of settings, which alter behaviour of the field in the application. This does not affect behaviour of the field via the API. For example, disabling edit prohibits changing a value of the datapoint in the application, but the value can still be modified through API.

Logical types

• Captured field represents information retrieved by the OCR model. If combined with edit option disabled, user can't overwrite field's value, but is able to redraw field's bounding box and select another value from the document by such an action.
• Data field represents information filled by extensions. This field is not mapped to the AI model, so it does not have a bounding box, neither it's possible to create one. If combined with edit option disabled the field can't be modified from the UI.
• Manual field behaves exactly like Data field, without the mapping to extensions. This field should be used for sharing information between users or to transfer information to downstream systems.
• Formula field This field will be updated according to its formula definition, see Formula Fields. If the edit option is not disabled the field value can be overridden from the UI (see no_recalculation).
• Reasoning fields This field will be updated according to its prompt and context. context supports adding related schema fields in a format of TxScript strings (e.g. field.invoice_id, also self.attr.label and self.attr.description are supported). If the edit option is not disabled the field value can be overridden from the UI (see no_recalculation).
• null value is displayed in UI as Unset and behaves similar to the Captured field.

Multivalue

Example simple multivalue:

{
  "category": "multivalue",
  "id": "po_numbers",
  "label": "PO numbers",
  "children": {
    ...
  },
  "show_grid_by_default": false,
  "min_occurrences": null,
  "max_occurrences": null,
  "rir_field_names": null
}

Example multivalue with grid configuration:

{
  "category": "multivalue",
  "id": "line_item",
  "label": "Line Item",
  "children": {
    ...
  },
  "grid": {
    "row_types": [
      "header", "data", "footer"
    ],
    "default_row_type": "data",
    "row_types_to_extract": [
      "data"
    ]
  },
  "min_occurrences": null,
  "max_occurrences": null,
  "rir_field_names": ["line_items"]
}

Multivalue is list of datapoints or tuples of the same type. It represents a container for data with multiple occurrences (such as line items) and can contain only objects with the same id.

Multivalue grid object

Multivalue grid object allows to specify a row type for each row of the grid. For data representation of actual grid data rows see Grid object description.

For example to distinguish two header types and a footer in the validation interface, following row types may be used: header, subsection_header, data and footer.

Currently, data extraction classifies every row as either data or header (additional row types may be introduced in the future). We remove rows returned by data extraction that are not in row_types list (e.g. header by default) and are on the top/bottom of the table. When they are in the middle of the table, we mark them as skipped (null).

There are three visual modes, based on row_types quantity:

• More than two row types defined: User selects row types freely to any non-default row type. Clearing row type resets to a default row type. We support up to 6 colors to easily distinguish row types visually.
• Two row types defined (header and default): User only marks header and skipped rows. Clearing row type resets to a default row type.
• One row type defined: User is only able to mark row as skipped (null value in data). This is also a default behavior when no grid row types configuration is specified in the schema.

Tuple

Example tuple object:

{
  "category": "tuple",
  "id": "tax_details",
  "label": "Tax Details",
  "children": [
    ...
  ],
  "rir_field_names": [
    "tax_details"
  ]
}

Container representing tabular data with related values, such as tax details. A tuple must be nested within a multivalue object, but unlike multivalue, it may consist of objects with different ids.

Updating Schema

When project evolves, it is a common practice to enhance or change the extracted field set. This is done by updating the schema object.

By design, Rossum supports multiple schema versions at the same time. However, each document annotation is related to only one of those schemas. If the schema is updated, all related document annotations are updated accordingly. See preserving data on schema change below for limitations of schema updates.

In addition, every queue is linked to a schema, which is used for all newly imported documents.

When updating a schema, there are two possible approaches:

• Update the schema object (PUT/PATCH). All related annotations will be updated to match current schema shape (even exported and deleted documents).
• Create a new schema object (POST) and link it to the queue. In such case, only newly created objects will use the current schema. All previously created objects will remain in the shape of their linked schema.

Use case 1 - Initial setting of a schema

• Situation: User is initially setting up the schema. This might be an iterative process.
• Recommendation: Edit the existing schema by updating schema (PUT) or updating part of a schema (PATCH).

Use case 2 - Updating attributes of a field (label, constraints, options, etc.)

• Situation: User is updating field, e.g. changing label, number format, constraints, enum options, hidden flag, etc.
• Recommendation: Edit existing schema (see Use case 1).

Use case 3 - Adding new field to a schema, even for already imported documents.

• Situation: User is extending a production schema by adding a new field. Moreover, user would like to see all annotations (to_review, postponed, exported, deleted, etc. states) in the look of the newly extended schema.
• Recommendation: Edit existing schema (see Use case 1). Data of already created annotations will be transformed to the shape of the updated schema. New fields of annotations in to_review and postponed state that are linked to extracted fields types will be filled by AI Engine, if available. New fields for already exported or deleted annotations (also purged, exporting and failed_export) will be filled with empty or default values.

Use case 4 - Adding new field to schema, only for newly imported documents

• Situation: User is extending a production schema by adding a new field. However, with the intention that the user does not want to see the newly added field on previously created annotations.
• Recommendation: Create a new schema object (POST) and link it to the queue. Annotation data of previously created annotations will be preserved according to the original schema. This approach is recommended if there is an organizational need to keep different field sets before and after the schema update.

Use case 5 - Deleting schema field, even for already imported documents.

• Situation: User is changing a production schema by removing a field that was used previously. However, user would like to see all annotations (to_review, postponed, exported, deleted, etc. states) in the look of the newly extended schema. There is no need to see the original fields in already exported annotations.
• Recommendation: Edit existing schema (see Use case 1).

Use case 6 - Deleting schema field, only for newly imported documents

• Situation: User is changing a production schema by removing a field that was used previously. However, with the intention that the user will still be able to see the removed fields on previously created annotations.
• Recommendation: Create a new schema object (see Use case 4). Annotation data of previously created annotations will be preserved according to the original schema. This approach is recommended if there is an organizational need to retrieve the data in the original state.

Preserving data on schema change

In order to transfer annotation field values properly during the schema update, a datapoint's category and schema_id must be preserved.

Supported operations that preserve fields values are:

• adding a new datapoint (filled from AI Engine, if available)
• reordering datapoints on the same level
• moving datapoints from section to another section
• moving datapoints to and from a tuple
• reordering datapoints inside a tuple
• making datapoint a multivalue (original datapoint is the only value in a new multivalue container)
• making datapoint non-multivalue (only first datapoint value is preserved)

Extracted field types

AI engine currently automatically extracts the following fields at the all endpoint, subject to ongoing expansion.

Identifiers

Document attributes

Amounts

Typical relations (may depend on local laws):

amount_total = amount_total_base + amount_total_tax
amount_rounding = amount_total - round(amount_total)
amount_due = amount_total - amount_paid + amount_rounding

All amounts are in the main currency of the invoice (as identified in the currency response field). Amounts in other currencies are generally excluded.

Tables

At the moment, the AI engine automatically extracts 2 types of tables. In order to pick one of the possible choices, set rir_field_names attribute on multivalue.

Tax details

Tax details table and breakdown by tax rates.

Line items

AI engine currently automatically extracts line item table content and recognizes row and column types as detailed below. Invoice line items come in a wide variety of different shapes and forms. The current implementation can deal with (or learn) most layouts, with borders or not, different spacings, header rows, etc. We currently make two further assumptions:

• The table generally follows a grid structure - that is, columns and rows may be represented as rectangle spans. In practice, this means that we may currently cut off text that overlaps from one cell to the next column. We are also not optimizing for table rows that are wrapped to multiple physical lines.
• The table contains just a flat structure of line items, without subsection headers, nested tables, etc.

We plan to gradually remove both assumptions in the future.