Creating a new validation for a data model
Permissions and endpoints
By default users with the Data Steward or Administrator role have the permission to create and manage data models and their associated validations via the FAIR API.
The full list of data model endpoints is here, the following are required for the process detailed in this article for creating a new dataset validation:
| Endpoint | Description |
| GET /datamodels/ | Returns a list of all data models available within a FAIR instance |
| GET /datamodels/{code}/validations | Returns a list of validations associated with a dataset |
| GET /datamodels/{code}/validations/{data_model_validation_code} | Returns the detail of a particular validation |
| POST /datamodels/{code}/validations | Adds a new validation to a data model |
| PATCH /datamodels/{code}/validations/{data_model_validation_code} | Used to enable or disable a validation |
Creating a new validation
We recommend using existing validations as a starting point for any new data validations. This section details the steps for basing a new validation on an existing validation for the curated OMOP data model.
- Viewing existing data models
GET /datamodels/
This will retrun a list of all available datamodels e.g:
{
"code": "omop",
"name": "OMOP",
"description": "Observational Medical Outcomes Partnership Common Data Model",
"system": true,
"enabled": true
},This provides a high level description of each available data model:
- code - data model code
- name - data model name
- description - description of the data model
- system - is the data model a curated system model or user created
- enabled - is the data model enabled for use by FAIR data owners
2. Viewing validations associated with a data model GET /datamodels/{code}/validations using the code of the relevant data model, in this case omop
This will return a list of all the validations associated with the OMOP data model:
{
"items": [
{
"code": "gold",
"name": "Gold",
"description": "This schema defines the structure of the OMOP v5.4 Common Data Model for the gold validation. All dictionaries exist with no additional dictionaries. All fields in each dictionary exists with no additional fields. Each field is of the correct type.",
"system": true,
"priority": 1,
"enabled": true
},
{
"code": "silver",
"name": "Silver",
"description": "This schema defines the structure of the OMOP v5.4 Common Data Model for the silver validation. All dictionaries exist with no additional dictionaries. All fields in each dictionary exists with no additional fields. Field types not checked.",
"system": true,
"priority": 2,
"enabled": true
},
{
"code": "bronze",
"name": "Bronze",
"description": "This schema defines the structure of the OMOP v5.4 Common Data Model for the bronze validation. All dictionaries exist, but additional dictionaries may be present. All fields in each dictionary exists. Additional fields may also be present. Field types not checked.",
"system": true,
"priority": 3,
"enabled": true
},
{
"code": "basic",
"name": "Basic",
"description": "This schema defines the structure of the OMOP v5.4 Common Data Model for the basic validation. All dictionaries exist, but the content of those dictionaries is flexible. Additional dictionaries may also be present.",
"system": true,
"priority": 4,
"enabled": true
}
],
"paging": {
"total": 4,
"page": 1,
"pages": 1
},
"additional": {}This provides a high level descripition for each available validation:
- code - validation code
- name - validaton name
- description - a description of the validation, as with the OMOP example above it is good practice to describe what is being validated in this field
- system - is this a curated system data validation or user created
- priority - this determines the order the validations will be shown to the data owner when they run a validation report in the FAIR UI. As you can see from the OMOP example above priorty is a descending scale, the highest priorty gold report is 1, the lowest priority basic report is 4.
- enabled - is the validation enabled
3. Viewing the details of a specific validation GET /datamodels/{code}/validations/{data_model_validation_code} using the code of the relevant data model and validation, in this case omop and bronze
This will return the full bronze validation schema. The Bronze schema checks that all OMOP CDM dictionaries are present and contain the expected columns, but non-OMOP dictionaries and columns may exist. The excerpts below show how this is implemented in the schema:
Dictionary level validation lists the OMOP dictionaries:
"unevaluatedProperties": false,
"properties": {
"dictionaries": {
"type": "object",
"required": [
"person",
"observation_period",
"visit_occurrence",
"visit_detail",
"condition_occurrence",
"drug_exposure",
"procedure_occurrence",
"device_exposure",
"measurement",
"observation",
"death",
"note",
"note_nlp",
"specimen",
"fact_relationship",
"location",
"care_site",
"provider",
"payer_plan_period",
"cost",
"drug_era",
"dose_era",
"condition_era",
"episode",
"episode_event",
"metadata",
"cdm_source",
"concept",
"vocabulary",
"domain",
"concept_class",
"concept_relationship",
"relationship",
"concept_synonym",
"concept_ancestor",
"source_to_concept_map",
"drug_strength",
"cohort",
"cohort_definition"Column level validation for the 'person' dictionary lists the fields for the dictionary:
"properties": { "person": { "type": "object", "required": [], "properties": { "person": { "type": "object", "required": [ "person_id", "gender_concept_id", "year_of_birth", "month_of_birth", "day_of_birth", "birth_datetime", "race_concept_id", "ethnicity_concept_id", "location_id", "provider_id", "care_site_id", "person_source_value", "gender_source_value", "gender_source_concept_id", "race_source_value", "race_source_concept_id", "ethnicity_source_value", "ethnicity_source_concept_id" ],
Because the Bronze schema does not validate field type each column has the type string e.g.
],
"properties": {
"person_id": {
"type": "string"
},
"gender_concept_id": {
"type": "string"
},
"year_of_birth": {
"type": "string"The Silver validation schema is more stringent than Bronze. It checks all OMOP CDM dictionaries and columns are present, but no additional dictionaries or additional columns are allowed, column types are still not checked.
Comparing Silver to Bronze shows how these additional checks manifest in the validation JSON.
Silver enforces the number of dictionaries by including the minProperties and maxProperties fields:
"unevaluatedProperties": false,
"properties": {
"dictionaries": {
"type": "object",
"minProperties": 39,
"maxProperties": 39,Similarly it also enforces the number of fields per dictionary :
"properties": {
"person": {
"type": "object",
"minProperties": 18,
"maxProperties": 18,
"unevaluatedProperties": false,4. Add a new validation to an existing data model POST /datamodels/{code}/validations
In this scenario the user wants to create an intermediate validation between the exisiting Bronze and Silver OMOP validations that validates the number of dictionaries, but not the fields in each dictionary. To achieve this the user can take a copy of the existing Bronze schema and update it to incldue the minProperties and maxProperties fields at the dictionaries level:
{
"code": "bronze-plus",
"name": "Bronze Plus",
"description": "This schema defines the structure of the OMOP v5.4 Common Data Model for the bronze plus validation. All dictionaries exist with no additional dictionaries. All fields in each dictionary exist. Additional fields may also be present. Field types not checked."",
"enabled": false,
"priority": 3,
"validation": {
"type": "object",
"required": [
"dictionaries"
],
"unevaluatedProperties": false,
"properties": {
"dictionaries": {
"type": "object",
"minProperties": 39,
"maxProperties": 39, This new validation can be added to the OMOP data model by posting the JSON to the POST /datamodels/{code}/validations endpoint using the omopcode.
Its enabled status can then be managed via PATCH /datamodels/{code}/validations/{data_model_validation_code}.