Smart Data Audit
Description
The Smart Data Audit API can be used to get detailed information about the data returned as part of the data inquiry response for a submission. The audit data can be retrieved for a data package or at a more granular level, i.e., for a single data point. The audit includes references from the different data sources for the data point in question.
Prerequisites
- The file is submitted for processing via the Universal Submit API.
- Once the transaction reaches the
Review_requiredstatus, the data is inquired for that transaction via the Data Inquiry API. - To retrieve the audit, a Data Package ID, or a Data Element ID / Data Element Key for the data element, is required.
- The Data Element ID (Data Point ID) is defined as part of the data package shared with you.
- The Data Element Key can be obtained from the Data Inquiry response.
Authentication
Step 1: Obtain a Bearer Token
The Smart Data Audit API uses the Insurance Intelligence Authenticate API to obtain a bearer token using your client_id, client_secret, and API key. The bearer token obtained from the Authentication API can be used to authenticate requests to the Smart Data Audit API.
Your Bold Penguin project team will provide you with a unique client_id and client_secret for each environment as well as an API key and Data Package ID (dp_id).
Step 2: Access the Smart Data Audit API
Data Package Audit
Get the audit for all the data points in a data package.
Endpoint
Staging
POST https://plb.di-beta.boldpenguin.com/universal-audit/v2/datapackage
Production
POST https://plb.ii.boldpenguin.com/universal-audit/v2/datapackage
Pass the following variables as part of the request
| Request Parameters | Comments |
|---|---|
| Authentication Token | The token obtained via the Authentication API |
| API Key | (Provided by Bold Penguin) |
| data_package_id | (Provided by Bold Penguin) |
| doc_reference_id | The tx_id returned as part of the Universal Submit response |
| tenant_id | (Provided by Bold Penguin) |
The response from the audit will contain the audit_data list, which has the audit details for every data point in the
data package. The following are the audit attributes for every data point.
| Response Parameters | Comments |
|---|---|
| data_element_id | Unique ID for the data element |
| data_element_name | The business name for the data element |
| destination_path | JSON path of the element in the data inquiry response |
| value_type | The type of the value (for example, STRING) |
| final_score | Final triangulated confidence score (BP score) |
| final_value | Final triangulated value |
| data_sources.source | The source name |
| data_sources.source_id | Identifier of the source configuration for the data point |
| data_sources.value | Value retrieved from the source |
| data_sources.value_type | Value type retrieved from the source |
| data_sources.trust_score | Trust score for the source |
| data_sources.confidence_score | Extraction confidence score |
When the data is extracted from multiple documents of the same classified document type, the dependency node further
refers to every sub document type.
| Response Parameters | Comments |
|---|---|
| data_sources.dependency.source | Source name. Examples: ACORD_DATA, APP_FORMS_DATA, SMART_FORMS_DATA, POLICY_FORMS_DATA, SOV_DATA, PDF_SOV_DATA, LOSS_RUN_DATA, DNB, CORTERA |
| data_sources.dependency.source_id | Identifier of the source configuration for the data point |
| data_sources.dependency.sub_doc_type | Sub document type. Examples: ACORD 125, ACORD 130, EML, MKTP |
| data_sources.dependency.trust_score | Trust score for the source |
| data_sources.dependency.value | Value retrieved from the source |
| data_sources.dependency.value_type | Value type retrieved from the source |
| data_sources.dependency.confidence_score | Extraction confidence score |
| data_sources.dependency.meta_data.key_lookup | The key to uniquely identify an item in the value, depending on the type of value: string: value list: items in the list object: keys in the object list of objects: keys in the objects inside an array list of lists: keys in the object inside an array |
| data_sources.dependency.meta_data.classified_file_id | Reference to the classified document from which the data was extracted |
| data_sources.dependency.meta_data.file_id | Reference to the file from which the data was extracted |
| data_sources.dependency.meta_data.file_name | Reference to the name of the file from which the data was extracted |
| data_sources.dependency.meta_data.page_num | Reference to the page number from which the data was extracted |
| data_sources.dependency.meta_data.coordinates | Reference to the coordinates from which the data was extracted (x1, y1 and x2, y2) |
Sample Smart Data Audit API Request
Request audit data using the Data Package ID
POST /universal-audit/v2/datapackage HTTP/1.1
Host: plb.di-beta.boldpenguin.com
x-api-key: xxxxxxxxxxx
Content-Type: application/json
Authorization: Bearer xxxxxxxxxx
Content-Length: 139
{
"doc_reference_id": "236fb681-c533-4b6c-ba65-05fafbbebcdb",
"data_package_id": "bookroll-us-c0001",
"tenant_id": "kowalski"
}
Sample Smart Data Audit Response
The response below is truncated to show the company_name and website data points.
{
"doc_reference_id": "236fb681-c533-4b6c-ba65-05fafbbebcdb",
"tenant_id": "kowalski",
"data":
{
"data_package_id": "bookroll-us-c0001",
"audit_data":
[
[
{
"data_element_id": "S-C000003",
"data_element_name": "Company Name",
"destination_path": "facts.company_name",
"value_type": "STRING",
"final_value": "Tesco Express",
"final_score": "93.27",
"data_sources":
[
{
"source": "SMD_SEED_DATA",
"source_id": "smd_company_name",
"trust_score": "7",
"value_type": "STRING",
"value": "Tesco Express"
},
{
"source": "CLEARBIT",
"source_id": "clearbit_company_name",
"trust_score": "6",
"value_type": "STRING",
"value": "Tesco PLC"
},
{
"source": "DNB",
"source_id": "dnb_company_name",
"trust_score": "10",
"value_type": "STRING",
"value": "TESCO STORES LIMITED"
},
{
"source": "BP_JACKIQ",
"source_id": "bp_jackiq_company_name",
"trust_score": "5",
"value_type": "STRING",
"value": "Tesco Express"
},
{
"source": "EML",
"source_id": "depn_sfd_company_name",
"trust_score": "7",
"value_type": "STRING",
"value": "TESCO",
"confidence_score": "96.50",
"dependency":
[
{
"source": "SMART_FORMS_DATA",
"source_id": "sfd_company_name_llm",
"sub_doc_type": "EML(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "TESCO",
"confidence_score": 96.5,
"meta_data":
[
{
"key_lookup": "TESCO",
"classified_file_id": "7a26a2a0-2bed-4747-8ad1-2b1127e13dc2",
"file_id": "63b26594-a176-4083-80e0-62c1e2141bbe",
"page_num": "1",
"file_name": "EML + MKPT + SOV.eml",
"coordinates":
{
"x1": 0.2396078431372549,
"y1": 0.03727272727272727,
"x2": 0.28823529411764703,
"y2": 0.04818181818181818
}
}
]
},
{
"source": "SMART_FORMS_DATA",
"source_id": "sfd_company_name_llm",
"sub_doc_type": "EML(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "TESCO",
"confidence_score": 96.5,
"meta_data":
[
{
"key_lookup": "TESCO",
"classified_file_id": "7a26a2a0-2bed-4747-8ad1-2b1127e13dc2",
"file_id": "63b26594-a176-4083-80e0-62c1e2141bbe",
"page_num": "1",
"file_name": "EML + MKPT + SOV.eml",
"coordinates":
{
"x1": 0.2396078431372549,
"y1": 0.03727272727272727,
"x2": 0.28823529411764703,
"y2": 0.04818181818181818
}
}
]
}
]
},
{
"source": "MKTP",
"source_id": "depn_mkpt_company_name",
"trust_score": "6",
"value_type": "STRING",
"value": "TESCO PLC",
"confidence_score": "96.00",
"dependency":
[
{
"source": "MKPT_DATA",
"source_id": "mkpt_company_name_llm",
"sub_doc_type": "MKTP(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "TESCO PLC",
"confidence_score": 96.0,
"meta_data":
[
{
"key_lookup": "TESCO PLC",
"classified_file_id": "a07ccecb-14fe-436e-b5d9-f02f1093b41f",
"file_id": "752fcb90-50e0-48da-9af4-37ef4e52dd0e",
"page_num": "1",
"file_name": "Tesco.pdf",
"coordinates":
{
"x1": 0.14862745098039215,
"y1": 0.8296969696969697,
"x2": 0.37333333333333335,
"y2": 0.8578787878787879
}
}
]
}
]
}
]
},
{
"data_element_id": "S-C001044",
"data_element_name": "Broker Email",
"destination_path": "options.broker_email",
"value_type": "STRING",
"final_value": "laura.razo@geninsbrokers.com",
"final_score": "79.11",
"data_sources":
[
{
"source": "EML",
"source_id": "depn_sfd_broker_email",
"trust_score": "9",
"value_type": "STRING",
"value": "laura.razo@geninsbrokers.com",
"confidence_score": "96.50",
"dependency":
[
{
"source": "SMART_FORMS_DATA",
"source_id": "eml_broker_email_llm",
"sub_doc_type": "EML(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "laura.razo@geninsbrokers.com",
"confidence_score": 96.5,
"meta_data":
[
{
"key_lookup": "laura.razo@geninsbrokers.com",
"classified_file_id": "7a26a2a0-2bed-4747-8ad1-2b1127e13dc2",
"file_id": "63b26594-a176-4083-80e0-62c1e2141bbe",
"page_num": "1",
"file_name": "EML + MKPT + SOV.eml",
"coordinates":
{
"x1": 0.16549019607843138,
"y1": 0.22272727272727272,
"x2": 0.41411764705882353,
"y2": 0.23363636363636364
}
}
]
}
]
}
]
},
{
"data_element_id": "S-C001043",
"data_element_name": "Broker State",
"destination_path": "options.broker_state",
"value_type": "STRING",
"final_value": "",
"final_score": "",
"data_sources":
[]
},
{
"data_element_id": "S-C000011",
"data_element_name": "State",
"destination_path": "facts.state",
"value_type": "STRING",
"final_value": "ENG",
"final_score": "93.61",
"data_sources":
[
{
"source": "DNB",
"source_id": "dnb_address",
"trust_score": "8",
"value_type": "STRING",
"value": "ENG"
},
{
"source": "SMD_SEED_DATA",
"source_id": "sd_address",
"trust_score": "7",
"value_type": "STRING",
"value": "Greater Manchester"
}
]
},
{
"data_element_id": "S-C000010",
"data_element_name": "City",
"destination_path": "facts.city",
"value_type": "STRING",
"final_value": "MANCHESTER",
"final_score": "93.61",
"data_sources":
[
{
"source": "DNB",
"source_id": "dnb_address",
"trust_score": "8",
"value_type": "STRING",
"value": "MANCHESTER"
},
{
"source": "SMD_SEED_DATA",
"source_id": "sd_address",
"trust_score": "7",
"value_type": "STRING",
"value": "Manchester"
},
{
"source": "CLEARBIT",
"source_id": "clearbit_address",
"trust_score": "6",
"value_type": "STRING",
"value": "Welwyn Garden City"
},
{
"source": "BP_JACKIQ",
"source_id": "bp_jackiq_address",
"trust_score": "7",
"value_type": "STRING",
"value": "Manchester"
},
{
"source": "EML",
"source_id": "depn_sfd_address",
"trust_score": "7",
"value_type": "STRING",
"value": "Manchester",
"confidence_score": "93.70",
"dependency":
[
{
"source": "SMART_FORMS_DATA",
"source_id": "sfd_company_address_llm",
"sub_doc_type": "EML(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "Manchester",
"confidence_score": 97.2,
"meta_data":
[
{
"key_lookup": "Manchester",
"classified_file_id": "7a26a2a0-2bed-4747-8ad1-2b1127e13dc2",
"file_id": "63b26594-a176-4083-80e0-62c1e2141bbe",
"page_num": "1",
"file_name": "EML + MKPT + SOV.eml",
"coordinates":
{
"x1": 0.08666666666666667,
"y1": 0.37393939393939396,
"x2": 0.15450980392156863,
"y2": 0.38212121212121214
}
}
]
}
]
},
{
"source": "MKTP",
"source_id": "depn_mkpt_address",
"trust_score": "6",
"value_type": "STRING",
"value": "Manchester",
"confidence_score": "90.90",
"dependency":
[
{
"source": "MKPT_DATA",
"source_id": "mkpt_company_address_llm",
"sub_doc_type": "MKTP(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "Manchester",
"confidence_score": 91.6,
"meta_data":
[
{
"key_lookup": "Manchester",
"classified_file_id": "a07ccecb-14fe-436e-b5d9-f02f1093b41f",
"file_id": "752fcb90-50e0-48da-9af4-37ef4e52dd0e",
"page_num": "5",
"file_name": "Tesco.pdf",
"coordinates":
{
"x1": 0.3819607843137255,
"y1": 0.2087878787878788,
"x2": 0.4807843137254902,
"y2": 0.22181818181818183
}
}
]
}
]
}
]
},
{
"data_element_id": "S-C000012",
"data_element_name": "Postal Code",
"destination_path": "facts.postal_code",
"value_type": "STRING",
"final_value": "M1 7ED",
"final_score": "93.61",
"data_sources":
[
{
"source": "DNB",
"source_id": "dnb_address",
"trust_score": "8",
"value_type": "STRING",
"value": "M1 7ED"
},
{
"source": "SMD_SEED_DATA",
"source_id": "sd_address",
"trust_score": "7",
"value_type": "STRING",
"value": "M1 7ED"
},
{
"source": "CLEARBIT",
"source_id": "clearbit_address",
"trust_score": "6",
"value_type": "STRING",
"value": "AL7 1TW"
},
{
"source": "BP_JACKIQ",
"source_id": "bp_jackiq_address",
"trust_score": "7",
"value_type": "STRING",
"value": "M1 7ED"
},
{
"source": "EML",
"source_id": "depn_sfd_address",
"trust_score": "7",
"value_type": "STRING",
"value": "M1 7ED",
"confidence_score": "93.70",
"dependency":
[
{
"source": "SMART_FORMS_DATA",
"source_id": "sfd_company_address_llm",
"sub_doc_type": "EML(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "M1 7ED",
"confidence_score": 95.1,
"meta_data":
[
{
"key_lookup": "M1 7ED",
"classified_file_id": "7a26a2a0-2bed-4747-8ad1-2b1127e13dc2",
"file_id": "63b26594-a176-4083-80e0-62c1e2141bbe",
"page_num": "1",
"file_name": "EML + MKPT + SOV.eml",
"coordinates":
{
"x1": 0.15764705882352942,
"y1": 0.37393939393939396,
"x2": 0.20313725490196077,
"y2": 0.38212121212121214
}
}
]
}
]
},
{
"source": "MKTP",
"source_id": "depn_mkpt_address",
"trust_score": "6",
"value_type": "STRING",
"value": "M1 7ED",
"confidence_score": "90.90",
"dependency":
[
{
"source": "MKPT_DATA",
"source_id": "mkpt_company_address_llm",
"sub_doc_type": "MKTP(AI)",
"trust_score": "10",
"value_type": "STRING",
"value": "M1 7ED",
"confidence_score": 93.9,
"meta_data":
[
{
"key_lookup": "M1 7ED",
"classified_file_id": "a07ccecb-14fe-436e-b5d9-f02f1093b41f",
"file_id": "752fcb90-50e0-48da-9af4-37ef4e52dd0e",
"page_num": "5",
"file_name": "Tesco.pdf",
"coordinates":
{
"x1": 0.48705882352941177,
"y1": 0.20909090909090908,
"x2": 0.548235294117647,
"y2": 0.2196969696969697
}
}
]
}
]
}
]
}
]
]
}
}