Skip to main content

Universal Submit

Description

Universal Submit is part of the Universal API package. This API allows you to submit prospect data for further processing. Data can be in eml, doc, pdf, and excel formats.

Authentication and Credentials

Step 1: Obtain a Bearer Token

The Universal Submit API uses the Authenticate API to obtain a bearer token using your client_id and client_secret. This bearer token can be used to authenticate requests using the Universal Submit API.

Accessing the Universal Submit API

Direct Submission for File Sizes Less Than 10MB

  • The Universal Submit API requires the file (eml/doc/pdf/xls etc.) to be passed along with the request for processing.
  • The below endpoint will work for file sizes less than 10MB.
  • To accommodate handling of files with sizes 10 MB or more the process of triggering the Universal Submit API is broken down into multiple steps as below.

Step 2: Submit the File for Processing

  • The call to the below endpoint will return a URL where the file can be uploaded.

Universal Submit URL

Staging

POST https://api-smartdata.di-beta.boldpenguin.com/universal/v3/universal-submit/file

Production

POST https://api.ii.boldpenguin.com/universal/v3/universal-submit/file

Pass the following variables as part of the request

Request ParametersComments
Authentication TokenThe token obtained in STEP 1
API Key(Shared with the Customer)
Content-Typemultipart/form-data
Content-Dispositionform-data; name="file"; filename="file"

Sample Universal Submit Request

POST /universal/v3/universal-submit/file HTTP/1.1
Host: api-smartdata.di-beta.boldpenguin.com
x-api-key: M4jHKvQ6dD4Y1sDiv7gEe6LIuqArhDin8XldfxoK
x-tenantId: kowalski
Authorization: Bearer NMav128jTZbbEK48o64UM94K
Content-Length: 258
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="Sample_File.xlsx"
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

(data)
----WebKitFormBoundary7MA4YWxkTrZu0gW

Sample Response

{
"access_token": "5ZhAzSrQCuyYct367GqfSnwE",
"token_type": "bearer",
"expires_in": 43200,
"refresh_token": "7EWVgH9bVdwBQTmeXH1HXJmV",
"scope": "",
"uid": "ee9e3e03-7261-4ec7-8988-1f88ba1ec2df",
"info": {
"name": "SmartData Integrations Service Client",
"email": null,
"first_name": "SmartData Integrations",
"last_name": "Service Client"
},
"extra": {
"raw_info": {
"user_id": "0715308c-db1c-4e63-b854-779bc93bc77c",
"tenant_id": "96a5f17b-ebc8-4459-aca7-b49ba6fbc752",
"tenant_name": "Kowalski",
"tenant_subdomain": "kowalski",
"tenant_theme": "default",
"tenant_type": null,
"primary_user_group_id": "0d5aef45-9b88-436c-a0ef-344b674bb6d3",
"user_group_ids": [
"0d5aef45-9b88-436c-a0ef-344b674bb6d3"
],
"name": "SmartData Integrations Service Client",
"email": null,
"first_name": "SmartData Integrations",
"last_name": "Service Client",
"phone": null,
"permissions": [
"::data/v2:get",
"::universal/v3/universal-submit:get",
"::universal/v3/universal-submit:post",
"::universal/v2/universal-submit:get",
"::universal/v2/universal-submit:post",
"::universal/v1/universal-submit:post",
"::universal/v1/universal-submit:get",
"::audit/v1/report:get",
"::location/v1/location_submit:post",
"::data/v1:get",
"::company/v1/quick_submit:post",
"::company/v1/company_submit:post"
],
"auth_uid": null,
"created_at": "2021-05-06T15:24:53.870Z",
"tenant_created_at": "2021-05-06T15:24:18.941Z",
"completed_steps": [],
"accepted_agreements": [],
"plan_sku": null,
"sub_status": null
}
}
}

Once the submission is completed, extracting the data from the document is a long running process. The different stages of the flow will transition the status of the transaction as below Submission_in_progress -> Scheduled -> Started -> Completed or Failed

Multi-step Processing for File Sizes > 10MB

The Universal Submit API requires the file (eml/doc/pdf/excel etc) to be passed along with the request for processing. To accommodate handling of files with sizes 10 MB or more the process of triggering the Universal Submit API is broken down into multiple steps as below

Step 2: Get the Fileupload URL

  • The call to the below endpoint will return a URL where the file can be uploaded.

  • The URL is valid only for 5 minutes.

  • The tx_id returned as part of the response is required to communicate with submit, status check and inquiry api’s in the context of this submission.

Universal Submit File Upload URL

Staging

POST https://api-smartdata.di-beta.boldpenguin.com/universal/v3/universal-submit/file-upload-url

Production

POST https://api.ii.boldpenguin.com/universal/v3/universal-submit/file-upload-url

Pass the below variables as part of the request | Request Parameters | Comments | | -------------------- | -------------------------------------------- | | Authentication Token | The token obtained in STEP 1 | | API Key | (Shared with the Customer) | | Filename | The full name of the file/eml that is being uploaded |

Sample File Upload URL Request

POST /universal/v3/universal-submit/file-upload-url HTTP/1.1
Host: api-smartdata.di-beta.boldpenguin.com
x-api-key: M4jHKvQ6dD4Y1sDiv7gEe6LIuqArhDin8XldfxoK
Authorization: Bearer NMav128jTZbbEK48o64UM94K
Content-Type: application/json
Content-Length: 70
{
"filename":"Sample File Name.eml"
}

Sample File Upload URL Response

{
"tx_id": "ca7496f7-a933-48d5-bccf-b1dd10f2b9e6",
"upload_url": "https://plb.di-beta.boldpenguin.com/file-transfer/upload-file?data=AAA...."
}

Step 3: Upload the File to AWS S3

  • Use the upload_url returned as part of Step 2 as the Host and upload the file.

Sample Request to Upload File

PUT /file-transfer/upload-file?data=AAAAuAECAwB..... HTTP/1.1
Host: plb.di-beta.boldpenguin.com
Content-Length: 231
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="Sample File Name.eml"
Content-Type: <Content-Type header here>

(data)
----WebKitFormBoundary7MA4YWxkTrZu0gW

Sample Response on File Upload

{
"data": "s3://riskgenius-riskgenius-beta-eu-uploadfiless3bucket-a7aqf1omt7i/003A21a5-1cc6-4a7f-b7cf-90be527d2209"
}

Step 4: Trigger the File Processing Flow

Endpoint

Staging

POST https://api-smartdata.di-beta.boldpenguin.com/universal/v3/universal-submit/file/{tx_id}

Production

POST https://api.ii.boldpenguin.com/universal/v3/universal-submit/file/{tx_id}

Pass the following variables as part of the request

Request ParametersComments
Authentication TokenThe token obtained in STEP 1
API Key(Shared with the Customer)
tx_idThe tx_id returned as part of Step 2: Get the file upload URL

Sample Universal Submit Request

POST /universal/v3/universal-submit/file/ca7496f7-a933-48d5-bccf-b1dd10f2b9e6 HTTP/1.1
Host: api-smartdata.di-beta.boldpenguin.com
x-api-key: M4jHKvQ6dD4Y1sDiv7gEe6LIuqArhDin8XldfxoK
Authorization: Bearer NMav128jTZbbEK48o64UM94K

Sample Universal Submit Response

{
"tx_id": "ca7496f7-a933-48d5-bccf-b1dd10f2b9e6",
"tx_start_time": "2022-05-31T19:48:46.356876+00:00",
"tx_end_time": "2022-05-31T19:48:52.370923+00:00",
"response_time_secs": 6.014047,
"tenant_id": "kowlaski",
"status": "Scheduled"
}

Upon submission completion, data extraction is a long running process. Each stage of the flow will transition the status of the transaction as below:

Submission_in_progress -> Scheduled -> Started -> Completed or Failed


Accessing the Universal Submit Status Check API

  • Once the file is submitted for processing, we can check the status by calling the Universal Submit Status Check API

Step 5: Access the Universal Submit Status Check API

Universal Submit Status Check URL

Staging

GET https://api-smartdata.di-beta.boldpenguin.com/universal/v3/universal-submit/status/{tx_id}

Production

GET https://api.ii.boldpenguin.com/universal/v3/universal-submit/status/{tx_id}

Pass the following variables as part of the request

Request ParametersComments
Authentication TokenThe token obtained in STEP 1
API Key(Shared with the Customer)
tx_idThis Id is returned as part of the Universal Submit response

Sample Universal Submit Status Check Request

GET /universal/v3/universal-submit/status/ca7496f7-a933-48d5-bccf-b1dd10f2b9e6 HTTP/1.1
Host: api-smartdata.di-beta.boldpenguin.com
x-api-key: M4jHKvQ6dD4Y1sDiv7gEe6LIuqArhDin8XldfxoK
Authorization: Bearer NMav128jTZbbEK48o64UM94K

Sample Universal Submit Status Check Response

{
"tx_id": "ca7496f7-a933-48d5-bccf-b1dd10f2b9e6",
"tenant_id": "kowalski",
"user_id": "9009f7fd-8920-4960-b2fb-f8c5b4bcaed4",
"tx_start_time": "2022-05-31T19:50:11.673863+00:00",
"tx_end_time": "2022-05-31T19:50:11.676083+00:00",
"response_time_secs": 0.00222,
"tx_status": "Started"
}

The status will begin as “Pending”, move to “Started”, and will then either have a status of “Completed” or “Failed”.

NOTE: Additionally, there will also be a c_report_id and l_report_id generated upon successful completion of data for enrichment

{
"tx_id": "8343ae00-ae0b-47c7-9abd-6ff05a75f982",
"tenant_id": "kowalski",
"user_id": "test",
"tx_start_time": "2021-06-28T16:18:35.139714+00:00",
"tx_end_time": "2021-06-28T16:18:35.505500+00:00",
"response_time_secs": 0.365786,
"status": "Completed",
"c_report_id": "CUS1187A9346D1C83129",
"l_report_id": "LUS1187A9346D1C8F76E"
}

Universal Submit Status Details URL

Staging

GET https://api-smartdata.di-beta.boldpenguin.com/universal/v3/universal-submit/status/{tx_id}/details

Production

GET https://api.ii.boldpenguin.com/universal/v3/universal-submit/status/{tx_id}/details

Pass the below variables as part of the request

Request ParametersComments
Authentication TokenThe token we get in STEP 1
API Key(Shared with the Customer)
tx_idThis Id is returned as part of the Universal Submit response

Sample Universal Submit Status Details Request

GET /universal/v3/universal-submit/status/ca7496f7-a933-48d5-bccf-b1dd10f2b9e6/details HTTP/1.1
Host: api-smartdata.di-beta.boldpenguin.com
x-api-key: M4jHKvQ6dD4Y1sDiv7gEe6LIuqArhDin8XldfxoK
Authorization: Bearer NMav128jTZbbEK48o64UM94K

Sample Universal Submit Status Details Response

{
"tx_id": "9fb6b3fe-5e90-4610-a2ab-dbea6656dd0e",
"tenant_id": "kowalski",
"user_id": "0715308c-db1c-4e63-b854-779bc93bc77c",
"tx_start_time": "2022-05-03T13:56:23.892802+00:00",
"tx_end_time": "2022-05-03T13:56:25.189395+00:00",
"response_time_secs": 1.296593,
"status": "Completed",
"flows": [
{
"document_name": "Sample File Name.eml",
"document_id": "bd78ba10-61f0-4238-be9f-a2fa45f9abd3",
"errors": [],
"classified_documents": [
{
"document_id": "d3149882-5c1a-420a-b904-c42348a23459",
"document_name": "Sample File Name.eml",
"document_type": "EML",
"number_of_pages_or_sheets": 1,
"startTime": "2022-05-31T19:55:44.582000",
"status": "Succeeded",
"review_id": "09db4e29-b39c-423c-b60b-15b0fc0f721d",
"endTime": "2022-05-31T19:58:27",
"review_url": "https://riskgenius-internal.di-beta.boldpenguin.com/genius-review/09db4e29-b39c-423c-b60b-15b0fc0f723d"
}
]
}
]
}

Universal Submit Status Response Structure

The following is the sample response structure returned from the Universal Status Inquiry API.

  • The HTTP response code would be 200 for all the successful requests.
  • Any failures because of authentication, authorization, service unavailable, rate limits exceeded; would return the standard HTTP Error
    Response codes.
  • Since there could be multiple documents as part of the request, the response will have metadata recorded for every document. This is arranged based on the classification of the documents. If any exceptions while processing the document are also recorded at the document level.
  • Any exceptions while orchestrating the processing of the document are also captured at the document level.
  • The tx_status aggregate represents the status of the submitted request. The values could be:
    • SUBMISSION IN PROGRESS
    • SCHEDULED
    • STARTED
    • REVIEW_REQUIRED
    • COMPLETED
    • FAILED
  • The structure of the error message will have an error_code and error_description. The error_description will have details about the exception. The details of different error codes are defined below.
{
"tx_id": "67611507-0687-4aec-aca1-0a9d3976a69c",
"tx_start_time": "2022-03-08T18:53:57.242Z",
"tx_end_time": "2022-03-08T18:54:11.075Z",
"response_time_secs": "13",
"data_package_id": "kowalski-us-c0006",
"tenant_id": "kowalski",
"user_id": "test",
"c_report_id": "CUS1187A9346D1C83129",
"l_report_id": "LUS1187A9346D1C8F76E",
"doc_reference_id": "9fb6b3fe-5e90-4610-a2ab-dbea6656dd0e ",
"seed_input": {
"company_name": "Somerville",
"address": "20 County Line Rd Branchburg NJ 08876-3498 UK"
},
"cleansed_input": {
"companyName": "SOMERVILLE ALUMINUM INC.",
"website": "www.somervillealuminum.com",
"phoneNumber": "8003463693",
"confidence_score": 0.9966666666666667,
"address": {
"streetAddr": "20 COUNTY LINE RD",
"city": "BRANCHBURG",
"state": "NJ",
"zipOrPostalCode": "08876",
"country": "US",
"countryISOCode": "",
"geoPosition": {
"latitude": 40.6076817,
"longitude": -74.7155544
}
},
"seedInput": {
"companyName": "SOMERVILLE ALUMINUM INC",
"address": {
"streetAddr": "20 County Line Rd",
"city": "Branchburg",
"state": "NJ",
"zipOrPostalCode": "08876",
"country": "US"
}
},
"dunsNumber": "011739927"
},
"data": {
"facts": {
"company_name": "SOMERVILLE ALUMINUM INC.",
"website": "www.somervillealuminum.com",
"address": "20 County Line Rd",
"city": "BRANCHBURG",
"state": "NJ",
"postal_code": "08876",
"duns_number": "011739927",
"legal_entity_type": "Corporation",
"year_founded": "1956",
"annual_revenue": "4947336.0",
"bankruptcy_indicator": "N",
"smart_company_credit_grade": "A",
"primary_naics_2012": [
{
"code": "444110",
"desc": "Home Centers"
}
],
"secondary_naics_2012": [
{
"document_name": "<doc_name>",
"document_id": "<doc_id>",
"errors": [{// error before processing
"error_code": "<err_code>",
"error_description": "<err_description>"
}],
"classified_documents": [
{
"document_type": "<doc_type>",
"number_pages_or_sheets": "<number>",
"processing_time": "<number>",
"status": "<status>",
"review_id": "<document_review_id>",
"review_url": "<document_review_url>",

"errors": [{// if error while document processing
"error_code": "<err_code>",
"error_description": "<err_description>"
}]
}
]
}
}

Supported Error Codes

Error CategoryHTTP Error CodeError CodeError Description
SECURITY_ERROR401Unauthorized
403Access Forbidden
429RATE_LIMIT_REACHED
BAD_REQUEST400ERR_0100INVALID_INPUT
ERR_0101INVALID_INPUT_SIZE_EXCEEDS_RANGE
ERR_0102UNSUPPORTED_MEDIA_TYPE
ERR_0103MISSING_MANDATORY_INPUT
ERR_0104DATA_NOT_FOUND
INTERNAL_SERVER500ERR_0200INTERNAL_SERVER
ERR_0201INTEGRATION_ERROR
ERR_0202CONNECTION_ERROR
ERR_0203TIMEOUT_ERROR
ERR_0204IO_ERROR
ERR_0205DB_ERROR
ERR_0300DOC_PROCESSING_ERROR
ERR_0301DOC_CLASSIFICATION_ERROR
ERR_0302DOC_EXTRACTION_ERROR
ERR_0303DATA_ENRICHMENT_ERROR
ERR_0401AI_CLEANSER_ERROR