Data Cleaning

Herald has logic to transform the values of risk, coverage, and admin parameters. These rules are applied when submitting values in an application, and when making a submission. These rules are enforced to ensure that every value submitted to an institution is formatted correctly.

Rules

This is the full list of rules that Herald applies to values. Below, you can see the list of rules applied based on the [.h-code-link]input_type[.h-code-link] of a parameter.

[.icon-circle-blue][.icon-circle-blue] This is the full set of rules as of 9/28/2023
Transformation Description Examples Notes
stringToNumber Convert string to number "1,000,000"1000000
"$100"100
Only cleans up correctly-spaced commas and single leading “$”. Does not handle "10,00" or "$$100".
handleNone Handles situations where people submit none of the above along with another selection ["None of the above", "Foobar"]["None of the above"]
cleanDomain Removes http:// and https:// from beginning "https://foo.com""foo.com"
trimWhiteSpace Removes leading and trailing whitespace " foo ""foo"
cleanPhone Removes non-digit characters and, if present, a leading 1 (country code) "+1 (222) 333-4444""2223334444"
cleanPostalCode Removes non-digit any anything after first 5 digits "02144-1234""02144"
emptyStringToNull Converts empty string to null ""null ⚠️ This only applies to address and claim_event today.

Input Types

Based on a parameters input type, the following rules will will be applied.

Input Type Transformation (s)
integer stringToNumber
currency stringToNumber
string trimWhiteSpace
email trimWhiteSpace
domain cleanDomain
select_many handleNone
phone cleanPhone
address.line1 trimWhiteSpace
address.line2 trimWhiteSpace
emptyStringToNull
address.line3 trimWhiteSpace
emptyStringToNull
address.organization trimWhiteSpace
address.city trimWhiteSpace
address.state trimWhiteSpace
address.postal_code cleanPostalCode
address.country_code trimWhiteSpace
claim_event.description trimWhiteSpace
emptyStringToNull