The third dimension that is critical to establishing data integrity is proper data controls or the ability to ensure what you actually captured or processed is correct. Proper data controls that must be established include:

Data that is reported and used in critical business decisions originates somewhere and thus it is crucial that data capture controls are properly implemented. For example, in a warranty system, when a dealer enters in reimbursements for warranty items, we must ensure that they do not enter values that exceed the possible range of values or enter in fraudulent warranty claims. There must be logic in the application to ensure that only valid claims are processed and that users are immediately notified when incorrect information is entered.

Similarly, when data is moved across the Internet, it is possible that someone may tamper with the data if proper controls are not placed on the data. This may compromise the integrity of that data. Consider the case of a subsidiary of a corporation that is on a separate network that must transmit data to a parent via the Internet. If this data is extremely sensitive, there is the potential that the data could be intercepted and altered by malicious individuals. In cases where the data is paramount to corporate integrity, companies should consider sending sensitive data via a virtual private network (VPN) or utilizing encryption techniques to mask the data.

Perhaps the data controls that are most important in the business intelligence world are the ones around data processing. In many of the business intelligence efforts that I've seen, very little emphasis is applied to validating the cleanliness and accuracy of processing. Let's start with data cleanliness. It's true that if data input controls were robustly implemented that data cleanliness would not be an issue. The truth is that data still comes in with junk values and must be dealt with appropriately in business intelligence systems.

Proper controls must be in place to ensure that improper records and values are filtered out before entering the data warehouse. Additionally, these improper records must be tracked and accounted for as part of the data verification process. For example, if 10 records out of 100 are excluded due to improper values, this must be tracked in the audit log and accounted for. The second piece of data processing controls are controls to ensure that the processing of data happened properly. In order to preserve integrity, we must track if input records were actually processed properly. This includes reporting and monitoring around record input counts and record output counts. Input/output verification can be performed either through hash token verification or simply counting input and output records. This goes for algorithm validation as well. Your data processing algorithms must be working properly, and controls are needed to validate this. By implementing controls during the data processing process, you can confirm that your results are right and thereby provide a structure for data integrity.

In the data world, ensuring that the data gets captured, processed and reported is only a small piece of the puzzle. You must ensure that those results are right and can be validated by proper controls. By building effective data controls into your design that is commensurate with the sensitivity of the data you are handling, people will have more trust in your data and therefore in the integrity of the results that you present.