Oracle Data Mining and Analytics

Analytics, such as data mining and statistics, produce interesting and useful results. However, can we trust this type of technology and the applications built on top of them?

Here is the issue: data mining and statistics do not produce "exact" results. In fact, the results one gets when asking the same question using different techniques are usually different. To top it all, computations can be order dependent and non-repeatable across different platforms. This is at odds with most computations usually performed in a database. These are repeatable and exact. For example, the number of sales per country is not order dependent and do not change due to differences in algorithms. In a way, we could compare these computations to bean counting. There is no guessing, estimations, or approximations. From this perspective, for most types of calculations, the database is a sophisticated and powerful bean counting device.

How can we come to terms with the "inexact" or approximated nature of the results generated by advanced analytical techniques? Interestingly enough, we are quite used to this type of behavior in our day-to-day life. It is not uncommon for us to enlist the services of an expert when dealing with complex tasks or activities. For example, we routinely seek advice from medical doctors and investment advisers. Also, expert advice is not exact and "repeatable." It is not unusual to get as many different answers as the number of experts we consult. How do we manage these interactions? Here is where trust comes into play, Trust is developed over time through a history of successful interactions with an expert or acquired through referral. In the latter case, a trusted party vouches for the quality of the expert. When we do not trust the expert, seeking a second opinion is a common way to handle the "inexact" nature of expert advices. This approach can be viewed as "expert pooling." In these cases, we either take the "average advice" or the majority advice. We also usually weight each expert's opinion by how much we trust the expert.

If we rely on experts that provide inexact advise, why not software? Working with analytical components requires a similar concept of trust. We need to feel reassured that the technology we are using, or a particular component, can be trusted. This can only be developed by using the technology over time and realizing that it works or, again, by referral. The later can be the reputation of a company, or the validation that an approach works for solving a type of problem. We can also apply "expert pooling" to analytical results. For example, we can use different techniques and take the average or majority result. We can weight each technique's contribution by how much we trust its results, as well. Suppose we want to estimate house value y based on house characteristics x. We could create two models (model1 and model2) using two different techniques. We could then compute the y as model1(x) * t1 + model2(x) * t2, where model1(x) and model2(x) are the estimates generated by model1 and model2 respectively, and t1 and t2 are how much we trust each model.

One way we can assign trust to a model is to compute its performance on a held aside data set not used for creating the model. Let's say e1 and e2 are the performances for models 1 and 2 on a held aside data set then we could compute t1 as e1/(e1+e2) and t2 as e2/(e1+e2).

In summary, in order to address the inexact nature of analytical software results, we need to develop an attitude similar to the one we use to deal with expert advice. We need to understand that these techniques are providing advices or answers to complex questions. Like expert advice, and unlike bean counting, there is uncertainty and margin for error in the answers. To manage the uncertainty, we need to cultivate trust and apply expert pooling strategies. Trust will increase as the use of analytical components become more prevalent. The latter requires making these techniques easier to use. This is the topic for a future post.

Let me know what you think is necessary to address the "inexact" or approximated nature of advanced analytical results. Do you think it is all a matter of trust?