No Bytes

Many analysts know it: You find a significant correlation - and then you wonder how secure the knowledge actually is. Of course, a good model inevitably requires an evaluation based on test data, because there is no way around it. But sometimes doubts remain - because even test data can be unreliable. At the beginning of this year, during my work, I found such a significant correlation (0.8 after Spearman). However, an evaluation was only possible to a limited extent, because due to external circumstances many attributes of the data can change in the respective department and not all changes are documented. Incompleteness in the dataset - that too will be known to analysts. Now I asked myself the question: how can I secure my knowledge nevertheless. The nature of the correlation was of central importance. The situation was similar to a well-insulated building in which the room temperature is dependent on both the outside temperature and a heating. Due to the thick insulation of t...

Meanwhile I am working as a data analyst for an industry company. One part of my job is to research production processes and reducing defects on the products to avoid them. Unfortunately during analysis with decision trees and logistic regression models a correlation very often is given, though a causality in many cases doesn't exist. And even if a causality exists, I have to consider the direction to avoid mistaking cause and impact. All these considerations are known under the term "Business Understanding". If you don't have the business understanding for your job, data mining and more simply analytics can get misleading very fast. Business Understanding an its spread Nearly every company that uses analytics and data mining has a strong concern in using the insights proper, because a wrong used information can cause bad consequences. This issue is that present in the heads of many managers, that they attach a lot of importance to business understanding. Th...