Different types of regression

I have always felt that regression is a very versatile tool. It can be used for measurement (to explain what happened), for analysis (to understand drivers) and for forecasting. It has a long history and still has relevance in our analytical suite of tools. 

Some of the evolution of regression is very interesting from the perspective of how shortcomings have been addressed. Some of the main arguments / shortcomings against regression are that it does not handle multicollinearity well (especially when you need driver analysis) and some of the assumptions (like the independence of the errors and the explanatory variables) that never seem to be satisfied. Research on these dimensions have led to improvements in methods that can handle these issues. There are three interesting ideas that I want to highlight in this week's blog post.  

There are many ways to handle multicollinearity in analysis. It's importance is driven by the fact that when one needs to measure the impact of key variables, it needs to be independent of other variables that could bias the measurement. Principal component analysis and factor analysis are options to handling multicollinearity but there are significant challenges in interpreting results after that. Latent class is a good way of handling this (and I will be discussing this in the future). Ridge (and Lasso) regression is a simple idea of handling multicollinearity in regression. Conceptually in Ridge regression, to handle multi-collinearity in the data, bias is introduced in the data. This has the effect of reducing the variance in the data which leads to better estimates from an analysis perspective. 

One other disadvantage of least squares regression is it's lack of flexibility. Variable transformations and interactions do add a lot of flexibility but there is one technique that adds a lot more flexibility. Local regression (also known as LOESS regression (or LOWESS - locally weighted least squares)) adds the flexibility that many machine learning techniques have. It does bring in some elements of computational intensity required to handle this but can add the flexibility to deliver interpretable results. Local regression basically creates local subsets to build models on and can hence manage very non-linear relationships well. 

One interesting issue in regression usage has been the difficulty in dealing with counter-intuitive results. Bayesian Regression provides an approach to formulate hypothesis that can be incorporated into the regression analysis. This can help bring in prior knowledge to play an important role in the analysis while minimizing very counter-intuitive results. Of course, as with all regression techniques, the modeler will need to use his intelligence to get to the best models.

In any case, there is a lot more to regression than meets the eye! 

Analytical software for analysts - are they way too complex?

Are there analytical software out there that actually make learning from data intuitive? I have experience with quite a few of these packages but none of them are intuitive for the average business analyst without making them useless after looking at data in one or two dimensions. While this is good for business, I must admit that it makes life difficult as the problems one has to tackle get quite mundane when responding to queries from the not so statistically literate. 

What would be the ideal requirements for one to actually be able to get ideas from data? Let us assume that the average user has a sense of the business he / she is dealing in. At the end of the analysis he should be able to get a sense of how to drive the business forward or at least has a good sense of what are some of the drivers that would explore further. Let us further assume that the average business user also has the ability to understand counter-intuitive results and can basically understand two dimension analysis and can possibly understand three dimension analysis but will be unable to move forward beyond that. 

Ideally when my business problems are well-defined (in the sense that I at least know what I want to solve initially even though I might realize that I need to solve something much larger later), then these tools should be able to at least drive some initial value for the analysts by incorporating these business requirements. But when I am sifting through data without a clue as to what I am looking for, how do I identify patterns that are meaningful and at the same time not require me to be in that business domain forever?

Regression analysis required significant understanding of the statistics to be able to confidently drive the analysis. CART / CHAID type algorithms are relatively easier to understand but I am not sure if there are decent implementations of a software that makes the learning from CHAID / CART intuitive. Bayesian networks or topological data analysis might be an answer but I have not worked enough with these to have a viewpoint on the implementation perspective. These are good with identifying patterns but do not necessarily make it easier for the business to get their reads better.

Ultimately I believe business problems need to be solved with the business context in mind and there are no general software that will enable that. Is it time for one to be created?

Can regression measure everything?

For a long time, I have been pushing for a better understanding of regression. Regression gives us insight into the multivariate relationship that exists in the world. It is difficult to visualize these relationships as the number of dimensions can exceed human imagination. For all that complexity though, regression is an ancient concept (by measure of the speed at which new techniques come into the analytics industry). Why has it not been adopted to understand the world a lot more?

I think due to the complexity in visualizing these relationships, there is resistance to using these ideas. I am adamant that people who show me anything think along those lines. There is a chance that some of these insights can be developed based on individual analyses slowly. It will be a challenge to ensure that you can highlight everything.

While people look at regression to tell us what will happen, I believe regression is a tool that is best used for measurement. The more complex the relationship, the more important is to ensure that we get the measurement framework right. The measurement of the impact of engine size on mileage is straightforward, but the measurement of the marketing spend on TV on its impact on sales is not so easy. Due to significant relationships between may contributing factors, teasing out the impact of TV marketing spend is a challenge that marketers have tried to solve with no easy solution. While these aspects might pose challenges to using a regression framework, there are a couple of other places where regression may be misleading.

1. When we have non-linear temporal relationships, a straightforward regression approach will not measure relationships accurately leading to misleading diagnoses.
2. When there is a feedback loop, regression usage might even lead to counter-intuitive relationships. While these relationships may not be difficult to recognize, they need to be measured with other techniques to get the right perspective.