Different types of regression

I have always felt that regression is a very versatile tool. It can be used for measurement (to explain what happened), for analysis (to understand drivers) and for forecasting. It has a long history and still has relevance in our analytical suite of tools. 

Some of the evolution of regression is very interesting from the perspective of how shortcomings have been addressed. Some of the main arguments / shortcomings against regression are that it does not handle multicollinearity well (especially when you need driver analysis) and some of the assumptions (like the independence of the errors and the explanatory variables) that never seem to be satisfied. Research on these dimensions have led to improvements in methods that can handle these issues. There are three interesting ideas that I want to highlight in this week's blog post.  

There are many ways to handle multicollinearity in analysis. It's importance is driven by the fact that when one needs to measure the impact of key variables, it needs to be independent of other variables that could bias the measurement. Principal component analysis and factor analysis are options to handling multicollinearity but there are significant challenges in interpreting results after that. Latent class is a good way of handling this (and I will be discussing this in the future). Ridge (and Lasso) regression is a simple idea of handling multicollinearity in regression. Conceptually in Ridge regression, to handle multi-collinearity in the data, bias is introduced in the data. This has the effect of reducing the variance in the data which leads to better estimates from an analysis perspective. 

One other disadvantage of least squares regression is it's lack of flexibility. Variable transformations and interactions do add a lot of flexibility but there is one technique that adds a lot more flexibility. Local regression (also known as LOESS regression (or LOWESS - locally weighted least squares)) adds the flexibility that many machine learning techniques have. It does bring in some elements of computational intensity required to handle this but can add the flexibility to deliver interpretable results. Local regression basically creates local subsets to build models on and can hence manage very non-linear relationships well. 

One interesting issue in regression usage has been the difficulty in dealing with counter-intuitive results. Bayesian Regression provides an approach to formulate hypothesis that can be incorporated into the regression analysis. This can help bring in prior knowledge to play an important role in the analysis while minimizing very counter-intuitive results. Of course, as with all regression techniques, the modeler will need to use his intelligence to get to the best models.

In any case, there is a lot more to regression than meets the eye! 

Analytical software for analysts - are they way too complex?

Are there analytical software out there that actually make learning from data intuitive? I have experience with quite a few of these packages but none of them are intuitive for the average business analyst without making them useless after looking at data in one or two dimensions. While this is good for business, I must admit that it makes life difficult as the problems one has to tackle get quite mundane when responding to queries from the not so statistically literate. 

What would be the ideal requirements for one to actually be able to get ideas from data? Let us assume that the average user has a sense of the business he / she is dealing in. At the end of the analysis he should be able to get a sense of how to drive the business forward or at least has a good sense of what are some of the drivers that would explore further. Let us further assume that the average business user also has the ability to understand counter-intuitive results and can basically understand two dimension analysis and can possibly understand three dimension analysis but will be unable to move forward beyond that. 

Ideally when my business problems are well-defined (in the sense that I at least know what I want to solve initially even though I might realize that I need to solve something much larger later), then these tools should be able to at least drive some initial value for the analysts by incorporating these business requirements. But when I am sifting through data without a clue as to what I am looking for, how do I identify patterns that are meaningful and at the same time not require me to be in that business domain forever?

Regression analysis required significant understanding of the statistics to be able to confidently drive the analysis. CART / CHAID type algorithms are relatively easier to understand but I am not sure if there are decent implementations of a software that makes the learning from CHAID / CART intuitive. Bayesian networks or topological data analysis might be an answer but I have not worked enough with these to have a viewpoint on the implementation perspective. These are good with identifying patterns but do not necessarily make it easier for the business to get their reads better.

Ultimately I believe business problems need to be solved with the business context in mind and there are no general software that will enable that. Is it time for one to be created?

To Bayes or not to Bayes

Why is this argument important? For a long time, the argument of the frequentist approach was that the the data generating mechanism had a distribution where the parameters that described the distribution were fixed. While this made sense initially (why would that parameter change in any case) and all you would do would be to estimate that parameter based on the data that you observe. However Bayesian inference came into the world much later and postulated (I am not sure who did it specifically) that I should be using any prior information that I have about the parameter estimate and not necessarily let it be driven purely by data.

While this in theory sounded quite radical initially, there have been significant contributions that have enabled this idea to be used successfully in very practical applications. Specifically, Bayesian regression is quite useful to build updating models by using continuous data collection mechanism as opposed to waiting till models deteriorate to the point of having to be rebuild. This can incorporate a good test and learn setup from a data input perspective. These models have very practical applications in credit scoring, churn analysis and customer acquisition.

The machine learning world took to Bayes theorem a lot more seriously than the statistical crowd. Algorithms which assumed prior knowledge and then were updated based on fresh data seemed to make a lot more sense. Spam filtering is one of the biggest application of this theorem. A general rule to define spam based on many emails can be a baseline, and the model can then be updated based on user characteristics and performance. This allows the spam filter to be very customized to the user.

Judea Pearl is one of the pioneers in looking at Bayesian Inference from a fresh new perspective. Graph theory has been in mathematics for quite a long time. However the usage of Bayesian theory enabled a fresh new perspective in this domain and Bayesian Networks is the result of this marriage. The network structure allows one to incorporate a lot more variables in the model and measure causal relationships which previously was only available in the time series domain (Will write on this later!).

The bottom line that I see is that the frequentist approach is outdated and we need to develop that perspective when looking at new models. This should be the way we think of incorporating models in the real world.