Quantitative Mehods: Which statistical test to use?

I wrote this post in another blog a couple of years ago and thought I should share this here as well:

If you are going to statistically analyse your data, knowing the right statistical test is absolutely essential. However, even if you have taken a couple of statistical courses, it is often difficult to decide which ones are the most relevant or correct tests.

If you are in that position you might find these links useful:

1. http://www.ats.ucla.edu/stat/mult_pkg/whatstat/default.htm
2. http://www.graphpad.com/www/Book/Choose.htm

Don't be surprised if your data can be analysed in many different ways depending on your objectives and hypotheses.

NVIVO 10 training

Last week, I attended the basic and advanced NVIVO 10 training sessions (2.5 hours each) on the 3rd and 6th. Some of my friends were quite surprised as I am a "quantitative" guy and one even joked "You are moving to the other camp". This actually underlines the philosophical divide that exists in academia (especially in the social sciences) between the positivists and the interpretivists in particular.

My research philosophy is grounded in positivism and my PhD Thesis was based upon extensive quantitative research, using structural equation modelling. Previously, I did not have the inclination (nor the time) to learn more about the various qualitative methods (and tools like NVIVO). However, after teaching research methods and the fact that a number of my dissertation students were using qualitative methods forced me to move outside my comfort zone. There also seems to be an increasing move towards research involving mixed methods which I feel is a good development. I am really into netnography, the process and term coined by Professor Robert V. Kozinets, Professor of Marketing, Schulich School of Business, York University.

Anyway, the session were led by Dr. Laura Venn, Co-founder and Director of Research, Innovative Futures Research, a company based in Warwick. I found the two session really useful and wish I had taken them much earlier. I have used qualitative methods to analyse discussions forums on the net (paper submitted for presentation at a Conference recently) and it would have made my life much easier. I had to download hundreds of webpages and code them the old fashioned way with a lot of cut and pasting. The ability to visualize connections by generating tag clouds, tree maps and cluster analysis are what attracted me in the first place. However a new amazing feature of NVIVO 10 is that it allows you to to actually capture all the information from the discussion forums and social media including Facebook and Twitter straightaway.

Laura did provide some examples and I know the time was quite limited but I wish we had more hands on training using actual exercises. I did learn a lot but will definitely need to play around more before I am really fluent with it. Fortunately QSR (the company which owns NVIVO) offers a number of free online webminars and I am attending one tomorrow.

You can download trial copies of the software (lasts for 30 days) here: http://www.qsrinternational.com/products_nvivo_free-trial-software.aspx

Sample size in Quantitative studies

Is your sample size sufficient for your quantitative analysis? This is a question that a lot of doctoral researchers ask. I guess it depends on the type of analysis used. My PhD research looked at blog readers and I used Structural Equation Modelling (SEM) to analyse my data. I used the following to justify my sample size.

• For factor analysis, according to Hair et al., (2010), a sample should preferably be more than 100 for factor analysis to proceed
• However,  according to Tabachnick and Fidell (2007 p. 613), it should be higher than 300 cases, the number that is considered “comfortable”.

If you are doing SEM:

• A ratio of ten responses per free parameters is required to obtain trustworthy estimates (Bentler and Chou, 1987).
• Others suggest a rule of thumb of ten subjects per item in scale development, is prudent (Flynn and Pearcy 2001).
• However, if data is found to violate multivariate normality assumptions, the number of respondents per estimated parameter increases to 15 (Bentler and Chou 1987; Hair, et al. 2006).

References:

BENTLER, P. M. & CHOU, C. P. (1987) Practical issues in structural modeling. Sociological Methods & Research, 16, 78.

FLYNN, L. & PEARCY, D. 2001. Four subtle sins in scale development: Some suggestions for strengthening the current paradigm. International Journal of Market Research  43: (4) 409-433.

HAIR, J. F., BLACK, W. C., BABIN, B. J. & ANDERSON, R. E. 2010. Multivariate Data Analysis: A Global Perspective, New Jersey, Pearson Prentice Hall.

TABACHNICK, B. G. & FIDELL, L. S. 2007. Using Multivariate Statistics (5th Ed.) New York, HarperCollins.

Non-normal data and SEM

There are a number of interesting discussions going on in the Doctorate Support Group on Facebook. One of the more recent discussion was started by one of the members who had normality issues with her data.

Normality is an issue because it is one of the basic assumptions required in order to carry out structural equation modelling (SEM) analysis (BYRNE, B. M. (2010) Structural equation modeling with AMOS: Basic concepts, application and programming, New York, Routledge: Taylor and Francis Group.).

Normality means that the distribution of the data is normally distributed with mean=0, standard deviation=1 and a symmetric bell shaped curve. Normally the Skewness and Kurtosis measures are checked:

• Skewness: value should be within the range ±1 for normal distribution.
• Kurtosis: Value should be within range ±3 for normal distribution.

So what happens when your data is non-normal?

You don't have to worry unless the departure from normality is very severe.

1. Several studies have shown that most data in social sciences has non-normal distribution.

Bentler, P.M., & Chou, C.-P. (1987). Practical issues in structural modeling. Sociological Methods & Research, 16, 78-117.

Barnes, J., Cote,J., Cudeck, R. and Malthouse, E. (2001). Checking Assumptions of Normality before Conducting Factor Analyses. Journal of Consumer Psychology, 10(1/2), pp. 79-81.

2. The ML estimator is considered relatively robust to violations of normality assumptions.

Diamantopoulos, A., Siguaw, J. & Siguaw, J. A. (2000). Introducing LISREL: A guide for the uninitiated, Sage Publications.
Bollen, K. A. (1989) Structural equations with latent variables, Wiley New York.

3. Monte-Carlo experiments found no major differences in terms of SEM analysis results using ML estimator on samples of different sizes and with different Kurtosis and Skewness levels.

Reinartz, W., Haenlein, M. & Henseler, J. (2009). An empirical comparison of the efficacy of covariance-based and variance-based SEM. International Journal of Research in Marketing, 26, 332-344.

4. Bootstrapping is increasingly being used to get around this issue.

Preacher, K. J. & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, p. 717.

5. Large sample size leads to reduction in the problem if multivariate non-normality.

Hair, J. F., Black, W. C., Babin, B. J. & Anderson, R. E. (2010) Multivariate Data Analysis: A global perspective Upper Saddle River, NJ, Pearson Education Inc.

Note: Severely non-normal data would probably need another alternative approach.