Friday, March 27, 2009

Week 12: Experiment and Quant. Description

Week 12 Blog Post

Experimental Study

Quick Lauer and Asher summary

True Experiments: According to Lauer and Asher, true experiments apply a treatment to a research subject in order to show cause an effect. Preferably, the researchers will control the variables and then change a variable(s) and observe the change in the research subject. Normally, the true experiment requires a control group to compare the reaction, changes, or differences with the group that received the treatment. The control and treatment groups are randomized. While not exclusive to a true experiment, a test hypothesis normally drives the variables involved. The hypothesis is also dependent on the result being able to dispel the null hypothesis. The results are them statistically compared to the likeliness of a type I/II error as well as the probability of chance.
Quasi-Experimental: Quasi-experimental studies differ from “true” experiments based on the inability to randomize, an inability to control all variables, and potentially unequal groupings. Often, and preferably, the quasi-experiments include a pretest to ensure that research subjects are comparable. While there is an attempt to generalize with the results, cause and effect should actually be correlation with variables. Like the true experiment, the quasi must also test a null hypothesis.

Caroll

Caroll’s method was a quasi-experiment testing the manual of a computer program as the independent variable. The study included a mall pretest, with 19 participants (office professional with some experience) using the two different manuals. The study lacked randomization (one reason for the “quasi” classification) and in addition did not have a “control group” that was given the same instructions without a manual. This would have strengthened the pretest because the manuals could potentially be statistically indistinguishable compared to a no-manual group. The second experiment tested more participants but also tested more variables. The limited number of participants (sample size) hindered the study and made the results a bit shaky considering the generalizations. Once again, claiming the minimal manual’s benefits seems premature due to the sample size and a comparison against a no-manual group. The subject needs to be analyzed further. Unlike the author’s claim of “less can be more” the study needs more to overcome the lack of sample and design. Also, isn’t there an issue about correlation is not causation?

Kroll

With another quasi-experimental study, Kroll uses a larger but stratified sample to determine how game rules are explained. Kroll uses 123 (non-random) students ranging from the 5th grade to freshmen college students. However, Kroll screened the participants and removed those who did not score high enough on a quiz. This was an unusual quirk and did not receive adequate explanation and appeared to be data manipulation/steering. The researchers filmed the students on multiple occasions on separate day, and the intercoder reliability was .76 with the explanatory approach. Again, the cause and effect was misapplied and it should have been that grade level has a strong correlation with the informative explanation. While the researcher may have been able to identify some correlation between student levels and explanation, the removal of observation/subjects lessens the values. In addition, I question how the students were taught them game. The video could have had different implications o learning style and age where the subjects understood the instructions differently.

Notarant/Cohen

And there are three quasi-experiments. Notarant and Cohen test communication styles on sales interactions. Using videotapes, videos include the different styles selling a stereo. While there are 80 subject, the explanation of the methodology was vague. They include a small sample (n=10) which I am unclear if that is the group size or the number of groups. The research subject (college students) were not random and were placed in various groups of unequal size (5-6). I not sure how the subjects were organized. While the authors randomize the subjects of the group, the subjects were not random… kinda misleading. In addition, the limited scope of age and higher female ratio limited the overall generalization ability of the study.

Quantitative Description

A super-quick Lauer and Asher summary

Quantitative descriptions seek to isolate, correlate, interrelate variables. It is not experimental since it does not describe a treatment to variables. For the most part, it is a statistical means for identifying relevant variables that is the qualitative research can also identify, but now allows for stronger generalization. (I just think the n=10 per variable is a lousy rule… there are other issues to deal with such as population size)

Faber
Faber has conducted a content analysis of popular media publications. He follows Huckin’s outline, which is actually a good approach (I used Huckin for the Client project). The project reduced the number from over 800 to 203 which strengthened/focused the project. He justified the units of analysis/themes. Finally the theme were compared based on subject and over time. Overall it was a well executed content analysis.

Golen
The better aspect of this research project was that the 25 variables (showing up along a broad range) were defined from the literature. Otherwise, the study is a bit of a mess. The “random” selection of the 10 breakout groups was weird, especially since there were 400 students in 3 classes… and 279 responses. He analysis was convoluted. The Likert scale scale results had large st. deviations. Also, stating 1 and “most of the time” and 5 as “never” seemed awkward. It also left looking at the data difficult (remembering that the lower scores meant more listening but then more laziness). Also, it was a difficult keeping the variables organized and some were not very well connected. It felt that the results were forced into the conclusions already established by the literature. It was a good set up to a project, but the execution of the research was awkward.