Meta-analysis
Name
Institution
Course
date Executive summary
Although the advantages of using meta-analysis seem obvious, critics have been raised. The critics entail the possibility of introducing bias into sampling findings and placing emphasis on personal effects. There are various disadvantages associated with meta-analysis as witnessed in this research. This paper analyses a meta-analysis of Psychodynamic Psychotherapy Outcomes. It evaluates the Effects of Research-Specific Procedures.
The first weakness of DeCoster (2009) paper analysis is personal bias in choosing and including existing studies in the analysis. In the DeCoster (2009), there is no database that includes all empirical studies on the topic of interest. It is also true that not every computer assisted search can identify all journal articles on the chosen topic. Many good studies are not available simply because they were never published. There might be a publication bias; to mean, important results are more likely to be published, whereas non-significant results are neglected into file drawers. Meta-analysis researchers in DeCoster (2009) paper need to set a clear and consistent standard for including all valid studies that meet this standard and to make a great effort in including all valid studies that meet this standard.
In the recommendation, DeCoster (2009) paper researchers also must avoid personal bias in deciding which studies from the literature to include in the analysis (Joel et al., 2012).
The second drawback of meta-analysis in DeCoster (2009) originates from the huge variation of present studies. Even on the same research topic and question, existing empirical studies may vary considerably in theoretical foundations and methodological issues, such as sampling strategy, measurements of interested variables, data analysis techniques, and the reporting formats and contents. It is evident that there exist considerable changes among various studies.
Evaluation of Meta-analysis
Various scholars have argued in defending meta-analysis by stating that it helps in synthesizing disparate researches. Various scholars have arguments that even though past research varies in their methodology, a meta-analysis which is well designed considers variations by treating them as moderator variables. Meta-analysis researchers ought to be careful in the aggregation of various studies with different participants and sampling methods. They should also consider operationalization and measuring variables of interests which was not carried out in DeCoster (2009) paper. When joining studies, scholars should be attentive to appropriate moderator variables that can result to alterations in research outcomes (Joel et al., 2012).
The other drawback of meta-analysis in DeCoster (2009) paper is its dependence on personal effects on various predictors on a dependent variable. In DeCoster (2009) paper, Meta-analysis methodically measures only personal relations between dependent and independent variables and cannot give a broad picture. Rosenthal and Dimatteo (2001) argue that this sample, systematic approach is essential in most research domains, stating that individual effects and correlations provide a foundation of building a comprehensive model that integrates many individual variables. In addition, meta-analysis tends to be a powerful tool for examining the combination and interactions of individual predictor variables. Such analysis is an essential condition for realizing multi-level and multi-factorial models. Meanwhile, meta-analysis scholars in DeCoster (2009) paper ought to be aware of the loss of information when they concentrate on particular impacts at a time in the analysis, also taking into consideration probable interactions among predictor variables.
Internal Validity
Although the list of threats to validity in DeCoster (2009) may seem devastating there is good reason to continue (Joel et al., 2012). Unlike the pilot’s checklist, on which any item overlooked could spell disaster, it is generally agreed by which any item overlooked could spell disaster. It is generally agreed by even the most finicky statisticians that a meta-analysis is not automatic failure. Suppose it has a shortcoming or two in validity; to fail, its validity problems must be serious or multiple. Nevertheless, after looking at the validity threats to meta-analysis, one can only agree with Ingram Olkin that doing a meta-analysis may be easy but doing one well is hard.
Whether the implementation of the primary studies and the meta-analysis justify the claims the researchers are making should be assessed. That is, whether the primary studies and meta-analysis actually test what the researchers say is being tested. Over a dozen threats to internal validity have been identified in DeCoster (2009) paper. In primary studies, most can be avoided by the use of proper methods, with random assignment of subjects to treatment and control conditions being the most obvious. In meta-analyses, scholars can protect against majority of the key threats to interior validity by using only principal studies, an action that was not carried out in DeCoster (2009) paper. The studies use randomization or make statistical modifications compensating for its lack.
Two other significant fears to the interior validity of meta-analysis in DeCoster (2009) paper comprise unfinished literature searchers, which produce samples of studies, and untrustworthy information coding from various studies. These threats can be guided against by more complete methods of collecting studies and by checks on coding discussed earlier, an action that was never carried out.
Number of studies
Piot, (2003) points out the possibility that poorly conducted studies included in his previously published meta-analysis could have artificially inflated or decreased the magnitude of the aforementioned effects sizes. However, due to a limited number of studies in the 2001 meta-analysis, Piot, (2003) was unable to statistically test the impact of study quality on effect size. To address this issue, Jones, (2006) re-analysed the now larger video game literature and identified studies that contained few methodological flaws. In order to identify methodologically sounds studies, Kimball, (2006) made use of a nine-item coding scheme. Included within this scheme were the following weaknesses: a nonviolent video game condition that involved the paying of a game that actually contained violence; a violent video game condition that contained little violence; difference between the violent and nonviolent conditions in terms of difficulty, frustration level, or generated interest, and so on. When Owens 2006 limited the revised meta-analysis to best practice studies, the effects sizes for aggressive behaviour, aggressive cognitions, and hostile affects and physiological arousal all increased. This indicates weaknesses of meta-analysis and so their applicability in DeCoster (2009).
The measurable process of DeCoster (2009) meta-analysis can be employed in addressing the challenges imposed by the presence of diverse research finding on a particular question. It allows researchers to combine numerical results from a number of studies, to accurately estimate descriptive statistics. The researchers are also able to explain the inconsistencies of findings in the literature, and to discover moderate variables for a dependent variable of interest (Rosenthal & DiMatteo, 2001). The major strength of DeCoster (2009) meta-analysis comes from its capacity to help researchers reach accurate and credible conclusions that other research approaches cannot provide(Joel et al., 2012). Such as one single primary study and qualitative or narrative literature review. Meta-analytic research strategy nonetheless has both benefits and drawbacks.
Moderator check
Lastly, the other drawback of meta-analysis as evident in DeCoster (2009) paper is its inadequate capability of including other variables that are melodramatically dissimilar from the current theory (Yang, 2002). Meta-analysis researchers cannot form new hypothetical ideas past the variables and study qualities that have not been involved in current studies. Although researchers may be able to discover different effects unless existing studies have reported relevant features (Joel et al., 2012). Consequently, a meta-analytic approach to theory building tends to be more applicable to a “research-analytic approach to theory” than “theory-than-research” strategy of theory building. Meta-analysis, consequently, has its constraints in mounting and validation of pioneering theory.
Reliability check
The principal difficulty of meta-analysis as witnessed in DeCoster (2009)is individual bias in choosing and including present studies in the analysis. In DeCoster (2009), there is no one database that entails all experiential studies in the examination. Not all computer-helped searches can classify all journal articles on the research topic. Various researches are unavailable since they are unpublished there can be a publication bias. That is, the importance in results has higher chances of being published, while non-important outcomes are relegated into file drawers (Rosenthal, 1979). Meta-analysis scholars are to set a perfect and reliable standard for containing experiential studies and to create a pronounced effort in entailing all valid studies that attain these conditions. Researchers should also evade individual bias in determining the studies from the past research to contain in the analysis.
The other drawback of meta-analysis in DeCoster (2009) comes from the huge alteration of present research. Although similar research theme and question, current experiential studies may fluctuate considerably in theoretic foundations and organizational issues, for example sampling approach, measurements of concerned variables, data analysis methods, and the reporting setups and subjects (Joel et al., 2012). It is evident that there are substantial differences among published researches considering quality of research. Subsequently, some scholars have argued against the practice of meta-analysis for mixing good and bad studies.
The other critics have been raised concerning the contrast of various kinds of studies as being same to mixing different research designs (Hunt, 1997). Rosenthal (2001), though, have dissimilar studies. The arguments are that even though studies change methodologically, a meta-analysis well designed considers such variances by considering them as moderator variables. When uniting studies, researchers are to pay attention to appropriate moderator variables that can cause variances in research outcomes.
External Validity
Hall and Rosenthal (1995) suggest three basic principles to guide meta-analysis, certain processes are vital to meta-analytic study. A typical meta-analysis has the following steps:
Define variables of interest, and formulate the research questions. This is evident is DeCoster (2009) paper.
Search the literature, and identify adequate empirical studies in a systematic way. Though there is a thorough literature search in DeCoster (2009), but the search is biased.
Construct validity has to do with whether the measure used to appraise the outcome is a trustworthy indicator of effect or if it distorts the true intervention-outcome connection. DeCoster (2009) meta-analysis, for example, examined how well a mental test of job abilities correlated with individuals’ actual job performance. Of the measures used in the primary studies, the mental test was known to be fairly reliable, while the job performance rating, based on a single supervisor’s opinion in each case, was generally unreliable.
The last group deals with threats to statistical validity in DeCoster (2009). These include errors in the way data are analysis is carried out. At the chief level of study, such error comprise in the way data are analyzed. At the principal level, such errors comprise using numerical tests that are unsuitable for the kind of data, exploiting on chance, and not recording statistical tests that were done. At the meta-analytic level, statistical drawbacks include unsuitable suppositions when effect sizes must be assessed, bias in changing effect sizes, and failing to weigh the studies’ results by samples size and other situations.
One of the most widely cited meta-analytic studies on media violence is Anderson, Berkowitz, (2003) systematic review of over 215 empirical studies. Each of the individual studies included in this meta-analysis assessed the negative effects of violent imagery seen in movies and on television on aggressive behaviour. Meta-analysis in DeCoster (2009) proved to be an important piece of work, as the impact of violent television and movies on aggressive behaviour produced medium-sized effects sizes, far larger than many critics had suspected. More recently, Funk et al., 2002 conducted a systematic review of the impact of violent video games on aggression and aggression-related constructs using 35 independent research projects. The results of this meta-analysis provided empirical support for the contention that violent video games influence aggressive behaviour, one of the most widely cited meta-analytic studies on media violence is Funk et al., 2002 systematic review of over 215 empirical studies. This shows strength of meta-analysis and so their applicability in DeCoster (2009)(Joel et al., 2012).
The other demerit of meta-analysis in DeCoster (2009) and other areas is in its dependence on personal effects on various predictors on a dependent variable (Joel et al., 2012). Meta-analysis methodically measures only personal relations amid dependent and independent variables and cannot afford a broad picture. Rosenthal (2001) argue that this is easy, methodical approach is important in majority of researches, stating that personal impacts and associations give a foundation for erecting a complete model that assimilates many personal variables.
Theoretical contribution
One advantage of DeCoster (2009) meta-analytic design is its capacity to integrate and synthesize current empirical studies for a research question. DeCoster (2009) meta-analysis allows researchers to integrate the existing empirical studies for a research question. Meta-analysis lets investigators to assimilate the present findings with some erudite tools for instance combined tests. Because different existing studies may come from various empirical areas, a combined test tends to cumulate the existing findings in a scientific way and thus to present the results with more generalizability (Joel et al., 2012). Researchers understand that it is crucial to conduct a literature review, yet they often get inconsistent or even conflicting findings. Qualitative or narrative review of the literature cannot deal with such findings, and thus sometimes such a review can be quite confusing.
DeCoster (2009) meta-analysis provides a cumulative view of a specific research topic by carefully analyzing similarities and differences of methodologies and findings across many studies. In other words, meta-analysis aims at getting a whole picture. By coding existing studies quantitatively, meta-analysis researchers can keep track of a large amount of potential information and then conduct a more detailed analysis (Joel et al., 2012). DeCoster (2009) meta-analysis can easily summarize multiple variables from hundreds of studies that most narrative reviews cannot handle and so its advantage. In addition, meta-analysis allows researchers to examine a wider range of relationships, interactions, and other such complex analyses that are normally not permitted under qualitative research techniques.
Moderator
The second benefit or strengths of DeCoster (2009) meta-analysis for getting good research outcomes originates forms its nature of analyzing the analysis. DeCoster (2009) meta-analysis does not only cumulate outcomes from personal studies but also can be using test multifaceted theories entailing various variables (Joel et al., 2012). Since social and administrative phenomenon seems to be complicated, diverse theories from different domains have been put in place to assist explain such phenomena. There might be various competing philosophies or hypothetical frameworks in one research domain. For instance, scholars can recognize various predictors for the efficiency of training in firms, including design of training, various methods of training, ability or task features, and evaluation features (Arthur et al., 2003). DeCoster (2009) meta-analysis gives essential method of estimating the relative effects of present predictors on the dependent variable and this gives aggregated experiential results for studying and judging present studies.
The other strengths of DeCoster (2009) meta-analysis employing meta-analysis is its ability to offer strategies for selection of variables and designing the research in coming studies. Meta-analysis uses the chosen literature with experiential evidences. Such outlook has various utility. For instance, investigators can use such data to reflect on the present design and determine some hopeful variables for future research. They researchers in DeCoster (2009)used meta-analysis in developing new theoretical and hypothetical ideas founded on experiential evidence shown in meta-analysis permits researchers to improve and validate new theoretical philosophies based on experiential evidence revealed in meta-analysis for example moderators and interactions impacts(Joel et al., 2012). In summary, meta-analysis permits researchers to improve other theoretical ideas basing on likely attributes and features of all imaginable current studies. This is to say, meta-analysis can follow a research then theory strategy of building theory (Reynolds, 1971). Comparing with other strategies, the chief advantage of meta-analysis is that it is based on a number of proven empirical studies as witnessed in DeCoster (2009). For instance, meta-analysis is based on published works and not only one piece of research.
The other advantage of employing meta-analysis as research method originates from its function in the continuous development refinement of present theory. Through discovering and examining the important moderators and likely interactions impacts, meta-analysis gives solid conclusions about entailing other proven variables or discarding old, less significant variables in the present theories and theoretical models. Even though the demerits of employing meta-analysis are obvious, critics have been raised, entailing the possibility of introducing bias into findings sampling (Joel et al., 2012). Meta-analysis researchers ought to be alert of drawbacks in connection with this method.
When adding up the current developments in the field of individually controlled learning on one hand, and the strengths and restrictions of meta-analysis on the other, some implications for additional research become deceptive.
The outcomes of DeCoster (2009) meta-analysis alone do not give enough bases for creating entire recommendations for an intercession. Obviously, other evidences not included in DeCoster (2009) are also needed (Joel et al., 2012). Succeeding chief research should be the re-evaluation of the integrative outcomes of the meta-analysis in an experimental setting. An intervention research ought to be established, which considers the outcomes of the meta-analysis of training studies dealing with the improvement of the educational self-regulation. In such a research, intervening with perfect characteristics ought to be assessed.
In the same way, DeCoster (2009) meta-analysis ought to be considered as an experiential study which synthesizes personal experiments quantitatively on the base of previous study objectives. Because meta-analyses are founded on a bigger sample size, the analysis tends to be more exact and dependable than any of the principal studies that they assess.
With regards to further exploration of the concept of self-regulated learning, it would be interesting to conduct a meta-analytical evaluation of the training studies. The evaluation should be aimed at fostering self-regulated learning. Furthermore, the evaluation should also be designed to synthesize on a smaller scale the experimental research on how learners acquire self-regulated learning skills. The main aim is in investigating single strategies more in depth, an action that was thoroughly carried out in DeCoster (2009). This is one of the greatest strengths of DeCoster (2009) meta-analysis (Joel et al., 2012).
Further, in order to advance the scope of the research in DeCoster (2009)’s field, conclusions established in adjacent research areas. The meta-synthesis of Hattie (2009), for instance, discovered that the effect of tutors on learners is bigger when equating the collaborative inspirations of learner, educator, instructional, and school factors. The outcome should be considered in the research on learners’ attainment of individual-regulation by examining the effect of educator feedback on learners’ individual regulation. In the assessment of training program, these issues should be combined into the involvements meant to foster individual-regulating learning.
Meta-analysis is no longer a new method. Among the main difficulties associated with meta-analysis is its dependence on the main studies involved. The information as reported by the authors is often not complete. For instance, information on certain important decisions taken may not be revealed(Joel et al., 2012). One way of solving this problem could be that authors provide the meta-analysis with their raw data when their study is being published (see e.g., Glass, 2000; Hattie et al., in press). This would allow the reviewers to work with these primary data and carry out additional analyses, whereby at least some of the limitations of meta-analyses as mentioned earlier could be overcome.
Until more sophisticated methods are established to synthesize literature, DeCoster (2009)meta-analysis approach can offer an important overview of the state of the art in a particular research field. They can also provide useful information about the effectiveness of interventions or, in more general field and provide useful information between two variables under investigation as witnessed in DeCoster (2009). Especially in a growing and evolving research field such as the area of self-regulated learning, procedures are needed that can structure the existing body of evidence and help to find comprehensive answers, while also facilitating the formulation of new hypotheses for future studies.
Interpretation of findings
In analysing the meta-analysis for validity, Rosenthal and Rubin ensured that their outcomes would not leak away under close examination. The validity of meta-analysis refers to the soundness of the original studies and the processes used in combining the data, and a minimum of three dozen possible validity leaks were identified in the processes. Studying all the likely faults, one can feel that the entire enterprise in DeCoster (2009) paper is desperate, but the condition is not bad, to alter metaphors, than that of an aircraft pilot reading a preflight (Joel et al., 2012).
References
Yang B., 2009.Research in Organizations c.21.Business & Economics / Management Science. Publisher Berrett-Koehler Publishers.
Swanson, R., 2005. Research in Organizations: Foundations and Methods of Inquiry. The Berrett-Koehler Organizational Performance Series. Publisher Berrett-Koehler Publishers.
Diana B. 1999. Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis : Methods for Quantitative Synthesis in Medicine: Methods for Quantitative Synthesis in Medicine. Volume 31 of Monographs in Epidemiology and Biostatistics.Publisher Oxford University Press.
Polit, Denise F., Cheryl T., 2013. Essentials of Nursing Research: Appraising Evidence for Nursing Practice. 8th Edition.Revised Publisher Lippincott Williams & Wilkins.
Ringquist E., 2013. Meta-Analysis for Public Management and Policy.Publisher John Wiley & Sons.
Harris M. Cooper, Larry V. Hedges, 1994. The Handbook of research synthesis, Volume 236.The Handbook of Research Synthesis.Publisher Russell Sage Foundation.
Joel M., Marc J., and Allan A., 2012.A Meta-Analysis of Psychodynamic Psychotherapy Outcomes.Evaluating the Effects of Research-Specific Procedures.
