Psychological Methods

Testing model nesting and equivalence.
When using existing technology, it can be hard or impossible to determine whether two structural equation models that are being considered may be nested. There is also no routine technology for evaluating whether two very different structural models may be equivalent. A simple nesting and equivalence testing (NET) procedure is proposed that uses random sample and model-reproduced moment matrices to evaluate both model nesting and equivalence. The analysis is “local” rather than “global” in nature, but its use with simulation or bootstrapping can imply global conclusions. Two standard applications of NET are to verify whether or not two proposed models are equivalent and whether a baseline model used in an incremental fit index is appropriately nested. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue.
In recent years, single-case designs have increasingly been used to establish an empirical basis for evidence-based interventions and techniques in a variety of disciplines, including psychology and education. Although traditional single-case designs have typically not met the criteria for a randomized controlled trial relative to conventional multiple-participant experimental designs, there are procedures that can be adopted to create a randomized experiment in this class of experimental design. Our two major purposes in writing this article were (a) to review the various types of single-case design that have been and can be used in psychological and educational intervention research and (b) to incorporate randomized experimental schemes into these designs, thereby improving them so that investigators can draw more valid conclusions from their research. For each traditional single-case design type reviewed, we provide illustrations of how various forms of randomization can be introduced into the basic design structure. We conclude by recommending that traditional single-case intervention designs be transformed into more scientifically credible randomized single-case intervention designs whenever the research conditions under consideration permit. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
K-balance partitioning: An exact method with applications to generalized structural balance and other psychological contexts.
Structural balance theory (SBT) has maintained a venerable status in the psychological literature for more than 5 decades. One important problem pertaining to SBT is the approximation of structural or generalized balance via the partitioning of the vertices of a signed graph into K clusters. This K-balance partitioning problem also has more general psychological applications associated with the analysis of similarity/dissimilarity relationships among stimuli. Accordingly, K-balance partitioning can be gainfully used in a wide variety of SBT applications, such as attraction and child development, evaluation of group membership, marketing and consumer issues, and other psychological contexts not necessarily related to SBT. We present a branch-and-bound algorithm for the K-balance partitioning problem. This new algorithm is applied to 2 synthetic numerical examples as well as to several real-world data sets from the behavioral sciences literature. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Killeen's probability of replication and predictive probabilities: How to compute, use, and interpret them.
P. R. Killeen's (2005a) probability of replication (prep) of an experimental result is the fiducial Bayesian predictive probability of finding a same-sign effect in a replication of an experiment. prep is now routinely reported in Psychological Science and has also begun to appear in other journals. However, there is little concrete, practical guidance for use of prep, and the procedure has not received the scrutiny that it deserves. Furthermore, only a solution that assumes a known variance has been implemented. A practical problem with prep is identified: In many articles, prep appears to be incorrectly computed, due to the confusion between 1-tailed and 2-tailed p values. Experimental findings reveal the risk of misinterpreting prep as the predictive probability of finding a same-sign and significant effect in a replication (psrep). Conceptual and practical guidelines are given to avoid these pitfalls. They include an extension to the case of unknown variance. Moreover, other uses of fiducial Bayesian predictive probabilities for analyzing, designing (“how many subjects?”), and monitoring (“when to stop?”) experiments are presented. Concluding remarks emphasize the role of predictive procedures in statistical methodology. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
A model-averaging approach to replication: The case of prep.
The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of traditional methods. Here we show that prep overestimates the probability of concurrence. This is because prep was derived under the assumption that all effect sizes in the population are equally likely a priori. In many situations, however, it is advisable also to entertain a null hypothesis of no or approximately no effect. We show how the posterior probability of the null hypothesis is sensitive to a priori considerations and to the evidence provided by the data; and the higher the posterior probability of the null hypothesis, the smaller the probability of concurrence. When the null hypothesis and the alternative hypothesis are equally likely a priori, prep may overestimate the probability of concurrence by 30% and more. We conclude that prep provides an upper bound on the probability of concurrence, a bound that brings with it the danger of having researchers believe that their experimental effects are much more reliable than they actually are. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Killeen's (2005) prep coefficient: Logical and mathematical problems.
In his article, “An alternative to null-hypothesis significance tests,” Killeen (2005) urged the discipline to abandon the practice of pobs-based null hypothesis testing and to quantify the signal-to-noise characteristics of experimental outcomes with replication probabilities. He described the coefficient that he invented, prep, as the probability of obtaining “an effect of the same sign as that found in an original experiment” (Killeen, 2005, p. 346). The journal Psychological Science quickly came to encourage researchers to employ prep, rather than pobs, in the reporting of their experimental findings. In the current article, we (a) establish that Killeen's derivation of prep contains an error, the result of which is that prep is not, in fact, the probability that Killeen set out to derive; (b) establish that prep is not a replication probability of any kind but, rather, is a quasi-power coefficient; and (c) suggest that Killeen has mischaracterized both the relationship between replication probabilities and statistical inference, and the kinds of claims that are licensed by knowledge of the value assumed by the replication probability that he attempted to derive. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Replication, prep, and confidence intervals: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).
This comment offers three descriptions of prep that start with a frequentist account of confidence intervals, draw on R. A. Fisher's fiducial argument, and do not make Bayesian assumptions. Links are described among prep, p values, and the probability a confidence interval will capture the mean of a replication experiment. The descriptions suggest the criticism of Maraun and Gabriel (2010) is unjustified. Iverson, Wagenmakers, and Lee (2010) discussed prep in terms of Bayesian model averaging. This went usefully beyond the dichotomous decision making of significance testing, but an extension to Bayesian estimation would be welcome. Lecoutre, Lecoutre, and Poitevineau (2010) referred to and extended their substantial research based on predictive approaches. Some of the links they make among p values, confidence intervals, and prep parallel links described earlier, although their conceptual framework is different. The interesting prep experiment in Psychological Science may be coming to a close; it suggests that statistical innovation, including that proposed by Iverson et al. (2010) and Lecoutre et al. (2010), is likely to be most successful if guided by cognitive evidence and supported by resources tailored for researchers generally. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
ppep Replicates: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).
Lecoutre, Lecoutre, and Poitevineau (2010) have provided sophisticated grounding for prep. Computing it precisely appears, fortunately, no more difficult than doing so approximately. Their analysis will help move predictive inference into the mainstream. Iverson, Wagenmakers, and Lee (2010) have also validated prep as a boundary condition in a model-averaging approach. I argue that the boundary is good; that the proper place for their assignment of subjective priors is to update personal belief; and that such assignment has no place in the evaluation of evidence, on which priors should be flat. Maraun and Gabriel (2010) have provided clarification, context, and critique of the derivation of prep. They concluded that prediction can never be precise without knowledge of population parameters and joint empirical distributions; that nature evolves; that a large initial effect will encourage replication attempts, which are therefore not independent of it; and that there is no substitute for a sustained program of replication attempts—all of which are prudent cautions. If statistics is to be the handmaiden of science and policy, however, rather than history, it ineluctably must speak to the future. The posterior predictive distribution is an underexploited tool in the analyst's kit that will serve that end, and prep is but one valid implementation of it. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
Regarding prep: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).
The sense that replicability is an important aspect of empirical science led Killeen (2005a) to define prep, the probability that a replication will result in an outcome in the same direction as that found in a current experiment. Since then, several authors have praised and criticized prep, culminating in the 3 articles in the current issue of Psychological Methods. In this article, Killeen's prep is reviewed, and the contributions of the current articles are summarized and discussed. An examination of the role of a measure of theoretical support such as prep in the acquisition of knowledge leads me to concur with Senn (2002) that prep is of little epistemological value. (PsycINFO Database Record (c) 2010 APA, all rights reserved)

Psychological Methods - Vol 15, Iss 2