Chapter 17 Conclusions

You’ve made it to the end of Experimentology, our (sometimes opinionated) guide to how to run psychology experiments.

17.1 Summarizing our approach

In this book we’ve tried to present a single unified approach to the why and how of running experiments. This approach begins with the goal of doing experiments:

Experiments are intended to make maximally unbiased, generalizable, and precise estimates of specific causal effects.

This formulation isn’t exactly how experiments are talked about the broader field, but we hope you’ve started to see some of the rationale behind this way of talking.

  • The emphasis on causal effects stems from an acknowledgement of the key role of experiments in establishing causal inferences (Chapter 1) and the importance of causal relationships to theories (Chapter 2).

  • A desire for specificity informs our preference for simple experimental designs (Chapter 9).

  • We select our sample in order to maximize the generalizability of our findings to some target population (Chapter 10).

  • We also try to minimize the possibility of bias through our decisions about data collection (Chapter 12) and data analysis (Chapter 11).

  • A focus on estimation (Chapter 5 helps us avoid some of the fallacies that come along with dichotomous inference (Chapter 6), and allows us to think meta-analytically (Chapter 16) about the overall evidence for a particular effect.

  • The desire for precision informs our choice of reliable and valid measures (Chapter 8).

Woven throughout this narrative is the hope that embracing open science values throughout the experimental process will help you maximize your work. Not only is sharing your work openly an ethical responsibility (Chapter 4), it’s also a great way to minimize errors while creating valuable products that both advance scientific progress and accelerate your own career (Chapter 13).

17.2 Forward the field

As we’ve learned at various points in this book, there’s a replication crisis (Open Science Collaboration, 2015), a theory crisis (Oberauer & Lewandowsky, 2019), and a generalizability crisis (Yarkoni, 2020) in psychology. Based on all these crises, you might think that we are pessimistic about the future of psychology. Not so.

There have been tremendous changes in psychological methods in the ten years since we began teaching our course (2012 – 2022). When we began, it was common for incoming graduate students to describe the rampant \(p\)-hacking they had been encouraged to do in their undergraduate labs. Now, students join the class aware of new practices like preregistration and cognizant of problems of generalizability and theory building. It takes a long time for a field to change, but we have seen tremendous progress at every level – from US government policies requiring transparency in the sciences all the way down to individual researchers’ adoption of tools and practices that increase the efficiency of their work while also decreasing the chances of error.

One of the most exciting trends has been the rise of meta-science, in which researchers use the tools of science to understand how to make science better. Reproducibility and replicability projects (reviewed in Chapter 3) can help us measure the robustness of the scientific literature. In addition, studies that evaluate the impacts of new policies (e.g., Hardwicke et al., 2018) – can help stakeholders like journal editors and funders make informed choices about how to push the field towards more robust science.

In addition to changes that correct methodological issues, the last ten years have seen the rise of “big team science” efforts that advance the field in new ways (Coles et al., 2022). Collaborations such as the Psychological Science Accelerator (Moshontz et al., 2018) and ManyBabies (Frank et al., 2017) allow hundreds of researchers from around the world to come together to run shared projects. These projects are enabled by open science practices like data and code sharing, and they provide a way for researchers to learn best practices via participating. In addition, by including broader and more diverse samples they can help address challenges around generalizability (R. A. Klein et al., 2018).

Finally, the last ten years have seen huge progress in the use of statistical models both for understanding data (McElreath, 2018) and for describing specific psychological mechanisms (Ma et al., 2022). In our own work we have used these models extensively and we believe that they provide an exciting toolkit for building quantitative theories that allow us to explain and to predict the human mind.

17.3 Final thoughts

Doing experiments is a craft, one that requires practice and attention. The first experiment you run will have limitations and issues. So will the 100th. But as you refine your skills, the quality of the studies you design will get better. Further, your own ability to judge others’ experiments will improve as well, making you a more discerning consumer of empirical results. We hope you enjoy this journey!


Coles, N. A., Hamlin, J. K., Sullivan, L. L., Parker, T. H., & Altschul, D. (2022). Build up big-team science. Nature Publishing Group.
Frank, M. C., Bergelson, E., Bergmann, C., Cristia, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Kline, M., Levelt, C., et al. (2017). A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy, 22(4), 421–435.
Hardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., Lenne, R. L., Altman, S. K., Long, B., & Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal cognition.
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams Jr, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahnı́k, Š., et al. (2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490.
Ma, W. J., Körding, K., & Goldreich, D. (2022). Bayesian models of perception and action: An introduction. unpublished.
McElreath, R. (2018). Statistical rethinking: A bayesian course with examples in r and stan. Chapman; Hall/CRC.
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., Castille, C. M., Evans, T. R., Fiedler, S., Flake, J. K., Forero, D. A., Janssen, S. M. J., Keene, J. R., Protzko, J., Aczel, B., … Chartier, C. R. (2018). The psychological science accelerator: Advancing psychology through a distributed collaborative network. Adv Methods Pract Psychol Sci, 1(4), 501–515.
Oberauer, K., & Lewandowsky, S. (2019). Addressing the theory crisis in psychology. Psychonomic Bulletin & Review, 26(5), 1596–1618.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
Yarkoni, T. (2020). The generalizability crisis. Behav. Brain Sci., 1–37.