Ullrich Köthe (computer science), Jens Kleesiek (radiology), Stefan Radev (psychology), Christoph Wiesinger (theology), WS 2019/2020
Thursdays, 14:15-15:45, Mathematikon A, 2nd floor, SR 2/103
Arguments are often supported by statistical analysis, but all too often these statistics are wrong. Sometimes, numbers and arguments are intentionally tweaked into desired directions, but even more often people arrive at incorrect conclusions despite of their best intentions. "The world is awash in bullshit.", as the seminar "Calling Bullshit" at the University of Washington (which inspired this course) puts it. We want to take a look behind the mechanisms of bullshit: How do people defend causal relationships, when there are in fact only spurious correlations? Which fallacies are common traps leading to incorrect arguments? Why is bullshit so powerful in practice? and, most importantly: How can we preserve a critical mind and counter bullshit?
Please register for the seminar via Müsli and send me an email with your favorite papers. Every participant needs to present a 40 minute talk (plus discussion) and hand-in a 10 page report (due 6 weeks after the talk), both in English. Participation gives 4 credit points according to computer science rules. Students of other fields (e.g. Physics) may also get 6 credits, if they invest considerable more work (e.g. implement and test the presented methods) and their faculty permits this regulation.
Schedule
| 17. October | Ullrich Köthe: Introduction | |
| Peter Hügel: How to lie with statistics (Huff) | Slides | Report | |
| 24. October | Florian Fallenbüchel: How to lie with charts (Jones) | Slides | Report |
| Marvin Ruder: The Visual Display of Quantitative Information (Tufte) | Slides | Report | |
| 31. October | Julius Drück: The Demon-Haunted World (Sagan) | Slides | Report |
| Claire Zhao Sun: Intersubjective reality (Harari, Sapiens) | Slides | Report | |
| 7. November | Valentin Wüst: Reasoning Fallacies | Slides | Report |
| Mustafa Ibrahim: Statistical Fallacies | Slides | Report | |
| 14. November | Sophie Jost: Biases in Questionaires | Slides | Report |
| Lennart Stipulkowski: "Flexible" Data Collection | Slides | Report | |
| 21. November | Meike Steinhilber: Reproducibility of Psychological Science | Slides | Report |
| Frederik Stegmüller: Shortcomings of p-values and multiple testing bias | Slides | Report | |
| 28. November | Climate Week Special | |
| Oliver Mehling: Climate skepticism | Slides | Report | |
| Peter Lippmann: Climate attribution science | Slides | Report | |
| 5. December | Hannes Kepler: Pitfalls of counterfactual inference | Slides | Report |
| 12. December | Lasse Becker-Czarnetzki: Is the winner really the best? | Slides | Report |
| 19. December | Karl Thyssen: Debunking myths effectively | Slides | Report |
| Mihai Verzan: Learning to avoid cognitive biases | Slides | Report | |
| 9. January | Jens Müller: Methods of epidemiology | |
| Raphael Hirsch: How many deaths did the Chernobyl desaster cause? | Slides | Report | |
| 16. January | Patrick Damman: The Book of Why 1 | Slides | Report |
| Jasper Henze: The Book of Why 2 | Slides | Report | |
| 23. January | Marina Walther: Health Effects of Smoking | Slides | Report |
| Aysegül Peközsoy: Populism | Slides | Report | |
| 30. January | Charlotte Boys: Communicating Uncertainty | Slides | Report |
| Duc Anh Phi: How to do better? | Slides | Report | |
| 6. February | Tara Butler: Rumor Cascades | Slides | Report |
Topics to Choose From:
Topic 1: The Classics
The techniques described in these books, although decades old, are still fully functional.
- (Speaker: Peter Hügel) Darrel Huff (1954): "How to Lie with Statistics" (PDF)
- Robert Thouless (1953): "Straight and Crooked Thinking" (PDF)
- Walter Krämer (1992): "So lügt man mit Statistik" (PDF?) (see also: Unstatistik des Monats by the same author)
- Kuzon et al. (1996): "The seven deadly sins of statistical analysis" (PDF)
- Alex Reinhart (2015): "Statistics done wrong" (HTML, PDF)
Topic 2: Case study: Graphs, Charts, and Maps
Visualization is a great way to lie with statistics. These books explain what can go wrong and how to do it right.
- (Speaker: Florian Fallenbüchel) Gerald Everett Jones (1995): "How To Lie With Charts"
- Alberto Cairo (2019): "How charts lie"
- Mark Monmonier (1996): "How to Lie with Maps" (PDF)
- (Speaker: Marvin Ruder) Edward Tufte (1983): "The Visual Display of Quantitative Information" (PDF) -- the classic on how to do it right
- Alberto Cairo (2016): "The Truthful Art: Data, Charts, and Maps for Communication" (PDF: first 40 pages, chapter 4, chapter 6, slides by the author)
- Comprehensive list of data visualization books (pick your favorite).
Topic 3: The Limits of Knowledge and the Power of Bullshit
What can science at best achieve? Why is bullshit so common?
- Karl Popper (1934): "Logik der Forschung" (PDF). Since this book is pretty long, you can also read "Grundprobleme der Erkenntnislogik".
- (Speaker: Julius Drück) Carl Sagan (1996): "The Demon-Haunted World" (PDF)
- (Speaker: Claire Zhao Sun) Yuval Noah Harari (2011/2014): "Sapiens" -- How shared beliefs made humankind rule the earth, falsehood of those beliefs notwithstanding.
Topic 4: Fallacies of Thinking
We frequently fall into thinking traps, and clever liars systematically exploit this.
- Wikipedia: List of Fallacies (with nice examples by David McCandless): work your way through the links and pick the most interesting stuff for your talk.
- Arthur Schopenhauer (ca. 1830): Die Kunst, Recht zu behalten
- (Speaker: Valentin Wüst) How we fool ourselves: reasoning fallacies
- Confirmation bias:
- Gambler's fallacy:
- Base rate fallacy:
- Overconfidence and hindsight bias:
- Moore and Healy (2007): "The Trouble with Overconfidence" (PDF)
- Dorota Skala (2007): "Overconfidence in Psychology and Finance – an Interdisciplinary Literature Review" (PDF)
- Dawson et al. (1988): "Hindsight Bias: An Impediment to Accurate Probability Estimation in Clinicopathologic Conferences " (PDF) -- overconfidence when the outcome is already known
- Expert failures:
- Camerer and Johnson (1991): "The Process-Performance Paradox in Expert Judgment: How Can Experts Know So Much and Predict So Badly?" (PDF)
- (Speaker: Mustafa Ibrahim) How numbers fool us: statistical fallacies
- Wikipedia: "Misuse of Statistics", Geckoboard: "Statistical Fallacies"
- Stephen Campbell (1974): "Flaws and Fallacies in Statistical Thinking"
- Tyson Holmes (2004): "Ten categories of statistical errors" (PDF)
- David Slutsky (2013): "Statistical Errors in Clinical Studies" (PDF)
- How we fool others: presentation tricks
- (Speaker: Sophie Jost) Choi and Pak (2005): "A Catalog of Biases in Questionnaires" (PDF) -- how to influence the outcomes of questionaires by asking the "right" questions
- Mark Battersby (2003): "The Rhetoric of Numbers: Statistical Inference as Argumentation" (PDF) -- how to beef up your results by "proper" rhetoric presentation of your numbers
- Fernandez-Duque et al. (2014): "Superfluous Neuroscience Information Makes Explanations of Psychological Phenomena More Appealing" (PDF) -- how to increase acceptance of your claims by adding superfluous information
Topic 5: The Replication Crisis
In some fields of science, a scary fraction of published papers (up to 90%) are not reproducible by others!
- John Ioannidis (2005): "Why Most Published Research Findings Are False" (PDF)
Leek and Jager (2016): "Is Most Published Research Really False?" (PDF) - Measuring the severity of the crisis:
- Begley and Ellis (2012): "Raise standards for preclinical cancer research" (PDF)
- (Speaker: Meike Steinhilber) Open Science Collaboration (2015): "Estimating the reproducibility of psychological science" (PDF) -- find that ~50% are not reproducible
Gilbert et al. (2016): "Comment on 'Estimating the Reproducibility of Psychological Science'" (PDF) -- claim that the above findings are wrong and there is no crisis - Camerer et al. (2018): "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015" (PDF)
- (Speaker: Frederick Stegmüller) The shortcomings of p-values and the multiple testing bias:
- Jacob Cohen (1994): "The Earth Is Round (p<0.05)" (PDF)
- John Ioannidis (2017): "What Have We (Not) Learnt from Millions of Scientific Papers with P Values?" (PDF)
- Szucs and Ioannidis (2017): "When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment" (PDF)
- Wasserstein et al. (2019): "Moving to a World Beyond p<0.05" (PDF)
- (Speaker: Lennart Stipulkowski) Devastating effects of "flexible" data collection and hind-sight claims:
- Norbert Kerr (1998): "HARKing: Hypothesizing After the Results are Known" (PDF)
"Registered Reports" -- prevent HARKing by announcing the hypothesis before data collection - Simmons et al. (2011): "False-Positive Psychology" (PDF)
- Shun-Shin and Francis (2013): "Why Even More Clinical Research Studies May Be False: Effect of Asymmetrical Handling of Clinically Unexpected Values" (PDF)
- Head et al. (2015): "The Extent and Consequences of P-Hacking in Science" (PDF)
- Norbert Kerr (1998): "HARKing: Hypothesizing After the Results are Known" (PDF)
- (Speaker: Victor Zimmermann) Publication bias -- only report your successes:
- Turner et al. (2008): "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy" (PDF)
- Dickersin and Chalmers (2011): "Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO" (PDF)
- Song et al. (2013): "Publication bias: what is it? How do we measure it? How do we avoid it?" (PDF)
- Nissen et al. (2016): "Publication bias and the canonization of false facts" (PDF) -- modelling the acceptance of bullshit as a Markov process
- (Speaker: Lasse Becker-Czarnetzki) The pitfalls of competitions:
- Detecting statistical errors in published papers:
- J. B. Carlisle (2012): "The analysis of 168 randomised controlled trials to test data integrity" (PDF)
- J. B. Carlisle (2017): "Data fabrication and other reasons for non-random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals" (PDF)
- Nuijten et al. (2015): "The prevalence of statistical reporting errors in psychology (1985–2013)" (PDF) -- present statcheck, computer program detecting statistical errors in PDFs
Topic 6: Correlation vs. Causation
Frustratingly often, people report causal relationships, when they in fact only observed correlations.
- (Speaker: Robert Fietz) Tyler Vigen (2015): "Spurious Correlations" (website)
- (Speaker: Robert Fietz) Calude and Longo (2016): "The Deluge of Spurious Correlations in Big Data" (PDF)
- (Speaker: Robert Fietz) Krämer and Gigerenzer (2005): "How to Confuse with Statistics or: The Use and Misuse of Conditional Probabilities" (PDF)
- (Speaker: Hannes Kepler) King and Zeng (2007): "When Can History Be Our Guide? The Pitfalls of Counterfactual Inference" (PDF)
- Imai et al. (2008): "Misunderstandings Among Experimentalists and Observationalists about Causal Inference" (PDF)
- Judea Pearl (2009): "Causal inference in statistics: An overview" (PDF)
- (Speakers: Patrick Dammann and Jasper Henze) Pearl and Mackenzie (2018): "The Book of Why" (book website)
Topic 7: Case study: Epidemiology
Epidemiology is the study and analysis of health and disease conditions in defined populations, using statistical methods.
- (Speaker: Pingchuan Ma) Methods of epidemiology
- Bonita et al. (2006): "Basic epidemiology" (PDF English, deutsch)
- Austin Bradford Hill (1965): "The Environment and Disease: Association or Causation?" (PDF)
- Richard Doll (2002): "Proof of Causality: Deduction from Epidemiological Observation" (PDF)
- Rothman and Greenland (2005): "Causation and Causal Inference in Epidemiology" (PDF)
- King and Bearman (2009): "Diagnostic change and the increased prevalence of autism" (PDF)
- Michael B. Bracken (2012): "Risk, Chance, and Causation: Investigating the Origins and Treatment of Disease" -- Demonstrating the difficulty of separating the hype from the hypothesis, this book clearly communicates how clinical epidemiology works.
- (Speaker: Marina Walther) The health effects of smoking:
- Cornfield et al. (1959): "Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions" (PDF) -- a pioneering paper of epidemiology.
praise for this paper on its 50th anniversary:- Joel Greenhouse (2009): "Commentary: Cornfield, Epidemiology and Causality" (PDF)
- Marcel Zwahlen (2009): "Commentary: Cornfield on cigarette smoking and lung cancer and how to assess causality" (PDF)
- George Davey Smith (2009): "Smoking and lung cancer: causality, Cornfield and an early observational meta-analysis" (PDF)
- David Cox (2009): "Commentary: Smoking and lung cancer: reflections on a pioneering paper" (PDF)
- Jan Vandenbroucke (2009): "Commentary on: ‘Smoking and lung cancer’ — the embryogenesis of modern epidemiology" (PDF)
- Barnes and Bero (1998): "Why Review Articles on the Health Effects of Passive Smoking Reach Different Conclusions" (PDF)
Lisa Bero (2005): "Tobacco industry manipulation of research" (PDF)
Hong and Bero (2002): "How the tobacco industry responded to an influential study of the health effects of secondhand smoke" (PDF)
Barnes et al. (2006): "The Tobacco Industry’s Role in the 16 Cities Study of Secondhand Tobacco Smoke: Do the Data Support the Stated Conclusions?" (PDF) - Cummings et al. (2007): "The Cigarette Controversy" (PDF) -- a study examining the controversy's history
- Cornfield et al. (1959): "Smoking and Lung Cancer: Recent Evidence and a Discussion of Some Questions" (PDF) -- a pioneering paper of epidemiology.
- (Speaker: Raphael Hirsch) How many deaths did the Chernobyl desaster cause? Claims in the literature differ between less than one hundred to over a million
- UNSCEAR (2008): "Health effects due to radiation from the Chernobyl accident" (Annex D of the 2008 UNSCEAR Report, PDF)
- Cardis et al. (2006): "Estimates of the cancer burden in Europe from radioactive fallout from the Chernobyl accident" (PDF)
- Yablokov et al. (2007): "Chernobyl: Consequences of the Catastrophe for People and the Environment" (PDF)
- (Speaker: Carola Schreier) Is vaccination dangerous?
- Salemi and D'Amelio (2010): "Could Autoimmunity Be Induced by Vaccination?" (PDF)
Netherlands Pharmacovigilance Centre Lareb (2016): "Long-lasting adverse events following immunization with Cervarix" (PDF) - The Informed Parent Co. (2019): Shouldn’t the after-effects of vaccination be discussed before? -- arguments of an opponent to measles vaccination
Naturheilmagazin (2019): Argumente absoluter Impfgegner
Robert Koch-Institut (2019): Antworten auf typische Einwände gegen das Impfen - Siddiqui et al. (2013): "Epidemiology of vaccine hesitancy in the United States" (PDF)
- Chandrakant Lahariya (2016): "Vaccine epidemiology: A review" (PDF)
- Horne et al. (2015): "Countering antivaccination attitudes" (PDF)
- Masaryk and Hatoková (2016): "Qualitative inquiry into reasons why vaccination messages fail " (PDF)
- Jonathan Kennedy (2019): "Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data" (PDF)
- Jonathan Kennedy (2019): "Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data" (PDF)
- Salemi and D'Amelio (2010): "Could Autoimmunity Be Induced by Vaccination?" (PDF)
- Health effects of air pollution
- Are smartphones dangerous?
- Manfred Spitzer: "Die Smartphone-Epidemie" (2019) and "Digitale Demenz" (2012). Spitzer is a master of unstatistic (see e.g. "Der Smartphone-Alarmist") -- can we figure out where he is right or wrong?
Topic 8: The Mechanisms of Bullshit
To fight fake information, it is crucial to understand how it spreads.
- Ong and Glantz (2001): "Constructing “Sound Science” and “Good Epidemiology”: Tobacco, Lawyers, and Public Relations Firms" (PDF) -- An attempt to establish “good scientific practices” that would make it impossible to conclude that environmental toxins caused diseases.
- (Speaker: Oliver Mehling) Climate skepticism:
- Michaels and Monforton (2005): "Manufacturing Uncertainty: Contested Science and the Protection of the Public’s Health and Environment (PDF) -- how skeptics question the validity of scientific evidence
- John Cook (2010): "The Scientific Guide to Global Warming Skepticism" -- how do climate change skeptics cast doubt on scientific evidence?
- Corner et al. (2014): "Uncertainty, scepticis, and attitudes towards climate change: biased assimilation and attitude polarisation" (PDF) -- psychological experiments on people's reactions to contradictory claims
- (Speaker: Tara Butler) Friggeri et al. (2014): "Rumor Cascades" (PDF) -- network analysis of how rumors spread on Facebook
- (Speaker: Tara Butler) Kwon et al. (2013): "Prominent Features of Rumor Propagation in Online Social Media" (PDF) -- similar, but on Twitter
- (Speaker: Aysegül Peközsoy) Silva et al. (2017): "The Elite Is Up to Something: Exploring the Relation Between Populism and Belief in Conspiracy Theories" (PDF)
- (Speaker: Aysegül Peközsoy) Rico et al. (2017): "The Emotional Underpinnings of Populism: How Anger and Fear Affect Populist Attitudes" (PDF)
- Angela Nagle (2017): "Die digitale Gegenrevolution" (book website)
Topic 9: How to do Better?
How to avoid mistakes in your analysis and communicate your results to the public.
- Carl Sagan (1996): "The Fine Art of Baloney Detection" (chapter 9 of his book "The Demon-Haunted World", PDF)
- Robert Pearson (2010): "Statistical Persuasion: How to Collect, Analyze, and Present Data...Accurately, Honestly, and Persuasively" (book website)
- (Speaker: Charlotte Boys) Communicating uncertainty:
- Stephens et al. (2012): "Communicating probabilistic information from climate model ensembles—lessons from numerical weather prediction" (PDF)
- David Spiegelhalter (2017): "Risk and Uncertainty Communication" (PDF)
- Willems et al. (2019): "Variability in the interpretation of Dutch probability phrases - a risk for miscommunication" (PDF) -- exactly how probable is an event that occurs "often" or "rarely"?
- (Speaker: Karl Thyssen) Debunking myths effectively: why it's hard and how it's done successfully?
- Cook and Lewandowsky (2011): "The Debunking Handbook"
- Lewandowsky et al. (2012): "Misinformation and Its Correction: Continued Influence and Successful Debiasing" (PDF)
- Nyhan and Reifler (2010): "When corrections fail: The persistence of political misperception" (PDF) -- the backfire effect
- Horne et al. (2015): "Countering antivaccination attitudes" (PDF)
- Masaryk and Hatoková (2016): "Qualitative inquiry into reasons why vaccination messages fail " (PDF)
- Farrell et al. (2019): "Evidence-based strategies to combat scientific misinformation" (PDF)
- (Speaker: Peter Lippmann) Climate attribution science: can we determine which extreme weather events are caused by global warning?
- Hannart et al. (2016): "Causal Counterfactual Theory for the Attribution of Weather and Climate-Related Events" (PDF)
- Theodore Shepherd (2016): "A Common Framework for Approaches to Extreme Event Attribution" (PDF)
- Ebert-Uphoff and Deng (2012): "Causal Discovery for Climate Research Using Graphical Models" (PDF)
- Myles Allen (2003): "Liability for climate change" (PDF)
- Herring et al. (2017): "Explaining Extreme Events of 2015 from a Climate Perspective" (PDF)
- (Speaker: Mihai Verzan) Learning to avoid cognitive biases:
- Croskerry et al. (2013): "Cognitive debiasing 1: origins of bias and theory of debiasing" (PDF)
- Croskerry et al. (2013): "Cognitive debiasing 2: impediments to and strategies for change" (PDF)
- Symborski et al. (2014): Missing: A Serious Game for the Mitigation of Cognitive Biases" (PDF)
Barton et al. (2015): The Use of Theory in Designing a Serious Game for the Reduction of Cognitive Biases" (PDF) - Daniel Willingham (2007): "Critical Thinking: Why Is It So Hard to Teach?" (PDF)
- see also: Daniel Kahnemann (2011): "Thinking, fast and slow" (PDF)
- (Speaker: Duc Anh Phi) Jordan Ellenberg (2014): "How not to be wrong: The Power of Mathematical Thinking" (PDF)
- Victoria Stodden (2015): "Reproducing Statistical Results" (PDF)
- Cook et al. (2018): "The Consensus Handbook" -- the range and importance of consensus to refute climate change skepticism
- Hans Rosling (2018): "Factfulness" (ebook) -- statistics and bubble graphs showing how the world has actually changed in the last decades (contrary to common belief)