## Ten Foundational Quantitative Reasoning Questions

### I. What do the numbers show?

- What do the numbers mean?
- Where are the numbers?
- Is there numerical evidence to support a claim?
- What were the exact figures?
- How can seeking and analyzing numbers illuminate important phenomena?

- How plausible is a possibility in light of back of the envelope calculations?

### II. How representative is that?

- What's the central tendency?
- "For instance is no proof."
- Mean, Mode, and Median.

- Interrogating averages:
- Are there extreme scores?
- Are there meaningful subgroups?
- Who's in the denominator?
- What's the variability (standard deviation)?

- What are the odds of that? What's the base rate?

### III. Compared to what?

- What's the implicit or explicit frame of reference?
- What's the unit of measurement?
- Per what?
- What's the order of magnitude?
- Interrogating a graph:
- What's the Y-axis? Is it zero-based?
- Does it K.I.S.S., or is it filled with ChartJunk?

### IV. Is the outcome statistically significant?

- Is the outcome unlikely to have come about by chance?
- "Chance is lumpy."
- Criterion of sufficient rarity due to chance: p < .05

- What does statistical significance mean, and what doesn't it mean?

### V. What's the effect size?

- How can we take the measure of how substantial an outcome is?
- How large is the mean difference? How large is the association?
- Standardized mean difference (d): d = (μ1-μ2)/σ

### VI. Are the results those of a single study or of a literature?

- What's the source of the numbers: PFA, peer-reviewed, or what?
- Who is sponsoring the research?
- How can we take the measure of what a literature shows?
- The importance of meta-analysis in the contemporary world of QR.

### VII. What's the research design (correlational or experimental)?

- Design matters: Experimental vs. correlational design.
- How well does the design support a causal claim?
- Experimental Design:
- Randomized Controlled Trials (RCT): Research trials in which participants are randomly assigned to the conditions of the study.
- Double blind trials: RCTs in which neither the researcher nor the patient know the treatment condition.

- Correlational Design: Measuring existing variation and evaluating co-occurrences, possibly controlling for other variables.
- Interrogating associations (correlations):
- Are there extreme pairs of scores (outliers)?
- Are there meaningful subgroups?
- Is the range of scores in a variable restricted?
- Is the relationship non-linear?

- Interrogating associations (correlations):

### VIII. How was the variable operationalized?

- What meaning and degree of precision does the measurement procedure justify?
- What elements and procedures result in the assignment of a score to a variable?
- What exactly was asked?
- What's the scale of measurement?
- How might we know if the measurement procedure is a good one?
- Reliability = Repeated applications of the procedure result in consistent scores.
- Validity = Evidence supports the use to which the measure is being put.

- Is the measure being manipulated or "gamed"? The iatrogenic effects of measurement.

### IX. Who's in the measurement sample?

- What domain is being evaluated? Who's in? Who's not?
- Is the sample from that domain representative, meaningful, and/or sufficient?
- Is the sample random?
- Are two or more samples that are being compared equivalent?

### X. Controlling for what?

- What other variables might be influencing the findings?
- Were these assessed or otherwise controlled for in the research design?
- What don't we know, and how can we acknowledge uncertainties?