CDIS 564 • Methods Reference

Validity

Whether your study measures what it claims and whether the results mean what you say.

Validity asks whether your conclusions are true. There are four kinds: internal, external, social, and statistical conclusion. In your methods proposal you address internal, external, and social validity. Statistical conclusion validity is addressed later, in the Data Analysis chapter, so it is explained here for background rather than as part of your methods. Expand each panel for what each kind means and how single case design protects it.
Use the Single-Case Design Rigor Planner to plan your fidelity, reliability, and validity decisions for the proposal.
1 The four kinds of validity ▼

Each kind of validity answers a different question about your study. You plan for all four. Internal, external, and social validity are addressed in your methods proposal; statistical conclusion validity is addressed in the Data Analysis chapter.

KindThe question it answers
InternalDid the intervention, and not something else, cause the change in behavior?
ExternalDo the results apply beyond this study, to other people, settings, and behaviors?
SocialDo the goals, the procedures, and the outcomes matter to the people affected?
Statistical conclusionAre the analyses appropriate, so the claim of an effect is sound?
2 Internal validity ▼

Internal validity is the confidence that your intervention caused the change, rather than maturation, practice, or some outside event. Single case design earns internal validity through its structure.

  • The participant is their own control. Each participant is measured under both a baseline condition and an intervention condition, so each person serves as their own comparison.
  • Repeated measurement across phases shows the behavior's pattern before and after the intervention is introduced.
  • A stable baseline establishes a predictable pattern the intervention must visibly break.
  • Replication of the effect, across participants, behaviors, or settings, and staggered in time, rules out the chance that an outside event happened to coincide with intervention.
A simple A-B design (baseline then intervention, one time) allows only correlational conclusions, because there is no replication to rule out coincidence. Multiple baseline and multiple probe designs replicate the A-to-B comparison across tiers, which is what raises confidence that the intervention is responsible.
Example The worked narrative study used a multiple probe design across participants. Intervention started with the next participant only after the first showed three consecutive data points that did not overlap with baseline. Because the change appeared each time intervention began, and only then, the design controls the main threats to internal validity. Fidelity of delivery also protects internal validity, because a change only means something if the intervention was actually delivered as planned.
3 External validity and generality ▼

External validity is the extent to which your findings hold beyond the exact conditions you studied. In single case design, generality is built up through replication and probed directly.

  • Across people: replicating the effect with several participants shows it is not specific to one person.
  • Across behaviors: showing the effect on more than one target behavior shows it is not specific to one skill.
  • Across settings: showing the effect in more than one context shows it is not tied to a single place.
  • Generalization probes: measuring with new materials, people, or settings tests directly whether the change transfers.
Example The worked narrative study measured generalization by having children retell a story from a different book series, using the same procedures as baseline. Transfer to untrained materials is direct evidence of external validity. Replicating the intervention across several participants adds to that generality.
4 Social validity ▼

Social validity is whether the study matters to the people it affects. It has three parts: the goals (are the targeted outcomes worth achieving), the procedures (are the methods acceptable to those involved), and the outcomes (is the change large enough to make a real difference in daily life).

Social validity is gathered from stakeholders, not from the researcher. Name who gives the feedback and the tool you use to collect it.

  • Who gives feedback: participants, families, teachers, clinicians, or other people positioned to judge whether the goals, procedures, and outcomes matter.
  • The tool: most often a rating questionnaire, commonly a Likert-type scale, sometimes with open comments or an interview.
Example In the worked narrative study, six speech-language pathologists watched randomly selected intervention videos and completed a Likert-type questionnaire rating the intervention's feasibility, effectiveness, and acceptability. Independent professionals rating the procedures and outcomes is a concrete way to gather social validity.
5 Statistical conclusion validity ▼

Statistical conclusion validity is whether your analysis actually supports the claim that an effect occurred. Most of it is addressed in the Data Analysis chapter, not in your methods proposal. It is explained here for background.

One part belongs in the methods: planning a randomization test. If you intend to use a randomization test, the random element has to be built into the design itself, for example randomly determining when the intervention begins or the order of conditions. Because that is a design decision, you state it in your methods.

Visual analysis and the effect size statistics below are how you analyze your data, so they belong in the Data Analysis chapter. Do not put visual analysis or effect sizes in your methods.

  • Visual analysis is the primary method. It examines level, trend, variability, overlap between phases, immediacy of effect, and consistency of the pattern across similar phases. It depends on a stable baseline and enough data points, with at least five per phase a common standard.
  • Effect sizes for single case data complement visual analysis. Because single case data rarely meet the assumptions of common parametric tests, nonoverlap and trend measures are used.
MeasureWhat it captures
Tau-UNonoverlap between baseline and intervention plus intervention trend; resistant to outliers and workable with few data points, and can be adjusted for a baseline trend
Percentage of nonoverlapping data (PND)The share of intervention points beyond the most extreme baseline point; simple but can over or underestimate
Improvement rate difference (IRD)The difference in improvement rates between phases
Example The worked narrative study combined visual analysis with descriptive statistics and Tau-U, interpreting Tau-U against published benchmarks for small, moderate, large, and very large effects. Namasivayam and colleagues (2024) reported PND and IRD with stated benchmarks for small, moderate, and large effects. Matching the analysis to single case data, rather than forcing a group-design test, is what protects statistical conclusion validity.
6 What to write in your proposal ▼

In your methods proposal, address internal, external, and social validity. Include each of these:

  • How your design protects internal validity (participant as own control, repeated measurement, stable baseline, replication).
  • How you build and test external validity (replication across people, behaviors, or settings, and generalization probes).
  • How you will gather social validity (who rates the goals, procedures, and outcomes, and with what tool).
  • If you intend to use a randomization test, the random element you are building into the design, for example randomly determining when the intervention begins or the order of conditions.

Visual analysis and effect sizes are how you analyze your data, so they belong in the Data Analysis chapter, not in your methods proposal.

□ Sources ▼