Complete study guide for Course Unit 9. Master the difference between Reliability and Validity, measuring internal consistency (Cronbach's Alpha), classifying threats to validity, and enforcing 5-year data retention protocols.
A study can be reliable without being valid, but it CANNOT be valid without being reliable.
The degree to which a measurement instrument yields consistent, reproducible results under unchanged experimental conditions.
CS Test: If you run a microbenchmark algorithm 100 times on identical hardware, do you get consistent execution timings?
The extent to which a test or metric actually measures what it claims to measure.
CS Test: Does measuring CPU cycles accurately represent overall user-perceived web application responsiveness (or does network RTT dominate)?
In survey research and Likert-scale questionnaire validation, internal consistency is evaluated using Cronbach's Alpha ($\alpha$):
| Type of Validity | Core Focus | Threat Example in CS Research | Mitigation Strategy |
|---|---|---|---|
| Internal Validity | Ensuring observed changes in dependent variable are strictly caused by independent variable. | OS background update or CPU thermal throttling skewing execution time during benchmark. | Isolate environment (fixed CPU affinity, disable cron/throttling, run 30+ trials). |
| External Validity | Extent to which results can be generalized to real-world production settings. | Evaluating a database engine only on tiny synthetic datasets in RAM, failing under disk swap. | Evaluate against industry standard benchmarks (e.g. TPC-C, YCSB workloads). |
| Construct Validity | Ensuring metrics properly represent theoretical concepts. | Using lines of code (LOC) as a metric for programmer productivity or software complexity. | Use validated metrics like Cyclomatic Complexity or Maintainability Index. |
| Conclusion Validity | Correctness of statistical inferences made from data. | Using a t-test on small, highly skewed sample data ($N=5$) with high variance. | Perform normality testing (Shapiro-Wilk) and select non-parametric tests if skewed. |
Scientific integrity requires that all underlying data, code, benchmark scripts, and configuration files be preserved for independent audit.
All raw experimental datasets, anonymized survey responses, script code, and container definitions (Dockerfiles) MUST be retained securely for a minimum of 5 years post-graduation to support post-publication audit and replication inquiries.