Phase 3 Exam Drill: Sampling, Data Collection & Analysis
90 Exam-Style Questions covering Modules 5 & 6 (Sampling techniques, Practice Q5 & Q7, statistical tests ANOVA/t-test, reliability vs validity, Cronbach's alpha, and data retention rules). Click "View Model Answer" to reveal the exact marking criteria.
Part 1: Foundational Questions (Questions 1 - 30)
P3-Q01Easy
Define Sampling in research.
NUST Model Answer [3 Marks]: Selecting a subset of individuals or data points from a population to estimate characteristics of the whole population.
P3-Q02Easy
State two reasons why researchers use sampling instead of testing an entire population.
NUST Model Answer [3 Marks]: 1. Saves time and financial costs, 2. Reduces hardware/network resource strain.
P3-Q03Easy
Define Simple Random Sampling.
NUST Model Answer [3 Marks]: A probability sampling technique where every element in the population has an equal and known chance of selection.
P3-Q04Easy
Define Stratified Sampling.
NUST Model Answer [3 Marks]: Dividing a population into distinct sub-groups (strata) and randomly sampling proportionally from each stratum.
P3-Q05Easy
Define Convenience Sampling.
NUST Model Answer [3 Marks]: Selecting readily accessible subjects or data without random selection (high risk of sampling bias).
P3-Q06Easy
Define Purposive / Judgmental Sampling.
NUST Model Answer [3 Marks]: Selecting subjects based on explicit expert criteria relevant to the study (e.g. senior security engineers).
P3-Q07Easy
Define Snowball Sampling.
NUST Model Answer [3 Marks]: Existing research subjects refer additional subjects who meet the target criteria.
P3-Q08Easy
What is Descriptive Statistics?
NUST Model Answer [3 Marks]: Summarizing and describing features of a dataset using mean, median, mode, variance, and standard deviation.
P3-Q09Easy
What is Inferential Statistics?
NUST Model Answer [3 Marks]: Drawing conclusions and making predictions about a population based on sample data analysis.
P3-Q10Easy
What is a Student's t-test used for in CS research?
NUST Model Answer [3 Marks]: Comparing the means of two experimental groups (e.g. openEuler latency vs Ubuntu baseline latency).
P3-Q11Easy
What is ANOVA (Analysis of Variance) used for?
Model Answer [3 Marks]: Comparing the means of 3 or more independent experimental groups simultaneously.
P3-Q12Easy
Define Reliability in research.
Model Answer [3 Marks]: The consistency and reproducibility of test results under identical conditions.
P3-Q13Easy
Define Validity in research.
Model Answer [3 Marks]: The accuracy of an instrument in measuring what it claims to measure.
P3-Q14Easy
What is Cronbach's Alpha ($\alpha$)?
Model Answer [3 Marks]: A statistical metric assessing internal consistency reliability of survey scales (acceptable threshold $\ge 0.70$).
P3-Q15Easy
State the mandatory raw data retention period under research integrity rules.
Model Answer [3 Marks]: 5 years after publication.
P3-Q16Easy
What is Thematic Analysis?
Model Answer [3 Marks]: A qualitative method for identifying, analyzing, and reporting patterns (themes) within textual data.
P3-Q17Easy
Define Sampling Frame.
Model Answer [3 Marks]: The complete list of population elements from which a sample is drawn.
P3-Q18Easy
What is Sampling Bias?
Model Answer [3 Marks]: Systematic error introduced when certain population members are more likely to be selected than others.
P3-Q19Easy
Define Standard Deviation.
Model Answer [3 Marks]: A metric measuring the amount of variation or dispersion of data points around the mean.
P3-Q20Easy
What is p-value in hypothesis testing?
Model Answer [3 Marks]: The probability that an observed experimental result occurred by random chance (significant if $p < 0.05$).
P3-Q21Easy
What is Type I Error?
Model Answer [3 Marks]: False Positive—rejecting a true null hypothesis.
P3-Q22Easy
What is Type II Error?
Model Answer [3 Marks]: False Negative—failing to reject a false null hypothesis.
P3-Q23Easy
Define Grounded Theory in qualitative analysis.
Model Answer [3 Marks]: Iteratively analyzing qualitative data to derive a theory grounded in empirical observations.
P3-Q24Easy
What is a 99th percentile ($p_{99}$) latency metric?
Model Answer [3 Marks]: The latency threshold below which 99% of all benchmark requests fall (captures worst-case tail latency).
P3-Q25Easy
State one threat to internal consistency.
Model Answer [3 Marks]: Ambiguous or contradictory survey questions.
P3-Q26Easy
What is Cluster Sampling?
Model Answer [3 Marks]: Dividing population into geographic/logical clusters and sampling entire clusters randomly.
P3-Q27Easy
Define Systemic Error.
Model Answer [3 Marks]: Predictable, non-random error introduced by miscalibrated hardware instruments or biased software.
P3-Q28Easy
Define Random Error.
Model Answer [3 Marks]: Unpredictable variations caused by environmental noise during measurement.
P3-Q29Easy
Why must raw survey data be anonymized before storage?
Model Answer [3 Marks]: To protect participant privacy and comply with ethics clearance guidelines.
P3-Q30Easy
What is a Likert Scale?
Model Answer [3 Marks]: A psychometric scale (e.g. 1-5 rating) used to quantify attitudes or satisfaction levels.
Part 2: Medium / Applied Questions (Questions 31 - 60)
P3-Q31Medium
PRACTICE Q7 DRILL: You want to survey student developers across 4 academic years (1st, 2nd, 3rd, 4th) on AI tool usage. Which probability sampling method should you choose and why? [8 Marks]
NUST Model Answer [8 Marks]: Method: Stratified Random Sampling. Justification: Ensures proportional representation across each academic year stratum (e.g. 25% from 1st yr, 25% from 2nd yr, etc.), preventing over-representation of a single class year and eliminating selection bias.
P3-Q32Medium
When should a researcher use an ANOVA test instead of multiple t-tests?
Model Answer [4 Marks]: When comparing 3 or more groups. Running multiple pairwise t-tests inflates Type I Error probability ($\alpha$ accumulation).
P3-Q33Medium
Explain why convenience sampling is acceptable in early prototype usability testing but invalid for statistical generalizability.
Model Answer [4 Marks]: Acceptable to catch quick UI bugs from available peers; invalid for generalizability due to un-controlled sampling bias.
P3-Q34Medium
How does CPU thermal throttling invalidate benchmark reliability?
Model Answer [4 Marks]: Introduces systemic clock speed drift across iterations, destroying test consistency under identical conditions.
P3-Q35Medium
Why is Cronbach's Alpha ($\alpha$) calculated on Likert scale data?
Model Answer [4 Marks]: To verify that all Likert items in a survey construct measure the underlying attitude consistently.
P3-Q36Medium
Distinguish between Purposive Sampling and Convenience Sampling.
Model Answer [4 Marks]: Purposive: Selected based on deliberate expert criteria. Convenience: Selected purely based on physical/logical availability.
P3-Q37Medium
Scenario: Testing a dark-web ethical hacker tool. Which sampling method fits best?
Model Answer [4 Marks]: Snowball Sampling—because the target population is hidden and hard to reach directly.
P3-Q38Medium
What is the consequence of violating the 5-year data retention rule?
Model Answer [4 Marks]: Inability to defend against allegations of data fabrication or audit inquiries, risking paper retraction.
P3-Q39Medium
Why is Mean insufficient without Standard Deviation when reporting execution latency?
Model Answer [4 Marks]: Mean masks extreme latency spikes (tail latency) that standard deviation and variance expose.
P3-Q40Medium
Explain how warm-up iterations protect test reliability.
Model Answer [4 Marks]: Populates hardware caches so measured iterations reflect steady-state execution rather than initial page faults.
P3-Q41Medium
Contrast Content Validity and Construct Validity.
Model Answer [4 Marks]: Content: Whether instrument covers all aspects of topic. Construct: Whether instrument correctly measures the theoretical construct.
P3-Q42Medium
What is the purpose of post-hoc tests (e.g. Tukey's HSD) after ANOVA?
Model Answer [4 Marks]: To identify exactly *which* specific group pairs differ significantly after ANOVA detects an overall difference.
P3-Q43Medium
Scenario: Sampling 1,000 network packets every 10th packet. What sampling technique is this?
Model Answer [3 Marks]: Systematic Probability Sampling (selecting every $k$-th element).
P3-Q44Medium
Why must qualitative interview scripts undergo thematic coding?
Model Answer [4 Marks]: To reduce raw transcript text into structured, evidence-backed themes.
P3-Q45Medium
What is the significance level $\alpha = 0.05$?
Model Answer [3 Marks]: 5% maximum acceptable risk of rejecting a true null hypothesis (Type I error).
P3-Q46Medium
Scenario: Testing openEuler driver latency with payload sizes 64B, 1KB, 64KB. Identify the Independent Variable.
Model Answer [3 Marks]: Payload Size (64B, 1KB, 64KB).
P3-Q47Medium
How does data anonymization prevent evaluation bias in user studies?
Model Answer [3 Marks]: Prevents researcher preconceptions regarding specific participants from affecting qualitative coding.
P3-Q48Medium
Explain the role of raw CSV/JSON logging in experiment validation.
Model Answer [3 Marks]: Provides un-altered empirical data logs for auditing and statistical scripting.
P3-Q49Medium
Why are high variance benchmarks undesirable?
Model Answer [3 Marks]: Indicates unstable execution conditions or un-controlled background process noise.
P3-Q50Medium
Distinguish between Parametric and Non-Parametric statistical tests.
Model Answer [4 Marks]: Parametric: Assumes normally distributed data (e.g. t-test, ANOVA). Non-Parametric: Used when data distribution is skewed (e.g. Mann-Whitney U).
P3-Q51Medium
State one cause of Type II errors in CS benchmarking.
Scenario: Surveying 10 close friends in class. Is this probability sampling?
Model Answer [3 Marks]: NO! It is Non-Probability Convenience Sampling.
P3-Q56Medium
What is the function of a codebook in qualitative analysis?
Model Answer [3 Marks]: Listing themes, definitions, and inclusion/exclusion rules for coding transcripts.
P3-Q57Medium
Why is $p < 0.05$ the standard threshold for statistical significance?
Model Answer [3 Marks]: Ensures less than 5% probability that observed differences were due to random chance.
P3-Q58Medium
How does random assignment reduce confounding in human-subject CS studies?
Model Answer [3 Marks]: Evenly distributes participant background differences across control and test groups.
P3-Q59Medium
What is Median and when is it preferred over Mean?
Model Answer [3 Marks]: Middle value of ordered dataset; preferred when data contains extreme latency outliers.
P3-Q60Medium
Explain the risk of selective data deletion in performance benchmarks.
Model Answer [3 Marks]: Deletes real worst-case performance data, constituting unethical data falsification.
Part 3: Hard / Defense & Synthesis Questions (Questions 61 - 80)
P3-Q61Hard
PRACTICE Q5 DEFENSE DRILL: Compare ANOVA and t-tests in a 3-way driver benchmark study (openEuler vs Ubuntu vs Debian). Why is running 3 separate t-tests statistically invalid? [15 Marks]
NUST Model Answer [15 Marks]: 1. Statistical Accumulation of Type I Error: If $\alpha = 0.05$ per test, running 3 separate t-tests increases total family-wise error rate to $1 - (1 - 0.05)^3 \approx 14.3\%$, inflating false positive risk. 2. ANOVA Superiority: One-Way ANOVA tests all 3 groups simultaneously under a single omnibus $F$-test at strict $\alpha = 0.05$. 3. Post-Hoc Verification: If ANOVA indicates significance, Tukey's HSD post-hoc test must be run to pinpoint pairwise differences safely.
P3-Q62Hard
Defend the requirement for a 5-year raw data retention policy against institutional auditors.
Model Answer [10 Marks]: Ensures full post-publication transparency, allows verification of claim validity, and protects authors against false allegations of data fabrication.
P3-Q63Hard
Evaluate how Cronbach's Alpha ($\alpha = 0.45$) invalidates survey conclusions in Chapter 5.
Model Answer [5 Marks]: $\alpha = 0.45$ falls far below acceptable 0.70 threshold, proving survey items are internally inconsistent and measurement results unreliable.
P3-Q64Hard
Contrast Probability and Non-Probability sampling in terms of external validity.
Model Answer [5 Marks]: Probability sampling provides high external validity through random selection; Non-probability limits external validity to the specific sample context.
P3-Q65Hard
Explain how dynamic frequency scaling in CPUs creates systemic measurement error.
Model Answer [5 Marks]: CPU scaling changes core clock speeds dynamically based on heat, introducing non-random measurement drift across benchmark iterations.
P3-Q66Hard
Critique a study that reports only mean latency without standard deviation or $p_{99}$.
Model Answer [5 Marks]: Flawed report: Mean hides extreme latency spikes and tail latency distributions critical for real-world system evaluation.
P3-Q67Hard
Synthesize how Stratified Sampling eliminates subgroup under-representation.
Model Answer [4 Marks]: Guarantees fixed, proportional sample allocations for every subgroup stratum before random selection.
P3-Q68Hard
Why is Purposive Sampling preferred for expert security architecture reviews?
Model Answer [4 Marks]: Random selection risks picking novices; purposive sampling deliberately targets domain experts with deep architectural insight.
P3-Q69Hard
Explain how a researcher proves internal consistency of qualitative thematic analysis.
Model Answer [5 Marks]: By establishing high inter-rater reliability ($\ge 80\%$ agreement) between independent coders using a standardized codebook.
P3-Q70Hard
Contrast Type I and Type II errors in automated biometric entry systems.
Model Answer [5 Marks]: Type I (False Acceptance): Impersonator granted entry (security breach). Type II (False Rejection): Valid student denied entry (system friction).
P3-Q71Hard
Why must benchmark runs be repeated at least 30-100 times?
Model Answer [4 Marks]: To satisfy Central Limit Theorem conditions, enabling normal distribution assumptions for parametric statistical tests.
P3-Q72Hard
Evaluate the risk of self-selection bias in online CS student surveys.
Model Answer [4 Marks]: Only highly motivated or highly frustrated students respond, skewing survey ratings away from true population mean.
P3-Q73Hard
Synthesize how reliability is a necessary but insufficient condition for validity.
Model Answer [5 Marks]: An instrument can be consistently wrong (reliable but invalid, e.g. miscalibrated clock); but an unreliable instrument can never be valid.
P3-Q74Hard
Explain the role of confidence intervals (95% CI) in reporting system throughput.
Model Answer [4 Marks]: Provides a range within which the true population throughput mean is 95% certain to lie.
P3-Q75Hard
Why is Snowball Sampling prone to community homophily bias?
Model Answer [4 Marks]: Participants tend to refer peers with identical attitudes and backgrounds, excluding diverse perspectives.
P3-Q76Hard
Critique a benchmark study that compares C++ on Linux with Java on Windows.
Model Answer [5 Marks]: Severely flawed: Confounders (OS architecture, JVM vs native runtime) are un-isolated, rendering language comparison invalid.
P3-Q77Hard
Explain how Tukey's HSD post-hoc test controls family-wise error rates.
Model Answer [4 Marks]: Adjusts critical mean difference thresholds to keep total Type I error risk at $\alpha = 0.05$ across all pairwise comparisons.
P3-Q78Hard
Differentiate between Construct Validity and Face Validity.
Model Answer [4 Marks]: Construct: Statistical proof instrument measures theoretical model. Face: Superficial appearance that instrument measures target metric.
P3-Q79Hard
Why must qualitative transcripts be coded before writing Chapter 5 results?
Model Answer [4 Marks]: Un-coded text lacks analytical structure; coding distills raw text into clear, thematic evidence for presentation.
P3-Q80Hard
Evaluate the impact of deleting 5 outlier benchmark runs without documenting reasons.
Model Answer [4 Marks]: Unethical data manipulation; outliers must be retained or explicitly justified under statistical outlier rules (e.g. 3 $\sigma$).
TRICK: Is a sample size of $N=30$ always sufficient for all CS benchmark experiments? Explain the examiner trap.
NUST Model Answer [5 Marks]: NO! Examiner Trap: While $N=30$ is a rule of thumb for Central Limit Theorem in basic statistics, high-variance systems (e.g. network latency or distributed cloud daemons) require thousands of runs ($N \ge 1,000$) to accurately capture tail latency percentiles ($p_{99}$).
P3-Q82EXAMINER TRICK
TRICK: Can a survey with Cronbach's Alpha $\alpha = 0.98$ be flawed due to redundancy?
Model Answer [5 Marks]: YES! Extremely high $\alpha > 0.95$ indicates item redundancy (asking the exact same question in slightly different words), shortening overall construct breadth.
P3-Q83EXAMINER TRICK
TRICK: Does Non-Probability Convenience Sampling ever yield statistically valid $p$-values?
Model Answer [4 Marks]: NO! Inferential $p$-value statistics assume random probability sampling. Applying them to convenience samples produces invalid statistical conclusions.
P3-Q84EXAMINER TRICK
TRICK: Is it ethical to delete raw benchmark logs after your paper is accepted for publication?
Model Answer [4 Marks]: NO! Data retention rules mandate preserving raw data logs for at least 5 years post-publication for audit inquiries.
P3-Q85EXAMINER TRICK
TRICK: Does high reliability guarantee high validity in benchmark logging?
Model Answer [5 Marks]: NO! A miscalibrated timer script can measure latency consistently wrong every single run (100% reliable, 0% valid).
P3-Q86EXAMINER TRICK
TRICK: If $p = 0.06$, can the researcher claim the hypothesis was "almost true"?
Model Answer [4 Marks]: NO! At threshold $\alpha = 0.05$, $p = 0.06$ is statistically non-significant. The null hypothesis cannot be rejected.
P3-Q87EXAMINER TRICK
TRICK: Can Stratified Sampling be conducted without knowing total population size?
Model Answer [4 Marks]: NO! Stratified sampling requires a complete sampling frame with known subgroup counts to calculate proportional stratum samples.
P3-Q88EXAMINER TRICK
TRICK: What is the examiner trap in reporting only the best 3 benchmark runs out of 50?
Model Answer [5 Marks]: Cherry-picking data constitutes unethical Data Falsification. All 50 runs must be included in mean and variance calculations.
P3-Q89EXAMINER TRICK
TRICK: Is Qualitative Thematic Analysis purely subjective with no validity rules?
Model Answer [4 Marks]: NO! Qualitative validity requires inter-rater reliability checks, codebooks, member checking, and audit trails.
P3-Q90EXAMINER TRICK
TRICK: What is the single biggest statistical mistake CS students make in Chapter 5?
Model Answer [5 Marks]: Claiming one system is "faster" based on raw sample means without performing statistical significance testing (t-test or ANOVA with $p$-values).