Complete study guide for Course Unit 4. Master Probability vs Non-Probability sampling methods, controlled benchmarking, Mining Software Repositories (MSR), surveys, interviews, and observation techniques.
Sampling is the selection of a subset of individuals, packet traces, or code commits from a target population to estimate overall population characteristics.
| Method | Mechanism | CS Benchmark / Study Example |
|---|---|---|
| Simple Random Sampling | Every item in the population has an equal non-zero chance of selection. | Selecting 500 network traffic logs randomly from a dataset of 1,000,000 packet traces using a PRNG seed. |
| Stratified Sampling | Population is divided into mutually exclusive strata (subgroups); samples are randomly drawn from each stratum proportionally. | Selecting open-source repos stratified by programming language (30% Python, 30% C++, 40% Rust) to evaluate vulnerability rates. |
| Systematic Sampling | Selecting every $k$-th element after a random starting point. | Sampling every 50th commit log from a Git repository history. |
| Method | Mechanism | CS Study Application & Risk |
|---|---|---|
| Purposive / Judgmental | Researcher handpicks subjects based on specific expertise criteria. | Selecting 10 senior Linux kernel maintainers to interview about lock contention bugs. High depth, non-representative. |
| Convenience Sampling | Selecting easily accessible participants or datasets. | Surveying fellow classmates in a CS lab. ⚠️ High risk of selection bias; findings cannot be generalized broadly. |
| Snowball Sampling | Existing subjects recruit future subjects from among their acquaintances. | Studying anonymous dark web malware developers or specialized hacker groups. |
In quantitative CS research, benchmarking measures hardware or software performance metrics under controlled test conditions.
Pin CPU core affinity (e.g., taskset -c 0), disable thermal throttling, terminate background cron jobs, and fix ambient room temperature.
Run minimum 30 execution trials ($N \ge 30$) to allow Central Limit Theorem application and calculate standard deviation and confidence intervals.
Compare against standard industry reference implementations (e.g., Snort IDS, Redis, GCC -O3 compiler outputs).
Always distinguish between latency (time per operation, e.g. milliseconds) and throughput (operations per unit time, e.g. req/sec). A system can achieve high throughput while suffering poor P99 tail latency.
Extracting empirical evidence from version control systems (GitHub, GitLab), issue trackers (Jira, Bugzilla), and mailing lists. Uses AST parsers (e.g., Tree-sitter) to analyze commit diffs and code churn.
Structured quantitative collection using Likert scales (1=Strongly Disagree to 5=Strongly Agree). Requires pilot testing on 5-10 users before full deployment.
Qualitative data collection. Structured (rigid protocol), Semi-structured (flexible probe questions), or Focus Groups (6-8 participants discussing UX or system workflows).