How to Assess QA Engineers: Building a Test Automation Hiring Process

Why QA hiring is broken

Most companies assess QA engineers like programmers with a QA badge. They ask algorithmic questions, run coding tests, and measure syntax speed. But QA engineering isn't programming with different tools. It's a different discipline: it's about risk, trade-offs, and judgment under constraint.

A candidate who's fast at Selenium might be terrible at prioritizing what to test. Someone who can't code Cypress scripts might excel at finding critical bugs through manual exploration. The assessment needs to match the job.

The three layers of QA assessment

Layer 1: Test case design (written, 45 minutes)

Give candidates a realistic feature spec and ask them to write 12–15 test cases they'd propose to a team. No code. Just cases.

What you're grading:

• Coverage: Do they identify happy path, error cases, edge cases, and state transitions?
• Clarity: Can someone else read and execute the test case without asking questions?
• Prioritization: Do they mark critical vs. nice-to-have, or do they treat all cases as equally important?
• Context awareness: Do they ask questions about the environment (production data, browser support, performance baseline)?

A strong submission includes: preconditions, test steps, expected outcomes, and why each case matters. A weak submission is either too granular (50 one-line cases) or too vague ("test the login").

This is the foundation. If they can't design a test case, automation frameworks don't matter.

Layer 2: Automation portfolio (take-home, 2–3 hours)

Send them a small, open-source project or a sandbox app. Ask them to write 4–6 automated test cases using their framework of choice.

What you're grading:

• Framework fluency: Can they write valid, runnable code in Selenium, Cypress, Playwright, or whatever you use?
• Test structure: Are tests independent, readable, and maintainable? Or are they coupled, hard-coded, and fragile?
• Robustness: Do they use waits, error handling, and selectors that won't break on minor UI changes?
• Documentation: Can they explain why they chose their approach?

Real talk: The technical quality of the automation matters less than the thinking behind it. A candidate who writes 3 solid test cases with good assertions is stronger than someone who writes 8 fragile ones that'll fail on every deploy.

The key question in follow-up: "If this test broke tomorrow, how would you debug it?" Their answer reveals whether they understand the code or just copy-pasted it.

Layer 3: Process & judgment (live interview, 60 minutes)

Combine this with a live round. Present a real scenario: "We have a 2-hour test suite. We want to cut it to 30 minutes. What do you do?" or "A feature launches in 2 weeks. Testing isn't done. What's your play?"

This is where senior engineers' interview loops apply to QA: you're not testing knowledge, you're testing judgment.

What you're measuring:

• Do they think strategically (risk, ROI, scope) or tactically (speed, coverage)?
• Can they push back on unrealistic timelines without being confrontational?
• Do they understand the business context (startup vs. regulated, user-facing vs. internal)?
• Can they articulate trade-offs without sounding wishy-washy?

Ask follow-ups. "Tell me about a time you had to choose between testing and shipping. What was the call?" Listen for honesty and reasoning, not heroics.

Optional layer: Code quality assessment

If your QA role includes reviewing test code, add a code review exercise. Give them a test suite with issues: flaky selectors, missing error handling, poor naming, over-assertion. Ask them to identify problems and suggest fixes.

This filters for people who can maintain code, not just write it.

How to assess at scale

For high-volume hiring, use assessment platforms that can:

• Grade test case designs on coverage, clarity, and completeness (not perfect—just good signal)
• Run submitted automation code in a sandbox and report on pass/fail and code quality metrics
• Record live coding interviews for async review if live rounds are too expensive

The goal isn't perfect objectivity. It's consistency and speed. A rubric that measures coverage, clarity, and trade-off articulation beats arbitrary gut feel every time.

Red flags during assessment

• Over-automation: "We should automate everything, especially the UI." — Suggests they don't understand ROI or maintenance cost.
• Under-automation: "Manual testing is more reliable." — Might not have hands-on experience with modern frameworks.
• Framework absolutism: "The only right tool is [Tool]." — Usually means limited exposure or stubbornness.
• Missing context: Test cases that don't consider data setup, environment, or dependencies. — Suggests they've only tested simple flows.
• No prioritization: All tests weighted equally, no mention of risk or criticality. — Red flag for judgment.

None of these are disqualifying alone. But if you see two or three together, the hire probably needs ramp-up investment.

When to hire test automation engineers vs. manual QA

This assessment approach works for both, but the weight shifts:

Test automation engineers (80% code, 20% strategy): Heavier on framework fluency and code structure, lighter on test case design.

Manual QA / exploratory testers (20% code, 80% strategy): Heavier on test case design, edge-case thinking, and judgment. Code fluency matters less.

The process interview applies to both. The trade-off thinking is what separates good hires from burnout hires in this discipline.

Building buy-in for QA assessment

If your engineering team has never run a structured QA interview, they'll push back: "We should just hire someone who's good at finding bugs." But "finding bugs" is outcome, not skill. You can't measure outcome until they're hired.

The three-layer assessment predicts who'll be good at finding bugs by measuring: how they think about coverage, how they code under constraint, and how they make judgment calls when something has to give.

Start with layer 1 (test case design). It's the fastest to grade and the most universally revealing. Add layers as you refine your process.