Lead AI Engineer – Agentic Test Automation
Location: Tysons, Virginia, USA
Duration: 4 Months (onsite)
Visa Status: Only GC and US Citizen
Job Description:
Must Have Qualifications: Must have 5+ years of experience and a strong AI development
background. Must have hands on experience building agentic workflows, automating testing using
AI, and working with tools such as GitHub, Copilot, and Claude.
Schedule: Standard
Shortlisting Deadline: July 23rd
Interview Information:
Rounds: 2 rounds
Duration: 30-60 mins
Interview Type: 1
1st round – virtual | 2nd round – onsite
Interview Placeholders: Targeting Jul 28th – Aug 4th
Interview Debrief: TBD – Will be scheduled with suppliers once all interview rounds are completed.
JOB DESCRIPTION
1) Agentic test automation foundation (reusable patterns + reference implementations)
• Design and implement agentic testing patterns that can be adopted by multiple
Underwriting teams (and later other domains).
• Create reference implementations (sample repos / templates) demonstrating:
o Test generation assistance (from requirements, APIs, contracts, schemas)
o Test maintenance assistance (auto-updating selectors/contracts, flaky test triage)
o Failure analysis assistance (root cause suggestions, log correlation, defect drafting)
• Establish a standard architecture for test code organization, tagging, data management,
and execution across UI + API + service layers.
2) Coverage standards, templates, and governance
• Define and publish coverage standards (what “good” looks like) including:
o Minimum coverage expectations by service/component
o Test type mix (unit vs API vs UI vs contract vs integration)
o Risk-based prioritization and traceability to requirements
• Provide templates usable across teams:
o Test plan templates
o Test case/spec templates (Gherkin-style or equivalent)
o Definition of Ready / Definition of Done quality checklists
• Create a scalable tagging/metadata strategy (e.g., feature, service, risk, priority, data
sensitivity) to support reporting and quality gates.
3) GenAI-assisted reporting and quality insights across microservices
• Build automated reporting that aggregates test + service data across multiple
microservices, such as:
o Test execution results (Karate/Playwright + CI runs)
o Service health signals (logs/metrics/traces if available)
o Defect signals (issue tracker metadata if available)
• Generate GenAI-driven summaries:
o Release readiness narratives
o Failure clustering and trend analysis
o “What changed?” insights (commit/PR correlation)
• Produce outputs consumable by engineering leadership and teams (dashboards,
markdown summaries in PRs, artifacts in CI).
4) “Quality gates” via agents
• Build automated review agents that evaluate user stories/requirements for minimum
required clarity and data before development/testing starts:
o Required fields present (acceptance criteria, testable outcomes, data needs,
dependencies)
o Ambiguity detection and missing edge cases
o Data/privacy considerations and environment needs
• Integrate gates into workflow (PR checks, issue templates, GitHub Actions) to reduce churn
and rework.
Required Technical Skills (must-have)
GenAI / LLM + agentic development
• Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning,
guardrails).
• Experience with prompting patterns, structured outputs (JSON schemas), evaluation, and
reducing hallucinations.
• Ability to design agent workflows for:
o Test generation/augmentation
o Requirements review and completeness validation
o Report generation and summarization
GitHub platform + GHCP (Copilot) for engineering workflows
• Strong proficiency with GitHub Copilot in day-to-day development.
• Deep experience with GitHub platform capabilities:
o GitHub Actions (CI/CD pipelines, reusable workflows, composite actions)
o PR checks, branch protections, CODEOWNERS, templates
o Automation via GitHub APIs/webhooks (as needed)
Test automation engineering (framework expertise)
• Advanced experience designing and implementing automation with:
o Karate (API testing, contract-like checks, data-driven testing, mocks)
o Playwright (UI automation, selectors strategy, parallelization, trace/video
artifacts)
• Strong understanding of test design and coverage:
o Happy path scenarios
o Negative/validation scenarios
o Edge/boundary scenarios
o Data setup/teardown strategies and test isolation
Cross-service reporting and data aggregation
• Proven ability to aggregate and normalize results from multiple microservices and multiple
pipelines.
• Experience producing actionable automated reports (trend analysis, failure clustering,
service correlation).
Automated requirements review agents
• Experience implementing automated checks that validate:
o Acceptance criteria completeness
o Required test data and environment dependencies
o Non-functional requirements (performance, security, observability) when
applicable
Deliverables / What success looks like (for the posting)
• A reusable agentic testing automation kit adopted by multiple teams.
• Published coverage standards + templates and onboarding documentation.
• A working GenAI-assisted reporting pipeline aggregating results across microservices.
• Automated quality gates integrated into GitHub workflows that measurably reduce story
churn.