Salary: $104,000 - 124,800 per year
Requirements:
- 4+ years of experience in software QA engineering, SDET, or AI/ML engineering, with a strong foundation in API testing and automation.
- 2+ years of hands-on experience evaluating production GenAI/LLM applications, including RAG-based or Agentic AI systems.
- Strong Python skills for test automation and CI/CD integration.
- In-depth understanding of AI quality, hallucination risk, guardrail evaluation, and adversarial or red-team testing.
- Ability to turn conceptual AI evaluation into practical, measurable, repeatable metrics.
- Experience with or familiarity with Python, AWS Bedrock, Azure OpenAI, Anthropic Claude, LangGraph, LangChain, Semantic Kernel, Promptfoo, Langfuse, and DeepEval.
Responsibilities:
- Lead quality assurance and evaluation strategy for our Agentic AI platform, with a focus on safety, hallucination risk, and PII/PHI protection.
- Develop automated evaluation tools in Python and integrate quality checks into CI/CD pipelines.
- Create practical, repeatable evaluation frameworks and test datasets for non-deterministic AI systems.
- Test the effectiveness of AI guardrails using functional, adversarial, and red-team approaches.
- Own the core software QA foundation for APIs, backend services, and user-facing workflows.
Technologies:
- Agentic AI
- AI
- AI Agents
- API
- AWS
- Azure
- Backend
- CI/CD
- Support
- LLM
- Langfuse
- Python
- RAG
- Cloud
More:
We are seeking a Senior GenAI QA Engineer for a live, production-grade Agentic AI platform in the healthcare domain. This hands-on, high-impact role involves close collaboration with senior AI engineers to support the safety and reliability of our evolving AI agents. The position is a three-month contract with a strong possibility of extension or conversion to hire. It is remote in Canada, with occasional on-site work in downtown Toronto; the listed work arrangement is hybrid remote in Toronto, Ontario. Compensation is $50–$60 per hour, with expected hours capped at 40 per week.
last updated 47 week of 2026
You apply on DevITjobs, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.
