About this role
Anaplan is hiring a QA Engineer focused specifically on quality assurance for GenAI systems.
The role will develop testing strategies, evaluation frameworks and quality metrics for LLM-powered applications, including conversational AI, agentic systems and GenAI workflows.
The QA Engineer will build automated tests, test datasets and monitoring systems to evaluate accuracy, relevance, safety, consistency and other quality dimensions of AI features.
Requirements
- Design and implement testing strategies for GenAI features.
- Test conversational AI, agentic systems and LLM-powered workflows.
- Develop automated prompt-testing and regression-test suites.
- Create evaluation frameworks for GenAI quality.
- Build and maintain test datasets and golden examples.
- Monitor AI quality and detect production degradation.
- Perform adversarial testing for hallucinations, biases and security vulnerabilities.
- Define acceptance criteria and quality gates with engineering teams.
- Build tools and frameworks for GenAI testing.
- Conduct user acceptance testing.
- Document testing procedures, known issues and quality metrics.
- Work with Product and Design teams to ensure AI features meet quality and UX standards.
Required:
- 3+ years of QA or test engineering experience.
- Strong understanding of GenAI, LLMs and prompt engineering.
- Experience with Python, JavaScript, Selenium or PyTest.
- Knowledge of functional, integration, regression, performance and security testing.
- API testing experience with Postman or REST Assured.
- Familiarity with CI/CD and automated testing.
Preferred:
- Conversational AI, chatbot or agentic-system testing.
- ML model evaluation.
- LangSmith, PromptFoo or Ragas.
- AI API performance/load testing.
- Responsible AI and security testing.
- Test management tools such as TestRail, Zephyr or Jira.