Job description
This role is for one of the Weekday's clients Salary range: Rs 2500000 - Rs 4000000 (ie INR 25 - 40 LPA) Min Experience: 5+ years Location: Bengaluru JobType: full-time We are looking for an innovative Agentic QA Engineer to define and build the next generation of quality assurance practices for AI-native platforms.
This is far more than a traditional QA role—you will combine software testing, automation engineering, and AI system validation to ensure platform reliability, scalability, and intelligent decision-making.
You'll be responsible for validating the platform's behavior, including its AI-driven orchestration, routing, and multi-agent workflows.
Rather than evaluating customer-built AI agents, your focus will be on ensuring the platform consistently makes reliable, secure, and efficient decisions under real-world conditions.
Key Responsibilities Platform Functional Testing Design and execute end-to-end functional tests covering agent onboarding, deployment, orchestration, and request routing.
Validate platform behavior across multiple supported agent frameworks and deployment scenarios.
Test failure handling, recovery mechanisms, and edge cases throughout the platform lifecycle.
QA Automation & Reliability Build and maintain automated regression test suites integrated into CI/CD pipelines.
Develop automated quality gates that prevent deployments when reliability metrics fall below acceptable thresholds.
Perform resilience testing by intentionally introducing failures such as service outages, database interruptions, network failures, and high traffic loads.
Define and monitor Service Level Objectives (SLOs) for platform reliability and availability.
Agentic Testing Develop testing methodologies for AI-driven platform behavior where deterministic outputs are not possible.
Validate intelligent routing decisions across multiple AI agents.
Test multi-agent orchestration to ensure correct information flow, execution limits, cost controls, and successful task completion.
Verify platform safeguards against infinite agent loops, resource overconsumption, security risks, and cross-tenant data leakage.
Build and maintain a suite of deterministic test agents with predictable behaviors to evaluate platform decision-making accurately.
Quality Engineering Create adversarial ("red team") testing strategies to uncover weaknesses in AI workflows and orchestration logic.
Convert execution traces into measurable platform health metrics such as routing accuracy, task success rate, execution cost, failure rates, and stuck-loop detection.
Investigate failure patterns and develop clear troubleshooting guides and operational playbooks.
Own quality assurance processes for platform releases and continuously improve testing strategies as the platform evolves.
Contribute toward developing platform testing capabilities that customers can leverage for validating their own AI agents.
Required Qualifications Strong experience in QA Engineering, SDET, Software Testing, or Site Reliability Engineering.
Proven expertise in building automated testing frameworks and integrating them into CI/CD pipelines.
Hands-on experience with test automation using Python or similar scripting languages.
Strong understanding of software quality principles, distributed systems, and cloud-native architectures.
Familiarity with AI/ML concepts and an understanding of how AI-powered systems behave in production environments.
Excellent analytical and problem-solving skills with an adversarial mindset for identifying edge cases and failure scenarios.
Strong communication skills with the ability to explain technical quality issues to engineering, product, and leadership teams.
Preferred Qualifications Experience working with Large Language Models (LLMs), AI agents, or autonomous AI systems.
Familiarity with agent frameworks such as LangGraph, CrewAI, AutoGen, or similar orchestration platforms.
Knowledge of distributed systems reliability and control plane architectures.
Experience with observability platforms such as OpenTelemetry, Grafana, Datadog, or similar monitoring tools.
Contributions to open-source QA, AI, or developer tooling projects.
Why Join Us?
Help define an entirely new discipline at the intersection of Quality Engineering, AI Systems, and Platform Reliability.
Build innovative testing methodologies for intelligent, AI-native platforms.
Work in a highly collaborative environment with significant ownership and technical autonomy.
Make a direct impact on the quality, scalability, and reliability of next-generation AI products.
Enjoy a flexible, remote-friendly work environment with competitive compensation and growth opportunities.
Must-Have Skills QA Automation Agentic Testing Test Automation Python Good-to-Have Skills Large Language Models (LLMs) LangGraph CrewAI AutoGen Site Reliability Engineering (SRE) Distributed Systems OpenTelemetry Grafana Datadog
This job post has been translated by AI and may contain minor differences or errors.
