Lead Software Engineer In Test – Agentic Ai (remote - Us)

About the Role

Lead quality engineering for agentic AI products, designing scalable automation frameworks and validating complex LLM workflows, RAG pipelines, and distributed orchestration systems across UI, API, backend, and AI layers. Drive testing strategy, CI/CD integration, and mentor QA engineers to ensure resilient, performant AI-driven applications.

Focuses on testing and validating agentic AI, LLM workflows and RAG pipelines—so it's about building and ensuring AI-driven systems work; leans into rapid AI iteration and AI dev tooling.

Location: United States

Type: Full Time

Posted: 1 days ago

Category: Software Engineering

Vibe Markers: 💻 Open Source, 🔥 Hot

Experience Level: Senior

Job Description

Role

The Lead Software Engineer in Test – Agentic AI will lead quality engineering strategy and execution for AI-powered applications and distributed systems. The role focuses on designing scalable automation frameworks, validating agentic AI behaviors, and ensuring high-quality delivery across UI, API, backend, orchestration, and cloud-native layers.

Key Responsibilities

• Design, develop, and maintain scalable automation frameworks and regression test suites for UI, API, backend, and AI-driven applications.
• Lead testing initiatives for Agentic AI systems including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
• Validate complex agent behaviors such as multi-agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
• Develop and execute functional, integration, regression, end-to-end, and performance testing strategies in cloud-native and distributed environments.
• Collaborate with Software Engineers, Product Managers, and DevOps to integrate testing throughout the SDLC and CI/CD pipelines.
• Build reusable automation solutions for APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI-driven business processes.
• Design and maintain test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
• Establish best practices for code quality, maintainability, scalability, dependency injection, observability, and testability.
• Lead validation of AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge-case conditions.
• Mentor QA engineers and develop scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.

Requirements

• Bachelor’s degree in Computer Science, MIS, Engineering, or a related technical field.
• 8+ years of experience in Software Development or SDET/SET roles.
• 5+ years hands-on experience developing automation frameworks using JavaScript/TypeScript or Java.
• Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.
• 2+ years testing AI-powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
• Hands-on experience validating agent orchestration workflows, conversational AI applications, multi-step reasoning systems, and AI-driven business processes.
• Experience testing AI orchestration frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies.
• Expertise in API testing, backend integration testing, UI automation, and end-to-end workflow validation.
• Solid understanding of automation architecture, design patterns, dependency injection, and inversion of control.
• Experience with cloud-native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes) and integrating automated tests into CI/CD pipelines.
• Experience validating asynchronous, event-driven, and distributed systems with emphasis on resiliency and fault tolerance.
• Strong analytical, debugging, troubleshooting, and problem-solving skills.
• Excellent organizational, communication, and stakeholder collaboration skills in Agile environments.
• Experience with AI observability, telemetry, RAG pipeline validation, vector databases, or prompt evaluation frameworks is preferred.

Preferred Qualifications

• Experience with Azure DevOps, GitHub Actions, or similar CI/CD platforms.
• Experience validating RAG pipelines, vector databases, and AI response accuracy.
• Knowledge of healthcare systems, HIPAA compliance, and regulated enterprise environments.
• Experience with event-driven architectures, messaging systems, and distributed data platforms (including SQL Server).
• Prior experience leading QA teams, mentoring engineers, and implementing enterprise test reporting and quality metrics.

Benefits

• Remote work
• Generous annual bonus opportunity
• 401(k) with employer match
• Flexible Time Off and 11 paid holidays
• “Your Day Your Way” cultural day
• LinkedIn Learning access
• Medical, Dental, and Vision plans
• Health Savings Account with employer match and Flexible Spending Account
• 100% company-paid parental leave (after 6 months)
• Company-paid life insurance and short/long-term disability
• Peer recognition program and Employee Assistance Program (free counseling)
• Corporate discounts on retail, travel, and entertainment

Work Environment

• Remote work
• Primarily computer-based work

Compliance

All activities must comply with EEO laws, HIPAA, ERISA, and other applicable regulations.

Tech Stack

JavaScript, TypeScript, Java, Playwright, Cypress, Postman, K6, LangGraph, CrewAI, AutoGen, Semantic Kernel, LLM, RAG pipelines, vector databases, Docker, Kubernetes, Azure, AWS, GCP, Azure DevOps, GitHub Actions, SQL Server, CI/CD, AI observability, prompt evaluation frameworks

Skills

Test Automation, Quality Engineering, Automation Framework Design, AI Testing, Agentic AI Validation, API Testing, UI Automation, Integration Testing, End-to-End Testing, Performance Testing, CI/CD Integration, Test Environment Management, Observability, Debugging, Troubleshooting, Mentoring, Stakeholder Communication, Agile, Cloud-native Practices, Resiliency and Fault Tolerance

Benefits

• Fully Remote
• Generous annual bonus opportunity
• 401(k) with Employer Match
• Flexible Time Off
• 11 paid holidays
• Your Day Your Way cultural day
• LinkedIn Learning access
• Medical Insurance
• Dental Insurance
• Vision Insurance
• Health Savings Account with employer match
• Flexible Spending Account
• 100% Company-Paid Parental leave (after 6 months)
• Company-Paid Life Insurance
• Short/Long Term Disability Insurance
• Peer Recognition Program
• Employee Assistance Program (free counseling)
• Corporate Discounts