Senior Applied Ai Engineer, Agent Quality & Evalu

About the Role

Flodesk is hiring a Senior Applied AI Engineer to own agent quality and evaluation for member-facing AI features. You will build the prompting, agent instructions, evaluation infrastructure, monitoring, and tooling needed to measure and improve production LLM/agent behavior across the product.

Building AI dev tools and evaluation for LLMs and agents; focuses on prompts, agent loops, and observability.

Location: United States

Salary: USD 150,000 - 250,000/year

Type: Full Time

Posted: 2 days ago

Category: AI/ML & Data

Vibe Markers: 💻 Open Source, 🔥 Hot

Experience Level: Senior

Job Description

Role

Flodesk is seeking a Senior Applied AI Engineer (Agent Quality & Evaluations) to report to the Head of Product, AI Systems and Core Experience. The role focuses on shaping AI-driven member experiences by owning prompts, agent instructions, quality definitions, evaluation criteria, and the continuous-improvement loop through technical tooling and evaluation infrastructure.

Key Responsibilities

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs (segmentation, scheduling, analytics, recommendations).
• Create, test, version, and document prompts and agent instructions with a clear change history and measurable impact.
• Define how context, member data, tools, and product state should be used; prototype agent loops, tools, and structured interfaces.
• Own AI evaluation practice: representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.
• Build and operate evaluation harnesses, monitoring, and quality dashboards to enable scalable measurement.
• Diagnose failures across prompts, context, orchestration, models, tools, data, and product code; fix issues or partner with owners.
• Set quality baselines and release gates for prompt, model, and agent changes; convert production failures and feedback into regression cases.
• Partner with product, design, marketing, and copy experts to encode product point-of-view while contributing product judgment.

Requirements

• Proven track record of building or meaningfully improving production LLM or agentic products (not just prototypes).
• Strong software engineering skills in Python, TypeScript, or similar languages to build internal tools, evaluation systems, and prototypes.
• Clear understanding of interactions among prompts, context, tools, agent loops, orchestration, and product state.
• Direct experience with LLM evaluation, observability, and experimental design; ability to combine automated checks with human judgment.
• Strong product sense to translate ambiguous member feedback into testable quality definitions and concrete system changes.
• Systems thinking and emphasis on reusable patterns to avoid short-term assumptions that hinder future surfaces.
• Clear communication across technical and nontechnical disciplines; able to explain tradeoffs involving quality, latency, reliability, and cost.

Bonus / Nice to have

• Experience building content-generation, creative, or marketing tools.
• Experience building multi-turn agents that use tools and preserve user intent across edits.
• Experience with Langfuse, Braintrust, or comparable evaluation and observability platforms.
• Experience working on products for small businesses, creators, or marketers.

Compensation & Benefits

• Base salary range: $150,000 - $250,000 (tiers noted: Tier 1 cities $165,000 - $250,000; other US locations $150,000 - $235,000).
• Fully paid health insurance for individual coverage.
• Parental leave: 16 weeks paid for non-birthing parents; 22 weeks paid maternity leave for birthing parents.
• Unlimited flexible time off.
• 401(k) match (US employees only).
• $1,000 annual stipend for learning and development.
• Remote-first company with globally distributed team and in-person hubs.

Tech Stack

Python, TypeScript, LLMs, Langfuse, Braintrust

Skills

Systems thinking, Software engineering, Prompt engineering, Evaluation & observability, Experimental design, Monitoring and diagnostics, Product sense, Prototyping, Cross-functional collaboration, Documentation and versioning, Quality assurance

Benefits

• Remote-first
• Fully paid health insurance (individual coverage)
• 16 weeks paid parental leave (non-birthing parents)
• 22 weeks paid maternity leave (birthing parents)
• Unlimited flexible time off
• 401(k) match (US employees only)
• $1,000 annual learning and development stipend