Staff Applied AI Engineer (Observability)

At Komodo Health, our mission is to reduce the global burden of disease. And we believe that smarter use of data is essential to this mission. That’s why we built the Healthcare Map — the industry’s largest, most complete, precise view of the U.S. healthcare system — by combining de-identified, real-world patient data with innovative algorithms and decades of clinical experience. The Healthcare Map serves as our foundation for a powerful suite of software applications, helping us answer healthcare’s most complex questions for our partners. Across the healthcare ecosystem, we’re helping our clients unlock critical insights to track detailed patient behaviors and treatment patterns, identify gaps in care, address unmet patient needs, and reduce the global burden of disease. 

As we pursue these goals, it remains essential to us that we stay grounded in our values: be awesome, seek growth, deliver “wow,” and enjoy the ride. At Komodo, you will be joining a team of ambitious, supportive Dragons with diverse backgrounds but a shared passion to deliver on our mission to reduce the burden of disease — and enjoy the journey along the way.

Healthcare in the U.S. is fragmented and inefficient. Komodo Health is working to fix that—with data.

By building the most complete view of patient journeys across the U.S., Komodo enables life sciences companies, payers, and providers to make better decisions that directly improve patient outcomes.To unlock the full potential of this data, Komodo is investing in AI-native infrastructure—systems that make AI reliable, scalable, and deeply embedded into how products are built and used.

We’re hiring Staff Applied AI Engineers to join a newly formed team focused on building the foundation that makes AI systems trustworthy at scale. This is a greenfield, backend-heavy systems role where you’ll define and build the core infrastructure that powers AI across the company. Your entry point is AI observability, evaluation, and reliability—but that is not the destination.

You’ll start by building the systems that make AI measurable and debuggable. Then you’ll expand into broader platform ownership: agent architectures, orchestration systems, and company-wide AI standards.

• Establish the observability foundation for AI systems across Komodo (logging, tracing, evaluation pipelines, feedback loops)
• Define how the organization measures LLM performance and reliability (failure modes, hallucinations, drift, system degradation)
• Build reusable infrastructure and frameworks adopted across multiple teams
• Set standards for AI system quality, monitoring, and production readiness
• Influence technical direction and platform decisions across the AI org
• Expand into broader AI platform ownership (multi-agent systems, orchestration layers, shared services)
• Mentor engineers in AI system design, debugging, and evaluation

• Design and build systems for:
  • Logging, tracing, and request visibility
  • Evaluation pipelines and benchmarking frameworks
  • Feedback loops for continuous system improvement
• Develop approaches to detect hallucinations, model drift, performance regressions and system failures
• Establish organization-wide standards for AI monitoring and evaluation

• Build foundational AI systems: agent frameworks, orchestration and tool-calling systems, and shared infrastructure for LLM-powered applications
• Contribute to company-wide AI architecture and standards
• Make high-impact build vs. buy decisions for AI tooling and platforms
• Partner cross-functionally to ensure systems are scalable, reliable, secure and production-ready

• →Strong experience building production AI systems (not just prototypes)
• →Deep expertise in LLMs and applied AI systems: tool/function calling, agent orchestration and prompt systems
• →Hands-on experience with:
  • →AI observability, evaluation, or reliability systems
  • →Monitoring, debugging, and performance analysis in production
• →Strong backend engineering skills: Python, APIs, distributed systems, or platform architecture
• →Proven ability to operate in ambiguous, undefined environments
• →Track record of owning systems end-to-end and driving technical direction
• →You will operate as a strategic AI leader—setting patterns, driving adoption, and ensuring Komodo stays at the forefront of applied AI innovation.

• Experience building internal observability platforms and valuation frameworks for LLM systems.
• Familiarity with: Request tracing, replay systems, or model monitoring
• Experience leading large-scale distributed data and compute systems (e.g., Spark, Databricks, Snowflake).
• Healthcare data experience (not required)

** Location flexible to NYC or SF hybrid, and remote

At Komodo, we're not just witnessing the AI revolution – we're leading it. This is a pivotal moment in time, where being first to market with AI transforms industries and sets the bar. We've already established industry leadership in leveraging AI to revolutionize healthcare, and we expect every team member to contribute. AI here isn't optional; it's foundational. We expect you to integrate AI into your daily work – from summarizing documents to automating workflows and uncovering insights. This isn't just about efficiency; it's about making every moment more meaningful, building on trust in AI, and driving our collective success.

Komodo Health has a hybrid work model with hubs in San Francisco, New York City, and Chicago. Roles vary — some can be performed from anywhere in the country, others are scoped to a specific region, and some are based near one of our hubs. For hub-based Dragons, we're building intentional in-office rhythms alongside the flexibility that's core to how we work. Whatever your setup, expectations will always be clear before you join.

Komodo Health provides equal employment opportunities to all applicants and employees. We prohibit discrimination and harassment of any type with regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws. 

By submitting your application, you acknowledge that you have read and understand Komodo Health’s Privacy Notice for Employees and Contractors.

This notice explains how we collect, use, and retain applicant data.