Principal AI Engineer - Office of the CTO
Posted Updated
We're looking for a Principal AI Engineer to architect, build, and harden the agentic AI systems that power our products. This is a hands-on, full-stack AI role: you'll work across the entire AI stack from the foundation (model lifecycle, experimentation, infrastructure) through shared services (agent orchestration, RAG and grounding, gateways and routing) up to the agents and workflows that reach customers. You'll set the technical direction for how we design agents, prove what works through rigorous evaluation, and turn promising prototypes into reliable, observable, governed production systems.
Here is how, through this role, you will contribute to BMC’s and your own success:
- Own the architecture for agentic systems end to end - reasoning and planning, tool/function calling, multi-agent coordination, routing, memory, and human-in-the-loop handoffs and define reusable patterns and reference implementations.
- Prototype new agentic capabilities quickly, then drive the ones that prove into production-grade systems; write production-quality code and set the engineering bar.
- Build and integrate across the stack: RAG and knowledge services, model/tool routing, prompt and context management, orchestration runtimes, and inference serving. Evaluate and adopt emerging models, frameworks, and protocols.
- Stand up the evaluation and experimentation strategy that gates what we ship, offline suites and golden datasets, regression tests in CI/CD, online evaluators on production traffic, calibrated LLM-as-a-judge graders, and A/B experiments to ensure safe and reliable deployments of agents in production.
- Define the metrics that matter (task success, grounded-ness, cost, safety, etc.) and define and build the tracing and observability to measure them across multi-turn interactions, closing the loop from error analysis to continuous improvement.
- Set technical direction across teams, mentor engineers, and translate complex architectures into clear guidance for partners and customers.