Senior Principal Machine Learning Engineer
Atlassian
Working at Atlassian
Atlassians have flexibility in where they work – whether in an office, from home, or a combination of the two. We can hire people in any country where we have a legal entity. Interviews and onboarding are conducted virtually, a part of being a distributed-first company.
About the role
Atlassian is seeking a Senior Principal Machine Learning Engineer to join our GenAI Platform organization, focusing on the quality and reliability of Rovo Chat.
Rovo is Atlassian’s AI teammate, embedded across our products to help teams search, understand, and act on their work. In this role, you will be the technical driver behind making Rovo Chat exceptionally accurate, trustworthy, observable, and reliable at scale. You will define what “great” looks like for GenAI chat quality, build the platforms and evaluation systems to measure it, and lead cross‑org efforts that materially improve customer outcomes and reduce incidents.
This work sits at the intersection of LLMs, retrieval-augmented generation (RAG), evaluation and quality frameworks, observability, and large‑scale production systems.
Your future team
You will join the GenAI Platform pillar within Central AI / Engineering‑AI, working closely with the Rovo Chat product and engineering teams.
Our mission is to:
Provide a central GenAI platform (models, infra, evaluation, safety, and tooling) that powers AI experiences across Atlassian.
Ensure Rovo Chat is a highly reliable, high‑quality assistant across Jira, Confluence, and the rest of our product suite.
Drive quality, observability, and debuggability for GenAI experiences, so we can quickly detect, root‑cause, and fix issues that impact customers, incidents, Disturbed tickets, and DoS escalations.
You’ll collaborate with:
Rovo Chat and Search & Conversation teams on chat UX and retrieval quality,
AI Fundamentals / AI Modeling / ML Platform on modeling, evaluation, training, and serving,
SRE / TechOps / Support (Disturbed / DoS) on reliability, incident response, and root‑cause tooling.
What you’ll do
As a Senior Principal Machine Learning Engineer, you will:
Set the bar for Rovo Chat quality & reliability
-
Define and evolve a north‑star quality and reliability framework for Rovo Chat, spanning:
Answer correctness, faithfulness, and grounding,
Safety and policy adherence,
Latency, robustness, and uptime,
Incident, Disturbed, and DoS impact.
Translate these into measurable metrics, SLAs/SLOs, and dashboards that are adopted across product and platform teams.
Build the evaluation & observability stack for GenAI chat
-
Design and lead implementation of end‑to‑end evaluation pipelines for Rovo Chat, including:
Offline evals (benchmarks, synthetic data, golden sets, human‑in‑the‑loop labeling),
Online evals (A/B tests, interleaving, guardrail metrics),
LLM‑as‑a‑judge and other automated evaluation techniques.
Drive observability and debuggability improvements (e.g., tracing, attribution, feature logging, and model behavior introspection) so engineers can quickly root‑cause regressions and incidents.
-
Partner with SRE/TechOps to connect evaluation and observability signals into incident management, improving:
% of incidents successfully root‑caused,
Disturbed ticket and DoS resolution efficiency.
Lead technical strategy for GenAI platform quality
Define and own technical roadmaps for GenAI platform features that directly impact Rovo Chat quality and reliability (e.g., retrieval quality, RAG orchestration, guardrails, safety filters, fallback strategies, model selection/routing).
-
Make high‑impact architecture decisions across:
LLM and RAG architectures,
Knowledge ingestion and retrieval,
Evaluation & monitoring infra,
Trust & Safety layers.
Identify and prioritize cross‑pillar investments (e.g., shared eval frameworks, reusable prompt libraries, safety and policy enforcement) that raise the bar across Atlassian AI.
Deliver high‑impact improvements to customer outcomes
Use data from incidents, Disturbed tickets, DoS escalations, and product telemetry to identify systemic quality and reliability gaps.
-
Lead multi‑team initiatives to:
Reduce production incidents and regressions,
Improve “first‑try success” rate of answers,
Decrease hallucinations and unsafe outputs,
Improve CSAT/NPS and key adoption/retention metrics for Rovo Chat.
Work closely with PMs and designers to ensure quality and reliability are visible, explainable, and trustworthy to customers.
Mentor, influence, and grow the AI community
Mentor senior/principal ML engineers and ML systems engineers across GenAI Platform and Rovo Chat.
Act as a technical thought partner to engineering and product leadership on GenAI quality and reliability strategy.
Contribute to AI best practices across Atlassian via design reviews, internal talks, and cross‑org forums.
On your first day, we’ll expect you to have
10+ years of industry experience in machine learning / applied AI, including shipping production systems at scale.
-
Deep hands‑on expertise with LLMs and/or large‑scale NLP systems, including at least one of:
Retrieval‑augmented generation (RAG),
Search & ranking / relevance,
Conversational AI / assistants / agents,
Evaluation and quality frameworks for LLM applications.
-
Strong coding skills in Python (and/or Java) with the ability to write performant, production‑quality code, plus:
Solid experience with Java/Kotlin and large‑scale data processing (e.g., Spark),
Familiarity with cloud environments (e.g., AWS, Databricks) and modern ML tooling.
-
Demonstrated experience designing and operating ML systems end‑to‑end, including:
Data pipelines and feature generation,
Training, evaluation, and deployment,
Monitoring, incident response, and iterative improvement.
-
A track record of technical leadership beyond a single team, such as:
Driving cross‑team/platform initiatives,
Making high‑impact architecture decisions,
Influencing roadmaps and org‑level priorities.
Ability to communicate complex ML concepts clearly to engineers, PMs, designers, and leadership, and to tell a compelling story with data.
A strong product sense and bias for pragmatism and iteration (80/20 mindset: knowing when “good and measurable now” beats “perfect later”).
It’s great, but not required, if you have
Master’s degree or PhD in Computer Science, Machine Learning, Statistics, or a related technical field.
-
Experience with:
LLM fine‑tuning, post‑training, and optimization (instruction tuning, preference optimization, safety tuning),
Model evaluation and guardrails (LLM‑as‑a‑judge, red‑teaming, safety frameworks),
High‑reliability systems in SaaS (SLOs, error budgets, incident command, post‑incident analysis).
Prior work on AI assistants or conversational experiences in a B2B SaaS or productivity setting.
Experience partnering with SRE / incident management / support to reduce MTTR, improve root‑cause coverage, and lower ticket volume through better tooling and automation.
Experience building observability and debuggability tools for ML or GenAI systems (e.g., tracing, experiment management, evaluation platforms).
Compensation
At Atlassian, we strive to design equitable, explainable, and competitive compensation programs. To support this goal, the baseline of our range is higher than that of the typical market range, but in turn we expect to hire most candidates near this baseline. Base pay within the range is ultimately determined by a candidate's skills, expertise, or experience. In the United States, we have three geographic pay zones. For this role, our current base pay ranges for new hires in each zone are:
Zone A: USD 267300 - USD 348975
Zone B: USD 241200 - USD 314900
Zone C: USD 222300 - USD 290225
This role may also be eligible for benefits, bonuses, commissions, and equity.
Please visit go.atlassian.com/payzones for more information on which locations are included in each of our geographic pay zones. However, please confirm the zone for your specific location with your recruiter.
Benefits & Perks
Atlassian offers a wide range of perks and benefits designed to support you, your family and to help you engage with your local community. Our offerings include health and wellbeing resources, paid volunteer days, and so much more. To learn more, visit go.atlassian.com/perksandbenefits.
About Atlassian
At Atlassian, we're motivated by a common goal: to unleash the potential of every team. Our software products help teams all over the planet and our solutions are designed for all types of work. Team collaboration through our tools makes what may be impossible alone, possible together.
We believe that the unique contributions of all Atlassians create our success. To ensure that our products and culture continue to incorporate everyone's perspectives and experience, we never discriminate based on race, religion, national origin, gender identity or expression, sexual orientation, age, or marital, veteran, or disability status. All your information will be kept confidential according to EEO guidelines.
To provide you the best experience, we can support with accommodations or adjustments at any stage of the recruitment process. Simply inform our Recruitment team during your conversation with them.
To learn more about our culture and hiring process, visit go.atlassian.com/crh.