Global Life Insurer - Asia AI Quality Evaluation Engineer

AI Quality & Evals Engineer

Regional AI Centre of Excellence | Global Life Insurer

Location: Hong Kong; candidates based in the Philippines or India may also be considered

Experience: 4+ years in software, ML or data engineering, including 2+ years focused on production LLM or agent quality, evaluation or safety

The opportunity

We are partnered with a global life insurer to build out its regional AI Centre of Excellence, supporting the adoption of generative AI and agentic systems across its Asian businesses. As these capabilities take on more complex workflows, engineering robust evaluation and operational controls will be essential to their success.

As an AI Quality & Evals Engineer, you will build the systems that assess AI behaviour, monitor live activity and intervene when necessary. This is a hands-on engineering role spanning evaluation infrastructure, streaming monitors, runtime controls and automated governance evidence.

What you will do

• Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria.
• Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve.
• Test agent-specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions.
• Develop adversarial and red-team suites covering prompt injection, data-exfiltration attempts and privilege escalation.
• Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies.
• Build streaming evaluations and risk-based monitors for quality, task success, latency, cost and behavioural drift.
• Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms.
• Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit.
• Share operational responsibility for the agent control platform, including on-call response and decisions to constrain or halt agent activity.

What you will bring

• Strong Python and engineering discipline across testing, typing, versioning and CI/CD.
• Proven experience building online evaluation over live production traffic.

Preferred: GCP/Vertex AI; event-driven architectures; MCP server development; AI performance and cost dashboards; and AI governance knowledge.

Why this role

You will help establish the production foundations of a regional AI capability, combining deep engineering ownership with exposure to modern ML, LLM and agent technologies in a complex enterprise environment.

Apply

Please share your CV and a brief overview of relevant experience for a confidential discussion.

Argyll Scott Asia is acting as an Employment Business in relation to this vacancy.