Principal Data Engineer

Professional Search Group ·www.professionalsearchgroup.com.au

Location Docklands, VIC, Australia
Work type Hybrid
Type Full time
Level Principal
Source Shazamme
Information Technology
Apply direct

Lead AI & Data Platform Engineer

 

  • Location: Melbourne (Hybrid Work Model)
  • Salary: $180,000 – $210,000 + Super
  • Industry: Financial Services
  • Contract Type: Permanent / Full-Time

 

About the Role

 

We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.

You will be our internal Databricks Expert, driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.

 

Key Responsibilities

 

1. Databricks Platform Excellence

  • Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
  • Enforce robust data governance and secure discovery across the enterprise using Unity Catalog.
  • Build, monitor, and scale production pipelines via Delta Live Tables.

 

2. Production AI & MLOps Infrastructure

  • Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
  • Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI, MLflow, Langchain, and Vector Databases.
  • Manage corporate feature stores and optimize model serving endpoints.

 

3. AWS Cloud Architecture & Infrastructure as Code (IaC)

  • Design secure, high-performance cloud environments using Terraform.
  • Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
  • Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.

 

4. Performance Tuning & Streaming

  • Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
  • Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
  • Architect real-time streaming pipelines leveraging Kinesis or Kafka.

 

Your Profile

 

To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.

 

Technical Requirements:

  • Core Languages: Mastery of PySpark, Python, SQL, and Scala.
  • AI Frameworks: Strong familiarity with PyTorch, TensorFlow, or Hugging Face.
  • DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
  • Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy, compliance, and risk frameworks).

 

What’s on Offer

 

  • Competitive Compensation: $180K – $210K base salary, depending on experience.
  • Hybrid Flexibility: A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
  • Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.

Frequently asked questions

Who is hiring for the Principal Data Engineer role?
Professional Search Group is hiring for the Principal Data Engineer position, a Shazamme client. Apply directly on the employer's career site.
Where is the Principal Data Engineer job located?
The Principal Data Engineer role with Professional Search Group is based in Docklands, VIC, AU. The role is hybrid-friendly.
Is the Principal Data Engineer role remote?
Yes — the Principal Data Engineer position at Professional Search Group is hybrid. Candidates based in AU are preferred.
Is the Principal Data Engineer role full-time or contract?
This is a full time position at Professional Search Group.
What experience level is the Principal Data Engineer role?
The Principal Data Engineer position is aimed at principal-level candidates.
How do I apply for the Principal Data Engineer role at Professional Search Group?
Apply directly on Professional Search Group's career page via the Apply button on this listing. ZammeJobs links straight through to the employer's ATS — no third-party form, no resume database.
Apply direct