Data Engineer (Databricks)

Data Engineer | Databricks - Melbourne (hybrid) 
 

Role Overview:

We are seeking an experienced Senior Data Engineer with extensive Databricks Lakehouse expertise to join a client’s Data Engineering Hub. 

The ideal candidate will have at least 3+ years of experience working with Databricks technologies and a strong background in data engineering, with a proven track record of designing and implementing scalable data solutions on Databricks and Lakehouse architecture.

Key Responsibilities:

• Develop solutions for data pipelines on Databricks, including pattern development to enable other engineers
• Develop and optimise notebooks using PySpark and Spark SQL
• Implement performance monitoring and tuning for data workflows
• Create and maintain data models to support business requirements
• Ensure data quality and integrity across all data pipelines
• Establish and implement best practices for data engineering, including data governance and security

Required Qualifications:

• Advanced PySpark development, including optimisation of wide transformations, shuffle reduction, and memory management
• Strong understanding of Spark execution plans, cluster configuration, and performance tuning
• Experience with Delta Lake, including ACID transactions, schema evolution, OPTIMIZE/VACUUM, and MERGE patterns
• Ability to design and implement Bronze ? Silver ? Gold Lakehouse architectures
• Proven experience as a Data Engineer with specific focus on Databricks
• Advanced SQL skills and experience with various database technologies
• Exceptional problem-solving abilities and attention to detail
• Experienced with Data Modelling, Normalisation and Dimensional Modelling
• Excellent communication skills and ability to work in a team environment