### **Lead Data Engineer – Insights Platform**
#### **Job Overview:**
We are seeking a **Lead Data Engineer** to take a pivotal role in ensuring that complex data sets delivered into our Insights Platform meet all functional and non-functional business requirements. In this role, you will design and maintain robust Databricks capabilities to extract, transform, and load (ETL) data from various sources into a unified data platform. Your leadership will drive the optimization of Databricks for performance, scalability, and cost-effectiveness, ensuring reliable and efficient data processing across the organization.
#### **Key Responsibilities:**
– **Data Integration and Transformation:**
– Design, implement, and maintain ETL processes in Databricks to integrate data from multiple sources into a cohesive data platform.
– Develop scalable and fault-tolerant data pipelines to ensure data integrity and quality throughout its lifecycle.
– **Platform Optimization:**
– Lead the optimization of Databricks environments to enhance performance, scalability, and cost-efficiency, ensuring robust and efficient data processing workflows.
– Continuously evaluate and implement new Databricks features and best practices to stay ahead of industry trends.
– **Team Leadership and Collaboration:**
– Manage and mentor a team of data engineers, data analysts, and data scientists, fostering a culture of collaboration and continuous improvement.
– Work closely with cross-functional teams to understand business requirements and translate them into effective data engineering solutions.
– **Platform Maintenance and Stability:**
– Oversee the maintenance and stability of Databricks pipelines, proactively identifying and resolving issues to ensure uninterrupted data flow.
– Develop monitoring and alerting mechanisms to detect and respond to pipeline failures or performance issues.
– **Documentation and Training:**
– Lead the documentation of data engineering processes and create training materials to support cross-functional teams in utilizing Databricks for data engineering tasks.
– Provide ongoing support and training to team members and stakeholders on best practices in data engineering.
– **Stakeholder Engagement:**
– Engage with strategic suppliers and account executives to stay informed on the latest Databricks features, capabilities, and industry trends.
– Collaborate with stakeholders to align data engineering efforts with business objectives and ensure high-quality data solutions.
#### **Qualifications and Experience:**
– **Educational Background:**
– B.S. or M.S. in Computer Science, Statistics, Business Analytics, Mathematics, or a related field.
– **Professional Experience:**
– Proven experience in managing teams of data engineers and data scientists, particularly in collaboration with third-party vendors.
– Demonstrated expertise in building and optimizing Databricks notebooks, workflows, and data pipelines.
– **Technical Skills:**
– Strong proficiency in SQL and Python/PySpark for data transformation, analysis, and scripting.
– Experience integrating Databricks with other Azure data services such as Azure Data Factory, Azure Synapse, and Azure DevOps.
– Knowledge of data modeling techniques, including the creation of relational and dimensional data models.
– **Industry Knowledge:**
– Familiarity with data governance best practices, including data quality management and master data management.
– **Soft Skills:**
– Excellent problem-solving skills with a proactive approach to identifying and resolving issues.
– Strong communication skills, both written and verbal, with the ability to document processes and present findings clearly.
– Ability to work effectively with cross-functional teams and stakeholders to achieve common goals.
– **Additional Attributes:**
– Passion for continuous learning and staying up to date with the latest developments in Databricks and data engineering.
—
This position offers an exciting opportunity to lead data engineering initiatives and make a significant impact on the organization’s data capabilities. If you’re passionate about data, innovation, and leadership, we’d love to hear from you!
