Job Description
We have a vacancy for a DevOps Data Engineer to build and operate data pipelines and Data Factory on AWS and Microsoft Azure Platforms.
The data engineer, which is an emerging role in the data and analytics team, will play a pivotal role in operationalising the most-urgent data and analytics initiatives for analytics use cases.
The bulk of the data engineer's work would be in building, managing and optimizing data pipelines, working with our data scientists and technical development team to move these data pipelines effectively into production for key data and analytics consumers (for example business/data analysts, data scientists or any user that needs curated data for data and analytics use cases).
Key Responsibilities
Data engineers need to guarantee compliance with data governance and data security requirements while creating, improving and operationalizing these integrated and reusable data pipelines.
Strong ability to integrate analytics and (or) data science results with business processes
This role will require both creative and collaborative working with the wider business.
It will involve evangelizing effective data management practices and promoting better understanding of data and analytics.
Data engineers will also be expected to collaborate with data scientists, data analysts and other data consumers and work on the models and algorithms developed by them in order to optimize them for data quality, security and governance and put them into production leading to potentially large productivity gains.
What do I need?
To be considered for this role, we would love you to have:
- Experience in DevOps on AWS/Azure and data management experience e.g. data profiling, large volume data handling.
- Extensive experience of developing using the AWS/Azure analytics components including Data Lakes, Power BI and Microsoft Visual Studio, Data Factory, HDInsight, Data Factory, SQL DB/DW, SSIS, ETL tooling and scripting languages: R, Scala, Python.
- Fully conversant with Agile and DevOps development methodology and concepts as applied to data driven analytics projects. Including CI/CD Coding, security testing best practice and standards.
- Experience with designing, building, and operating analytics solutions using AWS/Azure cloud technologies and in automated data driven testing.
