Data Engineer

Remote, Contract position

Minimum Requirements:

- BSc Engineering/ Computer Science/ relevant IT qualification

- 4+ years experience in a Data domain role (Data engineering) / Data modelling experience in relevant environment

- Data warehouse technical experience definition /implementation/ integration.

- Strong programming skills in Python and DBA skills (SQL/PSQL/DynamoDB or other).

- Experience with data pipeline and ETL tools and reporting/analytics tools including , but not limited to , any of the following combinations (1) SSIS and SSRS, (2) ETL Frameworks, (3) Data conformance, (4) Caching, (5) Spark (6) AWS data builds.

- Experience with data modelling, data governance, and data quality.

- Strong problem-solving skills and ability to work in a fast-paced environment.

- Strong communication skills and ability to work in a team.

- Expertise in Machine Learning (ML) and deep learning frameworks.

- Explaining the thinking behind simple ML algorithms.

- Proficiency in all aspects of model architecture, data pipeline interaction, and metrics interpretation.

Ideal:

- Experience with Big Data technologies such as Hadoop and Spark.

- Experience with containerization technologies such as Docker and Kubernetes.

Responsibilities:

- Develop and implement portfolio Data modelling, assurance and utilisation strategies and frameworks that align with enterprise approved governance, data and technology strategy and the Data COE.

- Lead the implementation of these strategies within the portfolio.

- Responsible for reporting activities at a portfolio level that provides insights into the portfolio data assurance landscape, strategy and roadmap and key metrics and indicators.

- Serve as though leader and guide in the data domain by sharing knowledge identifying problems, patterns, trends, and support the development of relevant BI and MI solutions.

- Design and implement scalable and robust processes for ingesting and transforming complex datasets.

- Contribute to the development of architectural frameworks, apply architecture principles, and drive the development of data architecture models within the organisation.

- Design and develop data models using dimensional modelling and data vault techniques and ensure stated business requirements are met by these models.

- Focused on data stewardship and curation, the data engineer enables the data scientist to run their models and analyses to achieve the desired business outcomes.

- Architect, train, validate and test advanced analytics / machine learning models, using enterprise-grade software engineering practices.

- Collaborate with data scientists and analysts to understand data requirements and ensure that data models and prompt engines function as expected and data is accessible and usable.

- Design, develops and maintain automated scalable data pipelines that improve estate performance, stability and auditability.

- These include data pipelines for ETL processing.

- Monitor and troubleshoot data pipeline issues.

- Define, implement and integrate with enterprise data lake and data warehouse solutions (cloud and on-premises).

- Ingest large, complex data sets that meet functional and non-functional requirements.

- Enable the business to solve the problem of working with large volumes of data in diverse formats, and in doing so, enable innovative solutions.

- Engineer data in the appropriate formats for downstream customers, risk and product analytics or enterprise applications.

- Proficiency in managing test data, ensuring data integrity, and maintaining data privacy and security standards.

- Provide technical leadership and mentorship to junior, intermediate, and senior data specialists.

- Leadership and Team Management

- Strong leadership skills to guide and mentor squads, setting clear goals and expectations for team members.

- Competence in managing and coordinating efforts, including resource allocation, workload distribution, and task prioritisation.

- Ability to foster a collaborative and innovative team culture that promotes excellence in data engineering practices.

- Stakeholder Communication

- Manage stakeholder communication, providing regular updates on data activities, milestones, and risks.

- Excellent communication and presentation skills for effectively conveying data status, data-driven insights, and recommendations to stakeholders at all levels.

- Ability to collaborate with cross-functional teams and provide visibility into data/modelling-related matters.

- Ethical and Compliance Awareness

- Understanding of ethical considerations in data engineering, including data privacy, security, and confidentiality.

- Awareness of industry-specific compliance standards, regulations, and best practices, and the ability to ensure adherence in data engineering processes.

- Skill in conducting ethical and compliant testing and data assurance activities.

- Continuous Learning and Adaptability

- Commitment to staying updated with emerging data engineering trends, technologies, and industry developments.

- Willingness to pursue certifications, training, and continuous learning opportunities to enhance and adapt data engineering skills.

- Ability to quickly adapt to evolving project requirements and data paradigms.

Originally posted on Himalayas