Senior Data Engineer
The Senior Data Engineer designs, implements and oversees maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information from structured and unstructured sources in a scalable, repeatable and secure manner. He/She assists data scientists with the extraction of valuable insights from data sets to derive valuable and actionable insights and recommendations that support business requirements. He involves in rollouts, upgrades, implementation and release of data system changes as required for streamlining of internal practices. He is proficient in database systems, scripting and programming languages required by the organization. He is also familiar with the relevant software platforms on which the solution is deployed on. The Senior Data Engineer possesses natural inclination for understanding business processes and relevant data requirements. He easily builds rapport with others and is able to put forth his ideas and recommendations in a persuasive manner, to influence stakeholders and decisions.
Skills and Competencies
Technical Skills & Competencies
Generic Skills & Competencies
Critical Work Functions and Key Tasks
• Assess the suitability of data structure to ensure availability, integrity,
quality, scalability and accessibility of data
• Translate the business' data needs into technical system requirements
• Evaluate suitability of technologies and products in database and
data processing for integration and storage
• Build data flow channels and processing systems to extract,
transform, load and integrate data
• Validate data extraction, preparation and processing systems
for accuracy of data and outputs
• Create data storage plans and solutions for information storage
and extraction
• Lead project rollouts, upgrades, implementation and release of
data system changes
• Analyse data system performance and develop solutions for improvements
• Build a metadata system to ensure documentation and cataloguing
of all available data
• Formulate data backup and fail-safe plans as part of business
continuity planning
• Ensure the security, privacy and anonymity of users accessing data systems
• Evaluate existing technologies and technology practices
• Resolve data integrity, performance, and availability issues
• Identify opportunities for improvements and optimisation
to systems and processes
• Automate processes focusing on repeatability and reliability