MehnaTi

Data Engineer

The Data Engineer supports the design, implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable and secure manner. He/She focuses on defining optimal solutions to data collection, processing and warehousing. He designs, codes and tests data systems and works on implementing those into the internal infrastructure. He focuses on collecting, parsing, managing, analyzing and visualizing large sets of data to turn information into insights accessible through multiple platforms. He is proficient in database systems, scripting and programming languages required by the organization. He is also familiar with the relevant software platforms on which the solution is deployed on. The Data Engineer is passionate about numbers and works with large data sets. He has a keenness for understanding business processes and resolving challenges in order to provide solutions with the help of clean and interlinked databases and architectures.

Skills and Competencies

Technical Skills & Competencies

Business Needs Analysis
Proficiency Level
"Document business requirements and identify basic needs as well as potential solutions"
2
Change Management
Proficiency Level
"Apply change control procedures in work processes, assess impact of change and develop communications to prepare stakeholders for the change"
3
Computational Modelling
Proficiency Level
"Identify and utilise appropriate statistical algorithms and data models to test hypotheses and derive patterns or solutions "
3
Configuration Tracking
Proficiency Level
"Label, track and document all configuration items and changes to software projects using standard tools and templates "
1
"Verify accuracy, completeness and currency of information in configuration logs and review unauthorized changes, diversions or inappropriate use of software assets "
2
Data Design
Proficiency Level
"Identify data requirements and support the design of database models, incorporating parameters, fields and mechanisms for the maintenance, storage and retrieval of data "
3

Generic Skills & Competencies

Leadership
Proficiency Level
Lead by example at team level. Encourage and guide others to adopt a point of view, make changes or take action. Provide a team environment that facilitates relationships building, teamwork and the development of others.
Intermediate
Developing People
Proficiency Level
Provide coaching to others to develop their skills and knowledge on their jobs to enhance performance.
Intermediate
Communication
Proficiency Level
"Articulate and discuss ideas and persuade others to achieve common outcomes "
Intermediate
Transdisciplinary Thinking
Proficiency Level
Co-relate material from diverse knowledge bases to guide decisions and policy making. Participate in reflective and trans-disciplinary communities within and outside the organization.
Intermediate
Computational Thinking
Proficiency Level
Modify existing computational models, tools and techniques to develop different solutions.
Intermediate

Critical Work Functions and Key Tasks

Identify business needs

• Identify suitable data structures based on business needs to ensure availability and accessibility of data 
• Determine technical system requirements based on data needs 
• Keep abreast of latest technologies and products in database and data processing software, and technologies 

Build and maintain data pipeline

• Assist in building scalable data pipelines to extract, transform, load and integrate data 
• Develop codes and scripts to process structured and unstructured data in real-time from a variety of data sources 
• Test data pipelines for scalability and reliability to process high data volume, variety and velocity 
• Consolidate and create data storage solutions for storage and retrieval of information 
• Develop prototypes and proof-of-concepts for data solutions 
• Monitor data system performance 
• Support the handling and logging of errors 
• Develop backup data archiving systems to ensure system continuity 
• Implement and monitor data security and privacy measures on existing data solutions 

Optimise solution performance

• Assist in the integration of data systems with existing infrastructure 
• Develop tools to improve data flows between internal and/or external systems and the data warehouse 
• Automate the data collection and analysis processes, data releasing and reporting tools 
• Test data system configurations to increase efficiency

More Information

Related Occupations

Get yourself a new skill

In this Path

Coming soon...