Lead Data Engineer - Platform Administration - Jobs in United Arab Emirates, Dubai

Job Description

Job PurposeThe Lead Data Engineer is a fully-participating member of a cross-functional team working autonomously on technology development and problem resolution in the Enterprise Data & Analytics space. The role involves the design, development, testing, implementation, support and maintenance of technical data and analytic solutions and products that support Emirates Airlines and the Emirates Group businesses - with a focus on technical solution design and end-to-end fit within the data architecture. The position demands in-depth expertise in the respective technology profiles, and involves collaboration with business stakeholders, product owners, delivery leads, architects and site reliability engineers to deliver high-quality product features successfully, and in a timely fashion.Job Outline:-Lead technical design, development and deployment for solutions within a team. Carry out effective technical design reviews to ensure that the right architecture patterns are used by the team. Conduct required source system analysis, data profiling, source-to-target mapping activities and data modelling (both logical and physical) to ensure delivery on business needs.-Complete data engineering coding and design tasks on problems of high scope and complexity. Demonstrate good coding principles and exercise good judgement in designing and building solutions. Conduct code review for peers. Ensure solutions adhere to published data privacy and cybersecurity principles.-Implement and practice fit-for-purpose estimation techniques, to promote iterative delivery. Help the team in refining estimates during sprint planning, with optimal assumptions and inter-dependency understanding.-Operate with a data-driven mindset. Help translate data and analytics requirements into data solutions based on the approved technical designs. Assist with data analysis activities such as source system analysis and data modelling, to ensure solutions deliver on business needs. Work with team members to envision solutions as a set of data products and data pipelines.-Ensure solutions being built are stable, scalable and maintainable. Enable test automation and ensure CI/CD pipelines are in good health. Implement monitoring of data applications and track product quality, performance and stability. Drive corrective, adaptive, preventative and perfective actions to maintain solution reliability and quality.-Collaborate with members of the cross-functional team to identify areas of inefficiency and propose solutions. Identify activities resulting in optimal resource utilization, cost reduction, technical debt remediation, service improvement and reuse value. Partner with architects and product owners to prioritize and implement such activities.-Update data inventories and registries as required to keep metadata and data lineage up-to-date, following agreed Data Governance standards, guidelines and principles.-Partner with fellow team members, Principal Data Engineers and Architects for developing and implementing data engineering best practices and modernization techniques, promoting solution reusability, process automation, built-in-quality, test-driven development, agile delivery, timely root cause analysis and blameless incident post-mortems. Key contributor in building and adopting data engineering playbooks for the relevant technologies, and ensuring adherence to said playbooks, as well as other published coding standards data technology blueprints.-Work effectively on incident resolution, conducting root cause analysis and handling business escalations. Engage other technical experts and vendor partners in the timely resolution of incidents. Maintain regular and transparent communication during incident management. Conduct post-mortem and lessons learned discussions.-Key contributor and catalyst in attracting and retaining data engineering talent. Handhold new joiners to ensure successful onboarding. Support engineering leadership for enturing. different working models for scaling up the data engineering team in line with delivery commitments.Qualifications & Experience- In a relevant field such as Computer Science, Computational Mathematics, Computer Engineering or Software Engineering.- Specialization or electives in a Data & Analytics field (e.g. Data Warehousing, Data Science, Business Intelligence) a nice-to-have.- Experience Data Engineering (Fewer years experience will be considered for Masters degree holders.- Minimum 5 years of Big Data Platform Administration experience out of which 3 years is mandatorily on Cloudera CDP platform.- Experience of cloud computing and data architecture concepts such as data modelling, Big Data storage, Lambda architecture, data pipelines and DevOps practices.- Deep understanding of batch and real-time data ingestion exposure to designing and coding pipelines that handle massive quantities of data (structured and unstructured), securely and in a timely fashion.- Deep understanding of system integration architecture, and demonstrated experience connecting and managing data sources, maintaining data continuity, data replication and data back-up using Big Data Platform management tools.- Experience in building reusable components.- Experience designing monitoring and alerting solutions for data pipelines and infrastructure resources.- Experience and knowledge of a wide variety of testing methods and tools covering functional, performance, regression, system integration and pipelines.- Operates with efficiency and automation in mind experience building reliable, re-usable automation frameworks (e.g. CI/CD), monitoring and maintenance dashboards.- Strong SQL querying and Linux/Python scripting skills required.- Airline industry experience a nice-to-have.Knowledge/Skills:- Strong ability to manage Big Data Platform (e.g. installations & configurations, data source management, upgrades and migration, monitoring and performance optimization, platform security & governance, automations) is mandatory- Operates with a You Code It, You Own It mindset (i.e. supports the products they build)- Demonstrated problem-solver able to design and document solutions independently.- Team player: able to collaborate with others to remove blockers, solve complex design problems and debug/resolve issues.- Able to deliver solutions (and associated value) interactively.- Is accountable and displays positive attitude.- Self-starter and has passion for exploring and learning new technologies, especially those in the Enterprise Data & Analytics space.Key Responsibilities Include:Lead Data Engineer (Platform) will work with Big Data Platform Management team to maintain, support, patch and implement key platform functionalities.. Cloudera platform maintenance and support. Applying Cloudera service packs. Upgrading Cloudera software to new release(s). Performance tuning, cluster resource optimization of cloud hosted solutions. Design, document Cloudera infrastructure maintenance and support run book. Creating, submitting, and tracking vendor support tickets. Work with distributed teams (DBA, Operating Systems, Application Production Support and DevOps Engineers). Managing Cloudera CDP components, Cloudera Manager, Kafka, Spark, YARN, Knox, HDFS. Handling Cluster sizes with multiple hosts.. Platform management automations Linux/Python shell script programming.. LDAP and Kerberos security authentication. Infrastructure backup and recovery. Working based on architectural principles, infrastructure engineering and integration.. Strong analytical and problem-solving skillsSalary & BenefitsJoin us in Dubai and enjoy an attractive tax-free salary and travel benefits that are exclusive to our industry, including discounts on flights and hotels stays around the world. You can find out more information about our employee benefits in the Working Here section of our website www.emirates.com/careers. Further information on whats it like to live and work in our cosmopolitan home city, can be found in the Dubai Lifestyle section.

Keyskills :
Data Modelling Data Architecture Cloud Computing

About Company

the Emirates Group employs over 103,363 staff from more than 160 nationalities. The Emirates Group’s extensive and diverse international portfolio includes the world’s largest international airline, Emirates, and one of the largest combined air services provider in the world, dnata. For the year financial year ending 2017-18, the Emirates Group posted a profit of AED 4.1 billion (US$ 1.1 billion).