Data Integration Developer
Date Posted: 21-Sep-2023 | Qualification: B.E/B.Tech, MCA, ME/MTech, Bsc, BCA
| Position: | Data Integration Developer |
| Skills Required: | Data Integration, Nifi, Cloudera |
| Employment Type: | Full Time |
| Location: | Riyaadh |
| Job Mode: | Hybrid |
| Experience: | 5 - 11 (years) |
Job Description:
Role Description
Responsible for designing and implementing ETL (Extract, Transform, Load) processes to integrate data from various sources into a data warehouse or data lake.
Responsibilities
Data Extraction
- Extract data from diverse sources like databases, APIs, flat files, and streaming data.
Data Transformation
- Cleanse, transform, and enrich data to ensure consistency and accuracy.
ETL Workflow Design
- Design ETL workflows and data pipelines to move and process data efficiently.
Data Integration
- Integrate data from multiple sources, ensuring data quality and consistency.
Data Mapping
- Map source data to target data structures and manage data schema changes.
Performance Optimization
- Optimize ETL processes for performance and scalability.
Error Handling
- Implement error handling and logging mechanisms for data quality control.
Data Loading
- Load transformed data into data warehouses or data marts for analysis.
Monitoring and Logging
- Set up monitoring and logging for ETL processes to detect and address issues.
Collaboration
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements.
Documentation
- Document ETL processes, data lineage, and metadata.
Data Governance
- Ensure data governance and compliance with data policies and regulations.
Skills and Qualifications
Must have
Apache Nifi and Spark
SQL and Scripting
- Strong SQL skills for data manipulation and scripting (Python, etc.).
Data Warehousing
- Knowledge of data warehousing concepts and technologies.
Data Quality Management
- Familiarity with data quality tools and best practices.
Problem-Solving Skills
- Ability to troubleshoot ETL process issues and optimize workflows.
