Data Integration Developer
Company: Teksands | Date Posted: 21-Sep-2023 | Qualification: B.E/B.Tech, MCA, ME/MTech, Bsc, BCA
||Data Integration Developer
||Data Integration, Nifi, Cloudera
||5 - 11 (years)
Responsible for designing and implementing ETL (Extract, Transform, Load) processes to integrate data from various sources into a data warehouse or data lake.
- Extract data from diverse sources like databases, APIs, flat files, and streaming data.
- Cleanse, transform, and enrich data to ensure consistency and accuracy.
ETL Workflow Design
- Design ETL workflows and data pipelines to move and process data efficiently.
- Integrate data from multiple sources, ensuring data quality and consistency.
- Map source data to target data structures and manage data schema changes.
- Optimize ETL processes for performance and scalability.
- Implement error handling and logging mechanisms for data quality control.
- Load transformed data into data warehouses or data marts for analysis.
Monitoring and Logging
- Set up monitoring and logging for ETL processes to detect and address issues.
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements.
- Document ETL processes, data lineage, and metadata.
- Ensure data governance and compliance with data policies and regulations.
Skills and Qualifications
Apache Nifi and Spark
SQL and Scripting
- Strong SQL skills for data manipulation and scripting (Python, etc.).
- Knowledge of data warehousing concepts and technologies.
Data Quality Management
- Familiarity with data quality tools and best practices.
- Ability to troubleshoot ETL process issues and optimize workflows.
Submit Your Latest CV Quick Apply