We are seeking a highly experienced Data Engineer with 8–10 years of expertise in designing and optimizing scalable ETL pipelines and enterprise data warehouse solutions across large and complex datasets.
Responsibilities
- Have 8-10 years of experience in the data ETL processes.
- Proficiency in SQL, R, PySpark, Scala and Python programming languages that relate to data processing
- Familiarity of PostgresSQL, SQL Server and MongoDB and various relational datababses
- Experience with data modelling, Machine Learning and experimenting ML models with MLOps is a huge plus
- Full Stack development experience with React JS, Next JS frameworks is a big plus
- Experience in designing and building flexible, efficient data pipelines from various data sources.
- Experience in fine tuning and optimizing the pipelines for quality and performance.
- Knowledge of Databricks, Snowflake or Microsoft data fabric is a plus
- Must have knowledge and experience in large datasets to meet both functional and non functional requirements
- Working experience with US clients is preferred.
- Working experience in Insurance data domain is a plus
- Knowledge of Apache Airflow, Cloud platforms - AWS(Including AWS data migration tool usage), Azure and GCP is a plus
- Working knowledge of Apache Spark and similar compute engines.
- Working knowledge of data mapping algorithms in transforming the data from source to destination.
