Senior Data Engineer

Job Description:

• Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, Python, and Scala
• Develop and optimize ETL/ELT processes for data ingestion, transformation, and processing across diverse data sources
• Work with Hadoop and its ecosystem to process and manage large volumes of structured and unstructured data
• Develop, troubleshoot, and optimize Spark applications to improve performance, scalability, and reliability
• Use Control-M to schedule, monitor, and manage batch jobs, workflow dependencies, and data processing pipelines
• Troubleshoot job failures, resolve data pipeline issues, and ensure timely completion of scheduled workloads
• Implement data validation, error handling, and quality checks to ensure data accuracy and consistency
• Collaborate with data architects, analysts, and cross-functional teams to understand requirements and deliver reliable data solutions
• Follow coding standards, testing practices, version control, and deployment procedures
• Support production deployments, incident resolution, and ongoing maintenance of data engineering solutions

Requirements:

• 8+ years of overall Data Engineering experience
• Strong hands-on experience with Databricks and Apache Spark
• Proficiency in Python and Scala for data processing and application development
• Hands-on experience with Hadoop and big data technologies
• Experience with Control-M for batch scheduling, job monitoring, workflow orchestration, and dependency management
• Strong understanding of ETL/ELT processes, data transformations, and distributed data processing
• Experience in performance tuning, debugging, troubleshooting, and production support
• Strong SQL skills and understanding of data management concepts
• Excellent analytical, problem-solving, and collaboration skills

Benefits:

• Medical
• Dental
• Vision
• Retirement Accounts
• Long and Short Term Disability
• Life Insurance
• HSA/FSA Accounts (USA)
• Discretionary time-off
• Continuous learning and hands-on exposure to emerging technologies
• Opportunities to build expertise in Generative AI, agentic AI, and next-generation data platforms
• Collaboration with global AI, data, engineering, and consulting talent
• Opportunities to lead strategic engagements and grow strategic client partnerships