Data Engineer

Job Description:

• Design, build, and maintain trusted risk data products and scalable batch and streaming data pipelines using cloud-native technologies
• Integrate data from risk platforms, control systems, vendor tools, operational applications, and external sources into governed, analytics-ready datasets
• Develop and optimize ETL/ELT workflows for automated data ingestion, transformation, validation, reconciliation, and delivery
• Build and manage data processing workloads in modern data lake and lakehouse environments, including Microsoft Fabric and OneLake
• Implement data quality, monitoring, lineage, and reconciliation processes
• Create curated datasets and analytical outputs for operational reporting, leadership dashboards, trend analysis, risk identification, and decision-making
• Optimize data architectures, schemas, and processing patterns for scalability, performance, resilience, and cost efficiency
• Develop reusable frameworks, engineering standards, CI/CD pipelines, automated testing, and operational monitoring
• Partner with analytics, product, risk, governance, security, and compliance stakeholders on data definitions, metrics, quality standards, and secure data access
• Participate in Agile delivery activities including sprint planning, backlog refinement, design reviews, and continuous improvement initiatives

Requirements:

• 4+ years of experience as a Data Engineer or in a similar role building and supporting production-grade data pipelines and data products
• Hands-on experience with Apache Spark for large-scale data processing and transformation
• Strong proficiency in Python, SQL, and modern data engineering best practices
• Experience developing ETL/ELT solutions within cloud-based data lake, lakehouse, or analytics platforms
• Experience designing and optimizing analytical data models supporting reporting, dashboards, operational insights, and advanced analytics
• Strong understanding of data quality, validation, monitoring, reconciliation, and production support processes
• Experience with CI/CD pipelines, version control, automated testing, and infrastructure-as-code concepts
• Ability to troubleshoot complex data quality, performance, and integration issues across multiple data sources
• Strong verbal and written communication skills
• Experience with Microsoft Fabric, OneLake, or similar modern analytics platforms preferred
• Experience working with risk, controls, compliance, audit, governance, or regulatory data domains
• Experience building real-time, near real-time, or event-driven data processing solutions
• Familiarity with data governance, metadata management, data lineage, access controls, and enterprise data quality frameworks
• Experience with orchestration and workflow management tools for data pipelines
• Experience supporting operational reporting, executive dashboards, trend analysis, advanced analytics, or machine learning initiatives
• Background investigation required
• Reliable internet with minimum speeds of 50 MB download and 5 MB upload
• Dedicated, private workspace free from distractions with appropriate desk and seating
• Allstate generally does not sponsor individuals for employment-based visas for this position

Benefits:

• Comprehensive technology setup including a laptop, monitors, headset, keyboard, and mouse
• Monthly connectivity reimbursement for employees eligible to work from home