Senior Databricks Engineer

Job Description:

• Design, develop, and deploy scalable data infrastructure
• Drive key data engineering initiatives using the Databricks platform and modern data warehousing
• Build data pipelines supporting the Department of State’s “4Ps” data lake
• Work with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs
• Architect and implement scalable, highly available data pipelines using Databricks and the Medallion Architecture
• Establish data quality standards and implement monitoring dashboards
• Write production-ready PySpark and SQL code for complex ETL/ELT processes
• Develop DLT and structured streaming pipelines
• Design and build data ingestion pipelines using RESTful API integrations
• Implement infrastructure and pipelines as code
• Automate deployments across Development, Staging, and Production using Databricks Asset Bundles and Azure DevOps
• Develop, package, and maintain custom Python libraries and modules
• Apply data warehousing concepts, including dimensional modeling and slowly changing dimensions
• Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs
• Share data securely with external BI tools such as PowerBI and SAP BusinessObjects
• Develop dashboards and capabilities natively within the Databricks environment
• Serve as a technical SME, conduct code reviews, establish best practices, and mentor staff members

Requirements:

• Active SECRET clearance
• Knowledge of Object Oriented Programming (OOP) concepts
• Proficiency in one or more programming or scripting languages: Java, C#, Python, JavaScript, or PowerShell
• 8+ years of dedicated Data Engineering experience
• At least 3+ years functioning in a senior capacity focused on the Databricks ecosystem
• Expert-level proficiency in PySpark and complex SQL
• Proven ability to create, package, and manage custom Python libraries, including building Wheel files
• Deep understanding of HTTP REST methods such as GET, POST, and PUT
• Hands-on experience extracting, paginating, and processing data from external API sources
• Deep theoretical and practical understanding of modern Data Warehousing concepts
• Hands-on experience building out the Medallion Architecture
• Proven experience automating Databricks deployments
• Hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps, including creating YAML pipelines, managing service principals, and handling multi-workspace deployments
• Strong familiarity with Azure Data Lake Storage Gen2 and Azure Key Vault
• Proficiency with Git-based version control workflows
• Concise and comprehensive written and oral communication skills for technical understanding and customer usage support
• Desired: Experience with Tableau, PowerBI, Jupyter Notebooks, d3js, Elasticsearch, SOLR, and Lucene
• Desired: Experience working with HR application (PeopleSoft) data for reporting purposes
• Desired: 3+ years of .NET, Java, or similar OOP experience
• Desired: Experience writing Oracle PL/SQL queries, packages, and procedures
• Desired: Experience architecting Data Vault 2.0 solutions

Benefits:

• Medical, dental, and vision insurance
• 401(k) retirement plan
• Paid time off
• Paid parental leave
• Life and disability insurance
• Flexible spending accounts
• Commuter benefits
• Tuition reimbursement