Data Engineer (Contract)

Data EngineerOur StoryOver the past several years, Able has grown immeasurably. We’ve also grown in the type of company that we are:Chapter 1: We were founded in 2013 as a product and engineering hub for a portfolio of early-stage start-ups. We grew up as an in-house/external hybrid shared services model. That allowed us to hone our skills and establish our operational and cultural foundation.Chapter 2: In 2019 we began to expand our vision. We began to grow outside of our inset partner base. We had good initial success meeting new partners, kicking off new relationships, and delivering high-value work.Chapter 3: In 2023, we moved into the next phase of a new chapter, an expansion of the ambition of Chapter 2. Our strategy for growth centers around two audiences:Venture Capital: VC firms are looking for trusted product and technology solutions to distribute seamlessly across their portfolios at scale.Private Equity: PE firms are looking for trusted solutions that can catalyze growth for their portfolio companies at scale.Chapter 3a: We are now in the next phase of Chapter 3, aligned to our mission and vision, and accelerated by the powers of applied AI. We believe that AI will be a powerful force in the end-to-end software development lifecycle. Specifically we are creating practices that – coupled with our world class talent – can deliver software significantly faster than legacy techniques. The result is increased value for our partners, who can dramatically increase the capacity of their product organizations. About the RoleSupporting the Director of Engineering and the broader Engineering team, this Data Engineer role will work cross-functionally with teams across Able while aligning closely with the Engineering discipline. This role will partner directly with a specific client and collaborate across Product, Design, and Engineering to deliver robust and scalable data solutions that meet critical business needs.Day-to-Day ResponsibilitiesStrategic Architecture LeadershipShape large-scale data architecture vision and roadmap across client engagementsEstablish governance, security frameworks, and regulatory compliance standardsLead strategy around platform selection, integration, and scalingGuide organizations in adopting data lakehouse and federated data modelsClient/Partner Value Creation Lead technical discovery sessions to understand client needsTranslate complex architectures into clear, actionable value for stakeholdersBuild trusted advisor relationships and guide strategic decisionsAlign architecture recommendations with business growth and goalsTechnical Architecture & ImplementationDesign and implement modern data lakehouse architectures with Delta Lake and DatabricksBuild and manage ETL/ELT pipelines at scale using Spark (PySpark preferred)Leverage Delta Live Tables, Unity Catalog, and schema evolution featuresOptimize storage and queries on cloud object storage (e.g., AWS S3, Azure Data Lake)Integrate with cloud-native services like AWS Glue, GCP Dataflow, and Azure Synapse AnalyticsImplement data quality monitoring, lineage tracking, and schema versioningBuild scalable pipelines with tools like Apache Airflow, Step Functions, and Cloud ComposerBusiness Impact & Solution DesignDevelop cost-optimized, scalable, and compliant data solutionsDesign POCs and pilots to validate technical approachesTranslate business requirements into production-ready data systemsDefine and track success metrics for platform and pipeline initiativesWhat We’re Looking ForThe ideal candidate will have:10+ years of data engineering experience with enterprise-scale systemsExpertise in Apache Spark and Delta Lake, including ACID transactions, time travel, Z-ordering, and compactionDeep knowledge of Databricks (Jobs, Clusters, Workspaces, Delta Live Tables, Unity Catalog)Experience building scalable ETL/ELT pipelines using tools like Airflow, Glue, Dataflow, or ADFAdvanced SQL for data modeling and transformationStrong programming skills in Python (or Scala)Hands-on experience with data formats such as Parquet, Avro, and JSONFamiliarity with schema evolution, versioning, and backfilling strategiesWorking knowledge of at least one major cloud platform:AWS (S3, Athena, Redshift, Glue Catalog, Step Functions)GCP (BigQuery, Cloud Storage, Dataflow, Pub/Sub) – nice to haveAzure (Synapse, Data Factory, Azure Databricks) – nice to haveExperience designing data architectures with real-time or streaming data (Kafka, Kinesis)Consulting or client-facing experience with strong communication and leadership skillsExperience with data mesh architectures and domain-driven data designKnowledge of metadata management, data cataloging, and lineage tracking toolsFamiliarity with healthcare standards (e.g., HL7, FHIR, DICOM) is a plusAwareness of international data privacy regulations and compliant system designNice to have:Master's degree in Computer Science, Data Engineering, or related fieldML Ops experience or integrating machine learning