Data Engineer

Reporting to the Manager, BI Development, the Data Engineer will be responsible for integrating enterprise wide data from Cofense’s Cyber Security based Applications, Micro-services and other disparate data sources. Essential Duties/Responsibilities include designing Data Platform and Architecture, developing Data Pipelines on Cloud Technologies, and securing data end-to-end by complying data privacy rules.RequirementsSubstantial experience with SQL and no-SQL OLTP databases and OLAP data warehousing technologies, especially AWS AuroraExperience data modeling and building data pipelines for multi-product and/or multi-department organizationsDevelop Data Pipelines on Cloud Technologies like Azure/AWS with well-defined tool frameworksAble to develop ETL code to stream data from disparate (structured and semi-structured) SaaS product data stores to Data Lake/Data Warehouse using Python, Azure/AWS Data Lake servicesAbility to write complex SQL scripts and automate them using PythonDevelop test cases and unit tests for key implementations of Data Platform by adhering to software engineering best practices and standardsSecure data end-to-end by complying data privacy rules while developing processes to move data across Applications/Data Sources and Data Lake/Data Warehouse, as well as while delivering data through SQL clients and BI tools.Experience building adhoc reporting APIs and service layer on top of underlying OLTP and OLAP databasesHelp integrate Data Platform with BI tools like Power BI, Tableau, Splunk etc.Ability to develop and interpret Entity Relationship Diagrams (ERD) across data sets in relational database systems as well as non-relational Data storesAble to do Data Mining and Identify trends, patterns, anomalies in complex data sets across multiple data sources/systems and present results without ambiguityDevelop data transformations to generate Facts, Summaries, Key metrics by applying business rulesets and aggregations using Python, SQL and other transformation toolsAble to review current processes related to data ingestion, transformation and statistical analysis and re-engineer themCollaborate with business users across Cofense’s departments in defining requirements, prioritize project work and deliver them timelyExpertise in SQL skills for data transformations, statistical analysis, and troubleshooting across more than one Database Platforms (MySQL, PostgreSQL, Redshift, Azure SQL Warehouse etc.)Expert in writing complex SQL scripts and automate them using PythonKnowledge of Data management on NoSQL DBs like DynamoDB, Mongo, and know-how of Big Data tools Hadoop, Spark, Kafka/Kinesis/SQS/Azure Queues or other messaging tools is huge plusAnalytical skills, with good at finding data trends/outliers, anomalies, and articulate complex information or data points with Business Users, Management, and individualsEnthusiasm to work with lot of data across disparate data sources and DatabasesHas strong sense of engineering craftsmanship, takes pride in the code they write.Has a sense of intellectual curiosity and a burning desire to learn is self-driven, actively looks for ways to contribute, and knows how to get things doneIs deliriously customer-focused both internal and external customersSees big picture impact and relationships among and across work unitsIdentifies complex technical problems and tries to resolve with minimal helpBenefits401k MatchingRetirement PlanTuition ReimbursementOriginally posted on Himalayas