SR. DATA ENGINEER

Job Overview

We are looking for a highly skilled Data Engineer to join our team and help build the next-generation analytics platform. The successful candidate will be responsible for designing, developing, and maintaining robust ETL solutions, working with complex data models, and creating data visualizations to support data-driven decision-making. This role requires expertise in both on-prem and cloud ESB architectures, with a focus on scalability, reliability, security, and performance.

Job Responsibilities

ETL & Data Warehouse Solutions:

• Design and develop ETL solutions using data warehouse best practices to support our Next Generation Analytics platform.
• Analyze business requirements, source data, and data models to determine the best methods for extracting, transforming, and loading data into data staging areas, warehouses, and other system integrations.
• Develop and deploy ETL job workflows with reliable error/exception handling and rollback strategies.
• Continuously adapt and optimize ETL code and jobs to accommodate changes in source data and evolving business requirements.

Data Integration & Analysis:

• Collaborate with stakeholders to analyze data requirements and outline effective solutions.
• Integrate and process large-scale data from multiple sources, ensuring data is clean, accurate, and accessible.
• Design and implement automated data audit and validation processes to ensure data integrity.

Architecture & System Design:

• Leverage deep working knowledge of both on-prem and cloud ESB architecture to address client requirements for scalability, security, performance, and reliability.
• Provide senior technical leadership in the design, architecture, integration, and support of the data sourcing platform.
• Ensure high availability, performance, scalability, and maintainability across all data solutions.

Data Visualization:

• Create and maintain interactive dashboards and reports using PowerBI, Tableau, or similar tools.
• Translate complex business requirements into clear, actionable data visualizations.
• Present data findings to both technical and non-technical stakeholders, ensuring effective communication of insights.

Database Management & Optimization:

• Manage and optimize databases to ensure efficient data storage and retrieval.
• Troubleshoot job errors, identify issues with job windows, and assist with database performance tuning and backups.

Collaboration & Communication:

• Work closely with data scientists, business analysts, and other cross-functional teams to translate business needs into data solutions.
• Proactively communicate innovative ideas and solutions, contributing to the continuous improvement of data processes.
• Provide support for testing, system integration, and transitions to support teams.

Documentation & Best Practices:

• Document all ETL processes, data flows, and system architecture.
• Create and update technical documentation to ensure smooth transitions and long-term maintainability.
• Develop and adhere to standards and procedural practices for ETL and data management processes.

Data Quality Assurance:

• Ensure data quality throughout the entire ETL process, implementing audits and feedback loops to maintain the integrity of source data.
• Perform and coordinate unit and system integration testing to validate data solutions.
   

Job Requirements

Technical Skills:

• ETL Tools: Experience with tools like Apache Spark, Talend, SSIS, and other ETL frameworks.
• Data Tools: Proficiency with data visualization tools like PowerBI, Tableau, or similar.
• Databases & Querying: Strong skills in SQL and experience with relational databases (e.g., MySQL, PostgreSQL, SQL Server).
• Programming: Familiarity with Python, R, or similar languages for data manipulation and automation.
• Cloud Platforms: Experience with AWS, Azure, or Google Cloud is a plus.

Experience:

• More than 6 years of proven experience in developing and maintaining ETL processes and data pipelines.
• Familiarity with data warehouse design and management.
• Experience working with large-scale data systems, both on-prem and in the cloud.
   

Soft Skills:

• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Attention to detail with a commitment to delivering high-quality, efficient data solutions.

Competencies