DATA ENGINEER II

Job Overview

• Develop and maintain data pipelines for efficient extraction, transformation, and loading (ETL) processes.
• Design and optimize data storage solutions, including Amazon Redshift data warehouses and Amazon S3 data lakes.
• Ensure data quality and integrity through validation, cleansing, and error handling processes.
• Collaborate with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets for business intelligence.
• Implement data security measures and access controls to protect sensitive information.
• Automate and improve data processes and workflows for scalability and efficiency.
• Monitor data infrastructure performance and address any issues promptly.
• Stay updated on industry trends and emerging technologies in data engineering.
• Document data pipelines, processes, and best practices to facilitate knowledge sharing.
• Continuously optimize data architecture to support the company's evolving data needs.
• Work cross-functionally to drive data-driven decision-making within the organization.

What your background should look like:

• BE(CS) with 5 years of experience in data engineering.
• Knowledge of AWS services, particularly Glue, Redshift and S3.
  • Data Storage and Management - Amazon S3, Amazon RDS and DynamoDB
  • Data Processing and ETL - AWS Glue
  • Data Warehousing - Amazon Redshift
  • Programming and Scripting:
    • Proficiency in languages like Python, Java, or Scala for data manipulation and automation.
  • Data Modeling and Architecture
  • Monitoring and Optimization - Using AWS CloudWatch for monitoring and optimizing data workflows.
• Proficiency in SQL and database management.
• Experience with ETL (Extract, Transform, Load) processes.

Competencies