Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.

Job Summary

For this role, we are seeking a Senior Data Engineer for our Client’s ETL Support Team to act as a technical pillar during our Client’s transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client’s systems by driving performance tuning, cost optimization, and automation initiatives.

Work Location: Noida, Sec 62

Working days per week: 5

Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week

Responsibilities:

• Job Monitoring: Monitor approximately 2,300 scheduled jobs (daily, weekly, monthly) to ensure timely and successful execution. Execute on-demand jobs as required by the business.
• Troubleshooting & RCA: Troubleshoot job failures, perform detailed root cause analysis (RCA), and provide clear documentation for all findings.
• Issue Resolution: Address and resolve bugs and data-related issues reported by the business team.
• Data Integrity: Verify source file placement in designated directories to maintain data integrity. Reload Change Data Capture (CDC) tables when structural changes occur in source systems.
• Data Synchronization: Help manage synchronization between external databases (including Teradata write-backs) and AWS Glue tables.
• Development: Assist in developing new solutions, enhancements, and bug fixes using AWS Glue, Lambda, and Step Functions.
• Stakeholder Support: Answer questions from the business and support User Acceptance Testing (UAT) inquiries.
• Decision Making: Make timely decisions to resolve issues, execute tasks efficiently, and escalate complex problems to senior or lead engineers as needed, all while maintaining agreed-upon SLAs.
• Documentation: Participate in knowledge transfer to internal resources and assist in creating necessary documentation and runbooks.

Qualifications:

• Experience: 3-5 years of relevant experience.
• Technical Skills: In-depth knowledge of AWS Glue, AWS Lambda, and AWS Step Functions.
• Core Concepts: A deep understanding of ETL processes and data warehouse design.
• Problem-Solving: Proven ability to troubleshoot data pipelines and perform root cause analysis (RCA).
• Should have worked on Python scripting
• Experience: Hands-on experience with Glue, Lambda, and Step Function development.
• Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
• Preferred Skills: Familiarity with Change Data Capture (CDC) processes, Teradata, and integration with SAS & RMJ events.