Senior Data Engineer

 Our customer is one of the world's largest technology companies based in Silicon Valley with operations all over the world. On this project, we are working with bleeding-edge big data technologies to develop a high-performance data analytics platform, which handles petabytes of data. We are looking for an enthusiastic and technology-proficient Senior Data Engineer, who is eager to participate in design and implementation of a top-notch big data solution that will be deployed at a massive scale.  

Responsibilities :  

• Build data pipelines to enable training of LLMs
• Work with technical and non-technical stakeholders to collect requirements, integrate with data systems and deliver large-scale datasets
• Operate in a complex multi-cloud data lakehouse, building end-to-end integrated workflows
• Work with privacy, security, and other policy stakeholders to design, review and implement solutions to automate compliant ETL for downstream teams

Requirements :  

• Strong understanding of machine learning principles, especially in the context of LLMs.
• 5+ years of experience in big data engineering
• Proficiency in Java, Scala, Spark
• Proficiency in PySpark and Golang(preferred)
• Experience building and leveraging large scale data infrastructure, map-reduce/hadoop , cloud-native deployments
• Experience with alerting, monitoring and remediation automation in a large scale distributed environment
• BS/BA or equivalent degree in computer science or similar (preferred)

• Opportunity to work on cutting-edge projects
• Work with a highly motivated and dedicated team
• Competitive salary
• Flexible schedule
• Full benefits including but not limited to medical, dental, vision
• Corporate social events
• Professional development opportunities