Data Engineer (Senior) ID53687

Department: Engineering
Specialization: Data Engineer
Experience: Senior
Technologies: Apache Spark Java
Client: Indeed
Technical flow: Data Engineer Java
Engineering technical flow: Data Engineer Java
Non-engineering technical flow: none
  • What you will do

  • Develop and maintain index builder products, including user session index builders, user session–derived index builders, and experimentation platform index builders;
  • Investigate and resolve reported issues related to index builders;
  • Assist with user inquiries regarding the platform and its datasets;
  • Improve index builder stability and reliability;
  • Support efforts to optimize compute costs across the platform;
  • Contribute to the Central Exposure Dataset effort, including building a consolidated dataset for experiment analysis;
  • Work toward meeting code freshness goals;
  • Persist Yarn logs and Spark history for terminated clusters;
  • Capture metrics from UserCohort;
  • Optimize resource allocation for platform infrastructure;
  • Help reduce the number of core instances for platform clusters;
  • Support the deprecation of legacy index builders used for experiment analysis.
  • Must haves

  • 4+ years experience in software development;
  • Bachelor’s degree in Computer Science or equivalent practical experience;
  • Significant practical experience with Java (4+ years);
  • Practical experience implementing Apache Spark jobs, including partitioning, grouping, joins, importing data into the cluster, and exporting data from the cluster;
  • Practical experience working with AWS, specifically AWS EMR (or ability to pick it up fast);
  • Upper-intermediate English level.
  • Nice to haves

  • Basic knowledge of Kubernetes;
  • Experience with Spark Operator;
  • Experience with Airflow;
  • Experience with Scala.

AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.

If you’re looking for a place to grow, make an impact, and work with people who care, we’d love to meet you!

About the project

The benefits of joining us

Professional growth

Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps

Competitive compensation

We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities

A selection of exciting projects

Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands

Flextime

Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.

Your AgileEngine journey starts here

1

2 min

Tell us about yourself

2

2 sec

Confirm requirements

3

30 - 60 min

Pass a short test

4

5 min

Record a short video

→ Introduce yourself on a video, instead of waiting for an interview

5

Live interview

Ace the technical interview with our team

→ Schedule a call yourself right away after your video is reviewed

6

Live interview

Final interview with your team

→ Get to know the team you will be working with

7

Get an offer

As quick as possible

Our geography

UTC-5
WASHINGTON DC USA
UTC-5
MIAMI USA
UTC-6
MEXICOMexico
UTC-5
ColombiaColombia
UTC-3
BrazilBrazil
UTC-3
ArgentinaArgentina
UTC+2
UkraineEurope
UTC+1
PolandEurope
UTC+0
PortugalPortugal
UTC+5:30
IndiaIndia

Apply for this position

Allowed Type(s): .pdf, .doc, .docx