Data Engineer (Lead) ID42105

• What you will do

• Provide BAU (Business-As-Usual) support for existing data products;
• Redesign and optimize existing workflows;
• Implement enhancements and new features using Python/Spark and React;
• Maintain clean, scalable, and well-documented code;
• Automate repetitive production scripts for BAU and remediation;
• Build dashboards tracing data flow from ingestion to distribution;
• Support release readiness activities (dry runs, pre-release validation, regression runs);
• Conduct post-release verification with business stakeholders;
• Suggest and implement platform and process improvements proactively;
• Document all work in Jira and Confluence;
• Participate in Agile ceremonies and scrums;
• Support knowledge transfer within the team.

• Must haves

• 7–12 years of experience;
• Strong proficiency in Python (Pandas, PySpark);
• Experience in data engineering and workflow optimization;
• Hands-on experience with Delta Tables and Parquet;
• Ability to take end-to-end ownership of features and tasks;
• Strong problem-solving and decision-making skills;
• Clear communication with distributed teams (India/US time zones);
• Experience working in Agile environments with a proactive improvement mindset;
• Upper-intermediate English level.

• Nice to haves

• Experience with Databricks;
• Familiarity with Apache Spark, DBT, Airflow;
• Knowledge of advanced Pandas optimizations;
• Experience with PyTest or DBT testing frameworks;
• Understanding of ESG or financial systems.

AgileEngine is one of the Inc. 5000 fastest-growing companies in the US and a top-3 ranked dev shop according to Clutch. We create award-winning custom software solutions that help companies across 15+ industries change the lives of millions.

If you like a challenging environment where you’re working with the best and are encouraged to learn and experiment every day, there’s no better place — guaranteed! 🙂