What you will do
- Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP, including BigQuery-based lakehouse layers and Dataproc-driven Delta Lake workflows;
- Actively participate in meetings with various stakeholders across data engineering, compliance, and business teams globally;
- Understand market data processing and transformation needs; build pipelines to acquire, normalise, transform, and release large volumes of financial data through the data factory;
- Design and implement bitemporal data models (valid-time + system-time) on BigQuery to support certified, regulatory-grade time-series datasets;
- Build, use, and maintain software testing frameworks (unit / non-regression / user acceptance) for data pipelines and transformation logic;
- Take complete ownership of solutions and assigned tasks, including ingestion pipelines, QA workflows, correction management, and audit trail implementation;
- Work in a collaborative manner with other team members and contribute to shared platform services rather than vertical-specific implementations;
- Have business acumen to understand financial concepts around reference data related to equities and other asset classes;
- Support teams across data and technology in implementing AI solutions and integrating their services with data science products and platforms, including AI-assisted ingestion, anomaly detection, and semantic search over the lakehouse using Vertex AI.
Must haves
- 6-8 years of experience in data engineering;
- Proficient in Python programming — data pipeline development, transformation logic, and automation scripts;
- Proficient in data query and analysis using SQL, with strong hands-on experience in BigQuery — partitioning, clustering, materialised views, and time-series query patterns at scale;
- Hands-on experience building and scheduling pipelines using Cloud Composer (Apache Airflow) — DAG authoring, SLA alerting, retry logic, and dependency management;
- Working knowledge of Dataproc (Apache Spark) — batch ingestion, Delta Lake merge operations, and incremental data processing;
- Proficient in AI-assisted development tools such as GitHub Copilot, Cursor, or others for accelerating code generation and enhancing developer productivity;
- Code versioning and collaboration using Git — branching strategies, pull request workflows, and pipeline-as-code practices;
- Familiarity with REST APIs — consuming external data vendor APIs and building service-layer integrations;
- Familiarity with GCP cloud technologies — Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, and VPC Service Controls;
- Upper-intermediate English level.
Nice to haves
- Basic knowledge of data manipulation and analysis libraries — pandas, PySpark, or equivalent;
- Basic knowledge of columnar storage, SQL-based querying, and time-series analytics (ClickHouse or equivalent);
- Familiarity with Dataplex for data discovery, lineage, policy tagging, and data quality rule management;
- Understanding of Change Data Capture (CDC) patterns using Datastream for replicating transactional data into BigQuery;
- Understanding of bitemporal data modeling concepts (valid-time and system-time) and the challenges of implementing them within BigQuery’s append-optimised design;
- Understanding of financial reference data — equities, fixed income identifiers, corporate actions, or index composition data;
- Familiarity with BigQuery cost management — slot reservations, query cost controls, and workload isolation using reservations and assignments;
- Exposure to CI/CD pipelines and infrastructure-as-code using Terraform for data platform deployments on GCP;
- Prior experience or projects involving LLMs and Agentic AI — particularly using Vertex AI for AI-assisted data quality, anomaly detection, semantic search, or natural language querying over structured datasets — is a strong plus.
We are looking for a Senior Data Analyst to design and maintain enterprise dashboards and reporting solutions using Tableau, while performing advanced analysis on complex datasets including compensation and HR data. You will use Python and SQL to clean, transform, and analyze large datasets in Snowflake, translating business questions into clear, actionable insights for both technical and non-technical stakeholders. The role requires strong independent judgment and the ability to navigate evolving, ambiguous business requirements.
About the role
The benefits of joining us
Professional growth
Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps
Competitive compensation
We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities
A selection of exciting projects
Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands
Flextime
Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.
Your AgileEngine journey starts here
2 min
Tell us about yourself
2 sec
Confirm requirements
30 - 60 min
Pass a short test
5 min
Record a short video
→ Introduce yourself on a video, instead of waiting for an interview
Live interview
Ace the technical interview with our team
→ Schedule a call yourself right away after your video is reviewed
Live interview
Final interview with your team
→ Get to know the team you will be working with
Get an offer
As quick as possible
