Senior Big Data Engineer

Department: Engineering
Specialization: Data Engineer
Experience: Senior
Technologies: AWS Big Data GCP Jenkins Spark
Locations: USA
Special referral bonus: No
Hot position?: Hot
Technical flow: Data Engineer
Engineering technical flow: Data Engineer
Non-engineering technical flow: none
  • What you will do

  • Work collaboratively with other engineers, architects, data scientists, analytics teams, and business product owners in an agile environment
  • Architect, build, and support the operation of Cloud and On-Premises enterprise data infrastructure and tools
  • Design and build robust, reusable, and scalable data driven solutions and data pipeline frameworks to automate the ingestion, processing and delivery of both structured and unstructured batch and real-time streaming data
  • Build data APIs and data delivery services to support critical operational processes, analytical models and machine learning applications
  • Assist in selection and integration of data related tools, frameworks, and applications required to expand platform capabilities
  • Understand and implement best practices in management of enterprise data, including master data, reference data, metadata, data quality and lineage
  • Must haves

  • Bachelor’s degree in computer science, information systems, math, engineering, or other technical field, or equivalent experience
  • 6 years of experience with Python or Java
  • 4+ years of experience in building data lake, cloud data platform leveraging cloud (GCP/AWS) cloud native architecture, ETL/ELT, and data integration
  • Three years of development experience with cloud services ( AWS,GCP,AZURE) utilizing various support tools (e.g. GCS, Dataproc, Cloud Data flow, Airflow(Composer), Kafka , Cloud Pub/Sub)
  • Expertise in developing distributed data processing and Streaming frameworks and architectures (Apache Spark, Apache Beam, Apache Flink, )
  • In-depth knowledge of NoSQL database technologies (e.g. MongoDB, BigTable, DynamoDB)
  • Expertise in build and deployment tools – (Visual Studio, PyCharm, Git/Bitbucket/Bamboo, Maven, Jenkins, Nexus, )
  • 5 years of experience and expertise in database design techniques and philosophies (e.g. RDBMS, Document, Star Schema, Kimball Model)
  • 5 years of experience with integration and service frameworks (e.g API Gateways, Apache Camel, Swagger API, Zookeeper, Kafka, messaging tools, microservices)
  • Expertise with containerized Microservices and REST/GraphQL based API development
  • Experience leveraging continuous integration/development tools (e.g. Jenkins, Docker, Containers, OpenShift, Kubernetes, and container automation) in a Ci/CD pipeline
  • Advanced understanding of software development and research tools
  • Attention to detail and results-oriented, with a strong customer focus
  • Ability to work as part of a team and independently
  • Analytical and problem-solving skills
  • Problem-solving and technical communication skills
  • Ability to prioritize workload to meet tight deadlines
AgileEngine is one of the Inc. 5000 fastest-growing companies in the US and a top-3 ranked dev shop according to Clutch. We create award-winning custom software solutions that help companies across 15+ industries change the lives of millions.If you like a challenging environment where you’re working with the best and are encouraged to learn and experiment every day, there’s no better place — guaranteed! 🙂

About the project

The benefits of joining us

Professional growth

Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps

Competitive compensation

We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities

A selection of exciting projects

Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands

Flextime

Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive.

Your AgileEngine journey starts here

1

Test task

We will review your CV and send you a test task via email

2

Intro Call

Our recruitment team will reach you to discuss available opportunities

WFH or a comfy office? Why not both?

International Projects

3

Technical Interview

You will have an interview with your future team lead

Our geography

UTC-5
WASHINGTON DC USA
UTC-5
MIAMI USA
UTC-6
MEXICOMexico
UTC-5
ColombiaColombia
UTC-3
BrazilBrazil
UTC-3
ArgentinaArgentina
UTC+2
UkraineEurope
UTC+1
PolandEurope
UTC+0
PortugalPortugal
UTC+5:30
IndiaIndia

About AgileEngine

Founded as a dev tool vendor with a 2-person team

2010

Opened a dev center in Ukraine

2012

Pivoted into outsourced product development

2014

Launched mobile and UI labs

2015

Got our first Inc. 5000 award

2016

Opened a dev center in Argentina

2017

Became a top-3 ranked custom software developer in DC, Ukraine, and Argentina

2019

Became the #1 software development company to hire in 2020

2020

Opened new dev centers in Mexico and Colombia, counting 500+ experts

2021

How we lead

A company where experts grow, hone their skills, and do what they love, AgileEngine is guided by these principles:

Stay agile and embrace changes

Thrive in a results-driven culture with individual autonomy

Innovate with fellow experts in a no-blame environment

Learn from mistakes and move on

Foster mutual trust and support

Our geography

UTC-5
WASHINGTON DC USA
UTC-5
MIAMI USA
UTC-6
MEXICOMexico
UTC-5
ColombiaColombia
UTC-3
BrazilBrazil
UTC-3
ArgentinaArgentina
UTC+2
UkraineEurope
UTC+1
PolandEurope
UTC+0
PortugalPortugal
UTC+5:30
IndiaIndia

Apply for this position

Allowed Type(s): .pdf, .doc, .docx