Data Pipeline Engineer

Kunde: B2B Search Engine

Kategorie: Backend
Veröffentlicht: 1. Sep 2021
Uplink-Provision: 3%, max. 6 Monate Mehr Infos »
Job-Typ: Recruiter Mehr Infos »


Our client who is a Berlin based startup working on a B2B search engine application, is in need of a Data Pipeline Engineer to support for a data pipeline which is implemented in microservices based on python and connected via gRPC.

They are developing their own microservices internally (9 total) but need help in a) Defining an expandable database structure to hold metadata b) Stitching together the whole pipeline.

The product works as a market intelligence software where their main aim is to collect unstructured public data about companies and connect it to their profile.

Some more details include:
- The pipeline should be implemented in python, ideally the services as well. Optionally, services could be implemented in typescript.
- Services should use gRPC for communication and docker containers for deployment.
- The pipeline should use Kafka/Faust for messages.
- They use poetry for managing packages and dependencies and protodep for proto files generation.
- They use an internal python package to handle common functions, including grpc connection.
- The current infrastructure utilizes MongoDB and Azure blobs for storage and Redis for caching.
- Deployment and CI are managed via GitHub actions to Kubernetes.

Must have skills
gRPC, Microservices, Python

Startdatum
Frühestens: sofort
Spätestens: 20. Sep 2021
Would ideally like to start as soon as possible

Laufzeit
2-4 Monate
Likely 3 months, could be extended

Auslastung
Voll- oder Teilzeit möglich
Ideally fulltime, but might work around 30 hours a week too

Remote
Komplett möglich
Full remote fine

Projektsprachen
English

Budget
75-80€ pro Stunde
It’s a startup, looking for reasonable rates
inkl. Uplink-Gebühr

Einsatzorte
Berlin


Dieser Job ist bereits geschlossen und Bewerbungen sind daher nicht mehr möglich.
Um in Zukunft Jobs wie diesen nicht mehr zu verpassen, melde dich jetzt bei Uplink an!

Jetzt Mitglied werden