Data Pipeline Engineer

Client B2B Search Engine
Category Backend
Published Sep 1, 2021
Uplink fee 3%, max. 6 months More info »
Job type Recruiter More info »

Our client who is a Berlin based startup working on a B2B search engine application, is in need of a Data Pipeline Engineer to support for a data pipeline which is implemented in microservices based on python and connected via gRPC.

They are developing their own microservices internally (9 total) but need help in a) Defining an expandable database structure to hold metadata b) Stitching together the whole pipeline.

The product works as a market intelligence software where their main aim is to collect unstructured public data about companies and connect it to their profile.

Some more details include:
- The pipeline should be implemented in python, ideally the services as well. Optionally, services could be implemented in typescript.
- Services should use gRPC for communication and docker containers for deployment.
- The pipeline should use Kafka/Faust for messages.
- They use poetry for managing packages and dependencies and protodep for proto files generation.
- They use an internal python package to handle common functions, including grpc connection.
- The current infrastructure utilizes MongoDB and Azure blobs for storage and Redis for caching.
- Deployment and CI are managed via GitHub actions to Kubernetes.

Must have skills
gRPC, Microservices, Python

Start date
Earliest: asap
Latest: Sep 20, 2021
Would ideally like to start as soon as possible

2-4 months
Likely 3 months, could be extended

Full- or parttime possible
Ideally fulltime, but might work around 30 hours a week too

Fully possible
Full remote fine

Language requirements

75-80€ per hour
It’s a startup, looking for reasonable rates
incl. Uplink fee

Onsite locations

This job is already closed and applications are therefore no longer possible.
Join Uplink now to not miss out on similar jobs in the future!

Join now