remote
Senior Software Engineer - Agentic Search Crawler - Nebius
Software Engineer
Lead the design and implementation of a high‑performance crawler for an AI‑powered search platform, leveraging Python, Node.js, and AWS services to ingest, process, and index massive data streams at scale.
About the role
Key Responsibilities
- Architect and develop a distributed crawler capable of handling billions of web pages, ensuring low latency and high throughput.
- Integrate crawling pipelines with AWS services (S3, Lambda, DynamoDB) and container orchestration (ECS/EKS) for scalable deployment.
- Collaborate with data scientists to refine search relevance models and feed real‑time indexing updates.
- Implement robust error handling, retry logic, and monitoring dashboards using Prometheus/Grafana.
- Optimize resource usage through containerization, auto‑scaling, and cost‑effective GPU/CPU allocation.
Requirements
- 5+ years of software engineering experience with a focus on large‑scale distributed systems.
- Proficiency in Python and Node.js, with solid knowledge of asynchronous programming.
- Hands‑on experience deploying applications on AWS, including Lambda, ECS/EKS, and S3.
- Strong background in containerization (Docker) and orchestration (Kubernetes).
- Excellent problem‑solving skills and a passion for building production‑grade, scalable services.
Skills
pythonnodejsawsdockerkubernetes