Job Description & Requirements
Be a Part of Something BIG!
- Design, build, productionize, and support scalable document parsing and content extraction pipelines on a modern hybrid cloud platform, handling structured and unstructured data including PDFs, PPTX, Excel, Word, images, scanned documents, videos, audios, charts, tables etc
- Design and implement OCR, VLM, and multimodal AI pipelines leveraging state-of-the-art models (e.g., GPT- [4/5]o, vision-language models, Text-to-Speech) for accurate content extraction, layout understanding, and table/chart interpretation
- Develop and optimize embedding models, including fine-tuning and domain adaptation to support organization-specific terminology, keywords, and semantic search use cases
- Implement knowledge ingestion, indexing, and retrieval pipelines for RAG (Retrieval-Augmented Generation) and agentic GenAI systems, ensuring high recall, precision, and low-latency retrieval
Make an Impact by:
- Build and maintain data ingestion pipelines for batch and streaming data sources using tools/library like Kafka, LlamaIndex, LangChain, PyMuPDF, MSAL etc.
- Implement layout-aware parsing to accurately preserve document structure, hierarchy, and reading order (using OCR, VLM, and multimodal AI solutions).
- Extract accurate numeric and semantic data from complex charts and irregular tables while preventing hallucinations.
- Assist in integrating data from diverse source systems (files, APIs, databases, streaming)
- Implement, fine-tune, and optimize embedding models to support organization-specific terminology, improving semantic search, retrieval accuracy, and downstream GenAI performance.
- Help maintain metadata and pipeline documentation for transparency and traceability
- Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform components
- Build and maintain knowledge base and RAG solution on cloud (Azure)
- Implement and operate knowledge base storage, lifecycle management and embedding/vectorization
- Contribute to automation efforts using version control and CI/CD workflows
- Apply basic data governance and access control policies during implementation.
Skills for Success:
- Bachelor’s degree in Computer Science, Engineering, or a related field
- 1–3 years of experience in data engineering or data platform development. Fresh graduates encouraged to apply as well.
- Proven ability to independently build basic batch or streaming document processing pipelines
- Hands-on experience with Python, PyTorch, PEFT, LoRA, OCR, VLM, LLM, prompting, Text-to-Speech models and SQL for data transformation and validation
- Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts