About the roleWe are looking for a Backend AI & Data Pipeline Engineer to own the end-to-end data processing infrastructure that powers Yuzee's intelligent course and job matching platform. You will design and maintain scalable, event-driven pipelines that process tens of thousands of daily records, generate semantic embeddings, and feed a growing knowledge graph used for personalised career pathway recommendations.What you'll doDesign and maintain three distinct processing pipelines — scheduled job ingestion, event-driven course processing, and a periodic knowledge graph builder — each with independent trigger logic and cost controlsGenerate and manage semantic embeddings via Amazon Bedrock (Titan v2), index them in MongoDB Atlas Vector Search, and calibrate similarity thresholds to ensure match accuracyBuild and maintain a knowledge graph linking jobs, courses, skills, and industries using FP-Growth association rules and archetype-to-SOC code mappingBuild and improve a two-stage discovery and matching API on AWS Lambda — vector retrieval first, then deep eligibility scoring with LLM re-rankingRight-size Fargate Spot instances and design resumable processing loops that tolerate interruption, keeping infrastructure costs under control as data volume scalesMaintain and improve daily job scrapers across multiple sources and build institution data scrapers with robust HTML cleaning pipelinesWhat we're looking for1+ years of backend engineering experience focused on data pipelines, ML infrastructure, or search systemsHands-on experience with AWS serverless and container services — Lambda, ECS Fargate, EventBridge, and Step FunctionsStrong Python skills — Pandas, async processing, bulk database operations, and text cleaningFamiliarity with vector databases and semantic similarity search; MongoDB Atlas Vector Search experience is a strong plusCost-conscious infrastructure mindset — you think in per-record compute costs, free tiers, Spot resilience, and right-sizingAbility to document and communicate complex architecture clearly to both technical and non-technical stakeholdersNice to haveExperience with knowledge graphs or association rule mining (FP-Growth, Apriori)Experience using LLMs for re-ranking or eligibility assessment on top of vector retrieval resultsBackground in edtech, jobtech, or recommendation/matching systems
