Navin Kumar

AI Engineer building agent systems that don't fail quietly

About

I build LLM systems and the machinery that keeps them honest. That means agentic orchestration that runs unattended, retrieval that actually grounds an answer, and evaluation harnesses that tell you when a model has quietly gotten worse.

Most of my work starts from the same observation: ML systems rarely fail with a crash. They fail quietly. A recommendation drifts across sessions, a detector's per-class recall collapses after a retrain, an agent reasons confidently from a false premise. So I spend as much time on benchmarks, failure analysis, and supervision as I do on the models themselves. CSE at IIIT Vadodara, currently shipping agents in production.

Experience

AI Engineer

MAY 2025 –

DigitalxCode

Built an agentic stock intelligence platform on GPT-4o with a custom MCP orchestration layer. Four discrete tools (bucket generation, RAG retrieval, LLM health validation, DB write) autonomously produce investment signals across 24 equity buckets daily with no human in the loop. Added a ChromaDB + text-embedding-3-small memory pipeline so the agent reasons about portfolio drift across sessions instead of repeating itself.

Shipped it as a containerised 5-service system on GCP Compute Engine, running FastAPI, Next.js, PostgreSQL, Redis and ChromaDB via Docker Compose, with APScheduler handling nightly generation and comparison jobs behind a live dashboard.

Separately, fine-tuned an RT-DETR detector on VisDrone (10 classes) for real-time drone perception: 90%+ mAP across 5,000+ frames, latency cut to under 33 ms (30+ FPS) through optimization, batching, and runtime profiling. Built the benchmark suite that regressed model quality version over version, and ran failure analysis on misclassified samples to target the next retraining cycle.

  • GPT-4o
  • MCP
  • ChromaDB
  • RT-DETR
  • PyTorch
  • FastAPI
  • GCP

Software Development Engineer Intern

JAN 2025 – MAY 2025

EaseMyGrowth

Owned a time-series forecasting service end to end, covering exploration, model selection, Dockerized packaging, GCP deployment, and integration into a client-facing dashboard. Built reproducible pipelines over 10,000+ row datasets with Pandas and SQL, and integrated LLM workflows producing structured outputs for downstream BI consumers.

  • Time-series
  • Pandas
  • Docker
  • GCP

Software Development Engineer Intern

MAY 2024 – JUL 2024

EaseMyGrowth

Used BERT for sentiment analysis over platform reviews, classifying them positive / negative / neutral so the team could read customer feedback at volume instead of sampling it. Wrote a Python scraper to pull Amazon product reviews as an additional corpus for that analysis and for market research, and worked alongside computer-vision projects that shaped how I approach data processing today.

  • BERT
  • NLP
  • TensorFlow
  • Python
  • Docker
  • GCP

Independent Builder, Agent Systems

2025 –

Hackathons & open source

Four shipped agent systems across the OpenAI long-range-autonomy and SitRep Future of Work hackathons: a circuit breaker for long-running agents, a post-meeting research agent, a telemetry correlation engine, and a RAG résumé optimizer. Each was built end to end, from engine to evaluation to a deployed interface.

Agentic incident investigation Recorded demo

IncidentForge

Evidence before confidence

A successful response can hide a duplicate charge. A restart can leave inventory reserved indefinitely. Built solo for the Nebius x NVIDIA Global AI Hackathon, IncidentForge investigates both failures across synthetic Order, Inventory and Payment services, compares hypotheses, and proposes bounded source repairs. NVIDIA Nemotron runs through Nebius Token Factory; isolated Sandboxes execute candidates and a separate verifier checks durable business effects.

The public viewer replays retained investigations, diffs and protected validation. It does not launch live inference. The frozen campaign retains all 12 attempts: 2 accepted, 2 rejected and 8 unresolved. Infrastructure timeouts prevent variant ranking; later successful examples do not establish repeat repair reliability. Every patch remains subject to human review.

  • Python
  • NVIDIA Nemotron
  • Nebius Token Factory
  • Sandboxes
  • Docker
  • SQLite
Agent reliability Live

AgentFuse

A logical circuit breaker for long-range agents

Long-running agents don't crash. They drift. AgentFuse sits above the execution graph, watches the telemetry every framework already emits, and trips a breaker the moment a failure mode crosses a threshold: an infinite tool loop, a slow slide off the original objective, a logical trap where the model reasons flawlessly from a false premise, or a budget quietly burning to zero. On a trip it freezes state and steers the agent back with a deterministic recovery ladder, which measured better than letting a reasoning model write the correction. When recovery isn't safe it escalates to a human instead of blindly retrying. The premise throughout: an agent that has drifted is the worst possible judge of whether it drifted. One engine, three runtimes.

  • OpenAI AgentKit
  • LangGraph
  • Python
  • Observability
Retrieval Live

Relevant Section Identification

Which pages of this PDF answer my question?

Give it a PDF and a natural-language query and it returns the pages and section headings that answer it. Prose, tables and figures take three separate chunking paths so structure survives, then EmbeddingGemma vectors fuse with BM25 at 0.6. Evaluated on 45 graded queries with 12 held out: nDCG@5 0.645 to 0.728, hit@3 82% to 93%.

The interesting part is what did not work. HyDE, cross-encoder reranking and metadata scoring were each built and measured, and each lost ground. Every point came instead from repairing what the index contained: reading tables from the embedded text layer rather than OCRing a picture of them, and stripping the running header that made all 40 pages match a query equally. Retrieval quality here was never a ranking problem, it was an extraction problem wearing a ranking costume.

  • EmbeddingGemma
  • ChromaDB
  • BM25
  • unstructured
  • pdfplumber
  • Cloud Run
Generative evaluation Live

Structured-Context Ad Generation

An evaluator you can actually check

Built for the G2 AI Engineering Hackathon. A product photo, a geography, a season and a line of copy go in; a 1024×1024 ad from Gemini comes out with a verdict on whether it honoured the request. The hard half is proving the checker catches a redrawn logo, a contradicted season, or 20% off rendered as 30% off.

On fixtures with ground truth known by construction it caught 5 of 6 defects with zero false accepts, abstaining on the sixth. It also surfaced a real weakness, rupee prices on non-Indian campaigns caught once in three, which the writeup reports as unresolved. 251 tests run offline.

  • Gemini image API
  • Python
  • Evaluation
  • Golden dataset
  • Offline replay
Adversarial ML Live

AI Defense Lab for Payment Security

Red team and blue team, wired into a loop

Built for the Mastercard Innovation Challenge 2026. A knowledge graph of payment-fraud vectors feeds an attack simulator, which feeds an XGBoost detector trained on tabular plus graph features. Missed detections route back as harder variants, so the detector's own blind spots choose what attacks it faces next. Across three runs the loop found a real generalization gap every time, retraining closed each one with no regression elsewhere, and 5 of 5 hand-crafted evasions were caught. Which gap it surfaces varies by run, so the writeup reports all three rather than the best one.

  • XGBoost
  • NetworkX
  • Knowledge graph
  • FastAPI
  • Cloud Run
Durable workflows Live

Artwork Exception Resolution Agent

Work that survives a crash or a slow reply

Resolves a blocker on a simulated sticker order end to end: inspect the file, clarify intent, make one eligible repair, verify it, prepare a proof. State is persisted at each step so a crash or a delayed customer answer resumes rather than restarts. A Go API and worker with a private Python image service, Postgres and object storage, all on Cloud Run with service-to-service auth by IAM.

The evaluation is the interesting part. Across 34 fixtures the agent passed all 34 and resolved 18, but once a deterministic script was given the same escalation ability it matched the agent exactly. The writeup says so plainly and favours the scripted path at this scope, rather than claiming the model won.

  • Go
  • GraphQL
  • Python
  • Cloud Run
  • Postgres
Computer vision

Human Activity Recognition in the Dark

LRCN for low-light video

An LRCN (CNN + LSTM) trained on the ARID low-light video dataset to recognise activity in near-dark footage. Accuracy moved from 72% to 91% through targeted preprocessing and model tuning, validated against loss curves, validation accuracy, and confusion-matrix analysis rather than a single headline number.

  • TensorFlow / Keras
  • CNN + LSTM
  • OpenCV
Audio ML

Natural Sound Recognition

RNN audio classification

An RNN-based classifier over engineered Librosa features, taking accuracy from 70% to 90% through architecture tuning, with structured evaluation across loss, accuracy, and per-class confusion to show where the gains actually came from.

  • Librosa
  • RNN
  • TensorFlow
Agent tooling Live

Dep Graph Generator

Which tool has to run before this one?

Reads a Composio toolkit catalog and emits a graph of which tools' outputs feed which tools' required inputs, so an agent knows what to call first. Literal field matching scores near zero: a comment tool wants issue_number, but the list tool returns Issue.number several $ref hops deep. The generator walks the schema graph, scores token matches corroborated by the owning type name, and sends only the 227 fields heuristics cannot resolve to an LLM in batches. Treating caller-supplied fields like owner and repo as context cut about 4,000 edges to 2,107 across 893 tools, and a two-tool synthetic catalog caught a filter that returned zero edges on small toolkits.

  • TypeScript
  • JSON Schema
  • OpenAI SDK
  • Force layout
  • GitHub Pages

Competitions & Hackathons

  1. Nebius x NVIDIA Global AI Hackathon 2026 Submission in progress

    IncidentForge

    Solo build for the Coding and Agentic Engineering track. A bounded Nemotron agent investigates distributed incidents and proposes repairs; independent protected checks evaluate the resulting business behavior. The recorded demo makes hypotheses, code changes, accepted candidates and limitations inspectable.

    2synthetic incident mechanisms
    12frozen campaign attempts
    7protected behavioral properties
  2. Kaggle Gemma 4 Developer Agent 2026 In progress

    Gemma 4 Agent Lab

    A coding agent on Google ADK for Gemma 4 31B: a coder that owns the patch, and a read-only analyzer it consults when navigation is unclear. A patch scores only if the hidden task's tests pass. V5 reached a verified Kaggle public score of 0.13, up from V3's 0.06. Development work covers reasoning-enabled inference, runtime budgets, and controlled evaluations with frozen, hash-verified releases.

    0.13V5 public score
    0.06V3 public score
    36offline CI checks
  3. G2 AI Engineering Hackathon 2026

    Structured-Context Ad Generation

    Problem statement 2, enriching image generation with structured context. Display ads generated with Gemini, paired with a blind evaluator whose job is to catch the ones that got the brief wrong. Full write-up in the project card above.

    5/6planted defects caught
    0false accepts
    251offline tests
  4. Mastercard Innovation Challenge 2026

    AI Defense Lab for Payment Security

    A closed red-team, blue-team loop for GenAI-powered payment fraud: a knowledge graph of attack vectors drives synthetic attacks, and the detector's own blind spots choose what it is attacked with next.

    5/5evasions caught
    3/3runs surfaced a real gap
    0regressions after retraining
  5. OpenAI Community Build Hackathon 2026

    AgentFuse

    Built for the Long-Range Autonomy challenge on multi-agent logical circuit breakers: a supervisor that halts a running agent when it loops, drifts off its goal, walks into a logic trap, or burns its budget. It kept going as a research project afterwards, and its own evaluation found a deterministic recovery ladder beating corrections written by a reasoning model, so the ladder is what ships.

    5/6real failures caught
    0/28false alarms on healthy runs
    1,018synthetic scenarios

Background

Education & Certifications

  • B.Tech in Computer Science & Engineering (2021–2025)
  • Indian Institute of Information Technology (IIIT) Vadodara
  • Machine Learning Specialization, DeepLearning.AI and Stanford (Sep 2023). Covers supervised learning, advanced learning algorithms, and unsupervised learning with recommenders and RL. Verify
  • Introduction to Generative AI, Google Cloud (Aug 2024). Verify
  • Fundamentals of Deep Learning, NVIDIA Deep Learning Institute

Retrieval Strategies

  • Vector search: dense cosine over EmbeddingGemma-300m vectors in ChromaDB
  • Vector stores: ChromaDB with cosine HNSW indexes, FAISS
  • Keyword search: BM25 across the same chunk set
  • Hybrid search: the two fused at alpha 0.6 over min-max normalised scores, chosen from a sweep showing a plateau across 0.5 to 0.8
  • Section expansion: a chunk inherits a decayed score from the best chunk under its heading, capped at four to stop one weak hit injecting a whole section
  • Page aggregation: chunk hits collapse to pages within 55% of the top score, capped at eight, since a fixed top-k is wrong in both directions
  • Metadata per chunk: page number, section heading, chunk type, table source
  • Measured and rejected on this corpus: HyDE with Qwen2.5-1.5B-Instruct (nDCG@5 0.710 to 0.684), cross-encoder reranking with BAAI/bge-reranker-base (0.728 to 0.727), and metadata as a separate scoring channel (0.710 to 0.704)

Document Processing

  • Layout-aware partitioning with unstructured hi_res, typing every block as title, prose, table or image
  • Three chunking paths that never share code, so structure survives: prose grouped by section title, tables to markdown, figures to caption plus OCR
  • Chunk sizing: 1000 minimum, 3000 soft, 5000 hard, with 500 characters of overlap applied between adjacent chunks and not only inside oversized ones
  • Tables read with pdfplumber from the embedded text layer rather than OCR of a rendered image, which recovered a symbol column the index was missing entirely
  • Generic boilerplate detection: lines appearing on 60% or more of pages are dropped, which cleared 226 of 852 elements and 57 false section titles
  • Tesseract OCR with stopword filtering for figure keywords

Models & Agents

  • Supervised fine-tuning (SFT): GPT-4o, BERT, RT-DETR
  • Agent frameworks: LangGraph, OpenAI AgentKit, LangChain, MCP orchestration
  • Tool-calling loops with a forced terminal tool, so every turn returns a structured object carrying a confidence score instead of free text
  • Human-in-the-loop routing: confidence computed from tool outcomes rather than self-reported, with risk-based escalation that overrides it
  • Heuristics first, model second: deterministic scoring resolves most cases and only the unresolved remainder goes to an LLM in batched calls, keeping cost off an O(n²) search
  • Providers behind one swappable interface (Groq, OpenAI, Anthropic), so changing model vendor is an implementation rather than a rewrite
  • Google ADK agent configs: a coder that owns the patch plus a read-only analyzer sub-agent, on Gemma 4 31B
  • vLLM inference serving with LiteLLM integration: reasoning-enabled generation, tool-call diagnostics, and bounded coding-agent runs
  • Image generation from structured context with Gemini 3.1 Flash Lite Image, at a fixed 1024×1024
  • Embeddings: google/embeddinggemma-300m, an asymmetric prompt-conditioned model at 768 dimensions, plus text-embedding-3-small
  • Cross-encoder: BAAI/bge-reranker-base
  • Generation: Qwen2.5-1.5B-Instruct for grounded answers and HyDE passages
  • Vision: llava-interleave-qwen-0.5b for figure captioning, chosen over llava-1.5-7b because 4 GB of VRAM could not hold the larger model

Evaluation & Reliability

  • Retrieval metrics: nDCG@k, hit@k, recall@k over graded relevance, where a complete answer and a passing mention score differently
  • Held-out splits: 12 of 45 queries never inspected while tuning anything
  • Generalization checks: a small synthetic out-of-domain input run beside the real one, which caught a filter that silently produced nothing on small toolkits
  • Baseline comparison: scoring an agent against a deterministic script on the same fixtures, and reporting the result when the script wins
  • Controlled fixtures with ground truth known by construction: abstentions counted apart from detections, and a false accept treated as the failure that matters
  • SWE-bench-style scoring, where a patch counts only if the repository's own tests pass
  • Paired coding-agent evaluations: verified solves, graded failures, ungraded attempts, and environment failures reported separately
  • JUnit control replay, checkout import provenance, sandbox cleanup checks, and mutation tests for evaluation guards
  • Failure taxonomy: queries tagged by kind (word, symbol, table, figure, multi) because those classes fail for different reasons and an average hides it
  • Detection metrics: mAP, per-class precision/recall, latency
  • Version-over-version quality regression across index builds
  • Schema-validated LLM output with Zod, plus per-request cost tracking
  • Runtime profiling: separating what the GPU helps (indexing, 7x) from what it does not (query time, 117 ms against 119 ms)

Technical Expertise

  • Languages: Python, Go, TypeScript, JavaScript, C/C++, SQL
  • ML: PyTorch, TensorFlow, Keras, Hugging Face, scikit-learn, OpenCV, Librosa
  • Data: Pandas, SQL, reproducible data pipelines
  • Graphs: JSON Schema $ref resolution, dependency and knowledge graphs with NetworkX, canvas force-directed visualisation
  • Backend: Go services, FastAPI, NestJS, Next.js, GraphQL, PostgreSQL with pgvector, Redis
  • Async work: BullMQ queues, idempotent webhooks, resumable background jobs, retry with backoff on flaky third-party APIs
  • Durable multi-step workflows: state persisted per step so a crash or a delayed external reply resumes instead of restarting
  • CI/CD: GitHub Actions, automated build and deploy pipelines
  • Containers & orchestration: Docker, Kubernetes
  • Deploys: GCP Cloud Run, Vercel, Neon, Supabase, Upstash, object storage with lifecycle expiry
  • Service-to-service auth with IAM rather than network ingress, plus server-side read-only guards on a public demo
  • Reproducible releases: frozen, hash-verified submission builds with CPU-only CI on Linux and Windows
  • Familiar with AWS (ECS/EKS), Git/GitHub, REST APIs

Let's build something reliable.

I'm open to AI Engineer roles and interesting problems in agent orchestration, retrieval, and model evaluation. The inbox is genuinely open.