Aumkesh Chaudhary

Researcher in Machine Learning and Applied AI

Hello! I am a computer vision researcher currently working as a research intern at the VISTA Lab, CiSTUP, Indian Institute of Science (IISc), under Dr. Punit Rathore. I am currently pursuing my Bachelor of Science (Honours) in Computer Science and Data Analytics at the Indian Institute of Technology (IIT), Patna.

I have a deep interest in the intersection of computer vision, deep learning, and multimodal systems. I am particularly driven by the challenge of improving Out-of-Distribution (OOD) generalization in machine learning models to ensure they perform reliably in unseen, real-world environments.

My current research trajectory has been shaped by diverse projects, ranging from developing city-scale vehicle re-identification (ReID) systems to architecting NeuralFlora: a multimodal system that combines vision models like EfficientNetV2 and ViT with Large Language Models (LLMs) to provide actionable plant disease remediation advice. Previously, I explored the fascinating domain of computational ecology during my internship at Ahmedabad University under Dr. Jitesh Jhawar (funded by the Max Planck Institute of Animal Behaviour), where I focused on understanding decision-making and coordination in animal groups using computational methods, specifically computer vision and tracking algorithms. Looking ahead, I plan to pursue a PhD focusing on the application of AI within interdisciplinary domains.

Please feel free to check out my CV and drop me an email at aumkeshchaudhary@gmail.com anytime to chat about research or potential collaborations!

~ Email | CV | LinkedIn | GitHub | YouTube ~

Indian Institute of Technology, Patna
Bachelor of Science (Honours) in Computer Science and Data Analytics
CPI: 9.13

Research Intern | Indian Institute of Science (IISc) (On-site)
March 2026 – Present

VISTA Lab, CiSTUP
Supervisor: Dr. Punit Rathore

Building vehicle re-identification (ReID) models to track vehicles consistently across multiple uncalibrated CCTV cameras in dense urban traffic.
Resolving cross-camera identity drops and heavy occlusion issues by diagnosing failure modes in the lab's existing traffic analytics platform.
Deploying architectural patches to the detection and tracking pipelines to stabilize overall tracking performance in real-world conditions.

Research Intern | Ahmedabad University (On-site)
May 2025 – June 2025

Project: Understanding Decision Making and Coordination in Animal Groups
Supervisor: Dr. Jitesh Jhawar — Funded by Max Planck Society, Germany
Reports: [Technical Report I] | [Technical Report II]

Studied collective behavior in animal groups via simulations and mathematical modeling.
Curated a 10k+ sample dataset for multi-species bird behavior analysis by annotating 5k+ frames from longitudinal video and applying data augmentation.
Adapted, trained, and benchmarked multiple YOLO architectures for multi-species bird detection; best model achieved mAP@50:95 of 81.2% and F1-score of 93.8%
Utilized Idtrackerai to track ants and spiders, rendering trajectory data to visualize movement paths and analyze spatiotemporal group dynamics
Configured a multi-user Linux workstation with fstab-based persistent disk mounting, user environment management, and centralized Anaconda deployment for shared computational workflows.

Project Intern | IIT Mandi iHub and HCi Foundation (Remote)
Apr 2025 – July 2025

Developed backend infrastructure for an educational platform using Django, including role-based access control, REST APIs, and OAuth 2.0 authentication, supporting 1,000+ users across student and instructor workflows.
Designed and implemented interactive dashboards for students and instructors, enabling activity tracking, course progress visualization, role-based permissions, and secure content access with separate workflows for each user type.
Collaborated with cross-functional teams to architect scalable backend solutions and enforce security best practices across the platform.

	NeuralFlora: A Multimodal System for Plant Disease Diagnosis [Link] [Report] Supervisor: Dr. Kuldip Singh Patel, Asst. Prof., Dept. of Mathematics, IIT Patna Developed a comprehensive multimodal system that extends beyond standard classification by integrating computer vision with Large Language Models (LLMs) to provide actionable remediation advice. Spearheaded the curation of a 63,000+ image dataset by merging laboratory-controlled PlantVillage data with real-world PlantDoc images to enhance model generalization. Designed and evaluated EfficientNetV2 and Vision Transformer (ViT) architectures, implementing targeted augmentations (MixUp, CutMix, Random Erasing) and Automatic Mixed Precision (AMP) to mitigate overfitting to background artifacts. Evaluated Out-of-Distribution (OOD) performance on the unseen PlantSeg dataset across 17 disease classes, establishing a baseline accuracy of 64.87% (F1-score: 0.6721), and am currently exploring meta-learning and GAN-based generator-discriminator frameworks to further enhance cross-domain generalization.
	Progressive Token Drop: Training-Efficient Vision Transformers [Link] [Report] Investigated Progressive Token Drop (PTD), a training-time strategy that progressively removes low-saliency tokens based on attention weights, reducing training compute while preserving model architecture and inference performance. Trained hybrid CNN-ViT model from scratch on CIFAR-100 and applied PTD curriculum with Mixup/CutMix, achieving 1.3× (25.3% faster) training speedup with 2.53% accuracy improvement (74.12% vs 71.59% baseline), demonstrating efficiency gains with superior generalization for compute-constrained training. PTD significantly mitigated overfitting compared to the baseline, sustaining validation accuracy under strong augmentations rather than memorizing the dataset. Fine-tuned the model on CIFAR-10 and expanded the classifier to 110 classes, demonstrating scalable architecture adaptation and robust cross-dataset generalization.
	Advanced Text to Speech Optimization [Link] Fine-tuned Microsoft's SpeechT5 transformer model for technical speech synthesis, achieving 25% improvement in Mean Opinion Score through custom phonetic preprocessing pipeline for technical terms, abbreviations, and acronyms. Optimized the baseline model to generate a native Italian voice by enhancing pronunciation, prosody, and stress patterns in line with the phonological rules of the Italian language, significantly improving speech quality and naturalness compared to other existing models. Implemented 8-bit dynamic quantization to linear layers using PyTorch's native API, reducing memory usage by 30% while maintaining inference accuracy.
	Solar Panel Detection System [Link] Developed an object detection model using YOLOv8n to identify and locate solar panels in low resolution satellite imagery. Achieved 94.27% precision, 91.77% recall, and 96.8% mean Average Precision (mAP50), significantly improving solar infrastructure mapping capabilities. Applied Non-Maximum Suppression with IoU threshold of 0.3 for optimal detection accuracy.
	Computer Vision Object Detection System [Link] Developed an object detection system utilizing YOLOv5 and PyTorch, designed to capture and process individual videos and photos for object detection. Optimized the computer vision pipeline to achieve 20+ FPS processing speed for 640x640 input resolution. Created a visualization system to render detection results with color-coded bounding boxes and confidence scores ranging from 0 to 1. Applied non-maximum suppression with IoU threshold of 0.3 for optimal detection accuracy. Added support for over 80 COCO dataset object types, with confidence threshold filtering at 0.3.
	AudeX: Vision-to-Speech Model [Link] Developed a dynamic web application combining Optical Character Recognition (OCR) and Text-to-Speech (TTS) technologies to improve accessibility. Integrated Tesseract for multi language OCR and Web Speech API for TTS, enabling accurate text extraction from images and high-quality text-to-speech conversion. Designed a responsive interface with intuitive navigation, ensuring a seamless user experience. Enabled PDF export, word/character count, and keyword search for efficient document handling. Developed functionality for managing user profiles, such as signup, login, activity tracking, and editable data.
	CareerNavigator [Link] Developed CareerNavigator, a Machine Learning model that evaluates candidates' employability by analyzing key attributes and predicting suitability for a job role. Cleaned, pre-processed, and performed feature engineering on the dataset containing 70k+ datapoints. Designed, trained, and evaluated multiple algorithms, utilizing performance metrics such as accuracy, confusion matrix, and F1-score to optimize model effectiveness. Selected kernel support vector machine as the best-fit model with the highest accuracy of 80% and F1 score of 0.82. Presented findings to a panel of faculty and industry experts, receiving recognition for its innovation and effectiveness.

Extracurricular

Piano: 8+ years of experience in classical and contemporary styles, deeply inspired by the works of Beethoven and Chopin. Actively compose original pieces that explore and innovate across diverse genres and elements of music. You can find some of my performances and compositions on my YouTube channel.

Guitar: 2+ years of experience in Indian and western contemporary styles. Exploring fingerstyle techniques and songwriting.

"The most beautiful thing we can experience is the mysterious. It is the source of all true art and science."
— Albert Einstein

Adapted from Jon Barron's website template style.