publications

2025

1. 

   AutoRev: Automatic Peer Review System for Academic Research Papers

   Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Harshit Gupta, Manav Chaudhary, and Vasudeva Varma

   Under review at the Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) , 2025

   Abs arXiv

   Generating a review for an academic research paper is a complex task that requires a deep understanding of the document’s content and the interdependencies between its sections. It demands not only insight into technical details but also an appreciation of the paper’s overall coherence and structure. Recent methods have predominantly focused on fine-tuning large language models (LLMs) to address this challenge. However, they often overlook the computational and performance limitations imposed by long input token lengths. To address this, we introduce AutoRev, an Automatic Peer Review System for Academic Research Papers. Our novel framework represents an academic document as a graph, enabling the extraction of the most critical passages that contribute significantly to the review. This graph-based approach demonstrates effectiveness for review generation and is potentially adaptable to various downstream tasks, such as question answering, summarization, and document representation. When applied to review generation, our method outperforms SOTA baselines by an average of 58.72% across all evaluation metrics. We hope that our work will stimulate further research in applying graph-based extraction techniques to other downstream tasks in NLP. We plan to make our code public upon acceptance.

2. 

   Screening of oral potentially malignant disorders and oral cancer using deep learning models

   Karishma Madhusudan Desai, Pragya Singh, Mahima Smriti, Vivek Talwar, Manav Chaudhary, George Paul, Subhas Chandra Kolli, Parisa Sai Raghava, Golla Vamshi Krishna, C. V. Jawahar, P. K. Vinod, Varma Konala, and Ramanathan Sethuraman

   Scientific Reports, May 2025

   Abs DOI

   Oral cancer though preventable, shows high mortality and affect the overall quality of life when detected in late stages. Screening techniques that enable early diagnosis are the need of the hour. The present work aims to evaluate the effectiveness of AI screening tools in the diagnosis of OPMDs and Oral cancers via native or web-application (cloud) using smart phone devices. We trained and tested two deep learning models namely DenseNet201 and FixCaps using 518 images of the oral cavity. While DenseNet201 is a pre-trained model, we modified the FixCaps model from capsule network and trained it ground up. Standardized protocols were used to annotate and classify the lesions (suspicious vs. non-suspicious). In terms of model performance, DenseNet201 achieved an F1 score of 87.50% and AUC of 0.97; while FixCaps exhibited F1 score of 82.8% and AUC of 0.93. DenseNet201 model (20 M) serves as a robust screening model (accuracy 88.6%) that can be hosted on a web-application in the cloud servers; while the adapted FixCaps model with its low parameter size of 0.83 M exhibits comparable accuracy (83.8%) allowing easy transitioning into a native phone-based screening application.

3. 

   ClarifyCoder: Clarification-Aware Fine-Tuning for Programmatic Problem Solving

   Jie JW Wu, Manav Chaudhary, Davit Abrahamyan, Arhaan Khaku, Anjiang Wei, and Fatemeh H. Fard

   Under review at the International Conference on Software Engineering (ICSE 2025) , May 2025

   Abs arXiv

   Large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, a significant gap remains between their current performance and that of expert software engineers. A key differentiator is that human engineers actively seek clarification when faced with ambiguous requirements, while LLMs typically generate code regardless of uncertainties in the problem description. We present ClarifyCoder, a novel framework with synthetic data generation and instruction-tuning that enables LLMs to identify ambiguities and request clarification before proceeding with code generation. While recent work has focused on LLM-based agents for iterative code generation, we argue that the fundamental ability to recognize and query ambiguous requirements should be intrinsic to the models themselves. Our approach consists of two main components: (1) a data synthesis technique that augments existing programming datasets with scenarios requiring clarification to generate clarification-aware training data, and (2) a fine-tuning strategy that teaches models to prioritize seeking clarification over immediate code generation when faced with incomplete or ambiguous requirements. We further provide an empirical analysis of integrating ClarifyCoder with standard fine-tuning for a joint optimization of both clarify-awareness and coding ability. Experimental results demonstrate that ClarifyCoder significantly improves the communication capabilities of Code LLMs through meaningful clarification dialogues while maintaining code generation capabilities.

2024

1. 

   Towards Understanding the Robustness of LLM-based Evaluations under Perturbations

   Manav Chaudhary, Harshit Gupta, Savita Bhat, and Vasudeva Varma

   In Proceedings of the 21st International Conference on Natural Language Processing (ICON), Dec 2024

   Abs HTML

   Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality annotators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.

2. 

   BrainStorm @ iREL at #SMM4H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets

   Manav Chaudhary, Harshit Gupta, and Vasudeva Varma

   In Proceedings of the 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks, Aug 2024

   Abs HTML

   The proliferation of LLMs in various NLP tasks has sparked debates regarding their reliability, particularly in annotation tasks where biases and hallucinations may arise. In this shared task, we address the challenge of distinguishing annotations made by LLMs from those made by human domain experts in the context of COVID-19 symptom detection from tweets in Latin American Spanish. This paper presents BrainStorm @ iREL‘s approach to the #SMM4H 2024 Shared Task, leveraging the inherent topical information in tweets, we propose a novel approach to identify and classify annotations, aiming to enhance the trustworthiness of annotated data.

3. 

   iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers

   Harshit Gupta, Manav Chaudhary, Shivansh Subramanian, Tathagata Raha, and Vasudeva Varma

   In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), Jun 2024

   Abs DOI

   This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models’ lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default commonsense associations and exhibit unconventional thinking. We propose a unique strategy to improve the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dynamic few-shot prompting techniques and introduce a model-generated reasoning strategy that utilizes the LLM‘s reasoning capabilities to improve performance. Our approach demonstrated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of performing as well as the human annotators, thus highlighting the efficacy of the proposed strategies.

2023

1. 

   iMETRE: Incorporating Markers of Entity Types for Relation Extraction

   N Harsha Vardhan and Manav Chaudhary

   Accepted at The 4th Workshop on Knowledge Discovery from Unstructured Data in Financial Services (KDF) at SIGIR (non-archival) , Jun 2023

   Abs arXiv HTML

   Sentence-level relation extraction (RE) aims to identify the relationship between 2 entities given a contextual sentence. While there have been many attempts to solve this problem, the current solutions have a lot of room to improve. In this paper, we approach the task of relationship extraction in the financial dataset REFinD. Our approach incorporates typed entity markers representations and various models finetuned on the dataset, which has allowed us to achieve an F1 score of 69.65% on the validation set. Through this paper, we discuss various approaches and possible limitations.