Publications

2023

1. EMNLP

   MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models

   Deepak Nathani, David Wang, Liangming Pan, and William Yang Wang

   2023

   Abs arXiv Code

   Language Models (LMs) have shown impressive performance in various natural language tasks. However, when it comes to natural language reasoning, LMs still face challenges such as hallucination, generating incorrect intermediate reasoning steps, and making mathematical errors. Recent research has focused on enhancing LMs through self-improvement using feedback. Nevertheless, existing approaches relying on a single generic feedback source fail to address the diverse error types found in LM-generated reasoning chains. In this work, we propose Multi-Aspect Feedback, an iterative refinement framework that integrates multiple feedback modules, including frozen LMs and external tools, each focusing on a specific error category. Our experimental results demonstrate the efficacy of our approach to addressing several errors in the LM-generated reasoning chain and thus improving the overall performance of an LM in several reasoning tasks. We see a relative improvement of up to 20% in Mathematical Reasoning and up to 18% in Logical Entailment.

2. Arxiv

   Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies

   Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and 1 more author

   2023

   Abs PDF

   Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. However, their efficacy is undermined by undesired and inconsistent behaviors, including hallucination, unfaithful reasoning, and toxic content. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. Techniques leveraging automated feedback – either produced by the LLM itself or some external system – are of particular interest as they are a promising way to make LLM-based solutions more practical and deployable with minimal human feedback. This paper presents a comprehensive review of this emerging class of techniques. We analyze and taxonomize a wide array of recent work utilizing these strategies, including training-time, generation-time, and post-hoc correction. We also summarize the major applications of this strategy and conclude by discussing future directions and challenges

2022

1. ACL

   Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings

   Kalpesh Krishna, Deepak Nathani, Xavier Garcia, Bidisha Samanta, and Partha Talukdar

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) May 2022

   Abs arXiv PDF Slides Website

   Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al. 2021) has attempted “few-shot” style transfer using only 3-10 sentences at inference for style extraction. In this work we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim. We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob. We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymization) all without retraining the model. Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages. To facilitate future research we crowdsource formality annotations for 4000 sentence pairs in four Indic languages, and use this data to design our automatic evaluations.

2. UMAP

   Walking with PACE - Personalized and Automated Coaching Engine

   Madhurima Vardhan, Narayan Hegde, Srujana Merugu, Shantanu Prabhat, Deepak Nathani, and 9 more authors

   In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization May 2022

   Abs PDF

   We design and implement a personalized and automated physical activity coaching engine, PACE, which uses the Fogg’s behavioral model (FBM) to engage users in mini-conversation based coaching sessions. It is a chat-based nudge assistant that can boost (encourage) and sense (ask) the motivation, ability and propensity of users to walk and help them in achieving their step count targets, similar to a human coach. We demonstrate the feasibility, effectiveness and acceptability of PACE by directly comparing to human coaches in a Wizard-of-Oz deployment study with 33 participants over 21 days. We tracked coach-participant conversations, step counts and qualitative survey feedback. Our findings indicate that the PACE framework strongly emulated human coaching with no significant differences in the overall number of active days, step count and engagement patterns. The qualitative user feedback suggests that PACE cultivated a coach-like experience, offering barrier resolution via motivational and educational support. We use traditional human-computer interaction approaches, to interrogate the conversational data and report positive PACE-participant interaction patterns with respect to addressal, disclosure, collaborative target settings, and reflexivity. As a post-hoc analysis, we annotated the conversation logs from the human coaching arm and trained machine learning (ML) models on these data sets to predict the next boost (AUC 0.73 ± 0.02) and sense (AUC 0.83 ± 0.01) action. In future, such ML-based models could be made increasingly personalized and adaptive based on user behaviors.

2020

1. ICLR

   Few-Shot Learning on Graphs via Super-Classes Based on Graph Spectral Measures

   Jatin Chauhan, Deepak Nathani, and Manohar Kaul

   In International Conference on Learning Representations May 2020

   Abs arXiv Blog Code

   We propose to study the problem of few shot graph classification in graph neural networks (GNNs) to recognize unseen classes, given limited labeled graph examples. Despite several interesting GNN variants being proposed recently for node and graph classification tasks, when faced with scarce labeled examples in the few shot setting, these GNNs exhibit significant loss in classification performance. Here, we present an approach where a probability measure is assigned to each graph based on the spectrum of the graphs normalized Laplacian. This enables us to accordingly cluster the graph base labels associated with each graph into super classes, where the Lp Wasserstein distance serves as our underlying distance metric. Subsequently, a super graph constructed based on the super classes is then fed to our proposed GNN framework which exploits the latent inter class relationships made explicit by the super graph to achieve better class label separation among the graphs. We conduct exhaustive empirical evaluations of our proposed method and show that it outperforms both the adaptation of state of the art graph classification methods to few shot scenario and our naive baseline GNNs. Additionally, we also extend and study the behavior of our method to semi supervised and active learning scenarios.

2. A Persistent Homology Perspective to the Link Prediction Problem

   Sumit Bhatia, Bapi Chatterjee, Deepak Nathani, and Manohar Kaul

   In Complex Networks and Their Applications VIII May 2020

   Abs PDF

   Persistent homology is a powerful tool in Topological Data Analysis (TDA) to capture topological properties of data succinctly at different spatial resolutions. For graphical data, shape and structure of the neighborhood of individual data items (nodes) is an essential means of characterizing their properties. We propose the use of persistent homology methods to capture structural and topological properties of graphs and use it to address the problem of link prediction. We achieve encouraging results on nine different real-world datasets that attest to the potential of persistent homology based methods for network analysis.

2019

1. ACL

   Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

   Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics May 2019

   Abs arXiv Blog Code

   The recent proliferation of knowledge graphs (KGs) coupled with incomplete or partial information, in the form of missing relations (links) between entities, has fueled a lot of research on knowledge base completion (also known as relation prediction). Several recent works suggest that convolutional neural network (CNN) based models generate richer and more expressive feature embeddings and hence also perform well on relation prediction. However, we observe that these KG embeddings treat triples independently and thus fail to cover the complex and hidden information that is inherently implicit in the local neighborhood surrounding a triple. To this effect, our paper proposes a novel attention based feature embedding that captures both entity and relation features in any given entity’s neighborhood. Additionally, we also encapsulate relation clusters and multihop relations in our model. Our empirical study offers insights into the efficacy of our attention based model and we show marked performance gains in comparison to state of the art methods on all datasets.

2018

1. ICML

   Solving Partial Assignment Problems using Random Clique Complexes

   Charu Sharma, Deepak Nathani, and Manohar Kaul

   In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 May 2018

   Abs arXiv Blog Code

   We present an alternate formulation of the partial assignment problem as matching random clique complexes, that are higher-order analogues of random graphs, designed to provide a set of invariants that better detect higher-order structure. The proposed method creates random clique adjacency matrices for each k-skeleton of the random clique complexes and matches them, taking into account each point as the affine combination of its geometric neighbourhood. We justify our solution theoretically, by analyzing the runtime and storage complexity of our algorithm along with the asymptotic behaviour of the quadratic assignment problem (QAP) that is associated with the underlying random clique adjacency matrices. Experiments on both synthetic and real-world datasets, containing severe occlusions and distortions, provide insight into the accuracy, efficiency, and robustness of our approach. We outperform diverse matching algorithms by a significant margin.