Unsupervised open-vocabulary action recognition with an autoregressive model

Abstract

Current works on zero/few- shot action recognition are largely based on contrastive approaches trained in a supervised manner to select an action class out of a predefined set. Instead, in this work, we propose a new paradigm for zero-shot action recognition based on autoregressive generation of a free-form action-specific caption describing the action occurring in the video. To this end, we propose to adapt an image-based pre-trained autoregressive Vision & Language (V&L) Model for action recognition. We firstly show that direct fine-tuning of an autoregressive model using the action classes suffers from severe overfitting. To alleviate this, we then introduce an unsupervised learning framework consisting of two key components: (a) an unsupervised method for adapting the autoregressive model to action/video data by means of pseudo-caption generation and self-training without using any action-specific labels; (b) a retrieval component for discovering a diverse set of pseudo-captions for each video. In the process, we show that both components are necessary to obtain high accuracy. Our model results in predictions that are fine-grained, interpretable, and naturally open-vocabulary. Importantly, when evaluated for zero- and few-shot action recognition, our approach matches or even outperforms contrastive learning-based methods.