RECAP: Elevating Audio Captioning with Retrieval-Augmented Models

By Aryaman PattnayakSep 22 2023Reviewed by Susha Cheriyedath, M.Sc.

In a paper submitted to the arxiv* server, researchers from the University of Maryland proposed REtrieval-Augmented Audio CAPtioning (RECAP), a novel technique to enhance audio captioning performance when generalizing across domains. The study demonstrates RECAP's capabilities on benchmark datasets and reveals unique strengths like captioning unseen audio events and complex multi-sound audios.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Audio captioning aims to generate natural language descriptions for environmental audio content instead of transcribing speech. Mapping audio to text aids several real-world applications across various fields. Most existing methods utilize encoder-decoder architectures with pre-trained audio encoders and text decoders. However, performance degrades significantly on out-of-distribution test domains, limiting usefulness in practice.

The researchers hypothesize that this challenge stems from distribution shifts in audio events across domains. For example, AudioCaps contains sounds like jazz music and interviews, which Clotho does not include. Real-world scenarios also involve emerging audio concepts within a domain over time. The researchers propose the novel RECAP technique to address this issue of poor generalization across domains.

Audio captioning has drawn increasing research attention as a way to describe audio content using natural language automatically. It has various applications across domains like surveillance, transportation, healthcare, etc. For instance, audio captioning systems could generate descriptions of sounds captured by smart speakers in a home environment or cameras at traffic intersections.

Unlike speech recognition, which focuses on transcribing human speech, audio captioning focuses on richer descriptions of environmental sounds and audio events. It involves listening to an input audio clip containing sounds like car honks, bird chirps, applause, and more while generating a textual caption that describes the audio content.

Overview of RECAP

The RECAP pipeline employs Contrastive Language-Audio Pretraining (CLAP) to encode audio and Generative Pretrained Transformer 2 (GPT-2) to decode text. For each audio, RECAP retrieves top similar captions from a datastore using CLAP to construct prompt text. This text conditions GPT-2 alongside the CLAP audio embedding via cross-attention layers. Only the cross-attention modules require training, freezing CLAP and GPT-2 for efficiency.

RECAP's design enables exploiting external text captions without training, reducing reliance on decoder weights. Pretraining CLAP on aligned audio-text gives it superior linguistic comprehension, further improving data efficiency. The authors motivate the RECAP approach to improve generalization performance by leveraging retrieval augmentation.

Earlier approaches to audio captioning involved rule-based and template-based methods that performed poorly for unconstrained audio. Recent progress has focused on data-driven deep learning techniques like encoder-decoder models. An audio encoder network processes the input audio in a typical encoder-decoder architecture. It generates a high-level representation to condition the language decoder to generate the caption text word-by-word.

Various choices have been explored for the audio encoder and text decoder, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), transformers, and sizeable pre-trained language models. Attention mechanisms allow the decoder to focus on relevant parts of the audio embedding while generating each word. However, models trained on limited paired datasets fail to generalize well to new distributions, motivating RECAP's design.

Evaluation of RECAP

RECAP was comprehensively evaluated on Clotho and AudioCaps benchmarks using different training regimes and datastore compositions. The authors also deeply analyzed model performance on complex real-world audio cases.

Experiments were conducted in three diverse settings:

(i) train and test on the same dataset,

(ii) train on one dataset, test on the other, and

(iii) train on both datasets and test individually. The data store contained source training set captions, target training set captions, a sizeable external pool, or combinations.

Performance was thoroughly compared against recent competitive baselines using standard metrics like Bilingual Evaluation Understudy (BLEU), Consensus-Based Image Description Evaluation (CIDEr), Semantic Propositional Image Caption Evaluation (SPICE), etc. The key research questions examined were RECAP's in-domain vs. cross-domain performance, the impact of the datastore composition, and capabilities on compositional audios.

The comprehensive experiments and evaluation provide insights into RECAP's capabilities under varying conditions, highlighting benefits compared to prior state-of-the-art approaches. The results help assess the effect of different training strategies and data store configurations.

Key Results

The results demonstrate that RECAP achieves state-of-the-art in-domain performance on Clotho and AudioCaps, outperforming existing methods. The advantages are more pronounced for cross-domain transfer, where RECAP substantially outperforms existing approaches when the datastore matches the target domain. The large external data store consistently boosts performance over just using source dataset captions.

Detailed analysis of model outputs on complex audio reveals unique benefits. RECAP generates better captions describing multiple sound events for compositional audios containing mixtures of sounds. It also manages to caption unseen audio concepts using relevant examples from the target domain's training set.

These qualitative examples prove that RECAP can generalize to caption novel sound events never observed during training through retrieval prompting. The capability to handle compositional audios also highlights RECAP's ability to describe multi-sound content.

Future Outlook

RECAP presents a significant advance for audio captioning, addressing the critical challenge of domain shift limiting current models' applicability. RECAP also showcases improved compositionality and novel event captioning through its prompting strategy.

However, some open questions remain for future work. Noisy or irrelevant retrieval examples can confuse caption generation if not filtered properly. Large-scale retrieval for prompting also raises computational and infrastructure challenges to enable low latency. Further analysis is required on how retrieval quality impacts overall performance. Another direction is improving captioning without domain-specific examples, where RECAP currently needs to catch up to specialized in-domain systems.

Nonetheless, RECAP provides a compelling starting point with its empirical strengths, simplicity, and training efficiency. Exploring tailored retrieval techniques and improved audio-text models are interesting future directions to build upon this work. The code and data released by the researchers should facilitate further progress in this exciting field.

In conclusion, the researchers propose RECAP, a new audio captioning approach utilizing retrieval-augmented generation. RECAP exploits CLAP's joint audio-text space and prompting with similar captions to enhance cross-domain generalization. Comprehensive experiments on Clotho and AudioCaps validate RECAP's advantages, like compositionality, novel event captioning, and significant gains in out-of-domain settings.

While some open challenges remain, RECAP provides a promising direction for developing widely usable audio captioning models. By alleviating domain shift issues, RECAP moves us closer to unlocking applications across diverse fields like smart cities, home assistance, and industrial monitoring. The researchers aim to refine retrieval algorithms and audio-text modeling techniques in future work.

This offers an important step towards robust audio captioning that can reliably generalize beyond training domains. RECAP's practical yet simple approach, combined with solid empirical evidence, make it a compelling solution for enabling the next generation of capable audio captioning systems across various real-world use cases.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.