Concept-based program search for Abstraction and Reasoning Corpus (ARC)

Abstract

The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intel- ligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs ex- cel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring func- tion, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over di- rect prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging general- ization problems like ARC. However, due to computational costs, LLM-based search is infeasible for real-time purposes. Additionally, ConceptSearch is based on iteratively providing in-context examples closer to solution, which limits its efficiency due to conditioning a black-box model only on ICL examples. Building on ConceptSearch, we are working on learning a Variational Autoencoder (VAE) to represent programs in rich and low-dimensional space using our proposed concept-based scoring function. In this program space, derivative-free optimization algorithms can be utilised for faster and more controlled search, thereby improving the efficiency. Further, we would like to explore test-time training this VAE for broad generalisation to novel tasks.