Comparing large language models with small language models for healthcare applications

Abstract

Large Language Models (LLMs) have revolutionized natural language processing applications, but their computational and memory demands often overshadow their utility in resource-constrained scenarios, particularly in healthcare. This study evaluates the performance of LLaMA, integrated with a Retrieval-Augmented Generation (RAG) layer and custom documents, alongside comparisons to other LLMs like OpenAI’s models and smaller language models (SLMs) such as Hungry Hungry Hippos. Employing VC dimensions as a robust evaluation framework, we analyze model accuracy, query response time, memory consumption, and computational resource utilization. The findings reveal that while LLMs exhibit unparalleled capabilities, SLMs demonstrate comparable potential with significantly lower resource demands, making them viable alternatives for specific healthcare applications. This report underscores the trade-offs between performance and efficiency, providing actionable insights for deploying AI in healthcare.