Behavior as a modality: a framework to enable automated persuasion

Abstract

Communication, as a system of messages, symbols, and cultural exchanges, is ubiquitous across all species. Scholars have argued that communication represents one of the most transformative evolutionary transitions in life’s history (Smith and Szathmary, 1997), alongside pivotal developments like chromosomal mechanisms, eukaryotic formation, sexual reproduction, and multicellular life. Its unique capacity to enable cooperation and facilitate the unlimited transmission of cultural information grants species an unprecedented form of adaptive flexibility (Kirby et al., 2008). Because of the critical role communication plays in the survival and advancement of the species, communication has been studied since the ancient times. The earliest known work on communication, called Precepts by Ptah -Hotep appeared more than 4500 years ago (2̃300 BCE) (Gray, 1946). Since then, communication has seen three distinct waves of intensified interest: the first one in Ancient Greece with great Sophists like Aristotle, Isocrates, and Plato producing seminal works like Rhetoric, Phaedrus, and Antidosis (Hackforth, 1972; Rapp, 2002; Norlin et al., 1928), the second one with the rise of print, the reformation, the Renaissance, and the European colonial pursuits (Mack, 2011), the third and most recent one during the Second World War (Briñol and Petty, 2012). We currently stand at the cusp of a fourth such phase, precipitated not by political upheaval (like the ideas of democracy or world war) or mechanical innovation (like the printing press and the steam engine), but by the unprecedented accumulation of digital content and behavioral data. This data now serves as the foundation for developing large language and diffusion models, which hold transformative potential for behavioral scientific inquiry. We will show in this thesis that these tools, while still in their infancy, have the potential to solve many problems considered ambitious in behavioral sciences. Communication is composed of seven modalities: the communicator, message, channel, time of receipt, receiver, time of behavior, and receiver’s behavior (Shan- non and Weaver, 1949; Lasswell, 1948, 1971). Critically, each communication turn’s behavior becomes the subsequent turn’s message, rendering communication a strategic interaction between sender and receiver aimed at optimizing shared or individual objectives (Smith and Harper, 2003). Examples like legal defense and prosecution, scientific discourse, mating, organizational communication, diplomacy, political propaganda, and culture (like folk songs and maxims), present different types of goals. This thesis explores behavioral sciences’ enduring mission—first articulated by Aristotle 2,500 years ago—of identifying and leveraging persuasive mechanisms (Rapp, 2002). The field has traditionally bifurcated into two epistemological approaches: explanation and prediction. Historically, behavioral scientists have sought explanations that can provide interpretable causal mechanisms behind human and societal functioning. However, societies and humans do not render them- selves to clean-cut equations and formulas, as is evidenced by the limited success of behavioral explanations in predicting behavior. The emergence of extensive digital behavioral repositories has consequently shifted focus towards more robust predictive methodologies. In this thesis, we start with the more traditional approach of behavior explanation, where we cover persuasion strategies in advertising images and videos. We construct the largest set of generic persuasion strategies based on theoretical and empirical studies in marketing, social psychology, and machine learning literature. We introduce the first dataset for studying persuasion strategies in advertisements. Further, we also introduce methods called universal adversarial triggers to mine behavioral models to understand what they learn. While persuasion strategies help a human correlate and understand content and behavior, universal adversarial triggers help understand what models learn, which makes them successful in predicting behavior. Next, we turn attention towards behavior prediction by constructing general behavior models. These models, similar to large language models, aim to understand behavior in general, as opposed to designed for a specific behavioral task We use the large repositories of digital analytics to train these models. The format of this data is the general communication model consisting of the communicator, message, time of message, channel, receiver, time of effect, and effect. We call these models, Large Content and Behavior Models (LCBMs). We further show that large language models, while being used as general purpose models for a variety of tasks in different domains, are unable to solve behavioral problems. We investigate the reason for this and find that while training LLMs, behavioral data is removed as noise due to which they lose the behavioral capabilities. We also show that after including the behavioral training data back leads to other positive side effects. Namely, we show that since behavior is an after effect of content (message), therefore, we can make inferences about content by looking at the receiver behavior. An example for this is blood pressure or eye dilation levels upon watching the movie Jurassic Park indicates the excitement level of different scenes. We show results for this hypothesis on more than 40 content understanding tasks across all four modalities of text, image, video, and audio. Finally, we make initial strides towards solving the problem of generating per-formant content. We show this both for performant text generation, by taking the illustrative case of the behavior of memorability, and images, by generating images that are more engaging. We also develop mechanisms to measure the engagement potential of text to image generation models. We show that existing metrics to benchmark the quality of text to image generation models are not correlated with engagement. We develop a model to measure the engagement potential of an image. We release the first automated arena to benchmark the engagement of text-to-image models. We rank several popular text-to-image models on their ability to generate engaging images and further encourage the community to submit their models to the arena.