Fingerprinting fine-tuned language models in the wild

Abstract

There are concerns that the ability of language models (LMs) to generate high quality synthetic text can be misused to launch spam, dis-information, or propaganda. Therefore, the re-search community is actively working on detecting whether a given text is organic or synthetic. While this is a useful first step, it is important to be able to further fingerprint the author LM to attribute its origin. Prior work on fingerprinting LMs is limited to attributing synthetic text generated by a handful (usually<10) of pre-trained LMs. However, LMs such as GPT2 are commonly fine-tuned in a myriad of ways (e.g., on a domain-specific text corpus) before being used to generate synthetic text. Thus, it is challenging to finger-printing fine-tuned LMs because the universe of fine-tuned LMs is much larger in realistic scenarios. To address this challenge, we study the problem of large-scale fingerprinting offline-tuned LMs in the wild. Using a real-world dataset of synthetic text generated by 108 different fine-tuned LMs, we conduct comprehensive experiments to demonstrate the limitations of existing fingerprinting approaches. Our results show that fine-tuning itself is most effective in attributing the synthetic text generated by fine-tuned LMs.