Stick to your role! Stability of personal values expressed in large language models

What is it about?

LLMs can role-play different personas by simulating their values and behaviour, but can they stick to their role whatever the context? Is simulated Joan of Arc more tradition-driven than Elvis? Will it still be the case after playing chess? We argue that context-dependence should be studied as a dimension of LLM comparison. We focus on the expression of values by simulated individuals and adapt the methodology from psychology towards that end.

Featured Image

Photo by Ant Rozetsky on Unsplash

Photo by Ant Rozetsky on Unsplash

Why is it important?

The standard way to study Large Language Models (LLMs) is to provide many different queries from similar minimal contexts (e.g. multiple choice questions). However, due to LLM’s highly context-dependent nature, conclusions from such minimal-context evaluations may be little informative about the model’s behaviour in deployment (where it will be exposed to many new contexts). We address this issue by studying context-dependence as another dimension of LLM comparison.

Perspectives