DiscoverBest AI papers explainedDo LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

Update: 2025-10-09
Share

Description

This paper assesses how well Large Language Models (LLMs) can infer, remember, and follow user preferences in long, multi-session conversations. The evaluation of 10 different LLMs using this benchmark revealed that current state-of-the-art models exhibit significant difficulty proactively following user preferences, with accuracy dropping below 10% in zero-shot settings within a short number of turns. The researchers conclude that while fine-tuning on PrefEval can improve results, the benchmark demonstrates LLMs still face challenges in personalized conversational abilities.

Comments 
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

Enoch H. Kang