As AI becomes better at improving its own capabilities, the challenge of taste in choosing the direction of future research is becoming increasingly important in the context of RSI. The report covers Anthropic's TASTE research and related discussions, raising questions about the differences between AI and human judgment.
As the automation of AI research advances, taste is attracting attention as an AI capability for judging the relative merits of research proposals. On 28 August, US-based Anthropic released TASTE, a measure of how appropriately research proposals are evaluated.
The study covered 92 pairs of research proposals. Fable 5, the best-performing AI model, achieved an accuracy rate of 60%, while the estimated agreement rate among human researchers was 77%, a difference of 17 percentage points.
The paper presents taste as a learnable capability, while explaining that there are limits to judging the future potential and importance of research proposals. It says similar challenges may arise when AI chooses the direction of research.
The report introduces OpenAI's methods for learning taste, the concept behind RLHF, and examples of research automation by The AI Scientist, and discusses how acquiring taste could affect the advancement of RSI.