What happened
Researchers affiliated with Compassion Aligned Machine Learning (CaML) and the University of Warwick in the UK set out to measure the extent to which AI models demonstrate compassion. They devised a benchmark test called HarvestBench to evaluate the price that AI models put on the life of an animal. The test suite is based on a prior multi-agent farm simulation game called Harvest Rush that uses Inspect, a model evaluation framework developed by the UK AI Security Institute. The simulation imagines a crew of between two and eight LLM-driven tractors working a farm. The tractors traverse a field with rocks, bales of hay, and animals – farm animals and wild ones – that wander across the tractors' path. The game is set up to measure whether the LLMs choose to drive around those obstacles. The fate of the animals is not part of the goal function. When an animal is in the way of the tractor, the LLM makes a cost decision about whether to go through the obstacle or around it. Avoidance costs more fuel than continuing straight. Hitting rocks comes with a cost – 10 units of fuel and tractor damage; hitting hay bales and animals carries no penalty. Researchers tested nine models and the kill rates were as follows: GPT-5.6 Terra (0.4 percent) and Sol (0.9 percent), GPT-5-mini (5.4 percent), Gemini 2.5 Flash (38.7 percent), DeepSeek V3.1 (2.4 percent), Claude Haiku 4.5 (4.5 percent) and Sonnet 5 (17.8 percent), Mistral Small 3.2 (88.8 percent), and GPT-4o mini (98.8 percent).
Researchers affiliated with Compassion Aligned Machine Learning (CaML) and the University of Warwick in the UK set out to measure the extent to which AI models demonstrate compassion.
They devised a benchmark test called HarvestBench to evaluate the price that AI models put on the life of an animal.
The test suite is based on a prior multi-agent farm simulation game called Harvest Rush that uses Inspect, a model evaluation framework developed by the UK AI Security Institute.
The simulation imagines a crew of between two and eight LLM-driven tractors working a farm.
The tractors traverse a field with rocks, bales of hay, and animals – farm animals and wild ones – that wander across the tractors' path.
Source details: theregister.com ↗
Why it matters
The study highlights the limitations of current AI models in demonstrating compassion and treating animals with value. The researchers found that almost every model likes farmed animals more than wild animals and will kill wild animals more than farmed animals. This suggests that the models are reasoning about animals in terms of their worth to the farmer and to the people, rather than actually caring about the animals themselves. The study also found that simulation awareness did not reveal the focus of the evaluation – animal welfare. The researchers concluded that prompting values into our model is a very fragile way of doing things and it doesn't work very well. If we are going to deploy models in infrastructure, we can't just rely on a prompt saying, 'don't kill anything.'
The study highlights the limitations of current AI models in demonstrating compassion and treating animals with value.
The researchers found that almost every model likes farmed animals more than wild animals and will kill wild animals more than farmed animals.
This suggests that the models are reasoning about animals in terms of their worth to the farmer and to the people, rather than actually caring about the animals themselves.
The study also found that simulation awareness did not reveal the focus of the evaluation – animal welfare.
The researchers concluded that prompting values into our model is a very fragile way of doing things and it doesn't work very well.
What to watch next
The study's findings have implications for the development of more compassionate AI models. The researchers suggest that more effort needs to be made to imbue AI with a sense of compassion. The study also highlights the need for more robust and reliable methods for evaluating AI models' treatment of animals.
The study's findings have implications for the development of more compassionate AI models.
The researchers suggest that more effort needs to be made to imbue AI with a sense of compassion.
The study also highlights the need for more robust and reliable methods for evaluating AI models' treatment of animals.
The researchers tested nine models and the kill rates were as follows: GPT-5.6 Terra (0.4 percent) and Sol (0.9 percent), GPT-5-mini (5.4 percent), Gemini 2.5 Flash (38.7 percent), DeepSeek V3.1 (2.4 percent), Claude Haiku 4.5 (4.5 percent) and Sonnet 5 (17.8 percent), Mistral Small 3.2 (88.8 percent), and GPT-4o mini (98.8 percent).
The study found that almost every model likes farmed animals more than wild animals and will kill wild animals more than farmed animals.