AI models are always surprising us, not just in what they can do, but also in what they can't, and why. An interesting new behavior is both superficial
Have they finally achieved consciousness and this is how they show it?!
No. The answer, as is usually the case with these things, is that we are anthropomorphizing a step too far. These models don’t care about what is and isn’t random. They don’t know what “randomness” is! They answer this question the same way they answer all the rest: by looking at their training data and repeating what was most often written after a question that looked like “pick a random number.” The more often it appears, the more often the model repeats it.
Such a clickbaity article.
Here’s the meat of it: