Actually this trick have been proven to be useless in a lot of cases.

LLMs don’t inherently know what they are because "they" are not themselves part of the training data.

However, maybe it’s working because the information is somewhere into their pre-prompt but if it wasn’t, it wouldn’t say « I don’t know » but rather hallucinate something.

So maybe that’s true but you cannot be sure.

If you believe 'leaked system prompts', it tends to be part of the system prompt.

I believe most of these came from asking the LLMs, and I don't know if they've been proven to not be a hallucination.

https://github.com/jujumilk3/leaked-system-prompts

It's injected into their system prompt

...which is useless when the model gets changed in-between responses.