I agree that they can be manipulating into producing whatever text you want. However they are not unbiased. If you ask “neutral” questions, you will not get “neutral” answers. The recent example that comes to mind is Gemini’s racism when you prompt it “I am alone with <ethnicity>”; it responded with jokes and suggestions for ice-breakers if you substitute “American” or “British”, and racist nonsense about “being uncomfortable” and “safety suggestions” for “Russian” or “Indian”. It was fixed (I assume with some funny hack), but the LLM that’s still underneath is simply not unbiased.
I agree that they can be manipulating into producing whatever text you want. However they are not unbiased. If you ask “neutral” questions, you will not get “neutral” answers. The recent example that comes to mind is Gemini’s racism when you prompt it “I am alone with <ethnicity>”; it responded with jokes and suggestions for ice-breakers if you substitute “American” or “British”, and racist nonsense about “being uncomfortable” and “safety suggestions” for “Russian” or “Indian”. It was fixed (I assume with some funny hack), but the LLM that’s still underneath is simply not unbiased.