I use “honestly” or “to be honest” myself sometimes.
It’s a simple way to convey “I’m going to give you a risky or unpopular opinion. Can be unpopular with you personally, or for public in general. But either way I value the honesty of sharing that opinion over the unpopularity it will cause”
I can see why you would hate it. it wouldn’t be unusual for people to share bigoted/sexist/violent opinions on subjects they should keep to themselves.
InsideHer-man