>> Its offensive, because it could offend a religious group.
This is the old tree, forest, sound, nobody there thing.
Is it offensive because it could offend, as you say, or, as the other side say, is it offensive only if somebody takes offence? And further, they might say, if you yourself don't take offense, then you can't declare it to be so?
I suspect the whole thing is further muddied because the common view of the word and its meanings are now confused with the legal word and its meanings; those two sets might overlap but inevitably won't be a perfect match.