I'd say

I wasn't mean to anyone.  It was they who started doing something to me.