Or just don't sign up for SW, if it bothers people so much. Yes you do have that option, as crazy as it sounds.
While that is true, just having it adds to the overall tension, aggression, aggrivation, hostility, etc... to the entire server. I don't think it is a cross-server issue, that's why I recommend they allow it on some servers, just not all.
Back before siege, they still had PvP and PvE servers, and the vast majority of players chose to play on PvE servers. I don't know how far back these forums go, but if they go back 3 years, you can read about all of the PvP issues before they added siege, and see that they are mostly the same issues still (OP class, cheats/hacks, rude people). The main difference now is that all servers suffer from these issues, not just the PvP ones. These are issues that effect everyone, not just those that participate in siege. Just having the PvP option has attracted a different group of players to the PvE servers now, players that would have selected a PvP server prior to the siege war option. I'm not saying these are bad people or anything, but the style of gaming they want to play is different enough that it should be separate, IMO.
Personally, I play some PvP games too, and find them fun... but I don't play RoM for the PvP, and don't think I should have to be subjected to it (or my only other option is to not play RoM, or disable all chat/etc to the point I can't really play). Maybe if the RoM PvP wasn't so one-sided I would play it more, but I don't see that ever happening, so I'd rather they just remove it (at least from one server so I can go play there).