When you can keep rogues and such off your back towers, then it shouldn't be a problem, but if one gets through, it doesn't take long for a single dps player to take it down.
Good things to keep outside would be: Buff towers, 1 of 2/2 of 3 procs, and I think the library (forget what that offers in a siege war)