This one has a lot of the pertinent details:
http://forum.us.runesofmagic.com/showthread.php?t=77049
If you just google for runes of magic damage formula or something like that, you get the best results. Sadly, I posted something that was very close to the magic damage formula at some point, but the forums have since done something that ate all old posts. I don't remember it in detail anymore, and some of the old versions of the equation that are still floating around are incorrect. Magic spells have a multiplier that scales with the skill level and is multiplied by the base cast time. You'll get something close to the correct multiplier by doing (1.5 + .03*skill_level)*cast_time, but I'm quite sure that those constants are off by a small amount.
The magic attack stat behaves the same way as decribed in the above link, and AOE skills generally have some multiplier that reduces their damage.