Same here with Skyrim, around level 30 I decided to go with Conjuration and destruction but I got killed very often, but the good thing is that I invested a lot in shields in early levels so I'm going with my shield and heavy armor but using destruction and conjuration, no swords or axes. So far it works very well.