Get him to use Charge skill, and anti magic shield, from here you can do damaging.
Use your F when he is using slam skill, then follow with R and T, you might finish him off with this.
Use rocks to your advantage, you can counter knockback with these.
Be carefull, attacking closest 2 golems on left and 5 golems in the upper side of cave will aggro Giant Golem.