I think that you were supposed to mostly rely on stealth. You don't have to fight the big guy in the castle -- you can get him very drunk take him down with one punch.

edit: just for fun, here is the scene -- https://youtu.be/9ivNLD75rAU?t=2854