Over the day 2000, I'd say with no hesitation: Praven. Also took 3 assaults for that one, with a 4-1 superiority in each, to take it down. You'll have to go down the stairs, still under fire, walk down a street, still under fire, up some stairs, still under fire, and then, finally, fight them and their rather decent meele stats, while they spawn around you in a very close quarters battle. Once up there, however, those damned Sarranid Master Archers will start spawning on a gate with a direct view of the wall, and start firing.

Getting up the ladder is hard, but feasible. Honorable mention goes to Ahmerrad, which I once also had to take from the Rohdoks, although it was garrisoned mostly by Sarranid troops. I took it once with a big and mixed army, mostly made of Huscalars, Mameluks and the odd Swadian Knight + several sarranid support troops, but it took 3 assaults, and by the time it fell, I had lost more than half my troops. The few crazy bastards who make it up the top of the ladder will face Rohdok Sargents, supported by yet another bunch of crossbowmen in a nearby roof (and spawn point). It being a Rohdok castle at the start of the game, means those 4 sides will likely be staffed with deadly marksmen, or at least hordes of damn good crossbowmen.

The damned assault ladder is somehow inside the courtyard meaning you have to walk towards the castle, taking fire from the main wall, then climb the ladder in the middle of a 4-way crossfire.

Mount and blade how to get a castle