Constructed over the site of a much weaker looking wooden castle by a pompous count, the fairytale fortress known as Gravensteen became a house of torture and injustice and was almost torn down later on due to the memory of those atrocities.
While a much smaller castle existed on the site prior to the construction of Gravensteen, it was the ostentatious Count Philip of Alcase who created the lovely, if imposing fortress that remains today. The fort was originally constructed as a symbol of the power of the Count of Flanders, it was Philip, who had fought in the Crusades and seen a number of grand castles in his travels, who insisted that the walls and towers of the castle be built to such a height that their very presence would let everyone know just how much money and influence their position held.
Eventually, the counts abandoned their seat in the 14th century and the castle became simply a prison and courthouse. It featured multiple courthouses and subterranean dungeons in which criminals and accused simply awaiting trial were kept in unlivable conditions and frequently tortured.
While the barbaric activities were abandoned as the area became more civilized in the ensuing centuries, the site never lost the taint of the horrors that were once perpetrated within its walls. By the end of the 1800s, the structure had become unsound and its reputation was doing nothing for it in the court of public opinion, thus Gravensteen was nearly razed.
Luckily a preservation group realized the architectural and (however horrible) historical importance and managed to save the castle. Today, the site has been greatly restored still retaining a decorative moat on three sides, and is open to the public, even featuring a torture museum, possibly as some sort of mea culpa. Despite its grim past, Gravensteen is still one of the most stunning historic castles remaining in the world.
Update May 2019: As of May 2018, the torture museum no longer exists. There were some renovations carried out and the implements of torture are now reduced to a few remaining examples displayed behind a plexiglass barricade.