The original stone keep at Yester, built before 1267, is generally considered to be by Sir Hugo de Giffard. A grandson of the first Laird of Yester, he served as a guardian of the young Alexander III of Scotland, and was by repute a magician and necromancer, known by many as "The Wizard of Yester".

It was in the under-croft of the castle that The Wizard was thought to practice his sorcery. 14th century chronicler John of Fordun mentions the large cavern in Yester Castle, thought locally to have been formed by magical artifice. Legend also supposed that Hugo was able, via a pact with the Devil, to raise a magical army to his aid, and use them to carry out his will. It is this army of hobgoblins that was considered to be the builders of Yester Castle.

Little remains of the castle today, but the subterranean Goblin Ha' (Gobiln Hall) is still largely intact. The Hall is described on the Royal Commission of the Ancient Monuments of Scotland website as follows:

"On the W of the enceinte are the remains of another range of buildings now only a few feet above ground, and from the most northerly of these, a stair leads down to the "Goblin Ha'", an oblong chamber 37ft by 13ft 2ins, built of ashlar.

The "Goblin Ha'" may date from the 13th century and the western range was probably built towards the end of the 14th century, while the masonry of the enceinte walls would appear to be of 15th century date."