Constructed between the 12th and 13th centuries, the castle fortress overlooking the picturesque town of Olvera was built as one of a series of signal stations on promotories in the region. Lookouts would signal using flashes of light from one tower to another.

The castle was seized from the original Moorish builders and occupants and redesigned and expanded under King Alfonso XI in 1327.