From failed port town to Civil War fort, the ruins of this former defense saw their finest day in the television age.