Originally built in 1868 by the New York newspaperman and land developer Lloyd W. Tomkins, this structure was a two-story home. But after David Brinkerhoff Ivison, president of the American Book Company, bought it in 1887 he turned the building into a three-story turreted mansion with 25 rooms including a music room and a carriage porch. In the 1930s, an indoor pool was installed on the second floor, the walls of which can still be seen in parts of the castle. 

Ownership of the building changed hands several times over the years, and in the 1970s, Fairleigh Dickinson University bought the castle and build a campus around it. The building was used as classroom space, and much of the original interior was covered with drop ceilings and partition walls. Fairleigh Dickinson operated on this campus from 1942 to 1994, when it outgrew the campus and sold it to Felician University (Formerly Felician College) in 1997. 

When Felician University moved into the campus in New Jersey, they set out to restore the building to its former glory. Today it houses many of the school’s administrative offices. The music room now serves as a chapel for the Catholic university’s weekly masses. Renovations were completed in 2013, and Felician welcomes visitors to the historic site (which was added to the National Register of Historic Places in 2004).