The College and Convent of Saint Francis Xavier has had a remarkably multi-hyphenated career. It was inaugurated in 1751 as a Jesuit-founded school to ordain priests and instruct the local Indigenous people in Spanish and Catholicism, but has since served many other purposes.

After it was abandoned by the Jesuits and left to fall in neglect, the Mexican government's expropriation of religious properties led to its acquisition in 1834, when it was turned into a State Penitentiary. Its prison remodel was designed by well-known Poblano architect and urban planner José Manzo, based on the then-cutting edge Cincinnatti Prison. After many stops and starts caused by an earthquake and foreign invasions of Puebla—first by the United States and then French troops, during which the building was used as army quarters—the prison opened in the 1890s.

After a few years exclusively as a prison, the building continued its identity crisis for nearly a century, going back to being used as army quarters in addition to a forensic laboratory, warehouse, epileptic, and military hospital. In 1984, it was finally decommissioned as a prison, and a public vote opted to transform it into a cultural center.  

Arts workshops and exhibits followed for a few decades before the center had its latest makeover. In 2017, the Centro Integral de Servicios San Javier (CIS San Javier) reopened. As of July 2019, the center offers public services such as water bill inquiries and issues driving permits to locals on behalf of the state government, as well as continues to house the State Archives. 

Of course, in a city whose many legends and tales have given it a status that is, well, legendary, such a storied location could not go without a local ghost or two. It is said that the government workers have heard everything from admonishments from a nun of the convent era to the cries and confessions of the prisoners of the former jail.