The Charles Sumner House in Boston’s Beacon Hill neighborhood is a National Historic Landmark, but the plaque marking the famous abolitionist’s birthplace is nestled into a brick wall several blocks away from the house where he lived for many years.

Charles Sumner was a United States Senator closely associated with the anti-slavery cause in the mid-19th century. After a particularly fiery (and personal) abolitionist speech on the Senate floor in 1856, he was severely beaten with a cane by an inflamed South Carolina Congressman. The attack is often considered a linchpin in the thickening pre-Civil War tensions between North and South.

Before Sumner’s cane-baiting political speeches, he embodied the life of a well-educated 19th century Bostonian. He was born and raised by abolitionists in Boston’s stately Beacon Hill neighborhood bordering the state capitol building, and he attended Harvard College and Harvard Law School. The brick townhouse where he resided for 30 years at 20 Hancock St. in Beacon Hill was designated a National Historic Landmark in 1973. However, the family didn’t move into this home until after Sumner had graduated from Harvard in 1830.

Less well-known than the building where Sumner spent three decades is the one in which he was born. About two and a half blocks southwest from the Charles Sumner house is a 2-foot oxidized plaque simply reading “Birthplace of Charles Sumner January 1811.” At the time of his birth, this section of Beacon Hill bordered the poor African-American section of the city. The Sumner family’s simple home welcomed Charles and his twin sister Matilda into the world on January 6, 1811.

The monument sits at eye-level on the building at 45 Myrtle St (now residential apartments), but takes a sharp eye to spot. It is slightly recessed into the wall and obscured by its weathered conditions.