The developers who tore down the iconic house called the public response “a bummer.”