Neue Galerie New York opened in 2001 in a distinguished home: a 1913 Beaux-Arts mansion designed by Carrère and Hastings, one of the preeminent architectural firms of its era. The house was commissioned by William Starr and Edith Warren Miller, both members of old and established American families. It was later occupied by Mrs. Cornelius Vanderbilt III. The mansion was purchased in 1994 by Serge Sabarsky and Ronald Lauder, with the intention of turning it into a museum for German and Austrian art. Architect Annabelle Selldorf was given the considerable task of transforming the building into a modern museum.

Architectural historian Andrew Dolkart recounts the colorful history of the building on its journey from mansion to museum. In the process, he illuminates the social changes of New York over the course of a century.