William Jay (1789-1858), John Jay’s younger son, inherited his father’s house and farm in 1829. He married Augusta McVickar in 1812; the couple had seven children. William Jay developed the farm, served for many years as a Westchester County judge, and was vice president of the American Bible Society. He became a figure of national importance as one of America’s leading abolitionists. An impassioned foe of slavery, he wrote brilliantly and forcefully for its abolishment. After his death, his fellow abolitionist Frederick Douglass delivered a twenty-seven page eulogy in his honor.

John Jay II (1817-1894), William’s son, carried on the family tradition of public service. At the age of seventeen, he became a member of the New York Young Men’s Anti-Slavery Society. A trained lawyer, he defended fugitive slaves in court, lectured at anti-slavery rallies, and supported the Underground Railroad. He also advocated for the passage of an international copyright law. From 1869 to 1874, John Jay II served as U.S. Minister to the Court of Austria-Hungary. He was one of the early leaders of the Republican Party in New York, and a founder of the Union League Club and of the Huguenot Society of America. He married Eleanor Kingsland Field in 1837; the couple had six children.

William Jay II (1841-1915) was the next generation of the family to own the property. He was called “the Colonel” on account of his rank during the Civil War; he had fought at Gettysburg, Mine Run, Chancellorsville, and Appomattox. In civilian life he was a lawyer, specializing in real estate law. He was an avid horseman, and was a founder and first president of the Coaching Club. William Jay II married Lucie Oelrichs in June 1878, with whom he had three children. Col. and Mrs. Jay were fixtures in the upper class, Gilded Age set known as “the Four Hundred.”

Eleanor Jay Iselin (1882-1953) was the only surviving child of William and Lucie; her two sisters died of diphtheria before they were grown. Called “Moppy” as a child on account of her unruly hair, she married Arthur Iselin in 1904. The couple was the last generation of the family to use the Homestead as their principal home. Eleanor called it “Bedford House.” Her most distinctive contribution to the home was architectural. In 1924, ground was broken for a masonry wing that more than doubled the size of the house. In 1946, deeply moved by the tragedy of World War II, Mrs. Iselin offered the estate as a location for the United Nations, but her offer was not accepted. Following her death in 1953, the property passed to her children, who arranged its sale to Westchester County for donation to the State of New York, to become an historical museum memorializing John Jay and his family.