This is Part II of a two-part series on Sylvester Manor that traces the fortunes of 15 generations of one family and how it parallels the history of Shelter Island and Northern slavery.
Following the deaths of Nathaniel and Grizzell Sylvester, the first generation of the family to own the Manor, their oldest son Giles took possession, becoming among the largest landowners and largest slave owners in the area. But Giles’ business skills weren’t what his father’s had been and he died in debt with the Manor lost to creditors.
His son Brinley stepped in and through lengthy litigation reclaimed the Manor. Working with Captain Benjamin L’Hommedieu, he successfully raised sheep and returned to exporting goods to the West Indies resulting in a restoration of the family’s wealth. Brinley entered politics, becoming a town supervisor, tax assessor, “overseer of the poor” and port collector. It was the last post that added to his wealth: He received a fee for every slave imported.
It was during this period that Jo, a West African slave, came to work for Captain L’hommedieu. In papers signed by Jo’s mother, the family was promised that in return for his service at Sylvester Manor until he was 21, Jo would be taught to read and write and learn a trade as a cooper, making casks for wine, flour, tobacco and other commodities.
It was a promise broken.
As the number of slaves increased in the North, their living conditions deteriorated. Despite reaching 21, the age at which his servitude was to have ended, records show that Jo remained at Sylvester Manor and was there 50 years later, known as “the old man,” when he died.
With Brinley’s death, Sylvester Manor went to his daughter Mary and her husband Thomas Dering after he went bankrupt. There, Mr. Dering began his own political dabbling and befriended another L’Hommedieu, Southold lawyer Ezra.
While continuing to sanction slavery, the Manor’s owners chafed under England’s heavy-handed rule, siding with Patriots who wanted independence. During the Revolutionary War, they saw much of Long Island under a brutal British occupation. By the time the war ended in 1783, George Washington would describe Long Island as “in a state of wretchedness.”
Determined to turn things around after the war years, the Brinley and L’Hommedieu sought, but never got support from, Thomas Jefferson to help restore farming.
They could have used some help since Shelter Island underwent further devastation by a crop-destroying Hessian fly invasion that took several years to abate so farming could begin anew.
Following the Revolutionary War, slavery continued despite rumblings about its immorality. Eventually, New York State “permitted” its voluntary end, but it took another 34 years to abolish slavery in the state by 1827. Some slaves secured their freedom early on and even acquired property, but most continued to work on their masters’ land.
During this period, Mary L’Hommedieu Gardiner and her husband Samuel acquired the Manor and by the mid-19th century, they had left it to their daughter Mary and her husband, Eben Norton Horsford, whose fortune lay not in farming the land but in his development of baking powder.
Mr. Horsford, Harvard University chemistry professor, worked with others on his product and formed the Rumsford Baking Powder Company in Rhode Island. The couple turned the responsibilities over to a farm manager and crew and the manor became the center of social life on Shelter Island, with trains and steamships bringing tourists and summer residents to the Island. Mr. Horsford sold off hundreds of acres of land intended to be developed as summer houses, weekend farms and elegant hotels.
He favored the Methodist Church that established a summer camp here and he encouraged wealthy hotel investors who founded Manhanset House.
By 1876, when the young country was engaged in patriotic celebrations, the Horsfords had their own more personal project — the chronicling of a family history, led by Cornelia Horsford, who later inherited the Manor. She collected family mementos and stories and restored the house and grounds. The family wanted not only to recognize its own history, but the history of all those who had inhabited the land, including its slaves.
A “rough, unfinished boulder installed to acknowledge the ‘Colored People of the Manor,’” was set down, but Indians and Africans played “mere supporting roles before being relegated to the mists of time,” according to the exhibit.
Fast forward to 2010 when Eben Fiske Ostby, the 15th generation to own Sylvester Manor, established a private nonprofit organization to preserve the Manor and grounds. The fields are now being planted “with an eye toward creating a sustainable future . . . and remembering its complex past.”