Reclusive millionaire Oliver Hazard Payne was as wealthy as he was enigmatic. The eight years this bachelor lived in West Park left a legacy that lingers today. This is a reprint of an article that we published in 1992.
Monasteries, apartment complexes and ranch houses line Ulster County’s west bank of the Hudson today. But once upon a time, some of the richest men and women in the world lived along what is now Route 9W. West Park sported residents like John Jacob Astor and Delores Ponce de Leon, daughter of the governor general of Cuba. The biggest mansion of them all belonged to an enigmatic recluse named Colonel Oliver Hazard Payne; today it’s tucked behind a small sign labeled Marist Brothers.
When the Colonel, as the people of West Park still refer to him, died in 1917, the only thing locals knew was that he was treasurer of Standard Oil. Payne’s holdings in that company were second only to John D. Rockefeller. Although he was childless, Payne’s eight years in West Park would have a far-reaching legacy, touching a multitude of lives: from the fabulously wealthy Whitney family to the son of John Burroughs to a future world heavyweight boxing champion.
When Payne was 70, he bought the former Astor estate, coupled it with another and commissioned society architects Carrère and Hastings to build him a massive Mediterranean palazzo.
Though Payne was actually a brigadier general, he preferred his more modest rank. Payne left Yale to fight for the Union Army during the Civil War. He was from a tightly knit, patriotic Ohio family; his father became a senator, and his brother became the mayor of Cleveland. Another relative penned the tune “Home Sweet Home.”
Dr. Karl Berntsen, a professor emeritus at Cornell Medical College, spent time in Payne’s ancestral Cleveland home, sifting through the Colonel’s letters. Payne’s multi-million-dollar benefactions created the medical school at which Berntsen taught.
Payne was shot from his horse during the Battle of Chickamauga. No one knows exactly where the bullet went. “Some people think he got shot in the genitalia, which might explain why he never had any children,” says Berntsen.
In any case, Payne was extremely modest, before and after the bullet. During the Civil War, he never allowed himself to be photographed; he would only submit to a sketch artist, and even then only in profile. “He viewed himself as unattractive,” Berntsen explains. “But Payne was very popular at all stages of his life, especially as an officer.”
Heading through Illinois, Payne allowed his troops to go back to Ohio and recruit their friends. Membership in the 124th Ohio Volunteers grew rapidly. “I am the idol of my troops,” Payne wrote his brother. “Payne wanted to do things right, and he did them,” says Berntsen, who has taken a liking to his research subject. Such attention to detail was noticed by one general, who termed Payne’s camp “a model of sanitation.”
Payne’s scrupulous nature was firmly formed by the time he went to prep school. He voluntarily mailed his father itemized expense reports, justifying the whereabouts of each dollar.
The Colonel wasn’t the most academic of the Paynes; his sister was. Flora Payne studied at Cambridge in special classes conducted only for women. She was a compulsive writer, well-traveled, outspoken and sharp-tongued. When Payne was wounded in the war, his sister wrote him, “You darling Oh – I hardly know which to write to, you or the bullet in you.” “Flora has a fine, masculine mind,” Payne wrote later about his favorite sibling.
Payne had so much affection for his sister that he fixed her up with his handsome Yale chum William C. Whitney. As a wedding present, he built them a house on Fifth Avenue in Manhattan, and gave them a million dollars in startup capital. Payne could spare the money: The man who was called affectionately “my little captain” by the soldiers he commanded emerged from the Civil War with heroic status and $20,000 from his father.
Just at the beginning of the acquisitive age of the Robber Barons, he became an industrialist. Payne got his feet wet with iron manufacturing and ended up in oil refining. After Rockefeller bought out his oil concern, Clark, Payne & Company, he became a ground-floor associate at Standard Oil. It was said that Payne and a handful of others controlled the American tobacco industry and helped start US Steel.
Payne eventually moved his base of operations from Cleveland to New York, where he spent a good deal of time with his sister Flora and her family. Although Payne loved children and lavished gifts upon Flora’s brood, he never married. He had already announced at the age of 18 that he would be “eventually, from choice and not necessity, obliged to choose an old bachelor’s life.”
In a strange turn of events, Payne did become a bachelor father. Flora Whitney died of heart trouble in 1893. Her husband remarried three years later, and Colonel Payne and William C. Whitney never spoke again. Although it was done in a hush-hush manner, Payne convinced two of the four Whitney children to turn their backs on their biological father and become Payne’s heirs.
Payne was very generous with his charges Payne and Pauline Whitney. When Payne Whitney married Helen Hay, the daughter of Secretary of State John Hay, the Colonel gave jewels to the bride and a home on Fifth Avenue to the newlyweds. He also granted them the use of his Georgia mansion, private railroad car and yacht.
No one knows why Payne and Whitney’s once-warm friendship turned into a cold-blooded feud. Some biographers have insinuated that Payne found out about a transgression between Whitney and a notorious burlesque actress. Others say that Payne thought Whitney’s remarriage showed a lack of respect for the deceased Flora.
Author Gore Vidal, in his best-selling novel Empire, insinuated that Payne fell in love with his brother-in-law and onetime Secretary of the Navy, Whitney. “This was a supposition on the Colonel’s treatment of Whitney, who was very handsome. It sounds to me like chagrin d’amour,” says Vidal, adding, “Whitney is the grandfather of my brother-in-law, Michael Whitney Straight.”
The archivist at the Cornell Medical Center in New York, Adele Lerner, points out that there is no hard evidence in the historical record to back Vidal’s claim, but that could be attributed to an era of extreme homophobia.
Payne went to great pains to keep his personal life out of the public eye, Lerner adds. He was “a tall, lean, quiet man who despised publicity and often made his gifts under the condition that he remain anonymous,” she explains, reading from a 1992 Cornell Medical College newsletter on the Payne/Whitney legacy.
Pulitzer Prizewinning biographer William A. Swanberg touched on the Payne/Whitney relationship in his biography, Whitney Father, Whitney Heiress. When Whitney collapsed in Washington in 1887, Oliver Hazard Payne hurried to his bedside. “He [Payne] was militarily erect, decisive and to outsiders so formal and possessed of such seeming hauteur that Henry Flagler, another Rockefeller partner, described him as kin to God,” Swanberg writes.
“He epitomized the Payne family solidarity,” Swanberg writes. “He loved his sister with an affection so tender that some – perhaps unfairly – were later to question its normality.”
West Parker Elizabeth Kelley Burroughs, granddaughter of naturalist John Burroughs, remembers Payne as a warm and open man. “Everybody said when the Colonel was young he was engaged,” Burroughs recalls, “but his fiancé died.”
Payne also owned a castle in Scotland, a mansion in Thomasville, Georgia, an estate named Paynehurst in Bermuda and a house on Fifth Avenue.
His West Park mansion was faced with imported French limestone and built around a huge central courtyard lined with frescoes. The mansion’s 40-plus rooms were walled in rich Circassian walnut, leather and ebony with gold tracery. There were greenhouses, formal gardens, huge barns, a gatehouse and a stone boathouse, where Payne’s Aphrodite, the largest yacht in America, was moored. Payne brought his Ming porcelains and Rubens and Turner paintings to West Park. He brought in tapestries, bronzes and busts. The Colonel spared no expense on his Hudson River villa.
In 1913, Burroughs’ father, Julian Burroughs, became the superintendent of Payne’s West Park estate. Since Julian Burroughs was selling Victrolas at the time, the job offer at the Colonel’s – which included a house, motorcar, maid and chauffeur – came as somewhat of a surprise. Andrew Mason, the estate’s previous superintendent, died either in what the Town of Esopus Story refers to as an automobile accident or from what Elizabeth Burroughs recalls as blood poisoning. The Colonel’s right-hand person, a Swedish woman named Miss Larson, adopted Mason’s orphaned daughter. Larson convinced the Colonel to hire Julian. “It was like a fairy tale,” Elizabeth Burroughs says. “I was so happy to grow up there and know beauty of that sort.”
From all reports in West Park, the Colonel was very generous. He sent Elizabeth Burroughs and her sister Ursula to Newport on his yacht. He gave Miss Larson the go-ahead to buy them expensive Fifth Avenue dresses and lunches at New York City’s Clermont. He bought them toys from F. A. O. Schwartz, magic lanterns and slides. The Colonel and their brother, John Burroughs, shared a birthday, so every year they shared a cake. Payne also gave the Burroughs boy a pedigreed Sealyham puppy from his nephew Payne Whitney’s kennels. Emily Burroughs, the children’s mother, received a bow-shaped diamond brooch from Tiffany’s after the Colonel noticed she didn’t wear a lot of jewelry. He also lent her his box at the Metropolitan Opera House.
But Julian Burroughs was by far the largest West Park beneficiary. “Those were my father’s best years,” recalls Elizabeth Burroughs with a bittersweet expression. “At Riverby [the family farm where Julian’s naturalist father John lived], it was always, ‘My boy.’ He was always dictated to by my grandfather.”
Julian designed large barns, a poultry farm, a boathouse and a dock for the Colonel. He commissioned custom-made iron gates, stone engravings and tile mosaics. Payne had total confidence in his inexperienced superintendent and never quibbled over price. “I would go into the library to see the Colonel and he would ask me how much money I needed,” Julian wrote. “Often I’d say 15 thousand and he’d write me out a check for 20.”
But the Colonel’s architectural firm from New York City had problems with the young superintendent. They disapproved of his sketches and told the Colonel that they would never dream of having anyone other than themselves design a building for his estate.
Payne had reason to doubt the competence of his society architects. The Hastings-designed central courtyard at times collected ten feet of snow. The imported French limestone couldn’t handle the Hudson Valley winters. “When I first went there, the limestone around the cellar windows was all crumbling,” Julian wrote. “So we cut out replacements of our native stone, which are good to this day.”
Payne’s affection for native stone and native folk is still remembered. Although he didn’t attend the nearby Ascension Church, the Colonel had it wired for electricity and built the sexton’s house. Payne threw a wedding reception for one of his servants on board the Aphrodite. Every year his employees sailed to West Point and enjoyed an onboard picnic. “Those were good days for West Park,” Elizabeth Burroughs says.
At the annual employees’ Christmas party, Payne gave out gold pieces to the children. Esopus resident John Mowell received a few of those coins when he was a boy. His father was a farmhand at Payne’s, and his grandfather worked in the estate quarry. “They also had a moving picture at Colonel Payne’s, where the Marist Brothers’ chapel is today,” Mowell says. “You could go to the movies every week if you wanted to.”
Mowell’s grandparents lived in a group of stone structures built to house the Payne estate workers. Because of their quaint design, these buildings on the western side of Route 9W were known as the English Village. Today they’re called Black Creek Apartments.
Mowell’s neighbor, Amy Markle, also remembers the glory days of Colonel Payne. One day, she took a peek inside the “Big House.” “There were solid-gold-handled doorknobs,” Markle recalls, “and a room was papered in silk that came from Japan. Money was no object. Everything was so big.”
Markle didn’t work on the estate; she only knew people who did. But Payne’s reputation preceded him. “The Colonel was sort of an eccentric, I heard,” says Markle. “He spent all this money on the place, and his nephew didn’t want it.”
Colonel Payne died June 27, 1917 at his home on 852 Fifth Avenue. He was 78 years old and had outlived his four siblings. The bulk of his huge estate went to his favorite nephew, Payne Whitney. The niece, Pauline Whitney Paget, who sided with the Colonel in his dispute with William C. Whitney, died of heart trouble in 1914.
To keep his will from being contested, the Colonel had to leave something to his estranged nephew, Harry Payne Whitney, William C. Whitney’s eldest son. Payne willed him a single painting that Harry had once praised. It was Turner’s Juliet and Her Nurse. Dorothy Whitney Straight, Harry’s other sister, didn’t get a penny.
Payne left the 645-acre Esopus estate and two million dollars to his other sister’s son, Harry Payne Bingham of Cleveland. When Bingham and the first of his three wives, Harriette Gown Bingham, moved into town, the trouble started. “It is almost a universal truth,” Julian Burroughs wrote, “that women like to change their surroundings, but men do not. Young Mrs. Bingham wanted to change everything.”
Mrs. Bingham “told Burroughs to hand over his plans for the unfinished barns to her architects. She told the workmen to destroy the four-by-five-foot stone engraving that Burroughs had designed with the Latin phrase Horae pereunt et puniunt, which means “The hours perish and punish.” She had a Bingham family crest put in its place.
The Burroughs family left the estate after less than a year, but not before they witnessed Mrs. Bingham’s renovations to the mansion. The gold tracery was chemically removed, and the ebony walls were painted a putty color. The long, custom-made silk curtains in the drawing room were dyed. “She was an utter fool,” says Elizabeth Burroughs. “She had so little knowledge about art. She really just wanted to be with her friends on Long Island.”
When Bingham couldn’t get his property taxes reduced, he gave the West Park estate to the Episcopal Diocese of New York in 1933. He was a Presbyterian at the time. Hundreds of workers in West Park had to find new jobs. Slowly, they cleared out of the English Village. “Dad worked there until the place closed up,” recalls Mowell. “Then he went to Hercules. Everybody was laid off. Mr. Buchanan, the man who was in charge of the chickens, was left with a brooder house and a thousand half-grown chicks. I think the estate must have turned them over to him.”
Before Bingham left, he donated a good deal of the Colonel’s art collection, including Rubens’ Venus and Adonis, to the Metropolitan Museum of Art, where, coincidentally, he became director. Bingham also became a director of the First National Bank of New York and an avid collector of marine life. The Bingham Oceanic Foundation at Yale was established with more than 3,000 of his specimens.
The Episcopalians turned a portion of Colonel Payne’s estate into the English Village for Convalescents. “They were sort of the nutty kind,” recalls Mowell. “I came home one night and saw a patient playing golf in the middle of 9W.” The place never caught on.
The Episcopalians started an experimental summer camp for delinquent and homeless kids from New York City in 1937. The big stone barns were turned into dormitories, and teepees lined the slopes by the Colonel’s former chicken farm. The 20 boys in the pilot program benefited so greatly that the camp caught the attention of Eleanor Roosevelt, Pulitzer Prizewinning author James Agee and the State Department of Welfare, which bought the camp from the church and turned it into the Wiltwyck School for Boys in 1942. The English Village was renovated into three dormitories, a dining hall and a gymnasium for the Wiltwyck School’s interracial, non-sectarian population of 8-to-12-year-old kids. Although the kids were placed in West Park by the courts, there were no bars at Wiltwyck.
It has been more than 35 years since former heavyweight champion Floyd Patterson walked the grounds of the old Wiltwyck School. At this point, he’s not sure exactly where it is. “Is it that left up there?” he asks hesitantly as his air-conditioned Lincoln heads up Route 9W.
Patterson’s squint widens with recognition as he pulls into Black Creek Apartments. He starts naming dorms, pointing to where the kids lined up for dinner, remarking that this was the first place he tried on boxing gloves. He spent one of his two Wiltwyck Christmases at Eleanor Roosevelt’s house in Hyde Park. He didn’t realize who she was. “To me she was just a nice lady,” he says.
Patterson came from a large, poor Bedford/Stuyvesant family. He ran away from home a lot, slept under the subway and stole at night. “I was a delinquent,” he says. “I had been in court a hundred times. The judge even knew my name. ‘What’s it now, Floyd?’ he used to say.”
Patterson came to Wiltwyck in September 1945. He and another kid stole a neighbor’s rowboat, paddled to Poughkeepsie and jumped a train to the City. The cops brought him back to West Park in three days. There he met a teacher named Vivian Costen, one of the most important people in his life. Every Friday, Costen asked her Wiltwyck pupils a question. Whoever answered correctly first got a bag of candy. One day, Patterson thought he knew the answer. “Oh no, don’t answer. You’re way off-key,” he said to himself. No one won the candy that day, and Costen told them the answer that Floyd already knew. Patterson ran out of the classroom crying, only to be followed by Costen. “I know you knew the answer, Floyd,” she said, giving him the bag of candy.
A little positive feedback went a long way, and Floyd started raising his hand like crazy. Costen had a soft spot for him. Once in a while, she let him stay at her place in New York City, where she took him to movies and restaurants. Floyd never forgot her. He gave her tickets to all his New York fights. She saw him win the Olympics at the age of 17. But she died before he became world heavyweight champion.
Patterson received his first dose of nature at the Wiltwyck School. Now the middle-aged Patterson wrestles with the brambles by Black Creek, navigating a log and jumping up to the dam with a boxer’s grace. He can’t hide his disappointment: The old swimming hole dried up a long time ago.
Further through the woods. he stops abruptly in awe, gaping at the huge, ivy-encrusted stone barns, the buildings that Julian Burroughs was working on when Colonel Payne died. To Patterson’s childhood memory they are “the castle.” “I always thought, ‘If I ever make it in this life, I’m going to live in a place just like this,’” Patterson says, eying the bucolic splendor around him. Eventually, he moved to New Paltz, where he continues to live today with his family. His adopted son, Tracy Harris Patterson, is the super bantamweight champion of the world.
The Episcopalians sold the rest of Colonel Payne’s estate to the Catholic Church in 1942. The Marist Brothers made a few renovations when they moved in with their religious preparatory students. The frescoes of the Muses, Pegasus and a youth lining the central courtyard were covered with paint. “The murals outside weren’t quite appropriate for what we had in mind, what with the nudes and all,” says Brother Stephen Martin. “So we painted over them, which was our prerogative.”
The Marist Brothers transformed the mansion’s Great Hall into a chapel, and made the other rooms dormitories and classrooms. They built cinderblock extensions to house future novitiates in 1952 on the huge Payne-era garage and electric plant. But the number of brother aspirants dwindled, and the West Park Marist Brothers ran out of students. The property became a retreat house for handicapped and mainstream students from New York City in 1971.
When Brother Martin took a group of high school kids on a field trip to Hyde Park’s Vanderbilt Mansion, the students ran to catch the river view. “‘That’s our mansion,” they screamed.
Due to financial considerations, the Marist Brothers sold the mansion and the boathouse to another reclusive millionaire, Raymond Rich, in the late 1980s. The brothers moved their activities over to Colonel Payne’s garage and power plant. “The opulence seemed almost inappropriate in today’s day and age,” Brother Martin says. “That’s not the way we Brothers live. When I used to visit my family during the summer, my room felt like a bandbox. Forget what happened to the brothers that came from apartments.”
Karen Merker was there in the beginning. She came up with her high school class from St. Helena’s in the Bronx in 1972. Today Merker teaches third grade at St. Joseph’s Elementary School in Kingston during the week and returns to the Payne estate every weekend to cook for hundreds of retreat guests. She remembers what it was like to slave away in Colonel Payne’s kitchen, where a few amenities from a bygone era remained: a two-floor scullery with a spiral staircase, a dumbwaiter and an elevator where the Brothers stored brooms. A speaking tube that once connected the Colonel in the dining room to the kitchen was there as well.
The intricate wooden carvings upstairs in the Great Hall intrigued Merker most. “I remember putting my finger in the mouth of a carved lion and there were ridges inside,” she says. “They went down to the last detail.”
Every week, Merker crossed an immense wrought-iron threshold with her bags of groceries. The door was so heavy, she recalls, that it literally ushered visitors inside. “I used to pretend that the mansion was my house and that 200 guests were coming for the weekend,” she says in her best mock-patrician voice.
Today, the Marist Brothers use the Payne buildings and grounds that they still own to host children each week during the summer. Some of the kids have terminal cancer. Others are developmentally disabled. “It was nice to have a big, beautiful mansion,” Merker adds, “but the spirit of the work is the real draw.”
The Colonel’s stone barns enjoyed a brief incarnation as a restaurant called the Creamery and a confederation of quaint gift shops called Wildwycke Village. A microfilm service for hospitals took up residence in the Colonel’s chicken farm, but it too has relocated.
The Colonel’s legacy continues to ripple down through the ages. In an ironic footnote, the Turner painting that he gave his estranged nephew, Harry Payne Bingham, sold for $6.4 million at auction in 1980, fetching the highest price for any work of art at the time. “I’m just staggered,” said the owner, Flora Whitney Miller.
The Whitney family continued to prosper, in part because of the Colonel’s wealth. Payne Whitney and his descendants continue to give heavily to the Cornell Medical College. Subsequent Whitneys have other success stories, including ownership of the New York Mets.
Miss Larson was set up with millions in stocks by Payne before he died. But, according to Elizabeth Burroughs, she was deceived by swindlers and left only with the house in Staatsburgh that the Colonel had helped her purchase and the more modest income that Payne’s heirs gave her.
The legendary Aphrodite had an even sadder fate. Not long before the Colonel died, he offered his favorite possession to the US Navy for wartime use. She was painted grey, and the contents of her Stanford White-designed interior were stored in a Long Island warehouse. Vincent Astor, whose great-uncle had owned Colonel Payne’s West Park estate before him, served as a junior officer on board. But while the Aphrodite was away, the Long Island warehouse burned, taking the Colonel’s memories with it.
Join the family! 
