The Touch of Tragedy

Library of Congress

George Washington Custis Lee (left) stands beside his father, General Robert E. Lee, and his father’s aide, Walter Taylor, in a photo taken shortly after war’s end.

It wasn’t easy being Robert E. Lee’s first child. George Washington Custis Lee—called Custis by the family—moved in prominent government and military circles during the Civil War, rose to the rank of major general, and took part in the last major campaign fought by the Army of Northern Virginia. After the war, Custis Lee served for decades as the president of Washington and Lee University, the “Lee” being his father. But he found little joy in or outside his work and was a lifelong bachelor. More than his two younger brothers, Custis bore the burden of being the son of the South’s greatest general.

Custis Lee was born at Fort Monroe in Virginia on September 16, 1832. His father and Mary Anna Randolph Custis had married the year before, and he was named after his maternal grandfather, George Washington Parke Custis, the owner of Arlington House plantation. His parents called him “Boo” as a child, a name that stuck until he was a teenager. His mother said he was the “happiest little creature you ever saw,” and the Lees lavished the kind of attention one might expect on a firstborn.¹

Library of Congress (Lee); National Portrait Gallery

Robert E. Lee (left) and George Washington Parke Custis

In time, the attention became overbearing. In 1837, writing from St. Louis, Missouri, his father admonished 5-year-old Custis for his “obstinacy,” demanding that his son “must comply” with his mother’s wishes.² In 1845, Lee was stern with his son after Custis’ brother Rooney had lost fingers on one hand in an accident. His father warned of “the fruits of [Rooney’s] disobedience,” adding that Custis hopefully would “never know the misery I now suffer.”³ Perhaps having known so little of his own father (the famous Revolutionary War officer Henry “Light-Horse Harry” Lee), who died when Robert was 11, he wished to be a conscientious parent. Custis, however, was never given tools to carve his own path.

Robert E. Lee and his wife, Mary, were what we might call today “helicopter parents.” A man with his own long list of regrets, Robert once told Custis that his greatest pleasure came from his son’s “good conduct & progress in learning.”⁴ To assure that “progress and learning” took place, Custis was given a good education—first at a school for classical studies in Alexandria headed by Reverend George A. Smith, then to study mathematics at Benjamin Hallowell’s school, which his father had attended. Custis proved an exceptional pupil.

Before Robert E. Lee left for the Mexican War, he wrote a will naming 14-year-old Custis as the executor of his estate. When Lee returned, he had no illusions about combat. “You have no idea what a horrible sight battle is,” he told his son.⁵ At various points in his life, Lee expressed regret at pursuing a military career.⁶ Nevertheless, Custis received an appointment to the U.S. Military Academy in 1850. He got in trouble his first year, when liquor was found in his room. He claimed the alcohol belonged to his roommate, but the authorities were not convinced and he might have been expelled had the secretary of war not overturned his conviction.

Robert E. Lee set a figurative and literal example for his son. While Custis was at West Point, his father was appointed the academy’s superintendent. Custis aspired to meet his father’s high standards and in 1854, graduated at the top of his class (his father had finished second in his), and like his father, with no demerits on his record.

Again, like his father, Custis joined the Corps of Engineers. His first assignment came in the spring of 1855 to Fort Clinch on Amelia Island, northeast of Jacksonville, Florida. The place is at the southern tip of the Sea Islands, which run into Georgia and South Carolina. Today, the fort is known for its recreation and natural beauty, but Custis complained back then about the heat, sparse population, pestering mosquitoes, and “sharks in abundance.”⁷

His next post, in 1857, was even farther away, to forts in San Francisco, California. At first, Custis was “quite disappointed” in the area, noting that the “streets and sidewalks are paved with boards: its most remarkable features at first sight are the absence of hoops [women] and the number of Chinese in the streets.” But he came to enjoy the mild climate, writing home that he was “very well pleased” to be there.⁸

Harper's New Monthly Magazine

Before his death in 1857, George Washington Parke Custis had named his grandson, Custis Lee, the inheritor of his estate, which included Arlington plantation (shown here in an illustration from 1853).

That October, George Washington Parke Custis died in Virginia. Over the course of his lifetime, he had owned nearly 200 enslaved people. At the time of his death, he had three plantations in operation, with Arlington being the most prominent. He named Robert E. Lee the estate’s executor and his oldest grandson, Custis, the estate’s inheritor. Custis asked his father to administer the estate in his place, wishing him to have “unlimited control over the whole property.”⁹ Running Arlington caused Robert E. Lee many headaches; he was no plantation manager and had difficulty turning a profit and governing the slaves. As Lee wrestled with the tangled threads of the Arlington estate, he hoped to get Custis appointed an engineer in the Washington, D.C., area. Eventually, Lee’s military duties called him away from Arlington to a post in the wilds of Texas.

Custis called California “charming country,” but complained of being “unbearable” and possessing a “peevishness due to my ill health and number of years, and carry about a face so gloomy as to stop the barking dogs.”¹⁰ His father, chasing Comanches across the Texas plains, was no better off. In October 1859, Robert E. Lee sadly wrote that the only pleasure he got was from his children (Custis was the oldest of seven). Custis by then had returned home, working in the Engineering Bureau in the District of Columbia and taking charge of Arlington. His mother complained that he was “very thin.”¹¹ And like her, Custis suffered from rheumatism.

National Portrait Gallery (Rooney Lee); Library of Congress

Rooney Lee (left) and Jefferson Davis

When civil war broke out, Custis sided with his family and the Confederacy. While he had been groomed his whole life for the army, his younger brothers would see far more combat than he did. Robert E. Lee Jr. was at the University of Virginia when Virginia seceded, and joined the army in 1862 and later served as a cavalry officer. William Henry Fitzhugh Lee, nicknamed Rooney, who had not gone to West Point, became a general and one of the best cavalrymen in the Army of Northern Virginia.

Custis continued doing what he did best: engineering, following orders, and not making trouble. His work on the fortifications near Richmond caught the attention of Jefferson Davis, who made him an aide-de-camp. Custis would have preferred a field command, but, as South Carolina diarist Mary Chesnut—then living in Richmond with her husband, an officer and aide to Jefferson Davis—wrote, he “must stay where he can do the most good.”¹²

For the first year of the war, Robert E. Lee served in Georgia, South Carolina, and western Virginia. While his work in Richmond did not have the excitement or prestige of a combat assignment, Custis was comfortable in the Confederate capital. He could assist the president, manage family affairs, and take care of his ailing mother (who was forced to move often during the war). The chaos of war, nevertheless, often upset the Lee family’s plans.

In early 1862, the Confederacy suffered major reverses in the West at Forts Henry and Donelson and at Shiloh. In the East, George B. McClellan was slowly moving a large Union army toward Richmond. In May, Custis was not sanguine about the situation. “We have shown but little enterprise and activity in this war so far,” he wrote his mother in May, “and not enough upon our own exertions.”¹³ By late June, though, Thomas J. “Stonewall” Jackson had won a series of inspiring victories in the Shenandoah Valley. The recent Confederate military draft swelled the ranks of the Army of Northern Virginia, now commanded by Custis’ father. In the bloody Seven Days fighting outside Richmond, Robert E. Lee drove back McClellan’s forces and reversed the Confederacy’s fortunes in Virginia.

The momentum shifted again in September when Robert E. Lee was forced back into Virginia following his failed invasion of the North and his bloody repulse at Antietam. Custis’ work took him out of Richmond around that time. His father had suffered a bad fall from his horse and needed assistance. Custis was with him and observed the Army of Northern Virginia’s retreat. He gave President Davis a sober assessment of the troops. “The men of our army have been in a measure worn out by long marches and hard fighting,” he noted, “and are poorly clad, with many of them without shoes.”¹⁴

Custis was a conscientious adviser and administrator, but he had little interest in rank or advancement. He had his father’s good looks and calm demeanor, but not his ambition and personality. Shy by nature, Custis apparently had no desire to deal with the stresses and politics of field command.

Virginia Military Institute

Shy by nature, Custis Lee (shown above in a wartime photo) did not share his famous father’s ambition. Of the Confederate war effort, he wrote his mother in 1862, “We have shown but little enterprise and activity in this war so far, and not enough upon our own exertions.”

His natural reserve may have shown in his timidity around women. In the summer of 1863, Custis was briefly engaged to Sarah “Sally” Warwick, the teenage daughter of a planter friend of the family. Custis’ father was dubious of the match, saying “[Sally] is as sweet as ever but I do not think has the least notion of marrying Custis.”¹⁵ He was correct in his assessment. The relationship didn’t last. In 1866, Sally married Major Richard Poor of Baltimore.

In June 1863, Custis was promoted to the rank of brigadier general, though he did not officially accept the promotion. He signed documents as a brigadier general, but this was merely to “give more weight to my authority & more dignity to the Local Defense Forces” he commanded.¹⁶ Custis might have been unwilling to accept his higher rank, but he took his duties seriously, drilling Richmond clerks who had never seen combat as if they were veterans.

In early 1864, Ulysses S. Grant, newly arrived in the eastern theater, was preparing for his spring offensive. Robert E. Lee needed every enlisted man and officer he could get. He wrote his son: “I think now is the time for you to take the field in some capacity.” Lee said his son had the “intelligence, energy, strength & the independence of the country at heart.”¹⁷ The preceding years had taken a great toll on generals, and Custis seemed a good candidate for promotion. Davis offered him command of Confederate forces in western Virginia, but Custis declined. His father, who had had little success in western Virginia early in the war, agreed with Custis’ decision. He wanted Custis to join the Army of Northern Virginia as an engineer. Davis blocked the move. “Our intercourse has been so pleasant and your service so very useful to me,” Davis wrote Custis, “that I should feel a two fold reluctance in parting from you.”¹⁸

So, rather than commanding a major combat force in 1864, Custis continued his usual duties of shoring up Confederate defenses and scouting enemy positions. He spent time between Wilmington, North Carolina, and New Market and Chaffin’s Bluff in Virginia. He commanded the Richmond forces during the controversial Dahlgren Raid—a failed attempt to free some 13,000 Union prisoners being held in the Confederate capital—in March 1864. Custis might have led a Confederate effort to free Rebel prisoners at Point Lookout in southern Maryland, but that plan was never carried out.

In October, Custis was promoted to major general and in December, he asked to be relieved of his duties in Richmond. He joined his father’s army in Petersburg for the spring campaign and was there, commanding an undermanned division, at the Confederate defeat at Sailor’s Creek on April 6, 1865. Custis was captured and held briefly by the Union army. In his report of the battle, Custis wrote, “my command was entirely surrounded” and that “to prevent useless sacrifice of life” he surrendered it.¹⁹ His brother Rooney noted that Custis “fought with a gallantry never surpassed” and that the army’s “defeat and surrender were inevitable.”²⁰ Fellow Confederate officers Richard Ewell and Robert Stiles praised Custis, Stiles calling him “a man of the highest character and an officer of the finest culture and a very high order of ability.” Stiles believed Custis “did not have a fair opportunity during the war” to show his abilities in a combat role.²¹

Matthew W. Paxton, a politician, newspaper editor, and one of Custis’ former students, noted that “it was no secret” that Custis “chafed under being given no opportunity to take a more active part” in the war, but the man who knew him best at the time, Jefferson Davis, said, “The only defect I found in him was his extreme diffidence in his own ability.”²² Perhaps Custis knew he lacked the killer instinct a general needed and was willing to yield to his own limitations rather than suffer possible disaster on the battlefield. His father had a commander’s natural aggressiveness, even bloodthirstiness—and yet made mistakes. Custis might have thought he was serving the Confederacy best in his role as Davis’ aide. By the end of 1864, he could see the Confederacy’s chances of victory were slight. Even if he joined the Army of Northern Virginia late, he could say he fought in the war’s last campaign.

Library of Congress

A Currier & Ives print, “Death of General Robert E. Lee,” depicts Custis Lee (far left)—together with a physician, a clergyman, and two of his sisters—at his father’s deathbed in 1870. The following year, Custis followed his father as president of the newly renamed Washington and Lee College.

After the war, Custis feared he might be tried for treason. In October 1865, he considered moving to Mexico as some Confederates, including prominent officers such as Edward Kirby Smith, chose to do. One observer wrote that Custis seemed “broken and disheartened” by the war.²³ Indeed, Custis said he had “lost everything” in the previous four years.²⁴ In poor health at the time, he stayed in Virginia with the rest of his family, then followed them to Lexington, where his father had been named president of the small, struggling Washington College. Custis began teaching engineering at nearby Virginia Military Institute. He also helped his father collect materials for his planned history of the Army of Northern Virginia. The book never materialized.

In November 1867, the Lees were subpoenaed to appear in court in Richmond to testify against Jefferson Davis, who faced possible trial for treason. Robert E. Lee gave evidence, but Custis did not. Davis was never tried for any of his crimes, nor were the Lees. In December 1868, President Andrew Johnson effectively pardoned all former Confederates. By February 1869, the Lees were free of any possible legal action against them for their role in the rebellion.

Custis enjoyed teaching at VMI. But his father’s death in October 1870 left Washington College without a leader. The trustees chose Custis to become the next president of the newly renamed Washington and Lee University. Once again, Custis found himself living in his father’s long shadow. He took the position and its $1,500 annual salary (equivalent to about $700 per week in 2023).

Despite the low pay, he stayed for 26 years. From 1879–1886, he taught mathematics, and teaching pleased him far more than did his administrative duties. While his students respected him, the trustees rejected his plans to design and construct new buildings. Custis steered Washington and Lee successfully through the financial “panic” of 1873, and though enrollment did not, the campus slowly expanded.

Custis was lonely in Lexington and frustrated by his position. He tried to relinquish the presidency multiple times—citing ill health—but his resignations were repeatedly refused. As a college president, Custis was congenial and reserved. “He had a most winning and benign smile,” one observer wrote of him, but added, “I never heard him laugh out loud.”²⁵ As he had been in his West Point days, Custis was meticulous. “His floors were the marvel of the town in their immaculate smoothness and cleanness,” the observer noted.²⁶ Custis was active in Grace Memorial Episcopal Church, but never became an official member.

There were rumors about his private life. Historian Bernice-Marie Yates has claimed the rumors of Custis Lee’s excessive drinking were “erroneous and insulting.”²⁷ She dismisses his documented drinking as having been within social norms. It would not have been surprising, however, that a man—with no wife or children, in a stressful job, and perhaps broken by a bloody and destructive war—might have taken solace in drink. Another rumor passed down through the years is that Custis fathered a son with an African-American maid in his service.

What is certain is that in 1897, at the end of his tenure at Washington and Lee, Custis’ father and mother had been dead for decades. One sister died during the war, another within weeks of their mother’s death in 1873. Rooney died in 1891. Custis himself suffered from arthritis, and added to his physical ailments were legal headaches. In 1874, he had sued the federal government to reclaim Arlington plantation, which had been seized during the war by the United States Army. By that point, the grounds were covered with the graves of Union soldiers and impossible to reoccupy. But he took his case all the way to the U.S. Supreme Court, which in 1883 ruled in his favor. He then sold the property back to the government for $150,000.

Despite this windfall, Custis remained unhappy in Lexington. “I am utterly useless here,” he complained in his 1896 resignation letter. He departed Washington and Lee University on July 1, 1897.²⁸ He donated 500 books to the library, a grand piano, paintings, $5,000 for the chapel, and created a scholarship fund. He also left $10,000 to his brother Robert E. Lee Jr. and to his brother Rooney’s son George Bolling Lee.

Custis headed north to Fairfax County and Ravensworth, where he lived with Rooney’s widow, Mary Tabb Bolling Lee, her sister, and Custis’ nephew, Robert E. Lee III. When it came to his sister Mary’s wish to reclaim things belonging to the Lees via their Washington family connection, Custis, by then tired of legal battles over the “Mount Vernon Relics” as he called them, complained of her “incessant nagging.”²⁹

Like his father, Custis avoided public discussion of the war and its aftermath. He never wrote a memoir, and he has mostly avoided the attention of Civil War scholars. Relatively few of his wartime letters have survived. Custis had asked that his papers be burned after his death; someone may have made good on his request, or perhaps the fire that consumed Ravensworth in the 1920s took them. The wartime letters that do survive contain little about his personal life or politics.

Custis managed to outlive most of the people close to him. In 1905, his sister Mildred died. A month later, his cousin Fitzhugh Lee—cavalry general and former governor of Virginia—passed away. In December 1911, Custis fell and broke his hip, an injury that confined him to his bed and a wheelchair. He was 80 when he died on February 18, 1913. He was remembered as “a noble, modest, unassuming Christian gentleman.”³⁰ And yet, as fellow Confederate and head of the Virginia Historical Society W. Gordon McCabe wrote, “In the contemplation of his career, one cannot, indeed, escape the constant suggestion of the touch of tragedy.”³¹

His memorial service was held in Lexington. His surviving brother, Rob, was there (he would die the next year). So was Walter Taylor, who had served on Robert E. Lee’s staff. Custis was buried in the Lee crypt at Washington and Lee University.

In 1925, Henry St. George Tucker III, a congressman and lawyer, donated a portrait of Custis (painted by artist John P. Walker) to Battle Abbey, located at the present site of the Virginia Museum of History and Culture. It was on a historic date, January 19—Robert E. Lee’s birthday.

Did Custis Lee Have an African-American Son?

Virginia Military Institute (Lee); Washington and Lee Special Collections Library

Custis Lee in 1875 (left) and Hugh Williams in 1918

When I was doing research on Custis Lee (1832–1913) some years ago, a longtime archivist in Lexington, Virginia, said it was rumored that Lee had fathered an African-American son. The man’s name was Hugh Williams and he worked as a barber. While race mixing was common in Custis’ time, did this rumor have any truth to it?

There is some circumstantial evidence to support the story. Hugh Alexander Williams was born November 17, 1873, in Lexington, to Charlott A. Williams. (His death certificate notes his father is “unknown.”) His mother worked as a domestic in town when Custis Lee was president of Washington and Lee University (1871–1897). Williams died in 1956, at 82, having spent a lifetime in Lexington, where he ran a barber shop and was a respected member of the community. A photograph shows him with other members of an African-America lodge in their Sunday best.

In the census records, Williams is listed as “mulatto,” the designation for mixed race. His draft card notes he was tall and of medium build. So, too, was Custis Lee.

Williams married another Virginian, Geneva White, age 19, on October 12, 1899, coincidentally an anniversary of Robert E. Lee’s death. Geneva outlived Hugh, dying in July 1965, in Stonewall Jackson Hospital, where he had died nine years before. They had no children and are buried in Evergreen Cemetery, the historically “black” cemetery in Lexington.

In photographs, Hugh Williams bears a striking resemblance to Custis Lee and even looks a little like Robert E. Lee. In one image when Hugh was in his mid-40s, he looks like Custis at around the same age, with an almost identical mustache and his hair parted the same way.

Custis Lee is not the first member of his family said to have fathered an African-American child. Early in the Civil War, after Union troops occupied the Lee home at Arlington, an enslaved person told a reporter that “master’s my father.” The reporter took “master” to mean Robert E. Lee, leading the reporter to say, “[T]his father and master now leads an army, the sole purpose of which is to establish a government founded on an institution which enslaves his own children, making his own flesh and blood saleable property!” But the enslaved person’s reference to “master” was to George Washington Parke Custis—the man who had built Arlington plantation—Robert E. Lee’s father-in-law. George Washington Parke Custis, after whom his grandson was named, was rumored to have fathered as many as 15 enslaved children at Arlington.

In the fall of 2018, I emailed professor Ted Delaney at Washington and Lee about Hugh Williams. Delaney, who died in 2020, was the first African American to head the history department at the university, and growing up in Lexington, knew the Williams and White families. “As I tell my students,” he wrote me, “miscegenation was not unusual in 19th century southern history. Hopefully, your article on GW Custis Lee speaks to his other foibles which may be more important [than] fathering Hugh Williams. Frankly, this surely does not compare with the Thomas Jefferson/Sally Hemmings story.” Custis Lee’s shortcomings as university president, he assured me, “is the more significant story.”

If Custis Lee fathered an African-American son, from a genealogical standpoint it would establish an African-American descendant in the Lee family, a grandson to Robert E. Lee. In a time when some people make dubious claims to being Lee’s descendant, it would be ironic if an African-American descendant had lived into the 1950s.

Fathering an African-American child would certainly complicate the scant details of Custis Lee’s personal life. We might wonder if Custis, the lifelong bachelor, had any kind of emotional relationship, consensual or exploitative, with Charlott. Hugh grew to adulthood while Custis was president of the local college in what was and remains a small town.

Without any documentary or DNA evidence, Custis Lee’s relationship to Hugh Williams remains only speculative. Like many a child born out of wedlock, Hugh may or may not have known who his father was. And like many such a father, that man may have wished to be unknown. —CEW

 

Colin Edward Woodward holds a doctorate in history from Louisiana State University. He is the author of Marching Masters: Slavery, Race, and the Confederate Army during the Civil War (UVA Press, 2014) and Country Boy: The Roots of Johnny Cash (University of Arkansas Press, 2022). He lives in Richmond, Virginia, where he is host of the American Rambler history and pop culture podcast.