If These Walls Could Talk: A Smith-McDowell House History

Buck House, prior to 1875

Just a few miles from George Vanderbilt’s grand Biltmore Estate is Buck House, a different kind of mansion – one that was nearly 50 years old when Vanderbilt began construction. Saddled on a ridge between two hills south of Asheville, ringed by picturesque mountains and overlooking the confluence of two great rivers, Buck House was constructed around 1840, over twenty years before the Civil War, primarily – and most likely exclusively – by slave labor. During a time when most people lived in wood frame houses, the imposing structure was composed of brick, a rarity in early 19th century Asheville. Today it is known as the Smith-McDowell House, the oldest surviving house in Asheville and the oldest brick structure in Buncombe County, North Carolina.

The mansion sits upon a plot of land, originally 308 acres in area, acquired by Colonel Daniel Smith in 1796 from William Stewart “for and in consideration of One hundred pounds lawful money….” Among the earliest settlers in the region, Smith’s second child is said to have been the first white baby born west of the Blue Ridge in North Carolina. It was this son, James McConnell Smith, who would later wed Mary “Polly” Patton, and build a business and real estate empire, including one few still-existing brick mansions from that time period in the area, Buck House.

The Smith's

James M. and Mary “Polly” Patton Smith

In 1833, James Smith built and operated a state-licensed toll bridge over the French Broad River that connected the Buncombe Turnpike with areas to the west of the river. The Turnpike was part of the Drovers Road that linked Greeneville, TN, and western North Carolina farmers to markets in South Carolina and Georgia. Smith’s Bridge brought in money that he then used to expand the general mercantile business that he ran with his son-in-law, William Wallace McDowell. He became one of North Carolina’s most influential and leading businessmen, owning the general store, a wagoner, a blacksmith, a tannery, a lumber yard, vast farm land, and the Buck Hotel. At one time, Smith owned more than 30,000 acres across Buncombe County.

1850 Slave Schedules, Buncombe County

There is little doubt that people of African descent enslaved by the family constructed Buck House in the 1840s and certainly cared for the Smith family and their vast landholdings. In 1850, Smith enslaved 44 people, including Bob – a tanner, Bob’s wife, Lydia, and her children Alexander Sy – a blacksmith, Bob Hardin, Catherine, Betsey, Caroline, and Moses; George – a shoemaker, and his wife Louisa, and her child, William; George’s sons, Miles and Charles; Alfred, Swan Lucy Ann, and Tom – a miller; Joe – a waggoner and his wife, TIlda, and her children, Alfred, Joe, Mary, Jane, and Vina; Peter, Charles, Clara, Arsela, Martha, and Robb; Jeff and his wife, Mary, and her child Samuel; Henry and Julia Ann, McCama and George, and their children; and Lucy, Harry, Mose, Rebecca, and Charlotte.

Though people enslaved in the South prior to Emancipation were not accounted for by name in the 1850 census, James M. Smith’s will gives us some written record of the people who were forced to serve him and his family. Though Smith typically only lists each person by name, and in a few cases occupation or family relation, in his will, one entry in an addendum to the original 1850 will reads:

All the balance of my negroes (except Phillip) I direct to be equally divided amongst my daughters now living…. My old man, Phillip, has long been a faithful servant and useful to myself and my family, I direct no labor be required of him, but that he be allowed to live with my wife or my son, John P. Smith, as he prefers, and as the law requires that he must have an owner, I give him as the property of my son, John P., in confidence that he will take care of him and protect him, and I give and bequeath to Phillip twenty-five dollars per annum as long as he lives for his comfort, to be paid out of my estate. If the legacy cannot legally take effect directly to him, I leave it to be paid to John P. Smith having a confidence that he will faithfully apply it as here intended. [Presumably, Phillip is the 66 year old man listed first on the 1850 slave schedule.]

W.W. and Sarah McDowell

When James Smith died in 1856, his son, John Patton Smith inherited Buck House, and many of the people once enslaved by his father. John died a little over a year later, at only 34 years old, and left no will. Without a clear line of inheritance, his sister, Sarah Smith McDowell and her husband, James Smith’s former business partner, William Wallace McDowell, purchased the house and 350 acres for $10,000.

On the eve of the Civil War, McDowell organized the Buncombe Riflemen, the first group of Confederate volunteers from western North Carolina. They fought at the battle of Big Bethel in Virginia. McDowell fell ill and was forced to return home and work in an administrative capacity. He eventually returned to the battle field, achieving the rank of Major in the Sixtieth North Carolina Regiment under the command of his brother, Colonel Joseph McDowell. At one point during the War, when the house was visited by Union troops, Sarah McDowell hid the family silver from the soldiers by burying it in the side yard.

When the Civil War ended, the McDowell family – no longer able to rely on the free labor of enslaved people – faced economic reversals. Buck House, along with 16 acres, was sold to Alexander Garrett in 1881. Garrett was an Irish immigrant who had made his fortune in America. He came to Asheville from St. Louis, Missouri, in hope that the mountain air would cure his daughter-in-law’s tuberculosis. Garrett updated the home, connecting the summer kitchen to the main house. He also founded Victoria, the wealthy community surrounding the house, becoming its mayor and building the Victoria Inn. Garrett later sold the house to his son, Robert, for $1.00.

Photo of the Carriage House and formal garden area

The formal garden area and the carriage house, circa 1913

Robert sold the Smith-McDowell House to Dr. and Mrs. Charles Van Bergen in 1898. Two years later, in 1900, Van Bergen commissioned the Olmsted Brothers to create a landscape plan for the property while adding a new carriage house that, according to one account, was designed by Richard Morris Hunt. Hunt died in 1895, therefore it is likely that the carriage house was designed by Richard Sharp Smith, supervising architect for the Biltmore Estate.

Subsequent owners of the house were Henrietta Bates McKee (friend of the Vanderbilts and President Roosevelt), millionaire businessman Brewster Chapman, and Herman Gudger (a friend of Chapman’s from a local Asheville family whose brother Francis was General Manager of DuPont during WWI and Vice-President of film-famed Goldwyn Corporation). Chapman hired Richard Sharp Smith to update the house in 1915, adding new flooring, a slate roof, and changing much of the interior woodwork.

In 1951, the Catholic Diocese purchased the house for a boy’s school dormitory. By 1974, the school had closed, and the Church became interested in selling the property. The then-dilapidated house and grounds were purchased by Asheville-Buncombe Technical Community College, and that same year, the Western North Carolina Historical Association negotiated a lease to restore the house as a museum and heritage center. By 1976, the Historical Association had pulled together enough funds to begin restorations and renovations, and the house was opened to the public in 1982.

Today, the restored Smith-McDowell House functions as the headquarters of the Western North Carolina Historical Association, a historic house museum, an exhibit space, and a center for youth history education. It is included on the National Register of Historic Places and is the finest surviving example of brick antebellum architecture in western North Carolina.