When James Madison died, he still owned about one hundred slaves. He freed none of them, not even Paul Jennings, his valet. Jennings could read and write, and in fact published the first White House memoir, declaring that Madison was “one of the best men who ever lived.” Modern biographers of Madison, such as Richard Brookhiser and Jeff Broadwater, have frankly acknowledged the shocking truth that such a politically astute and sensitive founding father utterly failed to address the problem of slavery seriously. But most, including not only Mr. Brookhiser and Mr. Broadwater, but also Kevin R. C. Gutzman, Andrew Burstein, and Nancy Isenberg, treat the issue of slavery as a thing apart, in separate chapters, instead dealing with the place of the “peculiar institution” in Madison’s life in the years after he left the presidency.
And yet there never was a time when James Madison (1751 – 1836), a third-generation slave owner, did not believe slavery was evil — or a time when he did not recognize the capabilities of African Americans. In 1791, Madison wrote admiringly about the “industry & good management” of a free African American landowner who could read, keep accounts, and supervise six white hired men on a 2,500-acre farm. In April 1800, Madison dined with Christopher McPherson, a confident and free African American, who came as a guest to Madison’s plantation home, Montpelier, to deliver books and letters that Madison and Jefferson sent to each other. During Madison’s terms as president, he often heard out his private secretary, Edward Coles, who objected to slavery as a violation of the natural rights doctrine that Jefferson and Madison espoused. In 1816, Jesse Torrey, a zealous abolitionist visited Montpelier and treated Madison to a tirade against slavery, afterwards sending a letter of apology — only to receive, in reply, a letter from Madison saying no apology was necessary. In 1824, Madison endured with good grace the disapproval of Lafayette, then on a triumphal tour of the United States, who visited Montpelier and told off the retired president, expressing disgust that both Jefferson and Madison, such champions of liberty, should still own slaves and support such a vile institution. In 1835, Harriet Martineau, an outspoken abolitionist and an old friend of Madison’s, visited him for the last time, afterwards reporting that her host “talked more on the subject of slavery than on any other, acknowledging, without limitations or hesitation, all the evils with which it has ever been charged.”
Like Madison himself, his biographers treat slavery as a kind of dirge, faintly heard off-stage and nearly drowned out by the stirring music of the freedom fighters making an American Revolution and the framers of the Constitution going about the glorious work of creating a democratic republic. Elizabeth Dowling Taylor, however, wants us to listen to that more troubled theme, and the result is a revelation. In A Slave in the White House: Paul Jennings and the Madisons, we’re asked to consider Madison as a “garden-variety slaveholder”: “He followed the basic patterns and norms for slaves’ living conditions and treatment that had long been established on Virginia plantations and like most owners respected the customary “rights” — such as Sundays off — that enslaved people had come to consider their due.” If it is not oxymoronic to say so, Madison was a humane slaveholder. He was also not very enterprising, in that his human holdings constituted — as they did for Jefferson — a losing economic proposition. As soon as her husband died, Dolley Madison, whose Quaker father had freed his slaves, sold off batches of her slaves in order to pay off debts.
Ms. Dowling crafts a narrative in which African Americans are virtually never out of sight. And that makes a great deal of sense: it is unlikely that Madison ever spent a day without relying on the services of a slave. He took at least one of them with him when he traveled. And Paul Jennings was the last one out the door, clutching some of Dolley Madison’s treasures, as the British advanced during the War of 1812 and set fire to the White House.
Harriet Martineau observed with some surprise how Madison could discourse on the evils of slavery, even as slaves served him at table. It is that Madison we see in Ms. Dowling’s narrative. Here is a sample sentence: “The Virginia Resolutions  was yet another appeal against tyranny that Madison drafted at the place where he lived with scores of slaves.” When Lafayette comes to Montpelier, Jennings is there beside Madison, listening, although we do not know what the slave thought. And this silence forces Ms. Dowling, all too often, to resort to what “must have been” going through Jennings’s mind. It is no wonder, then, that most historians and biographers are much more comfortable dealing with Madison’s well-documented mind. Thus Kevin R. C. Gutzman writes a stirring narrative, showing his subject’s dexterity as politician and statesman, while Andrew Burstein and Nancy Isenberg portray how well the tag-team of Madison and Jefferson served their country.
The concluding pages of Richard Brookhiser’s concise biography seem to come closest to revealing why the mild-mannered Madison both deplored slavery and supported it; started the War of 1812, even as he was trying to negotiate peace with the British; and fought stoutly for maintaining the Union, even as he remained very much a son of the South. Mr. Brookhiser sees Madison as the epitome of the legislative mind. Madison was the man of principles who made deals, making sure the words “slave” and “slavery” did not appear in the Constitution, but also paying off his Southern vote-counting brethren with the three-fifths compromise. Slaves were partial “persons” for purposes of exerting political power. This political accommodation jibed with Madison’s statement that slaves were part of his family, but only a “degraded” part.
The legislative mind, Mr. Brookhiser suggests, has trouble with the idea of exerting executive power. Since Madison believed that he could secure no agreement among slaveholders to abolish slavery — let alone arrange some kind of compact with the North — then nothing could be done short of shipping African Americans off to Liberia. But that strategy would work only if African Americans themselves consented, Madison argued, and most did not. And the cost of reimbursing slaveholders proved a problem too large for Madison’s limited capacity as an economist.
But there is an even more important factor to consider in exploring why Madison, a mover and shaker of public opinion when it came to engineering such triumphs as the “Federalist Papers” to support the Constitution, never mounted a credible campaign to abolish or even attenuate the institution of slavery. From 1780 to 1784, William Gardner, Madison’s slave, resided in Philadelphia with his master, who attended meetings there of the Continental Congress. Upon Madison’s return to Virginia, Madison left Gardner behind, writing that his factotum’s mind had been “tainted” with ideas — the “contagion of liberty,” as Elizabeth Dowling Taylor puts it. This episode is reminiscent of that scene in Frederick Douglass’s autobiography when his white mistress is advised not to teach him to read, because doing so will only give him “notions” that do not befit a slave.
Madison’s idea of the American polity had no place for educated black men and women, let alone the masses of freed slaves that he believed had trouble governing themselves. No matter which biography you read, all of them eventually disclose this fundamental fact: Madison did not believe that white and black Americans could live side by side on terms of equality and amity. His failure to imagine a world more capacious and tolerant than his own helps explain a good deal of subsequent history, and America’s resistance to the very practice of equality that Madison otherwise did so much to foster.