In 1834, he became pastor of Union Congregational Church in Boston, Massachusetts. He would remain in that position until his death in 1878. In 1850, he married again, to Sarah Brackett.
In 1854, he took a trip to the American South, and wrote a book entitled A South-Side View of Slavery (Boston, 1854). In the book, he lauded slavery as beneficial to the Negroes' religious character.This book was one of several polemic works he wrote. It caused a great sensation, and he received much hostile criticism. The book was attacked by abolitionists for its perceived moderation; the abolitionist newspaper The Liberator called it "as vile a work as was ever written, in apology and defence of 'the sum of all villanies'"