Andrew Murray was the second child of Andrew Murray Sr. (1794-1866), a Dutch Reformed Church missionary sent from Scotland to South Africa. He was born in Graaff Reinet, South Africa. His mother, Maria Susanna Stegmann, was of French Huguenot and German Lutheran descent.
Murray was sent to Aberdeen in Scotland for his initial education, together with his elder brother, John. Both remained there until they obtained their master's degrees in 1845. From there, they both went to the University of Utrecht where they studied theology. The two brothers became members of Het Réveil, a religious revival movement opposed to the rationalism which was in vogue in the Netherlands at that time. Both brothers were ordained by the Hague Committee of the Dutch Reformed Church on 9 May 1848 and returned to the Cape.
Murray married Emma Rutherford in Cape Town, South Africa, on 2 July 1856. They had eight children together (four boys and four girls).