Milhaud came from a wealthy, long-established Jewish-Provençal family. Although he was born in Marseille, he grew up in Aix-en-Provence.
His systematic musical education began at the age of 7 when he took violin lessons. His first compositions were written in 1905.
In 1912 he was introduced to the poet Paul Claudel with whom he would have a lifelong friendship and artistic collaboration. When Claudel was sent as French ambassador to the Brazilian capital of Rio de Janeiro in 1916, Milhaud who was released from service in World War I, accompanied him to South America as his attaché. There he became acquainted with Brazilian folklore and popular music which would greatly influence his music in the years that followed.
After the outbreak of World War II, Milhaud emigrated to the United States with his wife and became a teacher of composition at Mills College in Oakland (until 1971). From 1948 he also taught a composition class at the Paris Conservatory (until 1972). During the following period, he taught annually alternating on both continents.