Peirce was raised in a family of wealthy timber-land owners who were both cultivated and eccentric. His father, a recognized authority on Byzantine art, wrote several books on the subject in French. His mother was a would-be playwright and summer playhouse owner. His uncle, Waldo Peirce, was a prominent American painter and bohemian character.
Peirce attended, with no great distinction, Exeter, Stanford, and Harvard. At age 22 he married a Tahitian girl and moved to Tahiti, where he lived for the next 23 years. At various times he was a part owner, and sometimes accountant, for a mother-of-pearl button factory, a garden center, a one-hour laundry, and an import business.