The original palace was given to Edward II in 1305 by the Bishop of Durham, Anthony Bek, and used as a royal residence from the 14th to the 16th century. According to one account the incident which inspired Edward III's foundation of the Order of the Garter took place here. As the favourite palace of Henry IV it played host to Manuel II Palaiologos, the only Byzantine emperor ever to visit England, from December 1400 to January 1401, with a joust being given in his honour. There is still a jousting tilt yard. Edward IV built a Great Hall in the 1470s, and Prince Henry also grew up here; it was here that he met and impressed the scholar Erasmus. Tudor courts often used the palace for their Christmas celebrations.

With the grand rebuilding of Greenwich Palace, which was more easily reached by river, Eltham was less frequented, save for the hunting in its enclosed parks. In the 1630s, by which time the palace was no longer used by the royal family, Sir Anthony van Dyck was given the use of a suite of rooms as a country retreat. During the English Civil War, the parks were denuded of trees and deer. The palace never recovered. Eltham was bestowed by Charles II on John Shaw and- in its ruinous condition, reduced to Edward IV's Great Hall, the former buttery, called "Court House", a bridge across the moat and some walling- remained with Shaw's descendants as late as 1893.

The current house was built in the 1930s on the site of the original, and incorporates its Great Hall, which boasts the third-largest hammerbeam roof in England. Fragments of the walls of other buildings remain visible around the gardens, and the 15th century bridge still crosses the moat. In 1933 Sir Stephen and Lady Courtauld acquired the lease of the palace site and restored the Great Hall (adding a minstrels' gallery to it) while building an elaborate home, internally in the Art Deco style.

The stone bridge over the moat is said to be the oldest useable bridge in London.