Tonbridge’s first castle was a simple fort of earth and timber, thrown up – like hundreds of others – by Norman invaders for self-protection soon after they arrived in 1066. It stood on land overlooking the Medway crossing, which William the Conqueror had given to his kinsman Richard Fitz gilbert. Local labour would have been used to shift the 30,000 tonnes of earth required to form the moat and the motte – the ’castle mound’ which still survives. A wooden keep would have been built on top of the motte, with an adjacent area, the bailey (now the Castle Lawn), protected by a fence of stakes.

The wooden castle probably only survived about twenty years. Descendants of Richard Fitz gilbert gradually replaced the earth and timber structure with stone, repairing the effects of decay, fire and warfare and reinforcing their stronghold against improved attack methods. Their final castle had a noble gatehouse encircled by massive curtain walls connecting great towers at each corner, while a high shell crowned the motte. In places, the castle walls were almost three metres thick, with sandstone facings from nearby quarries. Kings fought and were entertained at Tonbridge Castle, archbishops quarrelled over its guardianship, and Henry III’s niece and Edward I’s daughter were mistresses of the castle. During Henry VIII’s reign, the gatehouse was "as strong a fortress as few be in England". During the Civil War, the Castle was strengthened and garrisoned. Thomas Weller, who owned it at that time, was a Parliamentarian. He joined other West Kent gentlemen in opposing local unrest at Sevenoaks in 1643, and there was a three-hour skirmish on the outskirts of Tonbridge at Hilden Brook. The river crossing was fought over but not; it seems, the castle itself – though the Roundhead garrison wreaked havoc on the grounds. Later Weller was ordered to put the castle beyond military use by dismantling its defences.

Later owner, John Hooker, sold stone from the castle to build locks on the river Medway, and in 1791 his son Thomas took more rock from the ruined walls to make the residence that now adjoins the Gatehouse. What was left of the once-proud fortress became, in turn, a private home, a military academy and a prep school until, in 1899, it was purchased by the Town Council. The residence became the Council Chamber and offices, and the grounds were opened to the public.

Nearly 900 years after its first construction, Tonbridge Castle saw military service once more as part of a defensive line against the possible German invasion in World War 2. Anti-tank defences and a machine-gun emplacement were constructed, and two pillboxes were built into the 13th-century walls.