Although its present appearance owes much to 19th-century alterations, the origins of The Wardrobe, 58 The Close lie in the 13th century, and the overall shape of a central hall (initially open to the roof) with two service wings remains. At first, the residence of one of the canons who served the Salisbury Cathedral later passed into the Bishop of Salisbury. It is probable that used as a storehouse for church treasures and relics and an administrative base for the bishop’s household, which led to its name of The Wardrobe, a title first recorded in 1543.

After use in the Second World War as a hostel of the Auxiliary Territorial Service (ATS), it was rented in 1945 by the Diocesan Training College for Schoolmistresses to provide accommodation for 25 students and two members of staff. This use continued until 1969. The building then remained empty for some years, with its future in doubt, but during the 1970s, negotiations took place for the building to become a museum for The Rifles (Berkshire and Wiltshire) Museum. Finally, in 1981, the Museum was officially opened by HRH Prince Philip, Colonel-in-Chief of the Duke of Edinburgh's Royal Regiment, on 29 July 1982.

In the 18th century, members of the Coles family divided the upper and lower halls, forming on the ground floor an entrance hall, staircase hall, and dining room, the arrangement which survives today. A central entrance replaced the large window on the courtyard side of the original lower hall. Visitors from all over the country now use the Rifles Museum