The song title was also relevant to the real find of Friday's archaeological search at the Priory - a 100 year old brooch made of real silver discovered by one of the children taking part in the open day.
The event was organised by Friends of the Priory, a group who are behind a plan to recreate a monastic garden at the priory, and involved a number of kids from Coldingham Primary School who had already had a taste of metal detecting earlier this year.
Bill Wyman said on Friday: "It is great to be here in Coldingham Priory to get involved in a local project of real historical value.
"The kids and teachers all have an immense pride and interest in their local heritage, so I am delighted to be here with them."
Bill Wyman's attendance at Coldingham was thanks to Richard Havers - a long time friend of 71-year-old metal detector Wyman.
Richard told the Berwickshire News: "The day went really well despite not finding any huge treasures. I think the only thing Bill found was a 50 pence! The most valuable thing to come from the day was how the kids enjoyed it. They were so enthusiastic and it really got them thinking. It is one of Bill's (Wyman) things that he believes metal detecting is a great way to get the kids out doing exercise."
Despite his lack of success on this occasion, Richard said Bill enjoyed the day, which was also part of promoting Historic Scotland and the potential new monastic garden, part of the 'Adopt a monument' scheme which would see a recreation of the ancient native species that Benedict monks would have kept at the Priory.
He said: "Bill had a great time and his 11-year-old daughter also enjoyed the day as she was invited to lunch by the Coldingham children. Bill heard about the project through Scottish Borders Council archaeologist Rory MacDonald who asked if Bill would be interested in coming to the open day and he was very enthuastic about it.
"Bill started in metal detecting when he moved to Suffolk and workmen in his garden found an old pot. It got him interested and he started metal detecting nearby, finding 300 coins and a Roman site in a random field.
"It is a wonderful project (the garden). Edgar's wall at Coldingham church (from the 11th century) is a real gem."
Helen Bradbury of The Council for Scottish Archaeology, who is the officer behind the 'Adopt a monument' scheme throughout Scotland said: "It was a fantastic day, a really good turnout and the kids really enjoyed it. They found a brooch which we believe is 100 years old and made of real silver, and a shaped piece of lead which was suggested was used as a hand guard. We don't know the dates yet of either piece but they have been sent away to be tested.
"We have submitted an application for funding for the garden and are quite hopeful of its success.
"It is a strong project and Friends of the Priory have put a lot of work into it. There is a huge community element behind the project, which is a very positive aspect."
The decision on funding for the garden will be known by the end of the year, with work hopefully to start in summer 2008.
"The garden will be open to everyone and we want to make it as physically and intellectually interesting as possible. The garden will try and recreate the daily lives of the Benedict monks in the garden, including orchard trees and medicinal plants used by them," added Helen.
Coldingham Priory's past is as chequered and interesting as Bill Wyman's ex-band mate Keith Richards'.
Built by King Edgar after a vision of St Cuthbert allowed him to regain the Scottish throne in 1098, the Priory was property of the church for over 500 years, and the priory housed monks within its walls.
Yet, in 1560 the reformation saw the Priory pass into the hands of a local landowner. It was allowed to become decrepit and was destroyed further by Oliver Cromwell who fired cannons at the building to dislodge Royalists hiding there.
Despite a new church being built in the 19th century, a wall still remains from the original 1098 builiding, named Edgar's Wall after the king who built it.