A study of 65 patients, 12 years after they were treated, found those who had radiotherapy were more likely to have problems with memory and attention.
Writing in The Lancet Neurology, the researchers said doctors should hold off using radiotherapy where possible.
One UK expert said doctors were cautious about using radiotherapy.
The patients in the study all had a form of brain tumour called a low-grade glioma - one of the most common types of brain tumour.
In these cases radiotherapy is commonly given after initial surgery to remove the tumour, but there is some debate about whether this should be done immediately or used only if the cancer returns.
It is known that radiation treatment in the brain causes some damage to normal tissue and the study's researchers suspected it could lead to decline in mental function.
A previous study in the same patients done six years after treatment found no difference in aspects like memory, attention and the speed at which people could process information, in those who had received radiotherapy.
But the latest research, carried out more than a decade after original treatment, did find significant variation in the results of several mental tests between those who had had radiotherapy and those who had not.
In all, 53% of patients who had radiotherapy showed decline in brain function compared with 27% of patients who only had surgery.
The most profound differences were in tests to measure attention.
With an average survival of ten years for this type of tumour, the researchers said patients undergoing radiotherapy were at considerable risk of developing problem years down the road.
One option for doctors would be to delay when patients received radiotherapy, reserving it in case the tumour returned, they advised.
"It always depends on the patient, but if it is possible to defer radiotherapy, maybe people should," said study leader Dr Linda Douw, from the Department of Neurology at VU University Medical Centre in Amsterdam.
But she added that more research was needed and there were trials under way to look at other treatments such as chemotherapy.
In an accompanying article, experts from the Mayo Clinic in Rochester, USA, said it was hard to draw conclusions because radiotherapy had improved since the patients in the study had been treated, but agreed more studies were needed.
Dr Jeremy Rees, a Cancer Research UK scientist at the National Hospital for Neurology and Neurosurgery Honorary said they would usually try to avoid giving radiotherapy to patients with low-grade glioma, unless the tumour was progressing or the patient had epilepsy not controlled on standard medication.
"Surgery is generally a preferred option with chemotherapy or radiotherapy coming into play at a later stage, if the glioma progresses.
"Continued research and increased knowledge about the disease is enabling us to treat it increasingly effectively while reducing side effects."(BBC)