Some colours may be harmonious when they are displayed together and some may not. The human eye can perceive harmony or disharmony right away as it is a natural instinctive response. We don’t have to think about it to judge it. I haven’t researched on the origin of this response but, even if it is cultural, it has been so conditioned that it has become natural.
It seems that harmony is related to the notion of order and disharmony to disorder. So colours that are harmonious must belong to a scheme. And if they are too different from a scheme, they would be seen as disharmonious.