It has always been known that the original temple of the Divine is within us. The outer temples are symbolic representations of certain inner realities. The Gita as well as the Upanishads speak about those who sacrifice to the gods for limited gains and those who offer all to the Divine for the very Highest Good, with temples serving their role in helping humanity connecting with the cosmic beings or the gods.
But the occult knowledge and the rites used to propitiate these deities have been hence lost, while mankind has become too much materialistic and desire-oriented, with psychological atmosphere surrounded by forces and energies that are of a lower vital nature. The true gods generally do not reside in such atmosphere and places. That is why most of these places are occupied by the forces of darkness that feed upon the human vital. These beings, unlike the true benevolent gods form a transactional relationship with man. They have powers to grant certain favours, mostly the temporary satisfaction of some petty vital desires or healing certain illnesses caused by adverse forces which they control. In return they seek to enslave the human soul with the purpose of using the human being as their instrument to spread falsehood and unconsciousness.
Of course, the human soul being a portion of the Divine cannot be enslaved but it can be temporarily imprisoned. Such a human being becomes the instrument of chaos and confusion upon earth. The stamp carries sometimes beyond one life. Since most human beings lack sufficient discernment to differentiate between the true gods and the lesser ones and are driven by desires and flock near wish-fulfilling shrines, the sage advice is to find the Divine within in the inner permanent temple. Even the true gods can at best provide only a partial help in our quest for Truth and Light and Freedom and Bliss and Peace and Love and Immortality. To discover the Immanent Divine within us we need to connect directly with the Source.
As far as the devotees of the Mother and Sri Aurobindo are concerned there is no need to go to temples or worship the gods, great or small. We can understand their place and role in the evolutionary process and leave others to follow them as they feel necessary but it has nothing to do with the Supramental transformation that Sri Aurobindo brings for us as the permanent remedy to all that yet baffles and limits and divides and subjects us to ignorance and suffering and death.