Mother Parvathi descended to Earth and chose this location to perform penance. The Devas, from the celestial world, came here in the form of plants and trees to worship the Goddess, as the area was a dense jungle at that time. Pleased with her devotion, Lord Shiva appeared before her, adorned with His Jadamudi (long hair styled as a crown), and thus earned the name Kavardesswarar. Patti, the calf of the divine cow Kamadenu, recognized the sanctity of this place, created a Linga, performed abhishekam (ritual bathing) with its milk, and prayed for Lord Shiva to remain there. As the Lord was worshipped by a cow, He was named Pasupatheeswarar.
The divine cow Kamadenu visited the shrine seeking relief from the curse of Sage Vasishta. In Sanskrit, cows are referred to as "Go," which led to the place being known as Govandhakudi (the palace where the cow came). The abhishekam performed by Kamadenu is depicted on the temple's flagpole.
The area is also known as Manikudam and Aswathavanam. According to scriptures, one of the two stones brought by Vayudeva in a competition with Adisesha is located here, while the other stone is at Nallur.