One Kilometer from the Temple is the vast palace built by Nayakas around 1550 AD and partly by the Marathas.The total area of the palace measures 10 acres. The fort of Thanjavur and maximum part of the recent palace built by the Raghu Natha Nayakar and Vijaya Ragava Nayakar. The palace has huge corridors, spacious halls, observational and arsenal towers and shady court yards of this vast labyrinthine building in the centre of the old town.  The palace has two towers and out of which one had been used for storing weapons. And another tower measures 34.8m height. The poorly marked entrance is a wide break in the eastern wall, which leads past a school and a police station. The palace entrance is off to left at the first junction, through the arched tunnel. Now inside the palace, Mannar Saroboji Saraswathi Mahal Library, Art Gallery and Sangeetha Mahal are situated.