As Baba did not like the slander of the villagers about His stay at Smt. Karnam Subbamma’s house, He went away to a cave in a hill located on the other side of the Chitravati river near Janakampalli. He stayed there for 6 months. When Smt. Subbamma heard that the Sidhgunananda Swami of Kothacheruvu wanted to take Baba with him and groom Baba to become his successor, she acted quickly and invited Baba to use a vacant plot of land on the western banks of Chitravati as His temporary residence. This land belonged to the Karnam family and had a small thatched hut built in it.
One night, miscreants set fire to the little hut where Baba was sleeping. A sudden downpour put out the flames and Baba was unscathed. Subsequently, Smt. Karnam Kamalamma and Smt. Subbamma offered a vacant plot of land, their family property, as a token of devotion to Baba. On the same day the sale deed was registered in Baba’s name at Bukkapatnam. On the 21st of July 1945, Smt. Karnam Subbamma called Sri Thunga Gangappa, Baba’s friend, and told him to help Baba construct a small house in the vacant land.
When the foundation work for the first Mandir began on Sri Krishnashtami in 1945, someone found a Lingam in the site. Baba immediately took it away, saying that it belonged to Him. He said that many Rishis had done penance in that place, and was hence charged with spiritual energy.