Manus Island refugees transferred to a hotel in Papua New Guinea's capital have been told their weekly allowance will be cut unless they volunteer to move into the community.
About 80 refugees in Granville hotel are under pressure to relocate to residential accommodation.
Along with about 200 other men, they were transferred to Port Moresby last month from the island where they had been detained by Australia since 2013.
During the last six years, about 1500 men had been held on Manus. Now only four or five refugees are thought to remain.
A notice given to the refugees this week in Granville said they would receive a weekly allowance of about $US30 when they volunteered to move.
But refugee Shaminda Kanipathi said the refugees already received this allowance and were told it would be cut unless they agreed to leave.
The men had not been told when or where they would be relocated, Mr Kanapathi said.
They had already been rejected for US resettlement and were afraid they would be abandoned in Port Moresby, he said.
"We are already in limbo and stressed. This sudden relocation without proper and adequate information makes us more stressed. This system always means to torture us and this is one of the aspects of their torture," Mr Kanapathi said.
Refugee advocate Ian Rintoul said an earlier attempt to house refugees in a Port Moresby suburb had failed.
Those refugees who tried to live in Morata were now destitute and had all been bashed and robbed more than once, Mr Rintoul said.
The hotel was guarded, but in the suburbs there would be no security, he said.
"Port Moresby is more dangerous than Manus Island. Immigration must lift the threat of cutting off income support to force refugees to move.
"This is another short-sighted proposal that puts refugee lives in danger that does nothing to provide them with a secure future after more than six years of mistreatment in offshore detention."