The governor of Manus province in Papua New Guinea has reportedly threatened refugees with consequences if they break curfew.
The 6pm to 6am curfew was imposed last week after a local women died in a car crash on Manus Island.
The car was being driven by a refugee from Afghanistan who has been charged with dangerous driving causing death.
The island's police commander David Yapu said the man did not have a drivers licence and had been drinking prior to the accident.
Mr Yapu said both the driver and another refugee in the car were being held in custody to protect them from angry relatives of the dead women.
Following the accident, the curfew was imposed at three centres housing about 650 refugees exiled to the island by Australia.
Some refugees questioned whether the curfew was legal given the PNG Supreme Court ruling of 2016 that their detention was unconstitutional.
A Sri Lankan refugee at West Haus said a refugee from Sudan broke curfew on Sunday which prompted a visit from the governor, Charlie Benjamin.
Mr Benjamin and a group of local men stormed into West Haus and threatened all the refugees, especially the man from Sudan, the Sri Lankan refugee said.
Security guards did not call the police or try to stop the local men from entering, he said.
A small number of refugees on Manus had purchased cars with money paid by the Australian government in compensation for illegal detention.
Most refugees were unable to obtain driving licences at this point, according to Mr Yapu.
The refugee charged over the woman's death was allegedly attacked by locals in April, when he was stabbed multiple times with a screwdriver.
Refugees said he had been suffering from mental illness.
The journalist and Kurdish refugee Behrouz Boochani said the woman's death was a tragedy.
"I'm deeply saddened by the news of the death of a Manusian girl in the recent car accident on Manus," Mr Boochani said.
"My thoughts are with her family and I'd like to pass on my sincerest condolences to them," he said.
"We are all living here together as one community, their pain is ours too."