Accusations from Prime Minister John Key that the Labour Party is "backing the rapists" have prompted a mass walk-out from Labour MPs in Parliament.
The question was about the government's response to the New Zealanders being held in the Christmas Island detention centre by Australia.
Trouble broke out at the centre after the death of an asylum seeker who had escaped. The facility houses both asylum seekers and New Zealanders awaiting deportation.
More on the trouble at the centre on Christmas Island
Mr Key told Parliament he and government ministers had been in touch with their Australian counterparts to make sure New Zealand detainees were being sent back to New Zealand as fast as possible, if that was what they had requested.
When he was challenged, Mr Key had been saying there were challenges with some individuals because they had been convicted of serious offences.
"Well it's not actually easy 'cause some of them are rapists, some of them are child molesters and some of them are murderers - people that the Labour Party are saying are more important to support than New Zealanders who deserve protecting when they come back here."
Australian authorities are still trying to restore order at the Christmas Island detention centre after disturbances broke out early on Monday (file). Photo: AFP
He then directed his comments directly towards Labour MP Kelvin Davis, who recently travelled to Christmas Island to meet the detainees.
"Mr Davis, if you want to put yourself on the side of sex offenders, go ahead, my son, but we'll defend New Zealanders!"
Those comments caused outrage on the cross-benches but Mr Key was not finished, yelling more comments across the Chamber.
"And what the Labour Party's saying is to hell with the rest of New Zealanders, these people should be put on a commercial aircraft and dispatched to New Zealand.
"Well, you back the rapists!"
At this point, he was cut off by Speaker of the House David Carter.
Labour MP Grant Robertson complained to the Speaker, saying he was deeply offended by the comments and wanted the Prime Minister to withdraw and apologise.
Mr Carter's refusal to order the Prime Minister to do so prompted about half of the Labour MPs in Parliament to walk out.
Listen to Andrew Little talk about the debate today
Labour MP Kelvin Davis recently travelled to Christmas Island to meet New Zealand detainees. Photo: RNZ / Alexander Robertson
Australia asked for assurances around deportations
Earlier, Mr Key defended his government's response to the riot at the detention centre and said there was no point sending New Zealand officials to the island at the moment, because they would just get in the way.
He said he was doing what he could.
"I can't stop Australia having the sovereign right to send someone to a detention centre even if I don't like the policy, and I don't like the policy.
"What I can do, and what I have done, is seek an assurance from [Australian Prime Minister] Malcolm Turnbull that he'll honour his commitment and have those people sent back as quickly as we can."
Mr Key also defended a New Zealand submission to the United Nations Human Rights Council, which contained no criticism of Australia's stance on asylum seekers.
This country had other ways of taking up matters with the Australians, he said.