The title of Housing Minister has disappeared from Cabinet in yesterday's reshuffle, and Labour says it's another way the government is denying there is a housing crisis.
Ms Adams has been given the social housing portfolio, previously held by Paula Bennett, and responsibility for Housing New Zealand, previously held by Mr English, while Dr Smith's title has changed to Minister for Building and Construction, rather than Building and Housing.
Under the change, Dr Smith will be responsible for the regulatory aspects of housing, like the overall number of houses that are being built, and large-scale construction projects such as Auckland's Special Housing Areas.
Ms Adams will oversee Housing New Zealand and will decide how the government spends money on housing and subsidies.
"What we've done is consolidate the portfolios." - Prime Minister Bill English
Prime Minister Bill English described the decision as "consolidating" the portfolio and said it was a logical move, given its vast scope.
"We've got a minister of social housing and Nick Smith is essentially a planning minister. Those are two complex functions and I wouldn't try to put them all together into one job - it would be bigger than half the rest of the cabinet put together."
Labour Party housing spokesperson Phil Twyford said the government wanted to avoid taking responsibility for failures in that area.
"I think it's really strange that in this reshuffle, not only are they denying the crisis is a problem, they're denying it exists by writing it out of the official list of ministerial responsibilities."
Dr Smith, who was the Building and Housing Minister, has come under fire for not doing enough to address soaring house and rental prices around the country, particularly in New Zealand's biggest city.
The average house price in Auckland hit $1 million earlier this year, while the Auckland City Mission's tally of homeless topped 200 for the first time in 12 years.
When asked how the division of repsonisbility would affect property and rent prices in Auckland, Mr English said that wasn't up to him.
"The government does not set house prices in Auckland. The Auckland City Council sets the rules that help determine those house prices."
Mr Twyford said the government needed to take ownership of what was sure to be a hot topic come next year's election.
"Demand-side factors like immigration, like the fact that our laws allow foreign buyers to speculate, like the fact that the tax system basically encourages people to speculate on property, only central government can control those policy issues, but National refuse to do anything about it."
In response, Mr English told Morning Report, "I don't know what they're talking about frankly".
The change to Dr Smith's portfolio title was "just putting it back to where it was".
Mr English told Nine to Noon the government's home building would "ramp up", with thousands built over the next 10 years, including in Northcote and Tamaki.
While the goverment set the rules, councils and the market had much of the power over housing, he said.
The biggest recent change was the Auckland Unitary Plan, which re-zones the city to provide more higher-density housing, close to transport, for population growth of up to 1 million by 2040.
"In the long run we need more houses ... [but] you can't just push a button and crank them up," Mr English said.
Community Housing Aotearoa chief executive Scott Figgenshow said there was no quick fix to make housing affordable for people on low incomes and addressing the problem required decisive action from the government, as well as charities and NGOs.
"We certainly look forward to sitting down with minister Amy Adams in the New Year," he said.
"One of the first questions would be 'how are we going to build a consistent pipeline that delivers a thousand new units of social and affordable housing each year, every year, for the next decade?'"