A memo from Housing New Zealand details an exit strategy for its presence in Queenstown.
The memo, released under the Official Information Act, relates to the sale last month of one of the agency's 10 homes in the resort town.
It said the town was a "high profile resort/lifestyle environment" with "reasonable work opportunities", which could be seen as "incompatible" with a social housing presence.
The comments have angered social housing providers in the town, including the Queenstown Lakes Community Housing Trust.
Its executive officer, Julie Scott, said it had a waiting list of 350 people struggling to find affordable accommodation.
"I just met with a family this morning and she was in tears. They've been in Queenstown since 1992, they've got three kids and they've got until 22 August and then they've got to be out of their property.
"They just can't find anywhere to go and it's just heartbreaking."
No plan to leave Queenstown - Housing NZ
There was no deliberate policy to exit Queenstown, Housing New Zealand chief operating officer Paul Commons said.
"What can happen over time is that, in some towns and cities where housing is quite affordable and there's plenty of rental stock, we tend to over time get fewer and fewer numbers and that's just the natural way it works.
"In some towns and smaller towns in particular, we do eventually exit them."
Mr Commons said only two people were currently on the waiting list for social housing in Queenstown.
The agency was in the game of providing social housing, not affordable housing, he said.
"We are housing some of the most vulnerable, and with the highest need. They're likely not to be in the workforce, they're likely to have significant financial pressures, they're likely to have some health issues.
"Queenstown has some housing pressures, in the general sense - but we're not housing typical working Kiwis who are trying to find a place in Queenstown."
Ms Scott said the only way Housing New Zealand could claim there was a lack of demand for social housing in Queenstown would be if officials were actively discouraging people from applying to the waiting list.
According to recent housing data, Queenstown is the least affordable town in New Zealand, with the ratio of median house price to median income reaching 11.3 to 1 - compared to 9.1 to 1 in Auckland.