Christchurch business owners say problems with beggars are escalating, after one of the city's trams was stopped and its driver abused over the weekend.
But homeless people say the finger is being pointed at them unfairly.
During the weekend, a tram full of tourists was stopped by a man who appeared homeless.
He verbally abused the driver when asked to move.
Workers on New Regent Street, a central city tourist hotspot, said it was not an isolated incident.
"It is scary because they are quite aggressive," said one business owner who asked to remain anonymous.
She wanted to extend her business hours during summer, but was afraid for the safety of her staff and customers.
Problems with people living on the street were not new, she said.
"A drunk homeless man came into the store asking for food, very aggressive as well... The police came that time but it took a while."
The tram company, Wood Scenic Line, did not return any of RNZ's phone calls today.
But RNZ understood there had been several other incidents, including a driver who had to stop his tram to remove a beggar's blanket.
That beggar then hurled verbal abuse at him in front of his passengers, telling him to "'eff off".
However one homeless man, Tepi, said homeless people were being unfairly targeted.
"It's just random people doing it, it might be a drunk jumping on the tram, but then next minute, 'Oh the homeless did it'," he said.
"They will always point the finger at us... If something happens in town they'll come straight for me."
Chris Chamberlain is the pastor of Oxford Terrace Baptist Church, a short walk away from New Regent Street.
His church offers free drop-in dinners for homeless people, as well as for people who can't afford food for themselves.
While some homeless people did cause problems, so did people from all walks of life, he said.
"There is always a bit of anti-social behaviour...[but] I don't think it is fair to target the homeless," he said.
"You only have to go downtown on Thursday, Friday, Saturday night and you'll find some very, very well-heeled people behaving badly."
Mr Chamberlain said the tram incidents highlighted the need to get people living on the street into homes.
"Some of these people fall into the category of needing housing, social services, and mental health care, and those things are really quite significantly stretched in Christchurch."