Catholic gay rights groups say the rejection by bishops of proposals for wider acceptance of gay people, backed by Pope Francis, is "very disappointing".
Pope Francis walks past Swiss guards at the Vatican. Photo: AFP
A draft paper issued at a Catholic Church synod in the Vatican had included text calling for greater openness towards homosexuals, and divorced Catholics who remarried.
But this failed to win two-thirds backing and was not in the final text.
The BBC reports the synod will reconvene next year.
The New Ways Ministry, a US Catholic gay-rights group, said it was "very disappointing that the synod's final report did not retain the gracious welcome to lesbian and gay people that the draft of the report included".
However, it said the synod's "openness to discussion provides hope for further development down the road".
Another group, DignityUSA, said: "Unfortunately, today, doctrine won out over pastoral need. It is disappointing that those who recognised the need for a more inclusive Church were defeated."
Correspondents say the text welcoming gay people and remarried Catholics had been watered down in the final version that was voted on - but it appears that they still met with resistance from conservatives.
All other parts of the draft report were accepted by the synod.
Christopher Lamb, from British Catholic journal The Tablet, said that the discussion at the synod was "a huge achievement in itself".
He said it was important to remember that many of the bishops at the synod were from countries where homosexuality is illegal.
"We have now got an acceptance that we need a new language in the Church when talking about gay couples and homosexuality in general," he added.
Speaking after the vote, Pope Francis told attendees that he would have been "worried and saddened" if there had not been "animated discussions" or if "everyone had been in agreement or silent in a false and acquiescent peace", AP news agency reported.
He also cautioned against "hostile inflexibility, that is, wanting to close oneself within the written word, and not allowing oneself to be surprised by God".
The earlier draft had said that homosexuals had "gifts and qualities to offer to the Christian community".
Conservative groups had described the earlier draft as a "betrayal".
About 200 bishops had attended the two-week synod on family issues at the Vatican.