The Fonterra Shareholders Council has rejected a proposed change to the co-operative's constitution that would shrink its board from 13 members to nine.
But the former directors who made the proposal said the council would have to reconsider when shareholders voted on the resolution at the end of November.
Former Fonterra director Greg Gent said the Shareholders Council did not understand how unwieldy and factionalised a large board could become.
"Thirteen for a board is a very large board - in fact it's vary rare.
"You get issues of splits, you get issues of where does the accountability lie because you have dominant and more passive individuals in large groups.
"All those sorts of factors lead to slowness in decision making. Management have got a process in place to get a leaner, fitter Fonterra and if you're going to do that you've got to start at the very top."
The council said the proposed process for reducing the number of directors presented a risk, as board members would have to stand down then seek re-election at the same time.
Mr Gent said the council was exaggerating.
"That's a total red herring. The appointed directors will remain. Farmers through time have been very responsible in how they vote.
"To think this would lead to a wholesale exit of the whole board, to me, borders on the ridiculous."
A committee is reviewing the co-op's governance, but Mr Gent said Fonterra should move swiftly to downsize its board as it did when cutting more than 700 jobs this year.
"The board, I guess, is conflicted in downsizing. It's not an easy thing to do because people lose positions. They committed to doing something to the Annual General Meeting in 2012. We've been waiting ever since and that's why we brought our resolution.
"Now we're told there's going to be another committee and another process which is almost your classic sort of government line - the problem is a bit hard let's form a committee. Our view is just get on with it and do it."
To change the constitution, 75 percent of Fonterra shareholders need to vote in favour of downsizing the board at next month's AGM.