Officials say transparency is critical to build trust in the government's most important data system, but the business case for expanding it is mostly blanked out (File photo). Photo: RNZ
Officials say transparency is critical to build trust in the government's most important data system, but the business case for expanding it is mostly blanked out.
Over two-thirds of the 95 pages in the business case to overhaul the Integrated Data Infrastructure, or IDI, are entirely (or a few almost entirely) redacted.
The black pen has been swept over all the options the government is looking at, all the costs to the taxpayer and the cost-benefit analysis.
Yet the business case report stated, "Transparency was highlighted as critical to building trust, with calls for clearer communication about data use, access, and safeguards."
This was after holding five workshops with iwi, industry and non-government organisations a few months back.
Some groups really did not trust what government was doing with data, the report noted.
Referring to feedback from the Data Iwi Leaders Group, Stats NZ said, "Trust and reciprocity were identified as foundational to any future data system."
A much more powerful IDI is crucial to the government's social investment approach, but there were barriers.
"The social license for expanded social investment is untested," said the report.
"There is a need to build robust data ethics practices and safeguards into the social investment approach (especially as this approach expands). This is important to maintain public trust in how government uses data and to ensure individuals and communities are comfortable sharing their data with government."
Both the indicative business case and a Cabinet paper in October - which was when the interim business case was approved and a detailed business case ordered up - had a lot to say about how urgent it was to transform the "clunky and slow" IDI.
It was the tool that "that brings it all together".
"The IDI is the only integrated data tool available to support the government's social investment approach," said the proactively released Cabinet paper.
Stats NZ says it needs to balance transparency with Cabinet rules and guidelines, government information management guidelines and with legal frameworks (File photo). Photo: RNZ /Dom Thomas
Why so much was blanked out
The mass blanking out of the report was to "maintain the constitutional conventions for the confidentiality of advice tendered by ministers and officials" to Cabinet, Stats NZ told RNZ in its response to an Official Information Act request (OIA). This is one of the grounds allowed under OIA law.
Business cases for government projects typically lay out the options and how they compare, and often they recommend one or the other.
Both the long-listed options and the short-listed ones are blanked entirely in the IDI report, as are the critical success factors.
If you wanted to read the "detailed analysis of long list options", too bad.
The "Economic Case" section was 19 pages long but only one page and two paragraphs of that survive for the public to read, and these illuminated little, other than to say doing a cost-benefit analysis was tricky.
A suggestion from a UK approach was that for every dollar invested you got $4 back. How does it pay off? For instance, in NZ data research inside the IDI helped spur more investment in driver training for young people, which was shown to cut how many ended up in court and costing society more.
The "Commercial Case" and the "Financial Case" in the business case were both entirely blanked out. The index showed these considered the funding model and "overall affordability".
Missing from view too, were the main risks anticipated from implementation, and the key constraints, dependencies and assumptions.
Uncharted territory
The scope of the "transformation" of the IDI could take it into uncharted territory, as the report briefly noted.
"All data in the IDI is de-identified, so while it can be used to analyse 'cohorts' of people, it cannot be used for case management or targeting services to individuals," it said.
"Any shift in how the IDI is used - for example, towards targeting services to individuals, would require significant legislative change and building strong social license for such a change."
Most of what is left unredacted and readable in the business case are the reasons why the overhaul was required, for instance, as the key testing ground for how to spend the $190 million Social Investment Fund.
The IDI has 15 billion rows of data, but can be refreshed only three times a year because it takes so long - 12-14 weeks per refresh. It underwent its biggest refresh in June this year that required 90 hours of staff overtime to complete on the final day before deadline.
"Data integration is labour-intensive, access is limited, and data coverage and quality are patchy," said the business case.
The Cabinet paper said demand was "increasing rapidly" particularly as the government expanded the so-called "Outcomes-Based-Contracting" model and the fledgling Social Investment Agency, and Whānau Ora commissioned more contracts through Te Puni Kōkiri.
The 15-year-old system was no match for this. "A single complex analysis within the IDI Data Lab can slow the system down for all users, turning a simple query from another user that would normally take seconds into a full day wait."
The Te Puni Kokiri building on Wellington's Lambton Quay. Photo: RNZ / DOM THOMAS
A concentration of labs
Data Labs are the only way to access and use the IDI which has no internet connection at all to protect its contents.
There are 40 labs, over half of those are in Wellington (22), while Auckland had nine. Sydney, Rotorua, Palmerston North and Hamilton had one or two labs each - but the South Island in total had just three.
The business case, what can be seen of it, does not talk about this geographical barrier to researchers.
It quoted them saying "real-time and on-demand access to integrated data was seen as critical" but little was said about how that might be tackled.
"Streamlining research approvals and improving access protocols were suggested to reduce barriers," it said.
A trust in Tai Rāwhiti has told RNZ about how it had to get expert help just to draw up its application for research approval, let along get hold of the coding and technical knowhow to design ways to get the data it was after once it was inside the Lab.
The IDI overhaul has been slowgoing. The indicative business case report was delivered a year overdue.
Stats NZ, in its OIA response, said it had also been delayed in developing a multiyear data and statistical programme as had been ordered, due to "competing priorities, including modernising the census and social investment".
Plus the dedicated data support team it was meant to have set up by October ran into problems signing contracts with other agencies, so instead it had been doing its own work improving the data flow in the IDI, among other things.
Stats NZ acting deputy chief executive - office of the chief executive Sarah Dwen said the agency "absolutely recognised" the need to build public trust and confidence in the work it does.
"Transparency and being open with communities are part of that, as are lots of other factors including reliability, visibility and accessibility.
"When it comes to transparency, we need to balance that with the requirement to keep some information confidential in order to comply with Cabinet rules and guidelines, government information management guidelines and with legal frameworks."
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.