This report summarises the methods, findings and lessons learnt from an Australian Government trial of Microsoft 365 Copilot, an artificial intelligence (AI) tool, at the Department of the Treasury.

The trial aimed to understand:

• whether and to what extent Copilot could be used as an example of generative AI
• whether it could be implemented safely
• what benefits and challenges it poses in the short and longer term. 

Further, the trial aimed to understand the effect of Copilot on:

• staff productivity
• efficiency
• process improvements
• employee‑related outcomes. 

The evaluation used a mixed-methods approach, with data collection from trial participants and their managers.

The trial found that generative AI can be deployed within the Treasury environment, but there were some limitations. Copilot was useful for some purposes, with almost two‑thirds of users finding it appropriate and beneficial for basic administrative tasks and work processes. However, Copilot’s value for more complex work tasks was limited. 

Access to Copilot had other unintended benefits, including improvements in accessibility, inclusion and work confidence. The report concludes with recommendations for how any future implementation of generative AI products in Treasury could be improved.

Recommendations

• Provide clear and specific use cases to distribute licences to staff who can demonstrate likely benefits and time savings.
• New AI products should be based on a phased approach.
• Include an assessment of the appropriate level of investment in education and training.
• Develop guidelines to support the transparent use.
• Monitor products over the long-term to determine potential impacts on quality and timeliness of work.
• Staff outcomes should be considered as important secondary outcomes.
• Conduct periodic assessments of whether emerging generative AI products may be better suited to Treasury’s security requirements and existing IT infrastructure.