“I would also say that there are risks that we can be overreliant on it, and there are risks that we will overly trust what it says.”

He continued that AI “definitely does hallucinate” and says things “very confidently” wrong, which makes it even harder to detect.

“If there is a rule of thumb, one that I've said a few times to people, is think of large language models such as ChatGPT as like an extremely confident undergrad – you need to sanity check it like you would with an inexperienced but eager uni student.”

Shelley Banton, head of technical for ASF Audits, said one major area of concern is that ChatGPT is starting to replace “the man in the pub” in terms of trustees getting their information.

“[Clients] are coming to professionals and saying, ‘Hey, my friend told me about this’, where in fact they've looked it up on ChatGPT and it's rubbish,” Banton said.

Forbes added that there are a few ways to ensure that professionals are using AI to its best advantage.

“As an example, sometimes we use different AI models to cross check and rank the responses of other AI models so you can try and get a second opinion in addition to your own,” he said.

“You have to act like a manager or supervisor. [Treat AI] like you're getting responses from your staff and review the work AI is doing for correctness.”

There are risks, especially when using AI in the SMSF sector, particularly if it is being used by members and trustees who may not understand the nuances of regulations.

“Members and trustees who are going to ChaGPT and getting advice on what they should be doing, it could be completely incorrect.”

“I'm a champion of experts in the industry. I think we need more, not less. There's definitely risks with people relying on things like references to the SIS legislation that does not exist, so you really have to know what you're asking AI and what you're expecting to get before you start typing.”