Under agenda item 2, our final evidence session this morning is on Audit Scotland’s “Quality of public audit in Scotland: Annual report 2024/25”. From Audit Scotland, I welcome back to the meeting Colin Crosby, chair of the board; Stephen Boyle, Auditor General for Scotland; Vicki Bibby, chief operating officer; and Owen Smith, interim audit director for audit quality and appointments. I open up the session to questions from members.
If I recall correctly, previous reports on the quality of public audit provided more information on audit inputs from your auditors, including internal auditors and external audit partners. Reports also included useful information from auditors who were surveyed for their opinions on, for example, whether they felt that they were able to carry out a high-quality audit, whether they had all the resources that were necessary to allow them to do their job and whether they were given the appropriate time, training and development to do so. It was quite helpful information, but it has not been included in this year’s report. Why not?
Good morning, Mr Greene. There are two reports. One is the report that you have before you, which is on the quality of public audit in Scotland. It is a summation of the totality of how well audit work has been delivered to the requirements set by Audit Scotland’s quality framework, which applies to Audit Scotland’s teams and the firms that we appoint.
We have the detail that you are asking for, and I will bring in colleagues in a moment who can talk you through some of that. However, as I mentioned, this is a summation report. Underlying it are transparency reports, which are produced by all the auditors who the Accounts Commission and I appoint. The reports set out how well they have done and how well they are performing against the external quality assessment of their work and any quality assessment that has been done by an in-house quality assurance team. They also set out, on the point that you are asking about, the views and experiences of the people delivering audit work.
I will pass to Vicki Bibby—Owen Smith might want to come in, as well—to give you some detail over and above what you have in the paper before you.
Independently, but with approval from the Auditor General and the Accounts Commission, the audit quality and appointments team—Owen’s team—sets up the framework for the audit quality standards. Section 5 of the report includes the audit quality survey based on what is asked. There were changes to the framework this year, which were agreed to by the Auditor General and the Accounts Commission. That is what the team went out and asked about, particularly in relation to the stakeholder perceptions in section 5. I will pass to Owen to give more detail on that aspect.
Good morning. We revised the audit quality framework last year with the approval of the Auditor General and the Accounts Commission. We had previously included staff surveys from all appointed auditors. However, in reality, it was never that easy to gather information, because it is done differently by different appointed auditors or audit firms. The transparency reports, which the Auditor General mentioned, include information on audit firms’ responsibility to ensure that their staff are adequately trained and have enough resources. That is because the international standard on quality management was brought in, on which we have an audit quality indicator to ensure compliance.
There are two reasons for not including that information. We could not get consistent information across all the appointed auditors and, to comply with the international standard on quality management, they were doing it anyway and reporting results through their transparency reports. We tried to streamline the “Quality of public audit in Scotland” report and allow the transparency reports to be the place where appointed firms discuss what they are doing to ensure that their staff are appropriately trained and have enough resources.
To clarify, does that mean that you no longer survey the auditors in external firms and you rely on those firms surveying their own staff?
We never surveyed the firms on that question. The approach relied on their existing procedures to meet the audit standards on whether their staff were—or perceived that they were—properly and appropriately trained, had enough resources and so on.
An important change for us was the bringing in, about two or three years ago, of the international standard on quality management. As I said, we have an audit quality indicator to ensure compliance with that, and we ask the Institute of Chartered Accountants in England and Wales to ensure that the arrangements in its review audit files are in line with what it expects to meet its standard. The standard is all about quality arrangements to ensure that audits are delivered to standard; it includes aspects such as resourcing. We removed it from the “Quality of public audit in Scotland” report because inconsistencies in the data that we were getting back meant that it was not able to be trended. However, that data is reported in the transparency reports for Audit Scotland and the six firms.
Where would I find information on internal Audit Scotland staff? What level of surveying has been done there? How useful or reliable is the data on their happiness or contentedness with what they are given to allow them to do their jobs?
As Owen said, elements of that are in the transparency reports. Separately, we undertake robust staff surveys. We are pleased to report that, as part of the Best Companies survey, which we have been doing for more than a decade, we have reached one star employer status, which we have only ever achieved during the years of the pandemic. That is a good result for us. The survey looks at how staff feel as part of the organisation and as part of their team and at whether they are appropriately trained to do their job. We got those results in March. We can get the data at high level and broken down by each team, and we take those details quite seriously. We also undertake health and wellbeing surveys, the next one of which will be done in September.
As a side question to that, in our earlier session we talked particularly about some of the costs involved, but I know that it has been a bit of a tough slog for many public organisations to try to bring people back together and get people back into the office. Is there a direction of travel on that for your organisation? Has it stabilised? Are you still trying to encourage people, particularly your auditors, to work face to face more? Has there been any resistance to that?
12:00
There is a real drive around the benefit of people being together but we have purposely taken a balanced approach because, equally, having an attractive hybrid approach is a way to encourage people to work with us. However, we are clear that hybrid working is not home working. The definition of hybrid working is a mix of home working and being in the office.
As I highlighted earlier, it is not unique to us, but in our role as a training provider, although we can do some of the more technical training online, people also need that onsite training. It is absolutely vital that they get face-to-face training on the relationships and the curiosity that you need as an auditor. We can corroborate that; the Institute of Chartered Accountants of Scotland results for our training are really strong and have improved year on year. The more that we do face-to-face training, the more that the results improve.
We are relatively small but we have 350 staff and bringing everybody together has a cost. We had a financial audit training session where we worked with a public sector host to bring our financial auditors together for training on some new standards, particularly on tricky areas such as asset valuation, and to show them what our new audit modernisation project would be like. We had a staff survey on that, and the training was seen as hugely valuable.
We are trying to take a balanced approach. I know that many organisations are going down the route of mandating the three days a week in the office. We are not there and we do not feel that we need to be there. We have been putting extra work into manager training on the expectations around face-to-face meetings. This year, we have introduced a new performance management framework that sets out the relationship between the manager and the individual and the expectation about the level of interaction. We are taking the approach of providing clarity around expectations without that top-down, draconian, “You must be in either three days or four days a week” approach.
There is a debate as to why not. It is a competitive working environment and other companies may offer different working practices that are more favourable to people, so you have to stay competitive, I guess.
I am happy to leave it there. I may have other questions on the audit quality later.
I have a couple of questions on audit delays. Paragraph 20 of the report tells us that 91 audits are delayed. In fact, when I look at it in a bit more detail, it tells me that 91 audits are late and “not making progress”, so they are not just delayed but stuck, it seems to me. I wonder whether you can address that. Can you also address this point? This is probably an unfair way of framing it but, if I could be simple in my approach, at the start of the report you talk about between 233 and 253 audits being completed; if there are 91 delayed audits, that is a ratio of between 35 to almost 40 per cent of audits that are delayed or “late and not making progress”. That is a huge proportion, is it not?
It is absolutely higher than we want it to be in terms of the number of audits. An audit does not sit in isolation. It is an audit of a set of annual reporting accounts of a public body, so we are talking about a public body being able to lay its audited annual reporting accounts in Parliament or being able to meet a deadline set by the Government or the Accounts Commission.
Of course it matters because timeliness of delivery is a feature of the accounts, their usefulness and the quality of the audit. To reassure the commission, we absolutely recognise that and we are looking at the matter. It is broadly consistent with what we knew was going to be a phased recovery period.
On some of the statistics, it might be helpful to set the context of what we are talking about. In the 2024-25 year, 91 per cent of all the national health service audits were delivered by the deadline of the end of June—that is, within that three-month window from the end of the financial year to the end of June. That is an increase from 74 per cent the previous year. For further education, it was 70 per cent, up from 67 per cent; central Government was 60 per cent, the same as the previous year, and for local government, it increased from 29 per cent of audits delivered—a really low percentage—up to 42 per cent.
It also matters how late the audits are. In some cases, we are talking about a few days or a few weeks. They are still late and that still matters, but 90 per cent or so were delivered by the end of the calendar year. By way of context, as we talked about in the earlier session, in some cases, we are still auditing over a three-year period. There are some audits going back to 2022-23. We have a few additional ones in 2023-24, with the majority now in 2024-25.
I have confidence, Mr Leonard, that we are progressing in terms of the phased recovery period, given the hugely disruptive effect of Covid on the delivery of audit work, and that we are on the right path. I can bring in Owen Smith to give a bit of detail on any specifics and what it means for an audit to be late if you would find that helpful. You are right that we are not yet where we want to be, but I am seeing signs of progress.
Before you bring in Owen Smith, can you expand not just on “late”—we all understand “late” and that there may be understandable reasons for lateness in the completion of an audit—but the expression “not making progress”? That is of much greater concern to me than audits being late.
Some of the wording is quite specific. Owen Smith can explain what that means.
We have brought in the phrase “making progress” to try to show either inertia or movement towards improvement. That means that the audit is a month better than the year before. If an audit is signed off a month ahead of where it was last year, it is making progress.
Audits might look as though they are stuck and not making progress for many reasons. It could be because firms are prioritising the ones that were on target and making sure that they do not become jeopardised. For instance, in the year ahead, I know that two firms plan on delivering all their audits on time; one firm had seven audits that were late last year but it is expecting only two to be late this year because it has prioritised staffing and resources, in partnership with the audit body, to make sure that it can bring the audit back on time.
There is no point in an auditor turning up and saying, “I am here to do an audit”, if the body is not ready. It is about that partnership and getting them aligned together. It might take a couple of years for some of these audits but it is about not jeopardising the ones that are on time and about making sure that the ones that need to catch up can do so at the right point in time. They will, I hope, be making progress in 2024-25.
But they are not like health and safety inspectors, who turn up unannounced, are they? I would expect that an audit is a planned operation between the public body that is being audited and the auditor—whether they are internal or from an external firm—who is carrying out the audit.
You are quite right. There is a partnership, between the auditor and the audited body, to deliver the audit. There is no merit in an auditor turning up to a deadline that has been set by the Accounts Commission, by me or the Parliament, if the public body is not ready to support the delivery of that audit.
On the language, the phrase “not making progress” does not necessarily mean that audit work is not happening. It can be—and in many cases it is—still happening, but it is not yet accelerating the recovery from the deadline to where we want that audit to be.
Forgive me for repeating myself, but, although there are concerns about some of the delivery timelines, across the piece we are seeing progress, especially compared with last year. As we have seen from Owen Smith’s and Vicki Bibby’s support for and engagement with auditors and audited bodies, the key focus is the handful of audits of public bodies that are not making the progress that we want them to. Where there is that level of concern, for good reason, we are keen to avoid a situation where we would be straddling more than one five-year audit appointment cycle. Considerable effort is being made to recover the deadlines for all audits as we come to the exit from the current cycle in two years’ time.
In paragraph 18 of the report, you refer to one outlier public body, which has not had its audit for the financial year 2022-23 completed and, therefore, not had one completed for 2023-24 and, therefore, not had one completed for 2024-25. Again, that rings alarm bells with me, both as convener of the Public Audit Committee and as a member of the commission. In the end, this is about public money. It is about assurance for that body, and the good governance of that body. I do not know how big that particular organisation is, but nonetheless we are talking about whether a proper audit of public money is being undertaken.
As Auditor General, I absolutely share your concern about that organisation, with which I am familiar. As I mentioned, timeliness of delivery and laying of accounts is a feature of the quality of assurance that is given to the Parliament that the money that was voted for and allocated to that public body was spent in accordance with the expectations at that time.
There are some mitigating circumstances in respect of that organisation. As you are convener of the Public Audit Committee, you will know well that, as Auditor General, I can exercise the option to lay a statutory report on the audit of a public body’s annual report and accounts. That is one of the reasons why the audit has not yet been completed or laid. I can say that all the audit work has been completed, and in respect of that audit we are going through clearance with the public body on the comments on our section 22 report.
On the wider recovery issue, as Owen Smith has alluded to, we have seen evidence that the audit process does not stop for 12 months and then pick up again. We are seeing an acceleration and a compression of the recovery period, so that a quick catch-up process is now being undertaken. The work for the year 2023-24 is well under way, if not completed, and that for 2024-25 will be completed shortly thereafter. We can see that the various interventions and steps taken are making a difference. However, your overall point is absolutely fair, which is that all of us want to see audits being completed in a timely way so that they are useful and relevant.
Perhaps I can finish where I started, by going back to your figures, which say that 91 audits are late—or, to use Mr Smith’s terminology, “not making progress” according to his yardstick. However, you also give us a breakdown that, of those 91 cases, 46 are late due to the auditor, 27 are due late to the body that is being audited and 18 are late in circumstances that are “beyond the control of either”. Again, that is your explanation. I am not quite sure what “beyond the control of either” means. How can something be beyond the control of either the auditor or the public body that is being audited? That baffles me.
All three of my colleagues want to come in, and I hope that they will be able to reassure you on those details.
Excellent.
I will pass to Owen Smith, then Vicki Bibby and then Colin Crosby.
The wording “beyond the control” is largely used of cases that were so late the year before that they could never have recovered within the year. That is not good, but a handful of those cases need to be looked at quite closely. The report says 18, but they are almost beyond the point where they can come back within the audit year in question. We are now in the third year of the five-year audit appointment cycle, which is where we expect recovery to start taking place. When we looked at the annual audit plans we found that 80 per cent of them were in on time, which reflects bodies’ plans to deliver the audits themselves on time, so there has been lots of good progress in year 3.
It can be very difficult to define exactly why an audit is late, which is why we have picked out the main reasons. The one that I mentioned earlier is about auditors prioritising audits that are already on time, and not jeopardising those that need recovery and work within the body if they are to come back in the next year or two to bring them back to operating on time. When the report says that the auditors are primarily responsible, that means that they will not prioritise the delayed audit at the risk of others that are already on time.
Given his expression, I wonder whether Mr Leonard wants to come in there.
12:15
I am sorry—that just reminds me of the narrative that we get on Ferguson Marine, which is that to get the Glen Sannox afloat we have to jeopardise progress on the Glen Rosa, through retrofitting parts from the Glen Rosa on to the Glen Sannox so that at least that vessel is afloat. However, what you described does not solve the problem of what is happening with the public body whose audit is being considerably delayed and jeopardised, does it? I understand that that is the way of working. The reason that I am making facial expressions is because it just reminds me of what we get told on other aspects of audit.
The report that you have before you describes a pragmatic phased recovery of audit delivery. Our engagement with the SCPA, and the support that you have provided us with in modernising our organisation and during the early stages of the pandemic, has led us to where we are today. We did not look to seek a significant injection of resource—even if it were available, which would have been debatable—whereby a pool of auditors would be waiting to come in to deliver public audit. That is regardless of the fact that, as Owen Smith set out, a public body might not be ready to deliver the audit through its own capacity and expertise.
Adopting something of a middle ground, by having a phased recovery, was the approach agreed among the Audit Scotland board, the Accounts Commission and myself to get ourselves to a point, by the end of this five-year cycle, whereby almost all the audits will be back on time and being delivered as expected. Broadly, I think that that was the right call. It allows for the fact that not all public bodies will have their audits delivered on time. As we have hoped to set out clearly today, in some cases that will be due to the capacity of the audit team or the audit provider, which has not been able to deliver in the way that they would have done before the pandemic. Therefore, at this stage we are in a place of recovery rather than having a perfect solution.
I add that, if we stick to the plan that we have, we should be on track for the end of this contract term. We have been working with the Auditor General, the Accounts Commission and the controller of audit to monitor which local government bodies are now on track when measured against the plan, and to provide for early intervention where they might slip. We have a rigorous approach to that.
We are also writing out to audited bodies to reinforce the importance and priority attached to producing a set of accounts. We do not want to find ourselves in the unfortunate position that has been reached south of the border. We have been working closely with the Ministry of Housing, Communities and Local Government on what its approach with the local audit office will be. It is really important that we highlight and stress the importance of the public audit process producing audited accounts. The Accounts Commission is closely watching the issue as well. I want to highlight that on its behalf—not that I speak for it—and to assure you that it has been looking at that very closely.
Mr Leonard, I am delighted that you have taken that line of questioning, which is similar to one that our board and the audit committee take and which is very fair.
I am not quite sure about the analogy with the Ferguson shipyard. However, if we take March 2024 as being the 2023-24 year-end, and if we take the number of audits delivered by 31 March, using your analogy we have 233 boats that are floating, working and approved, and a balance that are still at the dock. The real issue with that, which the board is increasingly examining, involves asking, “You have your critical path analysis of 253 audits. When are they going to land?” All the lateness criteria form part of our internal critical path analysis. Quite rightly, everybody internal to the process gets beaten up about lateness. However, the board has a slightly more holistic view of that and asks, “Are we are doing the bulk of the work that we say we will do, in the year in which we should be doing it? For the 8 per cent that have not been completed, what are the hiccups and hold-ups? Are they legitimate, or are they our fault?”
To that extent, according to critical path analysis, the key issue is that you cannot put the resources to the right place at the right time. However, because of the pandemic and one or two other things, we do not live in a perfect world where that can be put out automatically. That is why, in a way, the 233 cases out of 253 is a more meaningful number—certainly to the board—provided that we know what is happening to the ones that are behind, and why, and whether we can get them brought forward. That is what our line of questioning is. Therefore, it is a very valid and appreciated line of questioning that you are following, because it is absolutely at the forefront of the board’s thinking every time that we see this issue.
Thank you. I could continue, chair, but I do not think that time permits so I will pass back to you.
You are quite correct. Time is tight, and I ask for questions and answers to be tight, too.
I want to focus on the external quality review that the ICAEW conducted. I am aware that it looked at eight audits, five of which got a good bill of health. The other three, which were conducted by Audit Scotland’s in-house teams, included significant criticisms. Without repeating all the findings of those reviews, what were the reasons for those criticisms? What learning has taken place, and what actions have you put in place to weed out the organisations whose audits showed poor performance? We must bear in mind that only eight were looked at.
We have had a long-standing engagement with an external quality provider. We are now with the Institute of Chartered Accountants in England and Wales, and we were previously with the Institute of Chartered Accountants of Scotland. External assessment, validation or recommendations for improvement are integral to our audit quality framework. Importantly, for me and the Accounts Commission, that covers not just our financial audit but our performance audit and our public reporting, too, so that we have assurance that we when make judgments publicly, through the Parliament, they are reliable. The position is similar for our financial audit process, when the external quality provider gives us assurance on whether the audit opinion on public spending is reliable.
I will address the totality of the findings. On performance audit and best value, we are very pleased that we can see consistently strong results on our public reporting, our section 23 reports, and our best-value reporting. A score of two or one on each of those is very good. I am paraphrasing some of the language used in the findings, which is quite specific.
On our financial audits, we see a range of scores. For example, the exhibit on page 17 is on the quality of public audit reports. The audit services group of Audit Scotland has ticked every box, and we have scores of one, two, three and four for the financial audits. On the scores of one and two, again, it is very satisfying that the Institute of Chartered Accountants in England and Wales has said that our audit team is meeting the standards. Those are high standards. They are the same ones on which the regulator, ICAEW, will be holding all the big four, and the top 10 firms, to account on their audits of public limited companies, charities and every other commercial organisation. It is quite right that Audit Scotland has shown that it can meet those standards, and it is also satisfying. A score of three is not where we would want to be. It shows that improvements are required on the relevant audits.
I should point out that Audit Scotland audits a wide range of organisations, from the Scottish Government, large local authorities and health boards, down to some very small bodies and charities of local government. We require some of those bodies to meet the international standards on auditing. Therefore, for every audit that does not meet the required standard, the audit team is required first to do a lessons-learned or root-cause analysis—to find out what went wrong during that audit—and then to come up with an action plan of necessary improvements.
I will bring in Owen Smith if he wants to add anything, but I add first that, through the work of his team, we also examine whether there are any systemic issues with our audit approach that perhaps have been presenting as relating just to the body under review but might apply more widely. We will take that through our audit quality group and committee for them to look at our audit approach—again, so that we are satisfied that we are applying the learning points.
Given that it is what I do myself, day in and day out, I know that having their work reviewed can be stressful for teams; it can weigh heavily upon them to have their work checked. The purpose of the report is as much about giving assurance as it is about promoting learning. It is not necessarily about holding people to account directly but about supporting change and improvement across their work, especially through root-cause analysis. That is the approach that we adopt: if we get a tough score, we apply the learning from it and take that through into our audit work.
I will pass to Owen Smith if he wants to add anything.
The only point that I will add is that ICAEW identified a thematic issue in the previous year’s reviews on valuations, and Audit Scotland took action to improve that. Page 22 of the report includes a summary statement that that issue was not found this year. That shows, in a simple way, how inspection findings are taken seriously by the team in innovation and quality and passed out to audit teams to make sure that improvements are put in place for the next year’s audit. It was very satisfying that that issue had not happened again. That is reflected in the very good scores—top scores for a big council and an NHS board as well—because that was where issues had been found in the previous year. Therefore, that approach does work. It is never good to get a score of four, or even three, but the important thing is that the team does work to identify the root cause of why things went wrong and to improve them for the next year.
You also mentioned systemic issues. Were any such issues identified through the scores of three that revealed poor quality?
For this year, no. I used the word “thematic”, which is what we use in the report, but “systemic” is a good equivalent. There were issues for the scores of three on investment testing, sufficiency of audit evidence, and a very small point of auditing practice. Audit files should exist on their own for the audit opinion, and one audit had done too much cross-referencing to a bigger audit to which it was affiliated on the system, so it was not treated in a standalone way. There needed to be more evidence on file to back up the full flow of the auditor in getting to his opinion, but that was not there.
Should sample sizes should be increased in future years? Does a sample size of three from your own in-house teams provide a robust evidence base, or is that very much a random snapshot?
Broadly speaking, the sample sizes are right. We want to cover all the providers that the Accounts Commission and I appoint. The Audit Scotland in-house team will receive more because, as we have touched on, they cover two-thirds of the appointments. Because we cover such a wide range of bodies, some audits are undertaken by colleagues of a different level of seniority. Therefore, we have tried to match the smaller bodies, or the less complex ones, with our senior managers to give the audit opinion on those bodies, and an audit director will more likely sign a larger body.
I hope that this will be the final bit of reassurance that I will need to provide. In the event that we have not had the quality score that we want, which is a one or a two, we do a quick follow-up audit inspection of that audit and audit team to ensure that the root-cause analysis process has worked, the learning has been embedded and there are no repeat issues. Having evidence that the interventions are working and that the learning and development approach is correct broadly satisfies me that that level of sample feels about right for the time being, but we will keep that under review.
Thank you very much for that. At this point we will draw the meeting to a close. I thank Colin Crosby, the Auditor General, Vicki Bibby and Owen Smith for their evidence.
I close this meeting and wish everyone a great summer recess.
Meeting closed at 12:30.