Audit Sampling in SOC Examinations

In completing SOC 1 and SOC 2 examinations (and most other types of audits), there is testing involved to determine the operating effectiveness of controls. There are different types of tests that can be applied to testing controls (for more information on the five types of tests refer to our article, Five Types of Testing Methods Used During Audit Procedures), and to complete a majority of these tests there is a sampling of populations that are required. In this post, we cover what audit sampling is and provide guidance on how to apply audit sampling to get to a confident conclusion on the operating effectiveness of controls.

According to the AICPA (in SAS No. 122 AU-C Section 530), audit sampling is defined as “The selection and evaluation of less than 100 percent of the population of audit relevance such that the auditor expects the items selected (the sample) to be representative of the population and, thus, likely to provide a reasonable basis for conclusions about the population.” The entire AICPA audit sampling guide can be referenced here.

Why do Auditors Use Sampling? What is the Purpose?

The definition from the AICPA is a little wordy, but to summarize, as auditors, the purpose of audit sampling is to allow us to do the right amount of testing to confidently determine the operating effectiveness of controls. This does not mean we can always test 100 percent, or even have the capacity to. Therefore, sampling comes into play in testing. But what is the right amount and how do you figure that out?

As auditors we need to consider three primary areas when performing audit sampling: 1) sample method, 2) the sample size, and 3) tolerable rate of deviation.

What are the Different Types of Audit Sampling Methods?

There are four main types of audit sampling methods that are used when completing tests of controls in SOC 1 and SOC 2 examinations. The type of population, how it was generated, and the size of the population can have an impact on the type of audit sampling methodology that is chosen for testing. The four main types include:

• Simple Random Sampling – Every unit has the same probability of being selected. This type of sampling can easily be accomplished by assigning a number to each item in the population and then using a random number generator to randomly select numbers in the range of the population (there are online tools for this, apps, and even Excel formulas can be used to generate random numbers).
• Systematic Sampling – This method selects samples using internals which are a result of dividing the population of units by the sample size. For example, if there are 250 items in the population and 25 will be selected for testing, 250 is divided by 25 to come up with 10, therefore every 10th item in the population will be selected for testing.
• Haphazard Sampling – Similar to simple random sampling; however, random number generators or tools are not used, and selections are just made from the population without any bias.
• Block Sampling – Represents contiguous population items, for example, the five most recent transactions in a population or the five most recent events can be selected for testing. Block sampling would include testing 100 percent of a population.

Every SOC examination should follow one or more of these sampling methods for testing of the population. A walkthrough or inquiry only would not be sufficient to test all controls.

Statistical vs. Non-statistical Sampling

Statistical sampling requires that samples be selected at random, generally using a tool to generate random numbers. The simple random sampling method above would be considered statistical sampling.

Non-statistical sampling allows an auditor to use professional judgment when selecting samples. Non-statistical methods make a lot of sense when a population is very small, rather than spending the time setting up a statistical sample. While non-statistical sampling allows for auditor judgment, an auditor should always be careful not to include too much bias in selecting samples.

What is the Appropriate Sample Size?

There are a number of factors that need to be considered when determining the sample size.

1. The size of the population being tested.
2. The risk of the control. All the controls that the auditor has selected to test are significant controls, but there is a spectrum that exists regarding the significance of each control. It is important to consider the impact (qualitatively and quantitatively) if a control is not operating effectively.
3. How many deviations/failures would be acceptable in testing the specific control.

The tables below (Table 1 and Table 2) are what we use as guidelines when selecting our sample sizes in our SOC 1 and SOC 2 examinations. These tables align with the guidance set forth in the audit sampling guide from the AICPA.

Table 1 is used for larger sample sizes (250 or greater in the population) and shows recommended sample sizes to get to a minimum 90% confidence level. The table includes the sample sizes for up to two deviations and takes into consideration the risk of the control.
 

 
Table 2 gives further guidance sampling on less frequent operating controls and on smaller populations (transactional).
 

Audit Sampling Examples

Using the tables above a few examples would include:

• Example 1: A population of all employees is provided and consists of 389 people and you want to test that all employees are attending security awareness training. According to the table, expecting no deviations the initial sample would be 25 and simple random or haphazard sampling would likely be applied. If it is found that one of the 25 selected did not attend training the sample would be expanded to 40 people. If another deviation is found the sample would be expanded to 60. If another deviation is found sampling would stop and it would be determined that the control is not operating effectively.
• Example 2: The controls being tested states a monthly reconciliation is completed and you want to test that it was indeed completed monthly and reviewed by a manager. Using Table 2 above, you would select three months for testing. Using haphazard sampling you would pick three months from the year and test those months. Because the population is smaller, any deviations would be a failure of the operating effectiveness of the control.

Summary

The guidance from the AICPA is pretty extensive around audit sampling. SOC auditors should review their sampling methods to make sure they are aligned with the AICPA guidance when performing their examinations. Please contact us if you would like further information on sampling, testing methods, or any of the services we provide.