If you are beginning the process of looking into obtaining a SOC 1 or SOC 2 report you more than likely have a lot of questions. When speaking with prospects, many have questions related to the process of how a SOC 1 or SOC 2 audit is conducted – particularly questions with regard to the timing, the level of effort, cost, and how exactly the controls outlined within the report will be tested.
One of the commonly asked questions we receive, that I would like to focus on in this article, is: What is the difference between testing the design of a control vs. testing the operating effectiveness of a control. Similarly – how are both of these types of tests conducted during an audit?
Therefore, in this article, I will shed light on the difference between how an auditor would go about testing the design vs. the operating effectiveness of an organization’s internal controls, along with some specific examples.
What is the Difference Between Test of Design and Test of Effectiveness?
How do You Perform a Test of Design?
The test of design of an internal control would validate that the control that is stated to be in place by the organization has indeed been established and put in place.
An example test of design would be that an organization notes that they have controls around the hiring process, one control being that background checks are conducted on all new hires. In order for an auditor to test the design of this particular control, the auditor would look to see that a background check was conducted on one example recently hired employee. This information for the one example employee would confirm that: Yes, the organization has a process in place to perform background checks for new hires.
By confirming this, the audit organization would be able to validate and opine within a Type I report that the organization has designed the control they are claiming to have in place with regards to conducting background checks for new hires.
Another example would be controls around the change management process. If an organization is stating that they have a process in place to ensure all changes made to their production system are authorized through appropriate reviews, and testing prior to implementation; the design of this could be tested. In this particular case, in order to test the design of the control, typically the test procedure would be to validate that for a recent change implemented the following elements occurred:
- the change was reviewed (typically peer-reviewed)
- testing of the change occurred (typically automated testing and human testing)
- the change was approved by appropriate personnel
If this information outlined above is available for the example change, the auditor would be able to confirm that the change management internal control process was in place. In other words, this confirms the control has been designed as stated.
What is a Test of Effectiveness?
In short, the test of effectiveness of a particular internal control is whether or not the control operated consistently over a period of time in the past (typically 12 months). Specific examples of this to further clarify are outlined in the next section.
How Do You Measure Operating Effectiveness?
The test of operating effectiveness of a control is confirming that a control that is stated to be in place by the organization has been established for a period of time (typically 12 months). With a Type II report (Either SOC 1 or SOC 2) the test of the operative effectiveness of controls will be required.
Specific Examples of Operating Effectiveness Testing:
Going back to the background check example control noted above, we looked at how to test the design of the control. Now we can look at how an auditor would test the operating effectiveness of that same control. To test the operating effectiveness the auditor would need to look at a sample of new hires (more than one) across that last 12 months.
The auditor would then confirm that a background check had been conducted for each sampled new hire (vs just looking at one example, as is the case with testing the design of the control). By looking back in time, and testing a sample of new hires that were hired in the last 12 months, we can test the operation of the control. Hence, this sample testing method can identify whether the control ‘operated effectively’ and consistently over that period of time.
Let’s look at the change management example from above as well. If we wanted to test the operating effectiveness of the same control, again we would have to do sample testing. With sample testing, the auditor would obtain a population (i.e. a listing) of all of the system changes that occurred during the audit window (again, typically looking back 12 months). They would then select a sample of changes from that population. For each sample change selected, the auditor would look to confirm that key controls in the process (i.e. the peer review, testing, and approvals) occurred before each change sampled was moved to production.
Some other examples include quarterly account reviews or that new user accounts established were approved by authorized personnel prior to provisioning.
Final example – if an organization claims that they conduct quarterly account access reviews and would like to add this control to a Type 2 report, the operating effectiveness would be tested. To test, the audit organization would be required to look at a sample of documented account reviews and confirm that the reviews occurred throughout the course of the audit period (again, looking back, typically 12 months). In this particular case, typically 50 percent of the total population (4) would be reviewed to confirm with enough assurance that the account access reviews, in fact, did occur on a quarterly basis.
Summary
In summary, testing the design of a control is a ‘point in time’ test. Testing the operating effectiveness of an internal control is testing the control operation over a period of time (typically looking back 12 months), which would require sample testing. Given that, a Type I report where only the design of controls are tested would require less time and effort. This is especially in comparison to testing the operating effectiveness of controls over a period of time, as done by using sample testing in a Type II report.
However, testing the operating effectiveness in a Type II report gives the readers of the report greater assurance around whether an organization’s internal control environment is functioning properly. As such, a Type I report is conducted to identify the established control environment and is always a stepping-stone for a more rigorous Type II report.
The objective of this article is to shed light on this commonly inquired about topic – however, if you have any questions or would like more information about Linford & Co or our services, please contact us.
Linford & Co., LLP, founded in 2008, is comprised of professional and certified auditors with specialized expertise in SOC 1, SOC 2, HIPAA, HITRUST, FedRAMP and royalty/licensing audits. Our auditors hold CPA, CISA, CISSP, GSEC licenses and certifications. Learn more about our company and our leadership team.