Lori - Certainly it's nice to have a dedicated team from a resource perspective. However, I don't think it's bad to structure it the way you indicated. My main concern would be the risk of "higher priorities" causing you delays in actually getting to conduct the audits. The other factor to consider is the size of your organization. It's not unusual for companies to have a shared resource structure until they hit a certain size at which point a dedicated team seems warranted.
The audits themselves could be very successfully utilizing resources that really "know" the ins and outs. It's mostly the risk of distraction of day to day operations that would worry me in the big picture. I was in an environment recently where we did not have a dediciated team and that is exactly what happened and we delayed many of our audits from the original schedule. But if you can ensure that you will stick to an audit schedule without resources being pulled away, it could be successful. The bottom line is that it's better to do something than nothing. If for budgetary reasons, you structure it with resources that carry multiple responsibilities, I would suggest proceeding with it and documenting the beneficial outcomes in a way that may help you justify dedicated resources in the future if you experience the concern I have or if your institution grows to a size that warrants.