Restrict Group Organizer Access to Email Invites Only

"I don't want the group organizer to see every single user in the WordPress website. I only want them to send e-mail invities