Social dynamics are often driven by both pairwise (i.e., dyadic) relationships and higher-order (i.e., polyadic) group relationships, which one can describe using hypergraphs. To gain insight into the impact of polyadic relationships on dynamical processes on networks, we formulate and study a polyadic voter process, which we call the group-driven voter model (GVM), in which we incorporate the effect of group interactions by nonlinear interactions that are subject to a group (i.e., hyperedge) constraint. By examining the competition between nonlinearity and group sizes, we show that the GVM achieves consensus faster than standard voter-model dynamics, with an optimum minimizing exit time {\tau} . We substantiate this finding by using mean-field theory on annealed uniform hypergraphs with N nodes, for which {\tau} scales as A ln N, where the prefactor A depends both on the nonlinearity and on group-constraint factors. Our results reveal how competition between group interactions and nonlinearity shapes GVM dynamics. We thereby highlight the importance of such competing effects in complex systems with polyadic interactions.