Group theory is the study of mathematical structures called groups, which have properties that are used to classify molecules by symmetry.
The classification of a molecule according to its symmetry in turn allows us to analyse its molecular properties. In the field of quantum chemistry, group theory simplifies calculations, e.g. in determining if overlap integrals are zero and investigating the degree of degeneracy of a system. Group theory is also useful in predicting spectroscopic transitions, especially in electronic and vibrational transitions.
To fully understand the theory, we need to have some knowledge of abstract algebra and apply it to derive the great orthogonality theorem.