Gear faults manifest in the frequency spectrum as multi-order modulation sideband clusters phenomenon center on the meshing frequency and its higher-order harmonics, spaced by the gear rotation frequency. In order to automatically focus the fault side frequency components, a method of fault side frequency cluster extraction with penalty regression was proposed. Adaptive sparse group lasso regression self-data-driven strategy was used to determine the penalty coefficient size and update the spectrum weight online to find the fault sideband clusters. Based on the sideband weight coefficients obtained from sparse group lasso regression, a new index called the sparse group lasso sidebands indicator was proposed for the health monitoring of gear transmission systems, enabling the early fault warning and location of gear transmission systems. Results analysis show that the proposed method can provide more accurate gear early fault detection and fault location.