Gear faults manifest in the frequency spectrum as multi-order modulation sideband clusters centered on the meshing frequency and its higher-order harmonics, spaced by the gear rotation frequency. In order to automatically focus the fault side frequency components, a method of fault side frequency cluster extraction with penalty regression was proposed. The Adaptive sparse group lasso (ASGL) regression self-data-driven strategy was used to determine the penalty coefficient and update the spectrum weight online to find the fault sideband clusters. Based on the sideband weight coefficients obtained from Sparse Group Lasso (SGL) regression, a new index called the Sparse Group Lasso Sidebands Indicator (SSI) was proposed for the health monitoring of gear transmission systems, enabling the early fault warning and location of gear transmission systems. The results show that the proposed method can provide more accurate early fault detection and fault location results.