For the average consensus of multi-agent systems with lth-order chain integrator dynamics, it is important to build the LMI-based delay-dependent stability criterion with jointly-connected topologies. Using the idea of state decomposition, the condition was converted into verifying the stability of zero equilibrium of disagreement system. Considering multiple time-varying communication delays, common Lyapunov-Krasovskii functional was employed to analyze the stability of zero equilibrium. In order to relax the conservativeness, Free-weighting Matrices method was employed in the main results. After matrix order-reduced treatment, the tolerant upper bounds on communication delays were obtained through solving feasible linear matrix inequalities (LMIs). Numerical examples and simulations were presented to demonstrate the effectiveness of the proposed method. Different from the existing literature, the proposed stability criterion was characterized by lower conservativeness, simple formation of the solution, and wide range of time-varying delays. It can be applied to instruct the analysis and discussion of the average consensus of multi-agent systems with complicated communication conditions.