Self-organizing behaviors are widespread in nature. In order to simulate self-organizing behaviors of the fish school through learning, the fish school simulation environment model, the agent model and the reward mechanism were built, and a multi-agent reinforcement learning approach based on Hebbian trace and actor-critic framework was proposed as well. This approach uses Hebbian trace to enhance the swimming strategy learning with memory ability and realizes the distributed learning of multi-agent based on the homogeneous hypothesis. The simulation results show that the proposed approach can be applied to self-organizing behaviors learning of the fish school in the scenarios of leader-follower, autonomous wandering and navigation. Moreover, the characteristics of the fish school based on learning methods is similar to that based on Boids rules.