A technical architecture for user preference model is presented, and the nature of the problem represented within a Markov Decision Process(MDP) combined with adaptive reinforcement learning algorithm is displayed. We provided a possible candidate solution for user modeling dynamically to satisfy the user's expected preference based on minimal or missing information. It is also a exploration for the evaluation of the user experience when selecting service providers. Simulations of the user models show that the MDP model is effective for learning the user preference with multi-state profiles.