With the rapid development of internet technology, the transmission and access of image items have become easier and the volume of image repository is exploding. An efficient and effective query reformulation is needed for finding the relevant images from the database. Relevance feedback (RF) is an interactive process which refines the retrieval results to a particular query by utilizing the user’s feedback on previously retrieved images. Most of the existing approaches deal with hard feedback (relevant and nonrelevant) and focus on individual experience only. We propose to facilitate the use of soft feedback (involving excellent, fair, don’t care, and bad) to better capture user’s intention. To add this feature, all of the traditional RF techniques should be modified accordingly. Further, the meta-knowledge exploited from multiple users’ experiences can improve the performance of future retrieval results. We propose a soft association rule mining algorithm to infer image relevance from the collective feedback. The number of association rules is kept minimum based on confidence quantization and redundancy detection. Also, binary search and best-first search techniques are implemented to expedite the process of relevance inference from the association rules. The proposed model provides a more flexible interface for relevance feedback and the experimental results manifest that the retrieval performance of the proposed model is better than that of traditional methods.