Device-to-device (D2D) communication in cellular networks is defined as direct communication between two mobile users without traversing the base station (BS) or core network. D2D communication can occur on the cellular frequencies (i.e., inband) or unlicensed spectrum (i.e., outband). A high capacity IEEE 802.11-based outband device-to-device communication system for cellular networks is introduced in this paper. Transmissions in device-to-device connections are managed using our proposed medium access control (MAC) protocol. In the proposed MAC protocol, backoff window size is adjusted dynamically considering the current network status and utilizing an appropriate transmission attempt rate. We have considered both cases that the request to send/clear to send (RTS/CTS) mechanism is and is not used in our protocol design. Describing mechanisms for guaranteeing quality of service (QoS) and enhancing reliability of the system is another part of our work. Moreover, performance of the system in the presence of channel impairments is investigated analytically and through simulations. Analytical and simulation results demonstrate that our proposed system has high throughput, and it can provide different levels of QoS for its users.