Intra coding in H.264/AVC has significantly enhanced video compression efficiency. However, computation complexity increases by the rate-distortion (RD) based mode decision. This paper proposes a novel fast mode decision algorithm in H.264/AVC intra prediction and its VLSI architecture. A novel edge-detection pattern is proposed and both edge-detection technique and spatial mode prediction technique are combined together to reduce the number of intra 4×4 candidate modes from 9 to an average of 2.50. VLSI architecture of intra mode decision module is designed with TSMC 0.18µm CMOS technology. The maximum frequency of 285MHz is achieved and 13.1k NAND gates are required. High frequency, efficient processing cycle reduction and small area make this design to be an excellent accelerator for HDTV 1080p@30fps real time encoder.