To evaluate performance of a large scale digital system, the system is modeled by stochastic timed petri nets (STPN) and performance index is estimated through iteration of a petri net simulation. When the performance evaluation is carried out in a large distributed environment, parallel calculation with incidence matrices of STPN can make the petri net simulation more efficient. In this paper, we proposed an ordering method of incidence matrices to reduce the communication costs in the parallel calculation of matrix operations. For this purpose, we first proposed a method for blocking the incidence matrices based on a structure of a system by considering synchronous and asynchronous buses as composing elements. We also proposed a method for converting sub-matrices of incidence matrices to band matrices. This method is based on the bread first search of STPN represented by the sub-matrices. Finally, we demonstrated an effect of the ordering on the time for performance evaluation by carrying out the performance evaluation using incidence matrices generated by the proposed method. In addition, we showed that the ordering method could reduce the communication costs in a large distributed environment by estimating the use rate of calculation nodes.