Large-eddy simulations were carried out to simulate laboratory-scale isolated buoyant convection in unstratified water with shelf and slope topography in the presence of rotation and to compare and complement the experimental study of Jacobs & Ivey (1998) under the same conditions. The simulation code developed in this work was a three-dimensional incompressible Navier–Stokes solver and the simulation runs were performed on a distributed memory massively parallel computer, namely the IBM SP2, to study the effects of different applied heat fluxes and system rotation rates. We are able to show for the first time the detailed temporal evolution and spatial structure of the three-dimensional convective flow field. Rayleigh–Bénard instability in the form of circular concentric convective rings is recognized in the initiation process of the convection. The onset of Rayleigh–Bénard instability was investigated and the critical Rayleigh number was found to increase with Taylor number only when the Taylor number is greater than 5 × 103, where both non-dimensional parameters are based on the conductive layer thickness. The horizontally axisymmetric convective rings later break down and evolve into a quasi-two-dimensional vortex field. An azimuthal rim current develops around the periphery of the convective region. Our simulation results confirmed that the rim current velocity scales as Bt1/2/Hf3/2. Here B is the buoyancy flux applied over a bottom circular disk, f is the Coriolis parameter, t is the time and H is the distance between the tank bottom and the shelf. With increasing lateral temperature gradient the rim current undergoes a baroclinic instability. Our study of root-mean-square velocities in the convective region suggests that the transition from the buoyancy-flux-controlled to background-rotation-controlled flow occurred when the natural Rossby number Ro* became smaller than a critical value between 0.015 and 0.044. The simulation results of the convective overturning time, the wavelength of the baroclinic eddies and the density anomaly at steady state are all in reasonable agreement with the experimental data.