Frustration-induced strong quantum fluctuations, spin correlations, and interplay between competing degrees of freedom are some of the key ingredients that underlie exotic states with fractional excitations in quantum materials. Rare-earth-based two-dimensional magnetic lattices possessing a crystal electric field, spin-orbit coupling, anisotropy, and electron correlation between rare-earth moments offer a new paradigm in this context. Herein, we present crystal structure, magnetic susceptibility, and specific heat results accompanied by crystal electric field calculations on polycrystalline samples of ${\mathrm{Ba}}_3{R\mathrm{B}}_9{\mathrm{O}}_{18}$ $(R=\mathrm{Yb}, \mathrm{Er})$, in which $R^{3+}$ ions form a perfect triangular lattice. The localized $R^{3+}$ spins show neither long-range magnetic order nor spin-glass behavior down to 1.9 K in ${\mathrm{Ba}}_3{R\mathrm{B}}_9{\mathrm{O}}_{18}$. Magnetization data reveal pseudospin $J_{\mathrm{eff}}=1/2$ $\left({\mathrm{Yb}}^{3+}\right)$ degrees of freedom in the Kramers doublet state and a weak antiferromagnetic interaction between $J_{\mathrm{eff}}=1/2$ moments in the Yb variant. On the other hand, the effective moment ${\mu }_{\mathrm{eff}}=8.8{\mu }_B$ was obtained from the Curie-Weiss fit of the low-temperature susceptibility data in ${\mathrm{Ba}}_3{\mathrm{ErB}}_9{\mathrm{O}}_{18}$, which suggests the admixture of higher-crystal-electric-field states with the ground state. The Curie-Weiss fit of low-temperature susceptibility data for the Er system unveils the presence of a bit stronger antiferromagnetic interaction between ${\mathrm{Er}}^{3+}$ moments compared with its ${\mathrm{Yb}}^{3+}$ analog. ${\mathrm{Ba}}_3{\mathrm{ErB}}_9{\mathrm{O}}_{18}$ does not show long-range magnetic order down to 500 mK. Furthermore, our crystal electric field calculations based on the thermodynamic data suggest the presence of a small gap between the ground and first excited Kramers doublets. The broad maximum around 4 K in the specific heat at zero field is attributed to the thermal population of the first crystal electric field excited state in ${\mathrm{Ba}}_3{\mathrm{ErB}}_9{\mathrm{O}}_{18}$.

• Received 18 October 2021
• Revised 9 June 2022
• Accepted 22 August 2022

DOI:https://doi.org/10.1103/PhysRevB.106.104408