Abstract
We present a large-scale density matrix renormalization group (DMRG) study of the spin-1 SU(3) bilinear-biquadratic model on the square lattice, which was suggested to host a nematic spin liquid state in recent DMRG calculations. We report that this spin liquid appears to strongly compete with a three-sublattice magnetic order. To further study the competition between the two states, and the reason of the emergent nematic spin liquid, we included an additional next-nearest-neighbor SU(3) symmetric interactions along one of the two plaquette diagonal directions. This allows to tune the square-lattice model to the triangular-lattice model. By computing spin correlation functions and various order parameters, we find that the three-sublattice order may develop at infinitesimal additional new coupling, at least within the precision of our study. Compared with the previous findings that the nematic spin liquid is stable in extended parameter regions with additional couplings that respect the lattice symmetries of the square lattice, we argue that here the diagonal couplings, which frustrate the bipartite-lattice structure, rapidly suppress the two-sublattice fluctuations and the three-sublattice order thus wins. This numerical result is consistent with the conjecture that the nematic spin liquid emerges from the competition between two- and three-sublattice fluctuations.
