Absence of Jahn−Teller transition in the hexagonalBa3CuSb2O9 single crystalNaoyuki Katayamaa,1, Kenta Kimurab,2, Yibo Hanc,d, Joji Nasue,3, Natalia Drichkof, Yoshiki Nakanishig, Mario Halimb,Yuki Ishiguroh, Ryuta Satakea, Eiji Nishiboria,4, Masahito Yoshizawag, Takehito Nakanoi, Yasuo Nozuei,Yusuke Wakabayashih, Sumio Ishiharae, Masayuki Hagiwarac, Hiroshi Sawaa, and Satoru Nakatsujib,j

aDepartment of Applied Physics, Nagoya University, Nagoya 464-8603, Japan; bInstitute for Solid State Physics, University of Tokyo, Kashiwa 277-8581,Japan; cCenter for Advanced High Magnetic Field Science, Graduate School of Science, Osaka University, Osaka 560-8531, Japan; dWuhan National HighMagnetic Field Center, Huzhong University of Science and Technology, Wuhan 430074, China; eDepartment of Physics, Tohoku University, Sendai 980-8578,Japan; fInstitute for Quantum Matter (IQM) and Department of Physics and Astronomy, Johns Hopkins University, Baltimore, MD 21218; gGraduate Schoolof Engineering, Iwate University, Morioka 020-8551, Japan; hDivision of Materials Physics, Graduate School of Engineering Science, Osaka University,Toyonaka 560-8531, Japan; iDepartment of Physics, Graduate School of Science, Osaka University, Toyonaka, Osaka 560-0043, Japan; and jPRESTO(Precursory Research for Embryonic Science and Technology), Japan Science and Technology Agency (JST), 4-1-8 Honcho Kawaguchi, Saitama332-0012, Japan

Edited by Zachary Fisk, University of California, Irvine, CA, and approved June 19, 2015 (received for review May 11, 2015)

With decreasing temperature, liquids generally freeze into a solidstate, losing entropy in the process. However, exceptions to this trendexist, such as quantum liquids, which may remain unfrozen down toabsolute zero owing to strong quantum entanglement effects thatstabilize a disordered state with zero entropy. Examples of suchliquids include Bose−Einstein condensation of cold atoms, supercon-ductivity, quantum Hall state of electron systems, and quantum spinliquid state in the frustrated magnets. Moreover, recent studies haveclarified the possibility of another exotic quantum liquid state basedon the spin–orbital entanglement in FeSc2S4. To confirm this exoticground state, experiments based on single-crystalline samples areessential. However, no such single-crystal study has been reportedto date. Here, we report, to our knowledge, the first single-crystalstudy on the spin–orbital liquid candidate, 6H-Ba3CuSb2O9, and wehave confirmed the absence of an orbital frozen state. In stronglycorrelated electron systems, orbital ordering usually appears at hightemperatures in a process accompanied by a lattice deformation,called a static Jahn−Teller distortion. By combining synchrotronX-ray diffraction, electron spin resonance, Raman spectroscopy, andultrasound measurements, we find that the static Jahn−Teller distor-tion is absent in the present material, which indicates that orbitalordering is suppressed down to the lowest temperatures measured.We discuss how such an unusual feature is realized with the help ofspin degree of freedom, leading to a spin–orbital entangled quantumliquid state.

spin-orbital entanglement | quantum liquid state |synchrotron X-ray diffraction | Jahn–Teller transition | Ba3CuSb2O9

Quantum spin liquids have been widely recognized as a new stateof matter, as an increasing number of candidates with quantum

spin S = 1/2 have been found recently (1–4), a long time after thefirst proposal was made for the resonating valence-bond state (5).On the other hand, quantum liquids based on another electronicdegree of freedom, orbital, have been theoretically proposed (6).However, this type of liquid state has never been experimentallyconfirmed because the energy of orbital correlation is normally oneorder of magnitude stronger than spin exchange coupling, leading toan orbital ordering at a significantly high temperature accompaniedby a cooperative Jahn–Teller (JT) distortion. Nevertheless, if we canbring down the orbital energy to the same scale as for the spincoupling, it may lead to a novel spin–orbital entangled state, a“quantum spin–orbital liquid.” A possible spin–orbital entangledliquid state with dimer correlations has been theoretically discussedon a triangular lattice with singly occupied but triply degeneratet2g orbitals (7). In comparison with the t2g orbitals’ case, theexperimental realization of such a quantum spin–orbital liquidstate in the eg orbital system has been even more challenging(8), because eg orbitals more strongly couple to the JT modes.

Perovskite-type 6H-Ba3CuSb2O9 is a good candidate material forthe spin–orbital liquid state that has been theoretically proposed(9–11). Recently, we reported that spin–orbital short-range orderingoccurs in the short-range honeycomb lattice of Cu2+ with eg orbitaldegrees of freedom, as depicted in Fig. 1A (12), sharply contradictingthe previously reported crystal structure with a triangular lattice ofCu2+ (13). In addition to the confirmation of a dynamic spinstate down to 20 mK by muon spin spectroscopy (12, 14, 15),powder X-ray diffraction clearly indicates that even at lowtemperature, the hexagonal components remain, along withsome orthorhombically distorted components. In the hexago-nal phase, threefold symmetry exists for the Cu2+ sites, whichare surrounded by octahedrally coordinated oxygen, in-dicating the absence of a cooperative JT distortion. To ex-plain this unusual feature, we proposed two possible scenarios.

Significance

The quantum spin liquid state has been intensively pursued sinceAnderson proposed the resonating valence bond model. On theother hand, quantum liquids based on another electronic degree offreedom, orbital, has been believed unrealistic, because the energyscale of orbital correlation is normally one order of magnitudehigher than spin exchange coupling, resulting in an orbital orderingat a signicantly high temperature accompanied by a cooperativeJahn−Teller (JT) distortion. In this paper, we present striking com-plete suppression of the JT transition in the copper oxide, 6H-Ba3CuSb2O9 based on comprehensive structural studies, indicating therealization of the novel “spin–orbital liquid” state.

Author contributions: N.K., H.S., and S.N. designed research; N.K., K.K., Y.H., N.D.,Y. Nakanishi, M. Halim, Y.I., R.S., E.N., M.Y., T.N., Y. Nozue, Y.W., M. Hagiwara, andS.N. performed experiments; N.K., J.N., Y. Nakanishi, Y.W., S.I., M. Hagiwara, and S.N.analyzed data; and N.K., Y.W., S.I., M. Hagiwara, and S.N. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

Data deposition: The crystallography, atomic coordinates, and structure factors have beendeposited in the Cambridge Structural Database, Cambridge Crystallographic Data Cen-tre, Cambridge CB2 1EZ, United Kingdom (CSD reference nos. 943653 and 943654).1To whom correspondence should be addressed. Email: katayama@mcr.nuap.nagoya-u.ac.jp.2Present address: Division of Materials Physics, Graduate School of Engineering Science,Osaka University, Toyonaka 560-8531, Japan.

3Present address: Department of Physics, Tokyo Institute of Technology, Ookayama,Meguro, Tokyo 152-8551, Japan.

4Present address: Division of Physics, Faculty of Pure and Applied Science, Tsukuba Re-search Center for Interdiscriplinary Materials Science (TIMS) & Center for Integrated Re-search in Fundamental Science and Engineering (CiRfSE), University of Tsukuba, 1-1-1Tennodai, Tsukuba, Ibaraki 305-8571, Japan.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1508941112/-/DCSupplemental.

www.pnas.org/cgi/doi/10.1073/pnas.1508941112 PNAS | July 28, 2015 | vol. 112 | no. 30 | 9305–9309

PHYS

ICS

(i) A noncooperative static JT distortion appears. In this sce-nario, the local symmetry is lowered by a static JT distortion, asschematically shown in Fig.1B, but the overall hexagonal sym-metry remains. (ii) The static JT distortion is absent and, instead,a dynamic JT distortion appears, leading to a novel spin–orbitalliquid state, as depicted in Fig. 1C. These two possible scenarioscannot be distinguished by experimental results using powderspecimens alone, as was reported in the previous paper (12);a thorough structural study is required using a single crystalwithout orthorhombic components. Here, we report the com-prehensive study on a hexagonal single crystal of Ba3CuSb2O9that exhibits no cooperative JT transition down to low temperatures.This provides, to our knowledge, the first example of a copper3 d9 compound with no JT transition. Our results suggest a for-mation of a spin–orbital liquid state.

ResultsComprehensive Structural Studies Using Single-Crystalline Samples.First, let us present the results for the single-crystal X-ray dif-fraction measurements for the sample that retains the hexagonalsymmetry down to the lowest temperature. Fig. 2 A and B showsprofiles obtained at 300 K and 20 K, respectively. The peaksshow no signs of splitting or broadening down to 20 K. Herein-after, we call such a sample a hexagonal sample. The hexagonalsample can be refined well by using the centrosymmetric spacegroup P63=mmc for all temperatures. For P63=mmc, the three-fold symmetry is retained for Cu2+ sites, clearly indicating theabsence of the cooperative JT distortion.These observations are in sharp contrast with our previous

single-crystal X-ray diffraction study of 6H-Ba3CuSb2O9 (12).There, we reported that the Bragg peak splits into several sep-arate reflections upon decreasing temperature, as shown in Fig. 2C and D. This result indicates that the hexagonal P63=mmcsymmetry is lowered to the orthorhombic Cmcm symmetry. Weattribute this effect to a cooperative JT distortion induced by

uniform orbital ordering of Cu2+ ions (Fig. 1D). Hereinafter, wecall such a sample an “orthorhombic sample”. Although thecrystal structures are similar between hexagonal and ortho-rhombic samples with the same space group of P63=mmc athigh temperatures, the z coordinates of the crystallographicallyequivalent Cu–Sb sites slightly differ between them. The de-tails will be discussed in Structural Differences Between the Hexag-onal and Orthorhombic Samples. Note that for some hexagonalcomponents, ∼1–10% of the volume fraction remains in ortho-rhombic samples, even at the lowest temperature, as previouslyreported (12).The structural features that differentiate between the hexagonal

and orthorhombic samples can be also identified by powderX-ray diffraction. We prepared powder samples by crushing thecrystals used in the single-crystal X-ray diffraction experiments.Below 200 K, no signs of splitting occur for the hexagonal sampledown to 80 K, but the peaks split for the orthorhombic sample,as shown in Fig. 3 A and B, respectively. Using a cryostat, weconfirmed that the hexagonal sample retains hexagonal symme-try down to 13 K (Supporting Information). To further reveal theunusual state of the hexagonal phase, the local structure asso-ciated with the JT distortion should be studied.Electron spin resonance (ESR) is known to sensitively detect

the local orbital configuration through the anisotropy of the gfactor (16). For the magnetic field parallel to the ½2110� ([100])direction of the hexagonal (orthorhombic) sample, we find dis-tinct signals of ESR. For the orthorhombic crystal, the curvessplit below 200 K (Fig. 3D), corresponding to the hexagonal–orthorhombic structural phase transition found by the X-raymeasurement. In contrast, down to 3.5 K, the hexagonal sampleproduces the field derivative of a clear single Lorentzian signal(Fig. 3C). Fig. 3 E and F shows the angular dependence of the gfactors in the c plane obtained by fitting the ESR curves with thefield derivatives of single- or multiple-peak Lorentzian functions.Below 30 K, the orthorhombic sample produces three clear pe-riodic components (Fig. 3F), which originate from the three

Fig. 1. (A) Schematic view of the local structure for hexagonal and ortho-rhombic samples. (B) Schematic picture of a noncooperative static JT distortion.(C and D) Schematic pictures of spin-singlet formation in short-range honeycomblattices of Cu2+ for (C) hexagonal and (D) orthorhombic samples. For C, a spin–orbital entangled short-range-order state is expected. A pair of up and downarrows indicates a singlet state of the dimer based on the neighboring Cu2+

spins. At each site, an unpaired electron of Cu2+ occupies the dx2−y2 , dy2−z2 , ordz2−x2 orbital.

Fig. 2. Single-crystal X-ray diffraction profiles for (A and B) hexagonal and(C and D) orthorhombic samples. Insets in A and C are photographs of thetransparent brown single crystals for the hexagonal and orthorhombicsamples, respectively. The hexagonal samples are darker than the ortho-rhombic samples.

9306 | www.pnas.org/cgi/doi/10.1073/pnas.1508941112 Katayama et al.

types of domains corresponding to the three types of static JTdistortion naturally expected for a CuO6 octahedron. In contrast,for the hexagonal phases at all of the temperatures in the hex-agonal sample as well as above 200 K in the orthorhombicsample, the g factors are almost isotropic within the in-planedirections, as expected for a sample with hexagonal symmetry(Fig. 3 E and F). This result indicates that the hexagonal sampleretains its hexagonal symmetry at temperatures down to 3.5 K,which is low enough in comparison with both exchange couplingof ∼50 K and orbital correlation energy scale of ∼200 K, andthus confirms that the JT instability to stabilize an orthorhombicdistortion is fully suppressed in the present hexagonal samples.Raman spectroscopy and ultrasound measurements provide

other sensitive probes for the local structural symmetry and wereused to examine the temperature variation of the JT distortion(17–21). The overall Raman spectra of Ba3CuSb2O9 are definedby high disorder and broken local symmetry in the edge-sharingoctahedra, leading to a split in the shared-face oxygen modesaround 550 cm−1, as shown in Fig. 4 A and B. The spectra of the

orthorhombic sample display new bands at temperatures belowthe phase transition; for example, the polarized xx Ramanspectra shown in Fig. 4B exhibit a new band at 587 cm−1 uponcooling, indicating a lowered symmetry. In contrast, the spectraof the hexagonal sample, as shown in Fig. 4A, do not changeupon lowering the temperature, indicating the symmetry ismaintained. Fig. 4 C and D show the transverse C44 elasticconstant data collected using ultrasound measurement as afunction of temperature. Ultrasound measurement of the or-thorhombic sample clearly reveals a characteristic elastic soft-ening in C44 toward the phase transition, relevant to the orbitalsof dx2−y2, dy2−z2, and dz2−x2 (Fig. 4D). The solid line in Fig. 4Dis the theoretical elastic constant based on the followingCurie–Weiss-like formula for the cooperative JT effect: CΓ (T) =Cð0ÞΓ (T)(T - Tc)/(T - Θ). Here, Tc is the transition temperature

and Cð0ÞΓ is a background elastic constant mainly due to the

anharmonicity involved in the lattice vibrations. This fitting allowsus to estimate the positive paramagnetic Curie–Weiss constantof Θ ≈ 250 K, indicating a ferro-orbital interaction. This is fullyconsistent with the ferro-type orbital arrangements in ortho-rhombic samples, as shown in Fig. 1D. Conversely, a smoothcurve without any characteristic softening is observed down tothe lowest temperatures in the hexagonal sample, as shown inFig. 4C, again indicating the absence of symmetry lowering.Our ESR, Raman spectra, and ultrasound measurements are

fully consistent with those expected for hexagonal symmetry.Although the Raman measurement time scale (picoseconds) iscomparable or shorter than a typical dynamic JT time scale

Fig. 3. (A and B) Powder X-ray diffraction patterns of the (A) hexagonaland (B) orthorhombic samples. (C and D) Temperature-dependent ESR curvesalong ½2110� and ½100� of the (C) hexagonal and (D) orthorhombic samples.(E and F) Angular-dependent g factor in the c plane for (E) hexagonal and(F) orthorhombic samples. Inset in E indicates the Miller indices for thehexagonal (regular hexagon) and orthorhombic samples (distorted hexa-gon). The rotation angle of the magnetic field relative to the ½2110� or ½100�axis is given by φ.

Fig. 4. (A and B) Polarized xx Raman spectra of the (A) hexagonal and(B) orthorhombic samples in the range of oxygen stretching vibrations.Vertical dotted lines are at 587 cm−1. Spectra are shifted vertically for clarity.Inset in A shows polarized xx Raman spectra in the hexagonal phases of theorthorhombic and hexagonal samples at 240 K. Note the difference in fre-quencies of the Oinner stretching vibrations. Inset in B shows the local crystalstructure around a JT-active Cu2+ ion. (C and D) Elastic constants of thetransverse C44 for the (C) hexagonal and (D) orthorhombic samples. Theblack solid line indicates the theoretical fit with the Curie–Weiss-like formulafor the cooperative JT effect.

Katayama et al. PNAS | July 28, 2015 | vol. 112 | no. 30 | 9307

PHYS

ICS

(22, 23), the above results of the much slower probes, namelyESR (nanoseconds) and ultrasound measurements (microseconds),clearly exclude the noncooperative static JT distortion scenario ipresented above. Noncooperative static JT distortion must breakthreefold symmetry in each Cu2+ site and generate anisotropic gfactors within the in-plane directions in ESR experiments, simi-larly to the low-temperature phase of the orthorhombic sample,thus clearly inconsistent with our observations for the hexagonalsample. Significantly, orbital degree of freedom exists at hightemperatures in the present samples because the orthorhombicsample undergoes a cooperative JT transition at around 200 K,and, furthermore, a dynamic orbital state has been alreadyconfirmed at 290 K in a hexagonal sample using an inelasticX-ray scattering experiment (24). Thus, our study using thesingle crystals without any orthorhombic components has con-firmed that the hexagonal sample shows no signs of symmetrylowering on cooling and further demonstrates the absence of thenoncooperative static JT distortion in all temperature regions.

Structural Differences Between the Hexagonal and OrthorhombicSamples. To clarify what differentiates the hexagonal samples fromthe orthorhombic samples, we investigated the chemical and struc-tural differences between them. Our comprehensive studies using in-ductively coupled plasma atomic emission spectroscopy (ICP-AES)and X-ray diffraction experiments have revealed that the hex-agonal samples can be distinguished from the orthorhombicsamples by both Sb/Cu ratio and the high temperature a-axis pa-rameters, as shown in Fig. 5. Although their crystal structures aresimilar to the same space group of P63=mmc at high temperatures,they can be differentiated based on the Sb/Cu ratio, as the previouspaper suggested (12). As shown in Fig. 5B, the hexagonal samplesare located in the vicinity of the border of the orthorhombic phase,and the JT instability becomes enhanced with an increase in thea-axis parameter at 400 K and when the Sb/Cu composition ratio isslightly deviated from 2/1. When the Sb/Cu ratio is higher than 2/1,

some Sb ions should replace the Cu ions, and should break locallythe threefold symmetry of the short-ranged honeycomb lattice. Thissymmetry breaking inherent to the off-stoichiometry should inducethe orthorhombic distortion at low temperatures and thus lead to thelocalized valence bond solid state accompanied by the distortion. Incomparison with the orthorhombic samples, strong geometricalfrustration should exist in the hexagonal samples owing to the higherlocal symmetry, and should stabilize a quantum spin–orbital liquidstate. Although the Sb/Cu ratio was found to be always higher than2/1 in the orthorhombic samples as shown in Fig. 5A, the ortho-rhombic distortion would appear even if the Sb/Cu ratio becamesmaller than 2/1. The detailed theoretical considerations are given inSupporting Information.For further studying the local structure around JT-active Cu

ions, we refined the coordinates of crystallographically equiva-lent Cu–Sb sites independently using single-crystal X-ray dif-fraction. For the refinement, the Cu–Sb composition ratio isfixed to 1.0 for both the hexagonal and orthorhombic samples.Although we detected by chemical analysis (Supporting In-formation) a slight departure from unity in the Cu–Sb composi-tion ratio for the orthorhombic sample, this does not significantlyaffect the parameters obtained (Supporting Information). Usingthe bond valence sum technique, we estimated the valences ofseparated Cu and Sb to be close to 2+ and 5+, respectively, in-dicating the parameters obtained are reasonable. For the or-thorhombic sample, the z coordinates of the face-sharingoctahedral sites of Cu and Sb are very similar; however, for thehexagonal sample, the z coordinates of these sites differ. Asdepicted in Fig. 1A, the change in z coordinates can be inter-preted as the Cu–Sb dumbbell moving toward the Cu and slightlyelongating. Note that even with such a shift, the Cu–Sb dumbbellremains along the threefold axis for both structures, indicatingthe absence of a static JT distortion. The displacement of theCu–Sb dumbbell leads to longer Cu–Oinner bonds and shorterCu–Oouter bonds (Fig. 1A). In the high-temperature hexagonalphases, the different structure of face-sharing CuSbO9 octahedrafor the hexagonal and orthorhombic samples is also detectedby Raman spectroscopy (Fig. 4A, Inset). The vibrations of theshared-face oxygens shift clearly from 552 cm−1 and 576 cm−1 forthe orthorhombic sample to 535 cm−1 and 552 cm−1 for the hexa-gonal sample.By using the local structural parameters presented above, we

estimate the change in superexchange interactions between neigh-boring Cu ions for the hexagonal and orthorhombic samples. Theperturbational exchange processes are classified into two forms: twoholes located on a single Cu ion or on a single O ion in intermediatestates. The variations in the exchange constants that arise becauseof the change in the Cu–O–O angle are almost canceled out in thetwo processes. On the other hand, for both processes, decreasingthe Cu–O bond length increases the exchange constant. For the J1exchange interaction, depicted in Fig. 1A, Cu–Oouter bond lengthsare shorter in the hexagonal sample compared with the ortho-rhombic sample, which increases the transfer integral and therebyenhances the J1 exchange interactions. Note that the J1’ exchangeinteractions through Cu–Oinner–Oinner–Cu paths may be ignoredbecause they are much less than those through Cu–Oouter–Oouter–Cupaths owing to the long bond lengths.From these estimations of the superexchange interactions, we

hypothesize that in the hexagonal sample, the increase in ex-change interactions is related to the unusual suppression of astatic JT distortion. The orbital energy, i.e., the interaction en-ergy that lifts the orbital degeneracy and induces low-tempera-ture orbital ordered states, is typically much higher than the spinenergy due to the cooperative JT effect, resulting in a weakcorrelation between spin and orbital. However, the orbital en-ergy is reduced due to the spatially separated CuO6 octahedra. Ifthe magnetic exchange interactions are enhanced to the pointthat they are comparable with the orbital energy, the interplay

Fig. 5. (A) The Sb/Cu ratio vs. the lattice parameters at 400 K for severalsamples. The dotted line is a guide for the eye. The region within the av-eraged errors for the stoichiometric Sb/Cu = 2/1 is shown in gray. (B) Sampledependence on the composition ratio between hexagonal and orthorhom-bic phases by decreasing temperatures studied using several samples. Thevertical axis is the a-axis lattice constant obtained at 400 K in the high-temperature hexagonal phase, by which samples are distinguished. Eachcircle represents a temperature point for a powder X-ray measurement, andits color indicates the volume fraction of the hexagonal (white) and ortho-rhombic (black) phases.

9308 | www.pnas.org/cgi/doi/10.1073/pnas.1508941112 Katayama et al.

between spin and orbital would destabilize the conventionalorbital-ordered state, leading to a novel spin–orbital entangledstate. Indeed, such spin–orbital correlation has been alreadypointed out by using Huang scattering (24). According to thepresent structural study, the possible ground states have beenalready proposed theoretically, including a short-range resonat-ing-valence bond state (25, 26) and an emergent dimer stateinvolving nearest-neighbor spin singlets with orbital order (27).In particular, the latter scenario is inconsistent with the results ofour structural analysis because the valence bond solid stateshould be accompanied by a local structural distortion, whichis absent in the hexagonal sample. We hope that our successin growing the hexagonal single-crystalline samples without or-thorhombic components will stimulate further experimental in-vestigations toward the complete understanding of the novelquantum liquid state.To summarize, our experiments thus clarify the absence of a

static JT distortion and, instead the presence of a dynamical JTdistortion at least down to 3.5 K in the hexagonal samples. Theorthorhombic distortion appears when the Sb/Cu compositionratio slightly deviates from 2/1, probably because the presentsystem is quite sensitive to the symmetry breaking defects in-troduced by the off-stoichiometry. Normally, the lack of single-crystalline samples hinders detailed experimental investigationsto deepen our understanding of the exotic ground state, as in thecase of FeSc2S4 (28, 29). In this study, however, we have suc-cessfully obtained the single-crystalline samples that retain thehexagonal symmetry down to the lowest temperatures. Furtherstudies using the crystals will address the open questions on thequantum spin–orbital liquid state, such as the orbital dynamics,the resonant motions of singlet dimers, and the mechanism tostabilize such an exotic liquid state.

MethodsSingle crystals of Ba3CuSb2O9 were grown under oxygen atmosphere fromthe BaCl2-based flux at the Institute for Solid State Physics (ISSP), Universityof Tokyo. A mixture of polycrystalline sample and flux with a molar ratio of

1:10 was put inside a Pt crucible and heated up to 1,100 °C and cooled downto 800 °C with a cooling rate of 2 °C/h. The sample was then washed with hotwater to remove the remaining flux. Two types of single crystals wereobtained, depending on the growth condition. Namely, a small addition(9 mol %) of Ba(OH)2 to the BaCl2 flux was found to stabilize single-phasecrystals of the hexagonal samples, whereas the pure BaCl2 flux leads to thegrowth of single-phase crystals of orthorhombic samples. The combinationof both composition analyses by ICP-AES and X-ray structural analyses in-dicates that hexagonal and orthorhombic samples are both single phases and,respectively, stoichiometric and off-stoichiometric in terms of the Cu to Sbelemental ratio, as discussed. Single-crystal and powder X-ray diffraction ex-periments were performed at SPring-8 BL02B1 and BL02B2, respectively. Forsingle-crystal X-ray diffraction, a typical size of 40 × 40 × 20 μm3 were mea-sured with a wavelength of 0.35 Å. An X-band ESR apparatus (Bruker EMX EPRSpectrometer) was used for precise measurements of the temperature andangular dependencies of ESR spectra at about 9.3 GHz. Raman measurementswere done with a Jobin-Yvone T64000 triple monochromator spectrometerwith an Olympus microscope. The excitation line was the 514.5-nm line of aSpectra-Physics Ar-Kr laser. The probe was 2 μm in diameter. For low-tem-perature measurements, He-flow Janis Microcryostat was used. Velocity mea-surements of the ultrasonic transverse and longitudinal wave propagatingalong the c-crystallographic axis were performed with a frequency of 15 MHzand 30 MHz, respectively.

ACKNOWLEDGMENTS. The authors acknowledge Prof. C. Broholm for valu-able discussion. This work was supported by a Grant-in-Aid for Scientific Re-search (23244074, 242440590, 25707030, 23102702, 26287070, and 24540354),Program for Advancing Strategic International Networks to Accelerate theCirculation of Talented Researchers (R2604) from the Japanese Society for thePromotion of Science, PRESTO of JST (Japan Science and Technology Agency),the Global COE (Center of Excellence) Program (Core Research and Engineer-ing of Advanced Materials and Interdisciplinary Education Center for Mate-rials Science) (Grant G10) from the MEXT (Ministry of Education, Culture,Sports, Science and Technology), Core Research for Evolutional Science andTechnology of JST, and Department of Energy Grant for The Institute ofQuantumMatter DE-FG02-08ER46544. The synchrotron radiation experimentswere performed at SPring-8 with the approval of the Japan Synchrotron Ra-diation Research Institute (Proposals 2011B0083/BL02B1 and 2011B0084/BL02B2). The use of the Materials Design and Characterization Laboratory atthe Institute for Solid State Physics (ISSP) is gratefully acknowledged. Some ofthe numerical calculations were performed using the supercomputing facilitiesat ISSP.

1. Helton JS, et al. (2007) Spin dynamics of the Spin-1/2 kagome lattice antiferromagnetZnCu3(OH)6Cl2. Phys Rev Lett 98(10):107204.

2. Yamashita M, et al. (2010) Highly mobile gapless excitations in a two-dimensionalcandidate quantum spin liquid. Science 328(5983):1246–1248.

3. Shimizu Y, Miyagawa K, Kanoda K, Maesato M, Saito G (2003) Spin liquid state in anorganic Mott insulator with a triangular lattice. Phys Rev Lett 91(10):107001.

4. Okamoto Y, Nohara M, Aruga-Katori H, Takagi H (2007) Spin-liquid state in the S = 1/2hyperkagome antiferromagnet Na4Ir3O8. Phys Rev Lett 99(13):137207.

5. Anderson PW (1973) Resonating valence bonds: A new kind of insulator? Mater ResBull 8(2):153–160.

6. Feiner LF, Ole�s AM (2005) Orbital liquid in ferromagnetic manganites: The orbitalHubbard model for eg electrons. Phys Rev B 71(14):144422.

7. Normand B, Ole�s AM (2008) Frustration and entanglement in the t2g spin-orbitalmodel on a triangular lattice: Valence-bond and generalized liquid states. Phys Rev B78(9):094427.

8. Feiner LF, Ole�s AM, Zaanen J (1997) Quantum melting of magnetic order due to or-bital fluctuations. Phys Rev Lett 78(14):2799.

9. Mila F, et al. (2007) The emergence of resonating valence bond physics in spin orbitalmodels. J. Phys. Cond. Mat. 19(14):145201.

10. Moessner R, Sondhi SL, Chandra P (2001) Phase diagram of the hexagonal latticequantum dimer model. Phys Rev B 64(14):144416.

11. Corboz P, Lajkó M, Läuchli AM, Penc K, Mila F (2012) Spin-orbital quantum liquid onthe honeycomb lattice. Phys Rev X 2(4):041013.

12. Nakatsuji S, et al. (2012) Spin-orbital short-range order on a honeycomb-based lattice.Science 336(6081):559–563.

13. Köhl VP (1978) Crystal structure of the hexagonal compounds BaII3MeII SbV2O9. II.

Ba3 CuSb2O9. Z Anorg Allg Chem 442:280–288.14. Zhou HD, et al. (2011) Spin liquid state in the S = 1/2 triangular lattice Ba3CuSb2O9.

Phys Rev Lett 106(14):147204.15. Quilliam JA, et al. (2012) Singlet ground state of the quantum antiferromagnet

Ba3CuSb2O9. Phys Rev Lett 109(11):117203.16. Abragam A, Bleaney B (1970) Electron Paramagnetic Resonance of Transition Ions

(Clarendon Press, Oxford).

17. Kaplan MD, Vekhter BG (1995) Cooperative Phenomena in Jahn-Teller Crystals (Ple-num Press, New York).

18. Lüthi B (2007) Physical Acoustics in the Solid State (Springer, New York).19. Shuker R, Gammon RW (1970) Raman-scattering selection-rule breaking and the

density of states in amorphous materials. Phys Rev Lett 25(4):222–225.20. Iliev MN, Abrashev MV (2001) Raman phonons and Raman Jahn-Teller bands in

perovskite-like manganites. J Raman Spectrosc 32(10):805–811.21. Granado E, Sanjurjo JA, Rettori C, Neumeier JJ, Oseroff SB (2000) Order-disorder in

the Jahn-Teller transition of LaMnO3: A Raman scattering study. Phys Rev B 62(17):11304–11307.

22. Curtiss L, Halley JW, Wang XR (1992) Jahn-Teller effect in liquids: General principlesand a molecular-dynamics simulation of the cupric ion in water. Phys Rev Lett 69(16):2435–2438.

23. Kamaràs K, et al. (2013) Mott localization in the correlated superconductor Cs3C60

resulting from the molecular Jahn-Teller effect. J Phys Conf Ser 428:012002.24. Ishiguro Y, et al. (2013) Dynamical spin–orbital correlation in the frustrated magnet

Ba3CuSb2O9. Nat Commun 4:2022.25. Nasu J, Ishihara S (2013) Dynamical Jahn-Teller effect in a spin-orbital coupled system.

Phys Rev B 88(9):094408.26. Nasu J, Ishihara S (2015) Resonating valence-bond state in an orbitally degenerate

quantum magnet with dynamical Jahn-Teller effect. Phys Rev B 91(4):045117.27. Smerald A, Mila F (2014) Exploring the spin-orbital ground state of Ba3CuSb2O9. Phys

Rev B 90(9):094422.28. Chen G, Balents L, Schnyder AP (2009) Spin-orbital singlet and quantum critical point

on the diamond lattice: FeSc2S4. Phys Rev Lett 102(9):096406.29. Mittelstädt L, et al. (2015) Spin-orbiton and quantum criticality in FeSc2S4. Phys Rev B

91(12):125112.30. Vernay F, Penc K, Fazekas P, Mila F (2004) Orbital degeneracy as a source of frus-

tration in LiNiO2. Phys Rev B 70(1):014428.31. Thompson L, Lee P (2012) Decoupled spin chains in Ba3CuSb2O9. arXiv:1202.5655.32. Nasu J, Nagano A, Naka M, Ishihara S (2008) Doubly degenerate orbital system in

honeycomb lattice: Implication of orbital state in layered iron oxide. Phys Rev B 78(2):024416.

Katayama et al. PNAS | July 28, 2015 | vol. 112 | no. 30 | 9309

PHYS

ICS