HLS Logo

Humanity's Last Exam

Hugging FaceDatasetload_dataset("cais/hle")

Latest News

  • HLE Logo[04/03/2025]: HLE has been finalized with 2,500 questions.
  • HLE Logo[03/21/2025]: Bug Bounty closed. Thank you all for participating, we're working on finalizing bounty decisions!
  • HLE Logo[03/15/2025]: Late contributions closed. Thank you all for participating, we're working on reviewing!
  • HLE Logo[02/11/2025]: HLE Community Feedback Expansion - Bug Bounty. Finalized Original Authors and we are working on later contributors for co-authorship soon
CAIS Logo&Scale AI Logo

Long Phan*1, Alice Gatti*1, Ziwen Han*2, Nathaniel Li*1

Josephina Hu2, Hugh Zhang2, Chen Bo Calvin Zhang2, Mohamed Shaaban2

John Ling2, Sean Shi2, Michael Choi2, Anish Agrawal2, Arnav Chopra2, Ryan Kim1

Adam Khoja1, Richard Ren1, Jason Hausenloy1, Oliver Zhang1, Mantas Mazeika1

Summer Yue**2, Alexandr Wang**2, Dan Hendrycks**1

1Center for AI Safety, 2Scale AI

Dataset Contributors

Dmitry DodonovTung NguyenJaeho LeeDaron AndersonMikhail DoroshenkoAlun Cennyth StokesMobeen MahmoodOleksandr PokutnyiOleg IskraJessica P. WangJohn-Clark LevinMstyslav KazakovFiona FengSteven Y. FengHaoran ZhaoMichael YuVarun GangalChelsea ZouZihan WangSerguei PopovRobert GerbiczGeoff GalgonJohannes SchmittWill YeadonYongki LeeScott SauersAlvaro SanchezFabian GiskaMarc RothSøren RiisSaiteja UtpalaNoah BurnsGashaw M. GoshuMohinder Maheshbhai NaiyaChidozie AguZachary GiboneyAntrell CheatomFrancesco Fournier-FacioSarah-Jane CrowsonLennart FinkeZerui ChengJennifer ZampeseRyan G. HoerrMark NandorHyunwoo ParkTim GehrungerJiaqi CaiBen McCartyAlexis C GarretsonEdwin TaylorDamien SileoQiuyu RenUsman QaziLianghui LiJungbae NamJohn B. WydallisPavel ArkhipovJack Wei Lun ShiAras BachoChris G. WillcocksHangrui CaoSumeet MotwaniEmily de Oliveira SantosJohannes VeithEdward VendrowDoru CojocKengo ZenitaniJoshua RobinsonLongke TangYuqi LiJoshua VendrowNatanael Wildner FragaVladyslav KuchkinAndrey Pupasov MaksimovPierre MarionDenis EfremovJayson LynchKaiqu LiangAleksandar MikovAndrew GritsevskiyJulien GuillodGözdenur DemirDakotah MartinezBen PagelerKevin ZhouSaeed SooriOri PressHenry TangPaolo RissoneSean R. GreenLina BrüsselMoon TwayanaAymeric DieuleveutJoseph Marvin ImperialAmeya PrabhuJinzhou YangNick CrispinoArun RaoDimitri ZvonkineGabriel LoiseauMikhail KalininMarco LukasCiprian ManolescuNate StambaughSubrata MishraTad HoggCarlo BosioBrian P CoppolaJulian SalazarJaehyeok JinRafael SayousStefan IvanovPhilippe SchwallerShaipranesh SenthilkumaAndres M BranAndres AlgabaKelsey Van den HouteLynn Van Der SyptBrecht VerbekenDavid NoeverAlexei KopylovBenjamin MyklebustBikun LiLisa SchutEvgenii ZheltonozhskiiQiaochu YuanDerek LimRichard StanleyTong YangJohn MaarJulian WykowskiMartí OllerAnmol SahuCesare Giulio ArditoYuzheng HuAriel Ghislain Kemogne KamdoumAlvin JinTobias Garcia VilchisYuexuan ZuMartin LacknerJames KoppelGongbo SunDaniil S. AntonenkoSteffi ChernBingchen ZhaoPierrot ArseneJoseph M CavanaghDaofeng LiJiawei ShenDonato CrisostomiWenjin ZhangAli DehghanSergey IvanovDavid PerrellaNurdin KaparovAllen ZangIlia SucholutskyArina KharlamovaDaniil OrelVladislav PoritskiShalev Ben-DavidZachary BergerParker WhitfillMichael FosterDaniel MunroLinh HoShankar SivarajanDan Bar HavaAleksey KuchkinDavid HolmesAlexandra Rodriguez-RomeroFrank SommerhageAnji ZhangRichard MoatKeith SchneiderZakayo KazibweDon ClarkeDae Hyun KimFelipe Meneguitti DiasSara FishVeit ElserTobias KreimanVictor Efren Guadarrama VilchisImmo KloseUjjwala AnantheswaranAdam ZweigerKaivalya RawalJeffery LiJeremy NguyenNicolas DaansHaline HeidingerMaksim RadionovVáclav RozhoňVincent GinisChristian StumpNiv CohenRafał PoświataJosef TkadlecAlan GoldfarbChenguang WangPiotr PadlewskiStanislaw BarzowskiKyle MontgomeryRyan StendallJamie Tucker-FoltzJack StadeT. Ryan RogersTom GoertzenDeclan GrabbAbhishek ShuklaAlan GivréJohn Arnold AmbayArchan SenMuhammad Fayez AzizMark H InlowHao HeLing ZhangYounesse KaddarIvar ÄngquistYanxu ChenHarrison K WangKalyan RamakrishnanElliott ThornleyAntonio TerpinHailey SchoelkopfEric ZhengAvishy CarmiEthan D. L. BrownKelin ZhuMax BartoloRichard WheelerMartin StehbergerPeter BradshawJP HeimonenKaustubh SridharIdo AkovJennifer SandlinYury MakarychevJoanna TamHieu HoangDavid M. CunninghamVladimir GoryachevDemosthenes PatramanisMichael KrauseAndrew RedentiDavid AldousJesyin LaiShannon ColemanJiangnan XuSangwon LeeIlias MagoulasSandy ZhaoNing TangMichael K. CohenOrr ParadiseJan Hendrik KirchnerMaksym OvchynnikovJason O. MatosAdithya ShenoyMichael WangYuzhou NieAnna Sztyber-BetleyPaolo FaraboschiRobin RibletJonathan CrozierShiv HalasyamaniShreyas VermaPrashant JoshiEli MerilZiqiao MaJérémy AndréolettiRaghav SinghalJacob PlatnickVolodymyr NevirkovetsLuke BaslerAlexander IvanovSeri KhouryNils GustafssonMarco PiccardoHamid MostaghimiQijia ChenVirendra SinghTrần Quốc KhánhPaul RosuHannah SzlykZachary BrownHimanshu NarayanAline MenezesJonathan RobertsWilliam AlleyKunyang SunArkil PatelMax LamparthAnka ReuelLinwei XinHanmeng XuJacob LoaderFreddie MartinZixuan WangAndrea AchilleosThomas PreuTomek KorbakIda BosioFereshteh KazemiZiye ChenBiró BálintEve J. Y. LoJiaqi WangMaria Inês S. NunesJeremiah MilbauerM Saiful BariZihao WangBehzad AnsarinejadYewen SunStephane DurandHossam ElgnainyGuillaume DouvilleDaniel TorderaGeorge BalabanianHew WolffLynna KvistadHsiaoyun MillironAhmad SakorMurat EronAndrew Favre D.O.Shailesh ShahXiaoxiang ZhouFiruz KamalovSherwin AbdoliTim SantensShaul BarkanAllison TeeRobin ZhangAlessandro TomasielloG. Bruno De LucaShi-Zhuo LooiVinh-Kha LeNoam KoltJiayi PanEmma RodmanJacob DroriCarl J FossumNiklas MuennighoffMilind JagotaRonak PradeepHonglu FanJonathan EicherMichael ChenKushal ThamanWilliam MerrillMoritz FirschingCarter HarrisȘtefan CiobâcăJason GrossRohan PandeyIlya GusevAdam JonesShashank AgnihotriPavel ZhelnovMohammadreza MofayeziAlexander PiperskiDavid K. ZhangKostiantyn DobarskyiRoman LeventovIgnat SorokoJoshua DuerschVage TaamazyanAndrew HoWenjie MaWilliam HeldRuicheng XianArmel Randy ZebazeMohanad MohamedJulian Noah LeserMichelle X YuanLaila YacarJohannes LenglerKatarzyna OlszewskaClaudio Di FrattaEdson OliveiraJoseph W. JacksonAndy ZouMuthu ChidambaramTimothy ManikHector HaffendenDashiell StanderAli DasouqiAlexander ShenBita GolshaniDavid StapEgor KretovMikalai UzhouAlina Borisovna ZhidkovskayaNick WinterMiguel Orbegozo RodriguezRobert LauffDustin WehrColin TangZaki HossainShaun PhillipsFortuna SamueleFredrik EkströmAngela HammonOam PatelFaraz FarhidiGeorge MedleyForough MohammadzadehMadellene PeñaflorHaile KassahunAlena FriedrichRayner Hernandez PerezDaniel PydaTaom SakalOmkar DhamaneAli Khajegili MirabadiEric HallmanKenchi OkutsuMike BattagliaMohammad MaghsoudimehrabaniAlon AmitDave HulbertRoberto PereiraSimon WeberHandokoAnton PeristyyStephen MalinaMustafa MehkaryRami AlyFrank ReidegeldAnna-Katharina DickCary FridayMukhwinder SinghHassan ShapourianWanyoung KimMariana CostaHubeyb GurdoganHarsh KumarChiara CeconelloChao ZhuangHaon ParkMicah CarrollAndrew R. TawfeekStefan SteinerbergerDaattavya AggarwalMichael KirchhofLinjie DaiEvan KimJohan FerretJainam ShahYuzhou WangMinghao YanKrzysztof BurdzyLixin ZhangAntonio FrancaDiana T. PhamKang Yong LohJoshua RobinsonAbram JacksonPaolo GiordanoPhilipp PetersenAdrian CosmaJesus ColinoColin WhiteJacob VotavaVladimir VinnikovEthan DelaneyPetr SpeldaVit StriteckySyed M. ShahidJean-Christophe MourratLavr VetoshkinKoen SponseleeRenas BachoZheng-Xin YongFlorencia de la RosaNathan ChoXiuyu LiGuillaume MalodOrion WellerGuglielmo AlbaniLeon LangJulien LaurendeauDmitry KazakovFatimah AdesanyaJulien PortierLawrence HollomVictor SouzaYuchen Anna ZhouJulien DegorreYiğit YalınGbenga Daniel ObikoyaRai (Michael Pokorny)Filippo BigiM.C. BoscáOleg ShumarKaniuar BachoGabriel RecchiaMara PopescuNikita ShulgaNgefor Mildred TanwieThomas C.H. LuxBen RankColin NiMatthew BrooksAlesia YakimchykHuanxu (Quinn) LiuStefano CavalleriOlle HäggströmEmil VerkamaJoshua NewbouldHans GundlachLeonor Brito-SantanaBrian AmaroVivek VajipeyRynaa GroverTing WangYosi KratishWen-Ding LiSivakanth GopiAndrea CaciolaiChristian Schroeder de WittPablo Hernández-CámaraEmanuele RodolàJules RobinsDominic WilliamsonVincent ChengBrad RaynorHao QiBen SegevJingxuan FanSarah MartinsonErik Y. WangKaylie HausknechtMichael P. BrennerMao MaoChristoph DemianPeyman KassaniXinyu ZhangDavid AvagianEshawn Jessica ScipioAlon RagolerJustin TanBlake SimsRebeka PlecnikAaron KirtlandOmer Faruk BodurD.P. ShindeYan Carlos Leyva LabradorZahra AdoulMohamed ZekryAli KarakocTania C. B. SantosSamir ShamseldeenLoukmane KarimAnna LiakhovitskaiaNate ResmanNicholas FarinaJuan Carlos GonzalezGabe MaayanEarth AndersonRodrigo De Oliveira PenaElizabeth KelleyHodjat MarijiRasoul PouriamaneshWentao WuRoss FinocchioIsmail AlarabJoshua ColeDanyelle FerreiraBryan JohnsonMohammad SafdariLiangti DaiSiriphan ArthornthurasukIsaac C. McAlisterAlejandro José MoyanoAlexey ProninJing FanAngel Ramirez-TrinidadYana MalyshevaDaphiny PottmaierOmid TaheriStanley StepanicSamuel PerryLuke AskewRaúl Adrián Huerta RodríguezAli M. R. MinissiRicardo LorenaKrishnamurthy IyerArshad Anil FasiludeenRonald ClarkJosh DuceyMatheus PizaMaja SomrakEric VergoJuehang QinBenjámin BorbásEric ChuJack LindseyAntoine JallonI.M.J. McInnisEvan ChenAvi SemlerLuk GloorTej ShahMarc CarauleanuPascal LauerTran Đuc HuyHossein ShahrtashEmilien DucLukas LewarkAssaf BrownSamuel AlbanieBrian WeberWarren S. VazPierre ClavierYiyang FanGabriel Poesia Reis e SilvaLong (Tony) LianMarcus AbramovitchXi JiangSandra MendozaMurat IslamJuan GonzalezVasilios MavroudisJustin XuPawan KumarLaxman Prasad GoswamiDaniel BugasNasser HeydariFerenc JeanplongThorben JansenAntonella PintoArchimedes AprontiAbdallah GalalNg Ze-AnAnkit SinghTong JiangJoan of Arc XavierKanu Priya AgarwalMohammed BerkaniGang ZhangZhehang DuBenedito Alves de Oliveira JuniorDmitry MalishevNicolas RemyTaylor D. HartmanTim TarverStephen MensahGautier Abou LoumeWiktor MorakFarzad HabibiSarah HobackWill CaiJavier GimenezRoselynn Grace MontecilloJakub ŁuckiRussell CampbellAsankhaya SharmaKhalida MeerShreen GulDaniel Espinosa GonzalezXavier AlapontAlex HooverGunjan ChhablaniFreddie VargusArunim AgarwalYibo JiangDeepakkumar PatilDavid OutevskyKevin Joseph ScariaRajat MaheshwariAbdelkader DendanePriti ShuklaAshley CartwrightSergei BogdanovNiels MündlerSören MöllerLuca ArnaboldiKunvar ThamanMuhammad Rehan SiddiqiPrajvi SaxenaHimanshu GuptaTony FruhauffGlen ShermanMátyás VinczeSiranut UsawasutsakornDylan LerAnil RadhakrishnanInnocent EnyekweSk Md SalauddinJiang MuzhenAleksandr MaksapetyanVivien RossbachChris HarjadiMohsen BahaloohorehClaire SparrowJasdeep SidhuSam AliSong BianJohn LaiEric SingerJustine Leon UroGreg BatemanMohamed SayedAhmed MenshawyDarling DucloselDario BezziYashaswini JainAshley AaronMurat TiryakiogluSheeshram SiddhKeith KrenekImad Ali ShahJun JinScott CreightonDenis PeskoffZienab EL-WasifRagavendran P VMichael RichmondJoseph McGowanTejal Patwardhan

Late Contributors

Hao-Yu SunTing SunNikola ZubićSamuele SalaStephen EbertJean KaddourManuel SchottdorfDianzhuo WangGerol PetruzellaAlex MeiburgTilen MedvedAli ElSheikhS Ashwin HebbarLorenzo VaqueroXianjun YangJason PoulosVilém ZouharSergey BogdanikMingfang ZhangJorge Sanz-RosDavid AnugrahaYinwei DaiAnh N. NhuXue WangAli Anil DemircaliZhibai JiaYuyin ZhouJuncheng WuMike HeNitin ChandokAarush SinhaGaoxiang LuoLong LeMickaël NoyéIoannis PantidisTianbo QiSoham Sachin PurohitLetitia ParcalabescuThai-Hoa NguyenGenta Indra WinataEdoardo M. PontiHanchen LiKaustubh DholeJongee ParkDario AbbondanzaYuanli WangAnupam NayakDiogo M. CaetanoAntonio A. W. L. WongMaria del Rio-ChanonaDániel KondorPieter FrancoisEd ChalstreyJakob ZsambokDan HoyerJenny ReddishJakob HauserFrancisco-Javier Rodrigo-GinésSuchandra DattaMaxwell ShepherdThom KamphuisQizheng ZhangHyunjun KimRuiji SunJianzhu YaoFranck DernoncourtSatyapriya KrishnaSina RismanchianBonan PuFrancesco PintoYingheng WangKumar ShridharKalon J. OverholtGlib BriiaHieu NguyenDavid (Quod) Soler BartomeuTony CY PangAdam WeckerYifan XiongFanfei LiLukas S. HuberJoshua JaegerRomano De MaddalenaXing Han LùYuhui ZhangClaas BegerPatrick Tser Jern KonSean LiVivek SankerMing YinYihao LiangXinlu ZhangAnkit AgrawalLi S. YifeiZechen ZhangMu CaiYasin SonmezCostin CozianuChanghao LiAlex SlenShoubin YuHyun Kyu ParkGabriele SartiMarcin BriańskiAlessandro StolfoTruong An NguyenMike ZhangYotam PerlitzJose Hernandez-OralloRunjia LiAmin ShabaniFelix Juefei-XuShikhar DhingraOrr ZoharMy Chiffon NguyenAlexander PondavenAbdurrahim YilmazXuandong ZhaoChuanyang JinMuyan JiangStefan TodoranXinyao HanJules KreuerBrian RabernAnna PlassartMartino MaggettiLuther YapRobert GeirhosJonathon KeanDingsu WangSina MollaeiChenkai SunYifan YinShiqi WangRui LiYaowen ChangAnjiang WeiAlice BizeulXiaohan WangAlexandre Oliveira ArraisKushin MukherjeeJorge Chamorro-PadialJiachen LiuXingyu QuJunyi GuanAdam BouyamournShuyu WuMartyna PlomeckaJunda ChenMengze TangJiaqi DengShreyas SubramanianHaocheng XiHaoxuan ChenWeizhi ZhangYinuo RenHaoqin TuSejong KimYushun ChenSara Vera MarjanovićJunwoo HaGrzegorz LuczynaJeff J. MaZewen ShenDawn SongCedegao E. ZhangZhun WangGaël GendronYunze XiaoLeo SmuckerErica WengKwok Hao LeeZhe YeStefano ErmonIgnacio D. Lopez-MiguelTheo KnightsAnthony GitterNamkyu ParkBoyi WeiHongzheng ChenKunal PaiAhmed ElkhananyHan LinPhilipp D. SiedlerJichao FangRitwik MishraKároly Zsolnai-FehérXilin JiangShadab KhanJun YuanRishab Kumar JainXi LinMike PetersonZhe WangAditya MalusareMaosen TangIsha GuptaIvan FosinTimothy KangBarbara DworakowskaKazuki MatsumotoGuangyao ZhengGerben SewusterJorge Pretel VillanuevaIvan RannevIgor ChernyavskyJiale ChenDeepayan BanikBen RaczWenchao DongJianxin WangLaila BashmalDuarte V. GonçalvesWei HuKaushik BarOndrej BohdalAtharv Singh PatlanShehzaad DhuliawalaCaroline GeirhosJulien WistYuval KansalBingsen ChenKutay TireAtak Talay YücelBrandon ChristofVeerupaksh SinglaZijian SongSanxing ChenJiaxin GeKaustubh PonksheIsaac ParkTianneng ShiMartin Q. MaJoshua MakSherwin LaiAntoine MoulinZhuo ChengZhanda ZhuZiyi ZhangVaidehi PatilKetan JhaQiutong MenJiaxuan WuTianchi ZhangBruno Hebling VieiraAlham Fikri AjiJae-Won ChungMohammed MahfoudHa Thi HoangMarc SperzelWei HaoKristof MedingSihan XuVassilis KostakosDavide ManiniYueying LiuChristopher ToukmajiJay PaekEunmi YuArif Engin DemircaliZhiyi SunIvan DewerpeHongsen QinRoman PflugfelderJames BaileyJohnathan MorrisVille HeilalaSybille RossetZishun YuPeter E. ChenWoongyeong YeoEeshaan JainRyan YangSreekar ChigurupatiJulia ChernyavskySai Prajwal ReddySubhashini VenugopalanHunar BatraCore Francisco ParkHieu TranGuilherme MaximianoGenghan ZhangYizhuo LiangHu ShiyuRongwu XuRui PanSiddharth SureshZiqi LiuSamaksh GulatiSongyang ZhangPeter TurchinChristopher W. BartlettChristopher R. ScotesePhuong M. Cao

Affiliations

Introduction

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam, a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. The dataset consists of 2,500 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting.

Difficulty comparison across benchmarks

Compared against the saturation of some existing benchmarks, Humanity's Last Exam accuracy remains low across several frontier models, demonstrating its effectiveness for measuring advanced, closed-ended, academic capabilities.

Dataset

Humanity's Last Exam (HLE) is a global collaborative effort, with questions from nearly 1,000 subject expert contributors affiliated with over 500 institutions across 50 countries – comprised mostly of professors, researchers, and graduate degree holders.

Examples 1-2/8

Classics

Question:

Question image

Here is a representation of a Roman inscription, originally found on a tombstone. Provide a translation for the Palmyrene script.
A transliteration of the text is provided: RGYNᵓ BT ḤRY BR ᶜTᵓ ḤBL

Henry T

Merton College, Oxford

Ecology

Question:

Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.

Edward V

Massachusetts Institute of Technology

Samples of the diverse and challenging questions submitted to Humanity's Last Exam.

Quantitative Results

Accuracy. All frontier models achieve low accuracy on Humanity's Last Exam, highlighting significant room for improvement in narrowing the gap between current LLMs and expert-level academic capabilities on closed-ended questions.

Calibration Error. Given low performance on Humanity's Last Exam, models should be calibrated, recognizing their uncertainty rather than confidently provide incorrect answers, indicative of confabulation/hallucination. To measure calibration, we prompt models to provide both an answer and their confidence from 0% to 100%.

Judge Model: o3-mini | Dataset Updated: April 3rd, 2025
ModelAccuracy (%) ↑Calibration
Error (%) ↓
o3 logoo320.334.0
Gemini 2.5 Pro logoGemini 2.5 Pro18.471.0
o4-mini logoo4-mini18.157.0
o3-mini* logoo3-mini*13.480.0
Gemini 2.5 Flash logoGemini 2.5 Flash12.180.0
DeepSeek-R1* logoDeepSeek-R1*8.573.0
Claude 3.7 Sonnet (16K) logoClaude 3.7 Sonnet (16K)8.080.0
o1 logoo18.083.0
GPT-4.5 Preview logoGPT-4.5 Preview5.485.0
GPT-4.1 logoGPT-4.15.489.0
Claude 3.5 Sonnet logoClaude 3.5 Sonnet4.184.0
GPT-4o logoGPT-4o2.789.0

*Model is not multi-modal, evaluated on text-only subset.

Also available at SEAL LLM Leaderboards

Discussion

Future Model Performance

While current LLMs achieve very low accuracy on Humanity's Last Exam, recent history shows benchmarks are quickly saturated -- with models dramatically progressing from near-zero to near-perfect performance in a short timeframe. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. High accuracy on HLE would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or "artificial general intelligence." HLE tests structured academic problems rather than open-ended research or creative problem-solving abilities, making it a focused measure of technical knowledge and reasoning. HLE may be the last academic exam we need to give to models, but it is far from the last benchmark for AI.

Impact

By providing a clear measure of AI progress, Humanity's Last Exam creates a common reference point for scientists and policymakers to assess AI capabilities. This enables more informed discussions about development trajectories, potential risks, and necessary governance measures.

Citation

For any inquiries or feedback, please contact us at agibenchmark@safe.ai