SQL

Natural language to SQL query generation evaluates text-to-query fidelity and schema reasoning. This task is particularly relevant for analytics chat assistants and simplified database interfaces where users need to query data using natural language. Models must understand both the intent behind the question and the structure of the underlying database schema.

87
fireworks/deepseek-v3.1
Average duration
8036.217
Average tokens
893
Average cost
$0
100
1854.190659999972
611
opper_sql_sample_01
100
1879.4247380000115
629
opper_sql_sample_02
100
8842.519164000008
634
opper_sql_sample_03
100
2609.2546480000465
622
opper_sql_sample_04
100
1786.8949969999903
667
opper_sql_sample_05
100
41379.99856500005
694
opper_sql_sample_06
100
1497.5129470000184
626
opper_sql_sample_07
100
1560.3911670000343
623
opper_sql_sample_08
100
1569.6386639999673
631
opper_sql_sample_09
100
2018.128350999973
682
opper_sql_sample_10
100
129689.10741599996
852
opper_sql_sample_11
100
2066.4065889999392
871
opper_sql_sample_12
100
2172.2014419999596
902
opper_sql_sample_13
100
2607.3764060000713
912
opper_sql_sample_14
75
2225.901842999974
916
opper_sql_sample_15
0
2155.261067999959
946
opper_sql_sample_16
100
1838.300221000054
922
opper_sql_sample_17
75
4009.8865149999483
973
opper_sql_sample_18
50
1791.9658139999228
895
opper_sql_sample_19
100
2417.5146789999644
989
opper_sql_sample_20
100
2387.2609290000355
1059
opper_sql_sample_21
100
3572.4507610000273
1097
opper_sql_sample_22
100
3626.6308860000436
1183
opper_sql_sample_23
75
2290.060568999934
1122
opper_sql_sample_24
75
2220.1704240000026
1145
opper_sql_sample_25
75
2167.4728729999515
1092
opper_sql_sample_26
100
2409.7748879999017
1136
opper_sql_sample_27
75
2365.6847060000246
1204
opper_sql_sample_28
0
2032.5019389999852
1089
opper_sql_sample_29
100
2042.6287100000309
1076
opper_sql_sample_30