Перейти к основному содержимому

Интерфейс query_dump

В этой теме описывается, как использовать интерфейс query_dump для получения подробной информации об SQL-запросе и связанных с ним данных.

Если при выполнении SQL-запросов в Selena вы сталкиваетесь с любой из следующих проблем, вы можете использовать query_dump для получения подробностей SQL и отправки информации в техническую поддержку Selena для устранения неполадок:

  • При выполнении SQL-запроса или EXPLAIN возвращается Unknown Error.
  • При выполнении SQL-запроса возвращается сообщение об ошибке или исключение.
  • Выполнение SQL-запроса не так эффективно, как ожидалось, или план выполнения может быть оптимизирован (например, можно выполнить обрезку разделов или настроить порядок Join).

Обзор функции

Интерфейс query_dump возвращает информацию, на которую опирается FE при выполнении SQL, включая:

  • Оператор запроса
  • Оператор создания таблицы
  • Переменные сессии
  • Количество BE
  • Статистическая информация (минимальные, максимальные значения в столбце)
  • Информация об исключениях (аномальный стек)
  • Информация о стоимости Explain

Для обеспечения конфиденциальности данных мы обезличиваем метаинформацию, такую как имена баз данных, имена таблиц и имена столбцов. Мы также используем обезличенные метаданные для переписывания операторов запросов.

Обезличивание метаинформации включено по умолчанию. Если во время процесса обезличивания возникает исключение, используется исходная информация. Если необходимо обойти обезличивание, вы можете добавить "mock=false" в HTTP URI.

Синтаксис

HTTP Post

 fe_host:fe_http_port/api/query_dump?db=${database}&mock=${value} post_data=${Query}
wget --user=${username} --password=${password} --post-file ${query_file} "http://${fe_host}:${fe_http_port}/api/query_dump?db=${database}&mock={value}" -O ${dump_file}

Описание параметров:

  • query_file: файл, содержащий запрос
  • dump_file: выходной файл
  • db: база данных, в которой выполняется SQL-запрос. Параметр db является необязательным, если запрос включает use db. В противном случае он должен быть указан.
  • mock: включить или отключить обезличивание

Примеры

Отключение обезличивания

Команда:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch&mock=false" -O dump_file

Возвращаемые данные:

Данные возвращаются в формате JSON.

{
"statement": "select\n l_returnflag,\n l_linestatus,\n sum(l_quantity) as sum_qty,\n sum(l_extendedprice) as sum_base_price,\n sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n avg(l_quantity) as avg_qty,\n avg(l_extendedprice) as avg_price,\n avg(l_discount) as avg_disc,\n count(*) as count_order\nfrom\n lineitem\nwhere\n l_shipdate <= date '1998-12-01'\ngroup by\n l_returnflag,\n l_linestatus\norder by\n l_returnflag,\n l_linestatus ;\n\n",
"table_meta": {
"tpch.lineitem": "CREATE TABLE `lineitem` (\n `L_ORDERKEY` int(11) NOT NULL COMMENT \"\",\n `L_PARTKEY` int(11) NOT NULL COMMENT \"\",\n `L_SUPPKEY` int(11) NOT NULL COMMENT \"\",\n `L_LINENUMBER` int(11) NOT NULL COMMENT \"\",\n `L_QUANTITY` double NOT NULL COMMENT \"\",\n `L_EXTENDEDPRICE` double NOT NULL COMMENT \"\",\n `L_DISCOUNT` double NOT NULL COMMENT \"\",\n `L_TAX` double NOT NULL COMMENT \"\",\n `L_RETURNFLAG` char(1) NOT NULL COMMENT \"\",\n `L_LINESTATUS` char(1) NOT NULL COMMENT \"\",\n `L_SHIPDATE` date NOT NULL COMMENT \"\",\n `L_COMMITDATE` date NOT NULL COMMENT \"\",\n `L_RECEIPTDATE` date NOT NULL COMMENT \"\",\n `L_SHIPINSTRUCT` char(25) NOT NULL COMMENT \"\",\n `L_SHIPMODE` char(10) NOT NULL COMMENT \"\",\n `L_COMMENT` varchar(44) NOT NULL COMMENT \"\",\n `PAD` char(1) NOT NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`L_ORDERKEY`)\nCOMMENT \"OLAP\"\nDISTRIBUTED BY HASH(`L_ORDERKEY`) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\",\n\"in_memory\" = \"false\",\n\"enable_persistent_index\" = \"false\",\n\"replicated_storage\" = \"true\",\n\"compression\" = \"LZ4\"\n);"
},
"table_row_count": {
"tpch.lineitem": {
"lineitem": 3
}
},
"column_statistics": {
"tpch.lineitem": {
"L_TAX": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_SHIPDATE": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
"L_EXTENDEDPRICE": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_DISCOUNT": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_RETURNFLAG": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"L_LINESTATUS": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"L_QUANTITY": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
}
},
"explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: L_RETURNFLAG | 10: L_LINESTATUS | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Partition: UNPARTITIONED\n OutPut Exchange Id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Exchange Id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: L_DISCOUNT, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: L_QUANTITY, DOUBLE, false]\n | 6 <-> [6: L_EXTENDEDPRICE, DOUBLE, false]\n | 7 <-> [7: L_DISCOUNT, DOUBLE, false]\n | 9 <-> [9: L_RETURNFLAG, CHAR, false]\n | 10 <-> [10: L_LINESTATUS, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: L_TAX, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: L_DISCOUNT, DOUBLE, false]\n | 29 <-> [6: L_EXTENDEDPRICE, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: lineitem, rollup: lineitem\n preAggregation: on\n Predicates: [11: L_SHIPDATE, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_TAX-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_SHIPDATE-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
"session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
"be_number": 1,
"be_core_stat": {
"numOfHardwareCoresPerBe": "{\"10004\":104}",
"cachedAvgNumOfHardwareCores": 104
},
"exception": [],
"version": "main_querydump",
"commit_version": "0c4d8c8d3e"
}

Включение обезличивания (по умолчанию)

Команда:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch -O dump_file

Возвращаемые данные:

Обезличенные данные возвращаются в формате JSON.

{
"statement": "SELECT tbl_mock_001.mock_012, tbl_mock_001.mock_007, sum(tbl_mock_001.mock_010) AS mock_019, sum(tbl_mock_001.mock_005) AS mock_020, sum(tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) AS mock_021, sum((tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) * (1 + tbl_mock_001.mock_017)) AS mock_022, avg(tbl_mock_001.mock_010) AS mock_023, avg(tbl_mock_001.mock_005) AS mock_024, avg(tbl_mock_001.mock_004) AS mock_025, count(*) AS mock_026\nFROM db_mock_000.tbl_mock_001\nWHERE tbl_mock_001.mock_013 <= '1998-12-01'\nGROUP BY tbl_mock_001.mock_012, tbl_mock_001.mock_007 ORDER BY tbl_mock_001.mock_012 ASC, tbl_mock_001.mock_007 ASC ",
"table_meta": {
"db_mock_000.tbl_mock_001": "CREATE TABLE db_mock_000.tbl_mock_001 (\nmock_008 int(11) NOT NULL ,\nmock_009 int(11) NOT NULL ,\nmock_016 int(11) NOT NULL ,\nmock_006 int(11) NOT NULL ,\nmock_010 double NOT NULL ,\nmock_005 double NOT NULL ,\nmock_004 double NOT NULL ,\nmock_017 double NOT NULL ,\nmock_012 char(1) NOT NULL ,\nmock_007 char(1) NOT NULL ,\nmock_013 date NOT NULL ,\nmock_003 date NOT NULL ,\nmock_011 date NOT NULL ,\nmock_014 char(25) NOT NULL ,\nmock_015 char(10) NOT NULL ,\nmock_002 varchar(44) NOT NULL ,\nmock_018 char(1) NOT NULL \n) ENGINE= OLAP \nDUPLICATE KEY(mock_008)\nDISTRIBUTED BY HASH(mock_008) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\"\n);"
},
"table_row_count": {
"db_mock_000.tbl_mock_001": {
"tbl_mock_001": 3
}
},
"column_statistics": {
"db_mock_000.tbl_mock_001": {
"mock_017": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_013": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
"mock_005": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_004": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_012": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"mock_007": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"mock_010": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
}
},
"explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: mock_012 | 10: mock_007 | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Partition: UNPARTITIONED\n OutPut Exchange id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Exchange id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: mock_004, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: mock_010, DOUBLE, false]\n | 6 <-> [6: mock_005, DOUBLE, false]\n | 7 <-> [7: mock_004, DOUBLE, false]\n | 9 <-> [9: mock_012, CHAR, false]\n | 10 <-> [10: mock_007, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: mock_017, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: mock_004, DOUBLE, false]\n | 29 <-> [6: mock_005, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: mock_001, rollup: mock_001\n preAggregation: on\n Predicates: [11: mock_013, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_017-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_013-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
"session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
"be_number": 1,
"be_core_stat": {
"numOfHardwareCoresPerBe": "{\"10004\":104}",
"cachedAvgNumOfHardwareCores": 104
},
"exception": [],
"version": "main_querydump",
"commit_version": "0c4d8c8d3e"
}