Перейти к основному содержимому
Версия: 2.0.x

Интерфейс query_dump

Эта тема описывает, как использовать интерфейс query_dump для получения деталей SQL-запроса и связанной информации.

Если вы столкнулись с любой из следующих проблем при выполнении SQL-запросов в Selena, вы можете использовать query_dump для получения деталей SQL и отправки информации в техническую поддержку Selena для устранения неполадок:

  • Unknown Error возвращается при выполнении SQL-запроса или EXPLAIN.
  • Возвращается сообщение об ошибке или исключение при выполнении SQL-запроса.
  • Выполнение SQL-запроса не такое эффективное, как ожидалось, или план выполнения может быть оптимизирован (например, partition могут быть отсечены или порядок JOIN может быть скорректирован).

Обзор функции

Интерфейс query_dump возвращает информацию, на которую опирается FE при выполнении SQL, включая:

  • Оператор запроса
  • Оператор создания таблицы
  • Переменные сессии
  • Количество BE
  • Статистическую информацию (минимальные, максимальные значения в колонке)
  • Информацию об исключениях (стек ошибок)
  • Информацию о стоимости плана (Explain costs)

Для обеспечения конфиденциальности данных мы обезличиваем метаинформацию, такую как имена баз данных, таблиц и колонок. Мы также используем обезличенные метаданные для переписывания операторов запросов.

Обезличивание метаинформации включено по умолчанию. Если во время процесса обезличивания возникает исключение, используется оригинальная информация. Если обезличивание нужно обойти, вы можете добавить "mock=false" в HTTP URI.

Синтаксис

HTTP Post

 fe_host:fe_http_port/api/query_dump?db=${database}&mock=${value} post_data=${Query}
wget --user=${username} --password=${password} --post-file ${query_file} "http://${fe_host}:${fe_http_port}/api/query_dump?db=${database}&mock={value}" -O ${dump_file}

Описание параметров:

  • query_file: файл, содержащий запрос
  • dump_file: выходной файл
  • db: база данных, в которой выполняется SQL-запрос. Параметр db необязателен, если запрос включает use db. В противном случае он должен быть указан.
  • mock: включить или отключить обезличивание

Примеры

Отключение обезличивания

Команда:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch&mock=false" -O dump_file

Возвращаемые данные:

Данные возвращаются в формате JSON.

{
"statement": "select\n l_returnflag,\n l_linestatus,\n sum(l_quantity) as sum_qty,\n sum(l_extendedprice) as sum_base_price,\n sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n avg(l_quantity) as avg_qty,\n avg(l_extendedprice) as avg_price,\n avg(l_discount) as avg_disc,\n count(*) as count_order\nfrom\n lineitem\nwhere\n l_shipdate <= date '1998-12-01'\ngroup by\n l_returnflag,\n l_linestatus\norder by\n l_returnflag,\n l_linestatus ;\n\n",
"table_meta": {
"tpch.lineitem": "CREATE TABLE `lineitem` (\n `L_ORDERKEY` int(11) NOT NULL COMMENT \"\",\n `L_PARTKEY` int(11) NOT NULL COMMENT \"\",\n `L_SUPPKEY` int(11) NOT NULL COMMENT \"\",\n `L_LINENUMBER` int(11) NOT NULL COMMENT \"\",\n `L_QUANTITY` double NOT NULL COMMENT \"\",\n `L_EXTENDEDPRICE` double NOT NULL COMMENT \"\",\n `L_DISCOUNT` double NOT NULL COMMENT \"\",\n `L_TAX` double NOT NULL COMMENT \"\",\n `L_RETURNFLAG` char(1) NOT NULL COMMENT \"\",\n `L_LINESTATUS` char(1) NOT NULL COMMENT \"\",\n `L_SHIPDATE` date NOT NULL COMMENT \"\",\n `L_COMMITDATE` date NOT NULL COMMENT \"\",\n `L_RECEIPTDATE` date NOT NULL COMMENT \"\",\n `L_SHIPINSTRUCT` char(25) NOT NULL COMMENT \"\",\n `L_SHIPMODE` char(10) NOT NULL COMMENT \"\",\n `L_COMMENT` varchar(44) NOT NULL COMMENT \"\",\n `PAD` char(1) NOT NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`L_ORDERKEY`)\nCOMMENT \"OLAP\"\nDISTRIBUTED BY HASH(`L_ORDERKEY`) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\",\n\"in_memory\" = \"false\",\n\"enable_persistent_index\" = \"true\",\n\"replicated_storage\" = \"true\",\n\"compression\" = \"LZ4\"\n);"
},
"table_row_count": {
"tpch.lineitem": {
"lineitem": 3
}
},
"column_statistics": {
"tpch.lineitem": {
"L_TAX": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_SHIPDATE": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
"L_EXTENDEDPRICE": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_DISCOUNT": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"L_RETURNFLAG": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"L_LINESTATUS": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"L_QUANTITY": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
}
},
"explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: L_RETURNFLAG | 10: L_LINESTATUS | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Partition: UNPARTITIONED\n OutPut Exchange Id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Exchange Id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: L_DISCOUNT, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: L_QUANTITY, DOUBLE, false]\n | 6 <-> [6: L_EXTENDEDPRICE, DOUBLE, false]\n | 7 <-> [7: L_DISCOUNT, DOUBLE, false]\n | 9 <-> [9: L_RETURNFLAG, CHAR, false]\n | 10 <-> [10: L_LINESTATUS, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: L_TAX, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: L_DISCOUNT, DOUBLE, false]\n | 29 <-> [6: L_EXTENDEDPRICE, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: lineitem, rollup: lineitem\n preAggregation: on\n Predicates: [11: L_SHIPDATE, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_TAX-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_SHIPDATE-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
"session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"Selena\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
"be_number": 1,
"be_core_stat": {
"numOfHardwareCoresPerBe": "{\"10004\":104}",
"cachedAvgNumOfHardwareCores": 104
},
"exception": [],
"version": "main_querydump",
"commit_version": "0c4d8c8d3e"
}

Включение обезличивания (по умолчанию)

Команда:

wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch -O dump_file

Возвращаемые данные:

Обезличенные данные возвращаются в формате JSON.

{
"statement": "SELECT tbl_mock_001.mock_012, tbl_mock_001.mock_007, sum(tbl_mock_001.mock_010) AS mock_019, sum(tbl_mock_001.mock_005) AS mock_020, sum(tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) AS mock_021, sum((tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) * (1 + tbl_mock_001.mock_017)) AS mock_022, avg(tbl_mock_001.mock_010) AS mock_023, avg(tbl_mock_001.mock_005) AS mock_024, avg(tbl_mock_001.mock_004) AS mock_025, count(*) AS mock_026\nFROM db_mock_000.tbl_mock_001\nWHERE tbl_mock_001.mock_013 <= '1998-12-01'\nGROUP BY tbl_mock_001.mock_012, tbl_mock_001.mock_007 ORDER BY tbl_mock_001.mock_012 ASC, tbl_mock_001.mock_007 ASC ",
"table_meta": {
"db_mock_000.tbl_mock_001": "CREATE TABLE db_mock_000.tbl_mock_001 (\nmock_008 int(11) NOT NULL ,\nmock_009 int(11) NOT NULL ,\nmock_016 int(11) NOT NULL ,\nmock_006 int(11) NOT NULL ,\nmock_010 double NOT NULL ,\nmock_005 double NOT NULL ,\nmock_004 double NOT NULL ,\nmock_017 double NOT NULL ,\nmock_012 char(1) NOT NULL ,\nmock_007 char(1) NOT NULL ,\nmock_013 date NOT NULL ,\nmock_003 date NOT NULL ,\nmock_011 date NOT NULL ,\nmock_014 char(25) NOT NULL ,\nmock_015 char(10) NOT NULL ,\nmock_002 varchar(44) NOT NULL ,\nmock_018 char(1) NOT NULL \n) ENGINE= OLAP \nDUPLICATE KEY(mock_008)\nDISTRIBUTED BY HASH(mock_008) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\"\n);"
},
"table_row_count": {
"db_mock_000.tbl_mock_001": {
"tbl_mock_001": 3
}
},
"column_statistics": {
"db_mock_000.tbl_mock_001": {
"mock_017": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_013": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE",
"mock_005": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_004": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE",
"mock_012": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"mock_007": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE",
"mock_010": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE"
}
},
"explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: mock_012 | 10: mock_007 | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Partition: UNPARTITIONED\n OutPut Exchange id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Exchange Id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: mock_004, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: mock_010, DOUBLE, false]\n | 6 <-> [6: mock_005, DOUBLE, false]\n | 7 <-> [7: mock_004, DOUBLE, false]\n | 9 <-> [9: mock_012, CHAR, false]\n | 10 <-> [10: mock_007, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: mock_017, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: mock_004, DOUBLE, false]\n | 29 <-> [6: mock_005, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: mock_001, rollup: mock_001\n preAggregation: on\n Predicates: [11: mock_013, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_017-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_013-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n",
"session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"Selena\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}",
"be_number": 1,
"be_core_stat": {
"numOfHardwareCoresPerBe": "{\"10004\":104}",
"cachedAvgNumOfHardwareCores": 104
},
"exception": [],
"version": "main_querydump",
"commit_version": "0c4d8c8d3e"
}