Dejadoc & code duplication #2698

Over the past couple of weeks I have been working on dejadoc, which is a tool to find exact code duplications. It started out as a tool to find duplicated doctests, which is where it guarantees 100% accuracy, but then I just continued to have a few ideas and now it is a general code duplication tool specialized for Rust.

TL;DR: it does reduction to canonical form followed by execution graph isomorphisms for each context block larger than a given size.

I hope it is interesting, and will likely lead to some cleaup.

Ciao,
Luca

Parser, dialects, and AST

• Operator formatting arms in OperatorClassItem::fmt and OperatorFamilyItem::fmt (54 tokens, 5900b4f1). src/ast/ddl.rs:5061 · src/ast/ddl.rs:5297
• Repeated blocks in Expr::fmt (33 tokens, ea06ea64). src/ast/mod.rs:2151 · src/ast/mod.rs:2165
• PrimaryKeyConstraint::span and UniqueConstraint::span (95 tokens, 17ba4875). src/ast/table_constraints.rs:498 · src/ast/table_constraints.rs:572
• Alias keyword handling in MsSqlDialect::is_select_item_alias and is_table_factor_alias (37 tokens, 786b90d1). src/dialect/mssql.rs:148 · src/dialect/mssql.rs:182
• parse_infix in MySqlDialect and SparkSqlDialect (48 tokens, 1d1e6c9e). src/dialect/mysql.rs:103 · src/dialect/spark.rs:141
• Blocks in Snowflake parse_create_table and parse_create_database (31 tokens, 131fed19). src/dialect/snowflake.rs:1021 · src/dialect/snowflake.rs:1218
• Arms in parse_expr_prefix_by_unreserved_word (32 tokens, ad7f100d). src/parser/mod.rs:1695 · src/parser/mod.rs:1706
• Blocks in parse_cache_table (103 tokens, 54bd5405). src/parser/mod.rs:5658 · src/parser/mod.rs:5691
• Blocks in parse_function_arg and parse_aggregate_function_arg (38 tokens, 78e10075). src/parser/mod.rs:6391 · src/parser/mod.rs:6455
• Blocks in parse_create_operator_class and parse_operator_family_add_operator (39 tokens, 427630ff). src/parser/mod.rs:7622 · src/parser/mod.rs:11896
• Blocks in parse_create_table and parse_optimize_table (30 tokens, 644cb5fa). src/parser/mod.rs:8978 · src/parser/mod.rs:20621
• Blocks within parse_optional_column_option_inner (37 tokens, 9fb20c90). src/parser/mod.rs:9752 · src/parser/mod.rs:9788
• Blocks in parse_optional_column_option_inner and parse_optional_table_constraint (60 tokens, 498e55f2). src/parser/mod.rs:9812 · src/parser/mod.rs:10289

Tests

Any consolidation should retain the dialect-specific cases and assertions. A lexical match alone does not justify deleting coverage.

• parse_join_constraint_unnest_alias in BigQuery and PostgreSQL tests (68 tokens, bc48d2a7). tests/sqlparser_bigquery.rs:1786 · tests/sqlparser_postgres.rs:6070
• test_bigquery_trim and test_duckdb_trim (85 tokens, b4f1ce3a). tests/sqlparser_bigquery.rs:2456 · tests/sqlparser_duckdb.rs:860
• parse_delimited_identifiers in ClickHouse and SQL Server tests (205 tokens, d533ff35). tests/sqlparser_clickhouse.rs:158 · tests/sqlparser_mssql.rs:860
• Arms in parse_delimited_identifiers across five dialect test files (57 tokens, dd0c118a). tests/sqlparser_clickhouse.rs:165 · tests/sqlparser_mssql.rs:867 · tests/sqlparser_postgres.rs:4825 · tests/sqlparser_redshift.rs:110 · tests/sqlparser_snowflake.rs:1415
• Blocks in parse_use across six dialect test files (45 tokens, 4a50dddb). tests/sqlparser_clickhouse.rs:1485 · tests/sqlparser_databricks.rs:229 · tests/sqlparser_duckdb.rs:817 · tests/sqlparser_mssql.rs:1767 · tests/sqlparser_mysql.rs:604 · tests/sqlparser_snowflake.rs:3292
• First matching arms in parse_json_object and parse_mssql_json_array (30 tokens, 6628f171). tests/sqlparser_common.rs:1841 · tests/sqlparser_mssql.rs:1155
• Second matching arms in parse_json_object and parse_mssql_json_array (30 tokens, 7f486c49). tests/sqlparser_common.rs:1857 · tests/sqlparser_mssql.rs:1171
• assert_mysql_query_value arms in string-unescaping tests (37 tokens, dd3b9b9e). tests/sqlparser_common.rs:11801 · tests/sqlparser_common.rs:11864
• First matching arms in parse_create_table_like and parse_create_table_like_with_defaults (37 tokens, 9b33762e). tests/sqlparser_common.rs:19050 · tests/sqlparser_postgres.rs:685
• Second matching arms in parse_create_table_like and parse_create_table_like_with_defaults (37 tokens, 7eb8d06f). tests/sqlparser_common.rs:19067 · tests/sqlparser_postgres.rs:703
• test_select_wildcard_with_exclude in DuckDB and Snowflake tests (106 tokens, bc66e513). tests/sqlparser_duckdb.rs:156 · tests/sqlparser_snowflake.rs:1506
• parse_drop_trigger in SQL Server and MySQL tests (36 tokens, 885b1dde). tests/sqlparser_mssql.rs:2579 · tests/sqlparser_mysql.rs:4362
• parse_create_table_gencol in MySQL and SQLite tests (45 tokens, 2f1129d7). tests/sqlparser_mysql.rs:1403 · tests/sqlparser_sqlite.rs:318
• Blocks in parse_grant and parse_revoke (30 tokens, 0a0f9e04). tests/sqlparser_mysql.rs:4106 · tests/sqlparser_mysql.rs:4149
• Blocks in parse_create_index and parse_create_index_with_predicate (67 tokens, ab693f4d). tests/sqlparser_postgres.rs:3008 · tests/sqlparser_postgres.rs:3440
• Arms in test_snowflake_create_table_with_aggregation_policy (33 tokens, 57e152aa). tests/sqlparser_snowflake.rs:212 · tests/sqlparser_snowflake.rs:232
• Arms in test_snowflake_create_table_with_row_access_policy (33 tokens, f3df65a9). tests/sqlparser_snowflake.rs:252 · tests/sqlparser_snowflake.rs:274
• Arms in test_snowflake_create_table_with_tag (42 tokens, af56ee7c). tests/sqlparser_snowflake.rs:320 · tests/sqlparser_snowflake.rs:341
• Blocks in test_create_stage_with_file_format and test_copy_into_file_format (72 tokens, 0a513c29). tests/sqlparser_snowflake.rs:2123 · tests/sqlparser_snowflake.rs:2636 · tests/sqlparser_snowflake.rs:2678
• Blocks in test_create_stage_with_copy_options and test_copy_into_copy_options (46 tokens, 37e25c15). tests/sqlparser_snowflake.rs:2159 · tests/sqlparser_snowflake.rs:2713
• Arms in test_copy_into_file_format (78 tokens, 88464afb). tests/sqlparser_snowflake.rs:2636 · tests/sqlparser_snowflake.rs:2678
• Blocks in test_snowflake_stage_object_names_into_location and test_snowflake_stage_object_names_into_table (37 tokens, cfc10449). tests/sqlparser_snowflake.rs:2754 · tests/sqlparser_snowflake.rs:2777

Benchmarks

Review whether sharing setup would change what each benchmark measures.

• Closures in parse_table_factor_paren_chain and parse_function_arg_call_chain (33 tokens, 3c68771c). sqlparser_bench/benches/sqlparser_bench.rs:263 · sqlparser_bench/benches/sqlparser_bench.rs:287