Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQL: placeholder in identifier #3169

Open
Appalled opened this issue Sep 28, 2021 · 11 comments · May be fixed by #3172
Open

SQL: placeholder in identifier #3169

Appalled opened this issue Sep 28, 2021 · 11 comments · May be fixed by #3172

Comments

@Appalled
Copy link

Hi dear ctags developers,
I run into a problem while generating tags for the following SQL script with placeholder:

create table  database.tb_name${dt} as 
select col_a, col_b from database.tb_name

table 'tb_name${dt}' is not parsed as an identifier, and the same situation occurs to column name. Would you consider enhancing the sql parser to allow identifier with placeholder such as '${a_1}' ?
Thanks.

@masatake
Copy link
Member

masatake commented Sep 28, 2021

Where can I find the definition of SQL syntax for ${dt} ?

When extending a parser, I need information about the syntax.

such as '${a_1}' ?

"such as" is not enough to extend the parser.

@Appalled
Copy link
Author

Where can I find the definition of SQL syntax for ${dt} ?

When extending a parser, I need information about the syntax.

such as '${a_1}' ?

"such as" is not enough to extend the parser.

I didn't find the strict definition but a page in the hive language mannual:
LanguageManual VariableSubstitution.
Hope it could be helpful. Maybe I should search more.

@masatake
Copy link
Member

Thank you for the information.
Is the notation, ${var} is HiveQL specific?
I also want to know the file name extension for HiveQL. .sql?

masatake added a commit to masatake/ctags that referenced this issue Oct 2, 2021
Close universal-ctags#3169.

Some? dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
masatake added a commit to masatake/ctags that referenced this issue Oct 2, 2021
Close universal-ctags#3169.

Some? dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
@masatake
Copy link
Member

masatake commented Oct 2, 2021

See #3172.
You can try the pull request supporting ${var}. See the instructions written in
https://docs.ctags.io/en/latest/contributions.html#testing-a-pr-locally-before-being-merged
.

$ cat input.sql 
-- Based on issue #3169 opened by @Appalled

create table  database.tb_name${dt} as 
select col_a, col_b from database.tb_name;

create table  database.tb_name${dt}${dt0} as 
select col_a, col_b from database.tb_name;

create table  database.${dt1}tb_name${dt}${dt0} as 
select col_a, col_b from database.tb_name;

create table  database.${dt1}tb_name${dt}${dt0}Z as 
select col_a, col_b from database.tb_name;

create table  database.tb_${dt2}_name as 
select col_${key0}, col_${key1} from database.tb_name;
$ u-ctags --sort=no -o - input.sql 
tb_name${dt}	input.sql	/^create table  database.tb_name${dt} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}
tb_name${dt}${dt0}	input.sql	/^create table  database.tb_name${dt}${dt0} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}${dt0}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}${dt0}
${dt1}tb_name${dt}${dt0}	input.sql	/^create table  database.${dt1}tb_name${dt}${dt0} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}
${dt1}tb_name${dt}${dt0}Z	input.sql	/^create table  database.${dt1}tb_name${dt}${dt0}Z as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}Z
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}Z
tb_${dt2}_name	input.sql	/^create table  database.tb_${dt2}_name as $/;"	t
col_${key0}	input.sql	/^select col_${key0}, col_${key1} from database.tb_name;$/;"	E	table:tb_${dt2}_name
col_${key1}	input.sql	/^select col_${key0}, col_${key1} from database.tb_name;$/;"	E	table:tb_${dt2}_name
$ 

masatake added a commit to masatake/ctags that referenced this issue Oct 2, 2021
Close universal-ctags#3169.

Some? dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
masatake added a commit to masatake/ctags that referenced this issue Oct 2, 2021
Close universal-ctags#3169.

Some? dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
masatake added a commit to masatake/ctags that referenced this issue Oct 3, 2021
Close universal-ctags#3169.

Some? dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
masatake added a commit to masatake/ctags that referenced this issue Oct 4, 2021
Close universal-ctags#3169.

Some(?) dialects support shell-like variable substitution.
HiveQL is one of such dialects.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VariableSubstitution

With this change, the SQL parser accepts '${var}' as a part of an identifier.

TODO: `var` itself can be extracted as a reference tag.
Signed-off-by: Masatake YAMATO <[email protected]>
@Appalled
Copy link
Author

Appalled commented Oct 5, 2021

See #3172. You can try the pull request supporting ${var}. See the instructions written in https://docs.ctags.io/en/latest/contributions.html#testing-a-pr-locally-before-being-merged .

$ cat input.sql 
-- Based on issue #3169 opened by @Appalled

create table  database.tb_name${dt} as 
select col_a, col_b from database.tb_name;

create table  database.tb_name${dt}${dt0} as 
select col_a, col_b from database.tb_name;

create table  database.${dt1}tb_name${dt}${dt0} as 
select col_a, col_b from database.tb_name;

create table  database.${dt1}tb_name${dt}${dt0}Z as 
select col_a, col_b from database.tb_name;

create table  database.tb_${dt2}_name as 
select col_${key0}, col_${key1} from database.tb_name;
$ u-ctags --sort=no -o - input.sql 
tb_name${dt}	input.sql	/^create table  database.tb_name${dt} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}
tb_name${dt}${dt0}	input.sql	/^create table  database.tb_name${dt}${dt0} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}${dt0}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:tb_name${dt}${dt0}
${dt1}tb_name${dt}${dt0}	input.sql	/^create table  database.${dt1}tb_name${dt}${dt0} as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}
${dt1}tb_name${dt}${dt0}Z	input.sql	/^create table  database.${dt1}tb_name${dt}${dt0}Z as $/;"	t
col_a	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}Z
col_b	input.sql	/^select col_a, col_b from database.tb_name;$/;"	E	table:${dt1}tb_name${dt}${dt0}Z
tb_${dt2}_name	input.sql	/^create table  database.tb_${dt2}_name as $/;"	t
col_${key0}	input.sql	/^select col_${key0}, col_${key1} from database.tb_name;$/;"	E	table:tb_${dt2}_name
col_${key1}	input.sql	/^select col_${key0}, col_${key1} from database.tb_name;$/;"	E	table:tb_${dt2}_name
$ 

Sorry for the late reply. I was on vacation. It works like a charm, and thanks again for your immediate response.

@masatake
Copy link
Member

masatake commented Oct 5, 2021

Is the notation, ${var} is HiveQL specific?

I would like to get the answer to this question.
I wonder where this enhancement comes from. Do you want this enhancement because you are using HiveQL?

@Appalled
Copy link
Author

Appalled commented Oct 5, 2021

Is the notation, ${var} is HiveQL specific?

I would like to get the answer to this question. I wonder where this enhancement comes from. Do you want this enhancement because you are using HiveQL?

I believe it's not HiveQL specific. This problem emerges while I am using an in-company ad hoc query tool with spark and presto query engines on hive tables. In my experience, sql dialects of Spark, Presto and Hive all support variable substitution. And I just use '.sql' as file name extension.

@masatake
Copy link
Member

masatake commented Oct 5, 2021

Thank you.

@masatake
Copy link
Member

I looked into Presto a bit.
I run `grep '${' in the files copied from https://github.com/prestodb/presto/tree/master/presto-docs/src/main/sphinx .
I cannot find such a pattern used as part of SQL.

@masatake
Copy link
Member

masatake commented Oct 11, 2021

About, spark and presto, we cannot find the syntax definition for ${...}. So I would like to forget it.

I focus on Hive. I wonder whether we can say "ctags supports HiveSQL".

I had some questions.

And I just use '.sql' as file name extension.

I found .q is used. https://github.com/apache/hive/blob/ac4020789149115c71bcee9b85577fff7dd01f46/ql/src/test/queries/clientnegative/materialized_view_create_failure.q

How do you think about this?
Do you think ctags should recognize foo.q as a SQL file by default?

Just parsing ${...} is enough for supporting HiveSQL?
How about set var ... ?

I think the change in #3172 is too ad-hoc. So I wonder whether I should merge the pull request or not.

@masatake masatake changed the title SQL parser enhencement: placeholder in identifier SQL: placeholder in identifier Oct 17, 2021
@masatake
Copy link
Member

After seme researching I guess you write about hplsql. Am I correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants