AWS athena, Spark

Ignoring quotes in CSV while working in Athena , hive, spark SQL

The example below will ignore the the quates i.e if your data looks like

“a”, “b”, “c”

it will be processed as

a,b,c

Notice the serde name and the propreties

CREATE EXTERNAL TABLE IF NOT EXISTS  some_table (

a string, b string, c string

)ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.OpenCSVSerde’

WITH SERDEPROPERTIES (

  “separatorChar” = “,”,

  “quoteChar”     = “\””

)

LOCATION ‘s3://some_bucket/’;

 

 



——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/



Leave a Reply