java - Access dataBase using Apache-Spark-SQL -
hi i'm new learner apache spark using java this correct way or not? code working,but performance wise slow don't know 1 best approach access data every loop.
dataset<row> javardd = sparksession.read().jdbc(database_url, "sample", properties); javardd.todf().registertemptable("sample"); dataset<row> users = sparksession.sql("select distinct from_user sample "); list<row> members = users.collectaslist(); (row row : members) { dataset<row> userconversation = sparksession.sql("select description sample from_user ='"+ row.getdecimal(0) +"'"); userconversation.show(); }
try create set users , execute query
sparksession.sql("select description sample from_user in usersset);
this way execute 1 query pay overhead needed db connection once.
another approach if run spark on hdfs , 1 time query use tool sqoop load sql table in hadoop , use data natively in spark.
Comments
Post a Comment