RubyPDF Blog bigdata How to Connect Spark with PostgreSQL

How to Connect Spark with PostgreSQL

After installing SparkR, Let’s try to import data from PostgreSQL,

#tell the path of Spark
Sys.setenv(SPARK_HOME="/Users/steven/Applications/spark2")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
 
library(SparkR)
 
d.pg="org.postgresql:postgresql:9.4.1209.jre7"
sc <- sparkR.session(master="local[2]",sparkPackages=c(d.pg))
 
url<-"jdbc:postgresql://localhost:5432/steven?user=postgres&password="
driver<-"org.postgresql.Driver"
df.pg <- read.jdbc(source="jdbc", url=url, tableName ="public.mtcars",driver=driver)
 
printSchema(df.pg)
 
collect(df.pg)
#createOrReplaceTempView(df.pg,"mtcars")
#sql("select * from mtcars")
 
sparkR.session.stop()

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.