- Part 2

How to set virtualenv for a crontab?

Recent I was asked to batch run Python script in a virtualenv and also run in crontab. for example, pyspark_hello_world.py

import sys
from pyspark import SparkContext
from operator import add

sc = SparkContext()
data = sc.parallelize(list(sys.argv[1]))
counts = data.map(lambda x: (x, 1))\
.reduceByKey(add)\
.sortBy(lambda x: x[1], ascending=False)\
.collect()
for (word, count) in counts:
print("{}: {}".format(word, count))
sc.stop()

import sys

from pyspark import SparkContext

from operator import add

sc = SparkContext()

data = sc.parallelize(list(sys.argv[1]))

counts = data.map(lambda x: (x, 1))\

.reduceByKey(add)\

.sortBy(lambda x: x[1], ascending=False)\

.collect()

for (word, count) in counts:

print("{}: {}".format(word, count))

sc.stop()

I …

Using EDUP EP-N8508GS USB Mini WIFI adapter on Windows 2008r2

We have a Server installed Windows 2008r2, and I have to use wireless. I have 3 optional solution, Xiaomi Mini Wifi adapter, …

How to Connect Spark with PostgreSQL

After installing SparkR, Let’s try to import data from PostgreSQL,

#tell the path of Spark
Sys.setenv(SPARK_HOME="/Users/steven/Applications/spark2")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
 
library(SparkR)
 
d.pg="org.postgresql:postgresql:9.4.1209.jre7"
sc <- sparkR.session(master="local[2]",sparkPackages=c(d.pg))
 
url<-"jdbc:postgresql://localhost:5432/steven?user=postgres&password="
driver<-"org.postgresql.Driver"
df.pg <- read.jdbc(source="jdbc", url=url, tableName ="public.mtcars",driver=driver)
 
printSchema(df.pg)
 
collect(df.pg)
#createOrReplaceTempView(df.pg,"mtcars")
#sql("select * from mtcars")
 
sparkR.session.stop()

#tell the path of Spark

Sys.setenv(SPARK_HOME="/Users/steven/Applications/spark2")

.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))

library(SparkR)

d.pg="org.postgresql:postgresql:9.4.1209.jre7"

sc <- sparkR.session(master="local[2]",sparkPackages=c(d.pg))

url<-"jdbc:postgresql://localhost:5432/steven?user=postgres&password="

driver<-"org.postgresql.Driver"

df.pg <- read.jdbc(source="jdbc", url=url, tableName ="public.mtcars",driver=driver)

printSchema(df.pg)

collect(df.pg)

#createOrReplaceTempView(df.pg,"mtcars")

#sql("select * from mtcars")

sparkR.session.stop()

Currently browsing author

rubypdf, Page 2

How to set virtualenv for a crontab?

Using EDUP EP-N8508GS USB Mini WIFI adapter on Windows 2008r2

How to Connect Spark with PostgreSQL