For some reason, I have to convert sas data to hdfs then analyse with pyspark. after some research I found spark-sas7bdat is the best solution for me. This package allows reading SAS files in local or distributed filesystem as Spark DataFrames. Schema is automatically inferred from meta information embedded in the …
I am a coder, and focus on machine learning now. for some reason, I learned SAS 2 years ago, for a coder, SAS language is very strange, but I had to learn it. when I am familiar with it, I have to learn Python now. So I collect some useful …
Recent I was asked to batch run Python script in a virtualenv and also run in crontab. for example, pyspark_hello_world.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
import sys from pyspark import SparkContext from operator import add sc = SparkContext() data = sc.parallelize(list(sys.argv[1])) counts = data.map(lambda x: (x, 1))\ .reduceByKey(add)\ .sortBy(lambda x: x[1], ascending=False)\ .collect() for (word, count) in counts: print("{}: {}".format(word, count)) sc.stop() |
I …