pyspark报错 java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI
·
在这里我用的是spark3.0 + scala2.12版本
首先在cmd上启动pyspark
这里有一个小度量,第一次使用参数启动pyspark,以便它下载所有graphframe的jar依赖项,很多教程启动的时候并没有指定依赖包,这可能会发生错误: (根据你的spark版本去graphframe官网找到对应的下载命令)
官网链接:graphframes
比如我下载对应的0.8.0-spark3.0-s_2.12 后,将它放入spark启动时对应的文件目录下
在终端输入
pyspark --packages graphframes:graphframes:0.8.0-spark3.0-s_2.12 --jars graphframes-0.8.0-spark3.0-s_2.12.jar

完成
再次在spark中新建ipynb文件运行
sc.addPyFile("../graphframes-0.8.0-spark3.0-s_2.12.jar")
from graphframes import *
from pyspark.sql.functions import *
举个栗子:
# Vertics DataFrame
v = spark.createDataFrame([
("a", "Alice", 34),
("b", "Bob", 36),
("c", "Charlie", 37),
("d", "David", 29),
("e", "Esther", 32),
("f", "Fanny", 38),
("g", "Gabby", 60)
], ["id", "name", "age"])
# Edges DataFrame
e = spark.createDataFrame([
("a", "b", "friend"),
("b", "c", "follow"),
("c", "b", "follow"),
("f", "c", "follow"),
("e", "f", "follow"),
("e", "d", "friend"),
("d", "a", "friend"),
("a", "e", "friend"),
("g", "e", "follow")
], ["src", "dst", "relationship"])
# Create a GraphFrame
g = GraphFrame(v, e)
g.vertices.show()
g.edges.show()

已经不会报错了。
更多推荐
所有评论(0)