在这里我用的是spark3.0 + scala2.12版本

首先在cmd上启动pyspark
这里有一个小度量,第一次使用参数启动pyspark,以便它下载所有graphframe的jar依赖项,很多教程启动的时候并没有指定依赖包,这可能会发生错误: (根据你的spark版本去graphframe官网找到对应的下载命令)

官网链接:graphframes

比如我下载对应的0.8.0-spark3.0-s_2.12 后,将它放入spark启动时对应的文件目录下
在终端输入

pyspark --packages graphframes:graphframes:0.8.0-spark3.0-s_2.12 --jars graphframes-0.8.0-spark3.0-s_2.12.jar

在这里插入图片描述

完成
再次在spark中新建ipynb文件运行

sc.addPyFile("../graphframes-0.8.0-spark3.0-s_2.12.jar")
from graphframes import *
from pyspark.sql.functions import *

举个栗子:

# Vertics DataFrame
v = spark.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 37),
  ("d", "David", 29),
  ("e", "Esther", 32),
  ("f", "Fanny", 38),
  ("g", "Gabby", 60)
], ["id", "name", "age"])

# Edges DataFrame
e = spark.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
  ("f", "c", "follow"),
  ("e", "f", "follow"),
  ("e", "d", "friend"),
  ("d", "a", "friend"),
  ("a", "e", "friend"),
  ("g", "e", "follow")
], ["src", "dst", "relationship"])

# Create a GraphFrame
g = GraphFrame(v, e)

g.vertices.show()
g.edges.show()

在这里插入图片描述
已经不会报错了。

Logo

北京人形旗下天工造物具身智能开源社区,聚焦具身天工与慧思开物两大平台

更多推荐