[DO NOT MERGE] feat: LDBC benchmark and tests #547
Conversation
| if (Files.notExists(resourcesDir)) { | ||
| Files.createDirectory(resourcesDir) | ||
| } | ||
| val connection = LDBCUrl.openConnection() |
There was a problem hiding this comment.
We’re doing this manually to avoid a dependency, I take it?
There was a problem hiding this comment.
@rjurney For me, the perfect solution would be to grate the repository graphframes/graphframes-testing, place all the LDBC data here, add downloading scripts and use it as a git submodule in main graphframes. What do you think about it?
| .text(resourcesDir.resolve(properties.getProperty(s"graph.${caseName}.edge-file")).toString) | ||
| .withColumn("split", split(col("value"), " ")) | ||
| .select( | ||
| col("split").getItem(0).cast(LongType).alias(GraphFrame.SRC), |
There was a problem hiding this comment.
Same thing here, for CSV?
There was a problem hiding this comment.
I think you are right, I can read as a CSV with schema and a space as a separator.
| val enabled = spark.conf.getOption("spark.sql.adaptive.enabled") | ||
| try { | ||
| // disable AQE | ||
| spark.conf.set("spark.sql.adaptive.enabled", value = false) |
There was a problem hiding this comment.
Why disable AQE? I’ve found some algorithms can be painfully slow without it. Avoid bugs?
There was a problem hiding this comment.
This PR at the moment is for testing / sharing / discussions. I was trying to disable the AQE after thread in mailing list
SemyonSinchenko
changed the title
What changes were proposed in this pull request?
Why are the changes needed?