Context: The potential upgrade to Spark 2.3.0 in the issue-197 branch (#197).

As a first task, let's try to extract a domain link graph and convert it into a GraphX graph object - and from there we can get access to GraphX features...

Potentially interesting links:

• https://github.com/bhardwajank/graphx-overlapping-community
• https://github.com/Sotera/spark-distributed-louvain-modularity

An analysis that would be helpful to me is to produce values for weak and strongly connected components that output to Gexf. This would be a good thing to pull out of GraphX rather than Graphpass or igraph or whatnot.

Basically, this is a way to extract the "big ball" (giant component) out of a bunch of groups.

Note that Graphx (I called it Sparkx and that is wrong) is not compatible with Dataframes. GraphFrames may be an alternative: http://graphframes.github.io/

Note that Graphx (I called it Sparkx and that is wrong) is not compatible with Dataframes. GraphFrames may be an alternative: http://graphframes.github.io/

Given that GraphX is not comparable with the future direction of the project (Dataframes), should we close this issue?

@hardiksahl 's work provides some helpful new direction, but it will not resolve this issue in the short-run. Closing.

After looking at the issue, there is another way forward, using connected components and graphpass.

If we use GraphX instead of the usual, we will have access Page Rank values, connected components and strongly connected components.

In Graphpass, if the the node list > 50,000, then we can try filtering using just the largest connected component (and the data is all ready there, so it is not a huge run).

If the node list is still > 50,000 then we can use the graph of just the strongly connected components. In this case, it means that we have the "true core" of websites in interaction with each other. It is not a full representation of the graph, but it is a logical criteria for a "first pass" in the graph.

In other news, as a general feature, it may be possible to offer a Gexf that at least provides colour (using components) and size visualization features for a Gephi output. Positioning will still be random, but if someone does not want to use GraphPass, it's still better than the status quo.

Can we rewrite this issue, or close and open up some news ones? It seems like there are a few different things going on here, and it's not exactly clear what the definition of done is.

I’m a bit confused too. #245 seems to have resolved the major parts of this (both the assessing/investigating and revising GraphX support). If we go further down the Gephi output road that @greebie discussed above, that might be better served with more specific issues.

I see this as resolved. The rest is just a few notes on what we could do with it.