Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upKill Graphpass on graphs with greater than 40,000 nodes (Issue #22) #23
Conversation
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
greebie
Jun 8, 2018
Collaborator
Hi @ianmilligan1 I wonder if you can test that this works for you and then merge if you think it's working. Thanks.
|
Hi @ianmilligan1 I wonder if you can test that this works for you and then merge if you think it's working. Thanks. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Jun 11, 2018
Member
$ ./graphpass --file 5562-gephi.graphml --output /home/nruest/Dropbox --dir /home/nruest/Dropbox -g -q
>>>>>>> GRAPHPASSING >>>>>>>>
DIRECTORY: /home/nruest/Dropbox
STRLEN PATH: 20
OUTPUT DIRECTORY: /home/nruest/Dropbox
PERCENTAGE: 0.000000
FILE: 5562-gephi.graphml
METHODS STRING: d
QUICKRUN: 1
REPORT: 0
SAVE: 1
Running graphpass on file: /home/nruest/Dropbox/5562-gephi.graphml
Successfully ingested graph with 324009 nodes.
Graphpass can only conduct analysis on graphs with less than 40000 nodes.
Exiting...
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
greebie
Jun 11, 2018
Collaborator
Suggest we wait on details from 50k run before merge. I've also confirmed that the main chunks of memory usage are from the modularity calculation and the fruchtermann rheingold algorithm for node positioning.
|
Suggest we wait on details from 50k run before merge. I've also confirmed that the main chunks of memory usage are from the modularity calculation and the fruchtermann rheingold algorithm for node positioning. |
ruebot
referenced this pull request
Jun 12, 2018
Merged
Remove successful job criteria. Follow-on to #129. #138
ianmilligan1
requested changes
Jun 12, 2018
Once you update this, Ryan, I’ll do one last local test and I think it’s good to merge?
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
greebie
Jun 12, 2018
Collaborator
Added a verbose mode based on comments by @ruebot. Now if you want all the details in stdout you add a -v or --verbose. The print-outs are mostly for testing anyway.
It will still output messages on a FAIL situation however.
greebie
commented
Jun 12, 2018
•
edited
Edited 1 time
-
greebie
edited Jun 12, 2018 (most recent)
-
greebie
created Jun 12, 2018
edited
-
Jun 12, 2018 (most recent) -
Jun 12, 2018
|
Added a verbose mode based on comments by @ruebot. Now if you want all the details in stdout you add a It will still output messages on a FAIL situation however. |
greebie commentedJun 8, 2018
•
edited
Edited 1 time
-
greebie
edited Jun 8, 2018 (most recent)
-
greebie
created Jun 8, 2018
Upgrade gexf output to 1.3.
Kill graphpass on nodes > MAX_NODES (currently set to 40,000).
After doing some preliminary testing, the problem with large graphs sits with the walktrap modularity and fruchterman rheinhold algorithms.
For now we are going to cut off analysis for any graph with more than 40,000 nodes.
Note: graphpass will load the graph into memory and then quit (freeing the memory) if there are more than 40,000 nodes. It's possible also to check on file size instead, although igraph will quit if the load causes it to run out of memory.
Longer term solution is to try snap to see if we can deal with larger graphs that way.
To test:
./graphpass -f {FILENAME}.graphmlwhere FILENAME is a graph file with greater than 40,000 nodes.