Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dicussion: TensorFlow for Image Analysis #241

Closed
wants to merge 1 commit into from

Conversation

Projects
None yet
4 participants
@JWZ2018
Copy link
Contributor

commented Jun 29, 2018

Start setting up TensorFlow for image analysis


GitHub issue(s):
#240

This PR is mainly to have a discussion of how to proceed with add neural network inferencing for images.
The code sets up a couple functions in Python to use tensorflow to do this following the example from: https://docs.databricks.com/_static/notebooks/tensorflow-and-spark.html
Ideal usage eventually should be images.flatMap(apply_inference_on_batch) where images is the result of ExtractImageDetails
Couple issues I ran into:

  • The smaller issue: get_tensor_by_name('softmax:0') results in an error KeyError: "The name 'softmax:0' refers to a Tensor which does not exist. The operation, 'softmax', does not exist in the graph." I'm not yet familiar enough with TensorFlow. When I look this up online, a lot of people have they've encountered the same issue and said they have fixed it but did not post the solution.
  • The bigger issue: How to connect the scala and python parts together? The best would be if these python functions can be called from the scala. Otherwise re-implementing ExtractImageDetails in pyspark would be an option too.

Please suggest any ideas how to proceed.

Interested parties

@lintool @ruebot @ianmilligan1

Thanks in advance for your help with the Archives Unleashed Toolkit!

@ruebot

This comment has been minimized.

Copy link
Member

commented Jul 26, 2018

@lintool how do you want to proceed with this?

@JWZ2018

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

@ruebot @lintool I'm planning to pick this back up. I'm still not very familiar with tensorflow. Any suggestions greatly appreciated.

@ruebot

This comment has been minimized.

Copy link
Member

commented Aug 14, 2018

I have no experience with it 😆

I'll leave it to @lintool. But, once you move further along, I'm happy to do code review, and testing.

@lintool

This comment has been minimized.

Copy link
Member

commented Aug 16, 2018

I'll deal with this when I return from my travels...

@ianmilligan1

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

Pinging @lintool 😄

@ruebot

This comment has been minimized.

Copy link
Member

commented Oct 17, 2018

@lintool are we going to come back around to this one, or should we close it?

@lintool

This comment has been minimized.

Copy link
Member

commented Oct 20, 2018

I'm going to try and interest someone to work on it for a course project... :)

@ruebot

This comment has been minimized.

Copy link
Member

commented Jul 4, 2019

@lintool close this in favour of #318?

@ruebot

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

Superseded by #318

@ruebot ruebot closed this Jul 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.