I'm almost certain Jupyter Notebook works fine on HEAD. Basic PySpark and Dataframe support was added back around April.

I think @ianmilligan1 has some basic documentation around this hanging around. We haven't published any of this on the site because we're waiting to get to our next release. Once that happens it wiil. But in the interim, all we really have is outlined in closed and open issues: 1.0.0 Release of AUT, DataFrames and PySpark.

@SamFritz can you add this to our August 22nd agenda?

👍 yep, I think it's worth talking about notebooks - we've got them up and running for PySpark, but as an intermediate step it might be worth documenting using notebooks like this. I'll find some time to follow these steps before our meeting.

thanks @Natkeeran - tested and you can see how it's running on my end here:

A few quick notes:

• I used the most recent build (i.e. wget http://www-eu.apache.org/dist/zeppelin/zeppelin-0.8.0/zeppelin-0.8.0-bin-all.tgz)
• on Mac, I didn't have to tinker with anything in /opt but rather just edited the zeppelin-env.sh file in /conf

It's pretty straightforward except for editing the configuration file. We'll have to give some thought around how best to document that..

Oh, and scripts need to be compressed onto one line (the "paragraph" style we use with :paste in shell doesn't work, out of the box at least - I haven't dug deep to see if there is a workaround).

@ianmilligan1 Looks good. You can still use the paragraph style, but would need to add a brackets () as below.

val domains_list = (RecordLoader.loadArchives("/home/aut/data/*.gz", sc)
.keepValidPages()
.map(r => ExtractDomain(r.getUrl))
.countItems())

This will be resolved with updates to the user documentation (0.18.0 should be out next week), and Using AUT with PySpark (In progress).