GitHub issue(s):
#192

What does this Pull Request do?

Add support to the Tweet utility to use full_text from tweets.

How should this be tested?

I did this with Apache Spark 2.3.1:

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
import io.archivesunleashed.util.TweetUtils._

// Load tweets from HDFS
val tweets = RecordLoader.loadTweets("/path/to/replies.jsonl", sc)

// Count them
tweets.count()

// Extract some fields
val r = tweets.map(tweet => (tweet.id, tweet.createdAt, tweet.username, tweet.text, tweet.fullText, tweet.lang,
                             tweet.isVerifiedUser, tweet.followerCount, tweet.friendCount))

// Take a sample of 10 on console
r.take(10)

// Exiting paste mode, now interpreting.

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
import io.archivesunleashed.util.TweetUtils._
tweets: org.apache.spark.rdd.RDD[org.json4s.JValue] = MapPartitionsRDD[28] at filter at package.scala:65
r: org.apache.spark.rdd.RDD[(String, String, String, String, String, String, Boolean, Int, Int)] = MapPartitionsRDD[29] at map at <console>:50
res1: Array[(String, String, String, String, String, String, Boolean, Int, Int)] = Array((1004742470700322816,Thu Jun 07 15:10:46 +0000 2018,realDonaldTrump,"",When will people start saying, “thank you, Mr. President, for firing James Comey?”,en,true,52543149,46), (1005275688990052352,Sat Jun 09 02:29:36 +0000 2018,love4All_7,"",@realDonaldTrump Pleases look into my brothers case. Miskin Kamara, he's been locked up for 13+ years w...

If you want to use the same tweet set, it's here.

Additional Notes:

Once this is good to go, I'll take care of #194.

Interested parties

@ianmilligan1 @lintool