Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upMore complete Twitter Ingestion #194
Comments
ruebot
self-assigned this
Aug 10, 2018
ruebot
referenced this issue
Aug 10, 2018
Merged
Add support for full_text in tweets; resolve #192. #252
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Aug 10, 2018
Member
I think we can strike retweeted
from this list. Looking at a few of my really large datasets, I'm not seeing retweeted
equal anything but false
.
I think we can strike |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Sounds good to me @ruebot! |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Aug 10, 2018
Member
place
is an array. What do we want out of it?
Example:
$ zcat WomensMarch-20170123.json.gz | jq .place
{
"full_name": "Holden, MA",
"url": "https://api.twitter.com/1.1/geo/id/00173cd41f2b16d3.json",
"country": "United States",
"place_type": "city",
"bounding_box": {
"type": "Polygon",
"coordinates": [
[
[
-71.918466,
42.312204
],
[
-71.918466,
42.400332
],
[
-71.801795,
42.400332
],
[
-71.801795,
42.312204
]
]
]
},
"country_code": "US",
"attributes": {},
"id": "00173cd41f2b16d3",
"name": "Holden"
}
Example:
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Aug 10, 2018
Member
All of the entities
options are arrays too. So we'll have to map all those out as well.
"entities": {
"hashtags": [],
"symbols": [],
"user_mentions": [],
"urls": []
}
user_mentions
$ zcat WomensMarch-20170123.json.gz | head -n 1 | jq .entities.user_mentions
[
{
"indices": [
0,
15
],
"id_str": "18395177",
"screen_name": "solangeknowles",
"name": "solange knowles",
"id": 18395177
},
{
"indices": [
16,
28
],
"id_str": "800942537083068416",
"screen_name": "womensmarch",
"name": "Women's March",
"id": 800942537083068400
}
]
hashtags
$ zcat WomensMarch-20170123.json.gz | head -n 1 | jq .entities.hashtags
[
{
"indices": [
12,
21
],
"text": "BREAKING"
},
{
"indices": [
88,
99
],
"text": "TrumpLeaks"
},
{
"indices": [
100,
117
],
"text": "alternativefacts"
},
{
"indices": [
118,
124
],
"text": "amjoy"
}
]
urls
$ zcat WomensMarch-20170123.json.gz | head -n 1 | jq .entities.urls
[
{
"url": "https://t.co/psP0GzBgZB",
"indices": [
100,
123
],
"expanded_url": "http://www.trendinalia.com/twitter-trending-topics/singapore/singapore-today.html",
"display_url": "trendinalia.com/twitter-trendi…"
}
]
All of the
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
greebie
Aug 10, 2018
Contributor
- I think "country code," "full name" and "coordinates" for location.
- I think "id" & "screenname" for mentions
- Just the text for the hashtags. If people really care about where the hashtag was placed, they should probably use a twitter-specific tool.
- Expanded url is all we need from the urls.
|
added a commit
to ruebot/aut
that referenced
this issue
Aug 10, 2018
ruebot
referenced this issue
Aug 10, 2018
Merged
Add additional tweet fields to TweetUtils; partially address #194. #254
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Aug 10, 2018
Member
@SamFritz can we add a item for our August 22nd agenda to discuss how we should pull out these entities
? Basically, it's how we want to store them; array, list, comma delimited strings. In the twarc
tags.py
utility, we just have a nested loop to get at them. I'm assuming we don't want to do something like this to get the multiple .entities.hashtags.text
, entities.mentions.name
, etc.
...unless @lintool knows of some Scala tricks with json off the top of his head.
@SamFritz can we add a item for our August 22nd agenda to discuss how we should pull out these ...unless @lintool knows of some Scala tricks with json off the top of his head. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ruebot
Aug 10, 2018
Member
As for .geo
and .place
, here is an example with both in it:
"place":
{
"country_code": "US",
"url": "https://api.twitter.com/1.1/geo/id/67b98f17fdcf20be.json",
"country": "United States",
"place_type": "city",
"bounding_box": {
"type": "Polygon",
"coordinates": [
[
[
-71.191421,
42.227797
],
[
-71.191421,
42.399542
],
[
-70.986004,
42.399542
],
[
-70.986004,
42.227797
]
]
]
},
"full_name": "Boston, MA",
"attributes": {},
"id": "67b98f17fdcf20be",
"name": "Boston"
},
"geo":
{
"type": "Point",
"coordinates": [
42.33887,
-71.08839
]
}
As for "place":
{
"country_code": "US",
"url": "https://api.twitter.com/1.1/geo/id/67b98f17fdcf20be.json",
"country": "United States",
"place_type": "city",
"bounding_box": {
"type": "Polygon",
"coordinates": [
[
[
-71.191421,
42.227797
],
[
-71.191421,
42.399542
],
[
-70.986004,
42.399542
],
[
-70.986004,
42.227797
]
]
]
},
"full_name": "Boston, MA",
"attributes": {},
"id": "67b98f17fdcf20be",
"name": "Boston"
},
"geo":
{
"type": "Point",
"coordinates": [
42.33887,
-71.08839
]
}
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
greebie
Aug 10, 2018
Contributor
Good idea. My basic instincts suggest we either use classes, or just accept tuples. The coordinates thing seems complicated though and could just be a quagmire of nested Objects if we go the class route.
Good idea. My basic instincts suggest we either use classes, or just accept tuples. The coordinates thing seems complicated though and could just be a quagmire of nested Objects if we go the class route. |
greebie commentedApr 6, 2018
•
edited by ruebot
Edited 12 times
-
-
-
-
-
-
-
-
-
-
-
-
greebie
edited Apr 6, 2018
ruebot editedAug 10, 2018 (most recent)
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
ruebot editedAug 10, 2018
Threaded from #192
Going through the twitter ingestion script, it might be nice to include a few other things if possible from the api. Retweet_count for instance. Possibly things like "location" and "in_reply_to" depending on how far we want to go with that and ethical considerations.
Here's a short list from my perspective and maybe we can check the things we want to include, and then we can fix things all together:
retweeted (true / false)This is just a rough list of possible candidates. Not necessarily suggesting we should ingest them all or even one. Just think it's worth looking at while we fix TweetUtils.