Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @rhiever, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

Attempting PDF compilation. Reticulating splines etc...

👉 Check article proof 📄 👈

@rhiever, @zhampel this is where the review process takes place. There is information and check boxes at the top of this issue to guide you through the review process. Let me know if you have any questions.

@Kevin-Mattheus-Moerman I must disclose an affiliation I have with the author @jshermeyer of this submission. We are employees of the same parent organization 'IQT', though we have affiliations with separate independent laboratories within IQT. I leave it to you to decide whether I proceed as a reviewer.

@zhampel thanks for disclosing that. Can you provide a link to that organization IQT?

@Kevin-Mattheus-Moerman Sure thing: https://www.iqt.org

@whedon add @dmittman as reviewer

OK, @dmittman is now a reviewer

@zhampel thanks for mentioning your shared affiliation. I do not feel this is serious enough to exclude you from the review process. I'm confident that, now that we have three reviewers, and since review takes place in the open, we'll have a balanced review process. Thanks.

@rhiever, @zhampel, @dmittman thanks again for acting as reviewers. You all have a set of tick boxes at the top of this issue which will guide you through the review process (which you can tick if you've accepted this invitation). Please let me know if you have any questions. 🚀 🤖

@jshermeyer can you work on adding community/contributing guidelines (e.g. CONTRIBUTING.md)? I also recommend adding a code of conduct document (e.g. COC.md, you can find a good standard template here: https://www.contributor-covenant.org/). I recommend you link to the contributing guidelines in the readme and to the code of conduct in the contributing guidelines.

See also these resources:
https://help.github.com/articles/setting-guidelines-for-repository-contributors/
https://opensource.guide/code-of-conduct/

Examples:
https://github.com/atom/atom/blob/master/CONTRIBUTING.md
https://github.com/rails/rails/blob/master/CONTRIBUTING.md

@jshermeyer
Here are my initial comments on the paper.

Paper

General comments:

• I think it would be better to intro something like ‘user defined polygon identifying a region of interest (ROI)’ and use ROI for the remainder of the paper.
• Q: Any limits to ‘user defined area of interest’?
• Figures: please provide descriptive captions to the three figures.

Specific correction suggestions:

• I feel like the paper title could be smooth like this: 'Comet Time Series Visualizer: CometTS'
• that we call -> called
• (CometTS) -> (CometTS)
• a times series of satellite imagery -> time series satellite imagery.
• -> The tool aims to enable research into population estimation, land use detection, or natural disaster monitoring using a variety of data types.
• series of satellite imagery -> series of satellite images
• variation), -> variation)
• visualizes -> provides a visualization of
• from user drawn polygons. -> from a user defined region of interest.
• define GIS as geographic information systems
• can only be used to analyze individual -> are limited to analysis of individual
• CometTS then produces -> CometTS produces
• within the polygon -> region of interest
• of most relevance -> most relevant
• standard statistics -> standard statistical measures / standard test statistics

@Kevin-Mattheus-Moerman
Added a CONTRIBUTING.MD. Code of conduct seems overkill, if I need to post a code of conduct calling for basic civility in my little codebase I think the world is doomed.

@zhampel
Thanks for the notes. Will make some edits when another review comes in.

I think it would be better to intro something like ‘user defined polygon identifying a region of interest (ROI)’ and use ROI for the remainder of the paper.

Agreed, will make this change then when other reviews come back.

Q: Any limits to ‘user defined area of interest’?

Just computational strength, the larger the polygon the more slowly the processing. I believe CometTS will not like polygons that are larger than the imagery you provide, but it can tolerate and ignore nodata/ blank space/ masked areas.

Figures: please provide descriptive captions to the three figures.
Good idea.

@zhampel, @dmittman, can you provide an update on where we are in the review process? @rhiever when can we expect your contribution? Thanks!

Apologies for the delay @Kevin-Mattheus-Moerman. I've reviewed the reviewer guidelines, checked the conflict of interest statement and took a look through the submitted PDF and repo. I find the material just outside my area of expertise, being more about satellite imagery analysis with a time-series component than pure time-series visualization. Perhaps @lewismc (https://github.com/lewismc) might be a more appropriate reviewer than I.

@dmittman Thanks for your reply. I understand. Thanks for taking the time to check out this project, and for recommending another reviewer.
@lewismc would you be interested in reviewing this work for JOSS?

@Kevin-Mattheus-Moerman yes I would. Please give me a deadline for reviewing. I've just returned from some vacation.

@whedon add @lewismc as reviewer

@jshermeyer can you reply to @zhampel 's comments?

@rhiever, @lewismc can you provide an update on the review process? Let me know if you need help. Also please let us know in case you are no longer able to assist in this review. Thanks.

@Kevin-Mattheus-Moerman Thanks for staying on this. I'll do a full reply to all comments once all parties are done reviewing.

@rhiever has indicated he is no longer able to review this work

@whedon remove @rhiever as reviewer

OK, @rhiever is no longer a reviewer

@lewismc please let me know if you are able to proceed with this review. We could really use your help. Thanks.

@lewismc can you give an update on this review? Are you still able to help us? Thanks!

@lewismc Please can you inform us if you are still able to do this review. Thanks

I've e-mailed @lewismc to check if he is still able to review. Apologies for the delay encountered so far.

@jshermeyer although I will continue to try to contact @lewismc I suggest you start addressing comments by @zhampel to avoid further delays.

@jshermeyer in particular can you comment on whether you agree with @zhampel to migrate "... the functionality of your python notebooks to a more API-like structure..."?

@jshermeyer ☝️

👋 It looks like we're waiting here for the author, @jshermeyer, to come back with responses to how they are addressing the reviewer comments.

It also looks like the second reviewer, @lewismc, has gone MIA (unresponsive to multiple mentions here, plus emails from the editor). I suggest we proceed to a recommendation from the first reviewer and make the acceptance decision at that point.

@jshermeyer here are my comments regarding the structure of CometTS's code. I found your instructions easy to follow with the tools you have built. Your code does work as advertised, though I do think there are a changes that should be made to the formatting of the code, including commenting and other coding standards that should be upheld. Furthermore, I think your work could be made much stronger as an extensible project, by migrating the functionality of your python notebooks to a more API-like structure. I would also like to point out that a significant hurdle to properly reviewing the project is the lack of easy access to example datasets (you do point this out), which is beyond the scope of you as the author. To this end, the open source community encourages steps like CometTS to making these tools openly available indeed.

Thanks to Zig @zhampel for his helpful and thoughtful comments. Also thanks to the editors @Kevin-Mattheus-Moerman @labarba for pushing this on, I apologize for being an absentee on this. I've gradually added onto Comet to address all of his concerns.

A brief overview:
-Code totally reformatted to meet pep8 standards and reorganized for simplicity
-Now has a command line API for those who want to ignore the notebooks (would still recommend notebooks for easy plotting/viz).
-Installable via pip/conda
-Dockerized
-Pytests included for sensitive functions
-Added support for autoregressive integrated moving average modeling for future trend forecasting
-Includes an example test dataset of NPP VIIRS monthly composites over San Juan, PR.

Specifics:

MacOS fails to pip install due to gdal. Needs brew install gdal prior to pip install -r Requirements.txt. Works on my MacOS Mojave laptop.

Updated to have gdal install at the end. Also include support with docker and a conda environment for mac users. Would not recommend brew installing gdal, things get ugly fast when using brew/pip/conda.

DownThemAll add-on link points to a non-existent page.

Still works for old firefox, unfortunately isn't supported any more. I'll leave it up to an end user to get their own data. I now include sample VIIRS imagery for play.

Ensuring a proper directory structure is clear for the example you gave, but is that the case with other datasets? Not being a geospatial data expert, is there a resource that provides guidance on proper structuring, or perhaps a universal industry standard?

This is a standard for a few time-series methods, i.e. CCDC/YATSM- https://github.com/ceholden/yatsm

Had difficulty with GDAL install (who doesn’t?). Perhaps best to include gdal and subsequent deps on gdal at the end of the requirements list, so that all other packages install first. This may assist in debugging install.

Done.

Also recommend including link to building GDAL with python-bindings in Installation section. Had success using following: http://trac.osgeo.org/gdal/wiki/BuildingOnUnix

Multiple choices for installing GDAL now available.

Please include a python style .gitignore file that excludes the CometTS python notebooks.

Notebooks are pretty core for this project, but project is now pip/conda installable with an option for CLI only so this seems redundant.

Code Specifics

Recommend reformatting for python3 compatibility

Done.

Recommended to import functions explicitly, instead of * for ease of tracking

Done.

Header description for each fn/ Inputs and outputs description

Now have comments that describe inputs/outputs + github readme

All input argument declarations should be initialized to None, ‘’, etc

Some of these I like to keep to give a user what an example input would look like.

Run flake8 to format code properly, will make it easier to read too ;)

Done

Multi_studyAreas_FullStats.csv is in same dir as notebooks. Plot_Results.ipynb looks elsewhere for it. Consider moving it into the example data dir. Yet, when trying to use this example csv data, I get an error that there is no key count. Did work on my generated csv data though.

Code structure has been totally reorganized

For plotting, should have more scalable y-axis, as I chose a rather small city (Santa Fe, NM) to test

I've included a few helper notebooks for plotting only. A user will have to have a bit of background with matplotlib for tweaking some of the plotting features.

Difficult to debug errors, esp wrt formatting data issues

Not really sure how to make this easier, probably will require more iterations. Have added pytests to hopefully alleviate some stress.

What are the temp*.vrt files? Are they part of the project?

Temp outputs from some functions, can be ignored. I likely should clean these up but could be helpful for debug.

Need a set of tests for function verifications

Included for a good chunk of the code that could be sensitive. Also include test data.

@labarba thanks for helping here. In relation to:

It also looks like the second reviewer, @lewismc, has gone MIA (unresponsive to multiple mentions here, plus emails from the editor). I suggest we proceed to a recommendation from the first reviewer and make the acceptance decision at that point.

Agreed, however, although I do not feel there is evidence for a clear conflict of interest here (in fact the reviewer has been very thorough and detailed requirements), the reviewer @zhampel did point out to me that "...the author, Jake Shermeyer, is a member of a sister lab of my organization.". Hence I would prefer to find a replacement reviewer.

@chrismattmann @dmittman I believe you pointed out you might be able to review this work over at #866. Would greatly appreciate if you would be able to help (save the day! 🚀 ) with this review? The review has already started but we now need at least one additional/replacement reviewer.

@jshermeyer can you check if you need to update your paper to reflect all the changes you've made? Please run @whedon generate pdf here to regenerate your paper here.

No changes required.

@chrismattmann @dmittman 👋 do you think you can help review this work?
@danielskatz do you know these potential reviewers personally? We've had a reviewer drop out so we need a replacement. Thanks.

I sent an email in case they don't see the github notification, but it's also just 4:30 am in California ...

I find the material just outside my area of expertise, being more about satellite imagery analysis with a time-series component than pure time-series visualization.

And @chrismattmann replied to me by email that he doesn't have time currently

@turmon, @darth-pr - would either of you be able to step in and help with a review of this submission & this software for JOSS? (Or perhaps suggest someone else who might?)

See JOSS review criteria in case you are not aware of how JOSS works...