Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @Chilipp, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐️ Important ⭐️

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

Attempting PDF compilation. Reticulating splines etc...

👉 Check article proof 📄 👈

@Chilipp, @taqtiqa-mark - thanks for agreeing to review. (Note that @Chilipp will not be able to start for a week.)

If you have any questions, please ask. Otherwise, I'll expect that you will work through the checklists above and raise issues as you find items you cannot check off.

👋 @Chilipp, @taqtiqa-mark - I hope you can start reviewing this soon - at least, the code of conduct and conflict of interest boxes for each of you...

Yes, @danielskatz, I will start tomorrow. My apologies for the delay

👋 @taqtiqa-mark - are you able to get started too?

Hello @justinalsing! I am so far done with my first look into your software and I would like to have your feedback on some of the following points:

General checks

• Version: I could not find any version associated to your repository, is it written somewhere in your code?

Functionality

• Installation: I could not install it with the provided instructions (see
  justinalsing/dlmmc#3)

Documentation

• A statement of need: I understand that DLMs are great and useful to climate and especially atmospheric science. But I do not understand what the scope of your software is. Is it only that you provide the suite of four models? Since your software has a quite general name I wonder whether you will extend them in the future and aim for a large suite of DLMs to which others can contribute as well?
• Installation instructions: see
  justinalsing/dlmmc#3. Additionally I have the impression that you need to have a certain background in python and jupyter to be able to run through your tutorial. I strongly recommend to provide some statements about the previous knowledge you expect from the user of your software.
• Functionality documentation: Again, what are you aiming for? You provide instructions for how to use your models but if you aim for providing the possibility in extending the models, you should also give guidelines on how to extend them
• Automated tests: I could not find any. Could you please tell, how you test the functionality?
• Community guidelines: You have a short section in your README (https://github.com/justinalsing/dlmmc/blob/master/README.md#contributions-reporting-issues-and-feature-requests) but I strongly recomment to provide a CONTRIBUTING file (see https://help.github.com/articles/setting-guidelines-for-repository-contributors/). However this is not a formal requirement of JOSS as far as I know. Nevertheless, to my impression your are lacking clear guidelines on how to contribute and how to seek support

Software paper

• A statement of need: As noted above, is your aim to distribute the four models or to provide a framework that can serve as a suite for more DLMs?
• References: For the users of your package, please correct the citations in the model_descriptions.pdf. They are lacking, e.g. the full list of authors and DOIs.

👋 @justinalsing - note that we're waiting for your inputs here

Hi @CHILLIP, @danielskatz, @taqtiqa-mark

Thanks for all the feedback and sorry for the delay. Let me try to address your points:

Version thanks for pointing this out! I’ve just made a release, v1.0

Installation I saw you had problems installing with pip. The dependency that typically causes problems (as in your case) is pyston, since it needs to play nicely with the C++ compilers on your machine. For some it has worked with pip, for others it seems to work with conda but not pip. I’ve added a note to the README that says you can use “conda install (recommended) or pip” and another note, “Note: conda seems to be more stable for getting pystan to work than pip - if you installed with pip and compiling the stan models throws errors, try again with with conda.” Do you have any other suggestions for making the install instructions more solid?

I’ve also added a point in the README that prior knowledge of python/jupyter is needed to be able to fully follow the tutorial, as suggested. Agree this was a good idea.

Statement of need yes, I intend that the HMC-Kalman filtering framework can be used to expand the suite of models beyond those initially provided. Essentially the space of DLM models are characterized by a few model matrices; more-or-less all that needs modifying in the stan model code to implement a new one is to replace the model matrices and add any additional hyper-parameters. I’ve now provided a CONTRIBUTING file as you suggested giving details of how to contribute extended models to the suite. It does assume familiarity with stan, which I think is needed to contribute meaningfully. I’d be happy to give further details here if you feel they are needed and am very open to suggestions on how to improve this.

Automated tests In my mind the tutorial also serves this purpose: if the models compile and the tutorial runs through without error, you’re in good shape. I’ve added a note to the README that the tutorial also serves this purpose. However, if you feel additional tests are required I am open to suggestions..

Community guidelines see above re added CONTRIBUTING file.

References I’ve updated the references in the model_descriptions.pdf as suggested (thanks for pointing those out)

I hope this addresses your points and look forward to hearing your further thoughts



Thanks again!

My apologies for the delay. It took a while to find my first notes. which follow:

I've had a quick look at the model PDF and some of the repository and can say, while its fresh in my mind:

The use-case appears to be reasonably well motivated - implementing a known/published model, and a non-trivial software stack (e.g. parallel libraries).

My working assumption is the question of originality is out-of-scope - it suffices that the library implements a model already published. Consequently, while non-linear AR/(G)ARCH models are wide spread, I haven't investigated the question of whether this improves on them. However I would suggest that data sets used thought to have those properties from the R-project, would be used in the test suite.

Given the JOSS is software focused I'm placing all emphasis on the software engineering aspects, and almost non on the contribution to domain knowledge.

That non-trivial software stack nature suggests weight be given to the following observations:

• No test suite (some examples, but not formulated in a way that breaking changes would be flagged). I would expect tests using known null models, known data sets and results from a reference implementation, e.g. from R data sets package. I would also expect tests using data simulated from an alternate model - this would at least confirm the model recovers the true parameters (within a margin), and flag when anything breaks.

• No software documentation generated (that I can see). I would expect this to be created as part of the CI build.

• In general I expect a CI build for such opensource projects (CI services are free and ubiquitous) - this allows for quick validation of what works, as well as running the test suite and would generate up to date documentation.

• Vague instructions regarding installing parallel run requirements (unless pip installed are guaranteed to Just-Work on all platforms?). My bitter experience is that such general statements are used because it is so difficult to document the precise setup that works. @Chilipp's observations confirm this. Since parallel execution is an advertised feature I would expect a parallel test suite, and in the read-me an example of the speedup obtained using some known available data set. It is conventional to be explicit about which platforms the software is known to work on, and which platforms are supported - using CI builds would be one path to verify this, and would allow the author to work out exactly which version combinations work.

• cached pyc folders suggest pain lies ahead.

I'll update this with links to the issues opened against the upstream project, as I open them.

@danielskatz, for me blocking issues are:

1. Automated test suite, that at a minimum:

• Fits null and alternate specifications to simulated and empirical data. The tests would be simply checking parameters vales, and goodness of fit metrics. The 'true' or expected values should come from a reference implementation that is clearly identified, e.g. some R-project, etc. library result set, and the script used to generate the reference values would be part of the project.
• Parallel test suite that exercises the parallel calculations. This would likely be slow (i.e. while processing large data or models is faster than serial runs the parallel run would be expected to be 'slow' in calendar time - unit test times are generally aimed to be in the sub-seconds range) so would likely have its own test harness to prevent the TDD/BDD cycle from being slowed down.

2. Installation instructions for the platforms that are supported. My assumption is it is not the purpose of the reviewers to resolve all these issues across the various platforms? The warning note (should be more prominent) about conda instability and switching between pip/conda is really something that the package should resolve and provide the user with a painless path?
3. Documentation. Each function a user is expected to interact with should be documented. Ack. that many functions are likely to be upstream, but if the user is expected to interact with them, the package document should, at a minimum, provide links to the upstream documentation of the version in use.

Thanks @taqtiqa-mark - these details are very helpful.

In JOSS, we don't really have results like

Right now I'd be inclined to conclude: Major Revisions, resubmit.

We just say what's needed to be done, and then let the author decide to do this or not. In other words, we keep going until the submission is acceptable or we decide that it cannot be made so.

Hi @taqtiqa-mark ,

Thanks for your feedback.

I think it would help it I reiterate/clarify the scope of this code. The code is a suite of statistical models, written in stan, than can then be ran on user’s data using the pystan interface (provided by the stan team).

Along with the models, I provide a tutorial covering how to run the stan models using pystan — this is really to ease the way for users who are not familiar with pystan. For users who want to run many DLMs quickly but are not used to setting up MPI runs with python, I also provide a template for how to run models in parallel this way. The tutorial and MPI-run template are really examples of how to use the code to help users get up and running quickly, rather than central to the package. That is why, for example, I do not attempt to give openmpi install instructions that will work robustly for everyone — as you point out that is a difficult thing to do in practice.

Since the package is a suite of models implemented in a probabilistic-programming language, the model code “in of itself” does not have any “functionality” to be documented, ie., the model codes in stan_dlm_models.py are not a set of functions that can be called by the user. Rather, they are specifications of probabilistic models that are then passed to the pystan interface, which then runs them on user input data. I am intentionally relying on the pystan interface to run the models, since it already exists and is well documented. I could have implemented a higher level interface so that users would not have to interact with pystan at all, but this seemed unnecessary and would make maintaining the code more cumbersome.

There is a question of whether a suite of stan models constitutes a valid open source project. I believe it does for the following reasons:
— The (stratospheric science) community has explicitly asked for this, since the expertise/time to implement such models within the community is scarce, and the momentum to replace the industry-standard MLR with DLM is rapidly growing. The code is already being used by members of the community.
— The code is extendable in a meaningful way: people can implement new models based on the dlmmc code with relatively little effort, since it essentially just involves modifying the lines of code that specify the model matrices that define the space of DLM models. The bulk of the work (setting up the Kalman-filtering and smoothing, etc) has been coded up in a general-purpose way so does not need modifying, massively reducing the barrier to entry for people to implement new models.

I hope this focuses the discussion on what I should do to make this code ready for public consumption.

@taqtiqa-mark and @CHILLIP, you both raised automated tests and more robust install instructions. If you have some concrete ideas/specifications for what you think is needed here I’d greatly appreciate it.

The other related point you raised @taqtiqa-mark is validation. Validation of Bayesian codes is non-trivial, requiring either comparison to a known analytical posterior (hard in this case), comparing to another code, or running on a large ensemble of mock data vectors and looking at the statistical properties of the results (expensive). If you really feel this is a requirement I’d love to hear what you think would be most worthwhile here.

@danielskatz,

I've amended my comment o remove that observation.

@justinalsing makes the following observation:

There is a question of whether a suite of stan models constitutes a valid open source project

My assumption is that this question has been resolved in the affirmative by the (sub-)editor(s) - otherwise this would not have progressed to the review stage.
Please indicate if that assumption is correct.

If incorrect, are there JOSS guidelines to on what is a "valid open source project" (for the purposes of JOSS) to guide reviewers to address this issue in a way that results in consistent outcomes?

I'll address @justinalsing response separately - by editing my comments above to add links to issue in his repository - to prevent this thread from becoming spaghetti.

There is a question of whether a suite of stan models constitutes a valid open source project

My assumption is that this question has been resolved in the affirmative by the (sub-)editor(s) - otherwise this would not have progressed to the review stage.
Please indicate if that assumption is correct.

Yes, this is correct.

Also, @taqtiqa-mark - I think that rather than editing comments to address responses, it's better to just write a new comment, and, as you've said, to open issues in the source repository with links to them here. And so, to get an up-to-date status, can you (and also @Chilipp) please add new comments below that tell us what your current concerns are (linking to issues as needed)?

thanks @justinalsing for your detailed responses to my comments and the ones from @taqtiqa-mark ! I especially appreciate your work on the contribution guidelines.

There are still some open points and I would like to outsource them to other issues in your repository:

• Automated tests (justinalsing/dlmmc#5): As also stated by @taqtiqa-mark that they need to be implemented
• installation (justinalsing/dlmmc#3): Your installation instructions are still a bit vague. As a user, I do not want to test whether it works with pip or not, I simply want to install it and it should work
• references: thanks for implementing the detailed references. As noted by your colleague @pankajkarman (justinalsing/dlmmc#4), they have to be reviewed.

If you work on these issues, please directly comment in the issues in your repository as this is also recommended in the review process guidelines above

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Re: automated tests - The JOSS requirement is just "Are there automated tests or manual steps described so that the function of the software can be verified?" While automated tests are generally better for the software, manual steps are sufficient for JOSS acceptance. Specifically, adding good automated tests can be a longer-term activity than JOSS acceptance.