Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @dghoshal-lbl, @uellue it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

Reference check summary:

OK DOIs

- 10.1063/1.4812323 is OK
- 10.1007/s11837-013-0755-4 is OK
- 10.1016/j.commatsci.2012.02.005 is OK
- 10.1088/2515-7639/ab13bb is OK
- 10.1109/MCSE.2012.16 is OK
- 10.1002/cpe.3505 is OK
- 10.1016/j.commatsci.2015.09.013 is OK
- 10.21105/joss.01844 is OK
- 10.21105/joss.01844 is OK

MISSING DOIs

- None

INVALID DOIs

- None

👉 Check article proof 📄 👈

👋 @dghoshal-lbl & @uellue - thanks for agreeing to review this submission. Please be sure to read the comments above, and let me know if you have any questions. Basically, your job is to check the article proof and repository and check items off your checklist above. (And note that you will have to accept the invitation before you will be able to do so.)

If you see small problems that need to be discussed, feel free to discuss them here. But if you can, create a new issue in the target repository and link to this review thread in that issue to create a corresponding breadcrumb trail here.

I look forward to seeing how this review goes

Regarding "State of the field":

Comparison with Pika

After a quick side-by-side comparison between the RabbitMQ documentation and kiwiPy, it is not clear to me what the difference between the default Pika Python API and the kiwiPy API is. The code of the Pika-based RabbitMQ examples and the kiwiPy examles are quite similar, and at least to my eyes the kiwiPy version is not a noticeable improvement, different from what is claimed in the paper.

(edit: source code formatting)

#!/usr/bin/env python
import time

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='hello')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    time.sleep(body.count(b'.'))
    print(" [x] Done")


channel.basic_consume(
    queue='hello', on_message_callback=callback, auto_ack=True)

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

# -*- coding: utf-8 -*-
import time
import threading

import kiwipy

print(' [*] Waiting for messages. To exit press CTRL+C')


def callback(_comm, task):
    print((' [x] Received %r' % task))
    time.sleep(task.count('.'))
    print(' [x] Done')
    return task


with kiwipy.connect('amqp://127.0.0.1/') as communicator:
    communicator.add_task_subscriber(callback)
    try:
        threading.Event().wait()
    except KeyboardInterrupt:
        pass

Comparison with other work distribution and messaging systems

A quick list of major distributed work scheduling systems with Python API that might deserve a comparison:

• Dask futures https://docs.dask.org/en/latest/futures.html
• MPI
• ZeroMQ
• An endless list of distributed task queue systems: https://www.fullstackpython.com/task-queues.html
• An endless list of Python-based RPC systems: https://stackoverflow.com/questions/1879971/what-is-the-current-choice-for-doing-rpc-in-python

The paper and documentation should make much clearer what makes kiwiPy unique and for what particular use cases it is advantageous. Perhaps it can also reference the appropriate parts of RabbitMQ documentation since it is essentially a Python API for it, as far as I understood.

Regarding "References": A reference to the default "Pika" Python API for RabbitMQ https://pika.readthedocs.io/en/stable/ seems to be missing.

Regarding "Community guidelines": aiidateam/kiwipy#65

@danielskatz @muhrin This concludes my first round of review. Looking forward to your responses!

@uellue, many thanks for the helpful review. I'll coordinate with my coauthor and address the points you've raised.

👋 @muhrin - any update at this point?

We've addressed @uellue issue regarding contributor guidelines by adding wiki entry with is now linked to in the docs.
I've yet to find the time to address the other points raise but should find the time in the coming days. I'll report back as soon as I can.

Dear @danielskatz,

we once again thank @uellue for his thorough review which has helped to improve both the documentation and the paper itself. We have made the following changes to address the comments made:

• aiidateam/kiwipy@26b561a
  We have added a contributing guide to our wiki detailing how contributions to kiwiPy can be made (this is linked to from the documentation), thus closing aiidateam/kiwipy#65.

• aiidateam/kiwipy@56a6050
  A direct comparison to pika (the library used for tutorial on the RabbitMQ website) has been added to the documentation that both highlights the differences in in syntax and acts as a guide for transitioning to kiwiPy from pika. We believe this highlights that, for all but the most simple of cases, kiwiPy has more compact and pythonic syntax (e.g. contexts for resource lifetime scoping, use of futures), but perhaps more significantly we show that kiwiPy has a higher-level focus compared to pika making it more accessible to users not familiar with channels, exchanges, routing, etc.

• aiidateam/kiwipy@e10464d
  We recognise that we did not sufficiently emphasise the major benefit and differentiating factor of kiwiPy as compared to some existing libraries as highlighted in the review (Dask, ZMQ, MPI). Namely, that kiwiPy (via RabbitMQ) automatically persists messages (e.g. tasks) to disk allowing us to build loosely coupled systems where it is not expected that all clients (or even the broker) will be online at all times allowing long-running workflows (taking weeks or months) to complete as and when resources are available. This has been highlighted in the paper and a comparison has been added to a number of existing libraries.

This concludes our changes in response to the review.

We look forward to the continuation of the review process.

Dear @muhrin,

many thanks for the update! The new content is a great help for readers to understand the use cases and strengths of kiwiPy. The comparison with Pika and other systems sounds fair and objective. 👍 This fully addresses my previous comments and from my perspective the paper is good to go. 🚀 Congratulations!

CC @danielskatz

Thanks @uellue . Indeed, the library has certainly benefited from these additions. Thanks again for your review.

Happy to hear that you found it helpful! :-)

👋 @dghoshal-lbl - how is your review coming along?

I finished my first round of review for the software and the paper. Some of the paper related concerns have already been addressed. I have a couple of more questions regarding the paper:

i) Has the library been tested to show any significant performance overheads? Mainly because this adds another level of abstraction to RabbitMQ.

ii) Since one of the use-cases for kiwiPy is to restart partially completed workflows and HPC jobs, I was wondering if there are any specific configurations in kiwiPy that enables this or if this is a default behavior?

Regarding the software, I was able to install it but not test its functionality properly:

i) I could not find any module or scripts that would let me test the library.

ii) When I wrote my own client and server following the examples, I ran into the following error:
ImportError: cannot import name 'AsyncContextManager'

I also see that there are a few examples in the source tree. I can dig deeper into why I am getting this error, but I think it will be good to document the steps to quickly test the examples or provide scripts for validating the tests.

👋 @muhrin - I think we're waiting for your response to the issues @dghoshal-lbl has raised...