Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

basic parsl tutorial broken with 0.8.0a #1018

Open
danielskatz opened this issue Jun 5, 2019 · 2 comments

Comments

Projects
None yet
1 participant
@danielskatz
Copy link
Collaborator

commented Jun 5, 2019

With Parsl 0.8.0a, the tutorial fails in the Parallel Dataflow section (on both Ubuntu and MacOS):

@bash_app
def generate(outputs=[]):
    return "echo $(( RANDOM )) &> {outputs[0]}"

# App that concatenates input files into a single output file
@bash_app
def concat(inputs=[], outputs=[], stdout="stdout.txt", stderr='stderr.txt'):
    return "cat {0} > {1}".format(" ".join(inputs), outputs[0])

# App that calculates the sum of values in a list of input files
@python_app
def total(inputs=[]):
    total = 0
    with open(inputs[0], 'r') as f:
        for l in f:
            total += int(l)
    return total

# Create 5 files with random numbers
output_files = []
for i in range (5):
     output_files.append(generate(outputs=[os.path.join(os.getcwd(), 'random-{}.txt'.format(i))]))

# Concatenate the files into a single file
cc = concat(inputs=[i.outputs[0] for i in output_files], outputs=[os.path.join(os.getcwd(), 'all.txt')])

# Calculate the sum of the random numbers
total = total(inputs=[cc.outputs[0]])
print (total.result())

The error is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/anaconda/lib/python3.6/site-packages/parsl/dataflow/dflow.py in sanitize_and_wrap(self, task_id, args, kwargs)
    579                     try:
--> 580                         new_inputs.extend([dep.result()])
    581                     except Exception as e:

/anaconda/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    424             elif self._state == FINISHED:
--> 425                 return self.__get_result()
    426 

/anaconda/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    383         if self._exception:
--> 384             raise self._exception
    385         else:

/anaconda/lib/python3.6/site-packages/parsl/dataflow/futures.py in parent_callback(self, executor_fu)
    119                 if isinstance(res, RemoteExceptionWrapper):
--> 120                     res.reraise()
    121                 super().set_result(executor_fu.result())

/anaconda/lib/python3.6/site-packages/parsl/app/errors.py in reraise(self)
    162 
--> 163         reraise(t, v, tb)
    164 

/anaconda/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
--> 692                 raise value.with_traceback(tb)
    693             raise value

/anaconda/lib/python3.6/site-packages/parsl/app/errors.py in wrapper()
    171         try:
--> 172             return func(*args, **kwargs)
    173         except Exception:

/anaconda/lib/python3.6/site-packages/parsl/app/bash.py in remote_side_bash_executor()
     49         logging.error("Caught exception during formatting of app '{}': {}".format(func_name, e))
---> 50         raise e
     51 

/anaconda/lib/python3.6/site-packages/parsl/app/bash.py in remote_side_bash_executor()
     35         # Execute the func to get the commandline
---> 36         partial_cmdline = func(*args, **kwargs)
     37         # Reformat the commandline with current args and kwargs

<ipython-input-15-c2623b1950fb> in concat()
      8 def concat(inputs=[], outputs=[], stdout="stdout.txt", stderr='stderr.txt'):
----> 9     return "cat {0} > {1}".format(" ".join(inputs), outputs[0])
     10 

TypeError: sequence item 0: expected str instance, File found

During handling of the above exception, another exception occurred:

DependencyError                           Traceback (most recent call last)
<ipython-input-15-c2623b1950fb> in <module>()
     28 # Calculate the sum of the random numbers
     29 total = total(inputs=[cc.outputs[0]])
---> 30 print (total.result())

/anaconda/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/anaconda/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

/anaconda/lib/python3.6/site-packages/parsl/dataflow/futures.py in parent_callback(self, executor_fu)
    116 
    117             try:
--> 118                 res = executor_fu.result()
    119                 if isinstance(res, RemoteExceptionWrapper):
    120                     res.reraise()

/anaconda/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    423                 raise CancelledError()
    424             elif self._state == FINISHED:
--> 425                 return self.__get_result()
    426 
    427             self._condition.wait(timeout)

/anaconda/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

DependencyError: [23] Dependency failure from: [TypeError('sequence item 0: expected str instance, File found',)]```
@danielskatz

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 5, 2019

@benclifford suggests:

a quick ugly fix might be to replace
return "cat {0} > {1}".format(" ".join(inputs), outputs[0])
with
return "cat {0} > {1}".format(" ".join(list(map(str,inputs))), outputs[0])
ugly though.

@danielskatz

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 5, 2019

I can confirm this fix works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.