Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more

/parsl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically transfer task 'stdout' and 'stderr' #363

Open
annawoodard opened this issue Jul 1, 2018 · 4 comments
Labels
file-management

Comments

@annawoodard
Copy link
Collaborator

@annawoodard annawoodard commented Jul 1, 2018

We allow a user to pass stdout and stderr keywords to bash apps, but these have no effect if the app is run remotely. We should automatically transfer these files back. Related to #276 and #361.
@annawoodard

This comment has been minimized.

Sign in to view
Copy link
Collaborator Author

@annawoodard annawoodard commented Jul 23, 2018

This issue depends on #219
@annawoodard annawoodard changed the title Automatically transfer 'stdout' and 'stderr' Automatically transfer task 'stdout' and 'stderr' Aug 8, 2018
@kylechard

This comment has been minimized.

Sign in to view
Copy link
Collaborator

@kylechard kylechard commented Aug 8, 2018

Or support streaming of stdout/err. This would be useful for development/testing. E.g.,

Example from Francois:

@bash_app def run_process(job_dir, job_script, stdout=sys.stdout, stderr=sys.stderr): return f"cd {job_dir} && bash {job_script}
@annawoodard annawoodard mentioned this issue Aug 8, 2018
Open
@benclifford benclifford self-assigned this Jan 31, 2019
@benclifford

This comment has been minimized.

Sign in to view
Copy link
Contributor

@benclifford benclifford commented Aug 12, 2019

The test suite contains at least one test for stdout/stderr which makes assumptions about std file appending/location: test_stdout_append in parsl/tests/test_bash_apps/test_stdout.py.

This test is likely to not work in the presence of any kind of non-shared filesystem, which this issue is related to.
@benclifford

This comment has been minimized.

Sign in to view
Copy link
Contributor

@benclifford benclifford commented Aug 14, 2019

I think staging of these parameters should be made to be the same as for any other I put and output file, using a File object. Things like streaming should be implemented as regular staging mechanisms that can work for all files.
@benclifford benclifford removed their assignment Aug 14, 2019
This was referenced Aug 15, 2019
Closed
Open
benclifford added a commit that referenced this issue Oct 21, 2019
@benclifford
Add test mark for tests affected by #363
Loading status checks…
b9124cf 
Tests which inspect stdout/stderr cannot always expect
to find the stdout/stderr files on the submitting file
system. That is issue #363

This commit adds @pytest.mark.issue363, so that
pytest can be invoked with parameter:

   -k 'not issue363'

which skips tests affected in that way.

I have tested this with the AWS provider (as part of a
set of various other AWS test fixes and local configs).
@benclifford benclifford mentioned this issue Oct 21, 2019
Merged
yadudoc added a commit that referenced this issue Oct 24, 2019
@benclifford @yadudoc
Add test mark for tests affected by #363 (#1377)
7a9346d 
Tests which inspect stdout/stderr cannot always expect
to find the stdout/stderr files on the submitting file
system. That is issue #363

This commit adds @pytest.mark.issue363, so that
pytest can be invoked with parameter:

   -k 'not issue363'

which skips tests affected in that way.

I have tested this with the AWS provider (as part of a
set of various other AWS test fixes and local configs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
file-management
Projects
None yet
Milestone
No milestone
3 participants
@annawoodard @kylechard @benclifford
You can’t perform that action at this time.