Tree: 37c17b5ae3
-
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Make `connected_workers` consistent and add `connected_managers` (#902)
* Make `connected_workers` consistent Now the `connected_workers` method returns the number of connected workers. Fixes #656. * Add 'connected_managers' method
-
Tidy debug messages around heartbeat/task requests (#899)
benclifford committedApr 22, 2019 Issue #898 shows a confusing log message, where a heartbeat incorrectly logs that HEARTBEAT_CODE tasks have been requested, in addition to logging a heartbeat. This commit moves that log into the non-heartbeat path only, so that exactly one of the heartbeat or task request messages will be logged. The phrasing of the heartbeat log message is modified to be in the same sense as the task request message.
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Fix state machine in local provider (#896)
Previously, no state transition happened for completed processes: any process labelled as RUNNING would not trigger any of the state transitions. This commit removes the incorrect non-RUNNING restriction on transitions to COMPLETED or FAILED. This commit also extends the FAILED transition to happen with any non-zero return code, not just negative ones: this means that a FAILED transition should happen when a process exits with a failing return code. This commit also adds in a error statement when none of these state transitions fire. Prior to this commit, some failures to launch would manifest as hangs waiting for the launched code to start, as the failure would be ignored. After this commit, those failures to launch result in a COMPLETED or FAILED state, and so for example the HighThroughputExecutor loops attempting to start new workers repeatedly - still a hang, but a hopefully more debuggable hang.
-
Managers to report block_id on registration (#888)
Support for identifying managers by block_id, and putting managers in a block on hold Managers are given a block identifier at initialization that is reported back at registration. This allows the executor to track tasks/worker slots at the block level. The executor also has a _hold_block() function that puts all managers in a block on hold preventing further task scheduling, and making it safe to terminate the block.
-
Merge pull request #893 from Parsl/benc-post-892-docs
annawoodard committedApr 19, 2019 Update docs to reflect PR #892 import changes
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
-
Merge pull request #892 from Parsl/benc-more-base-imports
annawoodard committedApr 19, 2019 Expose newish app decorators through top level module
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
-
split top level parsl module export into multiple lines
benclifford committedApr 19, 2019 -
Unify callback paths for execution and memo-based results (#865)
benclifford committedApr 19, 2019 * Make memo-based callback look like execution-based callback This makes the memo-based callback happen via the (memo) Future, in the same way that the execution-based callback happens via an execution Future. This weakens the requirement that memo_fu have a result already at the point that this piece of code is reached: if it doesn't, the callback will be deferred until it does. In practice, this should not change user-exposed memo behavior - instead this is to simplify code. This is a step towards making memo- and execution-based completion callbacks be handled in entirely the same way, as part of pulling out pieces of PR #722. * Combine two exec callbacks into one Previously the exec_done callback was added in two places in launch_tasks, depending on whether the execution path or the memoization path was taken. This commit moves that callback addition to outside of launch_tasks, where we do not need to care where the launched task future has come from.
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
-
Merge pull request #887 from Parsl/monitoring-viz-recovery
annawoodard committedApr 16, 2019 Monitoring viz recovery
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Revert "Merge branch 'master' into monitoring-viz"
This reverts commit 741363b, reversing changes made to 0eb1ea2. The purpose of this merge and revert is to recover the commits made by @garri1105. These were unintentionally lost when his work was copied into a dedicated repository for the visualization tool.
-
Merge branch 'master' into monitoring-viz
annawoodard committedApr 16, 2019 -
Remove indirection of staging via File objects (#884)
benclifford committedApr 16, 2019 Prior to this, file objects pretty much just forwarded on staging requests to the data manager without doing anything interesting. This removes that indirection. It removes some of the need for a file to be associated with a data manager - something which sits awkwardly with wanting to reuse Files between DFKs, for example in interactive notebook use. Instead, each DFK has a reference to a data manager directly.
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Remove methods that should come from Future superclass (#879)
benclifford committedApr 16, 2019 All three methods (done(), exception() and add_callback()) will behave more consistently if using the Future superclass implementations rather than attempting to piggyback onto the parent implementations and behaving wrongly when there is no parent future.
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Add support for configuring prefetch capacity (#881)
* Add support for configuring prefetch capacity This is a squashed cherry pick with a few changes removed in order to just collect the changes related to prefetch capacity. It is based on the following commits: 75f9d4e 5a221f2 c89f3b2 * Set prefetch_capacity default to 0 It was already 0 in the executor, this sets it to zero for the manager. * Fix docstring with correct default * Removing self.block_id reference
-
Htex python version check #857 (#883)
* check the version until the micro level * add logging messages for compatible versions
-
Remove duplicate setting of local_path for Globus staging (#878)
Now local_path is set in the same way as for other staging methods, and some globus-specific special casing elsewhere in the code is removed. The way in which self.globus is initialised has to move around because of this change, and now happens in an explicitly named method. The globus-specific list of files in the data manager, which exists only for an undone TODO, is removed.
-
Make File not a subclass of `str` - a breaking user-facing API change (…
…#880) * Make File not a subclass of `str` - a breaking user-facing API change File was originally made a subclass of `str` to attempt to allow File objects to be used where file paths are expected - for example, with open(). Python 3.6 introduces a proper way to do this, using the __fspath__ protocol; and File already implements this. Python 3.5 does not support __fspath__, and only `str` or `byte` objects can be passed to `open`. Prior to this commit, File attempted to override the implementation of __str__ to dynamically provide a posix path. However, this does not always work: `open` uses the immutable superclass string value, not the value returned by `__str__`. When a `File` is constructed like this: `File("./out.txt")` then the immutable str value is (probably) correct and no error occurs. The existing test cases catch this. When a `File` is constructed like this: `File("file:///tmp/out.txt")` then the immuatable str value is that entire URI, even though invoking __str__() returns a reasonable posix path: `/tmp/out.txt` This case of opening a URI-based File is not tested in the current test suite. Because the immutable str value, and the result of __str__() do not agree, some awkward error messages arise. For example, an error message like this is reported: ``` Exception: [Errno 2] No such file or directory: /tmp/qwertz.out ``` even though /tmp/qwertz.out is somewhere writeable, because (according to `strace`) the OS open call was invoked with the immutable str value: ``` open("file:///tmp/qwertz.out", O_RDWR|O_CREAT|O_APPEND|O_CLOEXEC, 0666 <un finished ...> ``` This commit does not change behaviour for users of python >=3.6, but requires python 3.5 users to explicitly convert a File object to a path, using `str()`. * Fix documentation for #799 where File objs cannot be passed to open This commit puts explicit .filepath property accesses on to invocations of `open` * Update user guide
-
Merge pull request #876 from Parsl/benc-more-type-annotations
annawoodard committedApr 15, 2019 Add more `typeguard` annotations for end-user exposed calls
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
-
Add type hints for HighThroughputExecutor constructor
benclifford committedApr 14, 2019 encountering a problem here RussBaz/enforce#62 using optional[tuple with a None default value
-
Add type annotations for globus
benclifford committedApr 14, 2019 -
benclifford committed
Apr 14, 2019 -
Add type annotations for set_file/stream_logger
benclifford committedApr 14, 2019
-
Merge pull request #875 from Parsl/benc-enforce-to-typeguard
annawoodard committedApr 14, 2019 Switch from `enforce` to `typeguard` as it is more maintained
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
-
-
Fix broken #856 changes to interchange->executor exceptions (#860)
benclifford committedApr 13, 2019 Revert two changes from #856: in result_package, when tid == -1, the type of exception should be Exception after all (as was the case before #856). It is only when tid != -1 that the type of exception should be RemoteExceptionWrapper. In the third change in #856, it is necessary for an exception to be actually raised, rather than the exception object just be constructed, in order to pass it correctly with RemoteExceptionWrapper; and the parameters passed to RemoteExceptionWrapper were incorrect. This PR addresses both of those issues. * fix exception sending - when tid == -1, 'exception' should contain a serialised exception, not a remote exception wrapper; but when tid != -1, 'exception' should contain a RemoteExceptionWrapper * remove debugging * more fiddling with exceptions * remove debug changes to test * remove error message change, to go in other PR * fix flake8
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits -
Use enforce.py for runtime checking of top level Config (#862)
benclifford committedApr 13, 2019 This uses PEP484 type hints, the same annotations which mypy will statically check, but on suitably annotated functions, those type hints are enforced at runtime. Config based type errors are the main place where this could help users - see issue #855 and PR #832 Error messages resulting from this will look like this: E enforce.exceptions.RuntimeTypeError: E The following runtime type errors were encountered: E Argument 'strategy' was not of type <class 'str'>. Actual type was int. To use this fully, all of the functions for config-like code (i.e. constructors and helper functions that you'd expect to find in a config definition) should be type-hinted and the @runtime_validation decorator added - I have opened issue #873 to track that. Some suitable type hints might (or might not) be found in the benc-mypy branch (PR #863), where I've been shadowing the master code base but with type hints suitable for mypy. This complements mypy usage: mypy can verify that pieces of the parsl codebase correctly call other pieces of the parsl codebase, statically; enforce can verify that other people's code calls pieces of the parsl codebase correctly, dynamically. This PR also breaks mypy typechecking for Config - mypy does not have type information for the runtime_validation decorator, and so this PR tells mypy to ignore that decorator, meaning that decorated functions lose their type info as far as mypy is concerned. This could be addressed by providing a type stub to mypy for runtime_validation. This is a breakage of the spirit of PR #832 which was intended to allow static checking of configurations, for example ahead of submitting to a queue.
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits