Parsl/parsl

* Make `connected_workers` consistent

Now the `connected_workers` method returns the number of connected
workers. Fixes #656.

* Add 'connected_managers' method

Issue #898 shows a confusing log message, where a heartbeat
incorrectly logs that HEARTBEAT_CODE tasks have been requested,
in addition to logging a heartbeat.

This commit moves that log into the non-heartbeat path only, so
that exactly one of the heartbeat or task request messages will be
logged.

The phrasing of the heartbeat log message is modified to be in the
same sense as the task request message.

Previously, no state transition happened for completed processes: any
process labelled as RUNNING would not trigger any of the state transitions.

This commit removes the incorrect non-RUNNING restriction on transitions
to COMPLETED or FAILED.

This commit also extends the FAILED transition to happen with any non-zero
return code, not just negative ones: this means that a FAILED transition
should happen when a process exits with a failing return code.

This commit also adds in a error statement when none of these state
transitions fire.

Prior to this commit, some failures to launch would manifest as hangs
waiting for the launched code to start, as the failure would be ignored.

After this commit, those failures to launch result in a COMPLETED
or FAILED state, and so for example the HighThroughputExecutor loops
attempting to start new workers repeatedly - still a hang, but a
hopefully more debuggable hang.

Support for identifying managers by block_id, and putting managers in a block on hold

Managers are given a block identifier at initialization that is reported back at registration.
This allows the executor to track tasks/worker slots at the block level.
The executor also has a _hold_block() function that puts all managers in a block on hold
preventing further task scheduling, and making it safe to terminate the block.

Update docs to reflect PR #892 import changes

Expose newish app decorators through top level module

* Make memo-based callback look like execution-based callback

This makes the memo-based callback happen via the (memo)
Future, in the same way that the execution-based callback
happens via an execution Future.

This weakens the requirement that memo_fu have
a result already at the point that this piece of
code is reached: if it doesn't, the callback will
be deferred until it does.

In practice, this should not change user-exposed memo
behavior - instead this is to simplify code.

This is a step towards making memo- and execution-based
completion callbacks be handled in entirely the same
way, as part of pulling out pieces of PR #722.

* Combine two exec callbacks into one

Previously the exec_done callback was added in two
places in launch_tasks, depending on whether the
execution path or the memoization path was taken.

This commit moves that callback addition to outside
of launch_tasks, where we do not need to care where
the launched task future has come from.

Fixes #889.

Monitoring viz recovery

This reverts commit 741363b, reversing
changes made to 0eb1ea2.

The purpose of this merge and revert is to recover the commits made by
@garri1105. These were unintentionally lost when his work was copied into
a dedicated repository for the visualization tool.

Prior to this, file objects pretty much just forwarded on
staging requests to the data manager without doing anything
interesting.

This removes that indirection.

It removes some of the need for a file to be associated
with a data manager - something which sits awkwardly with
wanting to reuse Files between DFKs, for example in
interactive notebook use.

Instead, each DFK has a reference to a data manager directly.

All three methods (done(), exception() and add_callback())
will behave more consistently if using the Future superclass
implementations rather than attempting to piggyback onto
the parent implementations and behaving wrongly when there
is no parent future.

* Add support for configuring prefetch capacity

This is a squashed cherry pick with a few changes removed
in order to just collect the changes related to prefetch capacity.
It is based on the following commits:

75f9d4e
5a221f2
c89f3b2

* Set prefetch_capacity default to 0

It was already 0 in the executor, this sets it to zero for
the manager.

* Fix docstring with correct default

* Removing self.block_id reference

* check the version until the micro level

* add logging messages for compatible versions

Now local_path is set in the same way as for other staging
methods, and some globus-specific special casing elsewhere
in the code is removed.

The way in which self.globus is initialised has to move around
because of this change, and now happens in an explicitly
named method.

The globus-specific list of files in the data manager, which
exists only for an undone TODO, is removed.

…#880)

* Make File not a subclass of `str` - a breaking user-facing API change

File was originally made a subclass of `str` to attempt to allow File
objects to be used where file paths are expected - for example, with
open().

Python 3.6 introduces a proper way to do this, using the __fspath__
protocol; and File already implements this.

Python 3.5 does not support __fspath__, and only `str` or `byte`
objects can be passed to `open`.

Prior to this commit, File attempted to override the implementation
of __str__ to dynamically provide a posix path.

However, this does not always work: `open` uses the immutable superclass
string value, not the value returned by `__str__`.

When a `File` is constructed like this: `File("./out.txt")` then the
immutable str value is (probably) correct and no error occurs. The
existing test cases catch this.

When a `File` is constructed like this: `File("file:///tmp/out.txt")`
then the immuatable str value is that entire URI, even though invoking
__str__() returns a reasonable posix path: `/tmp/out.txt`

This case of opening a URI-based File is not tested in the current test
suite.

Because the immutable str value, and the result of __str__()
do not agree, some awkward error messages arise. For example,
an error message like this is reported:

```
Exception: [Errno 2] No such file or directory: /tmp/qwertz.out
```

even though /tmp/qwertz.out is somewhere writeable, because
(according to `strace`) the OS open call was invoked with the
immutable str value:


```
open("file:///tmp/qwertz.out", O_RDWR|O_CREAT|O_APPEND|O_CLOEXEC, 0666 <un
finished ...>
```

This commit does not change behaviour for users of python >=3.6, but
requires python 3.5 users to explicitly convert a File object to
a path, using `str()`.

* Fix documentation for #799 where File objs cannot be passed to open

This commit puts explicit .filepath property accesses on to
invocations of `open`

* Update user guide

Add more `typeguard` annotations for end-user exposed calls

encountering a problem here RussBaz/enforce#62
using optional[tuple  with a None default value

Switch from `enforce` to `typeguard` as it is more maintained

Revert two changes from #856: in result_package, when tid == -1, the type of exception should be Exception after all (as was the case before #856). It is only when tid != -1 that the type of exception should be RemoteExceptionWrapper.

In the third change in #856, it is necessary for an exception to be actually raised, rather than the exception object just be constructed, in order to pass it correctly with RemoteExceptionWrapper; and the parameters passed to RemoteExceptionWrapper were incorrect. This PR addresses both of those issues.

* fix exception sending - when tid == -1, 'exception' should contain a serialised exception, not a remote exception wrapper; but when tid != -1, 'exception' should contain a RemoteExceptionWrapper

* remove debugging

* more fiddling with exceptions

* remove debug changes to test

* remove error message change, to go in other PR

* fix flake8

This uses PEP484 type hints, the same annotations which mypy will statically check, but on suitably annotated functions, those type hints are enforced at runtime.

Config based type errors are the main place where this could help users - see issue #855 and PR #832

Error messages resulting from this will look like this:

E   enforce.exceptions.RuntimeTypeError: 
E     The following runtime type errors were encountered:
E          Argument 'strategy' was not of type <class 'str'>. Actual type was int.

To use this fully, all of the functions for config-like code (i.e. constructors and helper functions that you'd expect to find in a config definition) should be type-hinted and the @runtime_validation decorator added - I have opened issue #873 to track that.

Some suitable type hints might (or might not) be found in the benc-mypy branch (PR #863), where I've been shadowing the master code base but with type hints suitable for mypy.

This complements mypy usage: mypy can verify that pieces of the parsl codebase correctly call other pieces of the parsl codebase, statically; enforce can verify that other people's code calls pieces of the parsl codebase correctly, dynamically.

This PR also breaks mypy typechecking for Config - mypy does not have type information for the runtime_validation decorator, and so this PR tells mypy to ignore that decorator, meaning that decorated functions lose their type info as far as mypy is concerned. This could be addressed by providing a type stub to mypy for runtime_validation. This is a breakage of the spirit of PR #832 which was intended to allow static checking of configurations, for example ahead of submitting to a queue.

Remove most complexity from AppFuture __repr__