Introduce mypy stub for typeguard, and type-check parsl.load() (#874)

This allows mypy to understand the type signatures of functions decorated with @typeguard.typechecked, allowing the type signatures on such functions to be used by both enforce and mypy. Prior to this commit, adding @TypeChecked removed the ability of mypy to type-check calls to decorated functions, as it did not understand that the type of such a decorated function is the same as the original function. The added stub makes this declaration. The test-suite mypy calls are modified to use that stub file. The immediate motivation for this is to keep mypy passing when adding a type annotation for parsl.load; that type annotation is also added in this commit. This commit fixes some of problem described in the commit message for d7d9a25 about breaking mypy type checking for end user Configs in some places, but by no means all of it. However there is a mypy bug, python/mypy#5398, where decorated __init__ methods are not properly type-checked by mypy, which means that mypy typechecking is still missing on any decorated __init__ methods - which unfortunately is most of the user-facing type checked code. At time of writing, there are PRs open to fix this in mypy, so there is hope; and that does not impede this PR being merged.
Parsl · Apr 24, 2019 · e10c287b39b5d05a8c0fbfa99afcd16e9517cb44 · e10c287
1 parent 4df400e
commit e10c287b39b5d05a8c0fbfa99afcd16e9517cb44
Unified Split

Showing with 15 additions and 5 deletions.

+1 −1 .travis.yml

+10 −0 mypy-stubs/typeguard.pyi

+0 −3 mypy.ini

+4 −1 parsl/dataflow/dflow.py
diff --git a/.travis.yml b/.travis.yml
@@ -39,7 +39,7 @@ script:
     # This uses all of the configurations and tests as the base from which to
     # run mypy checks - these are likely to capture most of the code used in
     # parsl
-    - (for test in parsl/tests/configs/*.py parsl/tests/test*/test*; do mypy $test ; export MypyReturnCode=$? ; echo mypy return code is $MypyReturnCode ; if [[ "$MypyReturnCode" != 0 ]] ; then exit 1; fi; done ) ;
+    - (for test in parsl/tests/configs/*.py parsl/tests/test*/test*; do MYPYPATH=$(pwd)/mypy-stubs mypy $test ; export MypyReturnCode=$? ; echo mypy return code is $MypyReturnCode ; if [[ "$MypyReturnCode" != 0 ]] ; then exit 1; fi; done ) ;
 
       # do this before any testing, but not in-between tests
     - rm -f .coverage

diff --git a/mypy-stubs/typeguard.pyi b/mypy-stubs/typeguard.pyi
@@ -0,0 +1,10 @@
+
+# this type for runtime_validation is based on the
+# "Decorators that do not change the signature of the function" section of
+# https://github.com/python/mypy/issues/3157
+
+from typing import TypeVar, List
+
+Func = TypeVar('Func')
+
+def typechecked(f: Func) -> Func: ...
diff --git a/mypy.ini b/mypy.ini
@@ -1,9 +1,6 @@
 [mypy]
 plugins = sqlmypy
 
-[mypy-typeguard.*]
-ignore_missing_imports = True
-
 [mypy-non_existent.*]
 ignore_missing_imports = True
 

diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
@@ -5,13 +5,15 @@
 import pathlib
 import pickle
 import random
+import typeguard
 import inspect
 import threading
 import sys
 # import multiprocessing
 import datetime
 
 from getpass import getuser
+from typing import Optional
 from uuid import uuid4
 from socket import gethostname
 from concurrent.futures import Future
@@ -1027,7 +1029,8 @@ def clear(cls):
         cls._dfk = None
 
     @classmethod
-    def load(cls, config=None):
+    @typeguard.typechecked
+    def load(cls, config: Optional[Config] = None):
         """Load a DataFlowKernel.
 
         Args: