5.2. Abstract Protocol

  • Protocol describe an interface, not an implementation

  • Protocol classes should not have method implementations

  • Protocol can describe both methods and attributes

  • Since Python 3.8

  • PEP 544 -- Protocols: Structural subtyping (static duck typing)

5.2.1. Problem

>>> class User:
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class Admin:
...     def log_in(self): ...
...     def log_out(self): ...

>>> def auth(account):
...     account.login()
...     account.logout()

>>> mark = User()
>>> auth(mark)  # type checker: ok

>>> mark = Admin()
>>> auth(mark)  # type checker: error
Traceback (most recent call last):
AttributeError: 'Admin' object has no attribute 'login'. Did you mean: 'log_in'?

5.2.2. Solution: Abstract Class

  • Implementation is through the inheritance

  • Validation is dynamic (in the runtime)

>>> from abc import ABC, abstractmethod
>>>
>>>
>>> class Account(ABC):
...     @abstractmethod
...     def login(self): raise NotImplementedError
...
...     @abstractmethod
...     def logout(self): raise NotImplementedError

>>> class User(Account):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class Admin(Account):
...     def login(self): ...
...     def logout(self): ...

>>> def auth(account: Account):
...     account.login()
...     account.logout()
...     print('done')

>>> mark = User()
>>> auth(mark)  # type checker: ok
done

>>> mark = Admin()
>>> auth(mark)  # type checker: ok
done

5.2.3. Solution: Protocol

  • Implementation is through structural subtyping

  • No inheritance, just complying to the structure (having the same methods)

>>> from typing import Protocol
>>>
>>>
>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>>
>>> class User:
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class Admin:
...     def login(self): ...
...     def logout(self): ...

>>> def auth(account: Account):
...     account.login()
...     account.logout()
...     print('done')

>>> mark = User()
>>> auth(mark)
done

>>> mark = Admin()
>>> auth(mark)
done

5.2.4. Duck Typing

  • If it walks like a duck, and it quacks like a duck, then it must be a duck.

  • If it logins like an Account, and it logouts like an Account, then it must be an Account.

  • If it have method login like an Account, and it have method logout like an Account, then it adheres to the protocol Account.

If it walks like a duck, and it quacks like a duck, then it must be a duck.

If it logins like an Account, and it logouts like an Account, then it must be an Account.

If it have method login like an Account, and it have method logout like an Account, then it adheres to the protocol Account.

>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class User:
...     def login(self): ...
...     def logout(self): ...
...     def hello(self): ...
...     def goodbye(self): ...
>>>
>>> class Admin:
...     def login(self): ...
...     def add_user(self): ...
...     def del_user(self): ...

>>> def auth(account: Account):
...     account.login()
...     account.logout()
...     print('done')

>>> mark = User()
>>> auth(mark)  # ok
done

>>> mark = Admin()
>>> auth(mark)  # error (no method logout)
Traceback (most recent call last):
AttributeError: 'Admin' object has no attribute 'logout'

5.2.5. Case Study

You must always remember to close a file. Note, that the following implementation is erroneous:

>>> file = open('/tmp/myfile.txt', mode='wt')
>>> file.write('hello')
5
>>> file.close()

What if there is no space left on a disk? Write will raise an exception. When there is an exception, the thread exits. If that was the only thread, then the program exits and file is freed. However in a multithreaded environment it will result in deadlock. It means, that the file was not closed, and no other thread can write file (as it is blocked):

>>> file = open('/tmp/myfile.txt', mode='wt')
>>> file.write('hello')  # raises an exception: no space left on a disk
5
>>> file.close()  # will not execute

Context Manager solves this problem. Code inside of a context manager block will execute and if there is an error, the file will be closed anyway:

>>> with open('/tmp/myfile.txt', mode='wt') as file:
...     file.write('hello')
...
5

The following is the equivalent code. Context Manager has two methods __enter__() - called upon entering the context manager block (it should return self) and __exit__() - called upon exiting the context manager block. Method __exit__() is called even if there is an exception. In such case it information about an exception is passed as an argument:

>>> file = open('/tmp/myfile.txt', mode='wt').__enter__()
>>> try:
...     file.write('hello')
... except Exception as err:
...     file.__exit__(err.__class__, err, err.__traceback__)
... else:
...     file.__exit__()
...
5

Knowing all the information about Context Manager let's try to add this feature to our class:

>>> class User:
...     def hello(self):
...         print('hello')
>>>
>>>
>>> with User() as mark:
...     mark.hello()
...
Traceback (most recent call last):
TypeError: 'User' object does not support the context manager protocol

User adheres to the Context Manager protocol, because structurally covers all the methods required (it has __enter__() and __exit__() method. Hence it can be used in the with statement. Note, that there is no inheritance. User does not specify that it implements the protocol, it just simply does if there are proper methods. There is no abstract class nor interface here. All those rules are based on a fact, that your class has methods listed in the protocol:

>>> class User:
...     def hello(self):
...         print('hello')
...
...     def __enter__(self):
...         print('enter')
...         return self
...
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         print('exit')
>>>
>>>
>>> with User() as mark:
...     mark.hello()
...
enter
hello
exit

User adheres to the Context Manager protocol, because structurally covers all the methods required.

>>> 
... class ContextManager(Protocol):
...     def __enter__(self): ...
...     def __exit__(self, exc_type, exc_val, exc_tb): ...
...
... with <ContextManager> as <identifier>:
...     ...

5.2.6. Terminology

PEP 544 propose to use the term protocols for types supporting structural subtyping. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion [1].

This has the drawback that the term protocol becomes overloaded with two subtly different meanings: the first is the traditional, well-known but slightly fuzzy concept of protocols such as iterator; the second is the more explicitly defined concept of protocols in statically typed code. The distinction is not important most of the time, and in other cases we propose to just add a qualifier such as protocol classes when referring to the static type concept. [1]

If a class includes a protocol in its MRO, the class is called an explicit subclass of the protocol.

If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included in the MRO, the class is an implicit subtype of the protocol. (Note that one can explicitly subclass a protocol and still not implement it if a protocol attribute is set to None in the subclass, see Python data-model for details.) [1]

The attributes (variables and methods) of a protocol that are mandatory for other class in order to be considered a structural subtype are called protocol members. [1]

5.2.7. Explicit Subtyping

  • Email is explicit subclass of the protocol

If a class includes a protocol in its MRO, the class is called an explicit subclass of the protocol.

>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class User(Account):
...     def login(self): ...
...     def logout(self): ...

5.2.8. Structural Subtyping

  • If an object that has all the protocol attributes it implements it

  • Structural subtyping is natural for Python programmers

  • Matches the runtime semantics of duck typing

  • User is structural subtype of a protocol (it conforms to structure)

  • User is implicit subtype of the protocol Account (does not inherits)

  • User implement the protocol Account

  • User is compatible with a protocol Account

If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included in the MRO, the class is an implicit subtype of the protocol. (Note that one can explicitly subclass a protocol and still not implement it if a protocol attribute is set to None in the subclass, see Python data-model for details.) [1]

>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class User:
...     def login(self): ...
...     def logout(self): ...

5.2.9. What Protocols are Not?

  • At runtime, protocol classes is simple ABC

  • No runtime type check

  • Protocols are completely optional

At runtime, protocol classes will be simple ABCs. There is no intent to provide sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 Python steering committee states that protocols are completely optional [1]:

  • No runtime semantics will be imposed for variables or parameters annotated with a protocol class.

  • Any checks will be performed only by third-party type checkers and other tools.

  • Programmers are free to not use them even if they use type annotations.

  • There is no intent to make protocols non-optional in the future.

5.2.10. Runtime Checkable

  • By default isinstance() and issubclass() won't work with protocols

  • You can use typing.runtime_checkable decorator to make it work

The default semantics is that isinstance() and issubclass() fail for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime.

However, it should be possible for protocol types to implement custom instance and class checks when this makes sense, similar to how Iterable and other ABCs in collections.abc and typing already do it, but this is limited to non-generic and unsubscripted generic protocols (Iterable is statically equivalent to Iterable[Any]).

>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class User:
...     def login(self): ...
...     def logout(self): ...
>>>
>>>
>>> isinstance(User, Account)
Traceback (most recent call last):
TypeError: Instance and class checks can only be used with @runtime_checkable protocols

The typing module will define a special @runtime_checkable class decorator that provides the same semantics for class and instance checks as for collections.abc classes, essentially making them 'runtime protocols':

>>> from typing import Protocol, runtime_checkable
>>>
>>>
>>> @runtime_checkable
... class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...
>>>
>>> class User:
...     def login(self): ...
...     def logout(self): ...
>>>
>>>
>>> isinstance(User, Account)
True

5.2.11. Unions

>>> class CanLogin(Protocol):
...     def login(self): ...
>>>
>>> class CanLogout(Protocol):
...     def logout(self): ...

>>> def auth(account: CanLogin | CanLogout) -> None:
...     account.login()
...     account.logout()

5.2.12. Merging Protocols

>>> class CanLogin(Protocol):
...     def login(self): ...
>>>
>>> class CanLogout(Protocol):
...     def logout(self): ...

>>> class Account(CanLogin, CanLogout, Protocol):
...     ...

5.2.13. Interface Segregation Principle

  • S.O.L.I.D. Principles

  • ISP - Interface Segregation Principle

  • Clients should not be forced to depend upon interfaces that they do not use.

Clients should not be forced to depend upon interfaces that they do not use.

—SOLID, Interface Segregation Principle, Robert C. Martin

Instead of:

>>> class Account(Protocol):
...     def login(self): ...
...     def logout(self): ...

It is better to have:

>>> class CanLogin(Protocol):
...     def login(self): ...
>>>
>>>
>>> class CanLogout(Protocol):
...     def logout(self): ...

5.2.14. Generic Protocols

>>> from abc import abstractmethod
>>> from typing import Protocol, TypeVar, Iterator
>>>
>>>
>>> T = TypeVar('T')
>>>
>>> class Iterable(Protocol[T]):
...     @abstractmethod
...     def __iter__(self) -> Iterator[T]:
...         ...

5.2.15. Recursive Protocols

  • Since 3.11 PEP 673 –- Self Type

>>> from typing import Protocol, Iterable, Self

Traversing Graph nodes:

>>> class Graph(Protocol):
...     def get_node(self) -> Iterable[Self]:
...         ...

Traversing Tree nodes:

>>> class Tree(Protocol):
...     def get_node(self) -> Iterable[Self]:
...         ...

5.2.16. Modules as Implementations of Protocols

A module object is accepted where a protocol is expected if the public interface of the given module is compatible with the expected protocol. For example:

File myapp/config.py:

>>> database_host = '127.0.0.1'
>>> database_port = 5432
>>> database_name = 'users'
>>> database_user = 'mwatney'
>>> database_pass = 'Ares3'

File myapp/main.py:

>>> from typing import Protocol, runtime_checkable
>>>
>>>
>>> @runtime_checkable
... class DatabaseConfig(Protocol):
...     database_host: str
...     database_port: int
...     database_name: str
...     database_user: str
...     database_pass: str
>>>
>>>
>>> import myapp.config  
>>> isinstance(myapp.config, DatabaseConfig)  # type checker: ok

5.2.17. Use Case - 1

>>> from typing import Protocol
>>>
>>>
>>> class SupportsClose(Protocol):
...     def close(self) -> None:
...         ...

5.2.18. Use Case - 2

>>> from dataclasses import dataclass

>>> class SupportsWrite(Protocol):
...     def write(self): ...
>>>
>>>
>>> def print(*values, sep=' ', end='\n', file: SupportsWrite = None):
...     ...

>>> @dataclass
... class File:
...     filename: str
...
...     def write(self):
...         ...
>>>
>>> print('hello', 'world', file=File('/tmp/myfile.txt'))

5.2.19. Use Case - 2

File myapp/users.py

>>> PATH = '/api/v1/users/'
>>>
>>>
>>> def get(request):
...     ...
>>>
>>> def post(request):
...     ...
>>>
>>> def put(request):
...     ...
>>>
>>> def delete(request):
...     ...

File myapp/main.py

>>> from typing import Protocol
>>> import myapp.users  
>>>
>>>
>>> class Endpoint(Protocol):
...     PATH: str
...     def get(request): ...
...     def post(request): ...
...     def put(request): ...
...     def delete(request): ...
>>>
>>> def router_register(endpoint: Endpoint):
...     ...
>>>
>>> router_register(myapp.users)  # ok

5.2.20. Use Case - 3

File myapp/test.py:

>>> import logging
>>>
>>> URL = 'https://python3.info/index.html'
>>>
>>> def on_success(html: str) -> None:
...     assert '...' in html
>>>
>>> def on_error(error: Exception) -> None:
...     logging.error(error)

File myapp/main.py:

>>> from typing import Protocol
>>> import myapp.tests  
>>>
>>>
>>> class Test(Protocol):
...     URL: str
...     def on_success(self, html: str) -> None: ...
...     def on_error(self, error: Exception) -> None: ...
>>>
>>> def run(test: Test):
...     ...
>>>
>>> run(myapp.tests)  # ok

5.2.21. Use Case - 3

>>> from abc import abstractmethod
>>> from typing import Protocol
>>>
>>>
>>> class RGB(Protocol):
...     rgb: tuple[int, int, int]
...
...     @abstractmethod
...     def opacity(self) -> int:
...         return 0
>>>
>>>
>>> class Pixel(RGB):
...     def __init__(self, red: int, green: int, blue: float) -> None:
...         self.rgb = red, green, blue
...

Type checker will warn:

  • blue must be int

  • opacity is not defined

5.2.22. Use Case - 4

This use case demonstrate how mypy display information about invalid method. Cache protocol defines .clear() method which is not present in the DatabaseCache implementation. mypy will raise and error and fail CI/CD build.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: str) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')

$ python3 -m mypy myfile.py
myfile.py:16: error: Incompatible types in assignment (expression has type "DatabaseCache", variable has type "Cache")  [assignment]
myfile.py:16: note: "DatabaseCache" is missing following "Cache" protocol member:
myfile.py:16: note:     clear
Found 1 error in 1 file (checked 1 source file)

5.2.23. Use Case - 5

This use case demonstrate how mypy display information about invalid argument type to the .set() method. In Cache protocol .set() method has value: str. In the implementation .set() method has value: int. Despite this is only type annotation change, mypy will raise an error and fail CI/CD build.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: int) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
...
...     def clear(self) -> None:
...         ...
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')
>>> mycache.clear()

$ python -m mypy myfile.py
myfile.py:25: error: Incompatible types in assignment (expression has type "DatabaseCache", variable has type "Cache")  [assignment]
myfile.py:25: note: Following member(s) of "DatabaseCache" have conflicts:
myfile.py:25: note:     Expected:
myfile.py:25: note:         def set(self, key: str, value: str) -> None
myfile.py:25: note:     Got:
myfile.py:25: note:         def set(self, key: str, value: int) -> None
Found 1 error in 1 file (checked 1 source file)

5.2.24. Use Case - 6

This time example is valid and does not contain any errors. This will allow for success build in CI/CD system.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: str) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
...
...     def clear(self) -> None:
...         ...
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')
>>> mycache.clear()

$ python -m mypy myfile.py
Success: no issues found in 1 source file

5.2.25. References