7.3. Protocol Structural

• PEP 544 -- Protocols: Structural subtyping (static duck typing)

• Since Python 3.8

• Protocol describe an interface, not an implementation

• Protocol classes should not have method implementations

• Protocol can describe both methods and attributes

A class object is considered an implementation of a protocol if accessing all members on it results in types compatible with the protocol members.

All things protocol related resides in typing library in Protocol class:

>>> from typing import Protocol, Self, runtime_checkable, NamedTuple

Typical protocol implementation looks like that:

>>> class Message(Protocol):
...     recipient: str
...     body: str
...
...     def send() -> None: ...
...     def receive() -> Self: ...

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def delete(self, key: str) -> bool: ...
>>>
>>>
>>> class Database:
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def delete(self, key: str) -> bool: ...
>>>
>>>
>>> cache: Cache = Database()
>>> cache.set('name', 'Mark Watney')
>>> cache.get('name')
>>> cache.delete('name')

7.3.1. Example

In Python there is a Context Manager protocol. In order to conform to this protocol, your class needs to define two methods: __enter__() and __exit__(). When you have both of those methods, you can use it in the with statement. There is no checking if you have certain type or your class inherits from some kind of abstract. Just define two methods and your good to go.

>>> class MyFile:
...     def __enter__(self):
...         return ...
...
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         ...
>>>
>>>
>>> with MyFile() as file:
...     pass

Note, that there is no explicit information, that your code implements the protocol. This is called structural subtyping.

The intuitive implementation of the protocol might look like:

>>> class ContextManager(Protocol):
...     def __enter__(self): ...
...     def __exit__(self, exc_type, exc_val, exc_tb): ...

Which enables it use it in the with statement:

>>> cm: ContextManager
>>>
>>> with cm() as variable:  
...     ...

Note, that the above code is just only to demonstrate the example and it is not intended run. Executing it will result in SyntaxError exception.

7.3.2. Standard Library Protocols

• from collections.abc import *

• Container

• Hashable

• Iterable

• Iterator

• Reversible

• Generator

• Callable

• Collection

• Sequence

• MutableSequence

• ByteString

• Set

• MutableSet

• Mapping

• MutableMapping

• MappingView

• ItemsView

• KeysView

• ValuesView

• Awaitable

• Coroutine

• AsyncIterator

• AsyncGenerator

Table 7.2. Protocols

Abstract Base Class	Inherits from	Methods
Container		__contains__
Hashable		__hash__
Iterable		__iter__
Iterator	Iterable	__next__, __iter__
Reversible	Iterable	__reversed__
Generator	Iterator	send, throw, close, __iter__, __next__, __len__
Callable		__call__
Collection	Sized, Iterable, Container	__contains__, __iter__, __len__
Sequence	Reversible, Collection	__getitem__, __contains__, __iter__, __reversed__, __len__, index, count
MutableSequence	Sequence	__getitem__, __setitem__, append, reverse, extend, pop, __delitem__, remove, __iadd__, __len__, insert, __contains__, __iter__, __reversed__, index, count
ByteString	Sequence	__getitem__, __len__, __contains__, __iter__, __reversed__, index, count
Set	Collection	__contains__, __le__, __lt__, __eq__, __ne__, __iter__, __gt__, __ge__, __and__, __or__, __len__, __sub__, __xor__, isdisjoint
MutableSet	Set	__contains__, __iter__, clear, pop, remove, __ior__, __len__, __iand__, __ixor__, __isub__, add, discard, __contains__, __le__, __lt__, __eq__, __ne__, __iter__, __gt__, __ge__, __and__, __or__, __len__, __sub__, __xor__, isdisjoint
Mapping	Collection	__getitem__, __contains__, keys, items, values, __iter__, get, __eq__, __ne__, __len__
MutableMapping	Mapping	__getitem__, __setitem__, pop, popitem, clear, update, __delitem__, setdefault, __iter__, __len__, __getitem__, __contains__, keys, items, values, __iter__, get, __eq__, __ne__, __len__
MappingView	Sized	__len__
ItemsView	MappingView, Set	__contains__, __iter__
KeysView	MappingView, Set	__contains__, __iter__
ValuesView	MappingView, Collection	__contains__, __iter__
Awaitable		__await__
Coroutine	Awaitable, AsyncIterable	send, throw, close, __aiter__
AsyncIterator	AsyncIterable	__anext__, __aiter__
AsyncGenerator	AsyncIterator	asend, athrow, aclose, __aiter__, __anext__

7.3.3. Terminology

PEP 544 propose to use the term protocols for types supporting structural subtyping. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion [2].

This has the drawback that the term protocol becomes overloaded with two subtly different meanings: the first is the traditional, well-known but slightly fuzzy concept of protocols such as iterator; the second is the more explicitly defined concept of protocols in statically typed code. The distinction is not important most of the time, and in other cases we propose to just add a qualifier such as protocol classes when referring to the static type concept. [2]

If a class includes a protocol in its MRO, the class is called an explicit subclass of the protocol.

If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included in the MRO, the class is an implicit subtype of the protocol. (Note that one can explicitly subclass a protocol and still not implement it if a protocol attribute is set to None in the subclass, see Python data-model for details.) [2]

The attributes (variables and methods) of a protocol that are mandatory for other class in order to be considered a structural subtype are called protocol members. [2]

7.3.4. Explicit Subtyping

• Email is explicit subclass of the protocol

If a class includes a protocol in its MRO, the class is called an explicit subclass of the protocol.

>>> class Message(Protocol):
...     recipient: str
...     body: str

>>> class Email(Message):
...     sender: str
...     recipient: str
...     subject: str
...     body: str
>>>
>>>
>>> def send(message: Message):
...     ...
>>>
>>>
>>> email = Email()
>>> email.sender = 'mwatney@nasa.gov'
>>> email.recipient = 'mlewis@nasa.gov'
>>> email.subject = 'I am alive!'
>>> email.body = 'I survived the storm. I am alone on Mars.'
>>>
>>> send(email)  # will pass the checker

7.3.5. Structural Subtyping

• If an object that has all the protocol attributes it implements it

• Structural subtyping is natural for Python programmers

• Matches the runtime semantics of duck typing

• Email is structural subtype of a protocol (it conforms to structure)

• Email is implicit subtype of the protocol Message (not inherits)

• Email implement the protocol Message

• Email is compatible with a protocol Message

If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included in the MRO, the class is an implicit subtype of the protocol. (Note that one can explicitly subclass a protocol and still not implement it if a protocol attribute is set to None in the subclass, see Python data-model for details.) [2]

>>> class Message(Protocol):
...     recipient: str
...     body: str

>>> class Email:
...     sender: str
...     recipient: str
...     subject: str
...     body: str
>>>
>>>
>>> def send(message: Message):
...     ...
>>>
>>>
>>> email = Email()
>>> email.sender = 'mwatney@nasa.gov'
>>> email.recipient = 'mlewis@nasa.gov'
>>> email.subject = 'I am alive!'
>>> email.body = 'I survived the storm. I am alone on Mars.'
>>>
>>> send(email)  # will pass the checker

7.3.6. What Protocols are Not?

• At runtime, protocol classes is simple ABC

• No runtime type check

• Protocols are completely optional

At runtime, protocol classes will be simple ABCs. There is no intent to provide sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 Python steering committee states that protocols are completely optional [2]:

• No runtime semantics will be imposed for variables or parameters annotated with a protocol class.

• Any checks will be performed only by third-party type checkers and other tools.

• Programmers are free to not use them even if they use type annotations.

• There is no intent to make protocols non-optional in the future.

>>> class SMS(Protocol):
...     recipient: str
...     body: str
>>>
>>> class MMS(Protocol):
...     recipient: str
...     body: str
...     mimetype: str

>>> class MyMessage:
...     recipient: str
...     body: str
>>>
>>>
>>> a: SMS = MyMessage()  # Ok
>>> b: MMS = MyMessage()  # Expected type 'MMS', got 'MyMessage' instead

7.3.7. Covariance, Contravariance, Invariance

• https://www.youtube.com/watch?v=1IiL31tUEVk&t=445s

• Covariance - Enables you to use a more derived type than originally specified

• Contravariance - Enables you to use a more generic (less derived) type than originally specified

• Invariance - Means that you can use only the type originally specified.

• Invariance is important for example in np.ndarray, where all items must be exactly the same type

Covariance and contravariance are terms that refer to the ability to use a more derived type (more specific) or a less derived type (less specific) than originally specified. Generic type parameters support covariance and contravariance to provide greater flexibility in assigning and using generic types [1]

In general, a covariant type parameter can be used as the return type of a delegate, and contravariant type parameters can be used as parameter types.

>>> def check(what: int):
...     pass

>>> bool.mro()
[<class 'bool'>, <class 'int'>, <class 'object'>]

Covariance

Enables you to use a more derived type than originally specified [1]

>>> check(True)     # ok
>>> check(1)        # ok
>>> check(object)   # error

Contravariance

Enables you to use a more generic (less derived) type than originally specified [1]

>>> check(True)     # error
>>> check(1)        # ok
>>> check(object)   # ok

Invariance

Means that you can use only the type originally specified. An invariant generic type parameter is neither covariant nor contravariant [1]

>>> check(True)     # error
>>> check(1)        # ok
>>> check(object)   # error

>>> from typing import TypeVar
>>>
>>>
>>> T = TypeVar('T', int, float, covariant=False, contravariant=True)
>>>
>>> def run(x: T) -> T:
...     ...

Figure 7.2. Covariance. Replacement with more specialized type. Dog is more specialized than Animal. [3]

Figure 7.3. Contravariance. Replacement with more generic type. Animal is more generic than Cat. [3]

Figure 7.4. Invariance. Type must be the same and you cannot replace it. Animal cannot be substituted for Cat and vice versa. [3]

Example:

By default type annotation checkers works in covariant mode:

>>> def print_coordinates(point: tuple):
...     x, y, z = point
...     print(f'{x=}, {y=}, {z=}')

This means, that the point argument to the print_coordinates function could be either tuple or any object which inherits from tuple.

In the following example pt is invariant type as it is exactly as required, that is tuple:

>>> pt = (1, 2, 3)
>>> print_coordinates(pt)
x=1, y=2, z=3

NamedTuple inherits from tuple, so it could be used as covariant type:

>>> class Point(NamedTuple):
...     x: int
...     y: int
...     z: int
>>>
>>> pt = Point(1,2,3)
>>> print_coordinates(pt)
x=1, y=2, z=3

7.3.8. Default Value

>>> class User(Protocol):
...     firstname: str
...     lastname: str
...     group: str = 'admins'

7.3.9. Merging and extending protocols

>>> from typing import Sized, Protocol
>>>
>>>
>>> class Closable(Protocol):
...     def close(self) -> None:
...         ...
>>>
>>> class SizableAndClosable(Sized, Closable, Protocol):
...     pass

7.3.10. Generic Protocols

>>> from abc import abstractmethod
>>> from typing import Protocol, TypeVar, Iterator
>>>
>>>
>>> T = TypeVar('T')
>>>
>>> class Iterable(Protocol[T]):
...     @abstractmethod
...     def __iter__(self) -> Iterator[T]:
...         ...

7.3.11. Recursive Protocols

• Since 3.11 PEP 673 –- Self Type

• Since 3.7 from __future__ import annotations

• Future PEP 563 -- Postponed Evaluation of Annotations

>>> from typing import Protocol, Iterable, Self

Traversing Graph nodes:

>>> class Graph(Protocol):
...     def get_node(self) -> Iterable[Self]:
...         ...

Traversing Tree nodes:

>>> class Tree(Protocol):
...     def get_node(self) -> Iterable[Self]:
...         ...

7.3.12. Unions

>>> class Exitable(Protocol):
...     def exit(self) -> int:
...         ...
>>>
>>> class Quittable(Protocol):
...     def quit(self) -> int | None:
...         ...

>>> def finish(task: Exitable | Quittable) -> None:
...     task.exit()
...     task.quit()

7.3.13. Modules as implementations of protocols

A module object is accepted where a protocol is expected if the public interface of the given module is compatible with the expected protocol. For example:

File config.py:

>>> database_host = '127.0.0.1'
>>> database_port = 5432
>>> database_name = 'ares3'
>>> database_user = 'mwatney'
>>> database_pass = 'myVoiceIsMyPassword'

File main.py:

>>> from typing import Protocol
>>>
>>>
>>> class DatabaseConfig(Protocol):
...     database_host: str
...     database_port: int
...     database_name: str
...     database_user: str
...     database_pass: str
>>>
>>>
>>> import config  
>>> dbconfig: DatabaseConfig = config # Passes type check  

7.3.14. Callbacks

File myrequest.py:

>>> URL = 'https://python3.info'
>>>
>>> def on_success(result: str) -> None:
...     ...
>>>
>>> def on_error(error: Exception) -> None:
...     ...

File main.py:

>>> from typing import Protocol
>>>
>>>
>>> class Request(Protocol):
...     URL: str
...     def on_success(self, result: str) -> None: ...
...     def on_error(self, error: Exception) -> None: ...
>>>
>>>
>>> import myrequest  
>>> request: Request = myrequest  # Passes type check  

7.3.15. Runtime Checkable

• By default isinstance() and issubclass() won't work with protocols

• You can use typing.runtime_checkable decorator to make it work

The default semantics is that isinstance() and issubclass() fail for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime.

However, it should be possible for protocol types to implement custom instance and class checks when this makes sense, similar to how Iterable and other ABCs in collections.abc and typing already do it, but this is limited to non-generic and unsubscripted generic protocols (Iterable is statically equivalent to Iterable[Any]).

The typing module will define a special @runtime_checkable class decorator that provides the same semantics for class and instance checks as for collections.abc classes, essentially making them 'runtime protocols':

>>> class Message(Protocol):
...     recipient: str
...     body: str
>>>
>>>
>>> class Email:
...     sender: str
...     recipient: str
...     subject: str
...     body: str
>>>
>>>
>>> isinstance(Email, Message)
Traceback (most recent call last):
TypeError: Instance and class checks can only be used with @runtime_checkable protocols

>>> @runtime_checkable
... class Message(Protocol):
...     recipient: str
...     body: str
>>>
>>>
>>> class Email:
...     sender: str
...     recipient: str
...     subject: str
...     body: str
>>>
>>>
>>> isinstance(Email, Message)
False

The above example returns False because Email class defines only type annotations not fields (fields does not exist and therefore it is not and instance of a protocol). With methods is easier. Methods always exists.

Example below shows class with already filled information and therefore those fields exists.

>>> class Account(Protocol):
...     username: str
...     password: str
>>>
>>>
>>> class User:
...     username: str = 'Mark'
...     password: str = 'Watney'
...     groups: str = ['staff', 'admins']
>>>
>>>
>>> isinstance(User, Account)
Traceback (most recent call last):
TypeError: Instance and class checks can only be used with @runtime_checkable protocols

>>> @runtime_checkable
... class Account(Protocol):
...     username: str
...     password: str
>>>
>>>
>>> class User:
...     username: str = 'Mark'
...     password: str = 'Watney'
...     groups: str = ['staff', 'admins']
>>>
>>>
>>> isinstance(User, Account)
True

>>> @runtime_checkable
... class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def delete(self, key: str) -> bool: ...
>>>
>>>
>>> class Database:
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def delete(self, key: str) -> bool: ...
>>>
>>>
>>> cache: Cache = Database()
>>> cache.set('name', 'Mark Watney')
>>> cache.get('name')
>>> cache.delete('name')

7.3.16. Use Case - 0x01

>>> from typing import Protocol
>>>
>>>
>>> class SupportsClose(Protocol):
...     def close(self) -> None:
...         ...

7.3.17. Use Case - 0x02

>>> from dataclasses import dataclass

>>> class SupportsWrite(Protocol):
...     def write(self): ...
>>>
>>>
>>> def print(*values, sep=' ', end='\n', file: SupportsWrite = None):
...     ...

>>> @dataclass
... class File:
...     filename: str
...
...     def write(self):
...         ...
>>>
>>> print('hello', 'world', file=File('/tmp/myfile.txt'))

7.3.18. Use Case - 0x02

>>> from abc import abstractmethod
>>> from typing import Protocol
>>>
>>>
>>> class RGB(Protocol):
...     rgb: tuple[int, int, int]
...
...     @abstractmethod
...     def opacity(self) -> int:
...         return 0
>>>
>>>
>>> class Pixel(RGB):
...     def __init__(self, red: int, green: int, blue: float) -> None:
...         self.rgb = red, green, blue
...

Type checker will warn:

• blue must be int

• opacity is not defined

7.3.19. Use Case - 0x03

File myapp/view.py

>>> def get(request):
...     ...
>>>
>>> def post(request):
...     ...
>>>
>>> def put(request):
...     ...
>>>
>>> def delete(request):
...     ...

File main.py

>>> from typing import Protocol
>>>
>>>
>>> class HttpView(Protocol):
...     def get(request): ...
...     def post(request): ...
...     def put(request): ...
...     def delete(request): ...
>>>
>>>
>>> import myapp.view  
>>> view: HttpView = myapp.view  

7.3.20. Use Case - 0x04

This use case demonstrate how mypy display information about invalid method. Cache protocol defines .clear() method which is not present in the DatabaseCache implementation. mypy will raise and error and fail CI/CD build.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: str) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')

$ python3 -m mypy myfile.py
myfile.py:16: error: Incompatible types in assignment (expression has type "DatabaseCache", variable has type "Cache")  [assignment]
myfile.py:16: note: "DatabaseCache" is missing following "Cache" protocol member:
myfile.py:16: note:     clear
Found 1 error in 1 file (checked 1 source file)

7.3.21. Use Case - 0x05

This use case demonstrate how mypy display information about invalid argument type to the .set() method. In Cache protocol .set() method has value: str. In the implementation .set() method has value: int. Despite this is only type annotation change, mypy will raise an error and fail CI/CD build.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: int) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
...
...     def clear(self) -> None:
...         ...
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')
>>> mycache.clear()

$ python -m mypy myfile.py
myfile.py:25: error: Incompatible types in assignment (expression has type "DatabaseCache", variable has type "Cache")  [assignment]
myfile.py:25: note: Following member(s) of "DatabaseCache" have conflicts:
myfile.py:25: note:     Expected:
myfile.py:25: note:         def set(self, key: str, value: str) -> None
myfile.py:25: note:     Got:
myfile.py:25: note:         def set(self, key: str, value: int) -> None
Found 1 error in 1 file (checked 1 source file)

7.3.22. Use Case - 0x06

This time example is valid and does not contain any errors. This will allow for success build in CI/CD system.

>>> class Cache(Protocol):
...     def set(self, key: str, value: str) -> None: ...
...     def get(self, key: str) -> str: ...
...     def clear(self) -> None: ...

>>> class DatabaseCache:
...     def set(self, key: str, value: str) -> None:
...         ...
...
...     def get(self, key: str) -> str:
...         return '...'
...
...     def clear(self) -> None:
...         ...
>>>
>>>
>>> mycache: Cache = DatabaseCache()
>>> mycache.set('firstname', 'Mark')
>>> mycache.set('lastname', 'Watney')
>>> fname = mycache.get('firstname')
>>> lname = mycache.get('lastname')
>>> mycache.clear()

$ python -m mypy myfile.py
Success: no issues found in 1 source file

7.3.23. References

[1] (1,2,3,4)

https://docs.microsoft.com/en-us/dotnet/standard/generics/covariance-and-contravariance

[2] (1,2,3,4,5,6)

Levkivskyi, I. and Lehtosalo, J. and Langa, Ł. PEP 544 -- Protocols: Structural subtyping (static duck typing). Year: 2017. Retrieved: 2022-03-09. URL: https://www.python.org/dev/peps/pep-0544/

[3] (1,2,3)

Langa, Ł. Covariance/Contravariance/Invariance. Year: 2022. Retrieved: 2022-06-09. URL: https://www.youtube.com/watch?v=1IiL31tUEVk&t=445s