7.2. Metaprogramming New

• __new__ - object constructor

• __init__ - object initializer

object constructor

Method called when Python creates an instance of a class. Constructor is responsible for creating an object. Constructor returns an instance.

object initializer

Method called when Python creates an instance of a class. Initializer is responsible for initializing empty object with values. Initializer should return None.

In Python, __new__ and __init__ are two special methods that are used in the creation of objects. __new__ is a static method that is responsible for creating an instance of a class. It takes the class as its first argument and returns a new instance of the class. It is called before the __init__ method. __init__ is an instance method that is responsible for initializing the instance of a class that was created by the __new__ method. It takes the instance as its first argument and initializes its attributes. In other words, __new__ creates the object and __init__ initializes it. You can think of __new__ as the constructor of the class and __init__ as the initializer of the instance.

In Python by definition both methods __new__() and __init__() combined and called consecutively are constructors. This is something which is not existing in other programming languages, hence programmers has problem with grasping this idea. In most cases people will take their "experience" and "habits" from other languages, mixed with vogue knowledge about __new__() and call __init__() a constructor.

In Object Oriented Programming constructor is:

1. Special method defined by user, or inherited from parent class

2. Called automatically on object creation

3. Can set instance attributes with initial values

4. Works on not fully created object

5. Method calls are not allowed (as of object is not ready)

6. Cannot return anything

Method __new__ will always get called when an object has to be created. There are some situations where __init__ will not get called. One example is when you unpickle objects from a pickle file, they will get allocated (__new__) but not initialised (__init__) [1].

>>> class User:
...     def __new__(cls, *args, **kwargs):
...         print('New: before instantiating')
...         result = super().__new__(cls, *args, **kwargs)
...         print('New: after instantiating')
...         return result
...
...     def __init__(self, *args, **kwargs):
...         print('Init: initializing')
>>>
>>>
>>> mark = User()
New: before instantiating
New: after instantiating
Init: initializing

7.2.1. Init Method

• Initializer method

• for initializing object with initial values

• self as it's first parameter

• __init__() is called after __new__() and the instance is in place, so you can use self with it

• it's purpose is just to alter the fresh state of the newly created instance

Called after the instance has been created (by __new__()), but before it is returned to the caller. The arguments are those passed to the class constructor expression. If a base class has an __init__() method, the derived class's __init__() method, if any, must explicitly call it to ensure proper initialization of the base class part of the instance; for example: super().__init__([args...]) [3].

>>> class User:
...     def __init__(self):
...         print('init')
>>>
>>>
>>> mark = User()
init

7.2.2. New Method

• Constructor method

• cls as it's first parameter

• when calling __new__() you actually don't have an instance yet, therefore no self exists at that moment

• It takes the same arguments as __init__()

Called to create a new instance of class cls. __new__() is a static method (special-cased so you need not declare it as such) that takes the class of which an instance was requested as its first argument. The remaining arguments are those passed to the object constructor expression (the call to the class). The return value of __new__() should be the new object instance (usually an instance of cls) [2].

Typical implementations create a new instance of the class by invoking the superclass' __new__() method using super().__new__(cls[, ...]) with appropriate arguments and then modifying the newly-created instance as necessary before returning it [2].

__new__() is intended mainly to allow subclasses of immutable types (like int, str, or tuple) to customize instance creation. It is also commonly overridden in custom metaclasses in order to customize class creation [2].

>>> class User:
...     def __new__(cls):
...         print('new before')
...         instance = super().__new__(cls)
...         print('new after')
...         return instance
>>>
>>>
>>> mark = User()
new before
new after

7.2.3. New and Init

• __new__() is called before __init__()

• __new__() is responsible for creating the object

• __init__() is responsible for initializing the object

• __init__() is called on the object returned by __new__()

If __new__() is invoked during object construction and it returns an instance of cls, then the new instance's __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to the object constructor. If __new__() does not return an instance of cls, then the new instance's __init__() method will not be invoked [2].

>>> class User:
...     def __new__(cls):
...         print('new before')
...         instance = super().__new__(cls)
...         print('new after')
...         return instance
...
...     def __init__(self):
...         print('init')
>>>
>>>
>>> mark = User()
new before
new after
init

7.2.4. Parameters and Arguments

• __new__ and __init__ should have the same parameters

• You can use *args and **kwargs to pass any number of arguments

• Parameters - variables defined on method definition

• Arguments - values passed to method on invocation (call)

>>> class User:
...     def __new__(cls, firstname, lastname):
...         instance = super().__new__(cls)
...         return instance
...
...     def __init__(self, firstname, lastname):
...         self.firstname = firstname
...         self.lastname = lastname

You can use *args and **kwargs to pass any number of arguments. This is useful when you don't know how many arguments will be passed to the constructor.

>>> class User:
...     def __new__(cls, *args, **kwargs):
...         instance = super().__new__(cls)
...         return instance
...
...     def __init__(self, firstname, lastname):
...         self.firstname = firstname
...         self.lastname = lastname

7.2.5. Return Self Type

• __init__() is called on the object returned by __new__() if it returns an instance of the class

• super().__new__(cls) returns an instance of the class (preferred way)

• object.__new__(cls) returns an instance of the class

>>> class User:
...     def __new__(cls):
...         print('Constructing object')
...         instance = super().__new__(cls)
...         return instance
...
...     def __init__(self):
...         print('Initializing object')
>>>
>>>
>>> mark = User()
Constructing object
Initializing object

>>> class User:
...     def __new__(cls):
...         print('Constructing object')
...         instance = object.__new__(cls)
...         return instance
...
...     def __init__(self):
...         print('Initializing object')
>>>
>>>
>>> mark = User()
Constructing object
Initializing object

7.2.6. Return Other Type

• if __new__() does not return an instance of the same class, then __init__() will not be called

Return objects of other types from constructor:

>>> class User:
...     def __new__(cls):
...         instance = list()
...         return instance
...
...     def __init__(self):
...         print('Initializing object')  # -> is actually never called
>>>
>>> mark = User()
>>>
>>> type(mark)
<class 'list'>
>>>
>>> mark
[]

7.2.7. Case Study

class PDF:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying Word: {self.filename}')


file = PDF('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = Word('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = Word('myfile.docx')
file.display()
# Displaying Word: myfile.docx

def document(filename):
    name, extention = filename.split('.')
    match extention:
        case 'pdf':
            return PDF(filename)
        case 'doc' | 'docx':
            return Word(filename)
        case _:
            raise ValueError('Unknown file type')


class PDF:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying Word: {self.filename}')


file = document('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = document('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = document('myfile.docx')
file.display()
# Displaying Word: myfile.docx

class Document:
    def __new__(cls, filename):
        name, extention = filename.split('.')
        match extention:
            case 'pdf':
                return PDF(filename)
            case 'doc' | 'docx':
                return Word(filename)
            case _:
                raise ValueError('Unknown file type')


class PDF:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word:
    def __init__(self, filename):
        self.filename = filename

    def display(self):
        print(f'Displaying Word: {self.filename}')


file = Document('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = Document('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = Document('myfile.docx')
file.display()
# Displaying Word: myfile.docx

class Document:
    def __init__(self, filename):
        self.filename = filename

    def __new__(cls, filename):
        name, extention = filename.split('.')
        match extention:
            case 'pdf':
                return PDF(filename)
            case 'doc' | 'docx':
                return Word(filename)
            case _:
                raise ValueError('Unknown file type')


class PDF(Document):
    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word(Document):
    def display(self):
        print(f'Displaying Word: {self.filename}')


file = Document('myfile.pdf')
# RecursionError: maximum recursion depth exceeded

class Document:
    def __init__(self, filename):
        self.filename = filename

    def __new__(cls, filename):
        name, extention = filename.split('.')
        match extention:
            case 'pdf':
                return super().__new__(PDF)
            case 'doc' | 'docx':
                return super().__new__(Word)
            case _:
                raise ValueError('Unknown file type')


class PDF(Document):
    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word(Document):
    def display(self):
        print(f'Displaying Word: {self.filename}')


file = Document('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = Document('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = Document('myfile.docx')
file.display()
# Displaying Word: myfile.docx

class Document:
    def __init__(self, filename):
        self.filename = filename

    def __new__(cls, filename):
        name, extention = filename.split('.')
        plugins = cls.__subclasses__()
        for plugin in plugins:
            if extention in plugin.EXTENSIONS:
                return super().__new__(plugin)
        raise ValueError('Unknown file type')


class PDF(Document):
    EXTENSIONS = ['pdf']

    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word(Document):
    EXTENSIONS = ['doc', 'docx']

    def display(self):
        print(f'Displaying Word: {self.filename}')


file = Document('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = Document('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = Document('myfile.docx')
file.display()
# Displaying Word: myfile.docx

Adding new plugin does not require to change the code:

from abc import ABC, abstractmethod


class Document(ABC):
    def __init__(self, filename):
        self.filename = filename

    @property
    @abstractmethod
    def EXTENSIONS(self) -> list[str]:
        raise NotImplementedError

    @abstractmethod
    def display(self):
        raise NotImplementedError

    def __new__(cls, filename):
        name, extention = filename.split('.')
        plugins = cls.__subclasses__()
        for plugin in plugins:
            if extention in plugin.EXTENSIONS:
                return super().__new__(plugin)
        raise ValueError('Unknown file type')


class PDF(Document):
    EXTENSIONS = ['pdf']

    def display(self):
        print(f'Displaying PDF: {self.filename}')


class Word(Document):
    EXTENSIONS = ['doc', 'docx']

    def display(self):
        print(f'Displaying Word: {self.filename}')


file = Document('myfile.pdf')
file.display()
# Displaying PDF: myfile.pdf

file = Document('myfile.doc')
file.display()
# Displaying Word: myfile.doc

file = Document('myfile.docx')
file.display()
# Displaying Word: myfile.docx

Plugins can be hot-plugged. This means that you can attach a new plugin without reloading server code or application. Just define a class which conforms to plugin protocol (inherits from abstract base class Document) and it will work. No reloads nor restarts. That's it.

7.2.8. Use Case - 1

• Path

Note, that this unfortunately does not work this way. Path() always returns PosixPath:

>>> from pathlib import Path
>>>
>>>
>>> Path('/etc/passwd')
PosixPath('/etc/passwd')
>>>
>>> Path('c:\\Users\\Admin\\myfile.txt')  
WindowsPath('c:\\Users\\Admin\\myfile.txt')
>>>
>>> Path(r'C:\Users\Admin\myfile.txt')  
WindowsPath('C:\\Users\\Admin\\myfile.txt')
>>>
>>> Path(r'C:/Users/Admin/myfile.txt')  
WindowsPath('C:/Users/Admin/myfile.txt')

7.2.9. Use Case - 2

• Iris Factory

>>> from dataclasses import dataclass, field
>>> from itertools import starmap
>>>
>>>
>>> DATA = [
...     (5.8, 2.7, 5.1, 1.9, 'virginica'),
...     (5.1, 3.5, 1.4, 0.2, 'setosa'),
...     (5.7, 2.8, 4.1, 1.3, 'versicolor'),
...     (6.3, 2.9, 5.6, 1.8, 'virginica'),
...     (6.4, 3.2, 4.5, 1.5, 'versicolor'),
...     (4.7, 3.2, 1.3, 0.2, 'setosa'),
... ]
>>>
>>>
>>> @dataclass
... class Iris:
...     sepal_length: float
...     sepal_width: float
...     petal_length: float
...     petal_width: float
...     species: str = field(repr=False)
...
...     def __new__(cls, *args, **kwargs):
...         *measurements, species = args
...         clsname = species.capitalize()
...         cls = globals()[clsname]
...         return super().__new__(cls)
>>>
>>>
>>> class Setosa(Iris):
...     pass
>>>
>>> class Virginica(Iris):
...     pass
>>>
>>> class Versicolor(Iris):
...     pass
>>>
>>>
>>> result = starmap(Iris, DATA)
>>> list(result)  
[Virginica(sepal_length=5.8, sepal_width=2.7, petal_length=5.1, petal_width=1.9),
 Setosa(sepal_length=5.1, sepal_width=3.5, petal_length=1.4, petal_width=0.2),
 Versicolor(sepal_length=5.7, sepal_width=2.8, petal_length=4.1, petal_width=1.3),
 Virginica(sepal_length=6.3, sepal_width=2.9, petal_length=5.6, petal_width=1.8),
 Versicolor(sepal_length=6.4, sepal_width=3.2, petal_length=4.5, petal_width=1.5),
 Setosa(sepal_length=4.7, sepal_width=3.2, petal_length=1.3, petal_width=0.2)]

7.2.10. Use Case - 3

>>> import logging
>>> from datetime import datetime, timezone
>>> from uuid import uuid4
>>>
>>>
>>> class BaseClass:
...     def __new__(cls, *args, **kwargs):
...         instance = super().__new__(cls)
...         instance._since = datetime.now(tz=timezone.utc)
...         instance._log = logging.getLogger(cls.__name__)
...         instance._uuid = str(uuid4())
...         return instance
>>>
>>>
>>> class User(BaseClass):
...     def __init__(self, username, password):
...         self.password = password
...         self.username = username
>>>
>>>
>>> mark = User(username='mwatney', password='Ares3')
>>>
>>> vars(mark)  
{'_since': datetime.datetime(1969, 7, 21, 2, 56, 15, tzinfo=datetime.timezone.utc),
 '_log': <Logger User (INFO)>,
 '_uuid': '47fe947f-7219-446d-b59e-fa9afa1e8c81',
 'password': 'Ares3',
 'username': 'mwatney'}

7.2.11. Use Case - 4

>>> from datetime import datetime, timezone
>>> import logging
>>> from uuid import uuid4
>>> from abc import ABC, abstractmethod
>>>
>>>
>>> class BaseClass(ABC):
...     def __new__(cls, *args, **kwargs):
...         obj = object.__new__(cls)
...         obj._since = datetime.now(timezone.utc)
...         obj._uuid = str(uuid4())
...         obj._logger = logging.getLogger(cls.__name__)
...         return obj
...
...     def _log(self, level: int, id: int, msg: str):
...         self._logger.log(level, f'[{level}:{id}] {msg}')
...
...     def _debug(self, id:int, msg:str):    self._log(logging.DEBUG, id, msg)
...     def _info(self, id:int, msg:str):     self._log(logging.INFO, id, msg)
...     def _warning(self, id:int, msg:str):  self._log(logging.WARNING, id, msg)
...     def _error(self, id:int, msg:str):    self._log(logging.ERROR, id, msg)
...     def _critical(self, id:int, msg:str): self._log(logging.CRITICAL, id, msg)
...
...     @abstractmethod
...     def __init__(self):
...         pass
>>>
>>>
>>> class User(BaseClass):
...     def __init__(self, *args, **kwargs):
...         ...
>>>
>>>
>>> mark = User()
>>>
>>> vars(mark)  
{'_since': datetime.datetime(1969, 7, 21, 2, 56, 15),
 '_uuid': '83cefe23-3491-4661-b1f4-3ca570feab0a',
 '_log': <Logger User (WARNING)>}
>>>
>>> mark._error(123456, 'An error occurred')  
1969-07-21T02:56:15Z [ERROR:123456] An error occurred

7.2.12. Use Case - 5

• Injecting __new__ method with logging and tracing

>>> from datetime import datetime
>>> import logging
>>> from uuid import uuid4
>>>
>>>
>>> def trace(cls, *args, **kwargs):
...     instance = object.__new__(cls)
...     instance._log = logging.getLogger(f'{cls.__name__}')
...     instance._since = datetime.now()
...     instance._uuid = str(uuid4())
...     return instance

Without injecting:

>>> class User:
...     pass
>>>
>>> mark = User()
>>> vars(mark)
{}

With injecting:

>>> class User:
...     pass
>>>
>>> User.__new__ = trace
>>>
>>> mark = User()
>>> vars(mark)  
{'_log': <Logger myapp.User (INFO)>,
 '_since': datetime.datetime(1969, 7, 21, 2, 56, 15),
 '_uuid': '971827b5-5bc5-4a4b-9c8a-ecf5e4dbcfe3'}

7.2.13. References

[1]

Noufal Ibrahim. Python (and Python C API): __new__ versus __init__. Year: 2011. Retrieved: 2022-03-09. URL: https://stackoverflow.com/a/5143108

[2] (1,2,3,4)

Basic customization. object.__new__(cls[, ...]). Year: 2022. Retrieved: 2022-04-01. URL: https://docs.python.org/3/reference/datamodel.html#object.__new__

[3]

object.__init__(self[, ...]). Year: 2022. Retrieved: 2022-04-01. URL: https://docs.python.org/3/reference/datamodel.html#object.__init__

7.2.14. Assignments

# %% License
# - Copyright 2025, Matt Harasymczuk <matt@python3.info>
# - This code can be used only for learning by humans
# - This code cannot be used for teaching others
# - This code cannot be used for teaching LLMs and AI algorithms
# - This code cannot be used in commercial or proprietary products
# - This code cannot be distributed in any form
# - This code cannot be changed in any form outside of training course
# - This code cannot have its license changed
# - If you use this code in your product, you must open-source it under GPLv2
# - Exception can be granted only by the author

# %% Run
# - PyCharm: right-click in the editor and `Run Doctest in ...`
# - PyCharm: keyboard shortcut `Control + Shift + F10`
# - Terminal: `python -m doctest -v myfile.py`

# %% About
# - Name: OOP ObjectConstructor Syntax
# - Difficulty: easy
# - Lines: 6
# - Minutes: 5

# %% English
# 1. Define class `Point` with methods:
#    - `__new__()` taking `x` and `y` and returns new class instance
#    - `__init__()` takes `x` and `y` and stores them as attributes
# 2. Run doctests - all must succeed

# %% Polish
# 1. Zdefiniuj klasę `Point` z metodami:
#    - `__new__()` przyjmuje  `x` i `y` i zwraca nową instancję klasy
#    - `__init__()` przyjmuje `x` i `y` i zapisuje je jako atrybuty
# 2. Uruchom doctesty - wszystkie muszą się powieść

# %% Tests
"""
>>> import sys; sys.tracebacklimit = 0
>>> assert sys.version_info >= (3, 9), \
'Python 3.9+ required'

>>> from inspect import isclass

>>> assert isclass(Point)
>>> assert hasattr(Point, '__new__')
>>> assert hasattr(Point, '__init__')
>>> pt = Point(1, 2)
>>> assert type(pt) is Point
>>> assert pt.x == 1
>>> assert pt.y == 2
"""


# Define class `Point` with methods:
# - `__new__()` taking `x` and `y` and returns new class instance
# - `__init__()` taking `x` and `y` and stores them as attributes
# type: type
class Point:
    ...

# %% License
# - Copyright 2025, Matt Harasymczuk <matt@python3.info>
# - This code can be used only for learning by humans
# - This code cannot be used for teaching others
# - This code cannot be used for teaching LLMs and AI algorithms
# - This code cannot be used in commercial or proprietary products
# - This code cannot be distributed in any form
# - This code cannot be changed in any form outside of training course
# - This code cannot have its license changed
# - If you use this code in your product, you must open-source it under GPLv2
# - Exception can be granted only by the author

# %% Run
# - PyCharm: right-click in the editor and `Run Doctest in ...`
# - PyCharm: keyboard shortcut `Control + Shift + F10`
# - Terminal: `python -m doctest -v myfile.py`

# %% About
# - Name: OOP ObjectConstructor Passwd
# - Difficulty: easy
# - Lines: 8
# - Minutes: 8

# %% English
# 1. Iterate over lines in `DATA` and split line by colon
# 2. Create class `Account` that returns instances of `UserAccount` or `SystemAccount`
#    depending on the value of UID field
# 3. User ID (UID) is the third field, e.g.
#    `root:x:0:0:root:/root:/bin/bash` has UID equal to `0`
# 4. If UID is:
#    - below 1000, then account is system (`SystemAccount`)
#    - 1000 or more, then account is user (`UserAccount`)
# 5. Run doctests - all must succeed

# %% Polish
# 1. Iteruj po liniach w `DATA` i podziel linię po dwukropku
# 2. Stwórz klasę `Account`, która zwraca instancje klas
#    `UserAccount` lub `SystemAccount` w zależności od wartości pola UID
# 3. User ID (UID) to trzecie pole, np.
#    `root:x:0:0:root:/root:/bin/bash` to UID jest równy `0`
# 4. Jeżeli UID jest:
#    - poniżej 1000, to konto jest systemowe (`SystemAccount`)
#    - 1000 lub więcej, to konto użytkownika (`UserAccount`)
# 5. Uruchom doctesty - wszystkie muszą się powieść

# %% Hints
# - `str.splitlines()`
# - `str.split()`
# - `str.strip()`
# - `map()`

# %% Tests
"""
>>> import sys; sys.tracebacklimit = 0
>>> assert sys.version_info >= (3, 9), \
'Python 3.9+ required'

>>> from pprint import pprint

>>> result = list(result)
>>> pprint(result)
[SystemAccount(username='root'),
 SystemAccount(username='bin'),
 SystemAccount(username='daemon'),
 SystemAccount(username='adm'),
 SystemAccount(username='shutdown'),
 SystemAccount(username='halt'),
 SystemAccount(username='nobody'),
 SystemAccount(username='sshd'),
 UserAccount(username='mwatney'),
 UserAccount(username='mlewis'),
 UserAccount(username='rmartinez'),
 UserAccount(username='avogel'),
 UserAccount(username='bjohanssen'),
 UserAccount(username='cbeck')]
"""

DATA = """root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
nobody:x:99:99:Nobody:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
mwatney:x:1000:1000:Mark Watney:/home/mwatney:/bin/bash
mlewis:x:1001:1001:Melissa Lewis:/home/mlewis:/bin/bash
rmartinez:x:1002:1002:Rick Martinez:/home/rmartinez:/bin/bash
avogel:x:1003:1003:Alex Vogel:/home/avogel:/bin/bash
bjohanssen:x:1004:1004:Beth Johanssen:/home/bjohanssen:/bin/bash
cbeck:x:1005:1005:Chris Beck:/home/cbeck:/bin/bash"""

from dataclasses import dataclass


@dataclass
class SystemAccount:
    username: str

@dataclass
class UserAccount:
    username: str


# Parse DATA and convert to UserAccount or SystemAccount
# type: list[Account]
result = ...