Python internals, metaprogramming, and advanced usage

10 powerful but underused Python features, explained end-to-end

This article walks through ten of Python’s “hidden gems” in a structured, example-driven way: what they are, why they exist, how they work, and when you should (and shouldn’t) use them.

1. Overview and mental map

Python’s surface area looks simple: functions, classes, imports, loops. Underneath, though, it exposes a deep, programmable substrate: method resolution order, computed properties, bytecode, attribute interception, constrained object layouts, runtime code generation, weak references, dynamic scopes, lightweight context managers, and fully custom iteration.

The features covered here:

• 1. super() with multiple inheritance and the method resolution order (MRO)
• 2. @property for computed and validated attributes
• 3. dis for disassembling and understanding bytecode
• 4. __getattr__ and __setattr__ for dynamic attributes
• 5. __slots__ for attribute constraints and memory optimization
• 6. exec for dynamic code execution
• 7. weakref for lifecycle-aware references
• 8. globals() and locals() for scope manipulation
• 9. contextlib for class-free context managers
• 10. __iter__ and __next__ for custom iterators

For each, you’ll see the core idea, a runnable example, why it’s valuable, and typical use cases where the feature actually pays for its complexity.

---

2. super() with multiple inheritance

In multiple inheritance hierarchies, super() is not “call my direct parent.” It is “move to the next class in the method resolution order (MRO).” When every class in a hierarchy cooperates and uses super(), Python can automatically wire calls through all relevant ancestors without you manually stitching them together.

2.1. Example: cooperative multiple inheritance

class A:
    def method(self):
        print("A method")

class B(A):
    def method(self):
        print("B method")
        super().method()

class C(A):
    def method(self):
        print("C method")
        super().method()

class D(B, C):
    def method(self):
        print("D method")
        super().method()

d = D()
d.method()

Output:

D method
B method
C method
A method

2.2. What’s happening under the hood?

The key is the MRO of D, which you can inspect via D.__mro__ or D.mro():

D.mro() == [D, B, C, A, object] Call chain for d.method(): 1. D.method() # prints "D method" super() from D -> next in MRO after D -> B 2. B.method() # prints "B method" super() from B -> next in MRO after B -> C 3. C.method() # prints "C method" super() from C -> next in MRO after C -> A 4. A.method() # prints "A method" (A does not call super(), chain ends)

super() is context-sensitive: it looks at the class where it’s used and the MRO of the instance to determine the “next” implementation to call. This is what lets multiple parents contribute to a single logical operation.

2.3. Why this is a genuine gem

• Cooperative behavior: each class contributes to a method without hard-coding who comes before or after it.
• Extensibility: you can insert new mixins into the inheritance chain without rewriting everyone’s super() calls.
• Maintainability: the MRO algorithm, not hand-written calls, decides the order.

2.4. When to use it

Cooperative super() is most valuable in:

• Mixins that add behavior around a shared method (logging, validation, metrics).
• Frameworks that expect subclasses to extend but not replace base behavior.
• Complex hierarchies where manual parent calls would be brittle or unclear.

Always design cooperative hierarchies so that every class in the chain calling a method uses super() consistently; mixing direct parent calls and super() usually leads to surprising results.

---

3. @property for computed and validated attributes

The @property decorator turns a method into an attribute-like interface. This is more than syntactic sugar: it lets you attach validation, lazy computation, or derived logic behind a clean attribute access pattern.

3.1. Example: validated radius and computed area

class Circle:
    def __init__(self, radius):
        self._radius = radius

    @property
    def radius(self):
        return self._radius

    @radius.setter
    def radius(self, value):
        if value < 0:
            raise ValueError("Radius cannot be negative")
        self._radius = value

    @property
    def area(self):
        return 3.14 * (self._radius ** 2)

circle = Circle(5)
print(circle.area)    # 78.5

circle.radius = 10
print(circle.area)    # 314.0

3.2. What problem does this solve?

• Encapsulation: callers do circle.radius instead of circle.get_radius() and circle.set_radius().
• Validation: the setter enforces invariants (no negative radii).
• Derived values: the area property is computed from internal state and always stays in sync.

3.3. Advanced uses

Beyond simple validation, @property is useful for:

• Lazy loading: load data from disk or a database only when first accessed.
• Transparent caching: compute once, cache internally, invalidate on changes.
• API compatibility: start with a public attribute, then later migrate to a property without changing callers.

---

4. The dis module: disassembling Python bytecode

CPython compiles your source into bytecode instructions executed by the virtual machine. The dis module lets you inspect those instructions, which is invaluable when you’re tuning performance, understanding control flow, or just demystifying what Python actually runs.

4.1. Example: disassembling a simple function

import dis

def test_function(x):
    return x + 1

dis.dis(test_function)

Possible output:

  4           0 LOAD_FAST                0 (x)
              2 LOAD_CONST               1 (1)
              4 BINARY_ADD
              6 RETURN_VALUE

4.2. How to read this

• LOAD_FAST 0 (x): load local variable x.
• LOAD_CONST 1 (1): load constant 1.
• BINARY_ADD: perform x + 1.
• RETURN_VALUE: return the result from the stack.

4.3. Why this matters

For most code you never need this level, but it becomes crucial when:

• You’re analyzing how different constructs compile (comprehensions vs loops, inlining, constant folding).
• You’re writing tools that inspect or transform Python code (profilers, linters, custom VMs).
• You want to confirm what “actually happens” beyond semantics in the language reference.

---

5. __getattr__ and __setattr__ for dynamic attributes

__getattr__ is called when an attribute isn’t found the usual way. __setattr__ runs on every attribute assignment. Together, they let you intercept and control attribute access, turning an object into a programmable proxy, cache, or dynamic record.

5.1. Example: dynamic attribute handling

class DynamicAttributes:
    def __init__(self):
        # Important: avoid recursion by using object's __setattr__ directly
        object.__setattr__(self, "_attributes", {})

    def __getattr__(self, name):
        if name in self._attributes:
            return self._attributes[name]
        else:
            print(f"Dynamic fetch of {name}")
            return None

    def __setattr__(self, name, value):
        print(f"Setting {name} to {value}")
        self._attributes[name] = value

obj = DynamicAttributes()
obj.dynamic_attr = "Hello"
print(obj.dynamic_attr)   # prints: Setting..., then "Hello"

5.2. What’s going on?

• __setattr__: called for every obj.name = value. You decide how to store or validate.
• __getattr__: only called if the attribute wasn’t found in the normal places (__dict__, class attributes, descriptors).

In this example, we redirect attribute storage into a private dictionary _attributes, effectively turning the object into a dynamic mapping with an attribute-based interface.

5.3. Why this is powerful

• Dynamic objects: objects whose “shape” can change at runtime.
• Lazy loading: load the value from a database or remote API the first time the attribute is accessed.
• Data binding and tracking: automatically log or propagate attribute changes.

Be careful: incorrectly implemented __setattr__ or __getattr__ can easily cause infinite recursion or break tools that expect normal attributes. Use sparingly and clearly document the behavior.

---

6. __slots__ for constrained, memory-efficient objects

By default, Python instances store attributes in a per-object dictionary. This is flexible but costs memory and allows arbitrary attributes. __slots__ lets you define a fixed set of allowed attributes, eliminating the instance __dict__ and saving memory.

6.1. Example: constraining a Point

class Point:
    __slots__ = ["x", "y"]  # Only these attributes are allowed

    def __init__(self, x, y):
        self.x = x
        self.y = y

point = Point(1, 2)
print(point.x, point.y)   # 1 2

# This would raise an AttributeError:
# point.z = 3

6.2. Effects of __slots__

• No per-instance __dict__: attributes are stored in a compact structure.
• Attribute constraints: you cannot add attributes not listed in __slots__.
• Potential speed improvements: attribute access can be slightly faster.

6.3. Use cases

• Large numbers of small objects (e.g., millions of points, nodes, or records).
• Domain models where you want to forbid arbitrary attributes.
• Performance-sensitive pipelines or data processing systems.

__slots__ changes object layout: not all libraries or metaprogramming tricks expect this. Use where the constraints are a feature, not a surprise.

---

7. exec for dynamic code execution

exec executes dynamically constructed Python code. It’s extremely powerful and equally easy to misuse. When used carefully, it enables dynamic languages, DSLs, and systems that generate code at runtime.

7.1. Example: defining a function at runtime

code = """
def dynamic_function():
    return "Dynamically executed code!"
"""

exec(code)  # defines dynamic_function in the current global scope
print(dynamic_function())   # Dynamically executed code!

7.2. How exec works

exec takes a string of Python code (or a compiled code object) and executes it in a given namespace:

namespace = {}
exec("x = 10", {}, namespace)
print(namespace["x"])   # 10

You can supply custom globals and locals, giving you tight control over what the executed code can see and modify.

7.3. When exec is appropriate

• Dynamic evaluation systems and REPLs.
• Code generation where Python is the target language.
• Metaprogramming for rapid prototyping or tooling.

Never feed untrusted input to exec. It can run arbitrary code, access the file system, network, and more. Treat it as you would “remote code execution” in a security review.

---

8. weakref for lifecycle-aware references

A normal reference keeps an object alive. A weak reference does not: the object may be garbage-collected even if weak references still exist. The weakref module gives you tools to build caches, graphs, and observers that don’t interfere with memory reclamation.

8.1. Example: observing object lifetime

import weakref

class MyClass:
    pass

obj = MyClass()
weak_obj = weakref.ref(obj)

print(weak_obj())   # <__main__.MyClass object at ...>

del obj
print(weak_obj())   # None - the underlying object is gone

8.2. Why weak references are useful

• Non-owning references: reference an object without preventing its garbage collection.
• Caches: a cache can hold weak references so it doesn’t keep rarely used objects alive forever.
• Avoiding cycles: in some observer patterns or graphs, weak references help break reference cycles.

8.3. Typical patterns

You’ll often see weakref.WeakKeyDictionary or WeakValueDictionary used to associate metadata with objects without affecting their lifetimes. When the object disappears, the dictionary entry does too.

---

9. globals() and locals() for inspecting and manipulating scope

Python exposes the current global and local namespaces as dictionaries via globals() and locals(). This lets you inspect, and in some contexts modify, what names are bound to what values.

9.1. Example: manipulating globals and locals

a = 10

# Change global 'a' dynamically
globals()["a"] = 42
print(a)   # 42

def change_local():
    b = 5
    # Modifying locals() inside a function is CPython-implementation-dependent,
    # but for demonstration:
    locals()["b"] = 100
    print(b)   # Often still prints 5; behavior is not guaranteed across implementations

change_local()

9.2. Important subtleties

• globals(): returns the actual dictionary for the current module’s global scope. Mutations are reflected in name lookups.
• locals(): returns a dictionary representing the current local namespace, but mutating it inside a function is not guaranteed to affect local variables in all implementations.

9.3. When these are genuinely useful

• REPLs and shells: introspect or modify top-level names.
• Metaprogramming: dynamically define functions or variables in a specific module.
• Debugging tools: capture snapshots of local and global scopes at runtime for inspection.

While you can mutate scopes dynamically, leaning too heavily on this makes code much harder to reason about. Prefer explicit parameters and returns; reserve scope manipulation for tooling and specialized runtime systems.

---

10. contextlib: lightweight context managers without classes

A context manager defines a region of code with setup and teardown logic (e.g., opening and closing a file). Normally you write a class with __enter__ and __exit__, but contextlib.contextmanager lets you do this with a simple generator function.

10.1. Example: a minimal custom context

from contextlib import contextmanager

@contextmanager
def my_context():
    print("Entering the context")
    try:
        yield
    finally:
        print("Exiting the context")

with my_context():
    print("Inside the context")

Output:

Entering the context
Inside the context
Exiting the context

10.2. How it works

• The code before yield is the “enter” phase (allocate resources, acquire locks, start timers).
• The code after yield is the “exit” phase (cleanup, release locks, log results).
• The body of the with block runs where yield appears.

10.3. Where this shines

• Resource management: temporary configuration changes, timers, logging scopes.
• Testing utilities: quickly define temporary environments or patches.
• API ergonomics: expose “do X within this context” with very little boilerplate.

---

11. __iter__ and __next__ to build custom iterators

Python’s iteration protocol is simple: an object is iterable if it implements __iter__ returning an iterator; an object is an iterator if it implements __next__ and raises StopIteration when exhausted. Defining these lets you control how your objects are traversed.

11.1. Example: a simple range-like iterator

class MyIterator:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        self.current += 1
        return self.current - 1

my_iter = MyIterator(1, 5)
for num in my_iter:
    print(num)    # 1 2 3 4 5

11.2. Why not just use generators?

In many cases, a generator function is simpler:

def my_range(start, end):
    current = start
    while current <= end:
        yield current
        current += 1

But explicit iterator classes are useful when:

• You need stateful iterators with methods beyond iteration (e.g., reset(), peek()).
• You’re building complex iterable objects (trees, graphs, streaming sources) where controlling internal state matters.
• You want multiple different iterators over the same object (__iter__ returning new iterator objects).

---

12. Putting it all together

These features live at different layers of the language: object model (super(), @property, __slots__, custom iterators), runtime and internals (dis, weakref), and metaprogramming (exec, globals()/locals(), contextlib, dynamic attributes). They’re not for everyday scripting, but they unlock:

• Framework design: cooperative multiple inheritance, context managers, dynamic attributes.
• Performance tuning: __slots__, dis, weakref-based caches.
• Metaprogramming and tooling: exec, dynamic scopes, bytecode inspection.

The real power comes from combining them thoughtfully: a memory-efficient, slot-based domain object with computed properties; a context-managed, dynamically generated function; an iterator that lazily loads data under a property-backed facade. Used carefully, these tools let you adjust Python’s behavior at exactly the level you need—no more, no less.