14 Advanced Python Features

I mean that sounds smart and might seem correct to an uninformed observer that doesn’t try to run it, but try to actually run that and you’ll see that it errors in pyright and mypy. Unless I’m misunderstanding your comment, I think you’re suggesting this definition:

def g[T: str | int](v: T) -> T:
    return v * 2

Pyright gives this warning:

not_real.py:17:12 - error: Type "str | int" is not assignable to return type "T@g"
    Type "str | int" is not assignable to type "T@g" (reportReturnType)
1 error, 0 warnings, 0 informations

and mypy gives this warning:

not_real.py:17: error: Incompatible return value type (got "str | int", expected "T")  [return-value]
Found 1 error in 1 file (checked 1 source file)

Inside of the body, v‘s type is T (which in practice means str | int because it has so far not been narrowed), and v * 2 is str | int. Generics are just type hints, they’re not actually doing monomorphization or anything like that.

And besides, I could have picked this type of function as the original function to which we want to add type hints:

def f(a):
    if isinstance(a, str):
        return int(a)
    return str(a)

which you’d type hint as follows:

@overload
def f(a: int) -> str: ...

@overload
def f(a: str) -> int: ...

def f(a: str | int) -> str | int:
    if isinstance(a, str):
        return int(a)
    return str(a)

They’re open to bugs and mistakes in the implementation.

That’s sorta an argument against Python’s type hinting systems in their entirety. Once you use Any explicitly, or use Any implicitly by type hinting some but not all parameters, or use Any implicitly by leaving off the type parameter of a generic type, or use cast, or call any function that does those things, guess what, the call stack isn’t perfectly type checked. Using @overload means you’re adding a static hint to something that is fundamentally dynamically typed; @overload doesn’t introduce dynamic dispatch or monomorphization and doesn’t pretend to. It’s not a statically typed language. The bar isn’t “is this as rigorous as Rust/Haskell/whatever”, the bar is “does this meaningfully help us improve the quality of existing untyped Python codebases”.

I’m currently trapped in a love/hate war with __getitem__ and pyright where it wants me to support slices, or ints, but not both, and attempting to change its definition doesn’t do what I want out of it. It might have something to do with subclassing Sequence.

The last definition of a name “wins” at runtime, so you define all the overloads as annotated stub functions and then write the real one.

https://docs.python.org/3/library/typing.html#typing.runtime_checkable

runtime_checkable() will check only the presence of the required methods or attributes, not their type signatures or types.

If you’re referring to overloads, it works at runtime because there’s only one real implementation with a bunch of if statements to pick the correct argument set.

This is extremely cursed.

although I’m still not entirely comfortable with that walrus operator.

Personally, I use them often for branching where I need the result of the check. e.g.,

if var_exists := os.getenv("SOME_VALUE"):  # do the thing

or

match some_value := conditional():  # create a name binding without needing to `as name` in every branch

It’s small, but when used consistently, reduces the mental overhead of temporary bindings. Unfortunately, it does not stop the name being exported in the namespace because of Python’s scoping.

Yeah, NewType is not TypeAlias and AFAIK there’s no new syntax to declare a NewType.

I’m not sure I understand this complaint. What is a better design?

If I am the author of some function, am I not exposed to calling site choices only in the event of changes or in the event of variadic arguments (usu. when forwarding)? In the former case, can I not use positional or keyword-only arguments to eliminate this problem. In the latter, can I not use inspect.signature to eliminate the problem?

In other words:

def f(x, y): pass

# callee (`f`) cannot distinguish between:
f(123, 456)
f(123, y=456)
f(x=123, y=456)

As the author of f, if I want to change the order of arguments, this change affects callers.

def f(y, x): pass

f(123, 456)     # breaks
f(123, y=456)   # breaks
f(x=123, y=456) # works

As the author of f, if I want to change the name of arguments, this change affects callers.

def f(ecks, why): pass

f(123, 456)     # works
f(123, y=456)   # breaks
f(x=123, y=456) # breaks

I can use positional and keyword arguments to eliminate this problem: positional-only arguments can change names, keyword-only arguments can change positions. (With an import hook and a decorator, we could even design a utility that would allow us to version functions and adapt old API callers to the new API, but such a utility is probably not that useful in practice; more just calling convention is likely to change.)

def f(w, x, /, *, y, z): pass
def f(why, ecks, /, *, z, y): pass

f(123, 456, y=789, z=999)

In the case of variadic arguments, I can observe changes in calling convention.

def f(*args, **kwargs): pass

f(123, 456)     # args=(123, 456); kwargs={}
f(123, y=456)   # args=(123,);     kwargs={'y': 456}
f(x=123, y=456) # args=();         kwargs={'x': 123, 'y': 456}

However, what could I reasonably do with the arguments passed to f if there is no constraint on them, and I don’t even know what they might be? In general, the most useful thing I can do is forward them to another function which can make this distinction.

def f(*args, **kwargs): return g(*args, **kwargs)
def g(x, y): pass

f(123, 456)
f(123, y=456)
f(x=123, y=456)

This situation is covered by a previous example.

So the situation I want to consider is the use of *args and **kwargs where I observe (likely surgically) some portion. In this case, it may appear that I do have to know how the argument was passed in.

def f(*args, **kwargs):
    kwargs['y'] = kwargs.pop('y', 0) # default
    return g(*args, **kwargs)
def g(x, y): pass

f(123)        # works
f(x=123)      # works
f(123, y=456) # works
f(123, 456)   # breaks

However, this can be solved with inspect.signature.

from inspect import signature
def f(*args, **kwargs):
    bound = signature(g).bind_partial(*args, **kwargs)
    bound.arguments['y'] = kwargs['y'] = bound.arguments.get('y', 0) # default
    # optional: preserve original calling convention
    # args = (v for k, v in bound.arguments.items() if k not in kwargs)
    # kwargs = {k: bound.arguments[k] for k in kwargs}
    return g(*bound.args, **bound.kwargs)
def g(x, y, z=999): pass

f(123)        # works
f(x=123)      # works
f(123, y=456) # works
f(123, 456)   # works

the (default) way Python handled positional vs. keyword parameters has always seemed the worst of both worlds

the single biggest mistake in the language imho