2 things that you have to know to become a master of python

The concept of python language. Higher order functions and Dictionaries are two whales of python

Into

There are many programming languages. Some of them represent a bunch of different tools to manage memory. For example, C language is a more human-readable wrapper for ASM, whoever writes or saw MASM with macroses may mention that it’s so easy to make C language on them. Another type of programming languages made around some concept. This concept is a base for the language so that any more difficult structures of language is defined through the parts of the concept.

The python language is based on 2 things(like 2 whales 🐳 ):

🐳 Higher order functions — lambdas, data processing(map, reduce, etc or list comprehension), class methods, closures, decorators
🐳 Dictionaries — classes, meta classes, modules, etc

That’s it! Understanding those 2 things help to write any difficult framework or library on python. So who understand the idea of those 2 whales can just clap 👏 and skip reading this article. Who needs more examples, let’s go.

Disclaimer

All examples are on python 3 it only affects syntax not concept.

🐳 Thing #1: Higher order functions

It’s when you treat the function as an object so function could be passed to the another function or function could closure on the state. More detailed you can read here.

Let’s look to decorators through the prism of higher order functions

Start from simple decorator that track decorated function:

When python interpreter see declaration started with @ its translate it to the function call.

Where are such decorators may be used? In many scenarios where you need to register function for the further call from the framework. For example: Celery @shared_tasks or flask @app.route of course their decorators are much complex but the idea is the same register function to call inside a framework.

But what if I need to control not only the declaration of the function but also its execution. That means that instead of the function itself decorator should return some wrapper function where all execution control things are declared.

In the example above you may see that instead of the my_func decorator returns some wrapper function and this wrapper function closure to the state of decarator2 so that it knows about source function during the execution.

Such decorators use in many cases for example retrying. Also, you may play with function arguments and with the execution of the source function itself. For example look at Click as homework and try to repeat the Click decorators.

Knowledge about higher order functions in python helps to make any imaginable combinations of code.

🐳 Thing #2: Dictionaries

This thing is more difficult to understand than higher order functions. Basically, python looks at everything as at dictionary(dict) and provides tools to manage dicts. I’m not talking about primitives like 1 + 1. Basically, I wanna say that everything in python like modules, execution environment, classes, class instances and etc. are just dicts.

Let’s deep dive it steps by step.
When you start your program you have 2 dicts GLOBALS and LOCALS. The difference is the scope. GLOBALS is the global scope of your app, LOCALS is the local scope, for example scope of the function. Basically, LOCALS is the way to keep a state for higher order functions.

In another word when you define something, you tell python to make changes in dict.

Let’s go to a more complex thing like classes.
When you define class you tell python to create a dict and everything that you define inside it, put as keys and values to that dict. More mistakes happened when developers define class variables then change them in methods and thinking that it’s object fields, but they are classes. I’ll show this example later.

When you instantiate a class you tell python to make a new dict and call function from __init__ key of class definition dict with that new created dict as first arg for initialization.

In case if you have a chain of inheritance you have a list of dicts that python use for lookup. As you may understand when you call __init__ from parent class you just tell hey parent here is the instance dict put whatever data you need for further work.

Python methods are basically functions that call with the dict as the first arg the name of this arg usually is self but you may call it whatever you want for example pepa 🙂.

Here we come to the scariest part of python language — metaclasses. But I think those who understood everything that I said before having a clue what metaclasses are. With metaclasses, python gives you the ability not only to manage objects that instantiated from class but also class definition itself.

As you can see it’s possible to make a new class dynamically by providing the dict. When you are working on a framework you may have a lot of similar definitions like Pepa in example above. Why don’t apply the well-known approach with classes to them, because at the end everything is dicts. That’s how we have metaclasses

You may see metaclasses in frameworks where the class definition is used as some declaration. It’s mostly data mapping frameworks such as ORM for example Django models, web form mapping or JSON mapping, etc.

Also, it’s very helpful when you want to add some class-specific behavior like Django DoesNotExist exception is added to class from its metaclass so when you are catching exceptions for example User.DoesNotExist and Comment.DoesNotExists, for “except” operator they are different but DoesNotExist is defined once in metaclass.

Conclusion

Who read these lines and understood everything from this article now know python as master. At the end I wold like to confess there is the third thing it’s sugar :)

Higher order functions, Dictionaries and Sugar now you know everything about python