Python Deep Dives: Multiple Inheritances And Mixin Classes

I recently revisited three old posts on Django class-based views that I wrote for this blog, updating them to Django 3.0 (you can find them and noticed once again that the code base uses to increase code reuse. I also realised that mixins are not very popular in Python, so I decided to explore them, brushing up my knowledge of the OOP theory in the meanwhile. here mixin classes To fully appreciate the content of the post, be sure you grasp two pillars of the OOP approach: , in particular how it is implemented through inheritance, and . and contain all you need to understand how Python implements those concepts. delegation polymorphism This post about delegation this post about polymorphism # Multiple inheritance: blessing and curse ## General concepts To discuss mixins we need to start from one of the most controversial subjects in the whole OOP world: multiple inheritance. This is a natural extension of the concept of simple inheritance, where a class automatically delegates method and attribute resolution to another class (the parent class). Let me state it again, as it is important for the rest of the discussion: . inheritance is just an automatic delegation mechanism Delegation was introduced in OOP as a way to reduce code duplication. When an object needs a specific feature it just delegates it to another class (either explicitly or implicitly), so the code is written just once. Let's consider the example of code management website, clearly completely fictional and not inspired by any existing product. Let's assume we created the following hierarchy assignable reviewable item (assign_to_user, ask_review_to_user) ^ | | | pull request which allows us to put in only the specific code required by that element. This is a great achievement, as it is what libraries do for code, but on live objects. Method calls and delegation are nothing more than messages between objects, so the delegation hierarchy is just a simple networked system. pull request Unfortunately, the use of inheritance over composition often leads to systems that, paradoxically, increase code duplication. The main problem lies in the fact that inheritance can directly delegate to only one other class (the parent class), as opposed to composition, where the object can delegate to any number of other ones. This limitation of inheritance means that we might have a class that inherits from another one because it needs some of its features, but doing this receives features it doesn't want, or shouldn't have. Let's continue the example of the code management portal, and consider an , which is an item that we want to store in the system, but cannot be reviewed by a user. If we create a hierarchy like this issue assignable reviewable item (assign_to_user, ask_review_to_user) ^ | | | | +--------+--------+ | | | | | | issue pull request (not reviewable) we end up putting the features related to the review process in an object that shouldn't have them. The standard solution to this problem is that of increasing the depth of the inheritance hierarchy and to derive from the new simpler ancestor. assignable item (assign_to_user) ^ | | | | +------+--------------+ | | | | | | | reviewable assignable item | (ask_review_to_user) | ^ | | | | | | issue pull request However, this approach stops being viable as soon as an object needs to inherit from a given class but not from the parent of that class. For example, an element that has to be reviewable but not assignable, like a that we want to add to the site. If we want to keep using inheritance, the only solution at this point is to duplicate the code that implements the reviewable nature of the item (or the code that implements the assignable feature) and create two different class hierarchies. best practice assignable item +--------> reviewable item (assign_to_user) | (ask_review_to_user) ^ | ^ | | | | | | | CODE DUPLICATION | | | | +------+--------------+ | | | | | | | | | | | | V | | reviewable assignable item | | (ask_review_to_user) | | ^ | | | | | | | | | | issue pull request best practice Please note that this doesn't even take into account that the new might need attributes from , which prompts for another level of depth in the hierarchy, where we isolate those features in a more generic class. So, unfortunately, chances are that this is only the first of many compromises we will have to accept to keep the system in a stable state if we can't change our approach. reviewable item assignable item Multiple inheritance was then introduced in OOP, as it was clear that an object might want to delegate certain actions to a given class, and other actions to a different one, mimicking what life forms do when they inherit traits from multiple ancestors (parents, grandparents, etc.). The above situation can then be solved having inherit from both the class that provides the assign feature and from the one that implements the reviewable nature. pull request assignable item reviewable item (assign_to_user) (ask_review_to_user) ^ ^ ^ | | | | | | | | | | | | +------+-------------+ +----------------------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | issue pull request best practice Generally speaking, then, multiple inheritance is introduced to give the programmer a way to keep using inheritance without introducing code duplication, keeping the class hierarchy simpler and cleaner. Eventually, everything we do in software design is to try and separate concerns, that is, to isolate features, and multiple inheritance can help to do this. These are just examples and might be valid or not, depending on the concrete case, but they clearly show the issues that we can have even with a very simple hierarchy of 4 classes. Many of these problems clearly arise from the fact that we wanted to implement delegation only through inheritance, and I dare to say that 80% of the architectural errors in OOP projects come from using inheritance instead of composition and from using god objects, that is classes that have responsibilities over too many different parts of the system. Always remember that OOP was born with the idea of small objects interacting through messages, so the considerations we make for monolithic architectures are valid even here. That said, as inheritance and composition implement two different types of delegation ( and ), they are both valuable, and multiple inheritance is the way to remove the single provider limitation that comes from having only one parent class. to be to have ## Why is it controversial? Given what I just said, multiple inheritance seems to be a blessing. When an object can inherit from multiple parents, we can easily spread responsibilities among different classes and use only the ones we need, promoting code reuse and avoiding god objects. Unfortunately, things are not that simple. First of all, we face the issue that every microservice-oriented architecture faces, that is the risk of going from god objects (the extreme monolithic architecture) to almost empty objects (the extreme distributed approach), burdening the programmer with too a fine-grained control that eventually results in a system where relationships between objects are so complicated that it becomes impossible to grasp the effect of a change in the code. There is a more immediate problem in multiple inheritance, though. As it happens with the natural inheritance, parents can provide the same "genetic trait" in two different flavours, but the resulting individual will have only one. Leaving aside genetics (which is incredibly more complicated than programming) and going back to OOP, we face a problem when an object inherits from two other objects that provide the same attribute. So, if your class inherits from parents and , and both provide the method, which one should your object use? Child Parent1 Parent2 __init__ [...] [...] : class Parent1 () : def __init__ (self) : class Parent2 () : def __init__ (self) : class Child (Parent1, Parent2) # This inherits from both Parent1 and Parent2, which __init__ does it use? pass Things can even get worse, as parents can have different signatures of the common method, for example [...] [...] : class Parent1 # This inherits from Ancestor but redefines __init__ : def __init__ (self, status) : class Parent2 # This inherits from Ancestor but redefines __init__ : def __init__ (self, name) : class Child (Parent1, Parent2) # This inherits from both Parent1 and Parent2, which __init__ does it use? pass The problem can be extended even further, introducing a common ancestor above and . Parent1 Parent2 [...] [...] [...] : class Ancestor # The common ancestor, defines its own __init__ method : def __init__ (self) : class Parent1 (Ancestor) # This inherits from Ancestor but redefines __init__ : def __init__ (self, status) : class Parent2 (Ancestor) # This inherits from Ancestor but redefines __init__ : def __init__ (self, name) : class Child (Parent1, Parent2) # This inherits from both Parent1 and Parent2, which __init__ does it use? pass As you can see, we already have a problem when we introduce multiple parents, and a common ancestor just adds a new level of complexity. The ancestor class can clearly be at any point of the inheritance tree (grandparent, grand-grandparent, etc.), the important part is that it is shared between and . This is the so-called diamond problem, as the inheritance graph has the shape of a diamond Parent1 Parent2 Ancestor ^ ^ / \ / \ Parent1 Parent2 ^ ^ \ / \ / Child So, while with single-parent inheritance the rules are straightforward, with multiple inheritance we immediately have a more complex situation that doesn't have a trivial solution. Does all this prevent multiple inheritance from being implemented? Not at all! There are solutions to this problem, as we will see shortly, but this further level of intricacy makes multiple inheritance something that doesn't fit easily in a design and has to be implemented carefully to avoid subtle bugs. Remember that inheritance is an automatic delegation mechanism, as this makes what happens in the code less evident. For these reasons, multiple inheritance is often depicted as scary and convoluted, and usually given some space only in the advanced OOP courses, at least in the Python world. I believe every Python programmer, instead, should familiarise with it and learn how to take advantage of it. ## Multiple inheritance: the Python way Let's see how it is possible to solve the diamond problem. Unlike genetics, we programmers can't afford any level of uncertainty or randomness in our processes, so in the presence of a possible ambiguity as the one created by multiple inheritance, we need to write down a rule that will be strictly followed in every case. In Python, this rule goes by the name of MRO (Method Resolution Order), which was introduced in Python 2.3 and is described in by Michele Simionato. this document There is a lot to say about MRO and the underlying C3 linearisation algorithm, but for the scope of this post, it is enough to see how it solves the diamond problem. In case of multiple inheritance, Python follows the usual inheritance rules (automatic delegation to an ancestor if the attribute is not present locally), but the followed to traverse the inheritance tree now includes all the classes that are specified in the class signature. In the example above, Python would look for attributes in the following order: , , , . order Child Parent1 Parent2 Ancestor So, as in the case of standard inheritance, this means that the first class in the list that implements a specific attribute will be the selected provider for that resolution. An example might clarify the matter print( ) print( ) print( ) print( ) print( ) print( ) print(Child.__mro__) c = Child() c.rewind() c.open() c.close() c.flush() : class Ancestor : def rewind (self) "Ancestor: rewind" : class Parent1 (Ancestor) : def open (self) "Parent1: open" : class Parent2 (Ancestor) : def open (self) "Parent2: open" : def close (self) "Parent2: close" : def flush (self) "Parent2: flush" : class Child (Parent1, Parent2) : def flush (self) "Child: flush" As you can see, we can access the MRO of any class reading its attribute, and as we expected its value is . __mro__ ( , , , , ) So, in this case an instance of provides , , , and . When is called, the code in is executed, as this is the first class in the MRO list that provides that method. The method is provided by , while is provided by . If the method is called, the code is provided by the class itself, that redefines it overriding the one provided by . c Child rewind open close flush c.rewind Ancestor open Parent1 close Parent2 c.flush Child Parent2 As we see with the method, Python doesn't change its behaviour when it comes to method overriding with multiple parents. The first implementation of a method with that name is executed, and the parent's implementation is not automatically called. As in the case of standard inheritance, then, it's up to us to design classes with matching method signatures. flush ### Under the bonnet How does multiple inheritance work internally? How does Python create the MRO list? Python has a very simple approach to OOP (even though it ultimately ends with a mind-blowing ouroboros, see ). Classes are objects themselves, so they contain data structures that are used by the language to provide features, and delegation makes no exception. When we run a method on an object, Python silently uses the method (provided by ), which uses to reach the class from the instance, and to find the parent classes. The latter, in particular, is a tuple, so it is ordered, and it contains all the classes that the current class inherits from. here __getattribute__ object __class__ __bases__ The MRO is created using only , but the underlying algorithm is not that trivial and has to with the monotonicity of the resulting class linearisation. It is less scary than it sounds, but not something you want to read while suntanning, probably. If that's the case, the aforementioned by Michele Simionato contains all the gory details on class linearisation that you always wanted to explore while lying on the beach. __bases__ document # Inheritance and interfaces To approach mixins, we need to discuss inheritance in detail, and specifically the role of method signatures. In Python, when you override a method provided by an ancestor class, you have to decide if and when to call its original implementation. This gives the programmer the freedom to decide whether they need to just augment a method or to replace it completely. Remember that the only thing Python does when a class inherits from another is to automatically delegate methods that are not implemented. When a class inherits from another we are ideally creating objects that keep the backward compatibility with the interface of the parent class, to allow a polymorphic use of them. This means that when we inherit from a class and override a method changing its signature we are doing something that is dangerous and, at least from the point of view of polymorphism, wrong. Have a look at this example self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y super().__init__(pos_x, pos_y, size, size) super().resize(size, size) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : def move (self, pos_x, pos_y) : def resize (self, size_x, size_y) : class Rectangle (GraphicalEntity) pass : class Square (GraphicalEntity) : def __init__ (self, pos_x, pos_y, size) : def resize (self, size) Please note that changes the signature of both and . Now, when we instantiate those classes we need to keep in mind the different signature of in Square __init__ resize __init__ Square r1 = Rectangle( , , , ) r2 = Rectangle( , , , ) q1 = Square( , , ) 100 200 15 30 150 280 23 55 300 400 50 We usually accept that an enhanced version of a class accepts different parameters when it is initialized, as we do not expect it to be polymorphic on . Problems arise when we try to leverage polymorphism on other methods, for example resizing all objects in a list __init__ GraphicalEntity shape [r1, r2, q1]: size_x = shape.size_x size_y = shape.size_y shape.resize(size_x* , size_y* ) for in 2 2 Since , , and are all objects that inherit from we expect them to provide the interface provided by that class, but this fails, because changed the signature of . The same would happen if we instantiated them in a for loop from a list of classes, but as I said it is generally accepted that child classes change the signature of the method. This is not true, for example, in a plugin-based system, where all plugins shall be initialized the same way. r1 r2 q1 GraphicalEntity Square resize __init__ This is a classic problem in OOP. While we, as humans, perceive a square just as a slightly special rectangle, from the interface point of view the two classes are different, and thus should not be in the same inheritance tree when we are dealing with dimensions. This is an important consideration: and are polymorphic on the method, but not on and . So, the question is if we could somehow separate the two natures of being movable and resizeable. Rectangle Square move __init__ resize Now, discussing interfaces, polymorphism, and the reasons behind them would require an entirely separate post, so in the following sections, I'm going to ignore the matter and just consider the object interface optional. You will thus find examples of objects that break the interface of the parent, and objects that keep it. Just remember: whenever you change the signature of a method you change the (implicit) interface of the object, and thus you stop polymorphism. I'll discuss another time if I consider this right or wrong. # Mixin classes MRO is a good solution that prevents ambiguity, but it leaves programmers with the responsibility of creating sensible inheritance trees. The algorithm helps to resolve complicated situations, but this doesn't mean we should create them in the first place. So, how can we leverage multiple inheritance without creating systems that are too complicated to grasp? Moreover, is it possible to use multiple inheritance to solve the problem of managing the double (or multiple) nature of an object, as in the previous example of a movable and resizeable shape? The solution comes from mixin classes: those are small classes that provide attributes but are not included in the standard inheritance tree, working more as "additions" to the current class than as proper ancestors. Mixins originate in the LISP programming language, and specifically in what could be considered the first version of the Common Lisp Object System, the Flavors extension. Modern OOP languages implement mixins in many different ways: Scala, for example, has a feature called , which live in their own space with a specific hierarchy that doesn't interfere with the proper class inheritance. traits ## Mixin classes in Python Python doesn't provide support for mixins with any dedicated language feature, so we use multiple inheritance to implement them. This clearly requires great discipline from the programmer, as it violates one of the main assumptions for mixins: their orthogonality to the inheritance tree. In Python, so-called mixins are classes that live in the normal inheritance tree, but they are kept small to avoid creating hierarchies that are too complicated for the programmer to grasp. In particular, mixins shouldn't have common ancestors other than `object` with the other parent classes. Let's have a look at a simple example self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y self.size_x = size_x self.size_y = size_y rge = ResizableGraphicalEntity( , , , ) rge.resize( , ) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : class ResizableMixin : def resize (self, size_x, size_y) : class ResizableGraphicalEntity (GraphicalEntity, ResizableMixin) pass 5 4 200 300 1000 2000 Here, the class doesn't inherit from , but directly from , so gets from it just the method. As we said before, this simplifies the inheritance tree of and helps to reduce the risk of the diamond problem. It leaves us free to use as a parent for other classes without having to inherit methods that we don't want. Please remember that this happens because the classes are designed to avoid it, and not because of language features: the MRO algorithm just ensures that there will always be an unambiguous choice in case of multiple ancestors. ResizableMixin GraphicalEntity object ResizableGraphicalEntity resize ResizableGraphicalEntity GraphicalEntity Mixins cannot usually be too generic. After all, they are designed to add features to classes, but these new features often interact with other pre-existing features of the augmented class. In this case, the method interacts with the attributes and that have to be present in the object. Obviously, there are obviously examples of mixins, but since they would require no initialization their scope is definitely limited. resize size_x size_y pure ## Using mixins to hijack inheritance Thanks to the MRO, Python programmers can leverage multiple inheritance to override methods that objects inherit from their parents, allowing them to customise classes without code duplication. Let's have a look at this example self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y super().__init__(pos_x, pos_y, size_x, size_y) self.status = self.status = self.status b = Button( , , , ) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : class Button (GraphicalEntity) : def __init__ (self, pos_x, pos_y, size_x, size_y) False : def toggle (self) not 10 20 200 100 As you can see the class extends the one in a classic way, using to call the parent's method before adding the new attribute. Now, if I wanted to create a class I have two choices. Button GraphicalEntity super __init__ status SquareButton I might just override in the new class __init__ self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y super().__init__(pos_x, pos_y, size_x, size_y) self.status = self.status = self.status super().__init__(pos_x, pos_y, size, size) b = SquareButton( , , ) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : class Button (GraphicalEntity) : def __init__ (self, pos_x, pos_y, size_x, size_y) False : def toggle (self) not : class SquareButton (Button) : def __init__ (self, pos_x, pos_y, size) 10 20 200 which performs the requested job, but strongly connects the feature of having a single dimension with the nature. If we wanted to create a circular image we could not inherit from , as the image has a different nature. Button SquareButton The second option is that of isolating the features connected with having a single dimension in a mixin class, and add it as a parent for the new class self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y super().__init__(pos_x, pos_y, size_x, size_y) self.status = self.status = self.status super().__init__(pos_x, pos_y, size, size) b = SquareButton( , , ) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : class Button (GraphicalEntity) : def __init__ (self, pos_x, pos_y, size_x, size_y) False : def toggle (self) not : class SingleDimensionMixin : def __init__ (self, pos_x, pos_y, size) : class SquareButton (SingleDimensionMixin, Button) pass 10 20 200 The second solution gives the same final result, but promotes code reuse, as now the class can be applied to other classes derived from and make them accept only one size, while in the first solution that feature was tightly connected with the ancestor class. SingleDimensionMixin GraphicalEntity Button Please note that the position of the mixin is important as follows the MRO. As it is, the MRO of is , so, when we instantiate it the method is provided by , which in turn calls through the method of . The call in and the signature in match, so everything works. super SquareButton (SquareButton, SingleDimensionMixin, Button, GraphicalEntity, object) __init__ SingleDimensionMixin super __init__ Button super().__init__(pos_x, pos_y, size, size) SingleDimensionMixin def __init__(self, pos_x, pos_y, size_x, size_y): Button If we defined as SquareButton : class SquareButton (Button, SingleDimensionMixin) pass then the method would first be provided by , and its would call the method of . This would however result in an error, as the call in doesn't match the signature of . __init__ Button super __init__ SingleDimensionMixin super().__init__(pos_x, pos_y, size_x, size_y) Button def __init__(self, pos_x, pos_y, size): SingleDimensionMixin Mixins are not used only when you want to change the object's interface, though. Leveraging we can achieve interesting designs like super self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y super().__init__(pos_x, pos_y, size_x, size_y) self.status = self.status = self.status size_x = min(size_x, ) size_y = min(size_y, ) super().__init__(pos_x, pos_y, size_x, size_y) b = LimitSizeButton( , , , ) print(b.size_x) print(b.size_y) : class GraphicalEntity : def __init__ (self, pos_x, pos_y, size_x, size_y) : class Button (GraphicalEntity) : def __init__ (self, pos_x, pos_y, size_x, size_y) False : def toggle (self) not : class LimitSizeMixin : def __init__ (self, pos_x, pos_y, size_x, size_y) 500 400 : class LimitSizeButton (LimitSizeMixin, Button) pass 10 20 2000 1000 Here, the MRO or is , which means that when we initialize it the method is first provided by , which then calls through the method of , and through the latter the method of . LimitSizeButton ( , , , , ) __init__ LimitSizeMixin super __init__ Button __init__ GraphicalEntity Remember that in Python, you are never forced to call the parent's implementation of a method, so the mixin here might also stop the dispatching mechanism if that is the requirement of the business logic of the new object. # A real example: Django class-based views Finally, let's get to the original source of inspiration for this post: the Django codebase. I will show you here how the Django programmers used multiple inheritance and mixin classes to promote code reuse, and you will now hopefully grasp all the reasons behind them. The example I chose can be found in the , and in particular in two classes: and . code of generic views TemplateResponseMixin TemplateView As you might know, Django class is the ancestor of all class-based views and provides a method that converts HTTP request methods into Python function calls ( ). Now, the is a view that answers to a GET request rendering a template with the data coming from a context passed when the view is called. Given the mechanism behind Django views, then, should implement a method and return the content of the HTTP response. The code of the class is View dispatch CODE TemplateView TemplateView get context = self.get_context_data(**kwargs) self.render_to_response(context) : class TemplateView (TemplateResponseMixin, ContextMixin, View) """ Render a template. Pass keyword arguments from the URLconf to the context. """ : def get (self, request, *args, **kwargs) return As you can see is a , but it uses two mixins to inject features. Let's have a look at TemplateView View TemplateResponseMixin [...] [...] [...] : class TemplateResponseMixin : def render_to_response (self, context, **response_kwargs) : def get_template_names (self) [I removed the code of the class as it is not crucial for the present discussion, you can see the full class ] here It is clear that just adds to any class the two methods and . The latter is called in the method of to create the response. Let's have a look at a simplified schema of the calls: TemplateResponseMixin get_template_names render_to_response get TemplateView GET request --> TemplateView.dispatch --> View.dispatch --> TemplateView.get --> TemplateResponseMixin.render_to_response It might look complicated, but try to follow the code a couple of times and the whole picture will start to make sense. The important thing I want to stress is that the code in is available for any class that wants to have the feature of rendering a template, for example ( ), which receives the feature of showing the details of a single object by , which inherits from , overriding its method ( ). TemplateResponseMixin DetailView CODE SingleObjectTemplateResponseMixin TemplateResponseMixin get_template_names CODE As we discussed before, mixins cannot be too generic, and here we see a good example of a mixin designed to work on specific classes. has to be applied to classes that contain ( ), and while this doesn't mean exclusively classes derived from , it is clear that it has been designed to augment that specific type. TemplateResponseMixin self.request CODE View # Takeaway points Inheritance is designed to promote code reuse but can lead to the opposite result Multiple inheritance allows us to keep the inheritance tree simple Multiple inheritance leads to possible problems that are solved in Python through the MRO Interfaces (either implicit or explicit) should be part of your design Mixin classes are used to add simple changes to classes Mixins are implemented in Python using multiple inheritance: they have great expressive power but require careful design. # Final words I hope this post helped you to understand a bit more how multiple inheritance works, and to be less scared by it. I also hope I managed to show you that classes have to be carefully designed and that there is a lot to consider when you create a class system. Once again, please don't forget composition, it's a powerful and too often forgotten tool.