When a programming language uses “pass by reference”, it means that when you pass a variable as an argument to a function, you are actually passing a reference to the memory location of that variable, rather than a copy of its value. Because of this, any changes made to the variable inside the function will reflect on the original variable outside the function.
In simple terms, think of a variable as a box containing a value. Instead of handing over a separate box with the same contents (as is the case with “pass by value”), “pass by reference” gives someone a direct link or pointer to the original box. Any modifications made through this link will directly affect the contents of that original box.
It’s worth noting that not all languages strictly adhere to “pass by reference” or “pass by value”. Python, for example, has a unique behavior often termed as “pass by assignment” or “pass by object reference”. In Python’s case, whether the behavior seems like “pass by reference” or “pass by value” depends on whether the data type of the variable is mutable or immutable.
Python, as a high-level programming language, abstracts many of the intricate details of memory management to simplify the development process for programmers.
Yet, understanding how Python handles memory and variables can be invaluable in certain scenarios, such as debugging, optimizing performance, and ensuring efficient use of resources. In this exposition, we’ll dive deep into Python’s memory management mechanism and use code to bring these concepts to life.
Python Manages Memory and Variables by following methods
- Objects and Reference Counting
- Garbage Collection
- Garbage Collection
- Immutable vs Mutable Objects
- Memory Pools and Block Allocation
- Dynamic Typing
Now I will explain these, one by one in more detail. Let’s start with that.
1. Objects and Reference Counting:
In Python, everything is an object. Every integer, string, list, or function is an instance of an object.
x = 10 # Here, x is a reference to an object containing the integer 10
Python uses reference counting to track the number of references that point to an object.
y = x # Now, both x and y reference the same integer object
The memory for objects with zero references can be reclaimed.
2. Garbage Collection:
- Apart from reference counting, Python also has a garbage collector to detect and clean up objects that have circular references (i.e., they reference each other) and therefore would not be cleaned up by just using reference counting.
- The garbage collector mainly works by identifying unreachable objects and making sure their memory is released.
a = []
b = [a]
a.append(b) # a and b now reference each other, forming a circular reference
3. Variable Assignment:
- When a variable is assigned to an object in Python, the variable essentially points or references that object in memory.
- For instance, when you do
a = [1, 2, 3], the variableareferences the list object[1, 2, 3]in memory. - If you then do
b = a, bothaandbpoint to the same object in memory. This does not create a new list but merely a new reference to the same list.
list1 = [1, 2, 3]
list2 = list1 # list2 now references the same list object as list1
list2[0] = 99 # This will modify the original list object
print(list1) # Outputs: [99, 2, 3]
4. Immutable vs Mutable Objects:
- Python has both immutable objects (e.g., strings, integers, tuples) and mutable objects (e.g., lists, dictionaries).
- Immutable objects can’t be changed after they are created. Any operation that appears to modify them actually returns a new, modified object, leaving the original object unchanged.
- Mutable objects, on the other hand, can be modified in place. Therefore, changes made through one reference will be visible through all other references to the same object.
str1 = "hello"
str2 = str1
str2 = "world" # This creates a new string object, str1 remains unchanged
5. Memory Pools and Block Allocation:
- Python uses a system of memory pools for integer and small-sized object allocation. This system helps reduce the overhead of memory allocation by reusing blocks of memory.
- Large objects get their own dedicated memory chunks outside the pool.
# Most integers and small objects benefit from this pooled memory system
int1 = 5
int2 = 5 # int2 will likely point to the same memory location as int1
6. Dynamic Typing:
Variables in Python do not have a fixed type. Instead, they can reference objects of any type, and their type can change during the program’s execution. This dynamic typing nature can lead to more memory overhead because the objects have to store type information.
var = "hello" # var is a string
var = 100 # now, var is an integer
Understanding Python's Data Model
Python’s data model is the foundation upon which the entire language operates. It provides a consistent and unified framework for interacting with objects and data structures in the language.
By understanding the data model, one gains insights into the fundamental building blocks of the language and how they interact with one another. Let’s delve into the core aspects of Python’s data model:
Objects:
At the heart of Python’s data model is the notion that “everything is an object.” Whether it’s a basic type like an integer or a complex data structure like a list, everything in Python is an instance of some class or type.
x = 10 # x is an object of type 'int'
y = [1, 2, 3] # y is an object of type 'list'
Identity, Type, and Value:
Every object in Python possesses these three essential attributes:
- Identity: A unique identifier, typically the object’s memory address, which remains constant throughout the object’s lifetime. You can obtain an object’s identity using the
id()function. - Type: Represents the object’s data type (e.g., int, string, list). It determines the operations that the object supports. Once an object is created, its type cannot change. The
type()function returns an object’s type. - Value: The content or data the object holds. Depending on whether the object is mutable (like a list) or immutable (like an integer or string), this value might or might not be changeable.
z = "hello"
print(id(z)) # Outputs the identity (memory address) of z
print(type(z)) # Outputs: <class 'str'>
Attributes and Methods:
- Objects can possess attributes (data) and methods (functions) associated with them, depending on their type/class definition.
- Use the dot (
.) operator to access an object’s attributes and methods.
l = [1, 2, 3]
l.append(4) # 'append' is a method associated with the list object
Special Methods (Magic Methods or Dunder Methods):
- These are methods with double underscores before and after their names, such as
__init__,__str__, and__len__. - They provide a way to define how objects of a class behave with built-in operations like
print()or arithmetic operations. For example, the__len__method allows us to use thelen()function on objects of a class.
class MyClass:
def __len__(self):
return 5
obj = MyClass()
print(len(obj)) # Outputs: 5, thanks to the defined __len__ method
Protocols:
- Protocols are essentially the “contracts” that classes can choose to adhere to. By implementing certain dunder methods, a class says it supports a particular protocol.
- For example, an object is considered iterable if its class implements the
__iter__method, adhering to the iterable protocol.
Immutable vs Mutable Data Types
In Python, data types are categorized into two main categories based on whether the value they hold can be changed or not after they are created:
Immutable: These data types cannot be altered once they’ve been assigned a value. If you try to change their value, a new object is created instead. Common immutable types include:
- Integers
- Floats
- Strings
- Tuples
- Frozen sets
- Booleans
Mutable: These data types can be altered or modified after they’ve been assigned a value. Common mutable types include:
- Lists
- Dictionaries
- Sets
Examples:
Immutable Data Types:
String:
s = "hello"
print(id(s)) # Let's say this outputs: 140183202512144
s += " world"
print(id(s)) # Outputs a different identity, e.g., 140183202519184
In the above code, when we modify the string s, a new string object is created. The identity of s changes, confirming its immutability.
Tuple:
t = (1, 2, 3)
# t[0] = 99 # This would raise a TypeError, as tuples are immutable
Mutable Data Types:
List:
lst = [1, 2, 3]
print(id(lst)) # Let's say this outputs: 140183203080704
lst.append(4)
print(id(lst)) # Outputs the same identity: 140183203080704
Here, even after modifying the list, the identity remains the same, indicating that the list is mutable and the changes were made in place.
Dictionary:
d = {"name": "John", "age": 25}
print(id(d)) # Let's say this outputs: 140183203112576
d["age"] = 26
print(id(d)) # Outputs the same identity: 140183203112576
Similarly, the dictionary is mutable, as demonstrated by its consistent identity despite modifications.
Memory addresses and the id() function
In computer memory, every piece of data, be it a single integer or a complex data structure, is stored at a specific location. This location is often represented as a unique number called the memory address. Think of computer memory as a vast array of storage boxes, where each box has a unique number, and the contents of the box are the data values.
Python abstracts away many of the intricate details of memory management to make programming more straightforward. However, it does provide a way to peek behind the curtain and see the memory address of objects, and this is where the id() function comes into play.
The id() Function:
The built-in id() function in Python is used to obtain the “identity” of an object, which is unique and constant for the object during its lifetime. This “identity” is, in fact, the memory address where the object is stored.
x = 10
print(id(x)) # This might output a number like: 140721687216992
In the above example, the id() function returns the memory address of the integer object 10 that the variable x is pointing to.
Examples and Observations:
Immutable Objects:
For immutable objects like integers or strings, if two variables have the same value, they might point to the same memory address (for reasons of optimization). However, this is not guaranteed, especially for larger or more complex immutable objects.
a = "hello"
b = "hello"
print(id(a) == id(b)) # This might output: True
Mutable Objects:
Mutable objects like lists or dictionaries always have distinct memory addresses when created separately, even if their contents are the same.
list1 = [1, 2, 3]
list2 = [1, 2, 3]
print(id(list1) == id(list2)) # This will output: False
Variables and Memory Addresses:
When you assign one variable’s value to another, both variables point to the same memory address, especially in the case of mutable objects.
list1 = [1, 2, 3]
list3 = list1
print(id(list1) == id(list3)) # This will output: True
The Myth of Pass-by-Value or Pass-by-Reference in Python
One of the common points of confusion among new Python programmers is how arguments are passed to functions — is it by value or by reference? Let’s break this down.
Definitions:
Pass-by-Value:
- The called function receives a copy of the actual data, meaning changes made inside the function don’t affect the original data outside the function.
Pass-by-Reference:
- The called function receives a reference to the actual data, meaning changes made inside the function directly affect the original data outside the function.
Python's Approach: Pass-by-Object-Reference:
Python’s argument-passing strategy isn’t strictly pass-by-value or pass-by-reference. Instead, it’s often described as pass-by-object-reference. This means when you pass a variable to a function:
- You’re actually passing the reference (memory address) of the object the variable refers to, not the actual object itself.
- Whether the original data outside the function is affected by changes inside the function depends on the mutability of the object being referenced.
Examples:
Immutable Objects (like integers, strings):
For immutable objects, even though they are passed by object reference, changes inside a function won’t affect the original data. This behavior is similar to pass-by-value.
def modify_data(x):
print(f"Initial id inside function: {id(x)}")
x += 1
print(f"Modified id inside function: {id(x)}")
return x
num = 10
print(f"Original id outside function: {id(num)}")
new_num = modify_data(num)
print(f"Num after function call: {num}")
Here, the integer inside the function is a new object, and num outside remains unchanged.
Mutable Objects (like lists, dictionaries):
For mutable objects, changes inside a function directly affect the original data, akin to pass-by-reference behavior.
def modify_list(lst):
print(f"List id inside function: {id(lst)}")
lst.append(4)
numbers = [1, 2, 3]
print(f"Original list id outside function: {id(numbers)}")
modify_list(numbers)
print(f"List after function call: {numbers}")
Here, the list inside and outside the function refers to the same object, and changes in the function reflect outside.
Mutable Objects
In programming, when we say an object is mutable, we mean that its state or value can be changed after it has been created. This contrasts with immutable objects, whose state cannot be changed after creation. In Python, common mutable data types include lists, dictionaries, and sets.
1. Lists
Definition: A list in Python is an ordered collection of items that can be of any type. Lists are defined by enclosing the items (elements) in square brackets
[].
Example:
fruits = ["apple", "banana", "cherry"]
You can modify a list by adding, removing, or changing items. For instance:
fruits.append("orange") # Add an item
fruits[1] = "blueberry" # Change an item
2. Dictionaries
Definition: A dictionary in Python is an unordered collection of data values used to store data values like a map. It’s defined by enclosing items (key-value pairs) in curly braces
{}.
Example:
person = {"name": "John", "age": 25}
You can modify a dictionary by adding, removing, or changing items. For instance:
person["age"] = 30 # Change an item's value
person["gender"] = "male" # Add a new key-value pair
3. Sets
Definition: A set in Python is an unordered collection of unique items. Sets are defined by enclosing the items (elements) in curly braces {} or by using the set() constructor.
Example:
prime_numbers = {2, 3, 5, 7}
You can modify a set by adding or removing items. For instance:
prime_numbers.add(11) # Add an item
prime_numbers.remove(2) # Remove an item
Modifying Mutable Objects Within Functions
When you pass a mutable object to a function, the function works with a reference to the original object. As a result, any changes made to the object within the function are reflected in the original object outside the function.
Example: Modifying a list within a function:
def add_fruit(fruit_list, fruit):
fruit_list.append(fruit)
fruits = ["apple", "banana", "cherry"]
print(f"Original list: {fruits}") # Outputs: ['apple', 'banana', 'cherry']
add_fruit(fruits, "orange")
print(f"List after function call: {fruits}") # Outputs: ['apple', 'banana', 'cherry', 'orange']
Effect on the original list: In the above example, when we passed the fruits list to the add_fruit function and appended an “orange”, the original fruits list outside the function was modified. This illustrates the nature of mutable objects and how functions work with references to objects rather than copies.
Immutable Objects
Immutable objects, once created, cannot be changed, modified, or altered. Their state or value remains constant after their creation. This can be contrasted with mutable objects, which can be modified after creation.
Integers
Integers in Python represent whole numbers, both positive and negative.
Example:
num = 5
Although you can reassign the value of num to another integer, the integer object 5 itself is immutable.
Strings
A string in Python is a sequence of characters. Strings are defined by enclosing characters in quotes.
Example:
greeting = "Hello, world!"
Even if it seems you’re modifying a string using some operations, you’re in reality creating a new string.
Tuples
A tuple in Python is similar to a list but is immutable. Tuples are defined by enclosing the items (elements) in parentheses ().
Example:
coordinates = (4, 5)
You cannot change the elements of a tuple once it’s defined.
Trying to Modify Immutable Objects Within Functions
When you pass an immutable object to a function, the function works with its value, but any attempt to modify that object creates a new object.
Example: Trying to modify a string within a function:
def modify_string(s):
s = "Goodbye"
print("Inside function:", s)
greeting = "Hello"
print("Original string:", greeting) # Outputs: Hello
modify_string(greeting)
print("String after function call:", greeting) # Outputs: Hello
Why the Original String Remains Unchanged:
In the example above, when we pass the greeting string to the modify_string function, any modification inside the function doesn’t affect the original string. Instead, the variable inside the function gets associated with a new string object, leaving the original string untouched. This behavior exemplifies the immutable nature of strings.
Pitfalls and Common Mistakes
Confusion between reassignment and modification: For immutable types, reassignment (e.g.,
num = num + 1) creates a new object. It doesn’t modify the existing object. This is different from the behavior of mutable objects, where methods likeappendorextendchange the object’s state.Expecting immutables to act like mutables: A common mistake is expecting that operations on immutable types, especially within functions, will change the original data. As seen in the string example above, this isn’t the case.
Tuple with mutable items: While tuples themselves are immutable, they can contain mutable objects, like lists. This can lead to confusion because the tuple’s content can change if those mutable objects are modified.
tuple_with_list = ([1, 2, 3],)
tuple_with_list[0].append(4)
print(tuple_with_list) # Outputs: ([1, 2, 3, 4],)
Here, the list inside the tuple was modified, even though the tuple itself is immutable.
Modifying Mutable Objects vs Reassigning Reference
In Python, there’s a distinction between modifying the content of a mutable object and reassigning a variable’s reference.
Example:
Changing an entire list:
fruits = ["apple", "banana"]
fruits = ["cherry", "date"]
Changing an element inside the list:
fruits[0] = "grape"
In the first example, you’re reassigning the variable’s reference to a new list. In the second, you’re modifying the content of the original list.
Unintended Side Effects When Modifying Mutable Objects in Functions: When you pass mutable objects (like lists) to a function and modify them inside, you affect the original object.
def add_item(lst, item):
lst.append(item)
fruits = ["apple"]
add_item(fruits, "banana")
print(fruits) # ["apple", "banana"]
How to Mimic Pass by Value in Python
To prevent unintended modifications, you can create copies.
Copying Objects Using the Copy Module:
Shallow Copy: Copies the top-level object, but not the nested ones.
import copy
fruits1 = ["apple", ["banana", "cherry"]]
fruits2 = copy.copy(fruits1)
Deep Copy: Copies all nested objects.
fruits3 = copy.deepcopy(fruits1)
When to Use Shallow vs Deep Copies:
- Use a shallow copy when you want to clone the main object, but are okay with sharing nested objects.
- Use a deep copy when you want a fully independent clone.
Best Practices
Use Clear Function Documentation: Ensure the behavior of functions is documented, especially if they modify input arguments.
Return Modified Objects to Make Changes Explicit: Instead of modifying input directly, return the modified data.
Avoid Modifying Input Arguments Unless Necessary: Minimize side effects by not changing inputs unless it’s integral to the function’s purpose.
Real-World Examples
- Using Functions to Modify List Contents:
def process_data(data_list):
return [item * 2 for item in data_list]
- Using Functions to Modify Dictionary Key-Values:
def update_prices(prices_dict, discount):
return {key: value * (1 - discount) for key, value in prices_dict.items()}
Pitfalls in Real-world Scenarios and How to Avoid Them:
- Watch out for modifying mutable defaults in function arguments.
- Ensure that shared mutable objects are treated with caution.
Comparing Python’s Mechanism with Other Languages
Python vs C++: Unlike Python, C++ has explicit pointers and can directly use pass-by-value and pass-by-reference.
Python vs Java: Java uses pass-by-value, but when it comes to objects, it passes the value of the reference.
Summary and Key Takeaways
- Python doesn’t strictly use pass-by-value or pass-by-reference but rather pass-by-object-reference.
- The effects of modifying data in functions are influenced by mutability.
Further Reading and Resources
- Books: “Python Cookbook” by David Beazley, “Fluent Python” by Luciano Ramalho
- Articles: “Python Passing by Reference”, available on GeeksforGeeks and W3Schools
- Online Resources: Python’s official documentation on data models and the
copymodule.
F.A.Q.
Live Python Tutorials
Python is neither strictly pass-by-value nor strictly pass-by-reference. Instead, Python’s parameter passing is best described as “pass-by-object-reference”.
Integers and strings are immutable in Python. This means their values can’t be changed once they are created. Any modification creates a new object instead.
Lists are mutable objects. When you pass a list to a function, you’re actually passing a reference to that list. So, any changes made to the list inside the function reflect outside the function as well.
You can pass a copy (either shallow or deep) of the object to the function. This way, the original object remains unaffected by changes inside the function.
A shallow copy creates a new object, but doesn’t create copies of nested objects that reside within the original object. A deep copy, on the other hand, creates a new object and recursively adds copies of nested objects found in the original.
Yes, creating a deep copy can be slower, especially if the original object is large or has many nested objects. Use deep copies judiciously.
In C++, you can explicitly use pass-by-value or pass-by-reference. Java uses pass-by-value, but when passing objects, it’s the value of the reference that gets passed. Python’s mechanism is unique in that it’s always pass-by-object-reference.
Ensure that you’re not modifying mutable default values in function arguments and that shared mutable objects are treated with caution. Also, consult the tutorial’s section on common pitfalls and mistakes.
Refer to the “Further Reading and Resources” section of the tutorial. Books like “Fluent Python” provide in-depth discussions on Python’s data model and behaviors.
While the foundational ideas around mutability, references, and memory management are relevant in many programming contexts, the specific behaviors and best practices are tailored to Python.
