Go back

Memory Profiling in Python

CPU, Memory, and other issues are always present when profiling applications. Python apps, on the other hand, are prone to memory management issues. This is primarily due to the fact that Python is used in Data Science and Machine Learning applications and works with massive amounts of data. Furthermore, Python relies on its Memory Management system by default, rather than leaving it up to the user. However, this trust can be risky in some cases.

To do that, we will concern a special modulenamed‘memory-profiler’. This library is implemented by Fabian Pedregosa andthecontributors. Basically, it is a pure module that depends on the psutilmodulefrom the standard libraries inside Python but it is important to rememberthatthere are tones of different methods to profile memory usage such ascProfile,tracemalloc, etc.

The reason we picked this special library that it is easy touseand understandable structure to see line by line in our python file. Also,itprovides to log our memory usage easily.

Firstly, we need to install this package with pip below:

pip install –U memory_profiler

Then, we can start to check the sample implementation below:

We can now see the memory usage for this file, line by line, in the screenshot of our log output below:

NOTE: It should also be noted that these processes are donewith Python's version 3.9.2.

References: 

https://pypi.org/project/memory-profiler/

https://github.com/pythonprofilers/memory_profiler

https://towardsdatascience.com/profile-memory-consumption-of-python-functions-in-a-single-line-of-code-6403101db419?gi=f648adf9b303

https://pythonrepo.com/repo/pythonprofilers-memory_profiler-python-monitoring

Share

Our expertises