◀ Blog

Python Default Logging Module

In Python project, good logs is very essentail not only for debugging but also to provide insight to avoid issue and improve performance. In this articles, I will discuss the best practises for logging with Python and provide some code sample in 3 logging methods.

Good practices

  • Use the Python Standard Logging Module
  • Always keep date in your log file name
  • Always add some name to your log file name.
  • Always log time and date for every log event.
  • Always store your date as YYYYMMDD. Everywhere. In filename, inside of logfile. It greatly helps with sorting. Some separators are allowed (eg. 2009-11-29).
  • In general avoid storing logs in database. In is another point of failure in your logging schema.
  • If you have multiple environment or system always log environment or system id.

Create log

Logging Levels

The log level corresponds to the “importance” a log is given: an “error” log should be more urgent then than the “warn” log, whereas a “debug” log should be useful only when debugging the application.

There are six log levels in Python; each level is associated with an integer that indicates the log severity:

  • NOTSET=0
  • DEBUG=10 : for debugging purposes in development
  • INFO=20 : for something interesting—but expected—happens
  • WARN=30 : for something unexpected or unusual happens
  • ERROR=40 : for things that go wrong but are usually recoverable
  • CRITICAL=50 : for a serious scenario
In [1]:
import logging

logging.debug('debug message.')
logging.info('informational message.')
logging.debug('message for debugging purposes.')
logging.warning('warning!')
logging.critical('critial!')

WARNING:root:warning!
CRITICAL:root:critial!

The main components of the logging module are loggers, handlers, and formatters.

  • Handlers is the component that effectively writes/displays a log: Display it in the console (via StreamHandler), in a file (via FileHandler), or even by sending you an email via SMTPHandler, etc.
  • Loggers are the objects you call when you want to record a message in your application code.
  • Formatters are responsible for formatting the layout of log messages.

Configuring Logging

Programmers can configure logging in three ways:

  1. Creating loggers, handlers, and formatters explicitly using Python code that calls the configuration methods listed above.

  2. Creating a logging config file and reading it using the fileConfig() function.

  3. Creating a dictionary of configuration information and passing it to the dictConfig() function.

1. Creating loggers, handlers, and formatters

In [2]:
import os
import logging
from datetime import datetime

def create_logging():
  """ Create logging """
  logs_folder = 'logs/'
  if not os.path.exists(logs_folder):
    os.makedirs(logs_folder)

  log_dir = logs_folder + str(datetime.now().strftime('logfile_ws02_%Y%m%d.log'))
  logging.basicConfig(filename=log_dir,
                  level=logging.INFO,
                  format='%(asctime)s | %(name)s | %(levelname)s | %(message)s')
  console = logging.StreamHandler()
  console.setLevel(logging.INFO)
  formatter = logging.Formatter('%(asctime)s | %(name)s | %(levelname)s | %(message)s')
  console.setFormatter(formatter)
  logging.getLogger('').addHandler(console)
  logging.info('Started')
  logging.info('Log file : ' + str(log_dir))

create_logging()
logging.debug('debug message.')
logging.info('informational message.')
logging.debug('message for debugging purposes.')
logging.warning('warning!')
logging.critical('critial!')

WARNING:root:warning!

2020-09-18 04:08:34,780 | root | INFO | Started

2020-09-18 04:08:34,781 | root | INFO | Log file : logs/logfile_ws02_20200918.log

2020-09-18 04:08:34,784 | root | INFO | informational message.

2020-09-18 04:08:34,787 | root | WARNING | warning!

2020-09-18 04:08:34,789 | root | CRITICAL | critial!

2. Creating a logging config file

Creating a logging config file and reading it using the fileConfig() function.

logging.conf

[loggers]
keys=root

[handlers]
keys=fileHandler, consoleHandler

[formatters]
keys=logFormatter, consoleFormatter

[logger_root]
handlers=fileHandler, consoleHandler
level=NOTSET

[handler_consoleHandler]
class=StreamHandler
level=INFO
formatter=consoleFormatter
args=(sys.stdout,)

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=logFormatter
args=(__import__("datetime").datetime.now().strftime('logs/logfile_%%Y%%m%%d.log'), 'a')

[formatter_logFormatter]
format=%(asctime)s | %(name)s | %(levelname)-8s | %(lineno)04d | %(message)s

[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(message)s
In [3]:
import logging
from datetime import datetime
import logging.config


logging.config.fileConfig('logging.conf')

# create logger
logger = logging.getLogger('log-test')

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

2020-08-29 15:49:46,602 | INFO     | info message
2020-08-29 15:49:46,603 | WARNING  | warn message
2020-08-29 15:49:46,604 | ERROR    | error message
2020-08-29 15:49:46,605 | CRITICAL | critical message

logfile_20200829.log

2020-08-29 22:52:13,443 | log-test | DEBUG | 0013 | debug message

2020-08-29 22:52:13,443 | log-test | INFO | 0014 | info message

2020-08-29 22:52:13,443 | log-test | WARNING | 0015 | warn message

2020-08-29 22:52:13,443 | log-test | ERROR | 0016 | error message

2020-08-29 22:52:13,443 | log-test | CRITICAL | 0017 | critical message

3. Creating a dictionary of configuration information

In [4]:
import logging
import logging.config
from datetime import datetime

LOGGING_CONFIG = {
    'version': 1,
    'loggers': {
        '': {  # root logger
            'level': 'NOTSET',
            'handlers': ['fileHandler', 'consoleHandler'],
        },
        'handler_fileHandler': {
            'level': 'WARNING',
            'propagate': False,
            'handlers': ['fileHandler', 'consoleHandler' ],
        },
    },
    'handlers': {
        'consoleHandler': {
            'level': 'INFO',
            'formatter': 'formatter_consoleFormatter',
            'class': 'logging.StreamHandler',
            'stream': 'ext://sys.stdout',
        },
        'fileHandler': {
            'level': 'NOTSET',
            'formatter': 'formatter_logFormatter',
            'class': 'logging.FileHandler',
            'filename': datetime.now().strftime('logs/logfile_%Y%m%d.log'),
            'mode': 'a'
        }
    },
    'formatters': {
        'formatter_logFormatter': {
            'format': '%(asctime)s | %(name)s | %(levelname)-8s | %(lineno)04d | %(message)s'
        },
        'formatter_consoleFormatter': {
            'format': '%(asctime)s | %(levelname)-8s | %(message)s'
        },
    },

}

# Run once at startup:
logging.config.dictConfig(LOGGING_CONFIG)

# Include in each module:
logger = logging.getLogger('test-log')

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

2020-08-29 15:49:46,638 | INFO     | info message
2020-08-29 15:49:46,639 | WARNING  | warn message
2020-08-29 15:49:46,641 | ERROR    | error message
2020-08-29 15:49:46,642 | CRITICAL | critical message
In [ ]: