Logging to AWS CloudWatch Logs with a Python Logger

2020-07-18 13:50:05 | #sysadmin #programming

Tested On

  • Linux Ubuntu 20.04

Problem: You want a Python 3 logger module that allows you to output JSON-formatted logs that are compatible with stdout, and AWS Cloudwatch logs.

Solution: We're going to write our own logger module in Python 3 that is easy to import into any Python 3 project.

Creating the Project

We're going to set up a project skeleton with some necessary files and folders. This will allow you to install dependencies into the project and execute functions in the logger module through a main.py file.

First, execute the following commands:

cd ~
mkdir python-logging-to-cloudwatch
cd python-logging-to-cloudwatch
virtualenv -p python3 venv
source venv/bin/activate
touch main.py
touch logger.py

This will generate the following project files and activate the virtual environment.

▾ python-logging-to-cloudwatch/
  ▸ venv/
  logger.py
  main.py

Full Code For Our Reusable Python 3 Logger Module

Filename: logger.py

import os
import re
import sys
import json
import time
import logging


class JSONFormatter(logging.Formatter):
    def format(self, record):
        string_formatted_time = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created))
        obj = {}
        obj["message"] = record.msg
        obj["level"] = record.levelname
        obj["time"] = f"{string_formatted_time}.{record.msecs:3.0f}Z"
        obj["epoch_time"] = record.created
        if hasattr(record, "custom_logging"):
            for key, value in record.custom_logging.items():
                obj[key] = value
        return json.dumps(obj)


def striplines(m):
    m = re.compile(r'[\t]').sub(' ', str(m))
    return re.compile(r'[\r\n]').sub('', str(m))


def ex(e):
    logger.exception(striplines(e))


def info(msg):
    logger.info(msg)


def err(msg):
    logger.error(msg)


def debug(msg):
    logger.debug(msg)


logger = logging.getLogger(__name__)
logger.propagate = False  # remove default logger
if logger.handlers:
    for handler in logger.handlers:
        logger.removeHandler(handler)

handler = logging.StreamHandler(sys.stdout)
formatter = JSONFormatter()
handler.setFormatter(formatter)

logger.addHandler(handler)

if "DEBUG" in os.environ and os.environ["DEBUG"] == "true":
    logger.setLevel(logging.DEBUG)
else:
    logger.setLevel(logging.INFO)
    logging.getLogger("boto3").setLevel(logging.WARNING)
    logging.getLogger("botocore").setLevel(logging.WARNING)

Explanation Of The logger.py Code

Lines 1-6: imports all of the necessary dependencies, including the Logging facility for Python.

Lines 9-20: extends the logging.Formatter class with our own custom JSONFormatter class. This gives our logs some structure, allowing us to read the logging level, timestamp, and message with JSON notation.

Lines 23-25: defines a function that strips the linebreaks from messages, ensuring that each log statement corresponds to one message, and doesn't overflow into multiple Cloudwatch log messages.

Lines 28-41: defines convenient functions for capturing logs at each level. So to log an exception event, you would call ex('Example exception message'). And to log something informational, you would call info('Example informational message').

Lines 44-61: are all related to instantiating the logging facility which we store in the module's logger variable. We won't ever need to invoke this directly because we'll rely on our convenience functions, ex, info, err, and debug at the module level. If this sounds complicated, we have examples further down in this tutorial.

Lines 45-61: are very important for addressing an issue that many developers have faced, using Python logging with AWS Lambda, where Python logging doesn't seem to work in Coudwatch. These lines remove the default AWS Lambda Python runtime logging handler and overrides it with our own configuration.

Full Code For main.py

Filename: main.py

import logger

if __name__ == '__main__':
      logger.info('Information')
      logger.ex('Exception')
      logger.err('Error')
      logger.debug('Debug')

Running the Python Program

Now, run python main.py to execute the program and see the log statements printed to the console. You can import logger.py into any Python program you create. If you import it into a program you deploy to AWS Lambda, you will see log statements printed as JSON under Cloudwatch logs.

Comments

You must log in to comment. Don't have an account? Sign up for free.

Subscribe to Our Newsletter

Would you like to receive free whitepapers and other IT news? Just leave your email address below. You may opt out at any time.



Tell Us About Your Project









Contact Us

Do you have a specific IT problem that needs solving or just have a general IT question? Use the contact form to get in touch with us and an IT professional will be with you, momentarily.

Hire Us

We offer web development, enterprise software development, QA & testing, google analytics, domains and hosting, databases, security, IT consulting, and other IT-related services.

Free IT Tutorials

Head over to our tutorials section to learn all about working with various IT solutions.

Contact