Logging to AWS CloudWatch Logs with a Python Logger

2020-07-18 13:50:05 | #sysadmin #programming #aws #logging

Tested On

  • Linux Ubuntu 20.04
  • Windows 10
  • macOS Catalina

Problem: You want a Python 3 logger module that allows you to output JSON-formatted logs that are compatible with stdout, and AWS Cloudwatch logs.

Solution: We're going to write our own logger module in Python 3 that is easy to import into any Python 3 project.

How to Set Up the Cloudwatch Python Logger Project Files

How to Create Python Project Files with Windows 10 PowerShell 2.0+

cd ~
New-Item -ItemType "directory" -Path ".\python-cloudwatch-logger"
cd python-cloudwatch-logger
New-Item -ItemType "file" -Path . -Name "main.py"
New-Item -ItemType "file" -Path . -Name "logger.py"
virtualenv venv
.\venv\Scripts\activate

To verify that the virtual environment is active, make sure (venv) is in the PowerShell command prompt. For example, (venv) PS C:\Users\username\python-cloudwatch-logger>

How to Create Python Project Files with Linux Ubuntu 14.04+ or macOS

cd ~
mkdir python-cloudwatch-logger
cd python-cloudwatch-logger
virtualenv -p python3 venv
source venv/bin/activate
touch main.py
touch logger.py

To verify that the virtual environment is active, make sure (venv) is in the terminal command prompt.

This will create the following files and folders, and activate the virtual environment.

▾ python-cloudwatch-logger/
  ▸ venv/
  logger.py
  main.py

Full Code For Our Reusable Python 3 Logger Module

Filename: logger.py

import os
import re
import sys
import json
import time
import logging


class JSONFormatter(logging.Formatter):
    def format(self, record):
        string_formatted_time = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created))
        obj = {}
        obj["message"] = record.msg
        obj["level"] = record.levelname
        obj["time"] = f"{string_formatted_time}.{record.msecs:3.0f}Z"
        obj["epoch_time"] = record.created
        if hasattr(record, "custom_logging"):
            for key, value in record.custom_logging.items():
                obj[key] = value
        return json.dumps(obj)


def striplines(m):
    m = re.compile(r'[\t]').sub(' ', str(m))
    return re.compile(r'[\r\n]').sub('', str(m))


def ex(e):
    logger.exception(striplines(e))


def info(msg):
    logger.info(msg)


def err(msg):
    logger.error(msg)


def debug(msg):
    logger.debug(msg)


logger = logging.getLogger(__name__)
logger.propagate = False  # remove default logger
if logger.handlers:
    for handler in logger.handlers:
        logger.removeHandler(handler)

handler = logging.StreamHandler(sys.stdout)
formatter = JSONFormatter()
handler.setFormatter(formatter)

logger.addHandler(handler)

if "DEBUG" in os.environ and os.environ["DEBUG"] == "true":
    logger.setLevel(logging.DEBUG)
else:
    logger.setLevel(logging.INFO)
    logging.getLogger("boto3").setLevel(logging.WARNING)
    logging.getLogger("botocore").setLevel(logging.WARNING)

Explanation Of The logger.py Code

Lines 1-6: imports all of the necessary dependencies, including the Logging facility for Python.

Lines 9-20: extends the logging.Formatter class with our own custom JSONFormatter class. This gives our logs some structure, allowing us to read the logging level, timestamp, and message with JSON notation.

Lines 23-25: defines a function that strips the linebreaks from messages, ensuring that each log statement corresponds to one message, and doesn't overflow into multiple Cloudwatch log messages.

Lines 28-41: defines convenient functions for capturing logs at each level. So to log an exception event, you would call ex('Example exception message'). And to log something informational, you would call info('Example informational message').

Lines 44-61: are all related to instantiating the logging facility which we store in the module's logger variable. We won't ever need to invoke this directly because we'll rely on our convenience functions, ex, info, err, and debug at the module level. If this sounds complicated, we have examples further down in this tutorial.

Lines 45-61: are very important for addressing an issue that many developers have faced, using Python logging with AWS Lambda, where Python logging doesn't seem to work in Coudwatch. These lines remove the default AWS Lambda Python runtime logging handler and overrides it with our own configuration.

Full Code For main.py

Filename: main.py

import logger

if __name__ == '__main__':
      logger.info('Information')
      logger.ex('Exception')
      logger.err('Error')
      logger.debug('Debug')

Running the Python Program

Now, run python main.py to execute the program and see the log statements printed to the console. You can import logger.py into any Python program you create. If you import it into a program you deploy to AWS Lambda, you will see log statements printed as JSON under Cloudwatch logs.

Conclusion

That's the end of this tutorial. We hope you found it helpful. Make sure to check out our other tutorials, as well.

If you're interested in programs that carry out your computer tasks for you, take our Automation the Easy Way with Python course. This course teaches CSV and Excel file generation, API requests, website scraping, email delivery, task scheduling, and browser click, mouse, and keyboard automation. Automate your daily tasks, free up time, and get ahead, today.

Want To See More Exercises?

View Exercises View Courses

Comments

You must log in to comment. Don't have an account? Sign up for free.

Subscribe to comments for this post

Want To Receive More Free Content?

Would you like to receive free resources, tailored to help you reach your IT goals? Get started now, by leaving your email address below. We promise not to spam. You can also sign up for a free account and follow us on and engage with the community. You may opt out at any time.



Tell Us About Your Project









Contact Us

Do you have a specific IT problem that needs solving or just have a general IT question? Use the contact form to get in touch with us and an IT professional will be with you, momentarily.

Hire Us

We offer web development, enterprise software development, QA & testing, google analytics, domains and hosting, databases, security, IT consulting, and other IT-related services.

Free IT Tutorials

Head over to our tutorials section to learn all about working with various IT solutions.

Contact