QueueLogger and Python JSON Logger

Using QueueLogger with Python JSON Logger

When logging from multiple processes (via multiprocessing module), using QueueHandler is one approach with Python 2.

QueueHandler however sets exc_info attribute of a LogRecord to None since it is not "pickleable" (more on this later). This becomes a problem when you use python-json-logger to format your logs as JSON since it relies on exc_info being set. The result is you don't get exc_info in your logs. This is no however no longer true since this PR was merged. Sample Code.

Solution #1: Modify Python JSON logger

The first solution is to look for exc_text which is set by the standard Formatter class as per my PR and set that as the value of exc_info which means at least we get the string representation of the traceback. This PR is now merged.

Solution #2: Subclass QueueHandler

The second solution is it to subclass logutils.queue.QueueHandler as follows and add pickling support for exc_info via tblib:

# Bring in support for serializing/deserializing tracebacks
# needed by QueueHandler
from tblib import pickling_support

class QueueHandlerWithTraceback(logutils.queue.QueueHandler):
    """ QueueHandler with support for pickling/unpickling
        Tracebacks via tblib. We only override the prepare()
        method to *not* set `exc_info=None`
    def __init__(self, *args, **kwargs):
        logutils.queue.QueueHandler.__init__(self, *args, **kwargs)

    def prepare(self, record):
        record.msg = record.message
        record.args = None
        return record

Instead of logutils.queue.QueueHandler, we will then use QueueHandlerWithTraceback instead above (Sample Code).

当前网页内容, 由 大妈 ZoomQuiet 使用工具: ScrapBook :: Firefox Extension 从互联网中抓取并分享;
若有不妥, 欢迎评注提醒:

蟒营®编程思维提高班 Python版/第11期 正在报名

精品小班/ 每期<42人

扫描报名: 101camp12py

蟒营®式 原创课程


官网: py.101.camp

Reactivate Joy by Self-teching with You


关注公众号, 持续获得相关各种咨询:


任何问题, 随时邮件提问可也: