Using QueueLogger with Python JSON Logger
When logging from multiple processes (via multiprocessing module), using QueueHandler is one approach with Python 2.
QueueHandler however sets exc_info attribute of a LogRecord to None since it is not "pickleable" (more on this later). This becomes a problem when you use python-json-logger to format your logs as JSON since it relies on exc_info being set. The result is you don't get exc_info in your logs. This is no however no longer true since this PR was merged. Sample Code.
Solution #1: Modify Python JSON logger
The first solution is to look for exc_text which is set by the standard Formatter class as per my PR and set that as the value of exc_info which means at least we get the string representation of the traceback. This PR is now merged.
Solution #2: Subclass QueueHandler
The second solution is it to subclass logutils.queue.QueueHandler as follows and add pickling support for exc_info via tblib:
# Bring in support for serializing/deserializing tracebacks # needed by QueueHandler from tblib import pickling_support pickling_support.install() class QueueHandlerWithTraceback(logutils.queue.QueueHandler): """ QueueHandler with support for pickling/unpickling Tracebacks via tblib. We only override the prepare() method to *not* set `exc_info=None` """ def __init__(self, *args, **kwargs): logutils.queue.QueueHandler.__init__(self, *args, **kwargs) def prepare(self, record): self.format(record) record.msg = record.message record.args = None return record
Instead of logutils.queue.QueueHandler, we will then use QueueHandlerWithTraceback instead above (Sample Code).
