Django already has a mature logging system built on top of Python’s standard logging module. For most application logs, writing to stdout, files, or a centralized logging provider is enough. But sometimes you want a separate audit/security log that is stored outside the main application database. For example, you may want to record suspicious request methods, failed authorization checks, CSRF failures, webhook signature failures, rate-limit triggers, or other security-relevant events.

A common first instinct is to put logging logic directly inside middleware or views. That works for a small prototype, but it does not scale well across a project. A cleaner pattern is to expose a small project-wide logging API, configure a named logger in Django’s LOGGING setting, and attach a custom logging handler that writes to a separate PostgreSQL database.

This keeps the call sites simple:

1
2
3
4
5
6

record_error_event(
    "illegal_request_method",
    request,
    status_code=response.status_code,
    allowed_methods=response.get("Allow", ""),
)

while keeping the actual persistence backend — PostgreSQL, Redis, Celery, a file, or something else — behind a handler implementation.

Why use a logging handler?

Python logging is designed around a simple flow:

logger.warning(...)

• creates a LogRecord
• passes it to matching handlers
• each handler emits the record

So when you call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import logging

logger = logging.getLogger('error')

logger.warning(
    "unhandled exception",
    extra={
        "event_type": "unhandled exception",
        "path": request.path,
        "method": request.method,
        "status_code": 500,
    },
)

Python creates a LogRecord. The values passed through extra become attributes on that record. A custom handler can then access them with:

1
2
3

getattr(record, "path", None)
getattr(record, "method", None)
getattr(record, "status_code", None)

The handler’s emit() method is called automatically when the log event passes the logger’s level and filters. You do not call emit() yourself.

This is the central point: the project only emits semantic events, while the handler decides where those events go.

Django logging configuration

In Django, the correct place to configure loggers and handlers is settings.py.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,

    "handlers": {
        "logging_bytestaq": {
            "class": "path.to.logging.handlers.ErrorLoggingHandler",
        },
    },

    "loggers": {
        "error": {
            "handlers": ["logging_bytestaq"],
            "level": "WARNING",
            "propagate": False,
        },
    },
}

This tells Django that the named logger 'error' should send warning-or-higher records to ErrorLoggingHandler.

Then, anywhere in the project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import logging

error_logger = logging.getLogger('error')

error_logger.warning(
    "illegal_request_method",
    extra={
        "event_type": "illegal_request_method",
        "path": request.path,
        "method": request.method,
    },
)

When this code runs, Django/Python logging finds the configured handler and calls its emit() method.

One important warning: settings.py is for configuration, not runtime setup. Do not open PostgreSQL connections, create databases, run migrations, or call setup functions from settings. Django imports settings for many commands: runserver, migrate, makemigrations, shell, collectstatic, tests, and more. If settings opens external connections, those commands can fail before Django even starts.

If you want to know more about settings.py and how to organize it: Organize Django settings

A project-wide logging API

Instead of scattering raw logger.warning(...) calls everywhere, create a small wrapper function.

common/app_logging/logger.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import logging

error_logger = logging.getLogger('error')


def record_error_event(event_type, request=None, **data):
    payload = {
        "event_type": event_type,
        **data,
    }

    if request is not None:
        payload.update(
            {
                "path": request.path,
                "method": request.method,
                "ip_address": get_client_ip(request),
                "user_agent": request.META.get("HTTP_USER_AGENT", ""),
            }
        )

    error_logger.warning(event_type, extra=payload)


def get_client_ip(request):
    forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR")

    if forwarded_for:
        return forwarded_for.split(",")[0].strip()

    return request.META.get("REMOTE_ADDR")

Now call sites remain clean:

1
2
3
4
5
6

record_error_event(
    "illegal_request_method",
    request,
    status_code=response.status_code,
    allowed_methods=response.get("Allow", ""),
)

This gives you a stable internal API. Today it may write synchronously to PostgreSQL. Later it could push to Redis, Celery, Kafka, or a third-party telemetry system without rewriting every middleware and view.

Example middleware

A simple middleware for HTTP 500 responses could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

from common.app_logging.logger import record_error_event


class RequestMethodMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response


    def __call__(self, request):
        return self.get_response(request)


    def process_exception(self, request, exception):
        record_error_event(
            event_type='Unhandled exception',
            request=request,
            addt_message= f'{type(exception).__name__} - {str(exception)}',
        )

        return render(
            request,
            'errors/500.html',
            status=500,
        )

If you need to freshen up on HTTP response codes: HTTP Response Code Cheat Sheet

This middleware does not know about PostgreSQL. It only records an error event before the 500 response is returned to the user.

That separation is important:

Middleware should detect and describe the event. The logging system should decide how to store it.

Writing the custom handler

As mentioned already: If you want to log to a separate PostgreSQL database, you can implement a custom handler.

In this example we'll use psycopg2 to write to our PostgreSQL database. With this module we have to basic choices:

• ThreadedConnectionPool
• SimpleConnectionPool

For a normal synchronous Django stack, psycopg2.pool.ThreadedConnectionPool is more appropriate than SimpleConnectionPool. SimpleConnectionPool is intended for single-threaded use.

A Django application can serve requests concurrently across threads, so the threaded pool is the safer choice.

common/app_logging/handlers.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

import logging
import threading

from django.conf import settings
from psycopg2.pool import ThreadedConnectionPool


class ErrorLoggingHandler(logging.Handler):
    _pool = None
    _pool_lock = threading.Lock()

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def get_pool(self):
        if self._pool is None:
            with self._pool_lock:
                self._pool = ThreadedConnectionPool(
                    minconn=1,
                    maxconn=2,
                    dbname=settings.SECURITY_DB["dbname"],
                    user=settings.SECURITY_DB["user"],
                    password=settings.SECURITY_DB["password"],
                    host=settings.SECURITY_DB["host"],
                    port=settings.SECURITY_DB["port"],
                )

        return self._pool

    def emit(self, record):
        conn = None

        try:
            pool = self.get_pool()
            conn = pool.getconn()

            with conn.cursor() as cursor:
                cursor.execute(
                    """
                    INSERT INTO error_event_log (
                        event_type,
                        path,
                        method,
                        ip_address,
                        user_agent,
                        status_code,
                        allowed_methods,
                        created_at
                    )
                    VALUES (%s, %s, %s, %s, %s, %s, %s, now())
                    """,
                    (
                        getattr(record, "event_type", record.getMessage()),
                        getattr(record, "path", None),
                        getattr(record, "method", None),
                        getattr(record, "ip_address", None),
                        getattr(record, "user_agent", ""),
                        getattr(record, "status_code", None),
                        getattr(record, "allowed_methods", ""),
                    ),
                )

            conn.commit()

        except Exception:
            if conn is not None:
                conn.rollback()

            self.handleError(record)

        finally:
            if conn is not None:
                pool.putconn(conn)

This handler lazily creates a connection pool the first time a error event is emitted. After that, connections are borrowed from the pool and returned after each write.

Why use threading.Lock()?

The threading.Lock() is not there to prevent overlapping inserts. PostgreSQL can handle concurrent inserts, and the connection pool is designed to handle concurrent connection checkouts.

The lock only protects pool initialization:

1
2
3
4

if self._pool is None:
    with self._pool_lock:
        if self._pool is None:
            self._pool = ThreadedConnectionPool(...)        

Without this lock, two requests could hit the logger at the same time when _pool is still None. Both threads could observe _pool is None, and both could create a connection pool. That wouls create a race condition between the threads.

The double-check pattern prevents that:

1. First check: avoid locking after the pool already exists.
2. Lock: allow only one thread to initialize the pool.
3. Second check: confirm another thread did not already create the pool while this thread waited.

After the pool exists, the lock is not involved in normal writes.

You usually do not want this:

1
2

with self._pool_lock:
    cursor.execute(...)

That would serialize every log write and defeat the purpose of having a pool. The lock is for safe one-time initialization, not for write correctness.

One pool per process

A connection pool created this way is process-local. If you run four worker processes, each process has its own pool.

For example:

4 workers × maxconn=2 = up to 8 errorDB connections

That matters when sizing PostgreSQL connection limits. For low-volume error logging, a small pool is usually enough:

• minconn=1
• maxconn=2

or even maxconn=1 if you are comfortable serializing error writes per process.

Table design

A simple PostgreSQL table could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

CREATE TABLE error_event_log (
    id bigserial PRIMARY KEY,
    event_type text NOT NULL,
    path text,
    method varchar(16),
    ip_address inet,
    user_agent text,
    status_code smallint,
    allowed_methods varchar(255),
    created_at timestamptz NOT NULL DEFAULT now()
);

Use timestamptz, not plain timestamp, unless you have a specific reason not to. The default:

• created_at timestamptz NOT NULL DEFAULT now()

lets PostgreSQL assign the timestamp automatically, so your insert does not need to provide it.

Async logging

Asynchronous logging in Django only makes sense under certain conditions.

Most of Django's processes are synchronous so for example, if your middleware stack is mostly synchronous — including Django’s default middleware — one async database write at the end of the request does not buy much.

Django will already be adapting the request through sync code. An async PostgreSQL insert would add complexity without meaningfully improving throughput.

Async logging starts making sense when the whole request path is async, or when logging is decoupled from the request entirely. For a larger system, the architecture may become:

request path
    -> emit structured error event
    -> Redis/Celery/RQ/Dramatiq queue
    -> worker writes to PostgreSQL

That way the write happens somewhere else entirely where the stack is fully async by default.

That design is useful when error logging happens across many points in the project or becomes frequent enough that request latency matters. It also enables retries, buffering, batch inserts, backpressure, and isolation from error database outages.

For a low-volume 500 logger, a synchronous insert through a small PostgreSQL connection pool is usually simpler and good enough.

Practical recommendation

Use a simple layered design:

settings.py
    -> configures the logger

record_event(...)
    -> project-wide API for emitting error events

LoggingHandler.emit(...)
    -> writes records to the separate PostgreSQL database (or any other database)

ThreadedConnectionPool
    -> reuses connections safely across threads

threading.Lock()
    -> protects lazy one-time pool initialization only

This gives you clean call sites, a centralized event schema, separation between application data and logging data, and an easy path to a queue-based architecture later if the logging volume grows.