ClickHouse's unbounded system logs silently fill disks in self-hosted Langfuse and SigNoz

Self-hosted observability stacks have an unglamorous disk-eater: ClickHouse's own bookkeeping. A guide on dev.to explains why Langfuse and SigNoz deployments run out of disk even when trace volumes are tiny, and walks through a recovery in four steps: confirm the culprit, truncate the bloated tables, add retention limits, and clean up afterwards. The author tested the steps on stock clickhouse/clickhouse-server images 24.8, 25.12 and 26.9, with the SigNoz commands verified on 25.5 and 25.12.

Confirming the culprit

In most public reports the space goes not to application data but to ClickHouse's internal tables such as system.trace_log and system.text_log, which have no size limit by default. ClickStack has the same behaviour. The diagnostic is one read-only query against system.parts that lists the largest tables by on-disk size:

sql SELECT database, table, formatReadableSize(sum(bytes_on_disk)) AS size, sum(rows) AS rows, count() AS parts FROM system.parts WHERE active GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 15;

Run it with --readonly=1 so ClickHouse refuses any change to data. On Langfuse's docker compose setup the credentials are already set inside the container as CLICKHOUSE_USER and CLICKHOUSE_PASSWORD; in SigNoz the container is typically named signoz-clickhouse and the default user has no password. If system tables dominate the listing ahead of your actual data, ClickHouse's own logs are the problem.

Freeing the space now

Open a client with write access and truncate the offenders: system.trace_log, system.text_log, system.metric_log and system.query_log. This deletes only ClickHouse's diagnostics, not your traces, and needs no restart. One caveat from the guide: Langfuse reads system.query_log to follow some of its own queries, so empty it but do not disable it.

Tables larger than about 46.6 GiB — the default max_table_size_to_drop of 50 billion bytes, which applies to TRUNCATE as well — fail with error Code 359. On ClickHouse 23.12 and newer, the limit can be lifted for a single statement:

sql TRUNCATE TABLE system.text_log SETTINGS max_table_size_to_drop = 0;

Alternatively, create the flag file /var/lib/clickhouse/flags/force_drop_table with mode 666 and rerun the plain TRUNCATE. ClickHouse removes the flag after the first operation that needs it, so it must be recreated for each oversized table.

Why these tables balloon

ClickHouse's own documentation, cited in the guide, states that table growth is unlimited by default. Two defaults drive the large tables: the query profiler writes stack samples of running queries into trace_log, and text_log stores the server log at trace level. Default configs since 25.9 set a 30-day TTL only for processors_profile_log, not for the big ones.

The public record shows the scale. In Langfuse discussion #13123, system.trace_log had reached 66.86 GiB while the traces, observations and scores tables together used about 51 MiB; the maintainers chose not to override ClickHouse's defaults and documented the issue in an FAQ instead. In SigNoz issue #12050, more than 80 GB of system logs sat next to less than 500 MB of telemetry, and truncating recovered roughly 80 GB. SigNoz's newer Foundry installer sets TTLs, but older docker compose installs do not. A ClickStack Helm chart pull request, merged in September 2026, followed a report of a 10 Gi volume filling within ten days and adds a 7-day TTL.

Making the fix permanent

A TTL makes ClickHouse expire old log rows by itself. Order matters: when ClickHouse restarts with a new TTL it renames the old table, rows and all, to a name like trace_log_0 and creates a fresh table under the TTL. Truncating first keeps those copies small, and they can be dropped afterwards.

The retention rules go into a config.d XML file next to docker-compose.yml — an entry such as event_date + INTERVAL 7 DAY DELETE for each of query_log, trace_log, text_log, metric_log, asynchronous_metric_log and part_log — followed by a restart. A query against system.tables shows which logs currently lack a TTL. One trap stands out: opentelemetry_span_log is defined through an engine element in the stock config, and a separate ttl entry stops the server from starting (ClickHouse issue #88366). Its TTL belongs inside the engine definition.

When the disk is still full

If space does not return, the guide points next to Docker's own container logs, deleted rows still held on disk and stuck parts. It also warns Langfuse users to update Langfuse before moving ClickHouse to 26.8 or later.

Why it matters

Observability infrastructure is meant to watch systems, not starve them, yet this default does exactly that: a monitoring stack that looks healthy quietly consumes its own volume until the disk fills. The growth sits in system tables rather than user data, so it is invisible to anyone watching only application metrics, and neither Langfuse nor older SigNoz compose installs ship a cap. The remedy is cheap — one read-only query, a truncate, a small config file — and sequencing it correctly sidesteps the known traps, from the 50 GB drop guard to the engine-scoped TTL. Anyone self-hosting Langfuse, SigNoz, ClickStack or any other ClickHouse-backed stack should check system.parts before a disk alert does it for them.