| From: | Colin McDonnell <colinmcd94(at)gmail(dot)com> |
|---|---|
| To: | pgsql-bugs(at)lists(dot)postgresql(dot)org |
| Cc: | michael(at)paquier(dot)xyz, ju(dot)grigorev(at)ftdata(dot)ru |
| Subject: | Segfaults in 18 and 19 when pgstats can't grow its DSA (back-patch of d9de60c5e4) |
| Date: | 2026-11-20 21:26:59 |
| Message-ID: | CALdYYtTg2x4nNLsjX99acjbQj=oTo40CBXgDM1qzdzG7UbwzGQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-bugs |
On 18.7 and 19.0, once the cumulative statistics area needs a new DSM
segment and /dev/shm is full, the server segfaults. Autovacuum alone gets
there. Its anti-wraparound pass over a database of 3,100 tables leaves
stats entries with no body, and the next process to read one crashes:
another autovacuum worker, or the checkpointer writing the stats file at a
clean shutdown, which then ends in crash recovery.
This is the bug from the pgsql-hackers thread "DSA_ALLOC_NO_OOM vs
dsm_create ERROR leaving a half-initialized pgstats hash entry" [1], fixed
on master by d9de60c5e4 [2]. It stayed on master because d9de60c5e4 changes
the signature of pgstat_init_entry(), which pgstat_internal.h exports. The
attached patches make the same change on the stable branches without
changing any existing signature, so I'd like to ask for a back-patch.
Versions and platform:
- 18.7, Ubuntu package 18.7-1.pgdg24.04+5
- 19.0, Ubuntu package 19.0-1.pgdg24.04+1
- REL_19_STABLE at e22173e29b and REL_18_STABLE at fba1717d5a, built from
source
- Ubuntu 24.04 in Docker with --shm-size=64m, Linux 6.8.0 aarch64, glibc
2.39
- dynamic_shared_memory_type = posix (the default), no
shared_preload_libraries
Reproduction, as the postgres user with the binaries on PATH and /dev/shm a
64 MB tmpfs:
#!/bin/sh
set -e
D=/tmp/pgdata
initdb -D $D >/dev/null
pg_ctl -D $D -l $D/log -w start >/dev/null
# one database of 3,100 tables, the first 100 with a row, then 120,000 XIDs
createdb big
psql -X -q -d big <<'SQL'
SELECT format('CREATE TABLE t%s (id int PRIMARY KEY, v text)', g) FROM
generate_series(1, 100) g \gexec
SELECT format('INSERT INTO t%s VALUES (1, ''x'')', g) FROM
generate_series(1, 100) g \gexec
SELECT format('CREATE TABLE u%s (id int PRIMARY KEY, v text)', g) FROM
generate_series(1, 3000) g \gexec
SQL
echo 'SELECT txid_current();' > /tmp/xid.sql
pgbench -n -f /tmp/xid.sql -c 4 -j 4 -t 30000 postgres >/dev/null
# restart with empty stats, every database past autovacuum_freeze_max_age,
and 3 MB left in /dev/shm
pg_ctl -D $D -m fast -w stop >/dev/null
rm $D/pg_stat/pgstat.stat
cat >> $D/postgresql.conf <<'CONF'
autovacuum_freeze_max_age = 100000
autovacuum_naptime = 1s
CONF
fallocate -l $(( $(df -k --output=avail /dev/shm | tail -1) - 3072 ))K
/dev/shm/filler
pg_ctl -D $D -l $D/log -w start >/dev/null
# let the anti-wraparound pass run, then stop cleanly
sleep 60
pg_ctl -D $D -m fast -w stop >/dev/null
grep -c 'could not resize shared memory segment' $D/log
grep -E 'terminated by signal|Failed process was running|abnormal' $D/log
Expected: "could not resize shared memory segment" errors, then a clean
shutdown.
Actual, on the REL_19_STABLE build:
5617
LOG: checkpointer process (PID 3554) was terminated by signal 11:
Segmentation fault
LOG: abnormal database system shutdown
On the REL_18_STABLE build, three autovacuum workers crashed within that
minute as well, then the checkpointer:
LOG: autovacuum worker (PID 3068) was terminated by signal 11: Segmentation
fault
DETAIL: Failed process was running: autovacuum: VACUUM
pg_toast.pg_toast_41177 (to prevent wraparound)
(twice more)
LOG: checkpointer process (PID 3108) was terminated by signal 11:
Segmentation fault
LOG: abnormal database system shutdown
A worker crashes when a second worker in the same database reads an entry
the first one failed to create, so it depends on timing. In runs of the
same setup where two or three workers were already in that database when
the area filled, a worker crashed within 8 to 19 seconds on 18.7, 19.0 and
both branch tips, and again after each restart.
Backtraces from the 19.0 package. An autovacuum worker:
#2 LWLockAttemptLock at lwlock.c:774
#3 LWLockAcquire at lwlock.c:1215
#4 pgstat_lock_entry_shared at pgstat_shmem.c:709
#5 pgstat_fetch_entry at pgstat.c:1048
#6 pgstat_fetch_stat_tabentry_ext at pgstat_relation.c:490
#7 relation_needs_vacanalyze at autovacuum.c:3281
#8 table_recheck_autovac at autovacuum.c:2883
#9 do_autovacuum at autovacuum.c:2473
Its entry_ref has shared_stats = 0x0, and the hash entry is:
{key = {kind = 2, dboid = 16384, objid = 19133}, dropped = false, refcount
= {value = 2}, generation = {value = 0}, body = 0}
The checkpointer at a clean shutdown:
#3 __GI__IO_fwrite (buf=0x18, size=224, count=1, fp=...)
#4 write_chunk at pgstat.c:1607
#5 pgstat_write_statsfile at pgstat.c:1750
#6 pgstat_before_server_shutdown at pgstat.c:614
On 18.7 the worker crashed in pgstat_lock_entry() from
pgstat_relation_flush_cb(), flushing its pending stats at exit. A client
backend that reads 20,000 never-read tables with 2 MB of /dev/shm left,
then reads one that failed again, crashes the same way in
pgstat_relation_flush_cb().
Cause, as in [1]: pgstat_get_entry_ref() inserts the hash entry, then
pgstat_init_entry() allocates the body with DSA_ALLOC_NO_OOM. When the area
needs a new segment, make_new_segment() calls dsm_create(), and
dsm_impl_posix_resize() raises ERROR when posix_fallocate() fails with
ENOSPC. The entry stays live with body = InvalidDsaPointer. 16706f1f4d and
19ab0bae3a, which were back-patched, don't cover this case.
The patches add pgstat_alloc_entry_body() and pgstat_init_entry_ext(), and
pgstat_get_entry_ref() and pgstat_read_statsfile() allocate the body before
inserting the entry, as on master. pgstat_init_entry() keeps its signature
and behavior, and core no longer calls it. With them, the script above ends
in a clean shutdown on both branches, and in a 600-second run of the pass
that crashed within 19 seconds, autovacuum logged about 990,000 resize
errors without a crash. make check passes on both.
REL_15_STABLE through REL_17_STABLE have the same code, so patches for them
are attached too. Each builds, passes make check and runs the script above
to a clean shutdown.
[1]
https://stream.boxwatch.ru/message-id/flat/ddc3ecfb01ce4e9698b23cc59767f016(at)localhost(dot)localdomain
[2]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d9de60c5e47a7f7a5dcee050d119bda08ea85d55
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0001-Allocate-pgstats-entry-body-before-hash-insert-REL_19_STABLE.patch | application/octet-stream | 8.9 KB |
| v1-0001-Allocate-pgstats-entry-body-before-hash-insert-REL_18_STABLE.patch | application/octet-stream | 8.8 KB |
| v1-0001-Allocate-pgstats-entry-body-before-hash-insert-REL_17_STABLE.patch | application/octet-stream | 8.7 KB |
| v1-0001-Allocate-pgstats-entry-body-before-hash-insert-REL_16_STABLE.patch | application/octet-stream | 8.7 KB |
| v1-0001-Allocate-pgstats-entry-body-before-hash-insert-REL_15_STABLE.patch | application/octet-stream | 8.7 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Michael Paquier | 2026-11-20 22:18:38 | Re: Segfaults in 18 and 19 when pgstats can't grow its DSA (back-patch of d9de60c5e4) |
| Previous Message | Tom Lane | 2026-11-20 21:20:26 | Re: BUG #19924: unrecognized node type in create_plan_recurse: UNION, LIMIT, json_populate_recordset, inheritance |