diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index de84b77..dc39e94 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -5961,7 +5961,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - VACUUM performs a whole-table scan if the table's + VACUUM performs an eager freezing if the table's pg_class.relfrozenxid field has reached the age specified by this setting. The default is 150 million transactions. Although users can set this value anywhere from zero to @@ -6005,7 +6005,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - VACUUM performs a whole-table scan if the table's + VACUUM performs an eager freezing if the table's pg_class.relminmxid field has reached the age specified by this setting. The default is 150 million multixacts. Although users can set this value anywhere from zero to two billions, diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index 5204b34..7cc975d 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -352,9 +352,9 @@ Vacuum maintains a visibility map for each table to keep track of which pages contain only tuples that are known to be visible to all active transactions (and all future transactions, until the - page is again modified). This has two purposes. First, vacuum - itself can skip such pages on the next run, since there is nothing to - clean up. + page is again modified), and pages contain only frozen tuples. + This has two purposes. First, vacuum itself can skip such pages + on the next run, since there is nothing to clean up. @@ -438,28 +438,25 @@ - VACUUM normally skips pages that don't have any dead row - versions, but those pages might still have row versions with old XID - values. To ensure all old row versions have been frozen, a - scan of the whole table is needed. + VACUUM skips pages that don't have any dead row + versions, and pages that have only frozen rows. To ensure all old + row versions have been frozen, a scan of all unfrozen pages is needed. controls when - VACUUM does that: a whole table sweep is forced if - the table hasn't been fully scanned for vacuum_freeze_table_age - minus vacuum_freeze_min_age transactions. Setting it to 0 - forces VACUUM to always scan all pages, effectively ignoring - the visibility map. + VACUUM does that: a whole-table freezing is forced if + the table hasn't been ensured all row versions are frozen for + vacuum_freeze_table_age minus vacuum_freeze_min_age + transcation. The maximum time that a table can go unvacuumed is two billion transactions minus the vacuum_freeze_min_age value at - the time VACUUM last scanned the whole table. If it were to go - unvacuumed for longer than - that, data loss could result. To ensure that this does not happen, - autovacuum is invoked on any table that might contain unfrozen rows with - XIDs older than the age specified by the configuration parameter . (This will happen even if - autovacuum is disabled.) + the time VACUUM last scanned unfrozen pages. If it were to go + unvacuumed for longer than that, data loss could result. To ensure + that this does not happen, autovacuum is invoked on any table that might + contain unfrozen rows with XIDs older than the age specified by the + configuration parameter . + (This will happen even if autovacuum is disabled.) @@ -490,8 +487,7 @@ a regularly scheduled VACUUM or an autovacuum triggered by normal delete and update activity is run in that window. Setting it too close could lead to anti-wraparound autovacuums, even though the table - was recently vacuumed to reclaim space, whereas lower values lead to more - frequent whole-table scans. + was recently vacuumed to reclaim space. @@ -527,7 +523,7 @@ pg_database. In particular, the relfrozenxid column of a table's pg_class row contains the freeze cutoff XID that was used - by the last whole-table VACUUM for that table. All rows + by the last whole-table freezing for that table. All rows inserted by transactions with XIDs older than this cutoff XID are guaranteed to have been frozen. Similarly, the datfrozenxid column of a database's @@ -555,17 +551,18 @@ SELECT datname, age(datfrozenxid) FROM pg_database; VACUUM normally only scans pages that have been modified since the last vacuum, but relfrozenxid can only be advanced when the whole table is - scanned. The whole table is scanned when relfrozenxid is - more than vacuum_freeze_table_age transactions old, when - VACUUM's FREEZE option is used, or when all pages - happen to - require vacuuming to remove dead row versions. When VACUUM - scans the whole table, after it's finished age(relfrozenxid) - should be a little more than the vacuum_freeze_min_age setting - that was used (more by the number of transactions started since the - VACUUM started). If no whole-table-scanning VACUUM - is issued on the table until autovacuum_freeze_max_age is - reached, an autovacuum will soon be forced for the table. + frozen. Freezing occurs on the whole table once all pages of this relation + require it. In other cases such as where relfrozenxid is more + than vacuum_freeze_table_age transactions old, where + VACUUM's FREEZE option is used, VACUUM + can skip the pages that all tuples on the page itself are marked as frozen. + When all pages of table are eventually marked as frozen by VACUUM, + after it's finished age(relfrozenxid) should be a little more + than the vacuum_freeze_min_age setting that was used (more by + the number of transactions started since the VACUUM started). + If the advancing of relfrozenxid is not happend until + autovacuum_freeze_max_age is reached, an autovacuum will soon + be forced for the table. @@ -642,28 +639,28 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. pg_class.relminmxid stores the oldest possible multixact ID still appearing in any tuple of that table. If this value is older than - , a whole-table + , a table scan is forced. mxid_age() can be used on pg_class.relminmxid to find its age. - Whole-table VACUUM scans, regardless of - what causes them, enable advancing the value for that table. + When VACUUM scans all unfrozen pages, regardless of what causes + them, enable advancing the value for that table. Eventually, as all tables in all databases are scanned and their oldest multixact values are advanced, on-disk storage for older multixacts can be removed. - As a safety device, a whole-table vacuum scan will occur for any table - whose multixact-age is greater than - . Whole-table + As a safety device, a vacuum scan will occur for any table whose + multixact-age is greater than + . A vacuum scans will also occur progressively for all tables, starting with those that have the oldest multixact-age, if the amount of used member storage space exceeds the amount 50% of the addressable storage space. - Both of these kinds of whole-table scans will occur even if autovacuum is - nominally disabled. + Both of these kinds of scans will occur even if autovacuum is nominally + disabled. @@ -743,8 +740,8 @@ vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuple UPDATE and DELETE operation. (It is only semi-accurate because some information might be lost under heavy load.) If the relfrozenxid value of the table is more - than vacuum_freeze_table_age transactions old, the whole - table is scanned to freeze old tuples and advance + than vacuum_freeze_table_age transactions old, the table is + scanned to freeze old tuples and advance relfrozenxid, otherwise only pages that have been modified since the last vacuum are scanned. diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 85459d0..0bcd52d 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1423,6 +1423,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Estimated number of rows modified since this table was last analyzed + n_frozen_pages + integer + Number of frozen pages + + last_vacuum timestamp with time zone Last time at which this table was manually vacuumed diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c index 4f6f6e7..2a174a2 100644 --- a/src/backend/commands/vacuumlazy.c +++ b/src/backend/commands/vacuumlazy.c @@ -106,6 +106,8 @@ typedef struct LVRelStats BlockNumber rel_pages; /* total number of pages */ BlockNumber scanned_pages; /* number of pages we examined */ BlockNumber pinskipped_pages; /* # of pages we skipped due to a pin */ + BlockNumber vmskipped_frozen_pages; /* # of pages we skipped by all-frozen bit + of visibility map */ double scanned_tuples; /* counts only tuples on scanned pages */ double old_rel_tuples; /* previous value of pg_class.reltuples */ double new_rel_tuples; /* new estimated total # of tuples */ @@ -156,8 +158,9 @@ static void lazy_record_dead_tuple(LVRelStats *vacrelstats, ItemPointer itemptr); static bool lazy_tid_reaped(ItemPointer itemptr, void *state); static int vac_cmp_itemptr(const void *left, const void *right); -static bool heap_page_is_all_visible(Relation rel, Buffer buf, - TransactionId *visibility_cutoff_xid); +static void heap_page_visible_status(Relation rel, Buffer buf, + TransactionId *visibility_cutoff_xid, + bool *all_visible, bool *all_frozen); /* @@ -188,7 +191,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, MultiXactId mxactFullScanLimit; BlockNumber new_rel_pages; double new_rel_tuples; - BlockNumber new_rel_allvisible; + BlockNumber new_rel_allvisible, + new_rel_allfrozen; double new_live_tuples; TransactionId new_frozen_xid; MultiXactId new_min_multi; @@ -221,7 +225,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * We request a full scan if either the table's frozen Xid is now older * than or equal to the requested Xid full-table scan limit; or if the * table's minimum MultiXactId is older than or equal to the requested - * mxid full-table scan limit. + * mxid full-table scan limit. During full scan, we could skip some pages + * according to all-frozen bit of visibility map. */ scan_all = TransactionIdPrecedesOrEquals(onerel->rd_rel->relfrozenxid, xidFullScanLimit); @@ -253,7 +258,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * NB: We need to check this before truncating the relation, because that * will change ->rel_pages. */ - if (vacrelstats->scanned_pages < vacrelstats->rel_pages) + if ((vacrelstats->scanned_pages + vacrelstats->vmskipped_frozen_pages) + < vacrelstats->rel_pages) { Assert(!scan_all); scanned_all = false; @@ -274,15 +280,15 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * Update statistics in pg_class. * * A corner case here is that if we scanned no pages at all because every - * page is all-visible, we should not update relpages/reltuples, because - * we have no new information to contribute. In particular this keeps us - * from replacing relpages=reltuples=0 (which means "unknown tuple + * page is all-visible or all-frozen, we should not update relpages/reltuples, + * because we have no new information to contribute. In particular this keeps + * us from replacing relpages=reltuples=0 (which means "unknown tuple * density") with nonzero relpages and reltuples=0 (which means "zero * tuple density") unless there's some actual evidence for the latter. * - * We do update relallvisible even in the corner case, since if the table - * is all-visible we'd definitely like to know that. But clamp the value - * to be not more than what we're setting relpages to. + * We do update relallvisible and relallfrozen even in the corner case, + * since if the table is all-visible we'd definitely like to know that. + * But clamp the value to be not more than what we're setting relpages to. * * Also, don't change relfrozenxid/relminmxid if we skipped any pages, * since then we don't know for certain that all tuples have a newer xmin. @@ -295,10 +301,13 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, new_rel_tuples = vacrelstats->old_rel_tuples; } - new_rel_allvisible = visibilitymap_count(onerel); + visibilitymap_count(onerel, &new_rel_allvisible, &new_rel_allfrozen); if (new_rel_allvisible > new_rel_pages) new_rel_allvisible = new_rel_pages; + if (new_rel_allfrozen > new_rel_pages) + new_rel_allfrozen = new_rel_pages; + new_frozen_xid = scanned_all ? FreezeLimit : InvalidTransactionId; new_min_multi = scanned_all ? MultiXactCutoff : InvalidMultiXactId; @@ -319,7 +328,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, pgstat_report_vacuum(RelationGetRelid(onerel), onerel->rd_rel->relisshared, new_live_tuples, - vacrelstats->new_dead_tuples); + vacrelstats->new_dead_tuples, + new_rel_allfrozen); /* and log the action if appropriate */ if (IsAutoVacuumWorkerProcess() && params->log_min_duration >= 0) @@ -354,10 +364,11 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, get_namespace_name(RelationGetNamespace(onerel)), RelationGetRelationName(onerel), vacrelstats->num_index_scans); - appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins\n"), + appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped according to vm\n"), vacrelstats->pages_removed, vacrelstats->rel_pages, - vacrelstats->pinskipped_pages); + vacrelstats->pinskipped_pages, + vacrelstats->vmskipped_frozen_pages); appendStringInfo(&buf, _("tuples: %.0f removed, %.0f remain, %.0f are dead but not yet removable\n"), vacrelstats->tuples_deleted, @@ -480,9 +491,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * consecutive pages. Since we're reading sequentially, the OS should be * doing readahead for us, so there's no gain in skipping a page now and * then; that's likely to disable readahead and so be counterproductive. - * Also, skipping even a single page means that we can't update - * relfrozenxid, so we only want to do it if we can skip a goodly number - * of pages. + * Also, skipping even a single page according to all-visible bit of + * visibility map means that we might not be able to update relfrozenxid, + * so we only want to do it if we can skip a goodly number. On the other hand, + * we count both how many pages we skipped according to all-frozen bit of + * visibility map and how many pages we froze, so we can update relfrozenxid + * if the sum of two is as many as pages of table. * * Before entering the main loop, establish the invariant that * next_not_all_visible_block is the next block number >= blkno that's not @@ -492,18 +506,18 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * started skipping blocks, we may as well skip everything up to the next * not-all-visible block. * - * Note: if scan_all is true, we won't actually skip any pages; but we + * Note: if scan_all is true, we might not actually skip any pages; but we * maintain next_not_all_visible_block anyway, so as to set up the * all_visible_according_to_vm flag correctly for each page. * - * Note: The value returned by visibilitymap_test could be slightly + * Note: The value returned by visibilitymap_get_status could be slightly * out-of-date, since we make this test before reading the corresponding * heap page or locking the buffer. This is OK. If we mistakenly think - * that the page is all-visible when in fact the flag's just been cleared, - * we might fail to vacuum the page. But it's OK to skip pages when - * scan_all is not set, so no great harm done; the next vacuum will find - * them. If we make the reverse mistake and vacuum a page unnecessarily, - * it'll just be a no-op. + * that the page is all-visible/all-frozen when in fact the flag's just + * been cleared, we might fail to vacuum the page. But it's OK to skip + * pages when scan_all is not set, so no great harm done; the next vacuum + * will find them. If we make the reverse mistake and vacuum a page + * unnecessarily, it'll just be a no-op. * * We will scan the table's last page, at least to the extent of * determining whether it has tuples or not, even if it should be skipped @@ -518,7 +532,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, next_not_all_visible_block < nblocks; next_not_all_visible_block++) { - if (!visibilitymap_test(onerel, next_not_all_visible_block, &vmbuffer)) + if (!VM_ALL_VISIBLE(onerel, next_not_all_visible_block, &vmbuffer)) break; vacuum_delay_point(); } @@ -536,9 +550,13 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, bool tupgone, hastup; int prev_dead_count; - int nfrozen; + int nfrozen; /* # of tuples is frozen */ + int nalready_frozen; /* # of tuples is already frozen */ + int ntotal_frozen; /* # of frozen tuples in single page */ + int ntup_per_page; Size freespace; bool all_visible_according_to_vm; + bool all_frozen_according_to_vm; bool all_visible; bool has_dead_tuples; TransactionId visibility_cutoff_xid = InvalidTransactionId; @@ -554,8 +572,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, next_not_all_visible_block < nblocks; next_not_all_visible_block++) { - if (!visibilitymap_test(onerel, next_not_all_visible_block, - &vmbuffer)) + if (!VM_ALL_VISIBLE(onerel, next_not_all_visible_block, &vmbuffer)) break; vacuum_delay_point(); } @@ -569,14 +586,29 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, skipping_all_visible_blocks = true; else skipping_all_visible_blocks = false; + all_visible_according_to_vm = false; + all_frozen_according_to_vm = false; } else { - /* Current block is all-visible */ - if (skipping_all_visible_blocks && !scan_all && !FORCE_CHECK_PAGE()) - continue; + /* + * This block is at least all-visible according to visibility map. + * We check whether this block is all-frozen or not, to skip to + * vacuum this page even if scan_all is true. + */ + bool all_frozen = VM_ALL_FROZEN(onerel, blkno, &vmbuffer); + + if (scan_all && all_frozen && !FORCE_CHECK_PAGE()) + { + vacrelstats->vmskipped_frozen_pages++; + continue; + } + else if (!scan_all && skipping_all_visible_blocks && !FORCE_CHECK_PAGE()) + continue; + all_visible_according_to_vm = true; + all_frozen_according_to_vm = all_frozen; } vacuum_delay_point(); @@ -743,7 +775,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, empty_pages++; freespace = PageGetHeapFreeSpace(page); - /* empty pages are always all-visible */ + /* empty pages are always all-visible and all-frozen */ if (!PageIsAllVisible(page)) { START_CRIT_SECTION(); @@ -766,8 +798,10 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, log_newpage_buffer(buf, true); PageSetAllVisible(page); + PageSetAllFrozen(page); visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, InvalidTransactionId); + vmbuffer, InvalidTransactionId, + VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN); END_CRIT_SECTION(); } @@ -791,13 +825,15 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, all_visible = true; has_dead_tuples = false; nfrozen = 0; + nalready_frozen = 0; + ntup_per_page = 0; hastup = false; prev_dead_count = vacrelstats->num_dead_tuples; maxoff = PageGetMaxOffsetNumber(page); /* * Note: If you change anything in the loop below, also look at - * heap_page_is_all_visible to see if that needs to be changed. + * heap_page_visible_status to see if that needs to be changed. */ for (offnum = FirstOffsetNumber; offnum <= maxoff; @@ -945,8 +981,13 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, else { num_tuples += 1; + ntup_per_page += 1;; hastup = true; + /* Check whether this tuple is already frozen or not */ + if (HeapTupleHeaderXminFrozen(tuple.t_data)) + nalready_frozen += 1; + /* * Each non-removable tuple must be checked to see if it needs * freezing. Note we already have exclusive buffer lock. @@ -993,6 +1034,9 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, END_CRIT_SECTION(); } + /* Compute total number of frozen tuples in a page */ + ntotal_frozen = nfrozen + nalready_frozen; + /* * If there are no indexes then we can vacuum the page right now * instead of doing a second scan. @@ -1015,26 +1059,47 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, freespace = PageGetHeapFreeSpace(page); - /* mark page all-visible, if appropriate */ - if (all_visible && !all_visible_according_to_vm) + /* This page is all visible */ + if (all_visible) { - /* - * It should never be the case that the visibility map page is set - * while the page-level bit is clear, but the reverse is allowed - * (if checksums are not enabled). Regardless, set the both bits - * so that we get back in sync. - * - * NB: If the heap page is all-visible but the VM bit is not set, - * we don't need to dirty the heap page. However, if checksums - * are enabled, we do need to make sure that the heap page is - * dirtied before passing it to visibilitymap_set(), because it - * may be logged. Given that this situation should only happen in - * rare cases after a crash, it is not worth optimizing. - */ - PageSetAllVisible(page); - MarkBufferDirty(buf); - visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, visibility_cutoff_xid); + uint8 flags = 0; + + /* mark page all-visible, if appropriate */ + if (!all_visible_according_to_vm) + { + /* + * It should never be the case that the visibility map page is set + * while the page-level bit is clear, but the reverse is allowed + * (if checksums are not enabled). Regardless, set the both bits + * so that we get back in sync. + * + * NB: If the heap page is all-visible but the VM bit is not set, + * we don't need to dirty the heap page. However, if checksums + * are enabled, we do need to make sure that the heap page is + * dirtied before passing it to visibilitymap_set(), because it + * may be logged. Given that this situation should only happen in + * rare cases after a crash, it is not worth optimizing. + */ + PageSetAllVisible(page); + flags |= VISIBILITYMAP_ALL_VISIBLE; + } + + /* mark page all-frozen, if all tuples are frozen and not marked yet */ + if ((ntotal_frozen == ntup_per_page) && !all_frozen_according_to_vm) + { + Assert(PageIsAllVisible(page)); + + PageSetAllFrozen(page); + flags |= VISIBILITYMAP_ALL_FROZEN; + } + + if (flags) + { + MarkBufferDirty(buf); + visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr, + vmbuffer, visibility_cutoff_xid, flags); + } + } /* @@ -1045,9 +1110,14 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * that something bad has happened. */ else if (all_visible_according_to_vm && !PageIsAllVisible(page) - && visibilitymap_test(onerel, blkno, &vmbuffer)) + && VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)) { - elog(WARNING, "page is not marked all-visible but visibility map bit is set in relation \"%s\" page %u", + /* If the all-frozen is set then all-visible must be set */ + if (all_frozen_according_to_vm) + Assert(VM_ALL_FROZEN(onerel, blkno, &vmbuffer) && + VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)); + + elog(WARNING, "page is not marked all-visible (and all-frozen) but visibility map bit(s) is set in relation \"%s\" page %u", relname, blkno); visibilitymap_clear(onerel, blkno, vmbuffer); } @@ -1055,19 +1125,25 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, /* * It's possible for the value returned by GetOldestXmin() to move * backwards, so it's not wrong for us to see tuples that appear to - * not be visible to everyone yet, while PD_ALL_VISIBLE is already - * set. The real safe xmin value never moves backwards, but - * GetOldestXmin() is conservative and sometimes returns a value + * not be visible to everyone yet, while PD_ALL_VISIBLE (and PD_ALL_FROZEN) + * are already set. The real safe xmin value never moves backwards, + * but GetOldestXmin() is conservative and sometimes returns a value * that's unnecessarily small, so if we see that contradiction it just * means that the tuples that we think are not visible to everyone yet - * actually are, and the PD_ALL_VISIBLE flag is correct. + * actually are, and the PD_ALL_VISIBLE (and PD_ALL_FROZEN) flags are + * correct. * * There should never be dead tuples on a page with PD_ALL_VISIBLE * set, however. */ else if (PageIsAllVisible(page) && has_dead_tuples) { - elog(WARNING, "page containing dead tuples is marked as all-visible in relation \"%s\" page %u", + /* If all-frozen is set then all-visible must be set */ + if (PageIsAllFrozen(page)) + Assert(VM_ALL_FROZEN(onerel, blkno, &vmbuffer) && + VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)); + + elog(WARNING, "page containing dead tuples is marked as all-visible (and all-frozen) in relation \"%s\" page %u", relname, blkno); PageClearAllVisible(page); MarkBufferDirty(buf); @@ -1141,6 +1217,13 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, RelationGetRelationName(onerel), tups_vacuumed, vacuumed_pages))); + /* Report how many frozen pages vacuum skipped according to visibility map */ + ereport(elevel, + (errmsg_plural("skipped %d frozen page according to visibility map", + "skipped %d frozen pages according to visibility map", + vacrelstats->vmskipped_frozen_pages, + vacrelstats->vmskipped_frozen_pages))); + /* * This is pretty messy, but we split it up so that we can skip emitting * individual parts of the message when not applicable. @@ -1257,6 +1340,8 @@ lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer, OffsetNumber unused[MaxOffsetNumber]; int uncnt = 0; TransactionId visibility_cutoff_xid; + bool all_visible; + bool all_frozen; START_CRIT_SECTION(); @@ -1308,19 +1393,36 @@ lazy_vacuum_page(Relation onerel, BlockNumber blkno, Buffer buffer, * dirty, exclusively locked, and, if needed, a full page image has been * emitted in the log_heap_clean() above. */ - if (heap_page_is_all_visible(onerel, buffer, &visibility_cutoff_xid)) + heap_page_visible_status(onerel, buffer, &visibility_cutoff_xid, + &all_visible, &all_frozen); + if (all_visible) PageSetAllVisible(page); /* * All the changes to the heap page have been done. If the all-visible - * flag is now set, also set the VM bit. + * flag is now set, also set the VM all-visible bit. + * Also, if this page is all-frozen, set the VM all-frozen bit and flag. */ - if (PageIsAllVisible(page) && - !visibilitymap_test(onerel, blkno, vmbuffer)) + if (PageIsAllVisible(page)) { + uint8 vm_status = visibilitymap_get_status(onerel, blkno, vmbuffer); + uint8 flags = 0; + + if (!(vm_status & VISIBILITYMAP_ALL_VISIBLE)) + flags |= VISIBILITYMAP_ALL_VISIBLE; + + /* Set the VM all-frozen bit to flag, if needed */ + if (all_frozen && !(vm_status & VISIBILITYMAP_ALL_FROZEN)) + { + PageSetAllFrozen(page); + flags |= VISIBILITYMAP_ALL_FROZEN; + } + Assert(BufferIsValid(*vmbuffer)); - visibilitymap_set(onerel, blkno, buffer, InvalidXLogRecPtr, *vmbuffer, - visibility_cutoff_xid); + + if (vm_status != flags) + visibilitymap_set(onerel, blkno, buffer, InvalidXLogRecPtr, *vmbuffer, + visibility_cutoff_xid, flags); } return tupindex; @@ -1842,18 +1944,21 @@ vac_cmp_itemptr(const void *left, const void *right) /* * Check if every tuple in the given page is visible to all current and future * transactions. Also return the visibility_cutoff_xid which is the highest - * xmin amongst the visible tuples. + * xmin amongst the visible tuples, and all_frozen which implies that all tuples + * of this page are frozen. */ -static bool -heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cutoff_xid) +static void +heap_page_visible_status(Relation rel, Buffer buf, TransactionId *visibility_cutoff_xid, + bool *all_visible, bool *all_frozen) { Page page = BufferGetPage(buf); BlockNumber blockno = BufferGetBlockNumber(buf); OffsetNumber offnum, maxoff; - bool all_visible = true; *visibility_cutoff_xid = InvalidTransactionId; + *all_visible = true; + *all_frozen = true; /* * This is a stripped down version of the line pointer scan in @@ -1861,7 +1966,7 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut */ maxoff = PageGetMaxOffsetNumber(page); for (offnum = FirstOffsetNumber; - offnum <= maxoff && all_visible; + offnum <= maxoff && *all_visible; offnum = OffsetNumberNext(offnum)) { ItemId itemid; @@ -1877,11 +1982,12 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut /* * Dead line pointers can have index pointers pointing to them. So - * they can't be treated as visible + * they can't be treated as visible and frozen. */ if (ItemIdIsDead(itemid)) { - all_visible = false; + *all_visible = false; + *all_frozen = false; break; } @@ -1900,7 +2006,7 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut /* Check comments in lazy_scan_heap. */ if (!HeapTupleHeaderXminCommitted(tuple.t_data)) { - all_visible = false; + *all_visible = false; break; } @@ -1911,13 +2017,17 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut xmin = HeapTupleHeaderGetXmin(tuple.t_data); if (!TransactionIdPrecedes(xmin, OldestXmin)) { - all_visible = false; + *all_visible = false; break; } /* Track newest xmin on page. */ if (TransactionIdFollows(xmin, *visibility_cutoff_xid)) *visibility_cutoff_xid = xmin; + + /* Check whether this tuple is already frozen or not */ + if (!HeapTupleHeaderXminFrozen(tuple.t_data)) + *all_frozen = false; } break; @@ -1925,7 +2035,8 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut case HEAPTUPLE_RECENTLY_DEAD: case HEAPTUPLE_INSERT_IN_PROGRESS: case HEAPTUPLE_DELETE_IN_PROGRESS: - all_visible = false; + *all_visible = false; + *all_frozen = false; break; default: @@ -1934,5 +2045,6 @@ heap_page_is_all_visible(Relation rel, Buffer buf, TransactionId *visibility_cut } } /* scan along page */ - return all_visible; + if (!(*all_visible)) + *all_frozen = false; } diff --git a/src/test/regress/expected/visibilitymap.out b/src/test/regress/expected/visibilitymap.out new file mode 100644 index 0000000..87206b6 --- /dev/null +++ b/src/test/regress/expected/visibilitymap.out @@ -0,0 +1,22 @@ +-- +-- Visibility Map +-- +CREATE TABLE vmtest (i INT primary key); +INSERT INTO vmtest SELECT generate_series(1,10000); +-- All pages are become all-visible +VACUUM FREEZE vmtest; +SELECT relallvisible = (pg_relation_size('vmtest') / current_setting('block_size')::int) FROM pg_class WHERE relname = 'vmtest'; + ?column? +---------- + t +(1 row) + +-- Check whether vacuum skips all-frozen pages +\set VERBOSITY terse +VACUUM FREEZE VERBOSE vmtest; +INFO: vacuuming "public.vmtest" +INFO: index "vmtest_pkey" now contains 10000 row versions in 30 pages +INFO: skipped 44 frozen pages according to visibility map +INFO: "vmtest": found 0 removable, 56 nonremovable row versions in 1 out of 45 pages +\set VERBOSITY default +DROP TABLE vmtest; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index bec0316..2324420 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -110,3 +110,6 @@ test: event_trigger # run stats by itself because its delay may be insufficient under heavy load test: stats + +# page info map and vacuum test cannot run concurrently with any test that runs SQL +test: visibilitymap diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 7e9b319..df4c717 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -162,3 +162,4 @@ test: with test: xml test: event_trigger test: stats +test: visibilitymap diff --git a/src/test/regress/sql/visibilitymap.sql b/src/test/regress/sql/visibilitymap.sql new file mode 100644 index 0000000..365570b --- /dev/null +++ b/src/test/regress/sql/visibilitymap.sql @@ -0,0 +1,16 @@ +-- +-- Visibility Map +-- +CREATE TABLE vmtest (i INT primary key); +INSERT INTO vmtest SELECT generate_series(1,10000); + +-- All pages are become all-visible +VACUUM FREEZE vmtest; +SELECT relallvisible = (pg_relation_size('vmtest') / current_setting('block_size')::int) FROM pg_class WHERE relname = 'vmtest'; + +-- Check whether vacuum skips all-frozen pages +\set VERBOSITY terse +VACUUM FREEZE VERBOSE vmtest; +\set VERBOSITY default + +DROP TABLE vmtest;