Re: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

From: Ajin Cherian <itsajin(at)gmail(dot)com>
To: li jie <ggysxcq(at)gmail(dot)com>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Proposal: Filter irrelevant change before reassemble transactions during logical decoding
Date: 2024-05-22 04:17:54
Message-ID: CAFPTHDZZxQ_kCXi3mF8gdDQPJ7DzMe1Mgif1LODeMYyqD9QkDg@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Tue, Mar 19, 2024 at 1:49 PM Ajin Cherian <itsajin(at)gmail(dot)com> wrote:

>
>
>> Of course you can, but this will only convert disk space into memory
>> space.
>> For details, please see the case in Email [1].
>>
>> [1]
>> https://stream.boxwatch.ru/message-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t%3DuXz%2BZbg%40mail.gmail.com
>>
>> Regards, lijie
>>
>>
>
In some testing, I see a crash:
(gdb) bt
#0 0x00007fa5bcbfd277 in raise () from /lib64/libc.so.6
#1 0x00007fa5bcbfe968 in abort () from /lib64/libc.so.6
#2 0x00000000009e0940 in ExceptionalCondition (
conditionName=conditionName(at)entry=0x7fa5ab8b9842 "RelationSyncCache !=
NULL",
fileName=fileName(at)entry=0x7fa5ab8b9820 "pgoutput.c",
lineNumber=lineNumber(at)entry=1991)
at assert.c:66
#3 0x00007fa5ab8b7804 in get_rel_sync_entry (data=data(at)entry=0x2492288,
relation=relation(at)entry=0x7fa5be30a768) at pgoutput.c:1991
#4 0x00007fa5ab8b7cda in pgoutput_table_filter (ctx=<optimized out>,
relation=0x7fa5be30a768,
change=0x24c5c20) at pgoutput.c:1671
#5 0x0000000000813761 in filter_by_table_cb_wrapper (ctx=ctx(at)entry=0x2491fd0,

relation=relation(at)entry=0x7fa5be30a768, change=change(at)entry=0x24c5c20)
at logical.c:1268
#6 0x000000000080e20f in FilterByTable (ctx=ctx(at)entry=0x2491fd0,
change=change(at)entry=0x24c5c20)
at decode.c:690
#7 0x000000000080e8e3 in DecodeInsert (ctx=ctx(at)entry=0x2491fd0,
buf=buf(at)entry=0x7fff0db92550)
at decode.c:1070
#8 0x000000000080f43d in heap_decode (ctx=ctx(at)entry=0x2491fd0,
buf=buf(at)entry=0x7fff0db92550)
at decode.c:485
#9 0x000000000080eca6 in LogicalDecodingProcessRecord
(ctx=ctx(at)entry=0x2491fd0,
record=0x2492368)
at decode.c:118
#10 0x000000000081338f in DecodingContextFindStartpoint
(ctx=ctx(at)entry=0x2491fd0)
at logical.c:672
#11 0x000000000083c650 in CreateReplicationSlot (cmd=cmd(at)entry=0x2490970)
at walsender.c:1323
#12 0x000000000083fd48 in exec_replication_command (
cmd_string=cmd_string(at)entry=0x239c880 "CREATE_REPLICATION_SLOT
\"pg_16387_sync_16384_7371301304766135621\" LOGICAL pgoutput (SNAPSHOT
'use')") at walsender.c:2116

The reason for the crash is that the RelationSyncCache was NULL prior to
reaching a consistent point.
Hi li jie, I see that you created a new thread with an updated version of
this patch [1]. I used that patch and addressed the crash seen above,
rebased the patch and addressed a few other comments.
I'm happy to help you with this patch and address comments if you are not
available.

regards,
Ajin Cherian
Fujitsu Australia
[1] -
https://stream.boxwatch.ru/message-id/CAGfChW7%2BZMN4_NHPgz24MM42HVO83ecr9TLfpihJ%3DM0s1GkXFw%40mail.gmail.com

Attachment Content-Type Size
v2-0001-Reduce-useless-changes-before-reassemble-during-l.patch application/octet-stream 18.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message David G. Johnston 2024-05-22 04:31:07 Re: Ambiguous description on new columns
Previous Message Peter Smith 2024-05-22 04:12:48 Re: Ambiguous description on new columns