Hi David,
The patch fixes the cases from the report for me. Two small things.
1. simplehash rounds the size up to a power of 2, so the empty table
can still be twice hash_mem. With work_mem = 1GB, a plain EXPLAIN of
the recursive query zeroes 4 GB. Halving the limit fixes it.
hash_choose_num_buckets() does the same.
2. Plain EXPLAIN still builds the table. nodeAgg skips it.
The attached patch on top of yours does both.
Thanks,
Shihao