Hello, Itagaki-san, all

Sorry for my long mail, in advance.

From: "ITAGAKI Takahiro" <itagaki.takahiro@oss.ntt.co.jp>
> I tested the patch on Linux kernel 2.6.9-39, default settings, and ext3fs.
> Maybe pdflush daemons were strong enough to write dirty buffers in kernel,
> so step 2 was a main part and 3 was not.

I'm in favor of Mr. Stark's opinion.  After all, it seems that smoothing the throughput/response time is difficult as long as PostgreSQL uses fsync() and relies on OS for the I/O timing.  Even if bgwriter trickles write()s, it is highly possible that the transactions are caught in the storm caused by fsync()s.  I think it is difficult for PostgreSQL to appropriately tune the delay and the amount of work at each wakeup during checkpoint, because the actual I/O timing depends on the VM setting and the amount of available RAM.  Without O_SYNC, OS controls I/O timing.  How can we control what we don't try to control?
Again, please see the following excerpt from Oracle's manual.  Oracle has the parameters LOG_CHECKPOINT_INTERVAL and LOG_CHECKPOINT_TIMEOUT which are similar to checkpoint_segments and checkpoint_timeout respectively of PostgreSQL.  However, Oracle doesn't have parameters like bgwriter*.  What does this mean?  Do you think Oracle has some hidden parameters like bgwriter_* that work suitably for most environments, or auto-tunes them?

http://download-west.oracle.com/docs/cd/B19306_01/backup.102/b14191/rcmtunin004.htm
--------------------------------------------------
The foundation of Fast-Start Fault Recovery is the Fast-Start checkpointing architecture. Instead of conventional event-driven (that is, log switching) checkpointing, which does bulk writes, fast-start checkpointing occurs incrementally. Each DBWn process periodically writes buffers to disk to advance the checkpoint position. The oldest modified blocks are written first to ensure that every write lets the checkpoint advance. Fast-Start checkpointing eliminates bulk writes and the resultant I/O spikes that occur with conventional checkpointing.

With the Fast-Start Fault Recovery feature, the FAST_START_MTTR_TARGET initialization parameter simplifies the configuration of recovery time from instance or system failure. FAST_START_MTTR_TARGET specifies a target for the expected mean time to recover (MTTR), that is, the time (in seconds) that it should take to start up the instance and perform cache recovery. Once FAST_START_MTTR_TARGET is set, the database manages incremental checkpoint writes in an attempt to meet that target. If you have chosen a practical value for FAST_START_MTTR_TARGET, you can expect your database to recover, on average, in approximately the number of seconds you have chosen.


Note:

You must disable or remove the FAST_START_IO_TARGET, LOG_CHECKPOINT_INTERVAL, and LOG_CHECKPOINT_TIMEOUT initialization parameters when using FAST_START_MTTR_TARGET. Setting these parameters interferes with the mechanisms used to manage cache recovery time to meet FAST_START_MTTR_TARGET. 
--------------------------------------------------



I think it's time to seriously consider how to control I/O load by PostgreSQL.  That would be surely difficult, but commercial databases have overcome the difficulties.

From: "Bruce Momjian" <bruce@momjian.us>
> I think the bgwriter should keep track of how far it is to the next
> checkpoint, and use that information to increase write activity. 
> Basically now, during a checkpoint, the bgwriter does a full buffer scan
> and fsync's all dirty files, so it changes from the configuration
> parameter-defined behavior right to 100% activity.  I think it would be
> ideal if we could ramp up the writes so that when it is 95% to the next
> checkpoint, it can be operating at 95% of the activity it would do
> during a checkpoint.

Doesn't this mean that the transaction response time get worse as the checkpoint approaches?
Hmm... I think PostgreSQL needs to take more care to be gentle to transactions.  For example, auto-tuning the writing speed according to the speed of WAL accumulation.

I performed some simple tests, and I'll show the results below.  I'm glad if someone would be interested.
First, my environment is:

CPU: Pentium Xeon 3.6GHz * 2, HyperThreading enabled
RAM: 8GB (free command shows only 7.5GB, why?)
disk: Ultra320 SCSI
Data files and WAL are located on two separate disks connected to one SCSI board
The data files are stored on /dev/sde.
The WAL files are stored on /dev/sdd.
Both disks are formatted with ext3 file system.
OS: RHEL 4.0 for AMD64/EM64T
kernel: 2.6.9-42.ELsmp
PostgreSQL: 8.2.0 64-bit

Some common settings are:

[postgresql.conf]
  shared_buffers = 4GB
  # wal_buffers = 64KB
[/proc/sys/vm/dirty*]
  Untouched.  i.e. left as defaults

With various settings for bgwriter_* and checkpoint_*, I executed "pgbench -c16 -t100 postgres" five times in succession.  The scaling factor is 50, which creates 800MB of data.
The result of "iostat -x 5" is included in some tests.  WAL and data files are located on sdd and sde respectively.

(1) The default case
With default settings of bgwriter_* and checkpoint_*, the tps's were as follows:

235  80  226 77  240

You can see the drops probably due to checkpoints.  Are they really due to checkpoints?


[disk usage]
When the tps is low, the %util of disk for data files is high, and %util of disk for WAL is low.  Why is transaction logging is disturbed by cleaning and/or syncing activity?  While the bgwriter is fsync()ing, it does not lock any data structures that the transactions want to access.  Even though they share the same SCSI controller and bus, they are different disks.  The bandwidth does not appear to be exhausted, since Ultra320 is said to have 256MB band width in practice.

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 923.05  0.00 120.84    0.00 8349.50     0.00  4174.75    69.09     1.29   10.83   7.27  87.90
sde          0.00  35.27  0.00 21.44    0.00  453.71     0.00   226.85    21.16     3.59  167.36   3.21   6.89
sdd          0.00 877.64  0.00 117.76    0.00 7964.87     0.00  3982.44    67.63     1.19   10.11   7.21  84.91
sde          0.00  41.92  0.00 28.94    0.00  566.87     0.00   283.43    19.59     4.88  168.57   2.58   7.47
sdd          0.00 431.34  0.00 59.28    0.00 3921.76     0.00  1960.88    66.15     0.61   10.27   7.46  44.25
sde          0.00 1302.59  0.00 242.71    0.00 18503.79     0.00  9251.90    76.24  2017.52  995.53   2.34  56.85
sdd          0.00   3.61  0.00  0.60    0.00   33.67     0.00    16.83    56.00     0.26  438.33 324.33  19.50
sde          0.00   0.00  0.00 426.05    0.00    0.00     0.00     0.00     0.00  2697.42 4646.84   2.35 100.22
sdd          0.00   7.82  0.00  1.80    0.00   78.56     0.00    39.28    43.56     0.03   13.89  12.00   2.16
sde          0.00   0.60  0.00 343.29    0.00   19.24     0.00     9.62     0.06   822.85 9681.35   2.89  99.30
sdd          0.00 950.00  0.00 123.60    0.00 8587.20     0.00  4293.60    69.48     1.30   10.56   7.07  87.42
sde          0.00  28.40  0.00 22.00    0.00  401.60     0.00   200.80    18.25     2.53  115.06   3.13   6.88
sdd          0.00 794.81  0.00 116.57    0.00 7292.61     0.00  3646.31    62.56     1.24   10.60   7.85  91.56
sde          0.00  43.51  0.00 34.73    0.00  625.95     0.00   312.97    18.02     6.02  173.20   3.04  10.56
sdd          0.00 856.60  0.00 119.60    0.00 7808.00     0.00  3904.00    65.28     1.24   10.37   7.19  86.00
sde          0.00 1355.20  0.00 36.40    0.00 19227.20     0.00  9613.60   528.22    60.20  142.29   3.76  13.70
sdd          0.00   0.60  0.00  0.00    0.00    6.40     0.00     3.20     0.00     0.03    0.00   0.00   3.32
sde          0.00   0.00  0.00 448.40    0.00    0.00     0.00     0.00     0.00  3924.63 2591.94   2.23 100.02
sdd          0.00   0.00  0.00  0.40    0.00    1.60     0.00     0.80     4.00     0.19  559.50 476.50  19.06
sde          0.00   0.00  0.00 362.60    0.00    0.00     0.00     0.00     0.00  1836.17 7425.05   2.76 100.02
sdd          0.00 417.80  0.00 54.40    0.00 3779.20     0.00  1889.60    69.47     0.52    9.48   7.18  39.08
sde          0.00   1.00  0.00 197.40    0.00   25.60     0.00    12.80     0.13   268.59 11296.12   2.99  59.06
sdd          0.00 913.43  0.00 117.23    0.00 8243.69     0.00  4121.84    70.32     1.24   10.61   7.16  83.97
sde          0.00  29.06  0.00 21.84    0.00  407.21     0.00   203.61    18.64     2.88  132.03   2.70   5.89


(2) No write case
I disabled the bgwriter completely with the following settings:

bgwriter_delay = 200
bgwriter_* = 0
checkpoint_segments = 2100000000
checkpoint_timeout = 1h

The tps's were:

242  250  244  253  280

It seems no doubt that the activity of cleaning dirty buffers decreases performance greatly.  What is the real criminal, background writing or checkpointing?

[disk usage]
The data disk is not used, and the WAL disk is utilized well.

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 984.17  0.00 126.45    0.00 8884.97     0.00  4442.48    70.26     1.37   10.85   7.12  90.02
sde          0.00  25.85  0.00  5.41    0.00  251.70     0.00   125.85    46.52     0.66  121.41   8.44   4.57
sdd          0.00 975.85  0.00 124.35    0.00 8798.40     0.00  4399.20    70.75     1.29   10.43   7.12  88.58
sde          0.00  19.96  0.00  2.59    0.00  180.44     0.00    90.22    69.54     0.24   93.46  13.31   3.45
sdd          0.00 946.69  0.00 127.25    0.00 8593.19     0.00  4296.59    67.53     1.25    9.83   7.20  91.56
sde          0.00   7.21  0.00  1.00    0.00   65.73     0.00    32.87    65.60     0.01   14.60  10.00   1.00
sdd          0.00 890.40  0.00 126.40    0.00 8134.40     0.00  4067.20    64.35     1.17    9.23   7.17  90.62
sde          0.00  19.40  0.00  2.40    0.00  174.40     0.00    87.20    72.67     0.10   40.33   8.25   1.98
sdd          0.00 894.80  0.00 132.40    0.00 8217.60     0.00  4108.80    62.07     1.16    8.78   7.04  93.18
sde          0.00  14.80  0.00  2.60    0.00  139.20     0.00    69.60    53.54     0.09   33.23   6.23   1.62
sdd          0.00 946.80  0.00 131.60    0.00 8627.20     0.00  4313.60    65.56     1.18    8.99   6.77  89.06
sde          0.00  20.00  0.00  2.60    0.00  180.80     0.00    90.40    69.54     0.15   59.23   9.46   2.46
sdd          0.00 196.00  0.00 29.00    0.00 1798.40     0.00   899.20    62.01     0.27    9.27   7.53  21.84
sde          0.00   7.80  0.00  4.00    0.00   94.40     0.00    47.20    23.60     0.24   60.60   7.70   3.08
sdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.00   14.00  14.00   0.28
sde          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.00   18.00  18.00   0.36
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00


(3) No checkpoint case
I allowed the background writing with default bgwriter_* settings, and disabled only checkpoints as follows:

checkpoint_segments = 2100000000
checkpoint_timeout = 1h

The tps's were:

229  252  256  292  276

The background cleaning activity, at least with default settings, doesn't seem to have little impact on performance.

[disk usage]
This is similar to case (3); only WAL disk is well-utilized.

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 932.67  0.00 129.86    0.00 8500.20     0.00  4250.10    65.46     1.37   10.59   7.01  91.00
sde          0.00  24.65  0.00  6.61    0.00  250.10     0.00   125.05    37.82     0.60   90.67   6.21   4.11
sdd          0.00 981.16  0.00 125.45    0.00 8852.91     0.00  4426.45    70.57     1.21    9.66   7.18  90.02
sde          0.00  19.24  0.00  2.61    0.00  174.75     0.00    87.37    67.08     0.14   54.31   9.62   2.51
sdd          0.00 980.84  0.00 131.34    0.00 8897.41     0.00  4448.70    67.74     1.28    9.79   6.94  91.18
sde          0.00   7.19  0.00  1.00    0.00   65.47     0.00    32.73    65.60     0.02   16.60   9.20   0.92
sdd          0.00 950.50  0.00 124.75    0.00 8603.59     0.00  4301.80    68.97     1.24    9.92   7.18  89.56
sde          0.00  19.76  0.00  2.40    0.00  177.25     0.00    88.62    74.00     0.12   49.83   9.25   2.22
sdd          0.00 979.36  0.00 133.07    0.00 8901.00     0.00  4450.50    66.89     1.24    9.25   6.78  90.16
sde          0.00  14.23  0.00  2.40    0.00  133.07     0.00    66.53    55.33     0.09   39.42   7.25   1.74
sdd          0.00 904.00  0.00 125.60    0.00 8233.60     0.00  4116.80    65.55     1.19    9.56   7.31  91.80
sde          0.00  20.00  0.00  2.60    0.00  180.80     0.00    90.40    69.54     0.16   62.15  10.23   2.66
sdd          0.00  93.80  0.00 12.80    0.00  851.20     0.00   425.60    66.50     0.09    7.11   6.33   8.10
sde          0.00   7.00  0.00  0.80    0.00   62.40     0.00    31.20    78.00     0.01   18.25  12.25   0.98


(4) No fsync() case
I prevented fsync() from being executed by disabling smgrsync() in FlushBufferPool().  The postgresql.conf settings are the same as case (1).

The tps's were:

236  112  215  216  221

The drop isn't as big as the one in case (1), however, there is still great decrease.

[disk usage]
When the tps drops, the usage of data disk increases and the usage of WAL disk decreases greatly.

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 1004.01  0.00 122.65    0.00 9013.23     0.00  4506.61    73.49     1.38   11.21   7.35  90.16
sde          0.00  22.44  0.00  4.41    0.00  214.83     0.00   107.41    48.73     0.50  112.59   9.14   4.03
sdd          0.00 981.84  0.00 129.34    0.00 8891.02     0.00  4445.51    68.74     1.25    9.64   6.95  89.88
sde          0.00  19.36  0.00  2.79    0.00  177.25     0.00    88.62    63.43     0.13   48.07   8.07   2.26
sdd          0.00 483.20  0.00 66.00    0.00 4392.00     0.00  2196.00    66.55     1.24   16.94  14.55  96.00
sde          0.00 1344.80  0.00 214.80    0.00 19139.20     0.00  9569.60    89.10  2258.28 1188.81   2.44  52.48
sdd          0.00  25.40  0.00  6.00    0.00  252.80     0.00   126.40    42.13     1.03  192.40 166.47  99.88
sde          0.00   0.00  0.00 422.00    0.00    0.00     0.00     0.00     0.00  3071.61 4873.81   2.37 100.02
sdd          0.00   3.20  0.00  1.40    0.00   33.60     0.00    16.80    24.00     0.10   72.43  53.00   7.42
sde          0.00   0.00  0.00 328.00    0.00    0.00     0.00     0.00     0.00  1215.04 9908.95   3.05 100.02
sdd          0.00 699.20  0.00 101.40    0.00 6408.02     0.00  3204.01    63.19     0.97    9.49   6.95  70.46
sde          0.00   1.80  0.00 80.16    0.00   20.84     0.00    10.42     0.26    43.67 12828.41   3.04  24.35
sdd          0.00 870.00  0.00 122.60    0.00 7939.20     0.00  3969.60    64.76     1.26   10.30   7.21  88.36
sde          0.00  19.60  0.00 15.00    0.00  276.80     0.00   138.40    18.45     1.13   75.44   4.79   7.18
sdd          0.00 931.74  0.00 123.75    0.00 8442.32     0.00  4221.16    68.22     1.22    9.87   7.19  88.92
sde          0.00  35.33  0.00 24.35    0.00  477.45     0.00   238.72    19.61     3.19  130.98   2.77   6.75
sdd          0.00 117.03  0.00 17.03    0.00 1072.55     0.00   536.27    62.96     0.19   11.08   8.71  14.83
sde          0.00 1379.56  0.00 356.71    0.00 19608.82     0.00  9804.41    54.97  3323.09 1781.21   2.50  89.20
sdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.12  603.00 603.00  12.06
sde          0.00   0.00  0.00 406.40    0.00    0.00     0.00     0.00     0.00  2442.40 6098.31   2.46 100.02
sdd          0.00  49.70  0.00  7.58    0.00  461.48     0.00   230.74    60.84     0.09   11.95   9.18   6.97
sde          0.00   1.80  0.00 301.60    0.00   22.36     0.00    11.18     0.07   671.50 10983.53   3.02  91.06
sdd          0.00 876.71  0.00 124.50    0.00 8011.24     0.00  4005.62    64.35     1.21    9.67   7.08  88.11
sde          0.00  29.52  0.00 22.89    0.00  419.28     0.00   209.64    18.32     3.62  158.27   3.30   7.55


(5) No write by PostgreSQL, but fsync() by another program case
I disabled the bgwriter as case (2) so that PostgreSQL would not clean dirty buffers.  While running pgbench, I ran another program called write_fsync which opens a file without O_SYNC/O_DIRECT, writes 1000MB data sequentially in 8KB units, and finally fsync()s once.  The pseudo code is:

fd = open(path, O_CREAT | O_RDWR);
for (till 1000MB of data is written) {
    write(fd, buf, 8192);
}
fsync(fd);

The write_fsync program took 1 seconds to perform all write()s and 171 seconds to perform fsync().
I started write_fsync and pgbench simultaneously.

The tps's were:

9  223  260  283  292

The first run of pgbench seems to be stopped by fsync() of write_fsync program.  The first run took about 177 seconds (= 1600 transactions / 9 tps).  That is almost the same as the time write_fsync took to complete.

[disk usage]
The flood of I/O to sde is preventing the I/O to WAL disk caused by transactions?  Is fsync() exhausting the Ultra320 SCSI band width?  Oh, what an unfair I/O scheduling.  What a terrible fsync()!

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 174.80  0.00 22.40    0.00 1579.20     0.00   789.60    70.50     1.40   62.54  35.96  80.56
sde          0.00 25503.40  0.00 79.40    0.00 217808.00     0.00 108904.00  2743.17  6033.32 1878.37  10.17  80.76
sdd          0.00 220.00  0.00 30.20    0.00 2001.60     0.00  1000.80    66.28     1.38   45.51  32.73  98.84
sde          0.00 1062.00  0.00 73.60    0.00 9062.40     0.00  4531.20   123.13  8226.99 4554.48  13.59 100.02
sdd          0.00 273.20  0.20 35.40    1.60 2468.80     0.80  1234.40    69.39     1.34   37.62  27.69  98.58
sde          0.00 956.80  0.00 62.80    0.00 8166.40     0.00  4083.20   130.04  8211.84 7932.02  15.93 100.02
sdd          0.00  38.00  0.00  5.20    0.00  344.00     0.00   172.00    66.15     0.19   37.62  28.15  14.64
sde          0.00 1433.60  0.00 94.40    0.00 12238.40     0.00  6119.20   129.64  8208.65 12239.28  10.60 100.02
sdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.01   41.00  41.00   0.82
sde          0.00 1346.69  0.00 92.18    0.00 11491.78     0.00  5745.89   124.66  8225.32 13777.95  10.87 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1245.51  0.00 85.63    0.00 10628.34     0.00  5314.17   124.12  8192.77 18378.38  11.66  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1440.00  0.00 90.00    0.00 12288.00     0.00  6144.00   136.53  8209.04 22073.72  11.11 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1343.49  0.00 92.59    0.00 11466.13     0.00  5733.07   123.84  8225.70 27162.72  10.82 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1341.32  0.00 87.23    0.00 11445.91     0.00  5722.95   131.22  8192.85 29123.88  11.44  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1344.00  0.00 89.00    0.00 11468.80     0.00  5734.40   128.86  8209.05 34486.55  11.24 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1248.00  0.00 89.20    0.00 10649.60     0.00  5324.80   119.39  8209.54 36305.95  11.21 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1532.80  0.00 98.40    0.00 13081.60     0.00  6540.80   132.94  8209.67 42358.42  10.16 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1440.00  0.00 97.20    0.00 12288.00     0.00  6144.00   126.42  8209.06 42340.07  10.29 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1341.32  0.00 89.82    0.00 11445.91     0.00  5722.95   127.43  8192.69 47948.91  11.11  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1341.32  0.00 87.43    0.00 11445.91     0.00  5722.95   130.92  8192.84 51957.99  11.42  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1244.80  0.00 86.00    0.00 10624.00     0.00  5312.00   123.53  8208.91 55625.28  11.63 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1346.69  0.00 88.78    0.00 11491.78     0.00  5745.89   129.44  8225.61 59164.26  11.29 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 1440.00  0.00 98.40    0.00 12288.00     0.00  6144.00   124.88  8208.69 63661.97  10.16 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00  42.80  0.00 92.80    0.00  374.40     0.00   187.20     4.03  7971.70 66171.48  10.78 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 88.60    0.00    0.00     0.00     0.00     0.00  7527.88 70771.08  11.29 100.00
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 90.78    0.00    0.00     0.00     0.00     0.00  7094.69 74257.76  11.04 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 97.60    0.00    0.00     0.00     0.00     0.00  6608.81 79122.89  10.25 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 89.18    0.00    0.00     0.00     0.00     0.00  6158.38 83935.30  11.24 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 93.81    0.00    0.00     0.00     0.00     0.00  5685.10 85984.61  10.64  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 100.00    0.00    0.00     0.00     0.00     0.00  5210.90 85861.43  10.00 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 88.78    0.00    0.00     0.00     0.00     0.00  4732.95 93815.11  11.29 100.22
sdf          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 96.40    0.00    0.00     0.00     0.00     0.00  4266.87 100657.40  10.38 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 99.40    0.00    0.00     0.00     0.00     0.00  3785.82 101838.25  10.06 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 104.00    0.00    0.00     0.00     0.00     0.00  3263.02 104180.10   9.62 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 109.20    0.00    0.00     0.00     0.00     0.00  2742.11 103275.58   9.16 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 100.60    0.00    0.00     0.00     0.00     0.00  2215.01 110809.10   9.96 100.22
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 99.00    0.00    0.00     0.00     0.00     0.00  1715.49 114701.50  10.10 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 98.60    0.00    0.00     0.00     0.00     0.00  1216.70 118355.58  10.12  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 102.20    0.00    0.00     0.00     0.00     0.00   714.00 136172.56   9.79 100.02
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00 87.80    0.00    0.00     0.00     0.00     0.00   242.24 169761.75  11.39 100.04
sdd          0.00 881.96  0.00 120.24    0.00 8017.64     0.00  4008.82    66.68     1.21   10.09   7.15  85.99
sde          0.00   9.22  0.00  7.01    0.00   96.19     0.00    48.10    13.71     0.48 102139.60   8.03   5.63
sdd          0.00 886.60  0.00 122.80    0.00 8076.80     0.00  4038.40    65.77     1.23   10.02   7.38  90.58
sde          0.00  20.60  0.00  2.40    0.00  184.00     0.00    92.00    76.67     0.20   83.42  16.67   4.00
sdd          0.00 889.82  0.00 122.55    0.00 8097.41     0.00  4048.70    66.07     1.21    9.85   7.25  88.90
sde          0.00  24.55  0.00  2.79    0.00  220.36     0.00   110.18    78.86     0.16   48.64  12.50   3.49
sdd          0.00 1013.23  0.00 125.45    0.00 9114.23     0.00  4557.11    72.65     1.20    9.53   7.14  89.60
sde          0.00  17.23  0.00  3.21    0.00  161.92     0.00    80.96    50.50     0.10   38.75   6.88   2.20
sdd          0.00 965.13  0.00 131.66    0.00 8771.14     0.00  4385.57    66.62     1.22    9.25   6.73  88.66
sde          0.00  20.64  0.00  2.61    0.00  185.97     0.00    92.99    71.38     0.20   78.08  12.15   3.17
sdd          0.00 651.10  0.00 84.23    0.00 5881.04     0.00  2940.52    69.82     0.79    9.33   7.07  59.56
sde          0.00   6.99  0.00  0.80    0.00   62.28     0.00    31.14    78.00     0.01   16.25  11.50   0.92


(6) case (5) + O_SYNC by write_fsync
I did the same as case (5).  However, in this case, I specified O_SYNC to open() when running write_fsync program.
For information, write_fsync program took 849 seconds to complete all write()s and 0 seconds to perform fsync().

The tps's were:

97  114  126  112  125

Doesn't this look relatively steady?

[disk usage]
In this case, both data disk and WAL disk are utilized in almost full.  What does this mean compared to case the previous case?  fsync() is a tyrant?

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 370.26  0.00 41.52    0.00 3297.41     0.00  1648.70    79.42     1.29   30.95  19.99  82.99
sde          0.00 112.18  0.00 91.42    0.00 1670.26     0.00   835.13    18.27     1.01   10.90  10.85  99.20
sdd          0.00 398.80  0.00 58.20    0.00 3654.40     0.00  1827.20    62.79     1.39   23.89  16.69  97.12
sde          0.00  76.20  0.00 78.40    0.00 1195.20     0.00   597.60    15.24     1.94   24.75  12.70  99.54
sdd          0.00 397.00  0.00 58.40    0.00 3643.20     0.00  1821.60    62.38     1.29   22.09  16.77  97.94
sde          0.00  89.00  0.00 83.40    0.00 1379.20     0.00   689.60    16.54     1.10   13.29  11.95  99.70
sdd          0.00 435.87  0.00 59.12    0.00 3959.92     0.00  1979.96    66.98     1.34   22.56  16.05  94.89
sde          0.00  91.58  0.00 78.56    0.00 1361.12     0.00   680.56    17.33     1.07   13.59  12.69  99.68
sdd          0.00 428.54  0.00 55.69    0.00 3878.64     0.00  1939.32    69.65     1.41   24.71  17.48  97.33
sde          0.00  82.24  0.00 81.44    0.00 1309.38     0.00   654.69    16.08     1.00   12.32  12.23  99.60
sdd          0.00 455.00  0.00 57.60    0.00 4096.00     0.00  2048.00    71.11     1.38   24.58  16.64  95.82
sde          0.00  83.80  0.00 77.80    0.00 1292.80     0.00   646.40    16.62     1.02   13.12  12.83  99.78
sdd          0.00 448.30  0.00 56.11    0.00 4038.48     0.00  2019.24    71.97     1.36   24.14  17.02  95.51
sde          0.00  83.17  0.00 71.54    0.00 1236.07     0.00   618.04    17.28     1.09   15.31  13.92  99.60
sdd          0.00 432.00  0.00 57.60    0.00 3915.20     0.00  1957.60    67.97     1.35   23.45  16.34  94.12
sde          0.00  82.80  0.00 79.20    0.00 1297.60     0.00   648.80    16.38     1.29   16.27  12.58  99.66
sdd          0.00 444.40  0.00 52.80    0.00 3976.00     0.00  1988.00    75.30     1.27   23.92  18.05  95.30
sde          0.00  89.80  0.00 82.60    0.00 1379.20     0.00   689.60    16.70     1.06   12.83  12.06  99.64
sdd          0.00 371.66  0.00 57.88    0.00 3437.92     0.00  1718.96    59.39     1.31   22.68  16.85  97.54
sde          0.00  89.22  0.00 83.43    0.00 1381.24     0.00   690.62    16.56     1.04   12.45  11.94  99.66
sdd          0.00 466.53  0.00 63.13    0.00 4235.67     0.00  2117.84    67.10     1.26   19.99  15.31  96.63
sde          0.00  73.75  0.00 73.75    0.00 1179.96     0.00   589.98    16.00     1.01   13.62  13.56  99.98
sdd          0.00 329.86  0.00 42.89    0.00 2983.57     0.00  1491.78    69.57     1.23   28.44  22.43  96.17
sde          0.00  98.00  0.00 91.78    0.00 1518.24     0.00   759.12    16.54     1.03   11.26  10.88  99.84
sdd          0.00 437.05  0.00 57.17    0.00 3952.19     0.00  1976.10    69.13     1.32   23.11  16.77  95.88
sde          0.00  88.84  0.00 76.69    0.00 1324.30     0.00   662.15    17.27     1.30   16.97  12.92  99.08
sdd          0.00 420.24  0.00 60.52    0.00 3846.09     0.00  1923.05    63.55     1.26   21.10  16.15  97.76
sde          0.00  84.37  0.00 81.36    0.00 1325.85     0.00   662.93    16.30     1.07   13.11  12.26  99.76
sdd          0.00 104.20  0.00 15.80    0.00  958.40     0.00   479.20    60.66     0.32   20.19  16.82  26.58
sde          0.00 140.40  0.00 138.40    0.00 2230.40     0.00  1115.20    16.12     1.06    7.62   7.19  99.50
sdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.01   30.00  30.00   0.60
sde          0.00 158.40  0.00 158.60    0.00 2536.00     0.00  1268.00    15.99     1.00    6.30   6.28  99.56
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 152.71  0.00 156.91    0.00 2476.95     0.00  1238.48    15.79     1.02    6.50   6.36  99.86
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 150.30  0.00 151.70    0.00 2415.97     0.00  1207.98    15.93     1.00    6.60   6.56  99.48
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 157.88  0.00 158.88    0.00 2534.13     0.00  1267.07    15.95     1.00    6.30   6.25  99.24
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 158.12  0.00 159.12    0.00 2537.88     0.00  1268.94    15.95     1.03    6.44   6.28  99.98
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 150.20  0.00 151.20    0.00 2411.20     0.00  1205.60    15.95     1.01    6.67   6.60  99.76
sdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.01   37.00  37.00   0.74
sde          0.00 159.80  0.00 161.60    0.00 2571.20     0.00  1285.60    15.91     1.03    6.35   6.16  99.60
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 157.11  0.00 158.12    0.00 2521.84     0.00  1260.92    15.95     1.01    6.38   6.31  99.82
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 157.00  0.00 157.20    0.00 2513.60     0.00  1256.80    15.99     1.00    6.36   6.34  99.66
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 153.69  0.00 153.89    0.00 2460.68     0.00  1230.34    15.99     1.00    6.48   6.46  99.36
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 155.71  0.00 155.91    0.00 2492.99     0.00  1246.49    15.99     1.00    6.40   6.40  99.72
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 157.60  0.00 158.60    0.00 2529.60     0.00  1264.80    15.95     1.01    6.34   6.28  99.60
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 158.88  0.00 159.08    0.00 2543.71     0.00  1271.86    15.99     1.00    6.28   6.26  99.60
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 153.60  0.00 153.80    0.00 2459.20     0.00  1229.60    15.99     1.00    6.50   6.48  99.70
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 159.20  0.00 159.80    0.00 2552.00     0.00  1276.00    15.97     1.00    6.28   6.24  99.74
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 159.92  0.00 160.12    0.00 2560.32     0.00  1280.16    15.99     1.00    6.25   6.24  99.88
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 156.20  0.00 156.40    0.00 2500.80     0.00  1250.40    15.99     1.00    6.42   6.38  99.78
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 160.00  0.00 160.20    0.00 2561.60     0.00  1280.80    15.99     1.00    6.24   6.22  99.60
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 147.70  0.00 152.30    0.00 2400.00     0.00  1200.00    15.76     1.01    6.67   6.53  99.46
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 155.51  0.00 156.51    0.00 2496.19     0.00  1248.10    15.95     1.01    6.45   6.38  99.80
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 154.80  0.00 155.80    0.00 2484.80     0.00  1242.40    15.95     1.00    6.44   6.40  99.66
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 152.10  0.00 153.11    0.00 2441.68     0.00  1220.84    15.95     1.01    6.59   6.53  99.92
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 156.20  0.00 156.40    0.00 2500.80     0.00  1250.40    15.99     1.01    6.44   6.38  99.72
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 158.80  0.00 159.80    0.00 2548.80     0.00  1274.40    15.95     1.01    6.31   6.24  99.72
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 156.60  0.00 157.60    0.00 2513.60     0.00  1256.80    15.95     1.00    6.36   6.32  99.68
sdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde          0.00 158.60  0.00 158.80    0.00 2539.20     0.00  1269.60    15.99     1.00    6.30   6.28  99.70


(7) O_SYNC case
I made PostgreSQL use O_SYNC.  In this run, the postgresql.conf settings are the same as case (1), that is, background writing and checkpointing occurs normally.

The tps's were:

182  103  41  50  74

This is certainly low, and you will probably not want to face.  But I think this is the position where we have to start to provide the real operation steadiness.
I suppose we have to identify seriously what is stopping the transaction processing.  Just I/O load?  CPU can't work for transactions because of I/O?  I don't think so, or the phrase "too much I/O load is decreasing transaction performance" is vague.  If CPU could not work during I/O, is there any point in asynchronous I/O?

[disk usage]

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd          0.00 451.00  0.00 63.00    0.00 4113.60     0.00  2056.80    65.30     0.90   14.21   9.88  62.22
sde          0.00  59.80  0.00 34.80    0.00  756.80     0.00   378.40    21.75     0.54   15.59  14.01  48.74
sdd          0.00 888.38  0.00 114.23    0.00 8019.24     0.00  4009.62    70.20     1.27   11.13   7.73  88.26
sde          0.00  22.65  0.00 14.43    0.00  298.20     0.00   149.10    20.67     0.21   14.75  13.10  18.90
sdd          0.00 737.20  0.00 105.00    0.00 6739.20     0.00  3369.60    64.18     1.23   11.69   8.69  91.26
sde          0.00  34.00  0.00 21.20    0.00  440.00     0.00   220.00    20.75     0.27   12.92  12.90  27.34
sdd          0.00 419.56  0.00 60.48    0.00 3838.72     0.00  1919.36    63.47     0.65   10.69   7.81  47.21
sde          0.00  78.64  0.00 74.25    0.00 1224.75     0.00   612.38    16.49     0.63    8.41   8.31  61.72
sdd          0.00 100.80  0.00 15.43    0.00  931.46     0.00   465.73    60.36     0.37   24.08  17.56  27.09
sde          0.00 110.42  0.00 112.02    0.00 1779.56     0.00   889.78    15.89     1.12    9.99   8.69  97.37
sdd          0.00 270.34  0.00 45.69    0.00 2528.26     0.00  1264.13    55.33     1.08   23.32  16.44  75.13
sde          0.00  84.17  0.00 71.34    0.00 1244.09     0.00   622.04    17.44     1.20   16.75  13.87  98.94
sdd          0.00 340.36  0.00 55.07    0.00 3165.01     0.00  1582.50    57.47     1.43   26.04  15.03  82.74
sde          0.00  58.65  0.00 58.85    0.00  939.96     0.00   469.98    15.97     0.97   16.60  16.52  97.24
sdd          0.00  31.93  0.00  4.82    0.00  290.76     0.00   145.38    60.33     0.12   26.33  14.50   6.99
sde          0.00 127.51  0.00 127.71    0.00 2041.77     0.00  1020.88    15.99     1.00    7.87   7.81  99.76
sdd          0.00  88.80  0.00 17.60    0.00  852.80     0.00   426.40    48.45     0.30   16.77  14.99  26.38
sde          0.00 110.80  0.00 111.00    0.00 1774.40     0.00   887.20    15.99     1.00    8.98   8.96  99.42
sdd          0.00 213.40  0.00 45.20    0.00 2067.20     0.00  1033.60    45.73     0.81   17.96  14.63  66.14
sde          0.00  81.20  0.00 74.60    0.00 1246.40     0.00   623.20    16.71     1.01   13.43  13.33  99.42
sdd          0.00 254.00  0.00 46.80    0.00 2408.00     0.00  1204.00    51.45     1.14   24.37  15.86  74.24
sde          0.00  69.20  0.00 69.40    0.00 1108.80     0.00   554.40    15.98     1.00   14.42  14.29  99.18
sdd          0.00 106.20  0.00 17.20    0.00  985.60     0.00   492.80    57.30     0.37   21.60  15.05  25.88
sde          0.00 105.60  0.00 107.60    0.00 1705.60     0.00   852.80    15.85     1.09   10.10   9.23  99.30
sdd          0.00 101.00  0.00 14.66    0.00  926.91     0.00   463.45    63.23     0.37   24.78  15.45  22.65
sde          0.00 124.50  0.00 120.68    0.00 1963.05     0.00   981.53    16.27     1.10    9.12   8.24  99.46
sdd          0.00  22.91  0.00  3.78    0.00  211.95     0.00   105.98    56.00     0.13   35.63  20.11   7.61
sde          0.00 135.86  0.00 132.07    0.00 2141.83     0.00  1070.92    16.22     1.07    8.11   7.42  97.99
sdd          0.00 110.40  0.00 20.40    0.00 1048.00     0.00   524.00    51.37     0.42   20.31  15.51  31.64
sde          0.00 107.00  0.00 107.00    0.00 1712.00     0.00   856.00    16.00     1.00    9.35   9.25  98.96
sdd          0.00 372.06  0.00 55.89    0.00 3423.55     0.00  1711.78    61.26     1.30   23.35  16.64  93.01
sde          0.00  73.25  0.00 68.26    0.00 1132.14     0.00   566.07    16.58     1.19   17.50  14.32  97.78
sdd          0.00 400.80  0.00 60.40    0.00 3694.40     0.00  1847.20    61.17     1.29   21.19  15.50  93.62
sde          0.00  74.40  0.00 66.00    0.00 1123.20     0.00   561.60    17.02     1.23   18.55  14.91  98.40
sdd          0.00 147.70  0.00 25.85    0.00 1381.96     0.00   690.98    53.46     0.58   23.00  14.74  38.12
sde          0.00  97.39  0.00 97.60    0.00 1559.92     0.00   779.96    15.98     1.00   10.23  10.15  99.10
sdd          0.00 128.34  0.00 17.96    0.00 1175.25     0.00   587.62    65.42     0.47   25.48  16.88  30.32
sde          0.00 122.55  0.00 124.35    0.00 1975.25     0.00   987.62    15.88     1.09    8.76   7.96  99.00
sdd          0.00 331.06  0.00 54.71    0.00 3086.17     0.00  1543.09    56.41     1.16   21.40  15.84  86.67
sde          0.00  83.37  0.00 74.15    0.00 1260.12     0.00   630.06    16.99     1.19   16.10  13.06  96.81
sdd          0.00 335.20  0.00 52.60    0.00 3097.60     0.00  1548.80    58.89     1.32   25.11  17.59  92.54
sde          0.00  63.00  0.00 57.40    0.00  963.20     0.00   481.60    16.78     1.18   20.61  17.14  98.38
sdd          0.00 178.16  0.00 26.25    0.00 1636.87     0.00   818.44    62.35     0.58   22.24  14.09  36.99
sde          0.00 114.03  0.00 108.22    0.00 1777.96     0.00   888.98    16.43     0.99    9.15   8.83  95.51
sdd          0.00 995.20  0.00 85.20    0.00 8641.60     0.00  4320.80   101.43    85.62 1004.98  11.31  96.36
sde          0.00  35.80  0.00 36.60    0.00  579.20     0.00   289.60    15.83     0.54   14.64  13.74  50.28
sdd          0.00 233.80  0.00 35.60    0.00 2155.20     0.00  1077.60    60.54     0.70   19.59  13.15  46.82
sde          0.00  87.60  0.00 86.80    0.00 1395.20     0.00   697.60    16.07     0.99   11.42  11.36  98.62
