I am confirming the performance of the FS pipeline.
4 Files(CSV typed) are generated with 30,000 rows per second.
During the first 20 mins, almost data is loaded just in time.
However, after that, the pipeline works getting slower and can not receive the data continuously.
Is there any configuration factor in enhancing the performance?
I share my test environment below.
4 Hosts(16 CPUs, 64 GB RAM, 1TB DISK per host)
1 MA, 1 CA. 2 Leaves
1 Gbps Network Bandwidth
16 CPUs, 64 GB RAM, 1TB DISK(same as Memsql server)
Files: 40 files are generated per sec(each file is 550K)
Thank you in advance.