Hello all! Our CNC team and consultants have come up empty-handed so I'm reaching out to the masses for advice. I'm a sysadmin turned BA so forgive me if I initially leave out pertinent information.
We have a pretty hefty custom job that processes incredibly slowly at the end of the month after we do back-ups and IPL the system. It normally processes 3800 records per minute and it drops to about 320 per minute when we see this.
Fact 1: The job in question joins F1201, F1217, F1731 and F17311 to pull asset information. It finds these asset numbers on sales orders in F4211. It performs a simple equation to basically estimate when these assets will need to be used again and it writes data to a custom table.
Fact 2: This job normally processes about 3800 records per minute and completes in 3 hours.
Fact 3: When this job is processing slowly, it only processes about 320 records per minute. This is the source of our despair.
Fact 4: This job runs every day at 12:05am and no users are in the system at this time.
Fact 5: The number of records processed is effectively the same every day. (It increases by about 20 a day at most but we're currently at 518,000.)
Fact 6: We do not have debug logging turned on for this job. When I turn it on though it severely impacts performance, essentially to the degree we see when the job is running slowly in production.
CNC says that no other jobs are running at the same time as this one, there are no table locks, no zombie kernels, and they also say that there are no access paths being rebuilt during/after the IPL.
The only thing we've been able to link it to is the IPL and even that isn't always the case. Last week we IPL'd production and didn't see the job run slow until a couple of days later. Today in our prototype environment the job ran normal twice, they did the IPL, and now the job runs slow every time I kick it off. Something is bottlenecking or throttling this job and we still can't find the root cause. We've gone down all sorts of rabbit holes involving creating new indices and modifying the SQL statements it generates, but the crux I keep coming back to is that this job runs fine every single day with the same records and same code until they do the IPL and backups on the last weekend of the month. The following Monday all hell breaks loose because this job runs and never completes because it's so slow.
All hints, pointers, and messages of encouragement are appreciated. Thank you!
System: DB2 on flash, iSeries, JDE tools 9.1.5, 5 processors allocated to production
We have a pretty hefty custom job that processes incredibly slowly at the end of the month after we do back-ups and IPL the system. It normally processes 3800 records per minute and it drops to about 320 per minute when we see this.
Fact 1: The job in question joins F1201, F1217, F1731 and F17311 to pull asset information. It finds these asset numbers on sales orders in F4211. It performs a simple equation to basically estimate when these assets will need to be used again and it writes data to a custom table.
Fact 2: This job normally processes about 3800 records per minute and completes in 3 hours.
Fact 3: When this job is processing slowly, it only processes about 320 records per minute. This is the source of our despair.
Fact 4: This job runs every day at 12:05am and no users are in the system at this time.
Fact 5: The number of records processed is effectively the same every day. (It increases by about 20 a day at most but we're currently at 518,000.)
Fact 6: We do not have debug logging turned on for this job. When I turn it on though it severely impacts performance, essentially to the degree we see when the job is running slowly in production.
CNC says that no other jobs are running at the same time as this one, there are no table locks, no zombie kernels, and they also say that there are no access paths being rebuilt during/after the IPL.
The only thing we've been able to link it to is the IPL and even that isn't always the case. Last week we IPL'd production and didn't see the job run slow until a couple of days later. Today in our prototype environment the job ran normal twice, they did the IPL, and now the job runs slow every time I kick it off. Something is bottlenecking or throttling this job and we still can't find the root cause. We've gone down all sorts of rabbit holes involving creating new indices and modifying the SQL statements it generates, but the crux I keep coming back to is that this job runs fine every single day with the same records and same code until they do the IPL and backups on the last weekend of the month. The following Monday all hell breaks loose because this job runs and never completes because it's so slow.
All hints, pointers, and messages of encouragement are appreciated. Thank you!
System: DB2 on flash, iSeries, JDE tools 9.1.5, 5 processors allocated to production