We've recently moved our replication system to a new server, and spent some time creating new indexes to help performance. At some point, we've done the opposite. Each subscriber is taking hours to synchronize after changes are made at the publisher (130 subscribers). Hardware on the new server benchmarks very well (2x8core CPU, 88GB RAM, 6xSSD RAID10, 2xSSD RAID for replication index data files), and significantly outperforms the old server. So I'm not thinking the new hardware is the issue, and am leaning toward an indexing problem.We analyzed every replication filter and join, and created covering indexes for each table article filter, and stored the replication indexes in a separate ndf on a separate 2xSSD RAID1. When I execute the replication filters manually, the query plans show significant improvement. But recently we've made small amounts of over night changes at the publisher (something we've always done regularly), and the next subscriber sync takes an extremely long time. It seems to take forever when enumerating and downloading the changes (100 changes per batch). Each subscriber only has about 1000 changes to download, and this used to happen in a minute or two instead of an hour.The biggest difference I've noticed is that it's extremely CPU intensive, each connection to the publisher seems to pin the CPU for the entire hour. I'm not sure what this might indicate, is it possible it's doing table scans? I'm not sure exactly what replication does at the publisher when it's enumerating the changes, and at the stage of downloading the changes to the subscriber? I've tried using profiler, but I don't see any one task that takes this long to run, and the disk I/O is minimal the entire time. The longest running entries in profiler are calls to "exec sys.sp_MSenumchanges_belongtopartition..." for our largest table article join, but I'm not sure how to dig into that further?It reminds me of a poorly indexed query that the server is scanning, but is there some way for me to confirm this? Is there any other performance measurements I can do to see what exactly is taking so long and using so much CPU? Any other info I can post to help?EDIT: I should mention that the merge agents seem to be taking forever on my join filters. I have a central 'Client' article that is filtered to send certain clients to certain subscribers. There are about a dozen table articles joined to each 'Client' record, and it seems to jam up when calculating changes to the joined tables. The joins use 2 columns each (a FK reference and a 'Replicate' bit). Each join predicate is indexed individually on each of the child tables, and there is a covering index on both columns together with all other columns included.EDIT: Another thought, statistics are set to update automatically, and are also updated manually every night with an index rebuild maintenance plan.EDIT: Another important point I skipped over is that in moving servers we upgraded from SQL2K8 R2 to SQL2K12.
↧