From what you are describing I don't think there is anything "wrong" with your barrel. All my rifle barrels foul. There is much debate on barrel cleanliness and I'm the guy who cleans his barrels completely after each range session. Only to foul them once again just before hunting season to verify my rifle's zero. Consider this:
There is an article on the web site 6mmbr where a number of benchrest shooters discuss their cleaning regimen. Many of them advocate removing all copper then firing a fouling shot(s) to foul the bore before shooting for groups.
Contrast this with what John Barsness wrote in an article titled "A Clean Barrel": "For decades it was suggested that a center-fire barrel should be cleaned every 20 rounds or so to maintain best accuracy. I’ve only owned two center fires in the past 20 years that shot larger groups after 20 rounds. The rest could go anywhere from 50 to 500 rounds without showing any decrease in accuracy. The rifle that can go 500 rounds without cleaning is the heavy-barreled Remington Model 700 in .223 Rem. that had its throat smoothed by fire-lapping. During its busy lifetime, this rifle has been shot more than any other I’ve owned. Twice it has gone 500 rounds without cleaning, and at the end of that string still shot the same size groups as it shot after 20 rounds. It also cleaned quite easily with three to four patches soaked in any decent chemical solvent. The only reason I cleaned it after 500 rounds was to see what effect that would have on accuracy. As often happens, accuracy was slightly worse for the first group or two. After 10 rounds or so, groups shrunk to the same size the rifle shot after 500 rounds. Of course, not all barrels act this way, but some do, and most can take far more shooting than shooters expect and still remain accurate. During a visit to the Sierra bullet factory I had an interesting talk with Kevin Thomas, the firm’s test shooter. Kevin once performed a test of his own with a .308 Win. barrel. Generally he cleans Sierra’s test barrels every 50 rounds or so, but once he kept shooting a .308 barrel for 300 rounds without cleaning. It just kept shooting tiny groups, and since all this shooting was done in Sierra’s indoor tunnel there was no doubt about the results. “One interesting thing was the best groups were shot between 250 and 300 rounds after cleaning,” Kevin said. “So why did you clean the barrel after 300 rounds?” I asked. Kevin shrugged. “General principles?”
So, I honestly wouldn't worry about the fouling you are seeing. Whether you choose to remove or not is up to you but it is perfectly normal for barrels to foul...even after just a few rounds. Good shooting!