Setting the process's CPU affinity is generally a bad idea, and will usually hurt performance more than it will help. The OS is still going to preempt you to run other tasks, so you're not going to avoid context switches. But now, since you've told the OS that you only want to run on a single core, the OS can't schedule you to run on another core. You have to wait in line while the other threads get a chance to run.
The actual cost of context switches is difficult to quantify, but they're typically on the order of 1-3 µs. The cost of switching cores will likely be well under half of that. Of course, this is completely ignoring the fact that any switch (regardless of which core you end up on) is likely going to pollute the cache, and that is going to hurt performance a lot more than the switch itself. And all of this is
much less than the time wasted waiting for other threads.
The only reason you should ever set the CPU affinity is when accessing a highly core-dependent feature such as the RDTSC or CPUID instructions, and then you should only leave the affinity mask set as long as necessary.
TL;DR: Don't set the process CPU affinity if you care about performance, you're only hurting yourself.