RedBeard's Dev Blog

Everyone knows that when you have performance problems, you need to profile your application to find hot-spots and optimize them. XNA doesn’t give very good feedback about where time is spent; for CPU usage there are a number of tools and instrumentation samples which will tell you where time is being spent, but the GPU isn’t quite so blessed. You can use tools like PIX for Windows (PIX on Xbox is excellent but is only available to XDK developers), but most XNA developers care about Xbox performance where the tools are more limited; built-in instrumentation is possibly your best bet on Xbox. On Windows there are also options such as NVPerfHud which are vendor-specific but can tell you things that are hidden to all other programs & APIs.

For in-game instrumentation, we have to deal with whatever public APIs are exposed; what tools are available to use through XNA? D3D9 supports Timestamp queries which signal when they complete and record a timestamp for comparison against other events. XNA 4.0 Hi-Def exposes only Occlusion Query, which can apparently be hijacked to behave like Event queries, which will also signal when complete but without a recorded timestamp. That timestamp is essential in order to diagnose where time is being spent on the GPU, but we need to generate it on the CPU. If we just issue a bunch of queries throughout the frame and don’t try to check them until the end of the frame or the beginning of the next, there’s a good chance that a bunch of them will become signalled before you check them, and they’ll all get the same timestamp, so we need to poll the query continuously or at least frequently.

My approach, which I encapsulated in a helper class called GpuTrace:

1. Use a queue to track all issued queries. The GPU will execute them in-order, so only the oldest one is worth checking for completion at any time.
2. Track a named label with each issued query for diagnosis
3. At frame start, start a stopwatch and issue an occlusion query (Begin and End in a pair)
4. Issue an occlusion query before and after any major GPU milestone (Clear, scene.draw, post-proc), you can fence in a bunch of Draw calls with just one query
5. After each query is issued, poll the front of the queue to see if it’s completed yet
6. Whenever a query on the polling queue has completed, record the CPU stopwatch timestamp and put it aside, perhaps in a separate queue of timestamps
7. At the end of the frame, spin-wait for all active queries to finish (perform the previous step on each)
8. Examine the list of timestamps. The first one indicates the latency between issuing the query and seeing it complete. Deltas between two timestamps indicate the time spent by the GPU completing whatever activity was between them.

• Keep in mind that adding all these queries, constantly polling them, and waiting for the GPU to finish at frame end will negatively impact your overall framerate, but the integrity of the data provided should be reliable.
• You may also extend the occlusion query mechanism to separate the Begin/End calls, so the query will actually record the number of modified pixels during its execution.

Here is a screenshot of my in-game profilers, and another. The GPU profiling information gleaned from this approach is displayed in the upper right. The numbers displayed are the time since frame start until the noted event, and the time delta between an event and the previous event. Also visible on the left is a multi-threaded CPU profiler which I have borrowed and cleaned up from here; my cleanup was mostly focused on garbage collection. At the bottom of the screen is a framerate graph, which will show spikes and instability where a simple FPS number hides them; I’d like to do more work on profiling instrumentation to diagnose framerate spikes in particular.

Garbage collection in XNA can be a big deal if you care about framerate hitching, which is visually jarring and can cause input polling problems. After a long enough time, any per-frame allocation will eventually be subject to garbage collection, which will pause all threads and search memory for live and dead objects to make space for more allocations. The garbage collector only has an opportunity to run if you allocate memory, therefore you can control when garbage collection happens by only performing allocations during expected downtime, such as during loading screens. The same wisdom applies for writing real-time applications in native code, but .NET provides more of a safety net if you do things quick & dirty at first; the GC will clean up your mess whereas a native app might just crash when running out of memory. Reducing and optimizing your allocations can also improve your loading times and reduce your minimum-spec hardware requirements.

Profiling

The best tool I’ve used for diagnosing garbage allocation & collection is CLR Profiler. It has a few quirks: the allocation graph has some rendering bugs with the line traces, it has no option to “ignore this node”, and the histogram views don’t always allow enough resolution to see all the objects which have been allocated (ie if they’re less than 1KB).

With that said, the “histogram by age” view is quite useful for finding per-frame allocations; start up your game and get it into the desired state, then just let it run for a couple of minutes with no adjustments. After running a while, open up the “histogram by age” view and see if any allocations are younger than 1 minute. An option in the right-click context menu will even show you a visual call-stack of how those allocations happened. Note that if you exit your application, it will probably generate a bunch of garbage on shutdown, so it’s probably safe to ignore very-young allocations so long as the middle ground is clear.

Another useful view is the “allocation graph”, which will show you the list of all major allocations by basic type if you scroll all the way to the right, and the visual call-stack of how they were conjured as you look to the left. This view can be a little misleading if you have large chunks of one-time allocations for scratch memory pools, and there is no option to ignore or exclude specific nodes, but anything that bubbles up to the top should warrant investigation.

Major Sources of Garbage

… as discovered in my current codebase, with suggested corrective actions.

• String manipulation:
  • Never use String.Format, for multiple reasons: hidden StringBuilder allocation, value types get boxed, hidden ToString calls allocate strings, and the params arguments create an array.
  • Never concatenate strings with + operator.
  • Never append non-string values onto a StringBuilder, these methods call ToString under the hood.
  • Never call StringBuilder.ToString, it allocates a string to return.
  • Do all string manipulation with a pre-allocated capacity-capped StringBuilder, use custom functions for numeric conversion like itoa & ftoa.
  • SpriteBatch can render StringBuilder objects without calling ToString.
  • I use a custom StringBuilderNG class (NG = no garbage) which wraps a standard StringBuilder and forwards only the “safe” methods which generate no garbage, and implements new methods for custom conversion of int & float values. This approach is more prone to bugs, but itoa and ftoa are relatively easy to implement.
• DateTime.Now: replace with Stopwatch.Elapsed when used for profiling
• params[] arguments: pre-allocate an array of appropriate size, and populate it immediately before the call. Don’t forget to null out the entries afterwards to avoid dangling references.
• Value type boxing for IComparer<T> & IEnumerable<T>: implement CompareTo on the underlying type, and use an explicit value-type Enumerator like in List<T>. This also helps reduce virtual function calls.
• Worker threads typically need scratch memory to operate on, this can be pre-allocated in a pool, and threads can grab a chunk of scratch memory when they start up and release it when they’re done

Other Resources

• Gavin Pugh’s blog post: Avoiding StringBuilder Garbage
• CLR Profiler

I discovered recently that threads on the Xbox are not managed or scheduled across CPU cores automatically. You must call SetProcessorAffinity from within the target thread, and that thread will run only on the specific core (hardware thread) from then onwards. I wrote some helper code at the top of my worker-thread management function (m_Threads is the list of potentially active thread objects). My “do thread work here” implementation just spins waiting for work to arrive in a queue.

private static void DoThreadWork(object threadObj)
{
  Thread myThread = threadObj as Thread;
#if XBOX
  // manually assign hardware thread affinity
  lock (m_Threads)
  {
    // avoid threads 0 & 2 (reserved for XNA), and thread 1 (main render thread)
    int[] cpuIds = new int[] { 5, 4, 3};
    for (int n = 0; n < m_NumActiveThreads; ++n)
    {
      if (m_Threads[n] == myThread)
      {
        int cpu = cpuIds[n % cpuIds.Length];
        myThread.SetProcessorAffinity(cpu);
        break;
      }
    }
  }
#endif
  // do actual thread work here
  ...