Josh Einstein

Okay I’ve been tracking down a mysterious explosion in “Allocated Bytes/sec” performance counter for the entire day. I’ve been watching an application that’s been running happily in a live environment for several days now for the first time and I’ve been keeping an eye on the performance counters every now and then.

Well today I noticed something very unusual. The Allocated Bytes/sec was fluctuating between 250,000 and 600,000 bytes per second and Gen 0 collections were happening at a staggering 1.5 collections per second! Something was giving the garbage collector a workout. While the overall stability of the application was unaffected (because the Garbage Collector was doing its job after all) I found it unsettling and I needed to find out what was causing the allocations.

It took a few hours to realize that I had left another machine open to a web page which is using AJAX to constantly refresh statistical data from a remoting server (the one I’m trying to debug) but once I found that out, I was still perplexed as to why this statistical refresh could be causing so many allocations and GC’s. All I was doing is this:

LoggingStatistics stats = new LoggingStatistics( );
stats.QueuedTotal = Convert.ToInt32( _CounterQueueTotal.NextValue( ) );
stats.QueuedPerSecond = _CounterQueueRate.NextValue( );
stats.WritesTotal = Convert.ToInt32( _CounterWritesTotal.NextValue( ) );
stats.WritesPerSecond = Convert.ToInt32( _CounterWritesRate.NextValue( ) );
stats.CurrentQueueSize = Convert.ToInt32( _CounterQueueSize.NextValue( ) );
return stats;

As you can see, I am requesting the NextValue of 5 performance counters. The performance counters are instances within a Singleton object configured to live forever so the actual PerformanceCounter components themselves are not being re-initialized on every call. But as it turns out, calling NextValue is extremely resource hungry. Now I’ve never worked with performance counters prior to .NET so I’m not sure what is all involved in the interop, but half a meg of allocations and 1.5 garbage collections per second (the above method is called once per second, and the collections are happening more frequently) seems way too high for me.

So now I have to find an alternate solution. Because if I have two or more clients connected to the AJAX web page, those statistics will be multiplied. Quite simply, I can’t rely on performance counters for this.

Fortunately, the application acting as the remoting server delivering the performance counters is also the one managing their values. For the purpose of seamless integration with Windows Server, I am still going to have to update the performance counters but at least then I can manage the frequency. What I pain.

By the way I ran into this web page after I discovered the problem which confirms my findings:

http://www.devx.com/dotnet/Article/16099/1763/page/2

I have just decided to release “PowerTrace” a .NET assembly that contains a very cool component called “TerminalTraceListener” that allows you to quickly and easily debug your apps running in either production or staging environments.

What is PowerTrace?

PowerTrace is the name of the “product” although currently it only consists of one component. It’s also free with no royalties, no license fees, no support, and no code. So it’s not really a product. It is however, fully documented and extendable so you can inherit from it if you want. I just made it about an hour ago so take it for what it is.

What is TerminalTraceListener?

TerminalTraceListener is the name of the first (and currently only) component in the PowerTrace toolkit. The way it works is simple. You add the trace listener to the listeners collection either via an application configuration file or manually in code. This registers a trace listener which is a component that outputs trace information whenever you call System.Diagnostics.Trace.Write/WriteLine/etc. By wiring up listeners like this, you can output your trace information to a variety of destinations without changing your code. In fact, TerminalTraceListener will allow your application to expose a diagnostics server port without ANY changes to your code at all!

TerminalTraceListener will listen on a TCP/IP socket (you can change the addresses and ports that it will listen on, defaults to 8096, my home town’s zip code) for incoming connections from any telnet client. The trace messages aren’t actually sent with any telnet control codes, but any telnet client will allow you to “port into” your running application to get all of your trace messages delivered to your client in realtime.

Does TerminalTraceListener show me trace messages that happened before I connect?

Nope. Not yet anyway. Feel free to derive from it and provide that capability if you want.

Does TerminalTraceListener provide any type of security?

Nope. There is no authentication and the output is unencrypted. Again, you can derive from it and implement this functionality if you’d like but implementing encryption will probably make it pretty useless if your telnet client can’t unencrypt it.

Is there any documentation?

Yes! TerminalTraceListener is fully documented (only a handful of methods and classes) and included is a CHM help file with configuration examples and the XML documentation so that VS.NET’s intellisense will pick up the member descriptions.

Implementation Details and Other Stuff

The server is implemented using asynchronous sockets. It can accept as many clients as you want to connect to it and they will all receive the same output. The server will currently consume any text that you send back to it but it currently does not support any commands.

There is no installer for this component. You can stick it in the GAC or if you’re like me and avoid the GAC at all times, just XCOPY deploy it with your app. There are no dependencies other than the Framework itself. The binary is currently compiled as a .NET 2.0 image but I’m going to make a 1.1 image available soon. I’m just too tired right now.

Enough already, go download the component!

I posted this in the newsgroups a long time ago and several people have emailed me about how it helped them which is really great because the problem is extremely hard to track down. Hopefully by “cross-posting” it here on my web site I will make it even easier to find.

Newsgroups: microsoft.public.dotnet.framework, microsoft.public.dotnet.framework.aspnet, microsoft.public.dotnet.framework.performance, microsoft.public.dotnet.framework.windowsforms, microsoft.public.dotnet.languages.csharp, microsoft.public.dotnet.languages.vb, mic 
From: "Josh" <josheinstein@hotmail.com>
Date: Mon, 4 Aug 2003 16:34:53 -0400 
Local: Tues,Aug 5 2003 6:34 am  
Subject: FIX: Slow compilations when using strong name 

To all...

I have experienced this problem and recently so has my co-worker which led 
me to try to solve the problem. Before I give you the fix, let me give a 
brief overview of the problem. Hopefully this will be indexed efficiently by 
google so that others with this problem can quickly benefit from it.

The problem manifests itself when on a Windows NT 4.0 domain and building 
assemblies using a strong name key generated with the SN.EXE utility. I 
cannot identify when the problem started, but it was definitely related to 
the client and not the server because the problem occurred at different 
times on two different machines. (We're talking months apart...) The problem 
is that compilation takes an unacceptably long time (~15 seconds PER 
PROJECT). If you disable the network interface, the problem goes away. If 
you do not strong name your assemblies, the problem goes away.

I saw that one person solved the problem by changing the machine-based key 
containers option with the SN.EXE utility. While this fixed the problem on 
my machine (temporarily) it did not help at all on my co-worker's machine. I 
turned on auditing (for every failure) and found an event log entry in the 
security log that resembled the following:

Type: Failure Audit 
Source: Security 
Category: Detailed Tracking 
Event ID: 596

Backup of data protection master key. 
  Key Identifier:  (guid omitted) 
  Recovery Server: 
  Recovery Key ID: 
  Failure Reason:  0x3A

According to MSKB article 309408, this is from the DPAPI and there is a 
workaround detailed in article 331333. I know, it looks like it has nothing 
to do with VS.NET and the compilers, but in fact it does.

http://support.microsoft.com/default.aspx?scid=kb;en-us;309408


http://support.microsoft.com/default.aspx?scid=kb;en-us;331333

By adding the following registry value, the problem has been fixed on both 
mine and my coworker's workstations and I could not be happier. Below is the 
registry entry, but I suggest you read the KB articles before importing it 
because it changes the way the DPAPI functions on Windows XP on a NT 4 
domain.

Windows Registry Editor Version 5.00 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\Protect\ Providers\df9d8c 
d0-1501-11d1-8c7a-00c04fc297eb ]
"MasterKeyLegacyNt4Domain"=dword:00000001

On a side note, the fix also corrected this very annoying little problem I 
was having with Windows Messenger when I would open a new conversation 
window it would lag for a few seconds before allowing me to type. Perhaps it 
is related.

I hope I have made someone's day as I have certainly made my own. I was 
beginning to consider not strong naming anything anymore. :) 15 seconds per 
project in a solution with 8 or so projects was becoming quite annoying!!

Please email me if this fix helps you. I'd be happy to hear from you.