Visual Studio 2010 or WinDbg take longer than usual to load debug symbols.

Do you dislike a situation where applications on your machine have been working fine for weeks and months at a time and all of a sudden they begin to run very slowly? Well, I do too, especially when it comes to applications I use all day & every day.

At any given point in time I usually have Visual Studio and/or WinDbg running and today I noticed that my Visual Studio 2010 started to consume more CPU cycles than it does usually, and it was loading debug symbols for a release mode postmortem dump very, very slowly. First thing I did was to open the same dump in WinDbg, and guess what? WinDbg also was acting exactly the same. It was consuming approximately 30 to 40 percent of CPU cycles while in .reload /f /v and it was taking its sweet time.

On my machine, I use _NT_SYMBOL_PATH environment variable that controls my path to local symbol cache and remote symbol servers, and it is applicable to all debuggers that I use. I must point out that while I did come across several blogs that recommend against using this variable, it isn’t the root cause of the problem I had experienced today. After all, my debuggers were working fine last week (Friday) and not today (Monday).

So, what changed?

While I didn’t change anything, my machine is a member of a corporate domain so it is quite possible that “something” was changed on my machine for me.

Troubleshooting approach: simultaneous sniffer and process monitor capture on my machine while reloading symbols in VS 2010 or WinDbg.

Observations:

I noticed 3 to 7 second delays originating from my machine between each request to a remote symbol server (in this case: Microsoft public symbol server). Meaning: WinDbg or VS 2010 were blocked / busy doing something else before they even get to the point where they ask remote symbol server for a file, if it is missing in my cache.

sniffer

I also noticed excessive registry activity surrounding the registry keys shown below.

Continue reading

WinDbg exhibits a memory leak when you debug postmortem managed dumps

Howdy there!
My name is Olegas and I’ll be blogging here from time to time. Prashant has a very nice collection already and I’ll be adding my 2 cents every once in a while.

Recently I’ve come across an interesting behavior in WinDbg and I decided to look into it a bit further.

The scenario:
• You are debugging a managed dump using WinDbg 6.11.0001.404 on 32bit platform.
• You are trying to dump hundreds of managed objects to inspect their properties. You use a script similar, but not limited to
.foreach (MyObj {!dumpheap -MT 00687320 -short}) {!do MyObj}

The observed behavior:
• After dumping few dozen objects, WinDbg begins to report
<Note: this object has an invalid CLASS field>
Invalid object
• Your 32bt machine begins to respond very slowly and you notice excessive paging.
• Perfmon shows a behavior within WinDbg consistent with a memory leak
Perfmon_Graph

Continue reading

ASP.NET App Slow Response and Application Pool/AppDomain Recycle, Event message: Application is shutting down. Reason: Unknown – Windows Server 2003

Scenario
From time to time, asp.net application response is very slow on Windows Server 2003

Rants and the resolution

After turning on recycle events, logged message in application event log was Event message: Application is shutting down. Reason: Unknown. Slow response is always timed with this message in the application event log so that confirmed that Application Pool is terminating so no wonder asp.net response is slow from time to time.

However, the only missing piece was why? Since, the Reason is unknown :-) . This application pool is configured for web garden with 6 app pools in it so we decided to attach debugger in production box to 2 worker processes.

If you are just starting out with debugging or have not read John Robbins Book on debugging, I would like to stress the followings when using debugger in production environment

1. By Default, ADPlus  writes the call stack on first-chance exception. Walking call stack also results in Symbol loading, symbol loading along with the stack walking causes a performance hit when a debugger is attached. The last thing you want in production environment is to cause performance hit because of  debugger.

2. Don’t just use ADPlus script to attach a debugger to the worker process by name because it will attach the debugger to each worker process in your production server causing  further performance hit.

3. Don’t use DebugDiag in production environment unless you really have a good reason for it.

Continue reading

Caution when using System.IO.FileStream – ask what you need – System.UnauthorizedAccessException: Access to the path is denied

I have come across this issue quite a few times and this issue will be seen more often during deployment in QA/Production machine.

Its not unusual to see the below piece of code
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
XmlReader reader = XmlReader.Create(fs);
XmlSerializer serializer = new XmlSerializer(typeof(T));
Object o = serializer.Deserialize(reader);
}

Many times this will go unnoticed but if you look closely FileStream is using a constructor with FilePath and FileMode alone. System.IO.FileStream implementation of this constructor is

public FileStream(string path, FileMode mode) : this(path, mode, (mode == FileMode.Append) ? FileAccess.Write : FileAccess.ReadWrite, FileShare.Read, 0×1000, FileOptions.None, Path.GetFileName(path), false)
{
}

Default constructor asks for ReadWrite access, so now you see why your application is hosed in production, most of the developers use their system as admin user so of course they have the write access to the requested file. This is not to blame developers who run as admin because I can totally understand the pain.

By default, FileStream needs ReadWrite access that’s why System.UnauthorizedAccessException is thrown because on a production machine, User account under which asp.net worker process runs or a windows service or for that matter any process will not have the write access to a file by default.

Make sure, you ask for what you need. If your application needs only Read access to a file, make sure to specify that in your FileStream constructor. Don’t be GREEDY

using (FileStream fs = new FileStream(filePath, FileMode.Open,FileAccess.Read))

May be I would rather see a Read access by default in System.IO.FileStream implementation rather than ReadWrite.

LoadLibrary failed, Win32 error 0n193 “%1 is not a valid Win32 application.” Please check your debugger configuration and/or network access.

0:000> .loadby sos coreclr
The call to LoadLibrary(c:Program Files (x86)Microsoft Silverlight3.0.40624.0sos) failed, Win32 error 0n193
“%1 is not a valid Win32 application.”
Please check your debugger configuration and/or network access.

Make sure you are not using WinDbg 64 bit version. Silverlight is not 64 bit yet so even if you have a browser running on 64 bit os, sos dll for silverlight coreclr will fail to load on WinDbg 64 bit. Analyze your dump with WinDbg x86 version. I have WinDbg 32 bit and 64 bit both installed on my vista 64 bit os, although I still prefer XP or may be windows 7 from now on.

ProcDump sysiternals tool – really really helpful to create a memory dump based on CPU Usage

As described in Sysinternals documentation http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use) and unhandled exception monitoring. It also can serve as a general process dump utility that you can embed in other scripts.

You don’t need to write your own utility to create a memory dump by monitoring performance counter. Don’t forget to use the switch “-ma” to dump full memory(especially for .net app) because by default it only dumps thread and handle.

This is really helpful to get a memory dump based on CPU usage and we could probably get the memory dump without using ADPlus in most of the cases.

syntax to dump full memory given process id is

procdump <process id> -ma

syntax to dump full memory given process id and cpu usage 80%(threshold)

procdump <process id> -ma -c 80

Silverlight 3 OutOfBrowser(OOB) behind the scenes Explained, how to host a silverlight xap package OOB style without installing it – read/modify silverlight OOB metadata

You can find a sample on www.silverlight.net or google(bing) it. The purpose of this post is not to show you how to create a Out Of Browser(OOB) application using visual studio / Silverlight but what happens behind the scene and how you can re-use a silverlight xap package to deploy it on any machine by xcopy or a media drive and run it as a desktop application out of browser style.
First of All, creating a Silverlight Out of Browser Application is one line of code and the change in deployment manifest. Please refer to http://timheuer.com/blog/archive/2009/03/18/silverlight-3-offline-update-framework.aspx to understand silverlight offline/update behavior

In order to support Out of Browser in Silverlight, you just need to call Application.Current.Install() on user action as shown below.

private void Button_Click(object sender, RoutedEventArgs e)
{
Application.Current.Install();
}

When you install the application, it downloads the content(xap package) and generate the html file to host the silverlight app along with some metadata. Everything is generated at isolated storage, as shown below in Windows XP and server 2003

<systemdrive>:\Documents and Settings\<username>\Local Settings\Application Data\Microsoft\Silverlight\OutOfBrowser\<appid>

Continue reading

How to Revert back & forth between Silverlight 3 runtime and Silverlight 2 runtime – Ensuring That Your Silverlight 2 Applications Work with Silverlight 3

With the release of Silverlight 3, you may run into issue where silverlight 2 application may not work as expected. Please read this article on msdn Ensuring That Your Silverlight 2 Applications Work with Silverlight 3

You may need to revert to Silverlight 2 runtime to make sure that the issue has to do with the Silverlight 3 runtime. First of all in order for you to revert back and forth between Silverlight 2 and Silverlight 3 runtime, you should not uninstall Silverlight 2 runtime from your machine. Silverlight runtime gets loaded based on Silverlight plug in which is a COM dll(npctrl.dll).

COM Dll CLSID is “DFEAF541-F3E1-4c24-ACAC-99C30715084A” which remains same in silverlight 3 basically, you can’t run Silverlight 2 and Silverlight 3 runtime(CLR for Silverlight) side by side. Once you install Silverlight 3 runtime, browser running silverlight 2 app will load the Silverlight 3 runtime, you can verify that by looking at the loaded silverlight runtime dll in browser process space. By Default, Silverlight 3 will be installed in “C:\Program Files\Microsoft Silverlight\3.xxxx” so in order for you to revert back and forth all you need to do is register and unregister com dll.

You can simply create a batch file to do the job

for example

Batch file to revert to silverlight 2 runtime, click to download

regsvr32 /u /s “C:\Program Files\Microsoft Silverlight\3.0.40624.0\npctrl.dll”
regsvr32 /s “C:\Program Files\Microsoft Silverlight\2.0.40115.0\npctrl.dll”
Revert to silverlight 3 runtime, click to download

regsvr32 /u /s “C:\Program Files\Microsoft Silverlight\2.0.40115.0\npctrl.dll”
regsvr32 /s “C:\Program Files\Microsoft Silverlight\3.0.40624.0\npctrl.dll”