Look ma I figured out why my ETW EventSource isn’t tracing

The EventSource class in the framework 4.5 helps in writing custom ETW tracing.

When using EventSource class built within the framework, if the order of the methods don’t match ordinal number position in the class it would fail generating ETW traces. The EventSource has dependency on the order of the methods in the class.

This code would produce a valid ETW traces

This one would fail producing any ETW Traces.

The difference between them are the order of the methods. If you notice in the failing ETW tracing class the FailedTraceEvent is Second and the FailedDetailedEvent is first which is causing the trace not to be generated. The actual exception text would

Event FailedDetailedEvent is givien event ID 1 but 2 was passed to WriteEvent.

It’s one of those quirks that I ran into when building ETW tracing.

How to troubleshoot these kind of failures?

By default these kind of failed ETW exceptions would not be raised to be handled by the client code. The reason being,in production if you enable ETW tracing and all of a sudden the application crashes would not be something that we would want.

To troubleshoot this, use ETW tracing for exceptions. Use ETW to trace custom ETW failures. How cool is this?  My choice of tool is Perfview.


Within Perfview in the Events Window I enter Test|Ex in the filter text box which is shown in the above picture. FYI filter text box supports Regular expression. So by entering Test|Ex, I am filtering events with Test or Ex which for exceptions. With that information I could filter all the TestEvent’s and any exceptions that have been raised which shows ArgumentException.

The call-stack of the ArgumentException shows on the static contructor .cctor of  FailedEvent.


.NET, Security

Piracy in .NET Code – Part 2 – Even when the code is obfuscated

Continuing with my previous post, one of the biggest security holes I have noticed in certain application is using unsecure Network I/O communication, especially when activating license. I have seen software where they have used the best tool to obfuscate the code, it is extremely hard to disassemble this. But lose out by invoking a web service with plaintext xml for registration and communication. Like I mentioned in my previous post, I am not going to be discussing on how to solve this problem.

I have seen applications which would have a trial period after which the application pops-up a dialog box for activating license. Only if license text matches the format criteria (like xxx-xx-xxxx-xx) the send button enables. To circumvent the disabled button, some of the smart developers could enable the button within the debugger (windbg /cdb) or using a something like HawkEye . This is the first line of defense for the application.

The next thing is launching something like Fiddler and checking out the web service request /response for the activation request. I am sure most of us have dealt with fiddler, and if you are not aware, fiddler gives an option to have a break-point on response from the server. So it is easy to do a Man-in-the-middle attack by injecting your own response or someone could even hack the lmhost file to act as a server.

And just because it is plain text, I have usually seen a bool variable in the response being activated or not. And it is not hard for someone to update the response text and pass it as a response back from the server.

The updated xml might not comply with the response expected by the application, it might throw an exception like this

0:020> !pe

Exception object: 045b4314

Exception type:   System.Xml.XmlException

Message:          Unexpected end of file while parsing Test  has occurred. Line 13, position 81.

InnerException:   <none>

StackTrace (generated):


StackTraceString: <none>

HResult: 80131940

Because it is xml, usually these applications end up using XmlTextReader to parse it and from the exception someone could figure out what element is expected and build the xml out of this. FYI the XmlTextReader library is usually never obfuscated.

The idea behind this post is to let the dev’s understand the security risks and someone with little advanced knowledge can exploit the holes. This is not just applicable for software piracy, but in general, especially with the RIA growing rapidly we would want to consider the security aspect also.

.NET, PLINQ, Windbg

Visualizing Silverlight / .NET managed memory using Pivot

I got to see this amazing video on Pivot from Gary Flake.  Pivot is about visualizing data. It’s primarily for visualizing patterns and trends. What other way I would use other than analyzing managed memory using pivot. I spend a lot of time in windbg to look at memory for patterns, analyzing memory leaks.  And I also know not many of them want to learn the arcane commands of windbg. So I went ahead took the data from managed memory and ported it in to pivot format , which is cxml. I use the “!Traverseheap –xml” command from sos to generate the xml ,which I port it to cxml format.My initial thought was use dgml to visualize the data, but pivot is way better. Pivot can only be used in Vista+  OS.

FYI I know this is not for everyone, there are geeks who wouldn’t want to look at memory in a visual format, they prefer looking at in cdb.  I took Facebook Silverlight client app and got a snapshot of the memory for this demonstration

The few things that can be done with this are

  • Drill down objects by type with just one click : What I mean by this is , you could probably have 6000 odd types of objects and not instances (instances could be in thousands), but if you wanted to look at all the instance of customer objects  , it is as easy as one click from the filter.

Here is an example of filtering securitypermission objects in memory

  • Drill down members: The objects members can be looked up. So if you are looking at the customer object then you can view  strings, ints and all its members. It WILL NOT have the value for these members. It is not provided by the traverseheap command

  • Filter objects by namespace– All the objects belonging to namespaces can be drilled down.

  • Filter by object size

void Main()
 var startString =@"<?xml version=""1.0"" encoding=""utf-8""?>
<?xml-stylesheet type=""text/xsl"" href=""../pivot.xsl""?>
<Collection xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xmlns:xsd=""http://www.w3.org/2001/XMLSchema""
 Name=""Memory Visualization"" SchemaVersion=""1"" d1p1:Icon=""icon.ico""
 xmlns:d1p1=""http://schemas.microsoft.com/livelabs/pivot/collection/2009"" xmlns=""http://schemas.microsoft.com/collection/metadata/2009"">
 <FacetCategory Name=""Type"" Type=""String"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Address"" Type=""String"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Size""  Type=""Number"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Namespaces"" Type=""Link"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Members"" Type=""Link"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <Items ImgBase=""test.gif"">";

 var endString = @"</Items>

 var counter = 0;
 var outputFileName = "sl.cxml";
 var dict = new Dictionary<string,string>() { {"String","img5.jpg"},{"Object","img6.jpg"}};
 var getimg = new Func<string,string>((t) => dict.ContainsKey(t) ? dict[t] : "img8.jpg");

 var doc = XDocument.Load(@"c:\Temp\fb.xml");

 var types = (from i in doc.Descendants().Descendants().Descendants("type").AsParallel()
 let ns = i.Attribute("name").Value.SplitEx().ToList()
 select new {Id = int.Parse( i.FirstAttribute.Value),Name = ns.Last(),
 Namespaces = ns.Count()  > 2 ? ns.Take(ns.Count () -2) : ns.Take(ns.Count -1)}).ToList();
 var y = from i in doc.Descendants().Descendants().Descendants("object").AsParallel()
 let type = types.First (t => t.Id== int.Parse( i.Attribute("typeid").Value))
 select new {Address = i.FirstAttribute.Value,
 Type = type.Name,
 Namespaces = type.Namespaces,
 Size = int.Parse( i.LastAttribute.Value),
 children = from k in i.Descendants().AsParallel()
 from o in doc.Descendants().Descendants().Descendants("object").AsParallel()
 where k.FirstAttribute.Value == o.FirstAttribute.Value
 select new {Address = k.FirstAttribute.Value,Type = types.First (t => t.Id == int.Parse( o.Attribute("typeid").Value) ).Name}

 var x = from i in y.Take(2000).AsParallel()
 let increment = new Func<int>(() => Interlocked.Increment(ref counter))
 select new XElement("Item",new XAttribute("Img",getimg(i.Type)),new XAttribute("Id",increment()),new XAttribute("Name",i.Type),
 new XElement("Facets",
 new XElement("Facet",new XAttribute("Name","Type"),
 new XElement("String",new XAttribute("Value",i.Type))),
 new XElement("Facet",new XAttribute("Name","Size"),
 new XElement("Number",new XAttribute("Value",i.Size.ToString()))),
 new XElement("Facet",new XAttribute("Name","Address"),
 new XElement("String",new XAttribute("Value",i.Address))),
 new XElement("Facet",new XAttribute("Name","Namespaces"),
 i.Namespaces.Select (n =>new XElement("Link",new XAttribute("Name",n),new XAttribute("Href",string.Format("{0}#Link=EQ.{1}",outputFileName,n))))),
 i.children.Any() == false ? default(XElement):
 new XElement("Facet",new XAttribute("Name","Members"),
 i.children.Select (n =>new XElement("Link",new XAttribute("Name",n.Type),new XAttribute("Href",string.Format("{0}#Address=EQ.{1}",outputFileName,n.Address)))))
 File.WriteAllText(@"c:\temp\sl.cxml",string.Format("{0}{1}{2}",startString,x.Select (s => s).Aggregate(new StringBuilder(), (ag, n) => ag.Append(n)).ToString(),endString));
 Console.WriteLine ("Done");

static class Foo
 public static IEnumerable<string> SplitEx(this string s)
 if (s.Contains("`") == false)
 return s.Split(new []{'.'});
 var stringList = new List<string>();
 var index =0;
 var counter = 0;
 var splitString = "";
 while (counter < s.Length)
 index = s.IndexOf(".",counter);
 if (index < 0)
 splitString = s.Substring(counter,index -counter );
 if (splitString.Contains("`"))
 counter = index +1;
 return stringList;

The above code converts the xml format from !traverseheap to pivot xml format. This is CPU intensive and I have not profiled and fine tuned it. I use plinq for conversion primarily because I have quad-core box. I use pictures from powerpoint for visualizing  objects and I also wrote SplitEx method to avoid splitting namespaces that are part of the type name within generics “GenericEqualityComparer`1[[System.String, mscorlib]]”

And here is the command to get the xml input for the above code as input

!TraverseHeap -xml c:\temp\mem.xml
.NET, Windbg

.NET – How can debug=true extend the life time of local variable

I am sure there are quite a few blog posts that advocate on not set debug=true in production code. The reason being the optimization and performance are turned off. In this blog post I am specifically going to demonstrate how the lifetime of a variable would be extended based on this setting by the JIT Compiler. The JIT compiler explicitly extends the lifetime of the local variables until the end of the method, which might not be required because it is not being used anymore.

using System;
namespace ConsoleApplication2
 internal class Program
 private static void Main(string[] args)
 var c = new Customer {Age = 20, Name = "Ted"};
 internal class Customer
 public string Name { get; set; }
 public int Age { get; set; }

In the above sourcecode I have created a new instance of customer and I am not referring it anywhere after that. I am explicitly invoking GC.Collect so that the objects that aren’t in scope would be collected.

I have compiled this code within VS.NET using the default setting, which essentially is the debug mode. And here is the output after disassembling it, which adds the Debuggable attribute to the assembly

assembly: Debuggable(DebuggableAttribute.DebuggingModes.DisableOptimizations | DebuggableAttribute.DebuggingModes.EnableEditAndContinue | DebuggableAttribute.DebuggingModes.IgnoreSymbolStoreSequencePoints | DebuggableAttribute.DebuggingModes.Default)]

Let’s see how this attribute affects the local variable “c” lifetime. Launched the application and then attached it to Windbg.   Loaded the sosex extension

Here is the output from !mk command

0:000> !mk

Thread 0:

*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\bc19222db4406c472d9aa1f8b6e0f470\mscorlib.ni.dll

ESP              EIP

00:U 00000000002ee758 00000000774100da ntdll!ZwRequestWaitReplyPort+0xa

01:U 00000000002ee760 00000000772c2b08 KERNEL32!ConsoleClientCallServer+0x54

02:U 00000000002ee790 00000000772f5601 KERNEL32!ReadConsoleInternal+0x1f1

03:U 00000000002ee8e0 000000007730a922 KERNEL32!ReadConsoleA+0xb2

04:U 00000000002ee9c0 00000000772d9934 KERNEL32!zzz_AsmCodeRange_End+0x8bea

05:U 00000000002eea00 000007fef34417c7 clr!DoNDirectCall__PatchGetThreadCall+0x7b

06:M 00000000002eeab0 000007fedfb034a1 DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte*, Int32, Int32 ByRef, IntPtr)(+0x0 IL)(+0x0 Native)

07:M 00000000002eebd0 000007fee02af59a System.IO.__ConsoleStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, Int32, Int32 ByRef)(+0x53 IL)(+0xba Native)

08:M 00000000002eec40 000007fee02af402 System.IO.__ConsoleStream.Read(Byte[], Int32, Int32)(+0x5d IL)(+0x62 Native)

09:M 00000000002eeca0 000007fedfabe63c System.IO.StreamReader.ReadBuffer()(+0xa0 IL)(+0x5c Native)

0a:M 00000000002eecf0 000007fee0245630 System.IO.StreamReader.Read()(+0x21 IL)(+0x30 Native)

0b:M 00000000002eed30 000007fee02b7458 System.IO.TextReader+SyncTextReader.Read()(+0x0 IL)(+0x38 Native)

0c:M 00000000002eed80 000007ff001701cd *** WARNING: Unable to verify checksum for C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication2\bin\Debug\ConsoleApplication2.exe

ConsoleApplication2.Program.Main(System.String[])(+0x24 IL)(+0xad Native) [C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication2\Program.cs, @ 10,13]

0d:U 00000000002eede0 000007fef34810b4 clr!CallDescrWorker+0x84

0e:U 00000000002eee30 000007fef34811c9 clr!CallDescrWorkerWithHandler+0xa9

0f:U 00000000002eeeb0 000007fef3481245 clr!MethodDesc::CallDescr+0x2a1

10:U 00000000002ef0e0 000007fef3581675 clr!ClassLoader::RunMain+0x228

11:U 00000000002ef330 000007fef35817ac clr!Assembly::ExecuteMainMethod+0xac

12:U 00000000002ef5e0 000007fef3581562 clr!SystemDomain::ExecuteMainMethod+0x452

13:U 00000000002efb90 000007fef3583dd6 clr!ExecuteEXE+0x43

14:U 00000000002efbf0 000007fef3583cf3 clr!CorExeMainInternal+0xc4

15:U 00000000002efc60 000007fef3607365 clr!CorExeMain+0x15

16:U 00000000002efca0 000007fef9393309 mscoreei!CorExeMain+0x41

17:U 00000000002efcd0 000007fef9425b21 MSCOREE!CorExeMain_Exported+0x57

18:U 00000000002efd00 00000000772bf56d KERNEL32!BaseThreadInitThunk+0xd

19:U 00000000002efd30 00000000773f3281 ntdll!RtlUserThreadStart+0x1d

From the call-stack we are interested in the 0c frame. So I switch to the frame using !mframe 0c

The next command is to look at the locals using !mdv and here is the output

0:000> !mdv

Frame 0xc: (ConsoleApplication2.Program.Main(System.String[])):

[A0]:args:0x00000000026a2258 (System.String[])

[L0]:c:0x00000000026a2298 (ConsoleApplication2.Customer)

[L1]:<>g__initLocal0:0x00000000026a2298 (ConsoleApplication2.Customer)

Notice the local variable reference “c” which is of type Customer and so its details are using !mdt c

0:000> !mdt c

00000000026a2298 (ConsoleApplication2.Customer)

<Name>k__BackingField:00000000026a2278 (System.String: "Ted")

<Age>k__BackingField:0x14 (System.Int32)

This is something which usually isn’t surprising to see right.  But if I change the compilation settings to release mode then here is the locals for the same code using !mdv

0:000> !mdv

Frame 0xc: (ConsoleApplication2.Program.Main(System.String[])):



Notice the change we don’t see the variable “c” of type Customer anymore after the GC.Collect in the release mode.  The reason behind this is, in debug mode ,the JIT Compiler explicitly extended the lifetime of the local variable to the end of the method and where as in release mode it does not do it.

IntelliTrace, VS2010

Use IntelliTrace without Visual Studio .NET

IntelliTrace is one of the best things in Visual Studio 2010. You can do “time travel” through your managed code. John Robbins has cool cool blog post on the working on IntelliTrace. I knew I could use IntelliTrace in field where the customer does not have VS.NET. To use in field ,I had to figure out the dependencies for IntelliTrace. So I fired up Windbg attached to the IntelliTrace process which would dump the all the modules loaded

Symbol search path is: SRV*D:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 01210000 01218000   D:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\TraceDebugger Tools\IntelliTrace.exe
ModLoad: 77d20000 77ea0000   C:\Windows\SysWOW64\ntdll.dll
ModLoad: 716b0000 716fa000   C:\Windows\SYSTEM32\MSCOREE.DLL
ModLoad: 76fe0000 770e0000   C:\Windows\syswow64\KERNEL32.dll
ModLoad: 75de0000 75e26000   C:\Windows\syswow64\KERNELBASE.dll
ModLoad: 75b90000 75c30000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 77360000 7740c000   C:\Windows\syswow64\msvcrt.dll
ModLoad: 76000000 76019000   C:\Windows\SysWOW64\sechost.dll
ModLoad: 75cf0000 75de0000   C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75890000 758f0000   C:\Windows\syswow64\SspiCli.dll
ModLoad: 75880000 7588c000   C:\Windows\syswow64\CRYPTBASE.dll
ModLoad: 71050000 710b6000   C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll
ModLoad: 77440000 77497000   C:\Windows\syswow64\SHLWAPI.dll
ModLoad: 75910000 759a0000   C:\Windows\syswow64\GDI32.dll
ModLoad: 76cf0000 76df0000   C:\Windows\syswow64\USER32.dll
ModLoad: 774a0000 774aa000   C:\Windows\syswow64\LPK.dll
ModLoad: 759a0000 75a3d000   C:\Windows\syswow64\USP10.dll
ModLoad: 770e0000 77140000   C:\Windows\system32\IMM32.DLL
ModLoad: 77710000 777dc000   C:\Windows\syswow64\MSCTF.dll
ModLoad: 6b590000 6bbff000   C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
ModLoad: 70c00000 70cbe000   C:\Windows\system32\MSVCR100_CLR0400.dll
ModLoad: 6a7c0000 6b583000   C:\Windows\assembly\NativeImages_v4.0.30319_32\mscorlib\246f1a5abb686b9dcdf22d3505b08cea\mscorlib.ni.dll
ModLoad: 71b50000 71b60000   C:\Windows\Microsoft.NET\Framework\v4.0.30319\nlssorting.dll
ModLoad: 76df0000 76f4c000   C:\Windows\syswow64\ole32.dll
ModLoad: 6d2b0000 6db48000   C:\Windows\assembly\NativeImages_v4.0.30319_32\System\964da027ebca3b263a05cadb8eaa20a3\System.ni.dll
ModLoad: 74180000 74188000   C:\Windows\assembly\NativeImages_v4.0.30319_32\IntelliTrace\947780232db1934c92cdfdaf2433bb59\IntelliTrace.ni.exe
ModLoad: 59660000 59a9d000   C:\Windows\assembly\NativeImages_v4.0.30319_32\Microsoft.VisualStu#\c828bb166e9d0df0e9b44a0a7616624a\Microsoft.VisualStudio.IntelliTrace.ni.dll
ModLoad: 618b0000 6196e000   C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Runtime.Remo#\dc1f0dbf1d3ba856eccec90b62b55d79\System.Runtime.Remoting.ni.dll
ModLoad: 66330000 66883000   C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Xml\e997d0200c25f7db6bd32313d50b729d\System.Xml.ni.dll
ModLoad: 75c60000 75cef000   C:\Windows\syswow64\oleaut32.DLL
ModLoad: 70a10000 70a70000   C:\Windows\Microsoft.NET\Framework\v4.0.30319\clrjit.dll
ModLoad: 58fe0000 59085000   C:\Program Files (x86)\Common Files\Microsoft Shared\VSTS 10.0\Trace Debugger\TraceLogProfiler.dll
ModLoad: 77cf0000 77cf5000   C:\Windows\syswow64\PSAPI.DLL
ModLoad: 76020000 76c69000   C:\Windows\syswow64\SHELL32.dll
ModLoad: 752a0000 752a9000   C:\Windows\system32\VERSION.dll
ModLoad: 753e0000 753f6000   C:\Windows\system32\CRYPTSP.dll
ModLoad: 75360000 7539b000   C:\Windows\system32\rsaenh.dll
ModLoad: 753d0000 753de000   C:\Windows\system32\RpcRtRemote.dll
ModLoad: 650e0000 650e5000   C:\Windows\system32\shfolder.dll
ModLoad: 75300000 7535f000   C:\Windows\system32\sxs.dll
(159c.6bc): Break instruction exception - code 80000003 (first chance)
eax=7ef39000 ebx=00000000 ecx=00000000 edx=77dbf50a esi=00000000 edi=00000000
eip=77d3000c esp=0683fca4 ebp=0683fcd0 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
77d3000c cc              int     3

So from the above output I figure out the required files for running IntelliTrace without VS.NET is

  1. IntelliTrace.exe
  2. IntelliTrace.exe.config
  3. Microsoft.VisualStudio.IntelliTrace.dll
  4. TraceLogProfiler.dll

The next thing was to check the command line options for IntelliTrace

  1. start
  2. status
  3. stop
  4. run
  5. launch
  6. help

I knew the command line option should have something interesting. So when the IntelliTrace was running I looked at command line options in Process Explorer

"D:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\TraceDebugger Tools\IntelliTrace.EXE" run /n:"test.exe_00001330_01cae6f016c66e1e"
/cp:"C:\Users\naveen\AppData\Local\Microsoft\VisualStudio\10.0\TraceDebugger\Settings\aoqsbu4g.fpj" /f:"Test_00001330_100428_123018.iTrace"

When i tried to open the “C:\Users\naveen\AppData\Local\Microsoft\VisualStudio\10.0\TraceDebugger\Settings\aoqsbu4g.fpj”, I got this error

So I went to the directory and  just made a copy of the file and then I was able to open the file. It was a xml file which contains most of the settings. It is collection plan settings file. The things that I updated in this file were <LogFileDirectory useDefault=”false”>C:\temp\</LogFileDirectory> and <DeleteLogOnExit>false</DeleteLogOnExit> for my test run. There are lot of settings in this file, which can be tweaked based on need,do explore this file.

With all this information, the next thing was to get a clean Installation of  Windows without VS.NET, good thing I had my laptop which had that. So I copied the required dependencies mentioned above along with a simple test.exe  which I used for figuring out Watson Buckets and here are the contents of the directory

And here is the command line option to start tracing Test.exe

C:\Users\naveen\intellitrace\intellitrace>IntelliTrace.exe launch /cp:q0mmz2ch.ixp Test.exe

The /cp:q0mmz2ch.ixp is the collection plan file. Here is the output from the above command

C:\Users\Naveen\intellitrace\intellitrace>IntelliTrace.exe launch /cp:q0mmz2ch.i
xp Test.exe
Microsoft (R) Visual Studio Logger. Version 10.0.30319.1
Copyright (C) Microsoft Corporation. All rights reserved.
C:\Users\Naveen\intellitrace\intellitrace\IntelliTrace.exe run /name:c__temp_9ce
6b8f1-19a5-44b7-909d-512a4c74e632.itrace /logfile:C:\temp\9ce6b8f1-19a5-44b7-909
d-512a4c74e632.iTrace /buffersize:65536 /buffercount:512 /watch:-1 /help- /nolog
o+ /collectionplan:q0mmz2ch.ixp /hidden-
Logger name is ‘c__temp_9ce6b8f1-19a5-44b7-909d-512a4c74e632.itrace’
Log file path ‘C:\temp\9ce6b8f1-19a5-44b7-909d-512a4c74e632.iTrace’
Using 512 buffers of 65536 bytes each.
Logger started.
Logger started.
Press Ctrl+C to stop logging or use ‘IntelliTrace stop /name:c__temp_9ce6b8f1-19
a5-44b7-909d-512a4c74e632.itrace’ from another command line.
Starting process ‘C:\Users\Naveen\intellitrace\intellitrace\Test.exe’
Waiting for process to exit

Unhandled Exception: System.NullReferenceException: Object reference not set to
an instance of an object.
at WatsonTest.Test.Main(String[] args)
Process exited with exit code -532462766.
Process execution time: 5097 ms
Logger name is ‘c__temp_9ce6b8f1-19a5-44b7-909d-512a4c74e632.itrace’
Stopping logger

That’s cool. I was able to trace it without VS.NET on the box and  I brought the trace file back to dev machine I was able load it up in VS.NET and could see stacks. And also from the above output I figured out  there are other undocumented command line options.

.NET, .NET 4.0, SOS, Windbg

Exploring UnhandledException in .NET and Watson buckets

I wanted to understand about UnhandledExceptions in .NET because of the few questions that I saw in the CLR Forum ,which had watson buckets in the event viewer.  To get deep understanding  unhandled exception filter there is a article on MSDN from CLR Team.

In this post I will be demonstrating how to get the Watson Bucket from within your code . FYI this is the same information  you can get !WatsonBuckets from sos within Windbg whenever there is a termination of the .NET process. !WatsonBuckets is undocumented.  I am using the CLR hosting interfaces to get the watson bucket. Here is the code that throws an unhandled exception and invokes clr to get watson bucket info.

using System;
using System.Runtime.InteropServices;

namespace WatsonTest
 internal class Test
 private static void Main(string[] args)
 var t = AppDomain.CurrentDomain;
 t.UnhandledException += (s, e) =>
 var c = GetWatsonBuckets();
 throw new NullReferenceException();
 private static WatsonBuckets GetWatsonBuckets()
 var pParams = new WatsonBuckets();
 IClrRuntimeHost host = null;
 host = Activator.CreateInstance(Type.GetTypeFromCLSID(ClrGuids.ClsIdClrRuntimeHost)) as IClrRuntimeHost;
 if (host != null)
 var clrControl = host.GetCLRControl();
 if (clrControl == null)
 return pParams;
 var clrErrorReportingManager =
 clrControl.GetCLRManager(ref ClrGuids.IClrErrorReportingManager) as IClrErrorReportingManager;
 if (clrErrorReportingManager == null)
 return pParams;
 clrErrorReportingManager.GetBucketParametersForCurrentException(out pParams);
 return pParams;
 // BucketParameters Structure to get watson buckets back from CLR
 [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
 internal struct WatsonBuckets
 internal int fInited;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string pszEventTypeName;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param0;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param1;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param2;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param3;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param4;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param5;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param6;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param7;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param8;
 [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0xff)] internal string param9;

 internal static class ClrGuids
 internal static readonly Guid ClsIdClrRuntimeHost = new Guid("90F1A06E-7712-4762-86B5-7A5EBA6BDB02");
 internal static Guid IClrErrorReportingManager = new Guid("980D2F1A-BF79-4c08-812A-BB9778928F78");
 internal static readonly Guid IClrRuntimeHost = new Guid("90F1A06C-7712-4762-86B5-7A5EBA6BDB02");

 [Guid("90F1A06C-7712-4762-86B5-7A5EBA6BDB02"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
 internal interface IClrRuntimeHost
 void Start();
 void Stop();
 void SetHostControl(IntPtr pHostControl);
 IClrControl GetCLRControl();
 void UnloadAppDomain(int dwAppDomainId, bool fWaitUntilDone);
 void ExecuteInAppDomain(int dwAppDomainId, IntPtr pCallback, IntPtr cookie);
 int GetCurrentAppDomainId();

 int ExecuteApplication(string pwzAppFullName, int dwManifestPaths, string[] ppwzManifestPaths,
 int dwActivationData, string[] ppwzActivationData);

 int ExecuteInDefaultAppDomain(string pwzAssemblyPath, string pwzTypeName, string pwzMethodName,
 string pwzArgument);

 [Guid("9065597E-D1A1-4fb2-B6BA-7E1FCE230F61"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
 internal interface IClrControl
 [return: MarshalAs(UnmanagedType.IUnknown)]
 object GetCLRManager([In] ref Guid riid);

 void SetAppDomainManagerType(string pwzAppDomainManagerAssembly, string pwzAppDomainManagerType);
 // IClrErrorReportingManager to get watson bukets back from CLR
 [Guid("980D2F1A-BF79-4c08-812A-BB9778928F78"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
 internal interface IClrErrorReportingManager
 int GetBucketParametersForCurrentException(out WatsonBuckets pParams);

The watson bucket information has ten items

  1. AppName
  2. Version
  3. ?
  4. Assembly and Module Name
  5. Assembly Version
  6. ?
  7. MethodDef
  8. IL Offset
  9. Exception Type
  10. ?

In the above code WatsonBuckets is a structure to get watson bucket information back from CLR.  In the code I use 3 hosting interfaces, IClrRuntimeHost which is the main interface, the next interface is IClrControl which has GetCLRManager method to get ErrorReportingManger. IClrErrorReportingManager which has the method to get the exception buckets which is GetBucketParametersForCurrentException. I know that there isn’t going to be much of use for this code. But it gives me better understanding of  CLR integrates with watson when ever there is termination of an application.

.NET, .NET 4.0, SOS, SOSEX, Windbg

Exploring SOSEX and Windbg to debug .NET 4.0

With the latest release of sosex comes a new set of functions to debug. It is pretty awesome that one person (Steve) alone could pull of such cool things. In this blog post, I am just going to demonstrate how easy it is to debug managed code using sosex compared to sos.

using System;
using System.Collections.Generic;

namespace MemCheck
 internal class Test
 Dictionary<int, string> dict = new Dictionary<int, string>();
 private static void Main(string[] args)
 var p = new Test();
 for (int i = 0; i < 100; i++)
 p.dict.Add(i, i.ToString());

I like to keep the code simple , so it is easy to follow.  The debugging goal for today is to get the Dictionary values. First I am going to demonstrate it using sos and then using sosex.

As usual I start the app and then attach it to windbg.

.loadby sos clr

FYI in  .net 4.0 clr is the dll that has CLR implementation. In prior versions it used to be in mscorwks. The next command would look for the object Test in the memory

!dumpheap -type MemCheck.Test
0:000> !dumpheap -type MemCheck.Test
 Address               MT     Size
0000000002761e20 000007ff00054110       24
 total 0 objects
 MT    Count    TotalSize Class Name
 000007ff00054110        1           24 MemCheck.Test
 Total 1 objects

The next step is to dump the object

0:000> !do 0000000002761e20
Name:        MemCheck.Test
MethodTable: 000007ff00054110
EEClass:     000007ff00162350
Size:        24(0x18) bytes
File:        C:\Users\naveen\Documents\Visual Studio 2010\Projects\Test\bin\Debug\Test.exe
 MT    Field   Offset                 Type VT     Attr            Value Name
000007feec2b7a48  4000001        8 ...tring, mscorlib]]  0 instance 0000000002761e38 dict

Notice the dict object is in the 8th offset . To dump contents  dict object I would use the command !do poi(0000000002761e20+8) , which is pointer deference of Test object on it is 8th offset.  And here is the output

0:000> !do poi(0000000002761e20+8)
Name:        System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]]
MethodTable: 000007feec2b7a48
EEClass:     000007feebe113c0
Size:        88(0x58) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
 MT    Field   Offset                 Type VT     Attr            Value Name
000007feec27c7d8  4000bee        8       System.Int32[]  0 instance 0000000002764788 buckets
000007feecbd3dc8  4000bef       10 ...non, mscorlib]][]  0 instance 0000000002764ab8 entries
000007feec27c848  4000bf0       40         System.Int32  1 instance              100 count
000007feec27c848  4000bf1       44         System.Int32  1 instance              100 version
000007feec27c848  4000bf2       48         System.Int32  1 instance               -1 freeList
000007feec27c848  4000bf3       4c         System.Int32  1 instance                0 freeCount
000007feec2a5a48  4000bf4       18 ...Int32, mscorlib]]  0 instance 0000000002761ef0 comparer
000007feecc75f78  4000bf5       20 ...Canon, mscorlib]]  0 instance 0000000000000000 keys
000007feecc72078  4000bf6       28 ...Canon, mscorlib]]  0 instance 0000000000000000 values
000007feec275ab8  4000bf7       30        System.Object  0 instance 0000000000000000 _syncRoot
000007feec29a1b8  4000bf8       38 ...SerializationInfo  0 instance 0000000000000000 m_siInfo

And the dictionary object in turn stores them within an array which is again the 8th offset.  This time because we know it is an array we are going to use the !dumparray command on the memory location. The command to get the details is

!dumparray -details poi(poi(0000000002761e20+8)+8)
MT    Field   Offset                 Type VT     Attr            Value Name
 000007feec27c848  400047b        0             System.Int32      1     instance                   -1     m_value
 [195] 0000000002764aa4
 Name:        System.Int32
 MethodTable: 000007feec27c848
 EEClass:     000007feebe00890
 Size:        24(0x18) bytes
 File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
 MT    Field   Offset                 Type VT     Attr            Value Name
 000007feec27c848  400047b        0             System.Int32      1     instance                   -1     m_value
 [196] 0000000002764aa8
 Name:        System.Int32
 MethodTable: 000007feec27c848
 EEClass:     000007feebe00890
 Size:        24(0x18) bytes
 File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
 MT    Field   Offset                 Type VT     Attr            Value Name
 000007feec27c848  400047b        0             System.Int32      1     instance                   -1     m_value

Here is the partial output.

Now lets try and do the same thing using sosex. The one thing that I really like about  the new sosex is that  I can use names rather than pointer deference which is way much easier. Launched the app and then loaded sosex using the command

.load F:\Work\Tools\debuggers\sosex.dll

and then switched the thread from 4th to 0th thread using ~0s.  By default the debugger injects a thread into the process for debugging and that was the 4th thread. The next command I issued was to get stack trace

0:000> !mk
Thread 0:
 ESP              EIP
00:U 000000000015e408 0000000077bc00da ntdll!ZwRequestWaitReplyPort+0xa
01:U 000000000015e410 0000000077a72b08 KERNEL32!ConsoleClientCallServer+0x54
02:U 000000000015e440 0000000077aa5601 KERNEL32!ReadConsoleInternal+0x1f1
03:U 000000000015e590 0000000077aba922 KERNEL32!ReadConsoleA+0xb2
04:U 000000000015e670 0000000077a89934 KERNEL32!zzz_AsmCodeRange_End+0x8bea
05:U 000000000015e6b0 000007feed0317c7 clr!DoNDirectCall__PatchGetThreadCall+0x7b
06:M 000000000015e760 000007feec1d34a1 DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte*, Int32, Int32 ByRef, IntPtr)(+0x0 IL)(+0x0 Native)
07:M 000000000015e880 000007feec97f59a System.IO.__ConsoleStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, Int32, Int32 ByRef)(+0x53 IL)(+0xba Native)
08:M 000000000015e8f0 000007feec97f402 System.IO.__ConsoleStream.Read(Byte[], Int32, Int32)(+0x5d IL)(+0x62 Native)
09:M 000000000015e950 000007feec18e63c System.IO.StreamReader.ReadBuffer()(+0xa0 IL)(+0x5c Native)
0a:M 000000000015e9a0 000007feec915630 System.IO.StreamReader.Read()(+0x21 IL)(+0x30 Native)
0b:M 000000000015e9e0 000007feec987458 System.IO.TextReader+SyncTextReader.Read()(+0x0 IL)(+0x38 Native)
0c:M 000000000015ea30 000007ff00170213 MemCheck.Test.Main(System.String[])(+0x39 IL)(+0xf3 Native) [C:\Users\naveen\Documents\Visual Studio 2010\Projects\Test\Program.cs, @ 17,13]
0d:U 000000000015eaa0 000007feed0710b4 clr!CallDescrWorker+0x84
0e:U 000000000015eaf0 000007feed0711c9 clr!CallDescrWorkerWithHandler+0xa9
0f:U 000000000015eb70 000007feed071245 clr!MethodDesc::CallDescr+0x2a1
10:U 000000000015eda0 000007feed171675 clr!ClassLoader::RunMain+0x228
11:U 000000000015eff0 000007feed1717ac clr!Assembly::ExecuteMainMethod+0xac
12:U 000000000015f2a0 000007feed171562 clr!SystemDomain::ExecuteMainMethod+0x452
13:U 000000000015f850 000007feed173dd6 clr!ExecuteEXE+0x43
14:U 000000000015f8b0 000007feed173cf3 clr!CorExeMainInternal+0xc4
15:U 000000000015f920 000007feed1f7365 clr!CorExeMain+0x15
16:U 000000000015f960 000007fef8f13309 mscoreei!CorExeMain+0x41
17:U 000000000015f990 000007fef8fa5b21 MSCOREE!CorExeMain_Exported+0x57
18:U 000000000015f9c0 0000000077a6f56d KERNEL32!BaseThreadInitThunk+0xd
19:U 000000000015f9f0 0000000077ba3281 ntdll!RtlUserThreadStart+0x1d

FYI the command !mk has been part of sos from the initial version. I am interested in only looking at the code that I wrote so I would like to move stack frame to 0c which is MemCheck.Test.Main . To do that the command  is !mframe 0c, which moves to that stackframe. The reason to move the particular stack frame is to look for variables in the stack and the command to variables is !mdv , which display managed local variables

0:000> !mdv
Frame 0xc: (MemCheck.Test.Main(System.String[])):
[A0]:args:0x0000000002761dd8 (System.String[])
[L0]:p:0x0000000002761e20 (MemCheck.Test)
[L1]:i:0x0000000000000064 (System.Int32)
[L2]:CS$4$0000:0x0000000000000000 (System.Boolean)

Notice we see the local variable “p” which is of type MemCheck.Test. To display type p we issue the command !mdt p

0:000> !mdt p

0000000002761e20 (MemCheck.Test)

dict:0000000002761e38 (System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]])

I didn’t  have to get memory address , I am using the names which is very intuitive ,especially when we have to debug large application with N levels of nesting.So to get the dict values   from p the command to issue is !mdt -e p.dict

!mdt -e p.dict
[98] (System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.String, mscorlib]]) VALTYPE (MT=000007feec2b7b28, ADDR=0000000002765400)
 key:0x62 (System.Int32)
 value:0000000002765e48 (System.String: "98")
[99] (System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.String, mscorlib]]) VALTYPE (MT=000007feec2b7b28, ADDR=0000000002765418)
 key:0x63 (System.Int32)
 value:0000000002765e68 (System.String: "99")

Here is the partial output. Notice I never had to use a memory pointer or do a pointer deference .This is very similar to VS.NET debugging where I am used to the variable names compared the memory address.  Thanks to Steve for providing such a cool extension.