Decoding clr20r3 .NET exception – using mono cecil
I have often seen Devs trying to figure out the cause of the app crash without a memory dump. The only information that is available to analyze is the Windows Error Reporting message in the event viewer which would have “Event Name: CLR20r3″ along with Watson bucket information like this.
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0Problem signature:
P1: unhandledexception.exe
P2: 1.0.0.0
P3: 4ce1e0f1
P4: LibraryCode
P5: 1.0.0.0
P6: 4ce1e0f1
P7: 7
P8: 1f
P9: System.NullReferenceException
P10:
I will demonstrate the steps in identifying the code that caused the app to crash with the above information.Here is the explanation on the Watson Bucket items
- P1: unhandledexception.exe – is the Exe File Name
- P2:1.0.0.0 – is the Exe File assembly version number
- P3:4ce1e0f1- is the Exe File Stamp
- P4:LibraryCode- is the Faulting full assembly name
- P5:1.0.0.0- is the Faulting assembly version
- P6:4ce1e0f1- is the Faulting assembly timestamp
- P7:7- is the Faulting assembly method def
- P8:1f- is Faulting method IL Offset within the faulting method
- P9:System.NullReferenceException- is Exception type that was thrown
Here is the LibraryCode that is mentioned in P4 of the watson bucket
using System;
namespace LibraryCode
{
public class Foo
{
public Foo()
{
Console.WriteLine("Constructor");
}
public void Test()
{
Console.WriteLine("Test");
}
public string Bar(string test)
{
var x = test;
return x.ToUpper();
}
public string Bar1(string test)
{
var x = test;
return x.ToUpper();
}
public string Bar2(string test)
{
var x = test;
return x.ToUpper();
}
public string Bar3(string test)
{
var x = test;
return x.ToUpper();
}
public string Bar4(string test)
{
int j = 10;
for (int i = 0; i < 10; i++)
{
j += i;
}
var x = test;
return x.ToUpper();
}
}
}
And here is the code for the Main method calling the LibraryCode
static void Main(string[] args)
{
var f = new Foo();
var x = Console.ReadKey();
f.Bar4(null);
}
The most important items in the above watson bucket are 4,7 ,8 and 9. The item 4 is the assembly that was responsible for the crash which is “LibraryCode”. The item 7 is methoddef that threw the exception which is “7″. To identify the method we would have to dump the IL and here is the command to do that.
ildasm /tokens "C:\temp\LibraryCode.dll" /out=libcode.il
Open the libcode.il in a text editor and look for 06000007. The methoddef starts with 06 and 7 is the hex value and when converted to decimal it is still 7 and that’s how we ended with 06000007. The IL content for the corresponding method def
.method /*06000007*/ public hidebysig instance string
Bar4(string test) cil managed
{
// Code size 42 (0x2a)
With this we know the method that caused the app to crash.
The next step is to identify the faulting IL code within the method. The IL offset that caused the exception to be thrown is 1f (decimal value is 31), and here is the IL Code
IL_001d: ldarg.1
IL_001e: stloc.2
IL_001f: ldloc.2
IL_0020: callvirt instance string [mscorlib/*23000001*/]System.String/*01000013*/::ToUpper() /* 0A000012 */
IL_0025: stloc.3
IL_0026: br.s IL_0028
Now mapping the IL code back to C# shouldn’t be hard.
And If you are like me then you would probably want to automate things , so here is doing the same using Mono Cecil
AssemblyFactory.GetAssembly(@"C:\Temp\LibraryCode.dll")
.MainModule.Types.Cast<TypeDefinition>()
.ElementAt(1)
.Methods.Cast<MethodDefinition>().First(md => md.MetadataToken.RID == 7)
.Body.Instructions.Cast<Instruction>()
.Select (i =>
new {Offset = i.Offset,
OpCode = i.OpCode.ToString() ,
Operand = i.Operand != null ? i.Operand.ToString() : string.Empty} )
.Dump();
Notice the above code looks for methoddef “7″ which is the P7 item in the Watson bucket.The code could have just dumped 31st IL offset which is “ldloc.2″ but that would not help , I like to see the entire method to figure out the cause of the exception.
And here is the output from above code.
We cannot get the call-stack for the crash with just watson buckets.
Script to !SaveAllModules in .NET 4.0 SOS within Windbg
The .NET 4.0 sos doesn’t have save all modules (!SaveAllModules) command. It only has !SaveModule. Recently I was debugging a .NET 4.0 process for which I had to save all the modules. Here is a script that does !SaveAllModules.
!for_each_module .if ($spat ("${@#ImageName}","*.exe")) { !SaveModule ${@#Base} c:\temp\${@#ModuleName}.exe } .else { !SaveModule ${@#Base} c:\temp\${@#ModuleName}.dll }
Using Managed Code to debug Memory Dumps
I happened to notice the new DebugDiag 1.2 and it had COM based API for dbgeng. The sample code were in VB Script. I much comfortable writing managed code compared to VB script. So I decided to use COM based API in managed code.
Here are couple of ways to solve certain problems using this
- Parallel GC Roots :- Getting GC Roots from memory dump is the most time consuming because SOS is single threaded. I use PFX to do them in parallel.
- Reconstructing manged objects :- Creating an instance of an object by reading data from the memory dump.
Need to add reference to the COM Library
And in VS2010 (.NET 4.0) by default COM Interop types have Embed Interop Types turned on. I couldn’t compile the code with this option. I had to turn off Embed Interop types.
Few extension methods for the DbgObj
static class DbgExtensions {
public static DbgObj OpenDump(this DbgControlClass dbg, string dumpPath) {
var path = Environment.GetEnvironmentVariable("_NT_SYMBOL_PATH");
return dbg.OpenDump(dumpPath, path, path, null);
}
public static void LoadSOS(this DbgObj dbg){
// By default it only loads psscor2.dll and It will not work for .NET 4.0
dbg.UnloadExtensions();
// Will load sos based on the framework version
var sos = dbg.GetModuleByModuleName("clr") == null ? ".loadby sos mscorwks" : ".loadby sos clr";
dbg.Execute(sos);
}
public static IEnumerable<string> DumpHeap(this DbgObj dbg, string typeorMT, bool isMT = false) {
var parameter = isMT ? "-MT " : "-type ";
return dbg.Execute("!dumpheap -short " + parameter + typeorMT).Split(new[] { "\n" },
StringSplitOptions.RemoveEmptyEntries);
}
public static string GCRoot(this DbgObj dbg, string address) {
return dbg.Execute("!GcRoot" + address);
}
public static double ReadDouble(this DbgObj dbg, string address, string offset) {
return (double)Int32.Parse(
dbg.Execute(string.Format("dd {0}+{1} L1", address, offset)).Replace("\n", "")
.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries)
.ElementAt(1),
NumberStyles.AllowHexSpecifier);
}
public static string ReadString(this DbgObj dbg, string address, string offset) {
// The managed string in x86 starts at 8th offset
return dbg.ReadUnicodeString(ReadDouble(dbg, address, offset) + 8);
}
}
Parallel GC Roots
Anybody who is debugged memory dumps for leaks understands the pain of running gcroots within a loop. AFAIK sos is single threaded. I have had customers who had 24 way CPU’s who wanted to use all the CPU’s to debug memory leaks, but it wasn’t possible.
Here is a code that would make parallel gc roots possible
using System;
using System.Collections.Generic;
using System.Linq;
using System.Globalization;
using DbgHostLib;
namespace ConsoleApplication1 {
class Program {
static void Main(string[] args) {
var dump = new DbgControlClass().OpenDump(@"C:\TestClass.dmp").LoadSOS();
var testInstances = dump.DumpHeap("Test.TestClass");
var roots = testInstances.AsParallel().Select(testclass =>
new DbgControlClass().OpenDump(@"C:\TestClass.dmp").LoadSOS().GCRoot(testclass)).ToList();
Console.Read();
}
}
static class DbgExtensions {
public static DbgObj OpenDump(this DbgControlClass dbg, string dumpPath) {
var path = Environment.GetEnvironmentVariable("_NT_SYMBOL_PATH");
return dbg.OpenDump(dumpPath, path, path, null);
}
public static DbgObj LoadSOS(this DbgObj dbg){
// By default it loads psscor2
dbg.UnloadExtensions();
// Will load sos based on the framework version
var sos = dbg.GetModuleByModuleName("clr") == null ? ".loadby sos mscorwks" : ".loadby sos clr";
dbg.Execute(sos);
return dbg;
}
public static IEnumerable<string> DumpHeap(this DbgObj dbg, string typeorMT, bool isMT = false) {
var parameter = isMT ? "-MT " : "-type ";
return dbg.Execute("!dumpheap -short " + parameter + typeorMT).Split(new[] { "\n" },
StringSplitOptions.RemoveEmptyEntries);
}
public static string GCRoot(this DbgObj dbg, string address) {
var s = dbg.IsClrExtensionMissing;
return dbg.Execute("!gcroot " + address);
}
public static double ReadDouble(this DbgObj dbg, string address, string offset) {
return (double)Int32.Parse(
dbg.Execute(string.Format("dd {0}+{1} L1", address, offset)).Replace("\n", "")
.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries)
.ElementAt(1),
NumberStyles.AllowHexSpecifier);
}
public static string ReadString(this DbgObj dbg, string address, string offset) {
// The managed string in x86 starts at 8th offset
return dbg.ReadUnicodeString(ReadDouble(dbg, address, offset) + 8);
}
}
}
The above code loads a memory dump and looks for object type “Test.TestClass” and gets its addresses. Then gets GCRoots in parallel using the AsParallel option.
Reconstructing manged objects
Using the same API it is pretty easy to create an actual instance of a class from a memory dump. Here is the code for which I dumped the memory.
using System;
namespace Test {
class Program {
static Foo[] foo= new Foo[5];
static void Main(string[] args) {
for (int i = 0; i < 5; i++)
foo[i] = new Foo() { counter = i, Name = "Name " + i.ToString() };
Console.WriteLine(foo);
Console.Read();
}
}
class Foo {
public int counter;
public string Name;
public override string ToString() {
return string.Format("Counter :- {0} , Name :- {1} ", counter, Name);
}
}
}
Here is the memory structure of Foo
0:005> !do 00f1c660
Name: Test.Foo
MethodTable: 009b38bc
EEClass: 009b14a4
Size: 16(0×10) bytes
File: C:\Foo\bin\Debug\Foo.exe
Fields:
MT Field Offset Type VT Attr Value Name
79ba2978 4000002 8 System.Int32 1 instance 2 counter
79b9f9ac 4000003 4 System.String 0 instance 00f1c680 Name
Notice the variable “Name” is in the 4th offset and counter is in the 8th offset. I use these offsets to read its contents from the dump.Here is the code that recreates instances of Foo from the memory dump.
class Program {
static void Main(string[] args) {
var dump = new DbgControlClass().OpenDump(@"C:\temp\Foo.dmp").LoadSOS();
var foos = dump.DumpHeap(@"Test.Foo");
foos.Select(s => new Foo() { counter = (int)dump.ReadDouble(s, "0x8"), Name = dump.ReadString(s, "0x4") }).
ToList().ForEach(Console.WriteLine);
Console.Read();
}
}
And here is the output from the above code.
Counter :- 0 , Name :- Name 0
Counter :- 1 , Name :- Name 1
Counter :- 2 , Name :- Name 2
Counter :- 3 , Name :- Name 3
Counter :- 4 , Name :- Name 4
There is lot more to explore than what I have shown above. Happy debugging :)
Downloading PDC10 videos using the new async feature
I knew PDC10 has an OData endpoint which is http://odata.microsoftpdc.com/ODataSchedule.svc/ . The best part about OData is querying for specific data that we are looking for. And here is my OData url for filtering twitter hashtag #languages
http://odata.microsoftpdc.com/ODataSchedule.svc/Sessions()?$filter=startswith(TwitterHashtag,'%23languages')&$expand=DownloadableContent&$select=DownloadableContent
With the above OData feed I could get urls for low bandwidth mp4′s that I can download. And here is the sample code for filtering
var x =XDocument.Load(@"c:\temp\session.xml").Descendants().AsParallel().Where(xd => xd.Name.LocalName=="Url"
&& xd.Value.Contains("_Low.mp4")).Select (xd => xd.Value);
Now that I have the url’s ,here is the code to download the videos using the new async feature
using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using System.Xml.Linq;
namespace Test
{
class Foo
{
static void Main(string[] args)
{
DownloadAsync();
Console.Read();
}
static async void DownloadAsync()
{
var result = new WebClient().DownloadStringTaskAsync("http://odata.microsoftpdc.com/ODataSchedule.svc/Sessions()?$filter=startswith(TwitterHashtag,'%23languages')&$expand=DownloadableContent&$select=DownloadableContent");
var downloads = XDocument.Parse(await result).Descendants().AsParallel().
Where(xd => xd.Name.LocalName == "Url" && xd.Value.Contains("_Low.mp4")).
Select(xd => new WebClient().DownloadFileTaskAsync(xd.Value, Path.GetFileName(xd.Value)));
await TaskEx.WhenAll(downloads).ContinueWith(_ => Console.WriteLine("Downloading Complete"));
}
}
}
Dumping .NET strings to files using Windbg
In this post I would demonstrate how to dump strings from a memory dump /live process to a file. Recently I had to debug a process which had few big strings where I had to analyze its contents. The !dumpobj from sos would only dump partial strings. I had to dump few hundred XML strings that I had to analyze using some automation. And hence comes the script.
$$ Dumps the managed strings to a file
$$ Platform x86
$$ Naveen Srinivasan http://naveensrinivasan.com
$$ Usage $$>a<"c:\temp\dumpstringtofolder.txt" 6544f9ac 5000 c:\temp\stringtest
$$ First argument is the string method table pointer
$$ Second argument is the Min size of the string that needs to be used filter
$$ the strings
$$ Third is the path of the file
.foreach ($string {!dumpheap -short -mt ${$arg1} -min ${$arg2}})
{
$$ MT Field Offset Type VT Attr Value Name
$$ 65452978 40000ed 4 System.Int32 1 instance 71117 m_stringLength
$$ 65451dc8 40000ee 8 System.Char 1 instance 3c m_firstChar
$$ 6544f9ac 40000ef 8 System.String 0 shared static Empty
$$ start of string is stored in the 8th offset, which can be inferred from above
$$ Size of the string which is stored in the 4th offset
r@$t0= poi(${$string}+4)*2
.writemem ${$arg3}${$string}.txt ${$string}+8 ${$string}+8+@$t0
}
And to use the above script ,copy it to a file and invoke it within Windbg/cdb
$$>a<”c:\temp\dumpstringtofolder.txt” 6544f9ac 5000 c:\temp\stringtest
Parameters to the script
- 6544f9ac :- Is the MT to string.
- 5000 :- Is the min size of the string that I want to dump
- c:\temp\stringtest :- Is the path along with partial filename for each string item
The dumped contents would be in Unicode format and to view its contents use something like this
Console.WriteLine(ASCIIEncoding.Unicode.GetString(File.ReadAllBytes(@"c:\temp\stringtest03575270.txt")));
And here is a sample code that downloads big xml strings ,that can be used by the above script to dump its contents to a folder
using System;
using System.Net;
namespace Test
{
class Program
{
static void Main(string[] args)
{
var speakers = new WebClient().DownloadString("http://www.codemash.org/rest/speakers");
var sessions = new WebClient().DownloadString("http://www.codemash.org/rest/sessions");
Console.Read();
}
}
}
Dumping ASP.NET Session (x86 /x64) within Windbg
This post is going to be about dumping ASP.NET session objects using Windbg. I had recently answered a stackoverflow question in which someone wanted to dump ASP.NET session objects for 64-bit IIS (x64). I thought why not blog about the same which might be useful to others. The challenge is to write one script that should work in both x86/x64. FYI there is a script from Tess that does dump out the session contents, AFAIK it will not work on x64 and my script iterates through the array using the array length instead of using “.foreach /pS 2 /ps 99” which is somewhat cleaner.
Here is the script for dumping ASP.NET session objects within Windbg / CDB
$$$ Dump the ASP.NET Session objects within windbg/cdb
$$$ Platform : x86 / x64
$$$ Naveen Srinivasan http://naveensrinivasan.com
$$$ Usage: $$>a<"c:\Debuggersx86\dumpsession.txt" 000007fef4115c20
$$$ where 000007fef4115c20 is the MethodTable pointer System.Web.SessionState.HttpSessionState
r @$t9 = @$ptrsize
$$ $t9 register contains pointer size
$$ $t8 register contains the next offset of the variable
$$ $t7 register contains array start address
.if (@$ptrsize = 8 )
{
$$$ x64
r @$t8 = 10
r @$t7 = 20
r @$t6 = 10
}
.else
{
$$$ x86
r @$t8 = 6
r @$t6 = 8
r @$t7 = 10
}
.foreach ($obj {!dumpheap -mt ${$arg1} -short})
{
$$ The !dumpheap -short option has last result as --------------- and
$$ this .if is to avoid this
.if ($spat ("${$obj}","------------------------------"))
{}
.else
{
$$ $t5 contains refernce to the array which has key and value for the
$$ session contents
r$t5 = poi(poi(poi(poi(${$obj}+@$t9)+@$t6)+@$t9)+@$t9)
$$$ Iterating through the array elements
.for (r $t0=0; @$t0 < poi(@$t5+@$t9); r$t0=@$t0+1 )
{
.if(@$t0 = 0)
{
$$ First occurence of the element in the array would be in the 20 offset for x64 and 10 offset for x86
r$t1=@$t7
}
.else
{
$$ the rest of the elements would be in the 8th offset for x64 and 4th offset for x86
r$t1= @$t7+(@$t0*@$t9)
}
$$ Check for null before trying to dump
.if (poi((@$t5-@$t9)+@$t1) = 0 )
{
.continue
}
.else
{
.echo ************
$$ Session Key
.printf /ow "Session Key is :- "; !ds poi(poi((@$t5-@$t9)+@$t1)+@$t9)
$$ Session value
.printf /ow "Session value is :- ";!ds poi(poi((@$t5-@$t9)+@$t1)+@$t6)
}
}
}
}
Copy the above script in to a file and invoke the script like this within Windbg
$$>a<”c:\Debuggersx86\dumpsession.txt” 000007fef4115c20
Passing the MT of System.Web.SessionState.HttpSessionState as the script argument.
Within the script I am using the alias !ds for dumping strings instead of using !dumpobj.
To create the alias use this command in x64
as !ds .printf "%mu \n", 10+
and in x86
as !ds .printf "%mu \n", C+
Replace !ds with !do for dumping regular objects instead of strings.
Here is the output from the above script
0:022> $$>a<”c:\Debuggersx86\dumpsession.txt” 000007fef4115c20
************
Session Key is :- Name
Session value is :- Test
************
Session Key is :- Name1
Session value is :- Test1
If you are only interested in getting the session contents the above script should get you the answer you are looking for. The rest of the post is an explanation of how the script works.
I am going to start by explaining one of the important statement in the script “r$t5 = poi(poi(poi(poi(${$obj}+@$t9)+@$t6)+@$t9)+@$t9)” which gets the contents of the array that contains the session key and value.
The $obj is the loop variable that contains the object address for each Http Session object “.foreach ($obj {!dumpheap -mt ${$arg1} -short})”
If I dump the Http session object using !do
0:022> !do 000000013fe20c30
Name: System.Web.SessionState.HttpSessionState
MethodTable: 000007fef4115c20
EEClass: 000007fef3d73e00
Size: 24(0×18) bytes
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef40b59e8 4001f59 8 …IHttpSessionState 0 instance 000000013fe20bc0 _container
We can see the 8th offset contains the pointer to the “_container” object in x64 and in x86 it will be the 4th offset and that’s the reason we use poi(${$obj}+@$t9) which should work for both x86 and x64 because the value of @$t9 is the pointer size which will be 4 in x86 and 8 in x64.
The next step is to dump the “_container” which is equal to poi(${$obj}+@$t9)
0:022> !do poi(000000013fe20c30+8)
Name: System.Web.SessionState.HttpSessionStateContainer
MethodTable: 000007fef411e868
EEClass: 000007fef3d77348
Size: 64(0×40) bytes
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef7b77a80 4001f5a 8 System.String 0 instance 0000000000000000 _id
000007fef4086508 4001f5b 10 …ateItemCollection 0 instance 000000013fe20458 _sessionItems
000007fef4115390 4001f5c 18 …ObjectsCollection 0 instance 000000013fe209d0 _staticObjects
000007fef7b7ecf0 4001f5d 28 System.Int32 1 instance 20 _timeout
000007fef7b76c50 4001f5e 34 System.Boolean 1 instance 1 _newSession
000007fef411f8c8 4001f5f 2c System.Int32 1 instance 1 _cookieMode
000007fef411f798 4001f60 30 System.Int32 1 instance 1 _mode
000007fef7b76c50 4001f61 35 System.Boolean 1 instance 0 _abandon
000007fef7b76c50 4001f62 36 System.Boolean 1 instance 0 _isReadonly
000007fef411e7b0 4001f63 20 …essionStateModule 0 instance 000000013fce1bd8 _stateModule
Now that we have the SessionContainer, we would have to get the contents of “_sessionItems” which is in the 10th offset in x64.
Next step is to dump “_sessionitems” using !do poi(poi(000000013fe20c30+8)+10) and this is equal to poi(poi(${$obj}+@$t9)+@$t6).In the starting of the script @$t6 is set to 10 or 8 based on platform.
0:022> !do poi(poi(000000013fe20c30+8)+10)
Name: System.Web.SessionState.SessionStateItemCollection
MethodTable: 000007fef4086650
EEClass: 000007fef3d2fcf0
Size: 112(0×70) bytes
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef7b76c50 400117b 44 System.Boolean 1 instance 0 _readOnly
000007fef7b7e968 400117c 8 …ections.ArrayList 0 instance 000000013fe20830 _entriesArray
000007fef7b7fd88 400117d 10 …IEqualityComparer 0 instance 000000013fc6b270 _keyComparer
000007fef7b7f3d8 400117e 18 …ections.Hashtable 0 instance 000000013fe20858 _entriesTable
000007fef6f6f938 400117f 20 …e+NameObjectEntry 0 instance 0000000000000000 _nullKeyEntry
000007fef6f479b8 4001180 28 …se+KeysCollection 0 instance 0000000000000000 _keys
000007fef7b66840 4001181 30 …SerializationInfo 0 instance 0000000000000000 _serializationInfo
000007fef7b7ecf0 4001182 40 System.Int32 1 instance 3 _version
000007fef7b77370 4001183 38 System.Object 0 instance 0000000000000000 _syncRoot
000007fef7bbd028 4001184 a70 …em.StringComparer 0 shared static defaultComparer
>> Domain:Value 00000000010e2690:NotInit 0000000002e0a0a0:00000000ffae8cb8 <<
000007fef7b76c50 4001f67 45 System.Boolean 1 instance 1 _dirty
000007fef4108b20 4001f68 48 …n+KeyedCollection 0 instance 0000000000000000 _serializedItems
000007fef7b7aa30 4001f69 50 System.IO.Stream 0 instance 0000000000000000 _stream
000007fef7b7ecf0 4001f6a 60 System.Int32 1 instance 0 _iLastOffset
000007fef7b77370 4001f6b 58 System.Object 0 instance 000000013fe20818 _serializedItemsLock
000007fef7b7f3d8 4001f66 18e0 …ections.Hashtable 0 shared static s_immutableTypes
>> Domain:Value 00000000010e2690:NotInit 0000000002e0a0a0:000000013fe204c8 <<
Next field that we are interested in is “_entriesArray” which is in the 8th offset in x64. To dump its contents here is the command !do
poi(poi(poi(000000013fe20c30+8)+10)+8) which is equal to poi(poi(poi(${$obj}+@$t9)+@$t6)+@$t9
0:022> !do poi(poi(poi(000000013fe20c30+8)+10)+8)
Name: System.Collections.ArrayList
MethodTable: 000007fef7b7e968
EEClass: 000007fef7781ee0
Size: 40(0×28) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef7b65870 400094c 8 System.Object[] 0 instance 000000013fe3ddb8 _items
000007fef7b7ecf0 400094d 18 System.Int32 1 instance 2 _size
000007fef7b7ecf0 400094e 1c System.Int32 1 instance 2 _version
000007fef7b77370 400094f 10 System.Object 0 instance 0000000000000000 _syncRoot
000007fef7b65870 4000950 388 System.Object[] 0 shared static emptyArray
>> Domain:Value 00000000010e2690:00000000ffac6110 0000000002e0a0a0:00000000ffad19e0 <<
The next field we are interested is “_items” which is in the 8th offset in x64. Notice “_items” is an array and cannot be dumped using !dumpobj or !do. So this command “r$t5 = poi(poi(poi(poi(${$obj}+@$t9)+@$t6)+@$t9)+@$t9)” will set the array pointer to$t5.
Now that we have array containing the session items, we could have used !da to
dump the array contents with details using !da -details poi(poi(poi(poi(000000013fe20c30+8)+10)+8)+8)
0:022> !da -details poi(poi(poi(poi(000000013fe20c30+8)+10)+8)+8)
Name: System.Object[]
MethodTable: 000007fef7b65870
EEClass: 000007fef777eb58
Size: 64(0×40) bytes
Array: Rank 1, Number of elements 4, Type CLASS
Element Methodtable: 000007fef7b77370
[0] 000000013fe3dd98
Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
MethodTable: 000007fef6f6f938
EEClass: 000007fef6ce90b0
Size: 32(0×20) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef7b77a80 4001185 8 System.String 0 instance 000000013fe3dcf8 Key
000007fef7b77370 4001186 10 System.Object 0 instance 000000013fe3dd20 Value
[1] 000000013fe3ddf8
Name: System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
MethodTable: 000007fef6f6f938
EEClass: 000007fef6ce90b0
Size: 32(0×20) bytes
(C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef7b77a80 4001185 8 System.String 0 instance 000000013fe3dd48 Key
000007fef7b77370 4001186 10 System.Object 0 instance 000000013fe3dd70 Value
[2] null
[3] null
But notice it does not help much because we still cannot see the actual key and value. That is the reason for using a nested “.for” loop in the script which will iterate through the array contents. FYI @$t5 contains reference to the array.
The statement “.for (r $t0=0; @$t0 < poi(@$t5+@$t9); r$t0=@$t0+1 )” is standard for loop with one thing that is special which is poi(@$t5+@$t9). The poi(@$t5+@$t9) contains the reference to the size of the array. How do I know that? The answer is dd poi(poi(poi(poi(000000013fe20c30+8)+10)+8)+8)
0:022> dd poi(poi(poi(poi(000000013fe20c30+8)+10)+8)+8)
00000001`3fe3ddb8 f7b65870 000007fe 00000004 00000000
00000001`3fe3ddc8 f7b77370 000007fe 3fe3dd98 00000001
00000001`3fe3ddd8 3fe3ddf8 00000001 00000000 00000000
00000001`3fe3dde8 00000000 00000000 00000000 00000000
00000001`3fe3ddf8 f6f6f938 000007fe 3fe3dd48 00000001
00000001`3fe3de08 3fe3dd70 00000001 00000000 00000000
00000001`3fe3de18 f7b77370 000007fe 00000000 00000000
00000001`3fe3de28 00000000 80000000 f7b77a80 000007fe
Notice the 8th offset value is 00000004 which is the size of the array and for
more information look at the post on custom dump array http://naveensrinivasan.com/2010/06/24/custom-dumparray-windbg/
The first element in the array would be in the 20th offset in x64 and 10th
offset in x86 and that is the reason for the “.if(@$t0=0)”
.if(@$t0 = 0)
{
$$ First occurence of the element in the array would be in the 20 offset for x64 and 10 offset for x86
r$t1=@$t7
}
So the first time the value @$t1 would be 20. And the rest of the elements would be in the 8th offset in x64 and 4th offset inx86
.else
{
$$ the rest of the elements would be in the 8th offset for x64 and 4th offset for x86
r$t1= @$t7+(@$t0*@$t9)
}
So the second time it would be 20+(1*8) = @$t7+(@$t0*@$t9) which will be 28th offset.
The next statement “.if (poi((@$t5-@$t9)+@$t1) = 0 )” is null check , this would avoid dumping an object which has not be initialized. This is because not all elements in the array could have been initialized.
The !ds poi(poi((@$t5-@$t9)+@$t1)+@$t9) gets the session key which is in the 8th offset (look at the previous output from dumparray) and the !ds poi(poi((@$t5-@$t9)+@$t1)+@$t6) gets the session value which is in the 10th offset.
Here is my initial x64 specific script that I wrote.
foreach ($obj {!dumpheap -mt ${$arg1} -short})
{
$$ The !dumpheap -short option has last result as --------------- and
$$ this .if is to avoid this
.if ($spat ("${$obj}","------------------------------"))
{}
.else
{
$$ $t5 contains reference to the array which has key and value for the
$$ session contents
r$t5 = poi(poi(poi(poi(${$obj}+0x8)+0x10)+0x8)+0x8);
r$t1 = 0
.for (r $t0=0; @$t0 < poi(@$t5+0x8); r$t0=@$t0+1 )
{
.if(@$t0 = 0)
{
$$ First occurrence of the element in the array would be in the 20 offset
r$t1=20
}
.else
{
$$ the rest of the elements would be in the 8th offset
r$t1= 20+(@$t0*8)
};
$$ Check for null before trying to dump
.if (poi((@$t5-0x8)+@$t1) = 0 )
{
.continue
}
.else
{
.echo ************;
? @$t0
$$ Session Key
.printf "Session Key is :- "; !ds poi(poi((@$t5-0x8)+@$t1)+0x8);
$$ Session value
.printf "Session value is :- ";!ds poi(poi((@$t5-0x8)+@$t1)+0x10)
}
}
}
}
I had fun writing this script. Let me know if there is a better way to write this.
GC Start and Stop events in .NET using Windbg
I was recently showing someone the new ETW features in .NET especially the GC Event notification and I was asked if we can get this using Windbg.
So here is the sample code for the GC Collection
namespace GCStartStop
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
button1.Click += (s, b) => GC.Collect(2);
button1.Click += (s, b) => GC.Collect(1);
}
}
}
The goal is set to a break-point only when the collection count is 2. Here is a bp script for doing this.
bp clr!WKS::GCHeap::SuspendEE ".if (dwo(clr!WKS::GCHeap::GcCondemnedGeneration)==2) {.echo start of gen 2;g} .else {gc}"
The same thing can be done for clr!WKS::GCHeap::RestartEE.
When showing this to someone I was asked what does “EE” acronym in “SuspendedEE” ? “EE” is Execution Engine.
Get GC Information in Silverlight
I had earlier written a post on getting GC information on Silverlight using ETW. With that we would have to write code to parse the ETW csv file. In this post I am going to be using Perfmonitor to do this. This tools uses the same ETW under covers, but it does all the plumbing and gives a nice report , which is much easier to read. Here are the reports
To demonstrate this I used the bing’s world leader search page and here is the url
http://www.bing.com/visualsearch?q=World+leaders&g=world_leaders&FORM=Z9GE74#
Steps to get the GC information are
- Start a cmd or powershell as admin , this required to collect ETW tracing
- Browse the above mentioned webpage using IE
- Issue the command “PerfMonitor.exe /process:4180 start” where 4180 is the internet explorer’s process id
- Do the necessary actions
- Then issue “PerfMonitor.exe stop”
- The command to get the report “PerfMonitor.exe GCTime”. This will generate a report and open it in the browser
Perfmonitor is like xperf for managed code. This is non-intrusive and can collect some valuable information in production. This is an xcopy tool and does not need an install.
Script to load sos within Windbg based on .NET Framework version
I often debug .NET Framework v 2.0 / v 4.0 code within windbg. In v 2.0 the main clr dll was called “mscorwks.dll” and in v 4.0 it is called “clr.dll”. As many of you are aware , to load sos in v 2.0 we would have to enter “.loadby sos mscorwks” and in v 4.0 it would be “.loadby sos clr” . This was a pain for me. Came up with a script to automate loading sos based on clr version
!for_each_module .if(($sicmp( "@#ModuleName" , "mscorwks") = 0) ) {.loadby sos mscorwks} .elsif ($sicmp( "@#ModuleName" , "clr") = 0) {.loadby sos clr}
You can take it up a notch by setting a break-point within clr based on the .NET Framework version
!for_each_module .if(($sicmp( "@#ModuleName" , "mscorwks") = 0) ) {bp mscorwks!WKS::GCHeap::SuspendEE ".if (dwo(mscorwks!WKS::GCHeap::GcCondemnedGeneration)==2) {.echo start of gen 2}"} .elsif ($sicmp( "@#ModuleName" , "clr") = 0) {bp clr!WKS::GCHeap::SuspendEE ".if (dwo(clr!WKS::GCHeap::GcCondemnedGeneration)==2) {.echo start of gen 2}"}
Debugging .NET – mystery between DEBUG versus RELEASE within windbg
I am sure most of us have debugged applications that are build with debug turned on, which is obviously much easier compared to debugging release build (optimized code). In this post I am going to share one of my experiences of debugging release build code. I will demonstrate this with a simple Console Application.
Here is the code
using System;
namespace ConsoleApplication
{
class Program
{
static void Main(string[] args)
{
var x = 10;
var name = "naveen";
Console.WriteLine(name);
Console.Read();
}
}
}
I compiled it under release mode and launched it within the debugger. The goal is to have a break-point on Console.WriteLine(name). First thing was to set up a sx notification on load for mscorlib.
sxe ld:mscorlib
And when the break-point hits for the above event, then issued the following command to load sosex , sos and set an bp on Console.WriteLine which is nothing fancy
.load sosex;.loadby sos clr;!mbm *System.Console.WriteLine* "!mk";g
I would imagine that the break-point would hit and I would get a call-stack, but to my surprise this was the output
0:000> .load sosex;.loadby sos clr;!mbm *System.Console.WriteLine* “!mk”;g
The breakpoint could not be resolved immediately.
Further attempts will be made as modules are loaded.
(1090.11fc): CLR notification exception – code e0444143 (first chance)
Breakpoint set at System.Console.WriteLine().
Breakpoint set at System.Console.WriteLine(Boolean).
Breakpoint set at System.Console.WriteLine(Char).
Breakpoint set at System.Console.WriteLine(Char[]).
Breakpoint set at System.Console.WriteLine(Char[], Int32, Int32).
Breakpoint set at System.Console.WriteLine(System.Decimal).
Breakpoint set at System.Console.WriteLine(Double).
Breakpoint set at System.Console.WriteLine(Single).
Breakpoint set at System.Console.WriteLine(Int32).
Breakpoint set at System.Console.WriteLine(UInt32).
Breakpoint set at System.Console.WriteLine(Int64).
Breakpoint set at System.Console.WriteLine(UInt64).
Breakpoint set at System.Console.WriteLine(System.Object).
Breakpoint set at System.Console.WriteLine(System.String).
Breakpoint set at System.Console.WriteLine(System.String, System.Object).
Breakpoint set at System.Console.WriteLine(System.String, System.Object, System.Object).
Breakpoint set at System.Console.WriteLine(System.String, System.Object, System.Object, System.Object).
Breakpoint set at System.Console.WriteLine(System.String, System.Object, System.Object, System.Object, System.Object, …).
Breakpoint set at System.Console.WriteLine(System.String, System.Object[]).
(1090.11fc): CLR notification exception – code e0444143 (first chance)
That’s it. And I never got an hit for the break-point. I checked to make sure there was an actual breakpoint set by issuing a “bl” command. I could see there were break-points for Console.WriteLine. The next step was to disassemble the code. So got the instruction pointer from the !mk call-stack. Here is the output of !mk. FYI this is when the code is blocked on Console.Read
00:U 003def90 75d273ea KERNEL32!ReadConsoleInternal+0×15
01:U 003def98 75d27041 KERNEL32!ReadConsoleA+0×40
02:U 003df020 75caf489 KERNEL32!ReadFileImplementation+0×75
03:M 003df068 65651c8b DomainNeutralILStubClass.IL_STUB_PInvoke(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte*, Int32, Int32 ByRef, IntPtr)(+0×0 IL)(+0×0 Native)
04:M 003df0e8 65cbf7e8 System.IO.__ConsoleStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, Int32, Int32 ByRef)(+0×53 IL)(+0x8c Native) [f:\dd\ndp\clr\src\BCL\System\IO\__ConsoleStream.cs, @ 16707566,0]
05:M 003df110 65cbf6d0 System.IO.__ConsoleStream.Read(Byte[], Int32, Int32)(+0x5d IL)(+0x9c Native) [f:\dd\ndp\clr\src\BCL\System\IO\__ConsoleStream.cs, @ 131,13]
06:M 003df138 65608bfb System.IO.StreamReader.ReadBuffer()(+0xa0 IL)(+0x3b Native) [f:\dd\ndp\clr\src\BCL\System\IO\StreamReader.cs, @ 488,21]
07:M 003df154 65bcacc3 System.IO.StreamReader.Read()(+0x1b IL)(+0×23 Native) [f:\dd\ndp\clr\src\BCL\System\IO\StreamReader.cs, @ 302,17]
08:M 003df160 65cc5e9d System.IO.TextReader+SyncTextReader.Read()(+0×0 IL)(+0×19 Native) [f:\dd\ndp\clr\src\BCL\System\IO\TextReader.cs, @ 244,17]
09:M 003df170 0066009a *** WARNING: Unable to verify checksum for ConsoleApplication.exe
ConsoleApplication.Program.Main(System.String[])(+0×0 IL)(+0x2a Native) [c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs, @ 10,13]
0a:U 003df17c 661621db clr!CallDescrWorker+0×33
Next disassemble Main Method using the ip which is 0066009a
!u 0066009a
Here is the output
0:000> !u 0066009a
Normal JIT generated code
ConsoleApplication.Program.Main(System.String[])
Begin 00660070, size 2d
c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs @ 10:
00660070 55 push ebp
00660071 8bec mov ebp,esp
00660073 56 push esi
00660074 8b3530206b03 mov esi,dword ptr ds:[36B2030h] (“naveen”)
c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs @ 11:
0066007a e85170f864 call mscorlib_ni+0x2570d0 (655e70d0) (System.Console.get_Out(), mdToken: 060008cd)
0066007f 8bc8 mov ecx,eax
00660081 8bd6 mov edx,esi
00660083 8b01 mov eax,dword ptr [ecx]
00660085 8b403c mov eax,dword ptr [eax+3Ch]
00660088 ff5010 call dword ptr [eax+10h]
0066008b e8f0a55565 call mscorlib_ni+0x82a680 (65bba680) (System.Console.get_In(), mdToken: 060008cc)
00660090 8bc8 mov ecx,eax
00660092 8b01 mov eax,dword ptr [ecx]
00660094 8b402c mov eax,dword ptr [eax+2Ch]
00660097 ff500c call dword ptr [eax+0Ch]
c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs @ 13:
>>> 0066009a 5e pop esi
0066009b 5d pop ebp
0066009c c3 ret
And I see System.Console.get_Out instead of System.Console.WriteLine which I was totally surprised. This was the reason the break-point never hit. Next I wanted check the IL which was compiled , what we see above is jitted x86 mixed with IL. Here is the command to check the compiled IL. First I had to get the methodesc from the ip using !ip2md
!ip2md 0066009a
0:000> !ip2md 0066009a
MethodDesc: 002237f0
Method Name: ConsoleApplication.Program.Main(System.String[])
Class: 002213f8
MethodTable: 00223804
mdToken: 06000001
Module: 00222e9c
IsJitted: yes
CodeAddr: 00660070
Transparency: Critical
Source file: c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs @ 13
Here is from the methoddesc to IL
!dumpil 002237f0
0:000> !dumpil 002237f0
ilAddr = 012a2050
IL_0000: ldstr “naveen”
IL_0005: stloc.0
IL_0006: ldloc.0
IL_0007: call System.Console::WriteLine
IL_000c: call System.Console::Read
IL_0011: pop
IL_0012: ret
Which looks very similar to my C# code. So looks like CLR optimized the code ,converted the Console.WriteLine to Console.get_Out. To validate it restarted the app with this as the command for break-point
.load sosex;.loadby sos clr;!mbm *System.Console.get_Out* "!mk";g
And here is the output
00:M 0032ed58 655e70d1 System.Console.get_Out()(+0×0 IL)(+0×1 Native) [f:\dd\ndp\clr\src\BCL\System\Console.cs, @ 193,17]
01:M 0032ed60 003a007f *** WARNING: Unable to verify checksum for ConsoleApplication.exe
ConsoleApplication.Program.Main(System.String[])(+0×6 IL)(+0xf Native) [c:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication11\Program.cs, @ 11,13]
02:U 0032ed6c 661621db clr!CallDescrWorker+0×33
Now that I have solved this I wanted to check the same on the debug build (optimized -) . To validate if it was Console.get_Out or Console.WriteLine. So when mscorlib loaded here was my command to check this
.load sosex;.loadby sos clr;!mbm *Program.Main* "!u @eip";g
In the above command I am setting a break-point on Main method and when the break-point hits “!u @eip” will disassemble the ip, the @eip register will have the address of the current function. Here is the output from !u @eip
*** WARNING: Unable to verify checksum for ConsoleApplication11.exe
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 11:
01f10070 55 push ebp
01f10071 8bec mov ebp,esp
01f10073 83ec0c sub esp,0Ch
01f10076 894dfc mov dword ptr [ebp-4],ecx
01f10079 833d3c31360000 cmp dword ptr ds:[36313Ch],0
01f10080 7405 je 01f10087
01f10082 e8c85a5064 call clr!JIT_DbgIsJustMyCode (66415b4f)
01f10087 33d2 xor edx,edx
01f10089 8955f4 mov dword ptr [ebp-0Ch],edx
01f1008c 33d2 xor edx,edx
01f1008e 8955f8 mov dword ptr [ebp-8],edx
>>> 01f10091 90 nop
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 12:
01f10092 c745f80a000000 mov dword ptr [ebp-8],0Ah
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 13:
01f10099 8b0530200f03 mov eax,dword ptr ds:[30F2030h] (“naveen”)
01f1009f 8945f4 mov dword ptr [ebp-0Ch],eax
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 14:
01f100a2 8b4df4 mov ecx,dword ptr [ebp-0Ch]
*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_32\mscorlib\246f1a5abb686b9dcdf22d3505b08cea\mscorlib.ni.dll
01f100a5 e802706d63 call mscorlib_ni+0x2570ac (655e70ac) (System.Console.WriteLine(System.String) , mdToken: 06000919)
01f100aa 90 nop
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 15:
01f100ab e8b4c1ca63 call mscorlib_ni+0x82c264 (65bbc264) (System.Console.Read(), mdToken: 0600090a)
01f100b0 90 nop
c:\users\naveen\documents\visual studio 2010\Projects\ConsoleApplication11\Program.cs @ 17:
01f100b1 90 nop
01f100b2 8be5 mov esp,ebp
01f100b4 5d pop ebp
01f100b5 c3 ret
eax=003637f0 ebx=00000000 ecx=020fbc7c edx=00000000 esi=004bb2d0 edi=0016f400
eip=01f10091 esp=0016f3c8 ebp=0016f3d4 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
01f10091 90 nop
Notice in the above code it is Console.WriteLine and not Console.get_Out.
Here is one of gotchas of debugging optimized code.
