Making an Image Easier to Debug


I am doing security review for a managed application which is obfuscated. So I am doing a lot of   disassembling code at runtime using Windbg. One of the issues is that code gets JIT optimized because of the retail build. This makes it harder for me debug when mapping it back. Realized  that I could turnoff  JIT Optimization’s using the ini file.

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

Another use of feature which I guess wasn’t really intended for.


Updating .NET String in memory with Windbg


In this post I would show a simple trick to update .NET strings in memory with Windbg. The caveat is make sure the string that you’re updating is long enough to fit into the string buffer. If not there would be a memory corruption.

Here is a simple windows form application with title “Good”

The goal is to update the title from “Good” to “Bad”.


button1.Click += (s,b) => Text = _caption;

I am updating the title in the button click.

Here is the actual string object within the debugger

0:006> !do 0294d0a0
Name:        System.String
MethodTable: 59b9fb64
EEClass:     598d8bb0
Size:        22(0x16) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\
v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      Good
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
59ba2b30  40000ed        4         System.Int32  1 instance        4 m_stringLength
59ba1f80  40000ee        8          System.Char  1 instance       47 m_firstChar
59b9fb64  40000ef        8        System.String  0   shared   static Empty
    >> Domain:Value  004b0308:02941228 <<

I would be using the e  command to update the memory. The ezu command is used for updating  Null-terminated Unicode string .

Notice the first character starts in the 8th offset from the above. So we would have start updating the string only from the 8th offset. The first 8 bytes of object are for syncblock index and method table pointer.

Here is the command to update the string memory.

ezu 0294d0a0+8 “Bad”

And the updated form title.

Conditional BreakPoint based on callstack within Windbg – .NET


Someone recently asked me “How to have a break-point on a method based on certain function in the call-stack?”

Here is the sample code to demonstrate this

using System;
using System.Threading.Tasks;
using System.Data.SqlClient;
namespace Test
{
    class Program
    {
        string connectionString = @"Data Source=.\sqlexpress;Initial Catalog=Tfs_Configuration;Integrated Security=True";
        public void Bar()
        {
            using (var c = new SqlConnection(connectionString))
            {
                c.Open();
                var command = new SqlCommand(@"update [tbl_AccessMapping] set [DisplayName] = @param", c);
                command.Parameters.Add(new SqlParameter("param", "Bar"));
                command.ExecuteNonQuery();
            }
        }
        public void Foo()
        {
            using (var c = new SqlConnection(connectionString))
            {
                c.Open();
                var command = new SqlCommand(@"update [tbl_AccessMapping] set [DisplayName] = @param", c);
                command.Parameters.Add(new SqlParameter("param", "Foo"));
                command.ExecuteNonQuery();
            }
        }
        static void Main(string[] args1)
        {
            var s = new Program();
            Parallel.For(0, 2, (i) => s.Bar());
            Parallel.For(0, 2, (i) => s.Foo());
            Console.Read();
        }
    }
}

The requirement is to have a break-point on “ExecuteNonQuery” but it should break only if it is invoked from “Foo” and not from “Bar”.

Launched the exe within windbg and loaded sos,sosex and set a bp on System.Data.SqlClient.SqlCommand.ExecuteNonQuery suing !mbm

And when the break-point hits the first time updated the bp using

bs 0  $$>a<“d:\Debuggersx86\ConditionalBP.txt” Foo

Here are the contents of ConditionalBP.txt

ad /q Contains
aS /c Contains .shell -ci "!CLRStack" FINDSTR $arg1
.block {
            .if ($spat("${Contains}","*${$arg1}*"))
                {
                 !CLRStack
                }
           .else
                { 
                g
                }
     }
ad /q Contains

Saving Dynamic Assembly in .NET 4.0 using Windbg


I recently had to debug a .NET 4.0 process which was loading the dependent assemblies using the AppDomain.AssemblyResolve event. The dependent assemblies were stored within the executable. I couldn’t disassemble the code to look for the dependent assembly because the exe was obfuscated. FYI the dynamic assembly cannot be saved using !SaveModule and here is the reason for this read the comments especially from Evian. Unlike psscor2.dll the sos for .NET 4.0 does not have a !dumpdynamicassembly with a save option.

Here is the sample code to demonstrate this.


using System;
using System.Reflection;
using TestLib;
namespace Test
{
 class Foo1
 {
 int[] s = new int[2];
 int v = 100;
 public Foo1()
 {
 Console.WriteLine(new Class1().Foo());
 }
 static void Main(string[] args1)
 {
 AppDomain.CurrentDomain.AssemblyResolve += (sender, args) =>
 {
 String resourceName = "ConsoleApplication13." +new AssemblyName(args.Name).Name + ".dll";
 using (var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName))
 {
 Byte[] assemblyData = new Byte[stream.Length];
 stream.Read(assemblyData, 0, assemblyData.Length);
 return Assembly.Load(assemblyData);
 }
 };
 System.IO.File.Delete(@"C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication13\bin\Debug\TestLib.dll");
 var s = new Foo1();
 Console.Read();
 }
 }
}

I knew the Assembly had to be loaded using  System.Reflection.Assembly.Load(Byte[]) ,so ended setting a break-point on the method using command !mbm *Assembly.Load* on the launch of the executable.

Here are the output of args and local variables for the above break-point

0:000> !mdv
Frame 0x0: (System.Reflection.Assembly.Load(Byte[])):
[A0]:rawAssembly:0x025fc374 (System.Byte[])
[L0]:<?>

Notice the “rawAssembly” argument which has the assembly contents.  Here are the raw memory contents of the address using dd 0x025fc374

0:000> dd 0x025fc374
025fc374  6a764994 00001000 00905a4d 00000003
025fc384  00000004 0000ffff 000000b8 00000000
025fc394  00000040 00000000 00000000 00000000
025fc3a4  00000000 00000000 00000000 00000000
025fc3b4  00000000 00000080 0eba1f0e cd09b400
025fc3c4  4c01b821 685421cd 70207369 72676f72
025fc3d4  63206d61 6f6e6e61 65622074 6e757220
025fc3e4  206e6920 20534f44 65646f6d 0a0d0d2e

  1. 6a764994 :- Is the Array’s  Method Table
  2. 00001000 : – Is the Array size
  3. The rest are the array contents.

Unlike the reference type arrays, the value type arrays  don’t have a DWORD for Method table of its contents. With this information I could dump the contents from memory in to disk using .writemem command.


.writemem c:\temp\assembly.bin @ecx+8 L?(poi(@ecx+@$ptrsize)*@$ptrsize)

In x86 @ecx register contains argument for rawAssembly. The  @ecx+8 is the start  position of the first byte and that is the reason for using this as the start position for .writemem. The poi(@ecx+@$ptrsize) contains the array size which in our case is 0001000 and multiply it by @$ptrsize which is 4 in x86. The expression (poi(@ecx+@$ptrsize)*@$ptrsize) would in our case result to 4000 bytes.

The assembly.bin would contain data in hex format which has to be converted in to binary format. Here is the code to convert from Hex to Binary format.

Assembly.Load( File.ReadAllBytes(@"c:\temp\assembly.bin")
 .Select(x =>
 Convert.ToByte(
 int.Parse((x.ToString("X")),NumberStyles.HexNumber)
 )).ToArray())
.FullName.Dump();

Dumping Generic List in .NET within Windbg


Most of the code uses List<T> for storing items.  The present solutions don’ t have a way to dump List<T> within windbg. Even though sosex has an option to dump the List<T> using !mdt it still doesn’t meet the scripting requirements. For example here is an output using sosex “!mdt -e 029a91c0″

0:000> !mdt -e 029a91c0
029a91c0 (System.Collections.Generic.List`1[[Test.Foo, Test]])
Count = 2
[0] 029a9200 (Test.Foo)
[1] 029a9210 (Test.Foo)

I would have preferred to get the contents of the “Foo” object instead of just the address of Foo. So wrote a script to do that.


$$ pointer to the array within the List
r @$t5 = poi(${$arg1}+@$ptrsize)

 .if (@$ptrsize = 8 )
 {
    r @t7 = 20 
 } 
 .else 
 { 
    r @$t7 = 10
 }

 .for (r $t0=0; @$t0 &lt; poi(@$t5+@$ptrsize); r$t0=@$t0+1 )
 {
     .if(@$t0 = 0)
     {
         $$ First occurence of the element in the array would be in the 20 offset for x64 and 10 offset for x86
         r$t1=@$t7
     }
     .else
     {
         $$ the rest of the elements would be in the 8th offset for x64 and 4th offset for x86
         r$t1= @$t7+(@$t0*@$ptrsize)
     }
     $$ Check for null before trying to dump
     .if (poi((@$t5-@$ptrsize)+@$t1) = 0 )
     {
     .continue
     }
     .else
     {
     .printf &quot;%N \n&quot; ,poi((@$t5-@$ptrsize)+@$t1)
     }
 }                                  

This script should work in x86 and x64. To use the above script copy to a file and invoke it like this passing the address of List<T>

$$>a<"d:\Debuggersx86\dumplist.txt" 029a91c0

Here is the output from the above command.

0:000> $$>a<"d:\Debuggersx86\dumplist.txt" 029a91c0
029A9200
029A9210

Now with this script I can use !mdt to get the contents of the “Foo” object.

.foreach ($obj {$$>a<"d:\Debuggersx86\dumplist.txt" 029a91c0}) {!mdt $obj}

0:000> .foreach ($obj {$$>a<"d:\Debuggersx86\dumplist.txt" 029a91c0}) {!mdt $obj}
029a9200 (Test.Foo)
counter:0x1 (System.Int32)
Name:029a917c (System.String: "test")
029a9210 (Test.Foo)
counter:0x2 (System.Int32)
Name:029a9198 (System.String: "test2")

This is one of the scripts that I would use often. Hope it is useful to others also.

Why isn’t the !bpmd in sos / windbg not working?


I recently noticed another blog post refer to one of my post. The issue was, sos wasn’t enabling the break-points on non-jitted functions. The classic example being “Main”.  Thanks to Steve I have been using sosex and not sos for setting break-points.

From my previous post you can understand how CLR is using clrn/CLRNotificationException to notify sos/sosex on JIT. With this information when I looked at the rotor code, I noticed an interesting member variable “g_dacNotificationFlags”. So I decided to check the value of this variable when using !bpmd from sos and !mbm from sosex.


.if (dwo(mscorwks!g_dacNotificationFlags) = 0) {.echo bp not set } .else {.echo bp set}

It was “0” when using sos and “1” when using sosex. Now I had to change the value to “1” and check if the break-point becomes active when using sos’s !bpmd.  FYI I don’t have private symbols and haven’t seen CLR Code. Here is the code to set the value to “1”.


ed mscorwks!g_dacNotificationFlags 00000001

And not to my surprise the !bpmd seems to work for non-jitted function with the above hack. FYI we don’t have to resort to this to get !bpmd to work. If the !bpmd is set after load of mscorjit/clrjit it would work as expected.

Decoding clr20r3 .NET exception – using mono cecil


I have often seen Devs trying to figure out the cause of the app crash without a memory dump. The only information that is available to analyze is the Windows Error Reporting message in the event viewer which would have “Event Name: CLR20r3″ along with Watson bucket information like this.

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: unhandledexception.exe
P2: 1.0.0.0
P3: 4ce1e0f1
P4: LibraryCode
P5: 1.0.0.0
P6: 4ce1e0f1
P7: 7
P8: 1f
P9: System.NullReferenceException
P10:

I will demonstrate the steps in identifying the code that caused the app to crash with the above information.Here is the explanation on the Watson Bucket items

  1. P1: unhandledexception.exe – is the Exe File Name
  2. P2:1.0.0.0 – is the Exe File assembly version number
  3. P3:4ce1e0f1- is the Exe File Stamp
  4. P4:LibraryCode- is the Faulting full assembly name
  5. P5:1.0.0.0- is the Faulting assembly version
  6. P6:4ce1e0f1- is the Faulting assembly timestamp
  7. P7:7- is the Faulting assembly method def
  8. P8:1f-  is Faulting method IL Offset within the faulting method
  9. P9:System.NullReferenceException- is Exception type that was thrown

 

Here is the LibraryCode that is mentioned in P4 of the watson bucket

using System;

namespace LibraryCode
{
    public class Foo
    {
        public Foo()
        {
            Console.WriteLine("Constructor");
        }
        public void Test()
        {
            Console.WriteLine("Test");
        }
        public string Bar(string test)
        {
            var x = test;
            return x.ToUpper();
        }
        public string Bar1(string test)
        {
            var x = test;
            return x.ToUpper();
        }
        public string Bar2(string test)
        {
            var x = test;
            return x.ToUpper();
        }
        public string Bar3(string test)
        {
            var x = test;
            return x.ToUpper();
        }
        public string Bar4(string test)
        {
            int j = 10;
            for (int i = 0; i < 10; i++)
            {
                j += i;
            }
            var x = test;
            return x.ToUpper();
        }
    }
}


And here is the code for the Main method calling the LibraryCode

  static void Main(string[] args)
        {
            var f = new Foo();
            var x = Console.ReadKey();
            f.Bar4(null);
        }

The most important items in the above watson bucket are 4,7 ,8 and 9. The item 4 is the assembly that was responsible for the crash which is “LibraryCode”. The item 7 is methoddef that threw the exception which is “7”. To identify the method we would have to dump the IL and here is the command to do that.

ildasm /tokens "C:\temp\LibraryCode.dll" /out=libcode.il

Open the libcode.il in a text editor and look for 06000007. The methoddef starts with 06 and 7 is the hex value and when converted to decimal it is still 7 and that’s how we ended with 06000007. The IL content for the corresponding method def

.method /*06000007*/ public hidebysig instance string
Bar4(string test) cil managed
{
// Code size       42 (0x2a)

With this we know the method that caused the app to crash.

The next step is to identify the faulting IL code within the method. The IL offset that caused the exception to be thrown is 1f (decimal value is 31), and here is the IL Code

IL_001d:  ldarg.1
IL_001e:  stloc.2
IL_001f:  ldloc.2
IL_0020:  callvirt   instance string [mscorlib/*23000001*/]System.String/*01000013*/::ToUpper() /* 0A000012 */
IL_0025:  stloc.3
IL_0026:  br.s       IL_0028

Now mapping the IL code back to C# shouldn’t be hard.

And If you are like me then you would probably want to automate things , so here is doing the same using Mono Cecil

AssemblyFactory.GetAssembly(@"C:\Temp\LibraryCode.dll")
		.MainModule.Types.Cast<TypeDefinition>()
		.ElementAt(1)
		.Methods.Cast<MethodDefinition>().First(md => md.MetadataToken.RID == 7)
		.Body.Instructions.Cast<Instruction>()
		.Select (i => 
			new {Offset = i.Offset, 
			OpCode = i.OpCode.ToString() , 
			Operand = i.Operand != null ? i.Operand.ToString() : string.Empty} )
		.Dump();

Notice the above code looks for methoddef “7” which is the P7 item in the Watson bucket.The code could have just dumped 31st IL offset which is “ldloc.2″ but that would not help , I like to see the entire method to figure out the cause of the exception.

And here is the output from above code.

We cannot get the call-stack for the crash with just watson buckets.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: