The case of slow Visual Studio startup


In this post I would use Perfview /ETW to diagnose the delayed start-up of visual studio.

To analyze the problem start-up VS within Perfview as a run command

This would launch visual studio and collect etw traces. I have also enabled CodeMarkers , which is ETW traces for Visual Studio in case if you want to trace any extensions performance.

perfview-mainAfter it completes I choose the CPU Stacks and filtered with devenv.exe process.

On the CPU window I choose call-tree tab which displays the threads.perfviewcpuviewThe most amount of time is spent on the start-up thread and that is what we want to zoom into.

groupedcall-stacksWhen I expanded it does not show the information and everything is grouped into OTHER which does not help me.

The reason for that is perfview groups call-stacks for better viewing. I cleared the “groupparts” textbox and then expanded the start-up thread.

actualcallstacksFrom the call-stacks I could make almost 42% of time is spent on Xamarin and DevExpress extensions within VS. Now I could turn them off and have a better performance.

Perfview is great tool for identifying where the time is being spent!

Managed Stack Explorer using ClrMD


How often we run into an issue in the field where we just want to see the managed call-stack where the exception is or where the thread is hung. One of the options is debugger or something like ETW.

So I built a managed stack explorer

http://naveensrinivasan.github.io/ManagedStackExplorer/

Managed Stack Explorer provides call stack for .NET applications using managed code with thread local variables. The API as of now does not provide values for the local variables. When it is available we can update it.

It is a single executable without any other dll’s. It uses Costura to embed dll.

The debug shim works specific to processor and would not be able to get call-stacks if it is not. So x86 exe cannot get call-stacks of x64. That’s reason for x86 and x64 specific exe’s. There is no difference in code other than how it is compiled.

It is work in progress and I would love some feedback and code contributions.

The github site also has link to download the pre-complied executables.

Measure GC Allocations and Collections using TraceEvent


In this post I will explore  how we could use TraceEvent to measure our code (even at function level) for GC Allocations and Collections.

Save this with “.linq” extension and then open in  linqpad.

Here is the TL;DR

Why would I want to know GC events on a function level? Doesn’t the PerfMon counter  provide that information on an application level? Isn’t Premature optimization root of all evil?

Yes, for most of the part Premature optimization is not necessary. And PerfMon GC counter’s would give answers for the whole application. But it is usually after we build the application and when we start running into performance issues we start looking at them.

The motivation behind this are two things Measure Early and Often for Performance and Essential Truths Everyone Should Know about Performance in a Large Managed Codebase

If you haven’t read or watched the above video please do it. It let’s us know why and how to do it.

In the above video Dustin talks about Roslyn code base and how they used Perfview to measure Roslyn code base and identify potential bottlenecks early to avoid Perf issues.

One of the key differences between managed code and native code with respect to performance is GC. If GC is working hard then your application might not be able to get the performance that you are expecting. GC is good but if we don’t know which calls allocate what amount of data then it is an issue. Especially if you have a section code that is hit very often and which requires a lot of Perf, it is good know where the allocations are coming from. It is not explicit always.

In the above video Dustin shows few examples of Roslyn code where they were able to identify subtle issues that could allocate a lot when you are trying to get the most out of the code.There is also Roslyn Heap Allocation Analyzer which looks at the code help us identify allocations which isn’t necessary. It is a cool project.

I took one of the examples from the video as a motivation to check if  I could measure and make it a utility in my toolbox to help me when I need one.

In the above example I am trying look for a word “pede” in the lorem ipsum text. The code could get it using “foreach” or using the “Any” operator. I would like to run this few times to check what are the allocations and how long does it take. I used LINQPad as a scratch pad.

Here is the result of GC Allocations of using “Any” for 500 iterations and NOT the foreach

GCAllocations

The were 118 allocations of Enumerator and 146 allocations Func. GC usually allocates 100K each time it allocates that’s what is shown in the allocation amount column.

And here is GC Allocations when using “foreach”

GCAllocationWithForEach

There are hardly any new allocations compared to the previous one.

Here is the GC Collections when using  “Any”

GCCollection

There were 18 GC Collections using Any.

Here it is using foreach  and there were 0 collections.

ZeorCollections

Here is measure it time duration results using Any

MeasureWithAny

And here it is using Foreach

MeasureWithForEach

Look ma I figured out why my ETW EventSource isn’t tracing


The EventSource class in the framework 4.5 helps in writing custom ETW tracing.

When using EventSource class built within the framework, if the order of the methods don’t match ordinal number position in the class it would fail generating ETW traces. The EventSource has dependency on the order of the methods in the class.

This code would produce a valid ETW traces

This one would fail producing any ETW Traces.

The difference between them are the order of the methods. If you notice in the failing ETW tracing class the FailedTraceEvent is Second and the FailedDetailedEvent is first which is causing the trace not to be generated. The actual exception text would

Event FailedDetailedEvent is givien event ID 1 but 2 was passed to WriteEvent.

It’s one of those quirks that I ran into when building ETW tracing.

How to troubleshoot these kind of failures?

By default these kind of failed ETW exceptions would not be raised to be handled by the client code. The reason being,in production if you enable ETW tracing and all of a sudden the application crashes would not be something that we would want.

To troubleshoot this, use ETW tracing for exceptions. Use ETW to trace custom ETW failures. How cool is this?  My choice of tool is Perfview.

may-3-2015-etw-exception

Within Perfview in the Events Window I enter Test|Ex in the filter text box which is shown in the above picture. FYI filter text box supports Regular expression. So by entering Test|Ex, I am filtering events with Test or Ex which for exceptions. With that information I could filter all the TestEvent’s and any exceptions that have been raised which shows ArgumentException.

The call-stack of the ArgumentException shows on the static contructor .cctor of  FailedEvent.

may-3-2015-exception-callstack

Making an Image Easier to Debug


I am doing security review for a managed application which is obfuscated. So I am doing a lot of   disassembling code at runtime using Windbg. One of the issues is that code gets JIT optimized because of the retail build. This makes it harder for me debug when mapping it back. Realized  that I could turnoff  JIT Optimization’s using the ini file.

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

Another use of feature which I guess wasn’t really intended for.


					

Updating .NET String in memory with Windbg


In this post I would show a simple trick to update .NET strings in memory with Windbg. The caveat is make sure the string that you’re updating is long enough to fit into the string buffer. If not there would be a memory corruption.

Here is a simple windows form application with title “Good”

The goal is to update the title from “Good” to “Bad”.


button1.Click += (s,b) => Text = _caption;

I am updating the title in the button click.

Here is the actual string object within the debugger

0:006> !do 0294d0a0
Name:        System.String
MethodTable: 59b9fb64
EEClass:     598d8bb0
Size:        22(0x16) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\
v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      Good
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
59ba2b30  40000ed        4         System.Int32  1 instance        4 m_stringLength
59ba1f80  40000ee        8          System.Char  1 instance       47 m_firstChar
59b9fb64  40000ef        8        System.String  0   shared   static Empty
    >> Domain:Value  004b0308:02941228 <<

I would be using the e  command to update the memory. The ezu command is used for updating  Null-terminated Unicode string .

Notice the first character starts in the 8th offset from the above. So we would have start updating the string only from the 8th offset. The first 8 bytes of object are for syncblock index and method table pointer.

Here is the command to update the string memory.

ezu 0294d0a0+8 “Bad”

And the updated form title.

Who is is blocking my UI Thread? Diagnosing the cause using Windbg


It so happens most of the applications block the UI thread and do sync I/O, which is most common reason for “Not Responding” window. Here is a post http://blogs.msdn.com/b/nathannesbit/archive/2010/12/28/detecting-ui-thread-misuse.aspx that tries helping in detecting this. I like to handle this from bottom of the stack because we have a cool tool called debugger.

The approach is simple as having a break-point on a function like “KERNEL32!WaitFor*” and checking if the current thread is a UI thread. This could also be done for other functions like Sleep on UI thread by having a break-point on “KERNELBASE!SleepEx”.

Here are steps to determine if a thread is a STA thread / UI Thread. This is information is stored in TEB structure (thread environment block). Here is the output of the teb on the UI Thread

0:000> dt ntdll!_TEB @$teb
+0x000 NtTib            : _NT_TIB
+0x01c EnvironmentPointer : (null)
+0x020 ClientId         : _CLIENT_ID
+0x028 ActiveRpcHandle  : (null)
+0x02c ThreadLocalStoragePointer : 0x7efdd02c
+0x030 ProcessEnvironmentBlock : 0x7efde000 _PEB
+0x034 LastErrorValue   : 0
+0x038 CountOfOwnedCriticalSections : 0
+0x03c CsrClientThread  : (null)
+0x040 Win32ThreadInfo  : (null)
+0x044 User32Reserved   : [26] 0
+0x0ac UserReserved     : [5] 0
+0x0c0 WOW32Reserved    : 0x751b2320
+0x0c4 CurrentLocale    : 0x409
+0x0c8 FpSoftwareStatusRegister : 0
+0x0cc SystemReserved1  : [54] (null)
+0x1a4 ExceptionCode    : 0
+0x1a8 ActivationContextStackPointer : 0x001a07d0 _ACTIVATION_CONTEXT_STACK
+0x1ac SpareBytes       : [36]  “”
+0x1d0 TxFsContext      : 0xfffe
+0x1d4 GdiTebBatch      : _GDI_TEB_BATCH
+0x6b4 RealClientId     : _CLIENT_ID
+0x6bc GdiCachedProcessHandle : (null)
+0x6c0 GdiClientPID     : 0
+0x6c4 GdiClientTID     : 0
+0x6c8 GdiThreadLocalInfo : (null)
+0x6cc Win32ClientInfo  : [62] 0
+0x7c4 glDispatchTable  : [233] (null)
+0xb68 glReserved1      : [29] 0
+0xbdc glReserved2      : (null)
+0xbe0 glSectionInfo    : (null)
+0xbe4 glSection        : (null)
+0xbe8 glTable          : (null)
+0xbec glCurrentRC      : (null)
+0xbf0 glContext        : (null)
+0xbf4 LastStatusValue  : 0xc0000139
+0xbf8 StaticUnicodeString : _UNICODE_STRING “”
+0xc00 StaticUnicodeBuffer : [261]  “”
+0xe0c DeallocationStack : 0x00320000
+0xe10 TlsSlots         : [64] (null)
+0xf10 TlsLinks         : _LIST_ENTRY [ 0x0 – 0x0 ]
+0xf18 Vdm              : (null)
+0xf1c ReservedForNtRpc : 0x001d8c70
+0xf20 DbgSsReserved    : [2] (null)
+0xf28 HardErrorMode    : 0
+0xf2c Instrumentation  : [9] (null)
+0xf50 ActivityId       : _GUID {00000000-0000-0000-0000-000000000000}
+0xf60 SubProcessTag    : (null)
+0xf64 EtwLocalData     : (null)
+0xf68 EtwTraceData     : (null)
+0xf6c WinSockData      : (null)
+0xf70 GdiBatchCount    : 0x7efdb000
+0xf74 CurrentIdealProcessor : _PROCESSOR_NUMBER
+0xf74 IdealProcessorValue : 0x1010000
+0xf74 ReservedPad0     : 0 ”
+0xf75 ReservedPad1     : 0 ”
+0xf76 ReservedPad2     : 0x1 ”
+0xf77 IdealProcessor   : 0x1 ”
+0xf78 GuaranteedStackBytes : 0x1000
+0xf7c ReservedForPerf  : (null)
+0xf80 ReservedForOle   : 0x001ffd50

I have shown only the partial output because we are interested only in “ReservedForOle” member which is in the oxf80 offset. Within this structure in “0xc” offset contains the information on whether it is STA / MTA / Unkown and here is a write up on this from John Robbins http://www.microsoft.com/msj/1099/bugslayer/bugslayer1099.aspx.  Though the posts mentions STA as 0x80 and MTA as 0x140 with current version of windows value of STA is 81 and MTA is 141.
With this information it was pretty easy to create a script which will give us a call-stack if a UI Thread is blocking.

bm KERNEL32!WaitFor* ".if (poi(@$teb+0xf80) != 0) { .if (poi(poi(@$teb+0xf80)+0xc) = 81) {!clrstack;g} .else {g}} .else {gh}"
bp KERNELBASE!SleepEx ".if (poi(@$teb+0xf80) != 0) { .if (poi(poi(@$teb+0xf80)+0xc) = 81) {!clrstack;g} .else {g}} .else {gh}"

Here is a example call-stack from the above break-point which indicates that we are blocking on the UI Thread

OS Thread Id: 0x76c (0)
Child SP IP       Call Site
0029e74c 775c118e [InlinedCallFrame: 0029e74c]
0029e748 5affbc00 DomainBoundILStubClass.IL_STUB_PInvoke(System.Net.Sockets.AddressFamily, System.Net.Sockets.SocketType, System.Net.Sockets.ProtocolType, IntPtr, UInt32, System.Net.SocketConstructorFlags)
0029e74c 5afa72e4 [InlinedCallFrame: 0029e74c] System.Net.UnsafeNclNativeMethods+OSSOCK.WSASocket(System.Net.Sockets.AddressFamily, System.Net.Sockets.SocketType, System.Net.Sockets.ProtocolType, IntPtr, UInt32, System.Net.SocketConstructorFlags)
0029e7a4 5afa72e4 System.Net.Sockets.Socket.InitializeSockets()
0029e7f4 5afcc3ca System.Net.NetworkAddressChangePolled..ctor()
0029e808 5afcc326 System.Net.AutoWebProxyScriptEngine+AutoDetector.Initialize()
0029e838 5af7534d System.Net.AutoWebProxyScriptEngine+AutoDetector.get_CurrentAutoDetector()
0029e83c 5af75263 System.Net.AutoWebProxyScriptEngine..ctor(System.Net.WebProxy, Boolean)
0029e858 5af75202 System.Net.WebProxy.UnsafeUpdateFromRegistry()
0029e868 5af751c8 System.Net.WebProxy..ctor(Boolean)
0029e86c 5af74c79 System.Net.Configuration.DefaultProxySectionInternal..ctor(System.Net.Configuration.DefaultProxySection)
0029e8b0 5af748d2 System.Net.Configuration.DefaultProxySectionInternal.GetSection()
0029e8e4 5afcbf76 System.Net.WebRequest.get_InternalDefaultWebProxy()
0029e914 5afcbc86 System.Net.HttpWebRequest..ctor(System.Uri, System.Net.ServicePoint)
0029e92c 5afcbbcb System.Net.HttpRequestCreator.Create(System.Uri)
0029e938 5afcb772 System.Net.WebRequest.Create(System.Uri, Boolean)
0029e95c 5af94cad System.Net.WebRequest.Create(System.String)
0029e96c 0087056e WindowsFormsApplication1.Form1.<.ctor>b__0(System.Object, System.EventArgs) [C:\Users\naveen\Documents\Visual Studio 2010\Projects\WindowsFormsApplication1\WindowsFormsApplication1\Form1.cs @ 15]
0029e9ac 592b4ae8 System.Windows.Forms.Control.OnClick(System.EventArgs)
0029e9c4 592b70a2 System.Windows.Forms.Button.OnClick(System.EventArgs)
0029e9dc 59846174 System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
0029e9f8 598195b5 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
0029ea8c 59bda1bf System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0029ea90 59be18dd [InlinedCallFrame: 0029ea90]
0029eae4 59be18dd System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)
0029eb28 5931de00 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
0029eb34 593070f3 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0029eb3c 59307071 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0029eb50 59306fb6 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
0029ecf4 01010a35 [InlinedCallFrame: 0029ecf4]