Visualizing Silverlight / .NET managed memory using Pivot

I got to see this amazing video on Pivot from Gary Flake.  Pivot is about visualizing data. It’s primarily for visualizing patterns and trends. What other way I would use other than analyzing managed memory using pivot. I spend a lot of time in windbg to look at memory for patterns, analyzing memory leaks.  And I also know not many of them want to learn the arcane commands of windbg. So I went ahead took the data from managed memory and ported it in to pivot format , which is cxml. I use the “!Traverseheap –xml” command from sos to generate the xml ,which I port it to cxml format.My initial thought was use dgml to visualize the data, but pivot is way better. Pivot can only be used in Vista+  OS.

FYI I know this is not for everyone, there are geeks who wouldn’t want to look at memory in a visual format, they prefer looking at in cdb.  I took Facebook Silverlight client app and got a snapshot of the memory for this demonstration

The few things that can be done with this are

  • Drill down objects by type with just one click : What I mean by this is , you could probably have 6000 odd types of objects and not instances (instances could be in thousands), but if you wanted to look at all the instance of customer objects  , it is as easy as one click from the filter.

Here is an example of filtering securitypermission objects in memory

  • Drill down members: The objects members can be looked up. So if you are looking at the customer object then you can view  strings, ints and all its members. It WILL NOT have the value for these members. It is not provided by the traverseheap command

  • Filter objects by namespace– All the objects belonging to namespaces can be drilled down.

  • Filter by object size

void Main()
 var startString =@"<?xml version=""1.0"" encoding=""utf-8""?>
<?xml-stylesheet type=""text/xsl"" href=""../pivot.xsl""?>
<Collection xmlns:xsi="""" xmlns:xsd=""""
 Name=""Memory Visualization"" SchemaVersion=""1"" d1p1:Icon=""icon.ico""
 xmlns:d1p1="""" xmlns="""">
 <FacetCategory Name=""Type"" Type=""String"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Address"" Type=""String"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Size""  Type=""Number"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Namespaces"" Type=""Link"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <FacetCategory Name=""Members"" Type=""Link"" d1p1:IsFilterVisible=""true"" d1p1:IsMetaDataVisible=""true"" d1p1:IsWordWheelVisible=""true"" />
 <Items ImgBase=""test.gif"">";

 var endString = @"</Items>

 var counter = 0;
 var outputFileName = "sl.cxml";
 var dict = new Dictionary<string,string>() { {"String","img5.jpg"},{"Object","img6.jpg"}};
 var getimg = new Func<string,string>((t) => dict.ContainsKey(t) ? dict[t] : "img8.jpg");

 var doc = XDocument.Load(@"c:\Temp\fb.xml");

 var types = (from i in doc.Descendants().Descendants().Descendants("type").AsParallel()
 let ns = i.Attribute("name").Value.SplitEx().ToList()
 select new {Id = int.Parse( i.FirstAttribute.Value),Name = ns.Last(),
 Namespaces = ns.Count()  > 2 ? ns.Take(ns.Count () -2) : ns.Take(ns.Count -1)}).ToList();
 var y = from i in doc.Descendants().Descendants().Descendants("object").AsParallel()
 let type = types.First (t => t.Id== int.Parse( i.Attribute("typeid").Value))
 select new {Address = i.FirstAttribute.Value,
 Type = type.Name,
 Namespaces = type.Namespaces,
 Size = int.Parse( i.LastAttribute.Value),
 children = from k in i.Descendants().AsParallel()
 from o in doc.Descendants().Descendants().Descendants("object").AsParallel()
 where k.FirstAttribute.Value == o.FirstAttribute.Value
 select new {Address = k.FirstAttribute.Value,Type = types.First (t => t.Id == int.Parse( o.Attribute("typeid").Value) ).Name}

 var x = from i in y.Take(2000).AsParallel()
 let increment = new Func<int>(() => Interlocked.Increment(ref counter))
 select new XElement("Item",new XAttribute("Img",getimg(i.Type)),new XAttribute("Id",increment()),new XAttribute("Name",i.Type),
 new XElement("Facets",
 new XElement("Facet",new XAttribute("Name","Type"),
 new XElement("String",new XAttribute("Value",i.Type))),
 new XElement("Facet",new XAttribute("Name","Size"),
 new XElement("Number",new XAttribute("Value",i.Size.ToString()))),
 new XElement("Facet",new XAttribute("Name","Address"),
 new XElement("String",new XAttribute("Value",i.Address))),
 new XElement("Facet",new XAttribute("Name","Namespaces"),
 i.Namespaces.Select (n =>new XElement("Link",new XAttribute("Name",n),new XAttribute("Href",string.Format("{0}#Link=EQ.{1}",outputFileName,n))))),
 i.children.Any() == false ? default(XElement):
 new XElement("Facet",new XAttribute("Name","Members"),
 i.children.Select (n =>new XElement("Link",new XAttribute("Name",n.Type),new XAttribute("Href",string.Format("{0}#Address=EQ.{1}",outputFileName,n.Address)))))
 File.WriteAllText(@"c:\temp\sl.cxml",string.Format("{0}{1}{2}",startString,x.Select (s => s).Aggregate(new StringBuilder(), (ag, n) => ag.Append(n)).ToString(),endString));
 Console.WriteLine ("Done");

static class Foo
 public static IEnumerable<string> SplitEx(this string s)
 if (s.Contains("`") == false)
 return s.Split(new []{'.'});
 var stringList = new List<string>();
 var index =0;
 var counter = 0;
 var splitString = "";
 while (counter < s.Length)
 index = s.IndexOf(".",counter);
 if (index < 0)
 splitString = s.Substring(counter,index -counter );
 if (splitString.Contains("`"))
 counter = index +1;
 return stringList;

The above code converts the xml format from !traverseheap to pivot xml format. This is CPU intensive and I have not profiled and fine tuned it. I use plinq for conversion primarily because I have quad-core box. I use pictures from powerpoint for visualizing  objects and I also wrote SplitEx method to avoid splitting namespaces that are part of the type name within generics “GenericEqualityComparer`1[[System.String, mscorlib]]”

And here is the command to get the xml input for the above code as input

!TraverseHeap -xml c:\temp\mem.xml

Using LINQPad , PLINQ to grep for files

I use linqpad as my primary dev tool for iterative code development. Today I had to search the source code tree on my hard disk for certain keywords that were present in database. There is big impedance mismatch between database and the rest of the world. I would have to fetch the data from DB and had to use either powershell or cmd to look for them. This is fine if I have to look for one keyword the issue if I have few of them then it becomes a bigger issue.  I remember looking at a cool example from PFX about using parallel grep. Here is the code I modified it to include optional parameters and return a type

public class GrepResult
 public string File {get;set;}
 public int Line {get;set;}
 public string Text {get;set;}
 public static class Extension
 public static IEnumerable<GrepResult> Grep(string regexString, IEnumerable<string> wildcards,bool ignoreCase = true, bool recursive = false)
 var regex = new ThreadLocal<Regex>(() =>
 new Regex(regexString, RegexOptions.Compiled | (ignoreCase ? RegexOptions.IgnoreCase : RegexOptions.None)));

 var files = from wc in wildcards
 let dirName = Path.GetDirectoryName(wc)
 let fileName = Path.GetFileName(wc)
                from file in Directory.EnumerateFiles(
                String.IsNullOrWhiteSpace(dirName) ? "." : dirName,
                String.IsNullOrWhiteSpace(fileName) ? "*.*" : fileName,
                recursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly)
                select file;

 var matches = from file in files.AsParallel(). AsOrdered().WithMergeOptions(ParallelMergeOptions.NotBuffered)
               from line in File.ReadLines(file).Zip(Enumerable.Range(1, int.MaxValue), (s, i) => new { Num = i, Text = s, File = file })
               where regex.Value.IsMatch(line.Text)
               select line;
 foreach (var line in matches)
      yield return new GrepResult() {File = line.File, Line = line.Num,Text = line.Text} ;

I would compile this into a dll and add reference to linqpad.  And here is the code that actually using it

static void Main()
 Environment.SetEnvironmentVariable("LinqQueries",@"c:\Users\naveen\Documents\LINQPad Queries\*.*");
 var linqPath = Environment.GetEnvironmentVariable("LinqQueries");
 var x = from i in ConfigEntries
 select i.Name;
 x.Select (y => Extension.Grep(y.Name,new [] {linqPath})).Dump();

I usually set environment variables (I am setting the env variable just for demonstration) for my projects so that I don’t have type the path manually. In the above code I am getting the values from database and searching those keywords within my hard disk. I could have use the Findstr method via Util class within LINQPad , but the PLINQ  implementation can use all my quad cores. I search for code very often and this is going to be handy.


Get every new post delivered to your Inbox.

%d bloggers like this: