May 24 2013

LINQ Deferred Execution & Lambda Methods for providing Simple Stats (Part II)

Important!

This is part 2 in a series of posts on Linq & Lambda capabilities in C# 

Deferred Execution

So lets take a minute to talk about deferred execution. You may here this referred to as Lazy Execution as well. But in a nutshell what this means is that when you write a linq or lambda query against a collection or list, the execution of that query doesn’t actually happen until the point where you need to access the resuts. Let’s look at a simple example.

var ienum = Enumerable.Range(1, 10).ToList();

var query = from i in ienum
            where i%2 == 0
            select i;

ienum.Add(20);
ienum.Add(30);

SuperConsole.WriteLine(query);
//prints 2, 4, 6, 8, 10, 20, 30

So why does it print out 20 and 30. This is deferred execution in practice. At the point where you write your query (var query) the query is not actually executed against your datasource (ienum). After the query is setup, more data is added to your data source, and the query is only actually executed at the point where the results need to be evaluated (SuperConsole.WriteLine)

This holds true in a number of other Linq Scenarios. In Linq-to-Sql or Linq-to-Entity Framework, execution of the Sql Query is only sent to the database at the point where you need to evaluate your queries. It’s important to understand this so that queries don’t go out of scope before being executed, so that un-executed queries aren’t inadvertently passed to other parts or layers in your application and so that you don’t end up introducing N+1 problems where you think your working on data in memory but in actual fact, your performing multiple executions over and over in a loop. If you do need to make your queries “Greedy” and force them to execute there and then, you can wrap them in parenthesis and immediately call .ToList() on them to force the execution.

Min, Max, Count & Average

Linq has a number of convenient built in methods for getting various numeric stats about the data your working on. Consider a collection of Movies which you want to Query.

public class Movie
{
    public string Title { get; set; }
    public double Rating { get; set; }
}

...

var movies = new List
    {
        new Movie() {Title = "Die Hard", Rating = 4.0},
        new Movie() {Title = "Commando", Rating = 5.0},
        new Movie() {Title = "Matrix Revolutions", Rating = 2.1}
    };

Console.WriteLine(movies.Min(m => m.Rating));
//prints 2.1

Console.WriteLine(movies.Max(m => m.Rating));
//prints 5

Console.WriteLine(movies.Average(m => m.Rating));
//prints 3.7

Console.WriteLine(movies.Count);
Console.WriteLine(movies.Count());
//prints 3

Min, Max and Average are all fairly straight forward, finding the Minimum, Maximum and Average movie rating values respectively. It’s worth mentioning with regards the Count implementations that there are different “versions” of the count implementation depending on the underlying data structure you are operating on. The Count property is a property of the List class are returns the current number of items in that collection. The Count() method is an extension method on the IEnumerable interface which can be executed on any IEnumerable structure regardless of implementation.

In general LINQ’s Count will be slower and is an O(N) operation while List.Count and Array.Length are both guaranteed to be O(1). However in some cases LINQ will special case the IEnumerable parameter by casting to certain interface types such as IList or ICollection. It will then use that Count method to do an actual Count() operation. So it will go back down to O(1). But you still pay the minor overhead of the cast and interface call. Ref: [http://stackoverflow.com/questions/981254/is-the-linq-count-faster-or-slower-than-list-count-or-array-length/981283#981283]

This is important as well if you are testing your collections to see if they are empty. People coming from versions of .NET previous to Generics would use the Count or Length properties of a collection to see if they were empty. i.e.

if(list.Count == 0)
{ 
    //empty
}
if(array.Length == 0)
{
    //empty
}

Linq however provides another method to test for contents called Any(). It can be used to evaluate whether the collection is empty, or if the collection has any items which validate a specific filter.

if(list.Any()) //equivalent of count == 0
{ 
    //empty
}
if(list.Any(m => m.Rating == 5.0)) //if it contains any top rated movies.
{
    //empty
}

If you are starting with something that has a .Length or .Count (such as ICollection, IList, List, etc) – then this will be the fastest option, since it doesn’t need to go through the GetEnumerator()/MoveNext()/Dispose() sequence required by Any() to check for a non-empty IEnumerable sequence. For just IEnumerable, then Any() will generally be quicker, as it only has to look at one iteration. However, note that the LINQ-to-Objects implementation of Count() does check for ICollection (using .Count as an optimisation) – so if your underlying data-source is directly a list/collection, there won’t be a huge difference. Don’t ask me why it doesn’t use the non-generic ICollection… Of course, if you have used LINQ to filter it etc (Where etc), you will have an iterator-block based sequence, and so this ICollection optimisation is useless. In general with IEnumerable : stick with Any() Ref: [http://stackoverflow.com/questions/305092/which-method-performs-better-any-vs-count-0/305156#305156]

Next post, we’ll look at some different mechanisms for filtering and transforming our queries.

~Eoin C

Permanent link to this article: http://trycatch.me/linq-deferred-execution-lambda-methods-for-providing-simple-stats-part-ii/

May 23 2013

Handy LINQ & Lambda Methods and Extensions (Part I)

The System.Linq namespace contains a fantastic set of utility extension methods for filtering, ordering & manipulating the contents of your collections and objects. In the following posts I’ll go through some of the most useful ones (in my humble opinion) and how you might use them in your C# solutions

Important!

This is part 1 in a series of posts on Linq & Lambda capabilities in C# 

Before we start, here’s a handy static method to print your resulting collections to the console so you can quickly verify the results.

public class SuperConsole
{
    public static void WriteLine<T>(IEnumerable<T> list, bool includeCarriageReturnBetweenItems =false)
    {
        var seperator = includeCarriageReturnBetweenItems ? ",\n" : ", ";
        var result = string.Join(seperator, list);
        Console.WriteLine(result);
    }
}

Enumerable

The System.Linq.Enumerable type has 2 very useful static methods on it for quickly generating a sequence of items. Enumerable.Range & Enumerable.Repeat. The Range method allows you to quickly generate a sequential list of integers from a given starting point for a given number of items.

IEnumerable<int> range = Enumerable.Range(1, 10);
SuperConsole.WriteLine(range);
//prints "1, 2, 3, 4, 5, 6, 7, 8, 9, 10"

So why is this useful, well you could use it to quickly generate a pre-initialised list of integers rather than new’ing up a list and then iterating over it to populate it. Or you could use it to replicate for(;;) behavior. e.g.

for (int i = 1; i <= 10; i++) 
{     
    //DoWork(i); 
} 

Enumerable.Range(1, 10).ToList().ForEach(i =>
    {
        //DoWork(i)
    });

Repeat is similar but is not limited to integers. You can generate a Sequence of a given length with the same default value in every item. Imagine you wanted to create a list of 10 strings all initialised with a default string of “ABC”;

var myList = Enumerable.Repeat("ABC", 10).ToList();

Item Conversion

There are also a few handy ways to convert/cast items built into the System.Linq namespace. The Cast<T> extension method allows you to cast a list of variables from one type to another as long as a valid cast is available. This can be useful for quickly changing a collection of super types into their base types.

var integers = Enumerable.Range(1, 5);
var objects = integers.Cast<object>().ToList();

Console.WriteLine(objects.GetType());
SuperConsole.WriteLine(objects);

//prints
//System.Collections.Generic.List`1[System.Object]
//1, 2, 3, 4, 5

But what if a valid implicit cast isn’t available. What if we wanted to convert our collection of integers into a collection of strings with a ‘:’ suffix. Thankfully Linq has us covered with it’s ConvertAll Method on List

var integers = Enumerable.Range(1, 5);
var converter = new Converter<int, string>(input => string.Format("{0}: ", input));
var results = integers.ToList().ConvertAll(converter);

SuperConsole.WriteLine(results, true);
/*prints
    1:
    2:
    3:
    4:
    5:
    */

In the next post, we’ll look at some the lazy & deferred execution capabilities of LINQ and some useful methods for performing quick calculations and manipulations on our collections.

~Eoin C

Permanent link to this article: http://trycatch.me/handy-linq-lambda-methods-and-extensions-part-i/

May 21 2013

Implementing HTML Formatted Emails in the Enterprise Library Logging Block

Enterprise LibraryThe Microsoft Patterns & Practices Enterprise Library contains a number of useful applications blocks for simplifying things like DataAcces, Logging & Exception Handling in your .NET Applications. Recently we had a requirement to add HTML based formatting to the Email TraceListener in the Logging Application Block, something that’s unfortunately missing from the base functionality. Thankfully, Enterprise Library is an open source code plex project so implementing a custom solution is a relatively trivial task. The email tracelistener functionality is contained in 3 main files: EmailTraceListener – The actual listener which you add to your configuration EmailTraceListenerData – The object representing the configuration settings EmailMessage – The wrapper object around a LogMessage which gets sent via email. Unfortunately because of the the way these classes are implemented in the EnterpriseLibrary Logging Block, they are not easily extended due to dependencies on Private Variables and Internal classes in the EnterpriseLibaray Logging Assembly so they need to be fully implemented in your own solution.

Implementing a Solution

Step 1 was to take a copy of these three files and place them in my own Library Solution. I prefixed the name of each of them with Html; HtmlEmailTraceListener, HtmlEmailTraceListenerData and HtmlEmailMessage. Other code needed to be cleaned up including removing some dependencies on the internal ResourceDependency attributes used to decorate properties within the class & tidying up the Xml Documentation Comments. The main change was then to enable the IsBodyHtml flag on the mail message itself. This was done in the CreateMailMessage method of the HtmlEmailMessage

protected MailMessage CreateMailMessage()
{
	string header = GenerateSubjectPrefix(configurationData.SubjectLineStarter);
	string footer = GenerateSubjectSuffix(configurationData.SubjectLineEnder);

	string sendToSmtpSubject = header + logEntry.Severity.ToString() + footer;

	MailMessage message = new MailMessage();
	string[] toAddresses = configurationData.ToAddress.Split(';');
	foreach (string toAddress in toAddresses)
	{
		message.To.Add(new MailAddress(toAddress));
	}

	message.From = new MailAddress(configurationData.FromAddress);

	message.Body = (formatter != null) ? formatter.Format(logEntry) : logEntry.Message;
	message.Subject = sendToSmtpSubject;
	message.BodyEncoding = Encoding.UTF8;
	message.IsBodyHtml = true;

	return message;
}

Using your new solution

Once implemented it’s simply a matter of reconfiguring your app/web.config logging sections to use the new types you’ve created instead of the original Enterprise Library types. You need to change the type and listenerDataType properties of your Email Listener in the &gl;listeners@gt; section of your config.

<listeners>
      <!-- Please update the following Settings: toAddress, subjectLineStarter, subjectLineEnder-->
      <add name="EmailLog"
           toAddress="toAddress@example.com"
           subjectLineStarter="Test Console - "
           subjectLineEnder=" Alert"
           filter="Verbose"
           fromAddress="fromAddress@example.com"
           formatter="EmailFormatter"
           smtpServer="smtp.gmail.com"
           smtpPort="587"
           authenticationMode="UserNameAndPassword"
           useSSL="true"
           userName="fromAddress@example.com"
           password="Password"
           type="YourLibrary.YourNamespace.HtmlEmailTraceListener, YourLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=0000000000000000"
           listenerDataType="YourLibrary.YourNamespace.HtmlEmailTraceListenerData,  YourLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=0000000000000000"
           traceOutputOptions="Callstack" />
    </listeners>

You’ll also need to ensure that you’ve escaped your Html formatted textFormatter template in the formatters section of your code. i.e. replacing <html> with &lt;html&gt;

<formatters>
      <add name="EmailFormatter" 
              type="Microsoft.Practices.EnterpriseLibrary.Logging.Formatters.TextFormatter, Microsoft.Practices.EnterpriseLibrary.Logging, Version=5.0.414.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" 
              template="
                  &lt;html&gt;
                  &lt;body&gt;
                  &lt;table border=&quot;1&quot; style=&quot;border: solid 1px #000000; border-collapse:collapse;&quot;&gt;
                  &lt;tr&gt;&lt;td&gt;&lt;b&gt;Message&lt;/b&gt;&lt;/td&gt;&lt;td&gt;&lt;b&gt;{message}&lt;/b&gt;&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Local TimeStamp&lt;/td&gt;&lt;td&gt;{timestamp(local)}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Timestamp&lt;/td&gt;&lt;td&gt;{timestamp}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Title&lt;/td&gt;&lt;td&gt;{title}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Severity&lt;/td&gt;&lt;td&gt;{severity}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Category&lt;/td&gt;&lt;td&gt;{category}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Priority&lt;/td&gt;&lt;td&gt;{priority}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;EventId&lt;/td&gt;&lt;td&gt;{eventid}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Local Machine&lt;/td&gt;&lt;td&gt;{localMachine}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;AppDomain&lt;/td&gt;&lt;td&gt;{appDomain}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;LocalDomain&lt;/td&gt;&lt;td&gt;{localAppDomain}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Local Process Name&lt;/td&gt;&lt;td&gt;{localProcessName}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Local Process&lt;/td&gt;&lt;td&gt;{localProcessId}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Win32ThreadId&lt;/td&gt;&lt;td&gt;{win32ThreadId}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;ThreadName&lt;/td&gt;&lt;td&gt;{threadName}&lt;/td&gt;&lt;/tr&gt;
                  &lt;tr&gt;&lt;td&gt;Extended Properties&lt;/td&gt;&lt;td&gt;
                  &lt;table border=&quot;1&quot; style=&quot;border: solid 1px #000000; border-collapse:collapse;&quot;&gt;
                  {dictionary(&lt;tr&gt;&lt;td&gt;{key}&lt;/td&gt;&lt;td&gt;{value}&lt;/td&gt;&lt;/tr&gt;)}
                  &lt;/table&gt;&lt;/td&gt;&lt;/tr&gt;
                  &lt;/table&gt;
                  &lt;/body&gt;
                  &lt;/html&gt;
" />

    </formatters>

All done. Now you can happily send email log messages in HTML format via your Application Logging calls.

~EoinC

Permanent link to this article: http://trycatch.me/implementing-html-formatted-emails-in-the-enterprise-library-logging-block/

May 15 2013

Using Signlar to Publish Dashboard Data

SignalR

SignalR

Recently David Fowler announced the release of the Signlar 1.1.0 Beta Release. So I decided to do some dabbling to get a prototype application up and running. The solution is pretty simple. It uses a SignlaR hub to broadcast the current Processor % usage, and renders it in a nice visual graph using HighCharts.

 

Important!

The completed solution can be found on GitHub at https://github.com/eoincampbell/signalr-processor-demo 

First things first we’ll need a bare bones web application which we can pull in the relevant nuget packages into. I started with a basic empty web application running under .NET 4.5. Installing the signlar & highcharts packages is a breeze. Open up the PowerShell Nuget Console and run the following commands. HighCharts gets installed as a solution level package so you’ll need to manually copy the relevant JavaScript files to your scripts directory in your application.

Install-Package HighCharts
Install-Package Microsoft.AspNet.SignalR

The Hub

Signalr relies on a “Hub” to push data back to all the connected Clients. I’ve created a “ProcessorDataHub” which implements the Signalr Base Hub to manage this process. It contains a constructor for Initializing a static instance of my ProcessorTicker class, and a start method to start the thread within the ticker. The HubName attribute specifies the name which the hub will be accessible by on the Javascript side.

[HubName("processorTicker")]
public class ProcessorDataHub : Hub
{
    private readonly ProcessorTicker _ticker;

    public ProcessorDataHub() : this(ProcessorTicker.Instance) { }

    public ProcessorDataHub(ProcessorTicker ticker)
    {
        _ticker = ticker;
    }

    public void Start()
    {
        _ticker.Start(Clients);
    }
}

The ProcessorTicker

The heavy lifting is then done by the ProcessorTicker. This is instantiated with a reference to the Clients object, a HubConnectionContext which contains dynamic objects allowing you to push notifications to some or all connected client side callers. The implementation is fairlly simple using a System.Thread.Timer which reads the current processor level from a peformance counter once per second, and Broadcasts that value to the client side.

Since the Clients.All connection is dynamic, calling “updateCpuUsage” on this object will work at runtime, so long as the relevant client side wiring up to that expected method has been done correctly.

Clients.All.updateCpuUsage(percentage);

The Client Side

One change since the previous version of SignalR is the requirement for the developer to manually & explicity wireup the dynamically generated Javascript endpoint where SignalR creates it’s javascript. This can be done on Application Start by calling the RouteTable..Routes.MapHubs() method

protected void Application_Start(object sender, EventArgs e)
{
RouteTable.Routes.MapHubs();
}

Finally we’re ready to consume these published messages on our Client Page. Signlar requires the following javascript includes in the Head Section of your page.

<script type="text/javascript" src="/Signalr/Scripts/jquery-1.6.4.js"></script>
<script type="text/javascript" src="/Signalr/Scripts/jquery.signalR-1.1.0-beta1.js"></script>
<script type="text/javascript" src="/Signalr/signalr/hubs"></script>

With those inplace, we wire up our own custom Javascript function to access our ProcessorTicker, start the Hub on a button click, and begin receiving and processing the

    <script type="text/javascript">
    $(function () {
        var ticker = $.connection.processorTicker;

        //HighCharts JS Omitted..

        ticker.client.updateCpuUsage = function (percentage) {
            $("#processorTicker").text("" + percentage + "%");

            var x = (new Date()).getTime(), // current time
                y = percentage,
                series = chart.series[0];

            series.addPoint([x, y], true, true);
        };

        // Start the connection
        $.connection.hub.start(function () {
            //alert('Started');
        });

        // Wire up the buttons
        $("#start").click(function () {
            ticker.server.start();
        });
    });
</script>

The result is that I can fire up a number of separate browser instances and they’ll all get the correct values published to them from the hub over a persistent long running response. Obviously this an extremely powerful system that could be applied to Live Operations Systems where dash boards have traditionally relied on polling the server at some regular interval.

Live Processor Data to Multiple Browsers via SignalR

Live Processor Data to Multiple Browsers via SignalR

~Eoin C

Permanent link to this article: http://trycatch.me/using-signlar-to-publish-dashboard-data/

Apr 29 2013

When’s a Deep Dive not a Deep Dive ?

Global Windows Azure Bootcamp

This weekend, I attended the Global Windows Azure Deep Dive conference in the National College of Ireland, Dublin. Microsoft in conjunction This was a community organised event where Local & National IT Organisations, Educational Institutions & .NET Communities were running a series of events in parallel in a number of cities around the world. The purpose; Deep Dive into the latest technology available on Microsoft as well as take part in a massively parallel lab where participants from all over the world would spin up worker roles to contribute to 3D graphics rendering based on depth data from a KINECT. Alas, Deep it was not, and Dive we didn’t.

I suppose I can’t complain too much. You get what you pay for and it was a free event but I’d have serious reservations about attending this type of session again. Don’t get me wrong, I don’t want to sound ungrateful, and fair dues to the organisers for holding the event but if you’re going to advertise something as a “Deep Dive” or a “Bootcamp” then that has certain connotations that there would actually be some Advanced Hands-on learning.

Instead the day would barely have qualified as a Level 100 introduction to 2 or 3 Windows Azure technologies interspersed with Sales Pitches, Student Demo’s of their project work and filler talks relating to cloud computing in general. Probably most disappointingly we didn’t actually take part in the RenderLab experiment which kinda torpedoed the “Global” aspect of the day as well. You can see the agenda below. I’ve highlighted the practical aspects in Red.

Time Topic
0930 Welcome Dr Pramod – Pathak, Dean, School of Computing, NCI
0935 Schedule for the day  – Vikas Sahni, Lecturer, School of Computing,NCI
0940 How ISIN can help – Dave Feenan, Manager, ISIN
0945 Microsoft’s Best Practice in Data Centre Design – Mark O’Neill, Data Center Evangelist, Microsoft
1000 Virtual Machines – Demo and Lab 1 - Vikas Sahni, Lecturer, School of Computing, NCI
1100 Careers in the Cloud – Dr Horacio Gonzalez-Velez, Head, Cloud Competency Center, School of Computing, NCI
1110 Graduates available today - Robert Ward, Head of Marketing, NCI
1120 Break
1135 Web Sites – Demo and Lab 2 – Vikas Sahni, Lecturer, School of Computing, NCI
1235 Building the Trusted Cloud – Terry Landers, Regional Standards Officer for Western Europe, Microsoft
1300 Lunch
1400 Tools for Cloud Development – Colum Horgan, InverCloud
1410 Windows Azure Mobile Services – Overview and Showcase –  Vikas Sahni, Lecturer, School of Computing, NCI and Students of NCI
1440 Developing PaaS applications – Demo – Michael Bradford, Lecturer, School of Computing, NCI
1530 Break
1545 Windows Azure – The Big Picture – Vikas Sahni, Lecturer, School of Computing, NCI
1645 Q&A

Alas even the practical aspects of the day were extremely basic and the kinda of thing that most people in the room had done/could do in their own spare time.

  • During the Virtual Machines Lab, we spun up a Virtual Machine from the Windows Azure Gallery and remote desktop connected into it.
  • During the Websites Lab, we deployed a WordPress install… unless you were feeling brave enough to do something else. To be fair I hadn’t done a hands on GitHub Deploy of the code so that was interesting.
  • During the PaaS Application Demo… well it was supposed to be a Hello World web/worker role deployment but god love the poor chap he was out of his depth with Visual Studio and had a few technical hiccups and it was just a bad demo. Upshot was we ran out of time before there was an opportunity for any hands on time in the room.

At 15:30 we left… I didn’t have another lecture in me, although at least we’d had the common courtesy to stay that long. Half the room didn’t come back after lunch.

The takeaways; I know that alot of time and effort goes into these events, and particularly when they are free, that time and effort is greatly appreciated. But you need to make sure you get your audience right. If you advertise Advanced and deliver basic, people will be disappointed. That was clear from the mass exodus that occured during the day… I’m kinda curious to know if there was anyone around for the Q&A at all. I’ll be sure as heck checking the agenda on these type of events before committing my time to them in future. We aren’t currently using Windows Azure in our company yet, and embarrassingly I had been promoting it internally and had convinced several of my colleagues to give up their Saturday for it.

~Eoin C

Permanent link to this article: http://trycatch.me/whens-a-deep-dive-not-a-deep-dive/

Older posts «