Foo Theory

Introduction

In the previous part of this series, we talked about the importance of spatial data in our everyday lives. We talked about vector and raster data, and we explained that the current version of SQL Server 2008 focuses on two-dimensional vector data. Next, we introduced the new geography and geometry data types in SQL server 2008, and we took a quick detour into some important object-oriented principles.

In this article, we start out by taking a look at the class hierarchies for both the geometry and the geography data types in SQL Server 2008. We will focus on the concrete classes in the class diagram which are:

1. Point. 
2. LineString
3. Polygon

We will also take a look at the collection classes in the object hierarchy, which allow us to store multiple instances of each type.

Once we have a good grasp on the geometry and geography class hierarchy we will dive into the details of each concrete class. We will start our investigation by looking into the details of the Point and LineString class, and we will write a T-SQL script that highlights the most important features. In the process we will take a look at the three different data formats that can be used to represent spatial data:

1. The Well-Known-Binary  (WKB) format
2. The Well-Known-Text (WKT) format
3. The Geography Markup Language (GML) data format.

All of these formats are published as a standard by the Open Geospatial Consortium (OGC), which is the leading standards body for geospatial and location based services.

The Geometry class hierarchy

Overview

A class diagram showing the Geometry class and all of its sub-classes is shown below (note that the Geography class diagram is basically identical, with the Geography class at the root of the tree):

The darker-blue shaded classes are abstract base classes, so they cannot be directly instantiated in the database engine. The light-blue shaded classes are the concrete classes that we can use directly in our T-SQL code:

1. Point
2. LineString
3. Polygon

Each of these single-instance classes has a collection equivalent that can contain multiple instances:

1. MultiPoint
2. MultiLineString
3. MultiPolygon

The GeomCollection class is an additional collection class that can contain a mixture of instances of any type (Points, LineStrings and Polygons).

Supported Data Formats

All of the classes in the geometry and the geography class hierarchy can be represented by three different formats:

1. The Well-Known Binary (WKB) format. This is a byte-stream (binary) representation for a geography instance. This format is the preferred serialization format for those applications that need to store geospatial information in a compact, self-contained format.
2. The Well-Known Text (WKT) format. This is a compact, easy to read representation. Because of its user-friendly format it is the most commonly used representation for ad-hoc geospatial queries.
3. The Geography Markup Language (GML) format. This is an XML-based representation, which is best suited for including geospatial information in an XML document. This format is very useful for those applications that need to exchange geospatial information by means of an XML Web service.

All of the above formats are published as standards by the OGC. In the code samples that follow, we will be showing multiple examples of each format.

The Point Class

Basic Representation

A Point is a zero-dimensional object which represents a single, exact location. It always contains an X and Y coordinate, and can optionally can contain a Z (elevation) value, and a M (measure) value. The measure value is an arbitrary floating point measurement value that you can associate with the Point instance.

The sample T-SQL script (GeometryPoint.sql)illustrates some of the capabilities of the Point class. Throughout all of our demos, we will be working with a very simple table called GeometryDemo, which allows us to associate a Geometry instance with an integer ID:

   1: CREATE TABLE GeometryDemo

   2: (

   3:     ID    INT NOT NULL,

   4:     Value GEOMETRY NOT NULL

   5: );

In the following code sample, we insert a Point instance, and then perform a select to retrieve the point instance in a number of formats:

   1: -- Insert a simple Point with an

   2: -- X and Y coordinate

   3: INSERT INTO GeometryDemo (ID, Value)

   4: VALUES    (1, 'POINT(50 65)');

   5:  

   6: -- Get the X and Y coordinates and 

   7: -- the most relevant methods

   8: SELECT ID, Value.ToString() AS WKT, 

   9:            Value.STX AS [X Coordinate],                        

  10:            Value.STY AS [Y Coordinate],

  11:            Value.STAsBinary() as [Binary Representation],

  12:            Value.STasText() as [Text Representation]

  13: FROM GeometryDemo

  14: WHERE ID = 1;

In lines 3 and 4, we use the WKT format to insert a Point with an X coordinate of 50 and a Y coordinate of 65. The WKT format for a Point simply requires you to to specify the X and Y coordinates. Note that you should NOT use a comma separator between the two, which might seem counterintuitive at first.

In lines 8 through 14 we select the instance that we just inserted. Since Value is a geometry class instance, we need to use the  dot (".") notation to access properties and invoke methods on the object:

• In line 8 we invoke the ToString() method. This method will return the Well-Known Text (WKT) representation of the instance.
• In lines 9 and 10 we invoke the STX and STY properties to get the X and Y coordinates of our Point.
• In line 11 we invoke the STAsBinary() method to get the binary representation of the Point.
• Finally, in line 12 we invoke the STAsText() method, which will again return the WKT format of the Point.

One important note: although TSQL by itself is not case sensitive, the CLR methods and properties on the spatial data types are, so make sure that you spell the property and method names correctly! For example if you would spell "Value.STAsText()" as "Value.STasText()" you would get the following error message:

   1: Msg 6506, Level 16, State 10, Line 34

   2: Could not find method 'STasText' for type 'Microsoft.SqlServer.Types.SqlGeometry' in assembly 'Microsoft.SqlServer.Types'

From the above output, we see that the WKT representation for the Point is returned by both the ToString() and the STAsText() methods. We also see the binary WKB representation and the X and Y coordinates of the Point instance.

Using the Z and M coordinates

As we mentioned earlier, a Point can optionally contain a Z (Elevation) coordinate and a M (Measurement) value. Note that both the Z and M values should be floats. Below is a T-SQL example:

   1: -- Insert a Point with an X, Y, Z and

   2: -- M coordinate

   3: INSERT INTO GeometryDemo (ID, Value)

   4: VALUES (2, 'POINT(55 70 100.2 50)');

   5:  

   6: SELECT ID, Value.ToString() AS WKT, 

   7:            Value.STX AS [X Coordinate],                        

   8:            Value.STY AS [Y Coordinate],

   9:            Value.Z AS [Elevation],

  10:            Value.M AS [Measure]

  11: FROM GeometryDemo

  12: WHERE ID = 2;

OGC-compliant methods and Microsoft Extensions

The attentive observer probably noticed that some method and properties have an ST prefix (such as STX, STY and STAsText()), while other properties and methods do NOT have this prefix (for example: Z, M and ToString()). All methods and properties that are prefixed with ST are OGC-compliant methods, while the other methods and properties are Microsoft extensions to the OGC standard.

In the documentation you will notice that the OGC standard methods and the Microsoft extensions are cleanly separated:

In the above diagram we also see sections for static methods (both OGC and Microsoft extensions). We will take a look at static methods in a later section of this post.

Geography Markup Language (GML)

The only format that we have not worked with so far is the Geographic Markup Language (GML), let's write a query to retrieve both points as GML:

   1: --    Select both Points as GML

   2: --  Note that we are NOT retrieving the Z and M

   3: --  coordinates of the second point

   4: --  because they are NOT OGC compliant!

   5: SELECT ID, Value.AsGml() as [GML]

   6: FROM GeometryDemo;

The output of the query is shown below:

The standard XML namespace for GML will have the URN: http://www.opengis.net/gml.

There is also one important note that we can make from the above output: the Z and M coordinates that we used for the second point are NOT returned in the GML! That is because they are proprietary Microsoft extensions to the OGC standard.

Using the static Point methods

The methods and properties we have used so far were invoked on an instance of the Point class, so we used the "dot" (.) notation:

instanceName.Property

-or-

instanceName.MethodCall(... method arguments here ...)

Instance properties and methods are always tied to a particular object instance, in this case a Point instance. Besides instance methods, the spatial data types in SQL Server 2008 also use static methods and properties. Static methods and properties are tied to the class itself, and can thus be invoked without having an instance around. You use the "::" notation in combination with the class name to invoke a static method or property in T-SQL:

className::Property

-or-

className::MethodCall()

One frequent usage pattern for a static method is the Factory Pattern. The Factory Pattern returns a new instance of the class through a static method. This pattern is used in both geospatial types in SQL Server 2008. Below is a code example from the GeometryPoint.sql T-SQL script:

   1: /* ========================================================================= 

   2:    ======= Using the Static methods to create a Point Instance =============

   3:    ========================================================================= */

   4: DECLARE @Pnt1 Geometry, @Pnt2 Geometry, @Pnt3 Geometry, @Pnt4 Geometry, @Pnt5 Geometry

   5: DECLARE @xmlSnippet xml;

   6:  

   7: --    1. Create a Point from an X and Y coordinate and an SRID

   8: SET @Pnt1 = geometry::Point(30, 20, 0);

   9:  

  10: -- 2. Create a Point using the Parse method

  11: SET @Pnt2 = geometry::Parse('POINT( 90 23 56000 231)');

  12:  

  13: -- 3. Create a Point using the OGC STPointFromText method

  14: -- Notice the SRID at the end

  15: SET @Pnt3 = geometry::STPointFromText('POINT(20 25)', 0)

  16:  

  17: -- 4. Create a Point from GML 

  18: set @xmlSnippet = 

  19:     '<Point xmlns="http://www.opengis.net/gml"> <pos>34 23</pos> </Point>';

  20: SET @Pnt4 = geometry::GeomFromGml(@xmlSnippet, 0);

  21:  

  22: -- 5. Null Point

  23: SET @Pnt5 = geometry::[Null];

  24:  

  25: -- Now select all points

  26: SELECT @Pnt1.ToString() AS [From Arguments],

  27:        @Pnt2.ToString() AS [From Parse],

  28:        @Pnt3.ToString() as [From STPointFromText],

  29:        @Pnt4.ToString() AS [From Xml],

  30:        @Pnt5 AS [Null Point];

• In line 8, we use the static geometry::Point() method to create a Point instance by passing in the X, Y, Z and M coordinates.
• In line 11, we use the geometry::Parse() method to create a Point instance from Well-Known Text (WKT).
• In line 15, we use the geometry::STPointFromText() method to create a Point instance from Well-Known Text. The is the OGC equivalent of the geometry::Parse() method.
• In line 18, we have an example of how to create a Point instance from a Geographic Markup Language (GML) snippet, using the static geometry::GeomFromGml() method.
• Finally, in line 23 we create a NULL Point instance with the static geometry::[Null] property.

The output of the select statement is shown below:

As we can conclude from the above output, the expected Point instances were created by means of the static methods.

The Spatial Reference Identifier

In the previous code sample, you might have noticed the "0" after the @xmlSnippet parameter:

   1: set @xmlSnippet = 

   2:     '<Point xmlns="http://www.opengis.net/gml"> <pos>34 23</pos> </Point>';

   3: SET @Pnt4 = geometry::GeomFromGml(@xmlSnippet, 0);

This argument is the SRID, or the Spatial Reference Identifier. The SRID corresponds to a spatial reference system based upon the specific ellipsoid used for either flat-earth or round-earth mapping. Different projection systems exist, some of you might be familiar with the Mercator Projection that was used to create the maps that were in most classrooms in recent times. A good discussion of map projections can be found at: http://maps.unomaha.edu/Peterson/gis/notes/MapProjCoord.html.

Each projection is uniquely defined by its own SRID. You can use any projection you want, but you need to be aware that when you are comparing spatial instances using any of the methods (such as STIntersection()) you need to make sure that all objects have the same SRID otherwise you will get a run-time error!

That concludes part 2 of this series. In part three, we take a look at the LineString and the Polygon class, and we will use some visualization tools such as GeoQuery, Spatial Viewer, and SQL Server Management Studio's very own "Spatial Tab" to get a visual representation of our spatial data!

Series Abstract

In this post we take a look at the new spatial data type support in SQL server 2008. First, I will make a case for why you would want to integrate spatial support in your applications. Next we will take a look at the two core spatial data types in SQL Server:  the Geometry and Geography types. We will explain when to use each type, and we will take a look at the main application domains for the two types.

The spatial types are implemented as CLR types in the database engine. Since some database folks might be unfamiliar with CLR types, we will make a brief detour into SQL Server and CLR types. We will talk about the differences between static and instance methods, and explain the TSQL calling notation for each.

Both the Geometry and the Geography types are really just the top-level types of a rich object hierarchy. We will take a detailed look at the different classes in this hierarchy, and we will explore the methods and properties of each class by means of a number of TSQL scripts. A part of this exploration we will take a look at the three data formats that can be used to represent the spatial types: the SQL Server-native  Well-Know-Binary (WKB) format, and the OGC standard data types: the Well-Known-Text (WKT) and the Geography Markup Language (GML) . We will use TSQL scripts for each object to illustrate the different notations.

As always, a picture is worth a thousand words, and nowhere this is more the case as for spatial information. Therefore, I will use both the "Spatial Results" tab in SQL server and a number of third-party rendering tools such as SpatialViewer and GeoQuery to present a spatial query result.

After we have a good understanding of the spatial data types, we will put them to some practical use. We will use Virtual Earth to create applications in which we create mashups of Spatial data with a variety of business data. A large volume of GIS data is available on the public domain (some good sources are the US Census and USGS Web sites), but the format of this data is typically not compatible with SQL Server. Therefore, as part of this series we will create a library that will enable us to convert the "traditional" formats into a SQL Server 2008-compatible format.

The above abstract covers a wide variety of formats, therefore I am planing to spread out this article over a number of different posts. This first post will cover the need for spatial support in our applications, the basics of the Geometry and Geography data types and a quick OO primer.

Why Spatial Data?

These days it is hard to find any data that DOES NOT have a spatial aspect. A number of applications attempt to answer questions like the following:

1. Where are my customers located?
2. What cities have the highest accident rates?
3. What Florida counties have the highest flood risk? (OK, that one's easy to answer: "All of them"!)

As GPS devices become more prevalent, more and more data is geo-tagged. For example, a lot of modern mobile phones have both a camera and a GPS chip build in.

While the above applications use spatial data only as part of their overall data set, there are a number of applications that use mapping and spatial data as their primary output:

1. Consumer products such as Microsoft's Virtual Earth or Google maps.
2. The government publishes the census results as spatial data.
3. Utilities use mapping tools such as ESRI Server to plot the layout of electrical grid lines or underground gas lines.

While the above examples are pretty self-evident, spatial data also plays an import role in applications that we might not think about right away:

• When a warehousing application generates a pallet "pick run", it will use geospatial information to calculate the most optimal route.
• When an interior architect uses a CAD tool such as AutoDesk to perform modeling of an interior space, he/she is using spatial data quite extensively.
• Computer-aided manufacturing tools use spatial data to layout out parts on a piece of sheet metal.
• The program in a municipal kiosk uses spatial data and geometric algorithms to predict the arrival times of buses and trains.
• A multi-player computer game use spatial coordinates to keep track of the current locations of all game participants.

From the above discussion it is clear that a large number of application have a need to work with spatial data. Some common requirements for spatial data support are:

• The ability to store spatial coordinates directly in the database, preferably in the same tables as its associated data.
• The data types used for this spatial data should go beyond simple point coordinates. Most geospatial data consists out of a mixture of points, lines (both single-segment and multi segment), and closed shapes (typically referred to as polygons in the literature).
• The ability to perform a multitude of operations on this geospatial data. For example, a chip design application might want to assure that certain routes on a chip do not cross, other applications will have a need to calculate the area of a complex polygon etc.

In the next section will take a look at how SQL Server 2008 addresses the above requirements.

Spatial Data Support in SQL Server 2008

Types of Spatial Data

At the highest level, we recognize two major classes of spatial data:

1. Vector Data. Vector data is data expressed by a set of vertices and their relationship to one another. Common spatial features represented by vector data include:
   • Points.
   • Lines (where a line can have one or more segments)
   • Polygons. Polygons are typically used to represent areas and regions.
2. Raster Data. Raster data is data expressed as a matrix of cells. We typically recognize raster data as images. Within a spatial or GIS context, we see raster data manifested as:
   • Satellite images.
   • The Virtual Earth bird's eye images.
   • Google "street level" images.

An example of each type of spatial data is shown below

Sample Raster Image (a sample satellite heat map)	Sample Vector (in this case a simple line vector)

SQL Server 2008 does focus exclusively on Spatial Vector data.

The SQL Server 2008 Spatial Data Types

SQL Server 2008 introduces two new data types:

1. Geometry. The geometry data type is based on the Cartesian coordinate system,  based upon a "flat earth" representation model. In this model, a point is represented by an X, Y and optionally a Z coordinate. This usage domain of this data type is either:
   • The representation of simple coordinates in a two or three dimensional space. A example is the precise location of a pallet in a warehouse.
   • The representation of map coordinates, where distances are limited enough so that they are not affected by the round earth model.
2. Geography. The geography data point can store points, lines, polygon and collections of each of these, using a "round earth" model as opposed to the "flat earth" model used by the Geometry data type. Instead of using X and Y coordinates, the geography data type will use a latitude/longitude combination to represent a single point. Most of the GIS data available on the Web is latitude/longitude based, so the Geography data type is the data type you should used in most of you GIS applications, especially when you are dealing with longer distances where the shape of the earth becomes relevant.

Geometry : Cartesian coordinate system, "flat earth" model	Geography: Latitude/longitude coordinates, "round earth" model

The geometry data type conforms to the Open Geospatial Consortium (OGC) Simple features for SQL specification version 1.1.0. The OGC is a non-profit, voluntary concensus standards organization which is the leading consortium when it comes to the drafting and ratification of standards for geospatial and location based services. One of the standards published by the OGC is the Well-Know-Text specification (WKT) for spatial data types. We will investigate this format in detail in part 2 of the series.

CLR Objects in the Database Engine and the Database Developer

Both the geometry and geography data types are implemented as user-defined types (UDT's) in the database engine. These UDT's are implemented as .NET Common Language Runtime (CLR) types. Before we take a more detailed look how to use these types, it is important that we make sure that the reader is familiar with some basic OO principles. If you are already familiar with object-orientation and .NET types, please feel free to skip ahead to the next part in this series.

The core abstraction in the object-oriented world is the class. A class can be seen as the blueprint for a particular type. From this blueprint, a client can create any number of actual object instances. A class hides away its implementation details behind a set of publicly accessible properties and methods. This principle is called encapsulation or information hiding. For example, a Car class will encapsulate the behavior of an automobile. It might expose methods such as Start, Stop, SlowDown and SpeedUp, and it might provide properties such as CurrentSpeed and DaysTillNextOilChange. The Car class will hide the details of how it executes these methods and exposes these properties, enabling the client to be "blissfully unaware" of the implementation details of the Car class.

Classes can be related to one another. At a high-level we can recognize the following types of relationships:

1. The "has-a" relationship. For example, a Car class might have a SteeringWheel and a GasTank. This type of relationship is sometimes referred to as a "uses" relationship, for example the Car class "uses" the SteeringWheel class and "uses" the GasTank class.
2. The "is-a" relationship. To stay with our Car example, a SportsCar is a specialized type of Car,  which is specially equipped. It might have a Spoiler, a HighPerformanceEngine etc.. When classes have such a relationship they will use an OOP concept called inheritance to implement this relationship. In an inheritance implementation, the class from which we inherit is called the base class and the more specialized class is called the sub class. The sub class gets all of the functionality of the base class "for free", so it only needs to worry about implementing it own specialized behavior on top of the functionality of the base class. For example, the SportsCar class only implements the additional functionality that makes it a sports car.

An example of a "uses" ("has a") and an inheritance ("is a") relationship is shown in the figure below:

Some base classes only define a general abstract concepts or entities. These types of classes cannot be directly instantiated by the client, since they are not "feature complete". Such a class is called an abstract class The features defined by an abstract class are implemented by a sub class of the abstract class. Such a sub class that can be instantiated is called a concrete class. We will see in the next section that the geometry and geography classes are indeed defined as abstract classes in the database engine.

In the next part of this series, we will take a look at the class diagrams for both the geography and geometry classes, and we'll start writing some TSQL code!

Note: This post is a work in progress... not yet completed...

Overview

When Visual Studio 2008 and the .NET Framework 3.5 shipped in late 2007, the feature that got by far the most attention was Language Integrated Query (LINQ). This article will provide a quick overview of the LINQ technologies, and will then focus in on LINQ to XML, which is a subset of the LINQ technologies.

This article is structured as follows:

1. The next section will provide a short overview of LINQ in general. We take a look at what the main drivers were behind the creation of LINQ, and we will start looking at some of the specifics of LINQ to XML.
2. Once we have established a basic understanding of LINQ to XML, we will take a look at its object model. LINQ to XML introduces an easy to use, flattened object model, especially when compared to the XML DOM object model.
3. Next, we will take a look at what is in my opinion one of the most important features of LINQ to XML, and that is Functional Construction. Functional construction enables us to create an XML document in a very straightforward way. No longer do we need to use the elaborate XML DOM object model and write 100 lines of complex code, just to create a simple XML document that is just a few lines long.
4. After we have an understanding of how to create an XML document, we take a look at how we can use the features on the new LINQ to XML object model to query, iterate, manipulate and validate an XML document.
5. To illustrate how easy it is to leverage the LINQ to XML features in a real-world application, we will take a look at an example that  performs two-way binding of a rather complex XML document to a WPF tree control.
6. To round out our discussion, we will take a look at some practical tips and tricks, which enables us to take full advantage of LINQ to XML, and we will also mentioned some common pitfalls, and inform you how to save time not making the same mistakes as your truly ;-)!

Prerequisites

If you want to run the code samples that are associated with this article, you will need the following pre-requisites:

• Since we will be using LINQ, you obviously need to have a version of Visual Studio 2008 installed. If you install Visual Studio, you will automatically also install .NET 3.5 and the C# 3.0 compiler.
• We will be showing how you can use LINQ to SQL to create XML documents, so you will need to have SQL Server 2005 or the SQL Server 2008 beta installed. If you have the express edition installed, I recommend you also download SQL Server Management Studio express.
• We will be using the AdventureWorks sample database to illustrate how we can create an XML document from a SQL Server database, so you should have the latest version of the AdventureWorks database installed. If you currently don't have this database installed, you can navigate to: http://www.codeplex.com/MSFTDBProdSamples/Release/ProjectReleases.aspx?ReleaseId=4004, and download the appropriate version of the AdventureWorks*.msi file, and follow the instructions to install the database on your SQL server.

LINQ Overview

Background - Problem Description

Most of us write business applications to earn our living. All business  applications that I ever worked with deal with some type of data, and quite a number of them involved multiple heterogeneous data sources. 

In the early days (1980-1990) all of the data for an application was centralized in one file, and the data access capabilities were directly built into the language (I know that at this point, my friend Pete Miller is thinking back nostalgically about his FoxPro and DBASE days... ;-). The upside of this was that data querying and data manipulation were a core part of the programming experience, but the downside was that every platform (DBASE, FoxPro, FileMaker) would integrate these data access features in a completely different way, tailored to the capabilities of the underlying data access tool. As a result, programmers faced a steep learning curve when moving from one platform to another. Another drawback of these first-generation data access systems was scalability. All platforms were file-based, and therefore had a hard time scaling to multiple users and large data sizes.

In response to this and other issues, (1990s  through early 2000s) relational database systems (RDBMS) were created. All of these databases used an emerging query and DML language called SQL to access the data, and database programs migrated from being strictly tied to a particular data access system dialect, to using industry-standard SQL. The query and data manipulation features became again external to the core programming languages, and were typically made available through a set of external libraries (db-Library anyone?). This process was accelerated by the creation of ODBC, which provides a standard means of accessing any database that provides an ODBC driver.

In recent years, we have seen the emergence of Object-Relational Mapping (ORM) tools. The main philosophy behind the ORM movement was the fact that designers and programmers are dealing with objects to represent their data, but each time they have to persist or load these objects from a data store, they have to make a paradigm shift back to SQL and the relational model,  and the specific details associated with the database access libraries that are being used. An ORM tool, such as NHibernate takes over the responsibility of persisting and loading objects to the database in a transparent fashion.

While both SQL and their associated data access technologies (such as ADO.NET) and/or the newer ORM tools have simplified things significantly, we still have a number of challenges that remain:

1. Most real-world applications out there deal with other data types besides relational data. Indeed, in this connected world, we access a variety of XML data sources, we connect to RSS feeds, leverage Web Services using the SOAP or REST protocols, access data in Active Directory or some other LDAP-based data store, and so on. Each of these data sources have their own data access paradigms.
2. When working with "plain old objects", we have to use a very different API to  sort, filter, group our otherwise manipulate our objects, as compared to relational data. Wouldn't it be nice if we could have one standard API for these common tasks?
3. Often we have to perform complex data transformations and/or data shaping. The way in which these transformations are performed is often dependent on the type of the data. For XML we use XSLT, for relational data we use views or complex joins, for objects we use manual code etc. Again, it would be nice to have access to one standard approach for performing transformations.
4. What if we want to access data from a dynamic language, such as Ruby, IronPython, PowerShell, F# or any of the future DLR-based languages? Will these languages be able to use the same data access tool as the statically-typed .NET languages such as C# or VB.NET?

Microsoft's Answer: LINQ

At the core, LINQ is really a set of constructs, built into the language, which allow us to work with any type of data, be it relational, XML or plain old objects. LINQ is supported in both C# 3.0 and VB 9.0, which are compilers that shipped with Visual Studio 2008. In a way, LINQ brings us "back to the future", making querying and manipulating data  a core programming concept again. The main different with the "old school" languages such as FoxPro is that LINQ is fully independent of the type of data that is being accessed (object, relational data, XML, etc.), and the specific implementation of the data source (SQL Server, Oracle etc.).

The LINQ features in the C# 3.0 compiler are built on top of a number of other language enhancements such as:

1. Anonymous types
2. Anonymous methods
3. Type inference
4. Lambda expressions
5. Expression trees
6. Extension methods
7. Instance and collection initializers
8. Partial methods

A number of these features were addressed in some of my previous posts. Please refer to the blog archives for more information.

The .NET 3.5 Framework also provides LINQ support through a number of types, available in the System.*.LINQ namespaces. These types provide additional support on top of the compiler features.

A graphical overview of LINQ is shown below:

LINQ to XML

LINQ to XML is portion of LINQ that allows us to:

• Construct
• Traverse
• Manipulate
• Query
• Search

XML documents and fragments, using the standard LINQ API.

One of the main goals of LINQ to XML was to address the main shortcomings in the W3C XML DOM API, as implemented in the System.Xml.* .NET 2.0 namespaces, with a focus on the following areas:

• Simplify XML tree construction with functional construction.
• Eliminate document centricity in favor of element centricity.
• Simplify naming by eliminating prefixes from the API.
• Simplify Node value extraction.

The above topics have always been a hard area to deal with for any programmer working with XML documents. XML DOM code is unnecessary complex and bloated, and often unintentionally obfuscated. For example, it is not easy to imply the structure of the created XML document, when reading XML DOM document creation code.

Other issues that Microsoft wanted to address with LINQ to XML is to allow a developer to quickly move data extracted from a relational model to an XML representation, or from an object graph to an XML document. This current DOM API does not support constructs such as projections inside a XQuery, LINQ to XML provides an elegant solution to this problem.

In the .NET 3.5 framework, LINQ to XML is implemented in the System.Xml.Linq.dll assembly, and exposed through the System.Xml.Linq namespace.

The dependencies of the System.Xml.Linq.dll assembly is show in the figure below:

LINQ to XML Object Model

LINQ to XML was developed with Language-Integrated Query over XML in mind from the onset. It takes advantage of the standard query operators and adds query extensions specific to XML. Just as significant as the Language-Integrated Query capabilities of LINQ to XML is the fact that LINQ represents a consistent query experience across all LINQ enabled APIs and allows us to combine XML queries from other data sources. So, with one query, you can access data from:

1. Local objects in memory
2. An XML Data Source
3. One or more SQL Server data sources

The core classes that make up the LINQ to XML object model are listed below:

As you can conclude from the object model above, the number of classes involved has been dramatically reduced, resulting in a reduced learning curve. Actually I think that the biggest challenge in working with LINQ to XML is to unlearn some of the bad practices that we had to burn into our brain to make the W3C XML DOM work for us.

Some key issues regarding this object model are listed below:    

1. You can now work in a "document free mode" if you would like to do so. In some scenarios, you simply want to create or load some XML, manipulate and query it, and save it back. With the W3C DOM, you would be forced to create an XML document. In LINQ to XML, this is no longer the case. To perform the task listed, you could simply:
   • Create XElements directly (without having an XDocument involved at all)
   • Manipulate the XElements or XAttributes directly.
   • Save the resulting XML tree directly to a writer.
2. XML names have been greatly simplified. LINQ to XML goes out of its way to make XML names as straightforward as possible. One can say that the complexity of XML names does not originate in namespaces, but from XML prefixes. XML prefixes can be used for reducing the keystrokes required when inputting XML or making XML easier to read, however prefixes are just shortcuts for using the full XML namespace. On input LINQ to XML resolves all prefixes to their corresponding XML Namespace and prefixes are  not exposed at all in the programming API. In LINQ to XML, a XName represents a full XML name consisting of an XNamespace and the local name. Developers will usually find it more convenient to use the XNamespace rather than the namespace URI string.
3. An attribute (modeled by means of the XAttribute class) is no longer a subclass of the node class). It is now simply a XName-value pair, which is what it always should have been.

Functional Construction

Querying

WPF Data Binding

Tips and Tricks

Conclusion

Notes: Make sure how to construct an XML document from a LINQ database query

Introduction

Unless you have been living under a rock for the last week, you probably know that Google shipped the first beta-version of it's new browser, named "Google Chrome". I am not planning to start yet another debate here about why I think Google decided at this point that it needed to get in the Web browser business. You can read plenty of interesting comments on various blog sites, for example Paul Thurrot, from the Windows Supersite had some very interesting feedback on the subject in his Windows Weekly Podcast with Leo Laporte.

Most of the  features introduced by Chrome have an equivalent in Microsoft's upcoming IE 8 browser (beta 2 just shipped a couple of weeks ago). I do want to highlight one interesting feature of Chrome and that is the ability to create application shortcuts.

What is  a Chrome Application Shortcut?

A Chrome Application Shortcut is basically a quick, streamlined way to access a Web Application's functionality in it's own window. You can create a shortcut on either the desktop, the "Start bar", or the "Quick Launch" toolbar. Once the short cut is created, you can simply double-click the shortcut to launch the application in it's own window. Using shortcuts is a nice alternative to launching the full browser window, and entering the URL (either manually or through a "favorites" entry).

Also, if you use Google Gears, you optionally have the ability to access the application's functionality if offline mode (if so desired).

Walkthrough

To create a shortcut, first navigate to the URL of the Web Application. Next, click the Page menu next to the address bar, and select the "Create application shortcuts.." menu item, as is shown below:

In the Google Gears dialog box that comes up, you can select the location(s) for your shortcut. So in my case, I elected to create a shortcut to Gmail, and I opted to have a shortcut on both my desktop and the "Quick Launch" bar.

Note that Chrome will use the appropriate icon for your shortcut.

After creating your shortcut, I can simply double-click the shortcut, to open the selected Web Application as show below:

The application opens up in its own Chrome window, without any toolbar or address bar clutter, as shown below:

Table of Contents

Software as a service is a term is frequently making headlines in a number of trade journals. In all articles on the subject, Software as a Service is mentioned as the "next big thing", which is going to revolutionize the software industry as we know it.

This article will provide a high-level overview of the subject. The article structured as follows:

1. In the first section we introduce Software as a Service (know by its acronym Saas), provide a definition and discuss some fundamental topics that form the basis for the technology. Also, since the concept of a "tenant" and "multi-tenancy" is closely associated with Saas, we will include a definition for these terms in the same section.
2. The next section will address how Saas can mitigate the software acquisition risks for an organization.
3. Often, a (false) rumor has been spread throughout the industry, that SaaS eliminates the need for an client organization's IT staff. In our next section, we clarify this misconception, and show how SaaS can empower an IT organization to focus on new initiatives that contribute measurable, immediate value to the client's organization.
4. Next, we discuss the three core attributes that are essential for an effective single instance, multi-tenant application architecture.
5. Four different maturity models have been defined for a multi-tenant SaaS application, our next section discusses these different maturity levels and their specifics.
6. In the last section of this article, I provide the reader with some guidance of how the select the appropriate maturity level for their organization's  Saas application(s).

This article is mostly non-technical is nature. However, I am planning to follow-up this article with a number of technical articles, where we dive into some of the more challenging areas that make it possible to create a highly scalable, configurable and multi-tenant-efficient SaaS solution.

Introduction

When you ask a number of software professionals for a definition of the term "Software as a Service", you will get many different, and sometimes conflicting answers. Slowly tough, more and more experts are starting to agree on the fundamental principles that define Software as a Service, and the concepts that differentiate the technology from traditional, on-premise packaged software solutions.

The current, widely-accepted definition for Software as a Service can best be phrased as follows:

Saas is software deployed as a multi-tenant hosted service and accessed over the Internet.

Saas should not be confused with the services provided by traditional Application Service Providers (ASPs). An ASP typically provides a "shrink-wrapped" application to business users over  the Internet, and therefore can be better compared with a traditional on-premise IT application which has been deployed and hosted by a third-party. These applications were architected as a single-tenant application, and were not designed to share data and business process logic with other applications. Since we used the term tenant here I feel like another definition is in order:

A tenant is a client organization, which is using an application that has been deployed as a SaaS Application.

Multitenancy refers to the architectural principle, in which a single instance of the software runs on as Software as a Service (SaaS) vendor's servers, servicing multiple client organizations (tenants). Multitenancy can be contrasted with a multi-instance architecture, where separate software instances (or hardware systems) are set up for different client organizations. With a multitenant architecture, a software application is designed to virtually partition its data and configuration so that each client organization works with a different virtual application instance.

A typical SaaS application is offered either directly by the vendor or by an intermediary party called an aggregator, which bundles SaaS offerings from different vendors and offers them as part of a unified application platform.

I strongly believe that Saas is going to have a major impact on the software industry, because software as a service will change the way people build, sell, buy and use software. For this to happen though, software vendors need resources and information about developing Saas applications effectively.

When you take a look at our definition for Saas, it should become immediately clear that it does not prescribe any specific application architecture; it does not say anything about

Saas and Software Acquisition Risk Mitigation

Another area in which Saas applications tend to be different as compared to on-premise software is in their licensing model. On-Premise software applications typically have a high cost of entry, either because the application has to be developed from scratch by the internal or external IT staff, or in the case of a purchased application, because of the high up-front licensing fees. In contrast, SaaS application access is typically sold based upon a subscription model. The different subscription models offered by SaaS vendors are:

• A flat time-based fee (monthly, yearly etc.). This time-based fee will typically be much lower than the large up-front licensing fees for purchased on-premise applications.
• Usage based upon the metered activity by the customer, where 'usage activity' can be measured in a variety of ways, including number of business transactions, number of orders processed etc.

Some vendors will use a combination of both models, charging a flat time based fee in addition to some usage-based premium. In addition, most Saas vendors offer a "try before you buy" option, where a potential tenant has the opportunity to try out the application for a period of time before needing to make a commitment towards becoming a full-fledged customer.

In the traditional model, deploying large scale business-critical software systems (SAP anyone ? ;-) has been a major undertaking. As we mentioned earlier, deploying these systems across a large enterprise can cost hundred of thousands of dollars in up-front licensing cost, and usually requires a small (or sometimes large  ;-) army of IT personnel and consultants to customize and integrate it with with the organization's other systems and data. The time, staff and budget requirements of a deployment of this magnitude represents a significant risk for an organization of any size, and often puts such software out of reach of smaller organizations that would otherwise be able to derive great benefits  from the applications.

The subscription-based delivery model of SaaS applications changes this whole mindset. SaaS applications don't require the deployment of a large infrastructure at the customer's location, which eliminates or drastically reduces the upfront commitment of resources. With no significant initial investment to amortize, an enterprise that deploys a Saas application that turns out to produce disappointing results can walk away and pursue a different direction, without having to abandon an expensive on-premise infrastructure.

Does Saas eliminate the need for an IT organization?

One (wrong) conclusion that some people draw from the deployment of SaaS-based application is that it eliminates, or drastically reduces the need for an internal IT staff of an tenant's organization. Indeed with SaaS, the job of deploying  an application and keeping it running from day to day - testing and installing patches, managing upgrades, monitoring performance, ensuring high availability, and so forth - is handled by the provider.

But, rather than eliminating the need for an internal IT staff, by transferring the responsibility for the above-mentioned "overhead" activities to a third-party, the IT department can focus more on high-value activities that align with and support the business goals of the enterprise. Instead of being primarily reactive and operations-focused, the chief information offices (CIO) and IT staff can more efficiently function as technology strategists to the rest of the company, working with the business units to understand their business needs and advise them on how to best use technology to accomplish their objectives.

So, instead of being made obsolete by SaaS, the IT department has an opportunity to contribute to the success of the enterprise more directly than ever before. For example, based upon business requirements, the IT staff could create "composite applications" and "executive dashboards", which act as integrators and collectors of the valuable data contained in the different Saas applications. A successful service-centric IT organization can directly produce more value for the business by providing services that draw from both internal and external sources and align closely with the overall business goals of the organization.

The Three Attributes of a Single Instance Multi-Tenant Architecture

From an application's architect point of view, there are three key differentiators that separate a well-design SaaS application from a poorly designed one. A well-designed SaaS application is:

1. Scaleable
2. Multi-Tenant-efficient
3. Configurable

A graphical representation of these three important differentiators is shown below:

Scaling the application means achieving maximum  concurrently and using application resources more efficiently - for example, optimizing locking duration, statelessness, efficiently leveraging sharing pooled resources  such as thread and network connections, caching frequently used reference data, and partitioning large databases. Note that I am not stating here that a standard single-tenant application should NOT be designed  in a scaleable manner, but because of the resource-intensive nature of a multi-tenant application, these requirements are more amplified, and achieving maximum scalability is therefore even more important.

Multi-tenancy may be the most significant paradigm shift that an architect accustomed to designing isolated, single-tenant applications has to make. For example, when a user at one company accesses customer information by using a CRM application (e.g. SalesForce.com), service, the application instance that the user connects to may be accommodating users from dozens, or even hundreds, of other companies - all completely unbeknownst to any of the users. This requires an architecture that maximizes the sharing of resources across tenants, but in such a manner that the application is still able to differentiate data belonging to different customers. For example, in a multi-tenant environment, it might be prohibitive to allocate an expensive resource, such as a WIN32 process to a single tenant (we will have more to say about this topic in the next section). Also, the architect needs to ensure 100% data privacy for each tenant while maintaining an acceptable performance for each tenant.

Of course, if a single application instance on a single server has to accommodate users from several different companies at once, you can't simply write custom code to customize the end-user experience; anything you do to customize the application for one customer will change the application for other customers as well. Instead of customizing the application in the traditional sense, each customer uses metadata to configure the way the application appears and behaves for its users. The challenge for the SaaS Architect is to ensure that the task of configuring applications is simple and easy for the customers, without incurring extra development or operation costs for each configuration. Note that configuration is this context can apply to a number of different areas:

1. Allowing tenants to add custom fields or tables to the database.
2. Providing a mechanism by which a tenant can customize the user interface and the look and feel of the application (for example, in the case of a web application, the user should be able to upload custom a style sheet for the application).
3. Allowing tenants to customize the business process workflows, by including some type of process editor in the tenant's administrator user interface.

Saas as a Service-Oriented Architecture Maturity Model

In the above sections, we have discussed the business and technical benefits of Saas. The biggest impact of SaaS tough might be the fact that SaaS finally provides an organization with the right incentives for moving IT towards  a services-centric model or in other words: a "real", applied SOA strategy with concrete,  tangible benefits.

Broadly speaking, SaaS service maturity can be expressed using a model with four distinct levels. Each level is distinguished from the previous one by the addition of one of the three attributes listed above. A graphical level with the different maturity levels is shown below:

Maturity Level I : Ad Hoc / Custom

The first level of maturity is similar to the traditional application service provider (ASP) model of software delivery, dating back to the 1990s. At this level, each customer (or tenant) has its own customized version of the hosted application, and runs its own instance of the application on the hosts's servers. Architecturally, software at this maturity level is very similar to traditionally-sold LOB (Line-of-Business) software, in that different clients within the organization connect to a single instance running on behalf of its other customers. Note that at this maturity level, application maintenance tends to be complication and time consuming. If a common bug is found, the fix has to be integrated into the code base for each different instance, and each instance has to be individually tested, validated and re-deployed.

Typically, traditional client-server applications can be moved to a SaaS model at the first level of maturity, with relatively little development effort, and without re-architecting the entire system from the ground up. Although this level offers few of the benefits of a fully mature Saas solution, it does allow vendors to reduce cost by consolidating server hardware and administration.

Maturity Level II: Configurable

At the second level of maturity, the vendor hosts a separate instance of the application for each customer (of tenant). Whereas in the first level each instance is individually customized for the tenant, at this level all instances use the same code implementation, and the vendor meets customer's needs by providing detailed configuration options that allow the customer to change how the application looks and behaves to its users. Despite being identical to one another at the code level, each instance remains wholly isolated from all others.

Moving to a single code base for all of a vendor's customers greatly reduces a Saas application's service requirements, because any changes made to the code base can be easily provided to  all of the vendor's customers at once, thereby eliminating the need to upgrade or slipstream individual customized instances. However, repositioning a traditional application as SaaS at the second maturity level can require significant more re-architecting than at the first level, if the application has been designed for individual customization rather than configuration metadata. Therefore, it is important to note that if an enterprise architect is architecting a new application, which might be initially be deployed as a single-tenant application, but has the potential to be migrated to a multi-tenant solution, it is recommended that you take a close look at the architecture of your application, and build in some of the core "enabling" building blocks that would enable you to move to a multi-tenant solution without re-architecting the entire application.

Similarly to the first maturity level, the second level requires that the vendor provide sufficient hardware and storage to support a potentially large number of application instances running concurrently.

Maturity Level III: Configurable, Multi-Tenant-Efficient

At the third level of maturity, the vendor runs a single instance that serves every customer, with configurable metadata providing a unique user's experience and feature set for each one. Authorization and security policies ensure that each customer's data is kept separate from that of other customers; and, from the end user's perspective, there is no indication that the application instance is being shared among multiple tenants.

This approach eliminates the need to provide server space for as many instances as the vendor has customers, allowing for more more efficient use of computing resources that the second level, which translates directly into lower costs. A significant disadvantage of this approach is that the scalability of the application is limited. Unless partitioning is used to manage database performance, the application can be scaled only by moving it to a more powerful server (scaling up), until diminishing returns make it impossible to add more power cost-effectively.

Maturity Level IV: Scalable, Configurable, Multi-Tenant-Efficient

At the fourth and final level of maturity, the vendor hosts multiple customers on a load-balanced farm of identical instances, with each customer's kept separate, and with configurable metadata providing a unique user experience and feature set for each customer. A SaaS system is scalable to an arbitrarily large number of customers, because the number of servers and instances of the back end can be increased or decreased as necessarily to match demand, without requiring additional re-architecting of the application, and changes or fixes can be rolled out to thousands of tenants as easily as a single tenant.

Selecting the Appropriate Maturity Level for your Application

Given all of the above information, you might wonder what maturity level you should target for your SaaS application? Intuitively, one might expect the fourth level to be the ultimate goal for any SaaS application, but this is definitely not always the case. Indeed, it might be more helpful to think of SaaS maturity as a continuum between isolated data and code on one end, as shared data and code on the other, as is shown on the figure below:

Where your application should fall in this continuum depends on your business area, architectural experience and operational needs, and on customer considerations. As you'll be able to see event from this simple explanation, all of these considerations are interrelated to some degree.

1. Business Model. Does an isolated approach make financial sense? Forsaking the economic and management benefits of a shared approach means offering your application to the consumer at a higher cost; however; under some circumstances, it may be worth it to meet other needs. In addition, customers might have strong legal (e.g HIPPAA for the health care industry) or cultural resistance to an architectural model in which multiple tenants share access to an application, even if you can demonstrated that it does not place confidential data at risk. Ultimately, of course, you'll need a business model that shows how your  application can make money at whichever maturity level you've targeted.
2. Architectural Model. Can your application be made  to run in a single local instance, without a complete architectural overhaul? if you seeking to move a desktop-based or traditional client-server or distributed application to an Internet-based delivery system, then its architecture might be fundamentally incompatible with a single-instance, metadata-centric approach, and you may determine that it will never make financial sense to invest the development effort necessary to transform it into a fully mature SaaS application. If you are designing and building a net-native application from the ground up, you will probably have a lot more freedom to take a single-instance approach.
3. Operational Model. Can you guarantee your service level agreements (SLAs) without isolation? You (and your legal department ;-) should carefully example the obligations imposed by any existing SLAs that you have with your existing customers, with regard to considerations such as downtime, support operations, and disaster recovery, and determine whether these obligations can be met under an application architecture in which multiple unrelated customers share access to a single application instance.

Conclusion

This concludes our high-level overview of SaaS. Obviously this is a very rich topic, but I hope that in the course of reading this article, you got a good basic understanding of what SaaS is, what its benefits are to an organization, the different maturity levels, and how to select a maturity level that is appropriate for your organization.

During the course of the coming week, I will be following up on this article with a more technical paper, which will describe some of the architectural challenges associated with building a single-instance, multi-tenant efficient and scalable application, so stay tuned!

Introduction

In my previous post I introduced Team Foundation Server's customization and extensibility features. I mentioned how the TFS core functionality is exposed by means of the Team Foundation Core Services (TFCS), which are essentially a set of five different Web Services. The functionality of these web services is encapsulated by the Team Foundation Server Object Model (TFSOM), which is what Microsoft recommends you use to implement a Team Foundation extension.

The Team Foundation Object Model is implemented in a series of assemblies, some of which we will take a detailed look at in this post. We will explore some of the core objects in this object model, and we will build a small wrapper class library, called FooTheory.TFS.TFSConnectivity. We will use this class library as our foundation to create the following custom extensibility clients:

1. A PowerShell PSDrive provider called FooTheory.TFS.TFSProvider. This provider will allows us to treat the TFS repository as a logical drive, so we will be able to use our favorite commands such as  dir, cd, copy, new (or their "offical"cmdlet names: Get-ChildItems, Set-Location, Copy-Item, New-Item)  to manipulate TFS artifacts such as Work Items, Stored Queries, Team Projects, Reports etc.
2. An event listener called FooTheory.TFS.EventListener. This is a WCF-based service, which can subscribe to TFS events, and receive these events through HTTP. The other clients, such as the PSDrive provider and the WPF client will be able to host this service, enabling them to be informed of any changes to their TFS artifacts of interest.
3. A rich WPF-based client called FooTheory.TFS.TFSExplorer. This client will take full advantage of the graphical expressiveness of Windows Presentation foundation to allow the user to navigate the complete TFS object model, including Work Items, Reports, Source Code trees etc.

An overview of the assemblies is shown in the figure below:

To start out, we will take a look at the TFSOM client assemblies which need to reference in our class library in order to implement our functionality.

TFSOM Client Assemblies

To leverage the functionality of the TFSOM, we need references to the following assemblies:

If you have Team Explorer installed on your client, these assemblies should be located in the C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies location, together with all of the TFSOM assemblies. (If you have Orcas installed, you need to substitute "Visual Studio 9" for "Visual Studio 8" in the above path). Also, if you have Team Foundation Server 2008 installed, these assemblies will be the client assemblies for the Team TFS 2008, but these assemblies are 100% backwards compatible with TFS 2005 object model, so you can use the same assemblies to access both TFS versions.

In the next section we will take a look at the high-level design of our FooTheory.TFs.TFSConnnectivity custom class library.

The FooTheory.TFS.TFSConnectivity assembly

The main goal behind creating this assembly is to provide a simple API that provides access to some of the core functionality that we need to implement, for example:

• Connecting and authenticating to a Team Foundation Server Instance.
• Retrieving a list of all Team Projects .
• Retrieve the Process Template of a given Team Project instance.
• Retrieving the Microsoft Project mapping properties for a Team Project instance.
• Subscribing and unsubscribing to a TFS Event Type.
• Retrieving all stored Work Item Queries for a given project.
• Executing a stored query, retrieving the resulting Work Items in the process.

The main class in our assembly is the FooTheory.TFS.TFSConnectivity.TFSServer class. An overview class diagram of this class and its main relationships is shown in the figure below:

Note: The FooTheory.TFS.TFSConnectivity.dll assembly uses the Patterns & Practices Enterprise Library 3.1. Specifically, it leverages the Logging and Exception Handling blocks to enhance the robustness of the assembly. We'll have a few words to say about our usage of the Enterprise library at the end of this post.

Next, we will take a more detailed look at the class that is at the center of the Team Foundation Server Object Model, the Microsoft.TeamFoundation.Client.TeamFoundationServer class.

After we have a better understanding of this class, we will use our understanding of the TeamFoundationServer class to design and implement our TFSConnectivity assembly.

The TeamFoundationServer Object

The TeamFoundationServer object is the core object of the TFSOM. It contains the basic attributes of the Team Foundation Server that it is connected to. In addition, It provides access to the Team Foundation Core Services, as well as any other services that have been registered with Team Foundation Server, such as:

• Version Control
• Work Item Tracking
• Project Templates
• Event subscriptions

The TeamFoundationServer object is defined in the Microsoft.TeamFoundation.Client namespace, so you will need to have a reference to the Microsoft.TeamFoundation.Client.dll if your want to use it in your code.

You can get a reference to a TeamFoundationServer instance in one of two ways:

• You can use the TeamFoundationFactory.GetServer() method. This is the most commonly used strategy to create a TeamFoundationServer instance. The GetServer() method of the TeamFoundationServerFactory class supports the following overloads:
  • You can only pass in the name of the server:

public static TeamFoundationServer GetServer(string name);

You can pass in the name of the TFS server and an object that implements the ICredentialsProvider  interface:

public static TeamFoundationServer GetServer(
            string name, ICredentialsProvider fallbackCredentialsProvider)

• You can also choose to call the TeamFoundationServer constructor directly. The different overloads are shown below:

public TeamFoundationServer(String name);
public TeamFoundationServer(
    string name,
    ICredentials credentials);
public TeamFoundationServer(
    string name,
    ICredentialsProvider credentialsProvider);
public TeamFoundationServer(
    string name,
    ICredentials credentials,
    ICredentialsProvider credentialsProvider);

The main difference between these two approaches is the fact that the Factory method caches the instance by the Uri of the server. The first time you call GetServer() with a specific server Uri, the factory will create a new TeamFoundationServer instance, add  it to cache (indexed by the server Uri) and return it to the caller. The next time you call GetServer() with the same server Uri (regardless of the ICredentialsProvider that is passed in), you will get exactly the same Uri instance. 

The TeamFoundationServerFactory class was initially created to support the behavior of Team Explorer. In Team Explorer, you will notice that you will get the authentication dialog the first time your bring up the Explorer, but afterwards you are no longer prompted although behind the scenes, the team explorer code might have to call the Factory's GetServer() method multiple times. But, since the code is calling the GetServer() method with the same server Uri each time, you will NOT be prompted again.

When you call the TeamFoundationServer constructor directly, you are obviously creating a new instance each time.

As far as passing in credentials, you have basically two choices:

• You can pass in a reference to an object that implements the ICredentialsProvider interface. While you can pass in any instance that implements this interface, in a UI-based scenario you probably would want to use Microsoft.TeamFoundation.Client.UICredentialsProvider class. When you create a new instance of this class, it will automatically pop up the authentication dialog. So the following code:

                // Create an instance of the UICredentialsProvider.
                // Note that creating this instance will automatically
                // popup the TFS authentication dialog
                ICredentialsProvider provider = new UICredentialsProvider();

Will popup the following dialog:

• You can pass in any NetworkCredential instance, for example:

                // Create our Network Credentials
                NetworkCredential credentials = new NetworkCredential(TFSUserName, TFSPassword);

Implementation in the FooTheory.TFS.TFSConnectivity Library

Connecting to Team Foundation Server

The FooTheory.TFS.TFSConnectivity.TFSServer class contains an embedded instance of a TeamFoundationServer instance in the m_tfsServer field, and exposed through the TeamFoundationServer property:

        // This is our Team Foundation Server instance
        private TeamFoundationServer m_tfsServer;

        /// <summary>
        /// This read-only property returns our current
        /// TeamFoundationServer instance
        /// </summary>
        public TeamFoundationServer TeamFoundatonServer
        {
            get { return m_tfsServer; }
        }

The TFSServer class offers three overloads for the Connect() method, as shown below:

        public bool Connect(string serverName)

        public bool Connect(string serverName, NetworkCredential credentials)

        public bool Connect(string serverName, ICredentialsProvider credentialsProvider)

The first overload does not take any type of credentials, and will use the UICredentialsProvider to pop up the connect dialog shown in the previous section. The other overloads either take a NetworkCredential or an object that implements ICredentialsProvider. After successfully connecting and authenticating, a reference to the TeamFoundationServer object instance is stored in the m_tfsServer field.

The full implementations of the different Connect methods is shown below:

   1: /// <summary>

   2: /// This overload of the Connect Method will prompt

   3: /// the users for a set of credentials, by means of the

   4: /// UICredentialsProvider. It then calls our "generic"

   5: /// connect method, which takes an ICredentialsProvider

   6: /// as its second argument

   7: /// </summary>

   8: /// <param name="serverName"></param>

   9: /// <returns></returns>

  10: public bool Connect(string serverName)

  11: {

  12:     using (new Tracer(LoggingCategoryConstant.TFS_CONNECTIVITY_CONNECT))

  13:     {

  14:         //  Create an instance of our Creditials by means of the

  15:         //  UICredentialsProvider. Since we are using the 

  16:         //  UICredentialsProvider here, this  method will popup 

  17:         //  an authentication dialog for us and create a set 

  18:         //  of Credentials for us, which are passed to our other 

  19:         //  Connect overload that takes a generic 

  20:         //  ICredentialsProvider argument

  21:         // Create an instance of the UICredentialsProvider.

  22:         // Note that creating this instance will automatically

  23:         // popup the TFS authentication dialog

  24:         ICredentialsProvider provider = new UICredentialsProvider();

  25:         return Connect(serverName, new UICredentialsProvider());

  26:     }

  27: } // method Connect

  28:  

  29: /// <summary>

  30: /// This overload of the Connect Method takes a NetworkCredential.

  31: /// This method is typically used from a server client, which does

  32: /// not have the possibility to prompt the client for a username/password

  33: /// </summary>

  34: /// <param name="serverName"></param>

  35: /// <param name="credentials"></param>

  36: /// <returns></returns>

  37: public bool Connect(string serverName, NetworkCredential credentials)

  38: {

  39:     using (new Tracer(LoggingCategoryConstant.TFS_CONNECTIVITY_CONNECT))

  40:     {

  41:         //  Log our parameters

  42:         addLogEntry(

  43:             LoggingCategoryConstant.TFS_CONNECTIVITY_TRACE,

  44:             "Connect method with ServerName and Credentials Called",

  45:             EventIdentifier.ConnectToTfs,

  46:             EventPriority.StandardPriority,

  47:             TraceEventType.Verbose,

  48:             createCredentialsContextInfo(serverName, credentials));

  49:  

  50:         // Create our Network Credentials

  51:         NetworkCredential credentials = new NetworkCredential(TFSUserName, TFSPassword);

  52:  

  53:         // Here, we can pass the network credentials directly to the

  54:         // TeamFoundationServer constructor, so we have no need to used

  55:         // the factory here. You cannot use the factory methods with any 

  56:         // object instance that implements ICreditials, but it is supported

  57:         // in the standard constructor, as we are using it here

  58:         try

  59:         {

  60:             m_tfsServer = new TeamFoundationServer(serverName, credentials);

  61:             m_tfsServer.Authenticate();

  62:  

  63:             //  If we succcessfully authenticated, we can go ahead an extract

  64:             //  some reference to some commonly used service interfaces

  65:             if (m_tfsServer.HasAuthenticated)

  66:             {

  67:                 extractServiceInterfaces();

  68:             }

  69:         }

  70:         catch (Exception ex)

  71:         {

  72:             bool rethrow = ExceptionPolicy.HandleException(ex, ExceptionPolicyName.EXTERNAL_LIBRARY_POLICY);

  73:             if (rethrow)

  74:             {

  75:                 throw;

  76:             }

  77:         }

  78:     }

  79:  

  80:     // Return the authentication status

  81:     return m_tfsServer.HasAuthenticated;

  82:  

  83: } // method Conection

  84:  

  85: /// <summary>

  86: /// This is our "generic" Connect method. This method takes any object 

  87: /// instance that implements ICredentialsProvider. It uses the 

  88: /// TeamFoundationServerFactory GetServer method to retrieve the 

  89: /// TeamFoundationServer implementation

  90: /// </summary>

  91: /// <param name="serverName"></param>

  92: /// <param name="credentialsProvider"></param>

  93: /// <returns></returns>

  94: public bool Connect(string serverName, ICredentialsProvider credentialsProvider)

  95: {

  96:     using (new Tracer(LoggingCategoryConstant.TFS_CONNECTIVITY_CONNECT))

  97:     {

  98:         //  Log our parameters

  99:         addLogEntry(

 100:             LoggingCategoryConstant.TFS_CONNECTIVITY_TRACE,

 101:             "Connect method with ServerName and Credentials Called",

 102:             EventIdentifier.ConnectToTfs,

 103:             EventPriority.StandardPriority,

 104:             TraceEventType.Verbose,

 105:             createCredentialsContextInfo(serverName, credentialsProvider));

 106:  

 107:         // Create a TeamFoundationServer instance through the factory,

 108:         // and authenticate

 109:         try

 110:         {

 111:             m_tfsServer =

 112:                 TeamFoundationServerFactory.GetServer(serverName, credentialsProvider);

 113:             m_tfsServer.Authenticate();

 114:  

 115:             //  If we succcessfully authenticated, we can go ahead an extract

 116:             //  some reference to some commonly used service interfaces

 117:             if (m_tfsServer.HasAuthenticated)

 118:             {

 119:                 extractServiceInterfaces();

 120:             }

 121:         }

 122:         catch (Exception ex)

 123:         {

 124:             bool rethrow = ExceptionPolicy.HandleException(ex, ExceptionPolicyName.EXTERNAL_LIBRARY_POLICY);

 125:             if (rethrow)

 126:             {

 127:                 throw;

 128:             }

 129:         }

 130:  

 131:     }

 132:     // Return the authentication status

 133:     return m_tfsServer.HasAuthenticated;

 134:  

 135: } // method Connect

The first connect method uses the UICredentialsProvider class to retrieve the credentials from the user, and then calls the third overload of the Connect() method, which uses the TeamFoundationFactory.GetServer() method to connect to the Server, and the TeamFoundationServer's Authenticate() method to authenticate with the server.

The second Connect() overload accepts a NetworkCredential. Since the factory does not have an overload that takes an object that implement ICredentials, this method directly calls the TeamFoundationServer constructor, which does have an overload that accepts an ICredentials object instance.

Also, note that after successfully connecting and authenticating to the TFS server, all overloads call the extractServiceInterfaces() private method. This method extracts all relevant service interfaces from the TeamFoundationServer instance, and cache them as fields of the class, as is shown below:

        /// <summary>
        /// This method extract our most commonly used service interface
        /// from the TFS Server instance, and caches them in our private
        /// fields
        /// </summary>
        private void extractServiceInterfaces()
        {
            if (m_tfsServer != null)
            {
                m_commonStructureSvc = (ICommonStructureService)m_tfsServer.GetService(typeof(ICommonStructureService));
                m_workItemStore = (WorkItemStore)m_tfsServer.GetService(typeof(WorkItemStore));
                m_processTemplateSvc = (IProcessTemplates)m_tfsServer.GetService(typeof(IProcessTemplates));
                m_eventService = (IEventService)m_tfsServer.GetService(typeof(IEventService));
            }
        } // method extractServiceInterfaces

Note how we use the GetService(typeof(xxx)) pattern to retrieve each interface and/or object references (only the WorkItemStore is a object reference, all the others are interface references).

Retrieving Project Information

... coming soon ...

Introduction

A large number of developers are using Team Foundation Server for a variety of tasks such as Source Control, Bug Tracking, Requirements gathering, running nightly builds and managing the overall life cycle of a software development project.

Visual Studio Team System provides rich integration with Team Foundation Server through Team Explorer, which provides the basic connectivity to TFS. Users can use Source Control Management to manage their source code, manage Work Items, create Build types, run Unit Tests and Load Tests, run one of many SQL Server 2005 Reporting Services-based Team Reports etc.

In addition to Visual Studio, Office 2003 and Office 2007 also offer direct or indirect connectivity to Team Foundation Server. You can have Work Items show up as Tasks in Outlook 2007, you can export a Microsoft Project 2003 schedule to TFS, creating Work Items in the process, or you can export and import your Work Items to and from Excel 2007.

Even if you are working outside of the realm of the  .NET and the Microsoft development platform, you can use the TeamPrise plug-in for Eclipse to provide connectivity from your favorite Java development environment to Team Foundation Server.

What a lot of people don't realize is that Team Foundation Server also offers a powerful development object model, empowering the developer to create custom development tools, specialized clients, custom WebParts for a Team Project SharePoint portal and so on. Team Foundation Server was developed from the ground up with extensibility in mind. Indeed, TFS itself is build upon the Team Foundation Core Services, which are a set of Web Services used by Team Foundation Server itself to implement the different functional areas for Team Foundation Server.

As a developer, you will seldom have a reason to directly use the Team Foundation Core services. Indeed, Microsoft created the Team Foundation Server Object Model to increase the productivity of the TFS developer. The classes in this object model interact with the Web Services of the Team Foundation Server Core Services and make it easy for you to extend the functionality of Team Foundation Server in one of the following ways:

1. Create a custom TFS command-line tool.
2. Develop custom PowerShell cmdlets or PowerShell PSDrive providers that allow you to directly navigate the TFS repository.
3. Develop a SmartClient or Web client for Team Foundation Server.
4. Develop a custom check-on policy for your development team
5. Create a  plug-in for a third-party development tool.
6. Create a custom SharePoint WebPart for your Team Project SharePoint portal.
7. Implement custom Contiguous Integration (CI) tools.
8. Create new Team Foundation build tasks.

In article two of this series we will create a custom class library, which will directly interface with the Team Foundation Object Model. We will create a set of Units Tests for the classes and methods in this library, and we will create a simple console client to navigate the object model.

In part two of this article series, we will create a custom PowerShell PSDrive provider for TFS. This provider will allow us to navigate through the different artifacts in the TFS database, as if they were files and directories in the file system. We will be able to use the standard dir, cd, type etc. alias commands, together with the standard Get-ChildItem, Set-Location, Get-Content, etc..  PowerShell cmdlets to navigate the TFS structure. The implementation of this provider will leverage the class library that we create in this post.

In the final post of this series, we will build a custom WPF client for Team Foundation Server, which will allow us to browse, edit and create Team Projects and Work Items. In the course of implementing this client, we will have an opportunity to dive into some interesting WPF features, such as data binding, data templates and control templates. We will also leverage the TFS eventing service to keep our UI synchronized with the TFS repository at all times.

Extensibility versus Customization

Before we dive into the details of the TFS extensibility model, I would like to clarify some terms that tend to lead to confusion in the community. These terms are:

1. Customization. Customization involves modifying your environment, such as Visual Studio Team System or Team Foundation Sever, using the tools that are provided to you by that environment. It is important to note that there is no real coding involved in customization. Indeed, before you go fire up Visual Studio to write custom code to perform a certain task, I would encourage you to check out the Team Foundation Administrator's Guide in the Team Foundation Help documentation. In a lot of cases, you will notice that you can customize your TFS environment by using the existing toolset, or by simply modifying one or more of XML files. Also, I would encourage you to download the Team Foundation Server Power Toys, which extend the reach of the existing TFS toolset even further. Customization might include:
   • Modifying a Process Template.
   • Creating a new Work Item Type.
   • Customizing Team Foundation Build.
   • Modifying the Project Portal Template
   • Turning check-in policies on or off.
2. Extensibility. Extensibility is concerned with adding new functionality to your environment, which typically does involve coding. You can extend a Team Foundation Server installation by writing code to access the Team Foundation Server object model. This way you can build add-ins and integration components that round out the Team Foundation Server product line. TFS Extensions are typically created by internal IT departments or third-party independent software vendors (ISV's). Some extensibility examples are:
   • An adapter that integrates an existing toolset with Team Foundation Server. A good example is the TeamPrise plug-in for the Java Eclipse development environment.
   • A Customized Team Foundation Server development solution for a vertical industry (for example Healthcare-specific process templates, new Work Item Types with Healthcare-specific workflows etc.).
   • Custom explorer clients (like the one we will be developing in part II of this series).

Our focus of this series is on extensibility. I do have some ideas for future posts on TFS customization, so make sure you subscribe to our RSS feed!

The Team Foundation Core Services

The Team Foundation Core Services (TFCS) are a set of services running on the Application Tier of TFS, which allow us to access the different aspects of Team Foundation Server, including administration, security and events. TFCS enable customers and Microsoft Partners to extend Team Foundation functionality by developing Team System extensions and adapters to integrate third-party tools with Team System.

TFCS is made up our of five services:

1. Classification Service. This service provides access to Team Project information, and the different structures that make up a Team Project. In this article, we will mainly be focusing on this service.
2. Eventing Service. The Eventing Service allows Team Foundation Servers to communicate by raising events and listening for events. This is how tools like CruiseControl.Net are able to kick of a build in response to a check in. They use the Eventing Service to listen for these events, and react appropriately. We will use the Eventing service in our custom WCF client, to keep our displayed information synchronized with the TFS repository at all times.
3. Linking Service. This service provides the ability to link items together, such as Work Items and files that are version controlled. One interesting aspect of the Linking Service is that you can create a link between two artifacts that were created by different tools which have no knowledge of each other. This type of loose coupling is achieved by assigning Uri to each artifact in the TFS repository, and creating links between these Uri's.
4. Registration Service. The Registration Service allows a TFS component to discover other services registered on Team Foundation Server. Therefore, when you create a new TFS service, it should be registered on the service by using the Registration Service, so that it can be found and utilized by the other server components. You can compare the Registration Service to a UDDI registry for Web Services.
5. Security Service. The security service implements a unified security model used by Team Foundation Server to manage users, groups and permissions. If you create a custom TFS component, you have the ability to integrate your extension into this security model, and have your artifacts behave correctly within the TFS security model.

The above services are implemented as Web Services. Team Explorer and the different components of Team Foundation use the above services to implement their functionality. As I mentioned before, you will not often have a reason to call these web services directly. Instead, you will use  the Team Foundation Object Model to interact with these Core Services. This is what we will do to implement our Console, WCF and PowerShell clients.

An overview diagram is shown below:

In the next sections, we will take a more detailed look at the Classification Service. Future articles will take a look at the other services within the context of the overall article.

The Classification Service

 The classification service provides access to Team Project information, including the Team Project name and the Uri, as well as access to the areas and iterations of the project. The Area and Iterations sections of a Team Project form the structure of the project. This service allows you to view project information, and make changes to a project's structure. An example of using this service would be to populate the top nodes of a tree view that looks like the main tree view of team explorer, showing the names of all of the Team Projects on the server. From these nodes you could then use other services to create the other nodes in the tree (such as accessing the Work Items that belong to a Team Project, which is what we will be doing in our custom WCF client).

Some example of the services offered up by the Classification Servers are listed below:

• Creating a new Team Project.
• Delete an existing Team Project.
• Access the properties (name, Uri, process template Id and Project properties) of the Team Project.

The common pattern for accessing the object model is to first establish a connection to the server Once you have a connection to the server, you can query the server for the type of interface you would like to use. This model is not unlike the WF Runtime Model, where you can ask the Workflow Runtime for the different services, such as the persistence or threading service.

To access the Classificiation service, you query the server for the ICommonStructureService type, defined in the Microsoft.TeamFoundation.Server namespace in the Microsoft.TeamFoundation.Server.dll assembly.

In my next post, we will dive head-first into the Classification service, and the TeamFoundationServer object, which is at the root of the Team Foundation Object Model hierarchy.

The code for the article Writing Custom Cmdlets for Windows PowerShell is now available in the download section. The sample includes the code for the cmdlet itself, the unit tests, and a  class library with a sample pipeline input object.

Any feedback on either the article or the code is greatly appreciated!

Article Overview

In this article we will take a look at how to write a custom cmdlet for Windows PowerShell, which is the new command shell from Microsoft (formerly called "Monad"). Our cmdlet named "Get-Contact" will have the ability to retrieve contacts defined in Outlook 2007.

The full source code, inlcuding unit tests, installations scripts etc. can be found here.

The contents of this post is as follows:

1. In the first section, we will take a quick look at PowerShell. We look at the main drivers behind it's creation and what some of the key factors are that differentiate PowerShell from other well-known shells and scripting environments.
2. In the second section,we tell you how to download PowerShell if you don't have it installed yet (shame on you! ;-), and take a quick look at the PowerShell Extensions (PCX) library, which is an open-source project for writing custom PowerShell extensions.
3. Next we take a look at PowerShell cmdlets, we'll show some examples, and discuss Microsoft's naming standards.
4. Once we have a good understanding of cmdlets, we talk about the PowerShell SDK, and the PowerShell extension templates, which allow you to quickly get the "shell" (pun intended ;-) of a custom cmdlet up and running quickly.
5. Before we start the implementation of our cmdlet, I thought it would be a good idea to list the detailed requirements of our Get-Contact cmdlet, and lists its parameters and arguments.
6. Since we will be interacting with Microsoft Outlook 2007, we will make sure that you have the new Outlook 2007 Primary Interop Assembly installed. If not, I will walk you through the process of downloading and installing the PIA.
7. Once we have all these pre-requisites out of the way, we will finally be able to start writing the code of our cmdlet.
8. Finally, we'll write some scripts that test the functionality of our new cmdlet.

That quite a long list that we have here, so let's get rolling!

Microsoft PowerShell Overview

Windows PowerShell is the new command line and scripting language for Windows environments. While it has been designed and optimized for Windows, it is based on a rich shell heritage that stretches all the way back to the original Unix shells such as "csh", "ksh", and the more recent "bash" (the "Borne Again SHell"). Jeffrey Snover (the Windows PowerShell architect) and Bruce Payette (the lead designer of the language) both have a very extensive dynamic language background, and have been involved in the creation of a variety of command shells.

In this book "Powershell in Action" (which I highly recommend), Bruce Payette states that the overall goal of the PowerShell project was to provide the best shell scripting environment possible for Microsoft Windows.

I have yet to find somebody who really loved the old Windows Command line (cmd.exe). Maybe I should check some mental health institutions, because that's where some folks could have ended up if they tried to do some advanced scripting tasks with cmd.exe. All joking aside, as Bruce says in his book, the focus of Windows was always on the GUI and the average user, and not necessarily the computer professional and/or the systems administrator.

Initially, Microsoft created GUI tools (such as mmc.exe) to administer desktops and servers. But now that the power of the PC has increased, Windows is used in the corporate data center as well as on the user's desktop. In such a data center, the graphical point-and-click management approach that worked well for one machine does not quite scale. This limitation illustrates the need for a powerful, command-line based scripting environment.

The starting point for the PowerShell language was the grammar for the POSIX standard shell defined in IEEE specification 1003.2. One of the main enriching features that Microsoft added to this starting point was support for object-orientation. Indeed, PowerShell is the first fully object-oriented shell out there. I think that everybody is familiar with the typical "command pipeline" for command shells. In traditional scripting environment, only text is flowing through the pipeline from one command to the next. In PowerShell we have OBJECTS flowing between commands, so none of the semantics of the objects produced and consumed on either end of the pipeline are lost.

This also implies that PowerShell is fully compatible with the .NET type system. Actually, the .NET object model IS the object model in PowerShell, eliminating any impedance mismatch between the two. Because of this, PowerShell can leverage the intellectual capital captured in the vast universe of .NET code, including the .NET Base Class Library (BCL), and any other third-party .NET code. PowerShell also fully supports COM interoperability, ensuring us that we will be able to leverage any past investments in the COM and COM+ arena.

Installing PowerShell and the PCX Extensions

Windows Server 2008 will ship with PowerShell "in the box". You will also have PowerShell installed on any server that has a product installed that leverages PowerShell for it's administration implementation, such as:

• Exchange Server 2007.
• VMM (Virtual Machine Manager).
• Desktop Protection Manager.

Otherwise, you should first download PowerShell from this site. Running the installer is very straightforward, and only takes a couple of minutes. After the installation completes successfully, you can start an interactive PowerShell session as follows:

Start -> Programs -> Windows PowerShell

You also would want to download the Windows PowerShell 1.0 documentation pack.  It contains several documents ("Release Notes", a one page "Quick Reference" etc), but what I like the most is the "Windows PowerShell Primer", which is about 100 pages, and a good place to get started with PowerShell.

Next, I recommend that you download the PowerShell Community Extension (PCX) from CodePlex. This is a very useful set of cmdlets, providers, aliases, filters, functions and scripts that members in the beta-testing community had expressed interest in, but never made it into PowerShell v1.0. The functionality covers a wide number of areas, including:

• Compression (zip, tar etc.) functionality.
• A wide variety of XML extensions.
• Clipboard functionality.
• SMTP support.
• Terminal Session support.

PowerShell Cmdlets

A cmdlet is basically a lightweight command in the PowerShell language. I added the term "lightweight" here, because a cmdlet will NOT create a new process or AppDomain. Instead it  runs in the default AppDomain of the PowerShell process. When you compare a cmdlet to a "traditional shell command", such as a command in the Unix "C Shell (csh)", you will notice that they differ in the following ways:

• Cmdlets are instances of .NET classes, not stand-alone executables. This also implies that when we execute a cmdlet, we do not have to incur the overhead of process creation.
• Cmdlets are very easy to write in any .NET language.
• Cmdlets will typically not do their own parsing, error presentation or output formatting. Parsing, error presentation, and output formatting are handled by the Windows PowerShell runtime. This also guarantees consistency and ease of integration into new scripts. The advantages of this cannot be overstated. When you implement your custom cmdlet (as we will do shortly), you can use simple attributes to identify a property as a cmdlet argument, define whether it's a positional or named arguments, state that the parameter should support wildcards etc., without the need to do any extra programming.
• Cmdlets process input objects from the pipeline rather than from streams of text, and they typically deliver objects as output to the pipeline. This goes back to the fact that PowerShell is really a fully object-oriented shell.
• Cmdlets are record-oriented, processing a single object at a time. The PowerShell pipeline supports streaming, which improves performance, and limits user wait times, because the command on the right side of the pipeline can start processing the records produced by the left-side command as soon as they become available.

PowerShell uses a verb-noun pair for the names of cmdlets. For example, the Get-Process cmdlet built into PowerShell is used to retrieve a list of all processes that are current running on the host computer. The verb part ("Get") of the name identifies that action that the cmdlet performs. The noun part ("Process") of the name identifies the entity on which the action is performed (in this case, one or more Windows Processes).

Microsoft recommends that you follow the following naming rules when creating a new cmdlet:

1. When specifying the verb part of the name, it is strongly recommended that you use one of the predefined verb names provided by Windows PowerShell (as defined by the VerbsCommon enumeration). If you use these standards names, you will ensure consistency between the cmdlets that you create, and those provided by Microsoft, or third-parties, such as PCX.
2. Use only the present tense of a verb for a name. For example, use "Set" instead of "Setting".
3. To enforce consistency, do not use a synonym of an approved verb name. (or in other words, don't get creative ;-)!
4. Always use a simple, singular noun in cmdlet naming. For example, use "Get-Database" instead of "Get-Databases". While this might seem a bit strange at first it makes sense, because cmdlets might return either a single object or a collection, and you don't want to use different cmdlet names for both situations. So, Microsoft selected the singular name and used it consistently.
5. Use Pascal Casing for both verb and noun names.

The verb names page of the PowerShell SDK lists the common verbs defined in the VerbsCommon enumeration.

The PowerShell SDK

Ensuring that you have the SDK installed

The PowerShell SDK is an integral part of the Windows SDK. It consists out of a class library, documentation and tools which allow you to create:

• Custom cmdlets (which is the topic of this article).
• Custom PowerShell Providers (a PowerShell Provider allows you to navigate or browse a set of stored data, using a consistent set of cmdlets such as Get-Item, Set-Item, Copy-Item, Get-ChildItem etc.). This is something we might touch on in a future article.
• Custom PowerShell hosts. Your application can act as a host to PowerShell, and invoke command pipelines directly from the host. This is the approach used by the admin UI of applications such as Exchange 2007.
• Create custom PowerShell Runspaces. Runspaces provide the mechanisms that hosting application use to execute pipelines in a well-constructed, consistent manner, and are often used by custom PowerShell hosts.

If you have downloaded the Windows SDK, you will have the PowerShell SDK installed. If you don't have the Windows SDK installed, well, what are you waiting for? ;-). You can download the Windows SDK for Windows Vista and the .NET framework 3.0 from this location.

After installation, you can find a large number of samples in the directory (if you used the standard installation directory):

C:\Program Files\Microsoft SDKs\Windows\v6.0\Samples\SysMgmt\WindowsPowerShell

Examples for each of the categories of custom objects are included in this directory.

The Windows PowerShell Extension Templates for Visual Studio

These Visual Studio templates add a C# and VB.NET project template for building Windows PowerShell cmdlets and providers. Using these templates will save you a lot of startup time, they will add the correct references to your project, create a skeleton for your custom cmdlet or provider, and add a generated PSSnapIn class for you to your project (we will talk more about PSSnapIn classes in the next section). After you download and unzip the files, you will get a .vsi file for C# and VB.NET, as is shown below:

Run the template of your choice, for example the C# template. You will see the following selection screen:

Leave all options selected. After you click Next, you will get a warning dialog, informing you that the content of the install package is not signed:

Click Yes, and the install will continue. In the next screen, click Finish to the install the template:

The install will complete and we'll be ready to start coding. Click Close to close the install dialog box:

After the installation has been completed, you will now have a PowerShell option in the New Project dialog:

The Get-Contact cmdlet - Detailed Requirements

The Get-Contact cmdlet should retrieve the list of contact defined in the local instance of Microsoft Outlook 2007. If no parameters are specified, it should return the entire list of contacts. The instances returned should be of type Microsoft.Office.Interop.Outlook.ContactItem. This type is defined in the Microsoft Outlook 2007 PIA.

The user should be able to restrict the list of contacts returned by last name. A LastName parameter should be provided for this purpose. This parameter should be able to accept a single argument, or a list of arguments, as shown in the examples below:

13# Get-Contact -LastName Hensel | Format-Table FirstName, LastName, CompanyName -AutoSize

FirstName LastName CompanyName
--------- -------- -----------
Brian     Hensel   Statêra


14# Get-Contact -LastName Hensel, Banker, Felker | Format-Table FirstName, LastName, CompanyName -AutoSize

FirstName LastName CompanyName
--------- -------- -----------
Brian     Hensel   Statêra
Chris     Banker   Statêra
Donn      Felker   Statêra

The LastName parameter should be positional and optional, so the previous command could also be written as follows:

15# Get-Contact Hensel, Banker, Felker | Format-Table FirstName, LastName, CompanyName -AutoSize

FirstName LastName CompanyName
--------- -------- -----------
Brian     Hensel   Statêra
Chris     Banker   Statêra
Donn      Felker   Statêra

 The LastName parameter should also be able to accept wildcards. The syntax should be in accordance to the PowerShell wildcard specification. Below are a number of examples:

24# Get-Contact B* | Format-Table FirstName, LastName, Email1Address -AutoSize

FirstName LastName Email1Address
--------- -------- -------------
Eric      Boocock  /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=eboocock
Malcolm   Boswell  /o=STATORG/ou=FIRST ADMINISTRATIVE GROUP/cn=RECIPIENTS/cn=MBOSWELL
Chris     Banker   /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=cbanker
Tony      Blodgett /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=tblodgett


25# Get-Contact [CE]* | Format-Table FirstName, LastName, Email1Address -AutoSize

FirstName LastName Email1Address
--------- -------- -------------
Jeremy    Campbell /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=jcampbell
Erl       Egestad  /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=eegestad
Julie     Clint    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=jclint
Mike      Citro    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=mcitro


26# Get-Contact [m-x]* | Format-Table FirstName, LastName, Email1Address -AutoSize

FirstName LastName  Email1Address
--------- --------  -------------
Matthew   Ortiz     /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=mortiz
Kyle      Sanford   ksanford@statera.com
Wayne     Macdonald /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=wmacdonald
Bob       Mills     Bob_Mills@isagenix.net
Pete      Miller    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=pmiller
Jon       Robinson  jRobinson@statera.com

From the above list, you also notice that the wildcard expansion should be case-insensitive.

The Outlook 2007 Primary Interop Assembly

The Microsoft Office Outlook Primary Interop Assembly (PIA) allows a developer to write managed applications for Microsoft Office Outlook 2007. The name of the assembly is Microsoft.Office.Interop.Outlook.dll. To check if you already have this assembly installed in the GAC, open up File Explorer, and navigate to %WINDIR\Assembly, and check if the assembly is already loaded in the GAC:

Or, for the hard-core PowerShell fans, you can use the following command:

51# dir $env:windir\assembly\gac\Microsoft.Office.Interop.Outlook


    Directory: Microsoft.PowerShell.Core\FileSystem::C:\Windows\assembly\gac\Microsoft.Office.Interop.Outlook


Mode           LastWriteTime       Length Name
----           -------------       ------ ----
d----    12/16/2006  5:58 AM        <DIR> 12.0.0.0__71e9bce111e9429c

(note how we use the $env:windir environment variable to refer to the windows directory, this is basically equivalent to using %WINDIR% in cmd.exe, but of cour much "cooler" ;-).

If you DO NOT see see the Outlook PIA using the mechanisms illustrated above, follow the instructions listed below to install it (note: these steps are for Windows Vista, but should be very similar for Windows XP):

• Bring up Control Panel. In Control Panel, select Programs. In the next screen, select Programs and Features, and from the list, right-click Microsoft Office Outlook 2007, and click Change as is shown below:

• In the next dialog, click Add or Remove Features, and click Continue.
• In the next screen, make sure that you select ".NET Programmability Support" as shown below:

• Click Continue. The Configuration Progress dialog will show the installation progress.
• Click the Close button in the Installation Completed screen.

Implementing our Cmdlet

Creating the Cmdlet skeleton with the Visual Studio Template

To create the skeleton for your custom cmdlet, follow these steps:

• Start Visual Studio, and select File | New Project. Select a location, and in the Project Types tree, select Windows PowerShell. Pick a name for your project, in my case I selected the name FooTheory.GetContactCmdlet as is shown below:

• Click OK to create the project. The template will create a class library with the appropriate references added, and a snap-in class, as shown in the screen shot below:

The PSSnapin.cs file implements a PowerShell Snap-in. Windows PowerShell snap-ins provide a mechanism for registering cmdlets and/or providers with the shell, thus extending the functionality of the shell. A Windows PowerShell snap-in can register all the cmdlets and providers in a single assembly, or it can register a specific list of cmdlets and providers. In our case, we will only have our one Get-Contact cmdlet in assembly, so we will only need to register that specific cmdlet.

Since I called my assembly Footheory.GetContactCmdlet, the template named my snap-in class FooTheory.GetContactCmdletSnapIn, which is of course an invalid class name, so I changed it to just GetContactCmdletSnapIn, and I renamed its source file from PSSnapIn.cs to GetContactCmdletSnapIn.cs. The methods of the Snap-in provide information to the installer, such as the Name of the assembly, the Vendor etc. The wizard does a pretty good job of implementing these methods, so you only need to provide some tweaks, like filling in the name of the Vendor etc. The finalized version of my Snap-in class is shown below:

 1 using System;
 2 using System.Management.Automation;
 3 using System.ComponentModel;
 4 
 5 namespace FooTheory.GetContactCmdlet
 6 {
 7     /// <summary>
 8     /// This is our Snap-In class implementation. This class is
 9     /// used by the Installer, and by the Add-PSSnapIn command
10     /// when we register our cmdlet with the PowerShell runtime
11     /// </summary>
12     [RunInstaller(true)]
13     public class GetContactCmdletSnapIn : PSSnapIn
14     {
15         public override string Name
16         {
17             get { return "FooTheory.GetContactCmdlet"; }
18         }
19         public override string Vendor
20         {
21             get { return "www.FooTheory.com"; }
22         }
23         public override string VendorResource
24         {
25             get { return "FooTheory.GetContactCmdlet,"; }
26         }
27         public override string Description
28         {
29             get { return "Registers the CmdLets and Providers in this assembly"; }
30         }
31         public override string DescriptionResource
32         {
33             get { return "GetContactCmdlet,Registers the CmdLets and Providers in this assembly"; }
34         }
35     } // class GetContactCmdletSnapIn
36 }
37 

Notice also that the wizard added the following references to our project:

1. System.Management.Automation. This is the main PowerShell SDK assembly
2. System.Configuration.Install: Needed by the Snap-in for installation.

(the other assemblies are the "standard" assemblies added to any project).

At this point, I would recommend building your project to make sure that you have no compilation errors.

Adding our Cmdlet class

Now, we are ready to add our cmdlet class itself. Right-click the project and select Add | New Item. In the New Item dialog, you will notice that you have a choice of deriving from two different types of base classes:

1. The first option (Windows PowerShell Cmdlet), will derive our cmdlet from the System.Management.Automation.Cmdlet base class. Deriving from this class means that our cmdlet is using a minimum set of dependencies on the Windows PowerShell runtime. This has two benefits:
   • The footprint of our cmdlet will be smaller and if there are any changes in the Windows PowerShell runtime, our cmdlet will very likely not be impacted.
   • It is also a lot easier to unit-test a cmdlet that is derived from the standard Cmdlet class, since we can directly create an instance of it and invoke it from our test code, without having to test it from the PowerShell command line.
2. The second option (Windows PowerShell PSCmdlet) will derive our cmdlet from the System.Management.Automation.PSCmdlet class. If you derive from this class, you will have more direct access to the Windows PowerShell runtime environment. For example, from your cmdlet you can:
   1. Call scripts.
   2. Access Providers
   3. Access the current session state.

But this will also increase the size of your cmdlet, and it will tie our cmdlet more tightly to the current version of the PowerShell runtime.

In general, Microsoft recommends that you derive a custom cmdlet from the Cmdlet class, unless you absolutely have a need access to the PowerShell runtime. We have no such requirement, so we will derive our cmdlet from the System.Management.Automation.Cmdlet class. Select Windows PowerShell Cmdlet, and enter GetContactCmdlet.cs as the name for the source file, and click the Add button. This will add the GetContactCmdlet class to our project. This class contains the initial skeleton code for our cmdlet, which we will be customizing in our next section.

Customizing the Cmdlet Attributes

The default implementation of our cmdlet declares our class as follows:

namespace FooTheory.GetContactCmdlet
{
    [Cmdlet(VerbsCommon.Get, "GetContactCmdlet", SupportsShouldProcess = true)]
    public class GetContactCmdlet : Cmdlet
    {
      ...
    }

The CmdLetAttribute identifies a .NET class as a cmdlet and specifies the verb and noun pair used to invoke the cmdlet. The required parameters for this attribute include the following:

1. VerbName. This is the name that describes the action performed by the cmdlet. Windows PowerShell strongly recommends that cmdlets use only the verb names specified by the action name enumerators (VerbsCommon). Note that in our case, we are using VerbCommons.Get, which indicates that we are retrieving information.
2. NounName. This string is the name that describes the entity on which the cmdlet performs its action. The wizard will set this to the class name of our cmdlet, which in this case is not what we want. Since we are retrieving contact, we will use the string "Contact" here.

The optional SupportsShouldProcces named parameter indicates that our cmdlet supports calls to the ShouldProcess method, which provides the cmdlet with a way to prompt the user before an action that changes the system is performed. This is actually a very useful parameter, and I recommend that you support it in any of your cmdlets that you implement which modify the state of a system resource. It can avoid one of those "Oh Sh#$" situations that you could run into otherwise (I did not mean to do that, Oh my God, I'm fired  etc.. ). The ShouldProcess method is triggered by using the confirm switch parameter on a command, as is shown below for the Stop-Process cmdlet:

13# Stop-Process 2920 -confirm

Confirm
Are you sure you want to perform this action?
Performing operation "Stop-Process" on Target "wmpnscfg (2920)".
 Yes   Yes to All   No  [L] No to All   Suspend  [?] Help (default is "Y"):

Another very useful named parameter is ConfirmImpact. ConfirmImpact will trigger the cmdlet to show what action would be performed if the user would execute the command, without actually performing the action. It is triggered by the WhatIf switch parameter, an example with the Stop-Process cmdlet is shown below:

15# Stop-Process 2920 -whatif
What if: Performing operation "Stop-Process" on Target "wmpnscfg (2920)".

As you can see, the WhatIf switch parameter shows us the action(s) that would be trigger by the execution of the command without actually executing it. I really recommend using this switch for any command that you just not 100% sure about ; -)..

Back to the Get-Contact cmdlet now. In the Get-Contact cmdlet case, we are only retrieving information, and we are not changing anything in the state of the system, so we can safely set the value of the ShouldProcess member to false.

Our final class declaration for our cmdlet will look as follows:

1 namespace FooTheory.GetContactCmdlet
2 {
3     [Cmdlet(VerbsCommon.Get, "Contact", SupportsShouldProcess = false)]
4     public class GetContactCmdlet : Cmdlet
5     {
6         ....
7     }
8 }

Implementing a Record Processing Methodology

For a cmdlet to participate in the Windows PowerShell environment, it must override the ProcessRecord() method listed below, and optionally the BeginProcessing() and EndProcessing() methods. The semantics of these methods are shown below:

• System.Management.Automation.Cmdlet.BeginProcessing: This method is called once for each command pipeline, before the first record is processed. You can use this method to provide pre-processing functionality, allocated unmanaged resources etc.
• System.Management.Automation.Cmdlet.ProcessRecord: This method is called multiple times, and is used to provide record-by-record functionality. This is where we will call into the main execution logic.
• System.Management.Automation.Cmdlet.EndProcessing: This method is called one, and is used to provide post-processing functionality, for example to free up unmanaged resources.

We will leverage these methods as follows:

1. In BeginProcessing, we will setup the connection with Outlook, creating the main Outlook Application class.
2. In ProcessRecord, we will retrieve the Outlook  2007 contacts, based upon the specified parameters, as defined in our list of requirements.
3. In EndProcessing, we will disconnect from Outlook 2007, freeing up any tied-up COM objects from the Outlook PIA.

I don't like to mix the "core" cmdlet code and the Outlook code, so in my project structure I created an Implementation folder, and added a GetContactsImpl.cs class, which will contain our "real" implementation, the interactions with the COM objects etc. This class is marked as internal, and contains three methods to correspond with the main cmdlet processing methods:

1. SetupOutlookConnection(): This method is called from our cmdlet's BeginProcessing method, and will initialize the connection to Outlook.
2. GetContacts(): This method retrieves the actual contacts from outlook, and is called from the cmdlet's ProcessRecord method. 
3. TeardownOutlookConnection(): This method is called from the cmdlet's EndProcessing method, and cleans up out Outlook application object, and any other tied-up COM objects.

The following class diagram shows the relationship between our cmdlet and it's implementation class:

Creating a Unit Testing Framework for our Cmdlet

Now that we have the skeleton of our cmdlet all setup, it would be a good idea to create a unit test project for it. That way, we can quickly test the functionality of the cmdlet right in Visual Studio without having to go the PowerShell console to perform the testing.

To add the Unit Test project, follow these steps:

• Right-click the solution, and click Add | New Project.
• Under the Visual C# tree branch, select Test, and in the list of templates click Test Project. Name the project FooTheory.GetContactCmdLet.Test, as is shown below:

• Click OK to create the test project.
• From the generated project files, delete the manual test (ManualTest1.mht), and the generated unit test (UnitTest1.cs).
• Right-click the project and select Add | Unit Test.
• In the Create Unit Test dialog, expand the tree until you get to the cmdlet, and select all of the members below the cmdlet (which are our three main methods: BeginProcessing, ProcessRecord and EndProcessing), as show below:

Click OK to create the unit test file. Visual Studio will create a skeleton unit test with a test method for each of your selected methods. By  default, the results of our tests will be set to "inconclusive" by default.

• To start out with, I cleaned up the generated the test code a bit, and created one test method which calls into all methods in sequence. The full code for the initial test is shown below:

 1 // The following code was generated by Microsoft Visual Studio 2005.
 2 // The test owner should check each test for validity.
 3 using Microsoft.VisualStudio.TestTools.UnitTesting;
 4 using System;
 5 using System.Text;
 6 using System.Collections.Generic;
 7 using FooTheory.GetContactCmdlet;
 8 
 9 using CmdLet = FooTheory.GetContactCmdlet.GetContactCmdlet;
10 
11 namespace FooTheory.GetContactCmdLet.Test
12 {
13     /// <summary>
14     ///This is a test class for FooTheory.GetContactCmdlet.GetContactCmdlet and is intended
15     ///to contain all FooTheory.GetContactCmdlet.GetContactCmdlet Unit Tests
16     ///</summary>
17     [TestClass()]
18     public class GetContactCmdletTest
19     {
20         private TestContext testContextInstance;
21 
22         /// <summary>
23         ///Gets or sets the test context which provides
24         ///information about and functionality for the current test run.
25         ///</summary>
26         public TestContext TestContext
27         {
28             get
29             {
30                 return testContextInstance;
31             }
32             set
33             {
34                 testContextInstance = value;
35             }
36         }
37 
38         [DeploymentItem("FooTheory.GetContactCmdlet.dll")]
39         [TestMethod()]
40         public void CompleteTest()
41         {
42             CmdLet target = new CmdLet();
43 
44             FooTheory.GetContactCmdLet.Test.FooTheory_GetContactCmdlet_GetContactCmdletAccessor accessor =
45                 new FooTheory.GetContactCmdLet.Test.FooTheory_GetContactCmdlet_GetContactCmdletAccessor(target);
46 
47             accessor.BeginProcessing();
48             accessor.ProcessRecord();
49             accessor.EndProcessing();
50         
51         } // method CompleteTest
52 
53     } // class GetContactCmdletTest
54 }

Do a full debug run on the test project to ensure that everything is working fine (it should be, because we are not doing anything yet.. ;-). So, now we are all set, we have our skeleton  cmdlet code, we have a unit test, we are ready to start implementing this puppy!

Finally, some "real" Coding: Creating the initial version of our Cmdlet

First, we need a add a reference to the Outlook 2007 PIA to our cmdlet project. Add a reference to the Microsoft Outlook 12.0 Object Library, and is shown below:

Click OK to add the reference. Include a using statement for Microsoft.Office.Interop.Outlook and add an an explicit using statement to the cmdlet implementation class source file, as shown below:

using Microsoft.Office.Interop.Outlook;
using Outlook = Microsoft.Office.Interop.Outlook;

Now, we can create an instance of the main Outlook Application class in our SetupOutlookConnection() method, and release it in the TeardownOutlookConnection() method. Our implementation class code so far is shown below:

 1 using System;
 2 using System.Collections.Generic;
 3 using System.Runtime.InteropServices;
 4 using Outlook = Microsoft.Office.Interop.Outlook;
 5 
 6 namespace FooTheory.GetContactCmdlet.Implementation
 7 {
 8     /// <summary>
 9     /// This class is our implementation class for the Get-Contact 
10     /// cmdlet.
11     /// </summary>
12     internal class GetContactsImpl
13     {
14         #region private fields
15         // This is our Outlook application class
16         private Outlook.ApplicationClass m_Application;
17         #endregion private fields
18 
19         internal void SetupOutlookConnection()
20         {
21             // Create an instance of our Application class
22             m_Application = 
23                 new Microsoft.Office.Interop.Outlook.ApplicationClass();
24         }
25 
26         internal void GetContacts()
27         {
28         }
29 
30         internal void TeardownOutlookConnection()
31         {
32             // Release our COM object
33             Marshal.ReleaseComObject(m_Application);
34         }
35 
36     } // class GetContactsImpl
37 
38 } // namespace FooTheory.GetContactCmdlet.Implementation

Note have we use Marshal.ReleaseComObject to release the m_application COM object instance, since we are dealing with COM interop here, because we are invoking the methods in the PIA through a RCW (Runtime Callable Wrapper).

At this point, it would be a good idea to run the unit test again to make sure everything is running OK.

Next, we will implement a "bare minimum" version of the GetContacts method. The code is shown below:

 1         /// <summary>
 2         /// This method returns our list of Contacts
 3         /// </summary>
 4         /// <returns></returns>
 5         internal List<ContactItem> GetContacts()
 6         {
 7             // Get the contacts folders
 8             MAPIFolder contactsFolder = 
 9                 m_Application.Session.GetDefaultFolder(OlDefaultFolders.olFolderContacts);
10 
11             // Create our output list of ContactItems and add the ContactItems
12             List<ContactItem> contacts = new List<ContactItem>();
13             foreach(ContactItem contactItem in contactsFolder.Items)
14             {
15                 contacts.Add(contactItem);
16             }
17 
18             return contacts;
19         }

In lines 8 and 9 we are getting a reference to the Contacts Folder, and then (line 13-15) we loop through each ContactItem and add it to our strongly-typed List.  Again, at this point I would recommend a quick unit test to make sure that your code is running fine.

Now, we are ready to change the main body of our cmdlet to output the list of ContactItems. To write a single object to the pipeline, the PowerShell SDK provides a System.Management.Automation.Cmdlet.WriteObject method. We use this method to write the List<ContactItem> instance to the pipeline. The first  argument to the method is our List<ContactItem> object, and the second parameter is set to true to indicate the our object is a collection, and that PowerShell should enumerate over the collection and send each object in turn through the pipeline.

We need to make sure that we add the following using statement to our main cmdlet code file:

using Microsoft.Office.Interop.Outlook;

The implementation our our ProcessRecord() method is now very simple:

1         /// <summary>
2         /// This is our main processing method
3         /// </summary>
4         protected override void ProcessRecord()
5         {
6             List<ContactItem> contacts = m_implementation.GetContacts();
7             WriteObject(contacts, true);
8         }
9 

So, we simply call into our implementation to get the list of ContactItem's, and the we use WriteObject() to write this instance to the pipeline. The PowerShell runtime will enumerate through each member of our collection, and stream each object through the command pipline in turn.

One important thing that I noticed when I unit tested this first, is that I would get a NotImplementedException on the WriteObject statement, while I would NOT get this error during testing in the PowerShell console. This is probably because the unit test environment is not a "real" cmdlet host environment. Therefore, I surrounded the WriteObject with a try/catch block during my testing period as follows:

        protected override void ProcessRecord()
        {
            List<ContactItem> contacts = m_implementation.GetContacts();
            
            //
            //  We get a NotImplemented exception during testing,
            //  hence the try/catch. I recommend that you remove the 
            //  try/catch after testing is complete!
            //
            try
            {
                WriteObject(contacts, true);
            }
            catch (NotImplementedException)
            {
            }
        } 

This should complete our initial, simple version of the cmdlet. Time to bring out the PowerShell console to do some "real" testing!

Testing our initial Cmdlet version in the PowerShell Console

Installing the Cmdlet

Before we can test our cmdlet, we first need to install the assembly which contains our cmdlet, and use the Add-PSSnapIn cmdlet to add the snap-in to our PowerShell console. And of course, since we are using PowerShell here, what other tools should we use to do this task? In the Installation sub-folder I have two scripts:

1. InstallCmdlet.ps1. This script installs our assembly, and adds the snap-in to the console.
2. UninstallCmdlet.ps1: This script removes the snap-in and performs an uninstall of our assembly.

The contents of InstallCmdlet.ps1 is shown below:

installutil ..\bin\debug\FooTheory.GetContactCmdlet.dll
Add-PSSnapIn FooTheory.GetContactCmdlet

 To register the cmdlet, follow these steps:

• Open a PowerShell prompt, and navigate to the Installation sub-directory of your project.
• Run  the InstallcmdLet.ps1 script, as is shown below:

Our cmdlet is now registered and ready to go. Note that you will have to run the registration script each time you start up a new PowerShell console, and you want to use the Get-Contact cmdlet. You will notice that you get tired of this real quick, so I recommend that  you run the installation script from your user profile script, which is typically located at:

C:\Users\<UserName>\Documents\WindowsPowerShell\profile.ps1

(Substitute <UserName> with your user name). For example, this is the code I added to the end of my profile.ps1 file:

# Register our Get-Contact cmdlet
Push-Location "c:\bhaelen\Posts\Custom cmdlets for PowerShell\Code\FooTheory.GetContactCmdlet\Installation" 
.\InstallCmdlet.ps1 > $null
Pop-Location

Notice also how I redirect the output of the install to $null, so we don't get the full output of the install and registration command each time we startup the PowerShell console.

Before we can start testing, we should first make sure that our Outlook 2007 instance is configured correctly.

Configuring Outlook

You can configure Outlook to allow or disallow programmatic access by selecting Tools | Marco | Security. In the Trust Center dialog, select Programmatic Access, as is shown in the screen shot below:

The default option is the second option (always warn me about suspicious activity). You can leave Outlook at this setting, but then you will get a prompt each time you run the Get-Contact cmdlet. The recommended setting is the first option (Warn me about suspicious activity when my antivirus software is inactive or out of date) so I recommend that you use this setting while you are testing the Get-Contact cmdlet.

Testing our cmdlet

To find out what kind of information you can get out of our Get-Contact cmdlet, run the following command in the PowerShell console:

Get-Contact | Get-Member -MemberType property

When you run this command, you will notice that you get a (very) long list of properties, a very small sub-set of which is shown below:

   TypeName: System.__ComObject#{00063021-0000-0000-c000-000000000046}

Name                         MemberType Definition
----                         ---------- ----------
Account                      Property   string Account () {get} {set}
Actions                      Property   Actions Actions () {get}
Anniversary                  Property   Date Anniversary () {get} {set}
Application                  Property   _Application Application () {get}
AssistantName                Property   string AssistantName () {get} {set}
AssistantTelephoneNumber     Property   string AssistantTelephoneNumber () {get} {set}
Attachments                  Property   Attachments Attachments () {get}
AutoResolvedWinner           Property   bool AutoResolvedWinner () {get}
BillingInformation           Property   string BillingInformation () {get} {set}
Birthday                     Property   Date Birthday () {get} {set}
Body                         Property   string Body () {get} {set}
Business2TelephoneNumber     Property   string Business2TelephoneNumber () {get} {set}
BusinessAddress              Property   string BusinessAddress () {get} {set}
BusinessAddressCity          Property   string BusinessAddressCity () {get} {set}
BusinessAddressCountry       Property   string BusinessAddressCountry () {get} {set}
BusinessAddressPostalCode    Property   string BusinessAddressPostalCode () {get} {set}
BusinessAddressPostOfficeBox Property   string BusinessAddressPostOfficeBox () {get} {set}
BusinessAddressState         Property   string BusinessAddressState () {get} {set}
BusinessAddressStreet        Property   string BusinessAddressStreet () {get} {set}
BusinessCardLayoutXml        Property   string BusinessCardLayoutXml () {get} {set}
BusinessCardType             Property   OlBusinessCardType BusinessCardType () {get}
BusinessFaxNumber            Property   string BusinessFaxNumber () {get} {set}
BusinessHomePage             Property   string BusinessHomePage () {get} {set}
BusinessTelephoneNumber      Property   string BusinessTelephoneNumber () {get} {set}
CallbackTelephoneNumber      Property   string CallbackTelephoneNumber () {get} {set}
CarTelephoneNumber           Property   string CarTelephoneNumber () {get} {set}
Categories                   Property   string Categories () {get} {set}
Children                     Property   string Children () {get} {set}
Class                        Property   OlObjectClass Class () {get}
Companies                    Property   string Companies () {get} {set}
CompanyAndFullName           Property   string CompanyAndFullName () {get}
CompanyLastFirstNoSpace      Property   string CompanyLastFirstNoSpace () {get}
CompanyLastFirstSpaceOnly    Property   string CompanyLastFirstSpaceOnly () {get}
CompanyMainTelephoneNumber   Property   string CompanyMainTelephoneNumber () {get} {set}
CompanyName                  Property   string CompanyName () {get} {set}
ComputerNetworkName          Property   string ComputerNetworkName () {get} {set}
Conflicts                    Property   Conflicts Conflicts () {get}
ConversationIndex            Property   string ConversationIndex () {get}
ConversationTopic            Property   string ConversationTopic () {get}
CreationTime                 Property   Date CreationTime () {get}
CustomerID                   Property   string CustomerID () {get} {set}

For example, if you want to retrieve the full name and the first email address of your contacts, you can run the following command in the console:

Get-Contact | Format-Table -Property FullName, Email1Address -Autosize

This produces a nicely formatted table which looks something like this:

FullName        Email1Address
--------        -------------
Matthew Ortiz   /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=mortiz
Kyle Sanford    ksanford@statera.com
Noah Dipasquale /o=STATORG/ou=FIRST ADMINISTRATIVE GROUP/cn=RECIPIENTS/cn=NDIPASQUALE
Wayne Macdonald /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=wmacdonald
Bob Mills       Bob_Mills@isagenix.net
Jeremy Campbell /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=jcampbell
Pete Miller     /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=pmiller
Aendenne Armour /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=aarmour
Jon Robinson    jRobinson@statera.com
Erl Egestad     /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=eegestad
Julie Clint     /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=jclint
Eric Boocock    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=eboocock
Sayward Flint   /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=sflint
Malcolm Boswell /o=STATORG/ou=FIRST ADMINISTRATIVE GROUP/cn=RECIPIENTS/cn=MBOSWELL
Mike Citro      /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=mcitro
Carol Jenner    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=cjenner
Ray Kwan        /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=rkwan
Brian Hensel    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=bhensel
Chris Banker    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=cbanker
Chad Kingsley   /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=ckingsley
Tony Blodgett   /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=tblodgett
Donn Felker     /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=dfelker

So, our custom cmdlet is working nicely, we can integrate it in the pipeline, it nicely cleans up after itself, so now we can go ahead and implement our more advanced features, starting with supporting arguments and parameters.

Adding Support for Arguments and Parameters

PowerShell command structure

In PowerShell, a basic command looks as follows:

command -parameter1 -parameter2 argument1, argument2

Where:

• Command: This is the name of the command. A command can be one of the following:
  • A cmdlet (like our Get-Contact)
  • A shell function
  • A script command
  • A native Windows command (for example: Notepad, Calc, etc.)

A command can be followed by zero or more parameters.

• parameter1. This is a parameter that takes no arguments, often called a switch parameter.
• parameter2. This is a parameter that takes two arguments
• argument1. First argument to parameter2
• argument2. Second argument to parameter2

Adding Parameter support to Get-Contact

As we mentioned in our requirements, we would like to enable our cmdlet to filter the returned list of contacts by the Last Name of the Contact. The user should be able to specify one or more names, and furthermore, should be able to use wildcards and regular expressions to specify the last name he/she is looking for (for example: give me all contacts which have a last name starting with A, B or C.).

To support this functionality, we will add a LastName property to our GetContactCmdlet class. The full declaration of this property is shown below:

 1         /// <summary>
 2         /// This is  our "LastName" parameter. This is a positional 
 3         /// parameter, which accepts an array of values, and supports
 4         /// wildcard expansion. This parameter can also accept input
 5         /// from the pipeline, and has a "ln" shorthand alias
 6         /// </summary>
 7         [Parameter(
 8             Position = 0,
 9             Mandatory = false,
10             ValueFromPipeline=true,
11             ValueFromPipelineByPropertyName = true,
12             HelpMessage = "This are the Last Names of the Contact. You can use wildcards")]
13         [ValidateNotNullOrEmpty]
14         [Alias("ln")]
15         public string[] LastName
16         {
17             get { return m_lastNames; }
18             set { m_lastNames = value; }
19         }
20 

The type of our LastName property is string[], because we want to allow the user to specify a list of last of last names, hence the declaration as an array. We backed this property with the m_lastNames private field. We adorned our property with a System.Management.Automation.ParameterAttribute, which identifies our property as a parameter of the Get-Contact cmdlet. The name of the parameter is the name of our property ("LastName").

In our cmdlet class, we will pass the m_lastNames array to the GetContacts() method of our implementation class, as is shown below:

 1         /// <summary>
 2         /// This is our main processing method, called
 3         /// once for each input record in the pipeline
 4         /// </summary>
 5         protected override void ProcessRecord()
 6         {
 7             //  Invoke our implementation method, passing in our array of last 
 8             //  names
 9             List<ContactItem> contacts = m_implementation.GetContacts(m_lastNames);
10             
11             //
12             //  We get a NotImplemented exception during testing,
13             //  hence the try/catch. I recommend that you remove the 
14             //  try/catch after testing is complete!
15             //
16             try
17             {
18                 WriteObject(contacts, true);
19             }
20             catch (NotImplementedException)
21             {
22             }
23 
24         } // method ProcessRecord

Our ParameterAttribute contains the following members:

• Postion. This member identifies the LastName parameter as a positional parameter with position 0. This means that the first argument the user enters on the command line will be automatically inserted for the parameter. This implies that the user can omit the name of the parameter itself, so the following commands will be semantically identical:

             Get-Command -LastName Hensel, Banker     
             Get-Contact Hensel, Banker     

• Mandatory. This member indicates whether or not the parameters is required when invoking the cmdlet. Clearly, in our case the parameter is optional.
• ValueFromPipeline. This member indicates whether our cmdlet parameter can take values from incoming pipeline objects. For example, if you have a file Names.txt, containing a list of LastNames, you could use the content of this file as the left-hand side of a pipeline as follows:

    14# Get-Content Names.txt | Get-Contact | Format-Table FullName, Email1Address -AutoSize     

    FullName        Email1Address
    --------        -------------
    Chris Banker    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=cbanker
    Brian Hensel    /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=bhensel
    Aendenne Armour /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=aarmour

The content of the Names.txt file is shown below:

    17# Get-Content Names.txt
    Banker
    Hensel
    Armour

Being able to process input from the pipeline opens up all kinds of possibilities for our cmdlet, since it now will be able to fully participate in the PowerShell pipeline processing mechanisms. I would definitely recommend always supporting this feature, since it does require no parsing work except for the adding the ValueFromPipeline member to the  [Parameter] declaration!

• ValueFromPipelineByPropertyName. If this property is set to true, the Windows PowerShell runtime will check incoming pipeline properties for a LastName property. If the incoming object has such a property, the runtime will bind the FullName parameter to the FullName property of the incoming object.

The sample code contains a class library assembly called FooTheory.SampleInputPipelineLib. This class library contains a simple class called Customer, which has a LastName property, and has a simple constructor that takes a lastName argument:

 1     public class Customer
 2     {
 3         private string  m_lastName;
 4 
 5         public string  LastName
 6         {
 7             get { return m_lastName; }
 8             set { m_lastName = value; }
 9         }
10 
11         public Customer(string lastName)
12         {
13             m_lastName = lastName;
14         }
15     }
16 

In the following PowerShell script we load this assembly, create an array of two customer objects, and use this array as the left-hand argument of a Get-Contact pipeline:

1 [System.Reflection.Assembly]::LoadFile($(Resolve-Path .).Path + "\FooTheory.SampleInputPipelineLib.dll") > $null
2 $myCustomer = @($(New-Object FooTheory.SampleInputPipelineLib.Customer("Hensel")),
3                 $(New-Object FooTheory.SampleInputPipelineLib.Customer("Banker")))
4 $myCustomer | Get-Contact | Format-Table -Property FirstName, LastName, CompanyName -Autosize

As you can see, this script produces the correct output:

79# .\SampleObjectTest.ps1

FirstName LastName CompanyName
--------- -------- -----------
Brian     Hensel   Statêra
Chris     Banker   Statêra

• HelpMessage: This argument provides a short description for this parameter.

In addition to the [Parameter] attribute, we also specify a attribute of type System.Management.Automation.ValidateNotNullOrEmptyAttribute. This attribute validates that the argument of an optional parameter is not null, an empty string or an empty collection.

The last attribute is of type System.Management.Automation.AliasAttribute. This attribute allows us to define an alias for the parameter, in this case we specified "ln" as an alias for "LastName", as is shown below:

5# get-contact -ln Banker | ft FullName, Email1Address -auto

FullName     Email1Address
--------     -------------
Chris Banker /o=StatOrg/ou=First Administrative Group/cn=Recipients/cn=cbanker

Adding the attributes as specified above is the only thing we need to do to get all this functionality! The only feature which requires just a little bit of work is support for wildcard arguments, which is a topic we will tackle next.

Important note: Whenever you are re-building your cmdlet, make sure that you exit all of your PowerShell console instances. Otherwise, you will not be able to successfully build your cmdlet, since it will be locked by the PowerShell instance(s).

Adding Support for Wildcard Arguments

To support wildcards and wilcard expansion, the PowerShell SDK offers the System.Management.Automation.WildcardPattern class. The matching process itself is controlled by the System.Management.Automation.WildcardOptions flags enumeration. The enumeration supports the following options:

1. Compiled. When this option is selected, the wildcard pattern is compiled into an assembly, which yields faster execution, but will increase startup time a bit.
2. IgnoreCase. When this option is selected, the matching is performed in a case-insensitive manner.
3. None. When this option is selected, no special processing is performed.

In our implementation, we will select a combination of the Compiled and IgnoreCase flags, so we will do a case-insensitive matching. To create an instance of the WilcardPattern class, you pass in the pattern, and the selected options, as is shown below:

                    // Loop over all names
                    foreach (string lastName in lastNames)
                    {
                        // Perform the  Wildcard expansion, and add the ContactItem
                        // if we have a match
                        WildcardPattern wildcard = new WildcardPattern(lastName, options);
                       .......
                    }

Here, we are looping over each last name in our array, and we create a WildcardPattern for each last name (which might contain a wildcard pattern).

The WildcardPattern class supports a IsMatch() method, which returns true if a match is found:

                        if (contactItem.LastName != null && wildcard.IsMatch(contactItem.LastName))
                        {
                            contacts.Add(contactItem);
                            break;
                        }

Here, we pass in the LastName of the current contact into the IsMatch() method, and add the contact to our list of results  when a match is found.

Our full implementation of the GetContacts() method is shown below:

 1 /// <summary>
 2         /// This method returns our list of Contacts
 3         /// </summary>
 4         /// <returns></returns>
 5         internal List<ContactItem> GetContacts(string[] lastNames)
 6         {
 7             // Create our output list of ContactItems 
 8             List<ContactItem> contacts = new List<ContactItem>();
 9 
10             // Get the contacts folders from the current Session
11             MAPIFolder contactsFolder = 
12                 m_Application.Session.GetDefaultFolder(OlDefaultFolders.olFolderContacts);
13 
14             // Set our wildcard options
15             WildcardOptions options = WildcardOptions.IgnoreCase |
16                                       WildcardOptions.Compiled;
17 
18             // Enumerate the Contact Items
19             foreach (ContactItem contactItem in contactsFolder.Items)
20             {
21                 // Did we want to filter on LastName?
22                 if (lastNames != null)
23                 {
24                     // Loop over all names
25                     foreach (string lastName in lastNames)
26                     {
27                         // Perform the  Wildcard expansion, and add the ContactItem
28                         // if we have a match
29                         WildcardPattern wildcard = new WildcardPattern(lastName, options);
30                         if (contactItem.LastName != null && wildcard.IsMatch(contactItem.LastName))
31                         {
32                             contacts.Add(contactItem);
33                             break;
34                         }
35                     }
36                 }
37                 else
38                 {
39                     // No filter, simply add the Contact Item
40                     contacts.Add(contactItem);
41                 }
42             } // foreach
43 
44             return contacts;
45 
46         } // method GetContacts

As you see, our implementation is rather simple, we check if a filter (lastNames array) was specified, if not, we add every contact, otherwise we just add those contacts that match our wildcard patterns.

Testing our Cmdlet

Below are a number of examples that test the wildcard processing of our cmdlet:

11# Get-Contact [a-c]* | Format-Table -Property LastName, FirstName, JobTitle

LastName                                FirstName                               JobTitle
--------                                ---------                               --------
Campbell                                Jeremy                                  VP of Strategic Resources
Armour                                  Aendenne                                Consultant
Clint                                   Julie                                   Office Manager
Boocock                                 Eric                                    CRM Practice Director
Boswell                                 Malcolm                                 Client Partner
Citro                                   Mike                                    Consultant
Banker                                  Chris                                   Consultant
Blodgett                                Tony                                    Consultant


12# Get-Contact [a-c]*, *er | Format-Table -Property LastName, FirstName, JobTitle

LastName                                FirstName                               JobTitle
--------                                ---------                               --------
Campbell                                Jeremy                                  VP of Strategic Resources
Miller                                  Pete                                    VP of Services
Armour                                  Aendenne                                Consultant
Clint                                   Julie                                   Office Manager
Boocock                                 Eric                                    CRM Practice Director
Boswell                                 Malcolm                                 Client Partner
Citro                                   Mike                                    Consultant
Jenner                                  Carol                                   Consultant
Banker                                  Chris                                   Consultant
Blodgett                                Tony                                    Consultant
Felker                                  Donn                                    Consultant


13# Get-Contact *e?? | Format-Table -Property LastName, FirstName, JobTitle

LastName                                FirstName                               JobTitle
--------                                ---------                               --------
Campbell                                Jeremy                                  VP of Strategic Resources
Boswell                                 Malcolm                                 Client Partner
Blodgett                                Tony                                    Consultant

Summary

This concludes the development of our cmdlet. As you can see, the Windows PowerShell scripting language is a very powerful, productive and highly customizable development environment. Developing custom cmdlets is very easy, since all of the hard work like parameter parsing, passing objects to the command line, wildcard expansion etc. is performed by the PowerShell runtime.

I hope you have enjoyed this article, and I encourage you to experiment with this environment, and I guarantee you that you will never go back to the "good old" Windows command prompt! And of course, feedback is always appreciated!

Technorati Tags: PowerShell - Outlook 2007 - COM Interop

Hello, it took a while, but I finally finished the "Expression Trees" post, which is part 6 of the "New Features in C# 3.0 which enable LINQ" series. I also uploaded the sample code for those who are interested. If you read the version of the article of a few weeks back, and you are interested in the topic, I would recommend taking a look at the latter part of the post, I added some material on how to build dynamic queries.

So, what's next? In part 7 (the final installment of this series), we will take a look under the hood of a LINQ provider. The last section of my post hints on how such a provides goes about dynamically building a domain-specific query, and executing this "query" when the first result is accessed. But in this upcoming post, we will really dig into the bowels of how this is done, look at the interfaces involved in this process, and who knows, maybe we even write our own provider?

Take care..

Introduction

In the previous installment of this series, we looked at lambda expressions. During our little trip down memory lane, we concluded that lambda expressions are a logical evolution of anonymous functions, which in turn are (to some extent) an evolution of delegates. In the context of this article, it is very important to understand that lambda expressions are always directly compiled into IL code by the compiler, so a lambda expression is a representation of a unit of executable code, it is not a data structure.

Below is an example of a lambda expression that calculates the volume of a box:

Func<double, double, double, double> boxVolume = 
                (width, length, height) => width * length * height;

Next, let's take a look at this line of code:

Expression<Func<double, double, double, double>> volumeExpr = 
                (width, height, length) => m_width * m_height * m_length;

Essential, what we are doing in this line of code is taking our earlier lambda expression, and supplying it as the type argument for a generic type called Expression<T>. Expression<T> is defined in the System.Linq.Expressions namespace, and looks as follows:

 1     // Summary:
 2     //     Represents a strongly typed lambda expression as a data structure in the
 3     //     form of an expression tree. This class cannot be inherited.
 4     //
 5     // Type parameters:
 6     //   TDelegate:
 7     //     The type of the delegate that the System.Linq.Expressions.Expression<TDelegate>
 8     //     represents.
 9     public sealed class Expression<TDelegate> : LambdaExpression
10     {
11         // Summary:
12         //     Compiles the lambda expression described by the expression tree into executable
13         //     code.
14         //
15         // Returns:
16         //     A delegate of type TDelegate that represents the lambda expression described
17         //     by the System.Linq.Expressions.Expression<TDelegate>.
18         public TDelegate Compile();
19     }
20 

When you look at the comments for the class, you notice that an Expression<T> represents a lambda expression as a data structure in the form of an expression tree. So, rather than directly compiling the lambda expression into executable IL that can directly evaluate the expression, what an Expression<T> does is create an in-memory tree of objects that represents the expression that was supplied as its type argument.

Now, when you look at the definition of Expression<T> you probably noticed it single method called Compile(). This method compiles the expression, and when the compilation is successful,  generates the "real" lambda expression, which then in turn can be executed, because it really IS IL, ready to run! To execute a compiled Expression<T> instance, just use it's Invoke() method, supplying the required parameters.

An overview of this two-step process is shown below:

The in-memory representation of an Expression<T> is an expression tree, as is shown in the above figure.

Microsoft actually made a visualizer available, which is very useful for analyzing the structure of an expression tree. A screen shot of the visualizer for our expression looks as follows:

Getting the Expression Tree Visualizer up and running in Orcas Beta 2

In Beta 2 of Orcas, the expression tree visualizer is NOT installed by default. But fear not, and follow these steps to get the visualizer to work:

• If you are running the Orcas Beta 2 VPC image, go to the directory: C:\Program Files\Microsoft Visual Studio 9.0\Samples\1033. Otherwise, go to the directory where you have Oracas installed.
• In this directory, you will find a file called CSharpSamples.zip. Unzip this file to your preferred directory (I created a sub-directory CSharpSamples under the above directory).
• After you unzipped the file, move to the sub-directory LinqSamples, in this sub-directory you will find another subdirectory called ExpressionTreeVisualizer. Go to this sub-directory.
• In the ExpressionTreeVisualizer sub-directory, you will find a solution called ExpressionTreeVisualizer.sln. Open and build this solution.
• After the build successfully completes, move to the \ExpressionTreeVisualizer\bin\Debug sub-directory. In this directory, you will find a file called ExpressionTreeVisualizer.dll. This is the actual visualizer dll used by Visual Studio.
• Copy the ExpressionTreeVisualizer.dll to the C:\Documents and Settings\Administrator\My Documents\Visual Studio 2008\Visualizers directory (if not logged in as administrator, substitute administrator with your user name. This is the location where all Visual Studio visualizers live for your Visual Studio installation.
• After you have completed the above steps, and have restarted Visual Studio, the following will happen when you are debugging an application with an Expression<T> instance, and you have a breakpoint on a unit of code with an Expression<T>:
  • You will see the "Looking glass" icon, indicating that a visualizer is available for the object under the cursor.
  • Click the looking glass icon.
  • The "Expression Tree Visualizer" selector will pop up, as shown below:

• Click the pop-up, and the visualizer will show the expression tree in all of its beauty, as shown earlier.

Creating Expression Trees

You can create an expression tree in two different ways:

1. You can write an Expression<T> as we did in our example in the previous section. by assigning an Expression<T> instance to a lambda expression. As we saw in the visual studio visualizer, the C# compiler automatically generated (in this case a quite elaborate) expression tree for us.
2. You can use the Expression API in the System.Linq.Expressions namespace to manually build your own expressions. While this quite a bit more complicated and time-consuming, it does open up all kinds of possibilities, as we will see in the "Dynamic Queries" section of this post.

We will take a closer look at each approach in the next sections.

Creating an expression tree based upon a lambda expression

The first approach to creating expression trees is fairly easy. You write a valid lambda expression, and supply it as the type argument to an Expression<T> instance. This is illustrated by the following code sample (this is the SimpleExample class in the SimpleExpressionTrees project of our sample code):

 1         /// <summary>
 2         /// This method shows how you can create an expression, based upon
 3         /// a Lambda Expression
 4         /// </summary>
 5         public static void CreateExpressionFromLamba()
 6         {
 7             // This is how you declare a lambda expression by itself
 8             Func<int, int, int> addLambda = (value1, value2) => value1 + value2;
 9 
10             // This is how you declare an equivalent lambda expression
11             Expression<Func<int, int, int>> lambdaExpression = (value1, value2) => value1 + value2;
12             Console.WriteLine("Lambda Expression: {0}", lambdaExpression.ToString());
13             dumpExpression(lambdaExpression);
14 
15             // Compile and execute the lambda expression in the Expression<T>
16             Func<int, int, int> lambda = lambdaExpression.Compile();
17             int v1 = 5;
18             int v2 = 10;
19             Console.WriteLine("Compiling and executing the expression");
20             Console.WriteLine("{0} + {1} = {2}", v1, v2, lambda.Invoke(v1, v2));
21 
22             // Note how I can use statements in a lambda expression
23             Func<int, int, bool> compareLambda = 
24                 (val1, val2) => { if (val1 == val2)  return true; else return false; };
25            
26             // The statement below will not compile, since you cannot use
27             // a lambda expressions with a statement to create an expression tree
28             //Expression<Func<int, int, bool>> compareExpression =
29             //    (val1, val2) => { if (val1 == val2)  return true; else return false; };
30         
31         } // method CreateExpressionFromLamba

In line 8, we create a simply lambda expression which adds the two integers passed as arguments. In line 11, I create an expression tree, based upon an identical lambda expression. Note also, how Expression<T> has a handy ToString() override which shows the lambda expression. The output of line 12 is shown below:

Lambda Expression: (value1, value2) => (value1 + value2)

So, what  we get is a neat representation of the lambda expression on which the expression is based. To add some functionality to this I created a simple little dumpBLOCKED EXPRESSION function which prints out the details of an expression. The code for this method is shown below:

 1         /// <summary>
 2         /// This method prints out the details of the passed-in Expression
 3         /// </summary>
 4         /// <typeparam name="T"></typeparam>
 5         /// <param name="expression"></param>
 6         private static void dumpExpression<T>(Expression<T> expression)
 7         {
 8             Console.WriteLine("Lambda Expression: {0}", expression.ToString());
 9             Console.WriteLine("Expression Details: ");
10             Console.WriteLine("\tExpression Body: {0}", expression.Body);
11             for (int index = 0; index < expression.Parameters.Count; index++)
12             {
13                 var parameter = expression.Parameters[index];
14                 Console.WriteLine("\tParameter {0}: '{1}', Type: {2}",
15                     index, parameter, parameter.Type.Name);
16             }
17             Console.WriteLine("\tNodeType: {0}", expression.NodeType);
18         }

This code sample shows some interesting properties of the Expression<T> class:

1. The Body property of the Expression<T> gives us the body of the lambda expression.
2. The Parameters property returns a ReadOnlyCollection<T> where T is of type ExpressionParameter. The code simply loops over each parameter, and prints the index, name and type of each parameter. 
3. Finally, the NodeType property provides the Node Type of the Expression. We'll have more to say about Node Types in the next section.

When we pass our expression to this function, we get the following output, which correctly represents our expression instance.

Lambda Expression: (value1, value2) => (value1 + value2)
Expression Details:
        Expression Body: (value1 + value2)
        Parameter 0: 'value1', Type: Int32
        Parameter 1: 'value2', Type: Int32
        NodeType: Lambda

When the invoke the Expression<T> Compile() method, we get our original lambda expression back, as is shown below:

Func<int, int, int> lambda = lambdaExpression.Compile();

Next, we can invoke the lambda  expression directly with it's Invoke() method:

            int v1 = 5;
            int v2 = 10;
            Console.WriteLine("Compiling and executing the expression");
            Console.WriteLine("{0} + {1} = {2}", v1, v2, lambda.Invoke(v1, v2));

which produces our expected output:

Compiling and executing the expression
5 + 10 = 15

When we go back to our first listing, we see that we can also create a lambda expression, based upon a statement, instead of a simple expression:

1             // Note how I can use statements in a lambda expression
2             Func<int, int, bool> compareLambda = 
3                 (val1, val2) => { if (val1 == val2)  return true; else return false; };

However, you cannot use a lambda expression with a statement to initialize an expression. If I would attempt to compile a function with the following statement:

            // The statement below will not compile, since you cannot use
            // a lambda expressions with a statement to create an expression tree
            Expression<Func<int, int, bool>> compareExpression =
                (val1, val2) => { if (val1 == val2)  return true; else return false; };

The you would get the following compilation error:

Error    1    A lambda expression with a statement body cannot be converted to an expression tree    SimpleExpressionTrees

Manually Creating an Expression Tree

The System.Linq.Expression namespace offers a rich API, which allows you to dynamically build an expression tree of arbitrary complexity, without the limitations mentioned in the previous section. For example, the code sample below, shows how to use this API to build exactly the same expression as we did earlier in the first example of the CreateExpressionFromLamba() method:

 1         /// <summary>
 2         /// In this example, we build exactly the same expression as we 
 3         /// did in the first example of the CreateExpressionFromLamba()
 4         /// function
 5         /// </summary>
 6         public static void ManuallyBuildBLOCKED EXPRESSION
 7         {
 8             Console.WriteLine("\n\nManually Building an expression");
 9 
10             //  First, create the parameter expressions
11             ParameterExpression val1Expr = Expression.Parameter(typeof(int), "value1");
12             ParameterExpression val2Expr = Expression.Parameter(typeof(int), "value2");
13 
14             // Create the "add" expression
15             BinaryExpression addExpr = BinaryExpression.Add(val1Expr, val2Expr);
16             
17             // Wrap the add expression in a Lambda expression
18             var lambdaExpr = 
19                 Expression.Lambda<Func<int, int, int>>(
20                     addExpr, new ParameterExpression[] {val1Expr, val2Expr});
21 
22             // Dump the expression
23             dumpExpression(lambdaExpr);
24 
25             // Compile and execute the lambda expression in the Expression<T>
26             Func<int, int, int> lambda = lambdaExpr.Compile();
27             int v1 = 5;
28             int v2 = 10;
29             Console.WriteLine("Compiling and executing the expression");
30             Console.WriteLine("{0} + {1} = {2}", v1, v2, lambda.Invoke(v1, v2));
31 
32         } // method ManuallyBuildExpression

As you can see, the code is definitely more verbose, but not necessarily complicated. First, we create two ParameterExpressions to represent the value1 and value2 parameters to the lambda expression. Next, we create an Add BinaryExpression, which adds both values together. The BinaryExpression class has 17 overloads, which allow you to create just about any binary expression you would ever need.

Finally, we wrap our expression into a lambda expression, which is ultimately what we were planning to do from the start. Note that we need to specify the correct lambda prototype as type argument. The function arguments are the Binary Add expression, and both of our Parameter Expressions.

To verify that we indeed created the correct expression, we passed our expression on to our trusted dumpBLOCKED EXPRESSION function. Here is the output:

Lambda Expression: (value1, value2) => (value1 + value2)
Expression Details:
        Expression Body: (value1 + value2)
        Parameter 0: 'value1', Type: Int32
        Parameter 1: 'value2', Type: Int32
        NodeType: Lambda

And indeed, the output of our manually created expression tree is identical to our first expression, created directly from a lambda expression! An naturally, we can compile and invoke the expression as we did in our previous example.

Using Expression Trees to create Dynamic  Query Expressions

One of the interesting applications of expression trees, is the enablement of more dynamic LINQ queries. The DynamicQueryWithExpression class of the sample code illustrates this. The constructor of this class generates an integer array with 20 elements, as is shown below:

1         public DynamicQueryWithExpressions()
2         {
3             for (int i = 0; i < 20; i++)
4             {
5                 m_myInts.Add(i);
6             }
7 
8             Console.WriteLine("\nDynamic Query Example");
9         } // constructor

The SimpleQuery() method performs a query against this list, retrieving all elements that are between 5 and 15 exclusive:

 1         /// <summary>
 2         /// This method executes a simple, very straightforward query
 3         /// </summary>
 4         public void SimpleQuery()
 5         {
 6 
 7             var results = from i in m_myInts
 8                             where i > 5 && i < 15
 9                                 orderby i
10                                     select i;
11 
12             showResults("SimpleQuery", results);
13         
14         } // method  SimpleQuery

The output from this query is shown below:

Results from query: QueryWithDynamicParameters
        7
        9
        11
        14

This query contains a very simple where clause that specifies our range. But what if we have a situation where we do not know at compile time what numbers we need? We might want to include some number, and at the same time, we might want to exclude some other numbers. Furthermore, the user might want to specify these numbers at runtime, through some type of user interface. In such a scenario, the above simple query construct will no longer work for us.

Scenario's such as the one describe above are a custom fit for custom expressions. As we mentioned before, we can construct any Expresssion<T> -based expression tree we want, of arbitrarily deep complexity. In the above example, what we have a need for is a way to build AND and OR expression "chains". One we can build these expression, we can build ourselves a "power where clause" we can use for our query.

I started writing a class to build these types of predicates, but then I came across a beautiful implementation by  Josheph Albhari, and I decided that this was so nicely done that there was no way to improve upon his code, so credit where credit is due. By the way, I really recommend Joseph Albhari's blog, he has some great C# 3.0 examples in there. The name of his class is PredicateBuilder, and the code is shown below:

 1 using System;
 2 using System.Collections.Generic;
 3 using System.Linq;
 4 using System.Linq.Expressions;
 5 
 6 namespace SimpleExpressionTrees
 7 {
 8     /// <summary>
 9     /// This class allows us to build AND and OR expressions 
10     /// </summary>
11     public static class PredicateBuilder
12     {
13         // These expressions define a true or false expression, which are
14         // used as "anchor" expressions for build or AND or OR predicates
15         public static Expression<Func<T, bool>> True<T>() { return f => true; }
16         public static Expression<Func<T, bool>> False<T>() { return f => false; }
17 
18         /// <summary>
19         /// This Extension method creates an OR Expression
20         /// </summary>
21         /// <typeparam name="T"></typeparam>
22         /// <param name="expr1"></param>
23         /// <param name="expr2"></param>
24         /// <returns></returns>
25         public static Expression<Func<T, bool>> Or<T>(this Expression<Func<T, bool>> expr1,
26                                                            Expression<Func<T, bool>> expr2)
27         {
28             // Invoke the second expression, passing in the parameters of the first expression
29             var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
30 
31             // Create the Or Expression, and wrap it in a Lamba expression
32             return Expression.Lambda<Func<T, bool>>
33                   (Expression.Or(expr1.Body, invokedExpr), expr1.Parameters);
34         
35         } // method Or
36 
37         /// <summary>
38         /// This Extension method creates an AND expression
39         /// </summary>
40         /// <typeparam name="T"></typeparam>
41         /// <param name="expr1"></param>
42         /// <param name="expr2"></param>
43         /// <returns></returns>
44         public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> expr1,
45                                                             Expression<Func<T, bool>> expr2)
46         {
47             // Invoke the second expression, passing in the parameters of the first expression
48             var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
49 
50             // Create the And Expression, and wrap it in a Lamba expression
51             return Expression.Lambda<Func<T, bool>>
52                   (Expression.And(expr1.Body, invokedExpr), expr1.Parameters);
53         
54         } // method And
55 
56     } // class PredicateBuilder
57 } //namespace SimpleExpressionTrees
58 

In lines 15 and 16, we have a True and a False static Expression, which can be used an "anchor points" for building an AND or and OR expression chain. The implementations of both the And and the Or methods is very elegant and straightforward. First (in lines 29 and 48), we create an InvocationExpression,  passing in the second expression and their parameters. Next, we create a BinaryExpression by performing and And or an Or  between the body of our first expression and the InvocationExpression created earlier (lines 32 and 51), and then wrap the result in a Lambda Expression. The above class can be found in the PredicateBuilder.cs class of the sample code.

Based upon this PredicateBuilder class, I created a method in the DynamicQueryWithExpressions class that can build an OR filter chain. The code for this method is shown below:

 1         /// <summary>
 2         /// This method builds up a dynamic expression with OR statements
 3         /// for each number that should be included. It uses the PredicateBuilder
 4         /// to build to expression
 5         /// </summary>
 6         /// <param name="numbersToInclude"></param>
 7         /// <returns></returns>
 8         private Expression<Func<int, bool>> filterByIntList(params int[] numbersToInclude)
 9         {
10             // Since we are building OR statements, we start out with a FALSE expression,
11             // otherwise our OR statement would always evaluate to true..
12             var predicate = PredicateBuilder.False <int> ();
13  
14             // Loop over each number to include, and add an OR predicate
15             // to the expression
16             for (int index = 0 ; index < numbersToInclude.Length; index++)
17             {
18                 int value = numbersToInclude[index];
19                 predicate = predicate.Or(i => i == value);
20             }
21             return predicate;
22 
23         } // method filterByIntList

In line 12, we set our "anchor expression", which should be:

• False if we are building an OR filter chain (otherwise our result would always be true).
• True is we are building an AND filter chain (otherwise our result would always be false)

We get the arguments to include in the numbersToInclude params array, so we simply need to loop over each element of the array, and invoke the predicate.Or() method with the lambda expression:

i => i == value

as argument. This lamda expression will guarantee that our value is included in the results if it is present in the source array. So, in effect what we are doing during this loop is building and expression which gets more and more complex, and finally includes all values that should be used in the Where clause of our query.

Below is then finally the query which uses this expression:

 1         /// <summary>
 2         /// This method extends our query with dynamic and and or elements
 3         /// </summary>
 4         /// <param name="numbersToFind"></param>
 5         public void QueryWithDynamicParameters(params int[] numbersToInclude)
 6         {
 7             Expression<Func<int, bool>> predicate = filterByIntList(numbersToInclude);
 8             Func<int, bool> isInArray = predicate.Compile();
 9 
10             var results = from value in m_myInts.Where(isInArray)
11                           where value > 5 && value < 15
12                           orderby value
13                           select value;
14 
15             showResults("QueryWithDynamicParameters", results);
16 
17         } // method QueryWithDynamicParameters

in line 7 be invoke our method to create our expression,  and in line 8, we compile the expression, which gives us back a lambda expression. Note that we selected to actually apply the lambda expression directly on the m_myInts array, so we can use the power of a "standard, static" where clause together with the power of a dynamically created expression.  The application of the Where condition on the array effectively invokes the following extension method of IEnumerable<T>:

 public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

The invocation of this method and the results are shown below:

 1             // These examples show how to create a dynamic
 2             // query with an Expression
 3             DynamicQueryWithExpressions dynQuery = new DynamicQueryWithExpressions();
 4             ....
 5             dynQuery.QueryWithDynamicParameters(new int[] { 2, 7, 9, 11, 14, 17 });
 6 
 7 Results from query: QueryWithDynamicParameters
 8         7
 9         9
10         11
11         14

As you see, all supplied values are included except for 2 and 17, because they do not satisfy the "static" where clause.

The above example is very simple, but you can see how you could easily build powerful expression generators with classes like the PredicateBuilder.

Expression Trees and LINQ Query Providers

Besides enabling you to create dynamic queries, expression trees are also used in a "deferred execution" mode in LINQ. For example, in LINQ to SQL, here is the sequence of events:

1. The user writes a method which contains a LINQ-to-SQL query.
2. The compiler compiles the SQL-like query into a sequence of method calls.
3. The compiler then compiles the series of method calls into an expression tree.
4. The expression trees is used by the LINQ-to-SQL infrastructure to create an optimized T-SQL statement. So, in this case, the expression tree is NEVER directly executed, it is simply used as a rich data structure, from which we can create a Transact-SQL query.
5. The first time the results of the query are actually accessed, the SQL Statement is sent to the designated SQL server and executed, and the results are returned to the client.

Note that the above description is greatly simplified. The next (and final) installment of this series will dive deep into the bowels of the interfaces and classes that enable all of this to happen. Also, note that this model is not necessarily limited to LINQ-to-SQL. Most (if not all) LINQ providers use an identical model where some type of query is created from the expression tree, and executed when the results of the query are first accessed.

Hello, I (finally) got around to uploading the code for part 5 of my C# 3.0 article series. I also finalized the text of the post, adding a section on extension methods and lambda functions at the end of the article.

If you read the article, you will notice my excitement around lambda expression, I think they really move C# ahead with leaps and bounds, and open up the possibilities of endless language extensions, especially combined with expression trees, which is curiously enough the topic of of my next post!

Stay  tuned!

Definition

A Lambda Expression is basically a logic evolution of the concept of anonymous functions, which was introduced with C# 2.0. Anonymous functions basically allow the body of a function to be written "in-line", where we would normally expect a delegate to be invoked.

Lambda expressions provide a much more concise, functional and targeted syntax for writing anonymous methods. In LINQ they are used all the time. A LINQ query is basically a tree of Lambda expressions. We will look at expressions trees in detail in the next post in this series.

Lambda expressions provide a very compact and type-safe way to write functions that can be passed as function arguments to another function for subsequent evaluation.

A Bit of history

In this section, we will look at the evolution in C# from delegates, to anonymous methods and finally to Lambda expressions. The code for this section can be found in the "SimpleExamples" project of the download code for this article.

In this "history" section, we will take a simple problem, which in this case is the indirect execution of a block  of code, and solve it using different methods:

• Delegates
• Anonymous Methods
• Lambda Expressions

Delegates

Since it inception C# and .NET 1.0 have supported the concept of a delegate. A delegate is basically a type-safe function pointer. Delegates specify a method to call and optionally an object to call the method on. Typically, delegate are used to implement callbacks and event listeners.

A simple example of a class that uses a delegate is shown below (this class is part of the code project mentioned earlier):

 1     /// <summary>
 2     /// This class invokes a method through a delegate
 3     /// </summary>
 4     public class ClassWithDelegate
 5     {
 6         // This is our delegate declaration
 7         private delegate void MyDelegate(int myInt, string myString);
 8 
 9         // This is the method which will invoke "Bar" through a delegate
10         public void InvokeMethodThroughDelegate()
11         {
12             MyDelegate myDelegate = new MyDelegate(Bar);
13             myDelegate(10, "test");
14         }
15 
16         private void Bar(int anInt, string aString)
17         {
18             Console.WriteLine("\tThis is a method invoked through a delegate, params: {0} and {1}",
19                 anInt, aString);
20         }
21 
22     } // class ClassWithDelegate

First, on line 7 we have our delegate declaration, which basically specifies the name of our delegate and the signature of the to-be invoked method. The method we want to invoke is Bar, the body of which is specified from line 16 through line 20 in the code.

In line 12, notice how we have to "new up" a new instance of our delegate, specifying the name of the method (Bar) as an argument. In the next line, we can then call the method through the delegate, passing in the function arguments.

Note that delegates are type safe, so if at line 12, I specified a method (say Foo) as the new argument, with a signature that did not match the signature specified by the  MyDelegate specification, I would have received a compiler error:

Error    1    No overload for 'Foo' matches delegate     SimpleExamples

While delegates definitely have their use, and will continue to serve us well in the future, in the above example the code is pretty verbose. Basically, all we want to do is execute a block of code, but I need my delegate declaration, my Bar method, I need to new up a Delegate instance before I finally can invoked the code through the delegate. Isn't there a better way? Yes, there is.. enter: "Anonymous Methods"!

Anonymous Methods

An anonymous method is a new feature in C# 2.0 that lets you define an anonymous (that is, nameless) method that can be called by a delegate. Just like a regular method, a anonymous method can take any number of parameters. An anonymous method is defined inline, and not as a member of any class.

Anonymous methods can be used anywhere that a delegate is expected.  You can pass an anonymous method into any method that accepts the appropriate delegate type as a parameter.

Going back to our example, here is the code for the "ClassWithAnonMethod" class with our sample project:

 1     public class ClassWithAnonMethod
 2     {
 3         // This is the delegate declaration
 4         private delegate void MyDelegate(int myInt, string myString);
 5 
 6         // This is the method which will calls an anonymous method
 7         // through a delegate
 8         public void InvokeMethodThroughAnonMethod()
 9         {
10             MyDelegate myDelegate = delegate(int anInt, string aString)
11             {
12                 Console.WriteLine("\tThis is the anonymous method, params: {0} and {1}",
13                     anInt, aString);
14             };
15 
16             myDelegate(10, "test");
17         }
18 
19     } // class ClassWithAnonMethod
20 

As you can see, our delegate declaration is still there, but our "Bar" method has now been replaced with an anonymous method, defined in lines 12 and 13. The anonymous method itself is still invoked through the delegate in the same way at line 16. This code is a lot more compact and easier to read in some situations (as long as the code in the anonymous method is limited to a couple of lines of code).

A Zen moment...

My last comment (about anonymous functions) again makes me think that it would be a good idea to write a closure post at the end of this series, where we do kind of a "reset", and look at where it would be appropriate to use each feature mentioned in this series. In the programming world, you basically have two types of people (I am over-simplifying here, I know):

1. The first type are the folks who love all of this "new stuff", and would go out of their way to use every feature possible.
2. The second group of folks just wants to get  the job done. From these folks you often hear comments like: 'this is very cool stuff, but it just need to get this project done, and this stuff is not going to help me get there any faster!"

I think both groups of people have a point, and the "sweet spot" is probably somewhere in between both opinions. I don't see myself driving to work every day, thinking: "Today I am going to write a killer lamba expression tree, and create a cool new query syntax" ;-), but I can see a use for each of the features we talk about so far, as long as it is applied in the right context. And of course, when it comes to LINQ, DLINQ and XLINQ itself, I think those feature will indeed change the way we work, and they will be incredible time savers indeed!

Ok, that was my short "Zen moment", sorry for the interruption, back to Lambda expressions" now!

Lambda Expressions

Lambda expression provide a more concise, functional syntax for writing an anonymous method. Compared to anonymous methods they provide a very compact and more type-safe way to write small functions that can be passed as arguments for subsequent evaluation. The Select, Where, Sort etc. clauses in LINQ are basically all written as Lambda expressions.

Below is a small code sample. This is the "ClassWithLambda" class from out sample project:

 1     public class ClassWithLambda
 2     {
 3         // This is the delegate declaration
 4         private delegate void MyDelegate(int myInt, string myString);
 5 
 6         public void InvokeMethodWithLambdaBLOCKED EXPRESSION
 7         {
 8             // Now our delegate uses a Lambda expression
 9             MyDelegate myDelegate = 
10                 (value1, value2) => Console.WriteLine(
11                     "\tThis is lambda expression, params: {0} and {1}", value1, value2);
12             
13             myDelegate(10, "test");
14         }

Note that we still have our Delegate declaration at line 4. The Lambda expression itself is on lines 10 and 11.

In C# 3.0 a lambda expression is syntactically written as a parameter list, followed by a =>token, and then followed by the expression or statement block to execute when the expression is invoked. In it simplest form, a lambda expression can look like this:

params => expression

You can omit the parenthesis around the parameters when you only have one parameter.

In our case, our lambda expression looks like this:

(value1, value2) => Console.WriteLine(....)

Note that I did not bother to specify the types of both value1 and value2. Now, there is nothing that prohibits me from doing so, so I could have written:

(int value1, string value2) => Console.WriteLine(....)

In the first example, the compiler will use type inference to  find out what the types of passed-in parameters are, so the run-time overhead for both types will be exactly the same.

The expression you write at the right side of the => token can be any expression or statement block, and it can consume the passed-in parameters.

Now, you might look at this and say "big deal", now what does  this offer me that I cannot do with anonymous method? Well, let's take a look at that in the next section!

Using Lambda Expressions as Parameters to Functions

This this section, we will look at one of the most powerful features of lambda expressions, which is its ability to be passed as parameters to functions in a type-safe way with automatic type inference. During this section, we are assuming that we are creating a Math Library, and we are going to pass in a function that performs the actual math operation as a lambda expression. We will start out simple, and add more powerful features as we go along.

Using hard-coded argument types

Below is our first example:

 1         //  This function takes a lambda as parameter with hard-coded
 2         //  types for the parameters (in this case all integers)
 3         public int PerformMathFunction(Func<int, int, int> lambdaExpression, int val1, int val2)
 4         {
 5             int result = lambdaExpression(val1, val2);
 6             Console.WriteLine("\tPerformMathFunction, val1: {0}, val2: {1}, result: {2}",
 7                 val1, val2, result);
 8 
 9             return result;
10         }

the parameter:

Func<int, int, int> lambdaExpression

specifies that we are passing in a lambda expression, which takes two integers (the first two parameters are integers), and returns an integer. In a Func specification like this, the last type is always the return type. So far example an expression like this:

Func<int, double> lambdaExpression

specifies that we are passing in a lambda expression that takes one parameter that is an integer, and returns a double, or even simpler:

Func<decimal> lambdaExpression

specifies that we are passing in a lambda expression that returns a double.

We are invoking the passed-in lambda expression as follows:

int result = lambdaExpression(val1, val2);

In this case we know that our lambda expression is returning an integer, so we can assign the result of our expression to an integer. The rest of the above method simply prints our the passed-in values and the result, and returns the result.

Below is an example of how you invoke the PerformMathFunction() method:

1             // In the remainder of this method, we show how you can pass lamda
2             // expressions as arguments to a method 
3             ClassWithLambdaExpressionAsParam paramClass =
4                 new ClassWithLambdaExpressionAsParam();
5 
6             // Simple examples without type inference
7             paramClass.PerformMathFunction((int val1, int val2) => val1 + val2, 10, 20);
8             paramClass.PerformMathFunction((int val1, int val2) => val1 * val2, 10, 20);

The lambda expressions that we pass in are:

(int val1, int val2) => val1 + val2    -and-

(int val1, int val2) => val1 * val2   

The above example produces the following output:

PerformMathFunction, val1: 10, val2: 20, result: 30
PerformMathFunction, val1: 10, val2: 20, result: 200

While the above example nicely illustrates how we can pass in a lambda expressions into a method, it has one main drawback: It only works with arguments of type Integer. In the next example, we will see how we can get around this.

Using Generic arguments Types

Below is a new version of our method:

 1         //  This function uses Type Inference ("T"), but forces all parameters to be of the
 2         //  same type, so both math parameters and the return type are of the same Type T
 3         public T PerformMathFunctionWithTypeInference<T>(Func<T, T, T> lambdaExpression, T val1, T val2)
 4         {
 5             T result = lambdaExpression(val1, val2);
 6             Console.WriteLine("\tPerformMathFunctionWithTypeInference, val1: {0}, val2: {1}, result: {2}",
 7                 val1, val2, result);
 8 
 9             Console.WriteLine("\t\tType of T: {0}", typeof(T).Name);
10 
11             return result;
12         }

Now, our passed-in lambda expression has changed to:

Func<T, T, T> lambdaExpression

Now, we are using a generically typed parameter, and the compiler will use type interference to determine what the type of T is. Note that we used the same type "T" everywhere, so we expect both input arguments and the return type to be of the same type. Compared to our previous example, our method has become a lot more flexible though, since we now can pass in an argument of any type, and the compiler will do the work of figuring out what the type is.

Note also that since our Method PerformMathFunctionWithTypeInference() now basically is a generic method, we need to specify the generic parameter <T> as part of its name:

public T PerformMathFunctionWithTypeInference<T>(...)

Note that our method also prints out the name of the type "T", so in our output, we will be able to verify that the compiler indeed inferred the correct type.

So, if we call this method as follows:

1             // In the remainder of this method, we show how you can pass lamda
2             // expressions as arguments to a method 
3             ClassWithLambdaExpressionAsParam paramClass =
4                 new ClassWithLambdaExpressionAsParam();
5 
6            // Advanced examples with type inference
7             paramClass.PerformMathFunctionWithTypeInference((val1, val2) => val1 / val2, 200.0, 5.0);

Our output will look as follows:

 PerformMathFunctionWithTypeInference, val1: 200, val2: 5, result: 40
         Type of T: Double

Because we passed in variables 200.0 and 5.0, the compiler correctly inferred that the type of T is indeed Double.

In our next example, we will take this example one step further, and vary the output type of the result.

Using different Generic types for the Arguments

As a next step, we will extend our example once more, and allow the return type of our lambda function to be different then the type of the passed-in parameters.  The code for the example is shown below:

 1         // This function also takes a lambda parameters, but it allows the caller to specify a
 2         // different type for the return type of the lambda (type "R"). Both parameters are expected
 3         // to be of the same type "T"
 4         public R PerformMathFunctionGenericFlexReturnType<R, T>(Func<T, T, R> lambdaExpression, T val1, T val2)
 5         {
 6             R result = lambdaExpression(val1, val2);
 7             Console.WriteLine("\tPerformMathFunctionGenericFlexReturnType, val1: {0}, val2: {1}, result: {2}",
 8                 val1, val2, result);
 9 
10             Console.WriteLine("\t\tType of T: {0}, Type of R: {1}", typeof(T).Name, typeof(R).Name);
11 
12             return result;
13         }

Notice the signature of our lambda expression:

Func<T, T, R> lambdaExpression

Indicating that the return type of our lambda expression is now the generic type "R", while the input parameters have the generic type "T". Notice also how we had to add the additional generic type parameter to our function signature:

public R PerformMathFunctionGenericFlexReturnType<R, T>(....)

In our function, we can now  use the return type "R" as the return type of the lambda expression. In this case, we also selected to print out both the name of the types of "T" and "R". An example invocation method is shown below:

1             // In the remainder of this method, we show how you can pass lamda
2             // expressions as arguments to a method 
3             ClassWithLambdaExpressionAsParam paramClass =
4                 new ClassWithLambdaExpressionAsParam();
5 
6            paramClass.PerformMathFunctionGenericFlexReturnType((val1, val2) => (int)(val1 / val2), 5.0, 2.0);

So, our lambda expression is in this case:

(val1, val2) => (int)(val1 / val2)

When we run this sample, we get the following output:

PerformMathFunctionGenericFlexReturnType, val1: 5, val2: 2, result: 2
        Type of T: Double, Type of R: Int32

Because our passed-in lambda expression did a cast of the result to integer, the compiler correctly inferred that the type of "R" is indeed System.Int32.

Lambda Expressions and Extension Methods

Introduction

We talked about extension methods in part 4 of this series. Extension methods can basically be defined for any type, and long as the type itself is not static. We you combine Lambda expressions with extension methods, you get a very powerful brew. It is fact this combination that is at the very core of the query syntax of LINQ.

Microsoft has create a large number of extension methods for the IEnumberable<T> type. IEnumerable<T> is a logical choice since this interface is supported by most generics collections in C#. All of these extensions are defined  in the namespace System.Linq, in the static class Enumerable:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Runtime.CompilerServices;

namespace System.Linq
{
    // Summary:
    //     Provides a set of static (Shared in Visual Basic) methods for querying objects
    //     that implement System.Collections.Generic.IEnumerable<T>.
    public static class Enumerable
    {
        // Summary:
        //     Applies an accumulator function over a sequence.
        //
        // Parameters:
        //   source:
        //     An System.Collections.Generic.IEnumerable<T> to aggregate over.
        //
        // Parameters:
        //
        //   func:
        //     An accumulator function to be invoked on each element.
        //
        // Type parameters:
        //   TSource:
        //     The type of the elements of source.
        //
        // Returns:
        //     The final accumulator value.
        //
        // Exceptions:
        //   System.ArgumentNullException:
        //     source or func is null.
        //
        //   System.InvalidOperationException:
        //     source contains no elements.
        public static TSource Aggregate<TSource>(this IEnumerable<TSource> source, Func<TSource, TSource, TSource> func);
        ......
       }
}

Now, I wanted to find out how many extensions methods there were exactly, and give the reader some idea of how many different overloads there were for every method, but instead of doing this manual, I decided to write a little LINQ query (you know I had to  ;-)!. This is the code:

 1         public static void DetermineNumberOfExtensionMethodsForIEnumerable()
 2         {
 3             // Write the total number of methods
 4             Console.WriteLine(
 5                 "Total number of methods in System.Linq.Enumerable: {0}",
 6                     typeof(System.Linq.Enumerable).GetMethods().Length);
 7 
 8             // This query gets the name and number of overloaded methods
 9             var overloads =
10                 from method in typeof(System.Linq.Enumerable).GetMethods()
11                 where method.IsStatic == true
12                 group method by method.Name into m
13                 orderby m.Key
14                 select new
15                 {
16                     MethodName = m.Key,
17                     OverloadCount = m.Count()
18                 };
19 
20             Console.WriteLine(Environment.NewLine);
21 
22             // Write out the overloads
23             foreach (var overload in overloads)
24             {
25                 Console.WriteLine(
26                     string.Format("Method: '{0}', # of overloads: {1}",
27                     overload.MethodName, overload.OverloadCount));
28             }
29 
30         } // method DetermineNumberOfExtensionMethodsForIEnumerable
31