EntityFramework – Grouping by Date Ranges

If you’ve ever created an outstanding balance report or other report that deals with aggregating data into date ranges, you’ll know that it isn’t immediately obvious how to structure your query, whether using SQL or LINQ (at least, it wasn’t to me).

My initial thought was to run multiple queries (one for each time range) and munge the results together. However, an elegant solution is to use SQL’s CASE expression to group date ranges together.

Let’s say you wanted a report that summed the amount of unpaid invoices in 20 day groupings (0-19 days past due, 20-39 past due, 40+). You could write something like this:

SELECT DaysSinceDueRange, SUM(Amount)
FROM (SELECT CASE WHEN GETDATE() - DueDate < 20 THEN 0
		  WHEN GETDATE() - DueDate BETWEEN 20 AND 39 THEN 20
		  WHEN GETDATE() - DueDate > 39 THEN 40
             END AS DaysSinceDueRange,
             Amount
       FROM Invoices
       WHERE Unpaid=1) inv
GROUP BY DaysSinceDueRange

This is really elegant, but then the question becomes how to do this with an ORM like EntityFramework. There are a couple of tricks required here:

  1. To do the date comparisons, EntityFramework requires the usage of System.Data.Entity.DbFunctions.DiffDays method (in EF 6 – it used to be in System.Data.Objects.EntityFunctions). If you try to do something like (DateTime.Now – invoice.DueDate).TotalDays, you’ll get an exception “DbArithmeticExpression arguments must have a numeric common type” because the subtraction operator is not defined for Dates in SqlServer.
  2. To do CASE / WHEN / THEN / END in EntityFramework, you have to make use of a lot of ternary operators. It can be kind of ugly, but if you write your code well enough, it should be fairly readable (or at least as readable as the SQL expression).

Here is an example of the SQL above translated into LINQ:

Context.Set<Invoice>()
       .Where(inv => inv.Unpaid)
       .Select(inv => new
       {
           DaysSinceDueRange = DbFunctions.DiffDays(inv.DueDate, DateTime.Today) < 20 ? 0 :
                               DbFunctions.DiffDays(inv.DueDate, DateTime.Today) >= 20 && 
                                   DbFunctions.DiffDays(inv.DueDate, DateTime.Today) < 40 ? 20 : 
                               40,
           Amount = inv.Amount
       }).GroupBy(inv => inv.DaysSinceDueRange)
       .Select(g => new
       {
           DaysSinceDueRange = g.Key.DaysSinceDueRange,
           Amount = g.Sum(inv => inv.Amount)
       });

Of course, you can get more complicated in a hurry, but I think this is a pretty elegant way to handle grouping data by date ranges.

EntityFramework Performance and IEnumerable vs IQueryable

Working in the .Net world, you get pretty used to dealing with IEnumerable collections. However, you have to be aware of performance issues that can arise when using them with EntityFramework. Sometimes I forget about IQueryable because LINQ to Entities obfuscates much of the difference of retrieving objects from a database vs dealing with an in-memory collection and IQueryable is pretty specific to dealing with querying a database.

When using the Repository pattern, one of the things I love to do is add a flexible “Find” method to the repository. Below is an example:

public partial class Order
{
    public int ID { get; set; }
    public string CustomerItemNumber { get; set; }
    public DateTime? ShipDate { get; set; }
    public int OrderTypeID { get; set; }
}

public class FindOrdersReqeust
{
    public IEnumerable<int> OrderIDs { get;set; }
    public IEnumerable<string> CustomerOrderNumbers { get; set; }
    public IEnumerable<int> OrderTypeIDs { get; set; }
    public bool? HasShipDate { get; set; }
    public DateTime? ShipDateBefore { get; set; }
    public DateTime? ShipDateAfter { get; set; }
    
    public FindOrdersReqeust()
    {
        OrderIDs = new List<int>();
        CustomerOrderNumbers = new List<string>();
        OrderTypeIDs = new List<int>();
    }
}

//I would normally implement an interface here, but for the sake of brevity am excluding from this example
public class OrderRepository
{
    private IDbContext context;
    public OrderRepository(IDbContext context)
    {
        this.context = context;
    }

    public IEnumerable<Order> Find(FindOrdersRequest request)
    {
        IEnumerable<Order> orders = context.Set<Order>().AsEnumerable();
        if(request.OrderIDs.Any())
        {
            orders = orders.Where(o => request.OrderIDs.Contains(o.ID));
        }
        if(request.CustomerOrderNumbers.Any())
        {
            orders = orders.Where(o => request.CustomerOrderNumbers.Any(x => o.CustomerOrderNumber.Equals(x)));
        }
        if(request.OrderTypeIDs.Any())
        {
            orders = orders.Where(o => request.OrderTypeIDs.Contains(o.OrderTypeID));
        } 
        if(request.ShipDateAfter.HasValue)
        {
            orders = orders.Where(o => o.ShipDate.HasValue && o.ShipDate >= request.ShipDateAfter);
        }
        if (request.ShipDateBefore.HasValue)
        {
            orders = orders.Where(o => o.ShipDate.HasValue && o.ShipDate <= request.ShipDateBefore);
        }
        if(request.HasShipDate.HasValue)
        {
            orders = orders.Where(o => o.ShipDate.HasValue == reqeust.HasShipDate.Value);
        }

        return orders;
    }
}

public class OrderService
{
    private OrderRepository orderRepository;
    public OrderService(OrderRepository orderRepository)
    {
        this.orderRepository = orderRepository;
    }

    public void GetUnshippedOrders()
    {
        return orderRepository.Find(new FindOrdersRequest()
        {
            HasShipDate = false
        }).ToList();
    }
}

The major problem with this code is highlighted in the example above – namely that AsEnumerable() call on the context.Set<T> will enumerate every row from the database in full. The Sql generated will be equivalent to a SELECT *, and will probably look something like:

SELECT [Extent1].[ID] AS [ID], 
    [Extent1].[CustomerOrderNumber] AS [CustomerOrderNumber], 
    [Extent1].[ShipDate] AS [ShipDate]
    FROM [dbo].[Orders] AS [Extent1]

Now, you might not notice if you have 10 rows, but if you have 1,000,000, you’ll notice as your application burns to the ground and consumes all the memory on whatever server it’s running on.

So, an easy fix is to change that one line to IQueryable / AsQueryable() like so:

IQueryable<Order> orders = context.Set<Order>().AsQueryable();

Now we get the benefits of deferred execution until ToList is called on the results of the Find method from OrderRespository. The SQL generated will now be something like:

SELECT [Extent1].[ID] AS [ID], 
    [Extent1].[CustomerOrderNumber] AS [CustomerOrderNumber], 
    [Extent1].[ShipDate] AS [ShipDate],
    [Extent1].[OrderTypeID] AS [OrderTypeID]
    FROM [dbo].[Orders] AS [Extent1]
    WHERE [Extent1].[ShipDate] IS NULL

This is a huge improvement already, but it can be much better than this. In the OrderRepository class, if we also make the return type IQueryable, we can then further query the database before pulling the results into memory.

public IQueryable<Orders> Find(FindOrdersRequest request)
{
    //the rest of the method remains the same
}

This distinction is important, and I will provide an example.

After I add this “Find” functionality to my repositories, I tend to build reports using those methods, which frequently utilize .GroupBy() after filtering. If we were to leave the Find method as returning an IEnumerable<Order> collection, we would find that the SQL generated would not be what we wanted.

For example, let’s say I now wanted a report that showed the number of shipped and unshipped orders by OrderTypeID. I would add a method to the OrderService as such:

//assume that an object ReportItem exists with properties as defined below
public IEnumerable<ReportItem> GetUnshippedOrdersByTypeReport()
{
    var report = new List<ReportItem>();
    var results = orderRepository.Find(new FindOrdersRequest(){ HasShipDate = false })
                                 .GroupBy(order => new 
                                 {
                                     OrderTypeID: order.OrderTypeID
                                 })
                                 .Select(g => new
                                 {
                                     OrderTypeID: g.Key.ID,
                                     ShippedOrderCount: g.Sum(x => x.ShipDate.HasValue),
                                     UnshippedOrderCount: g.Sum(x => !x.ShipDate.HasValue)   
                                 });
    foreach(var result in results)
    {
        report.Add(new ReportItem()
        {
            OrderTypeID: result.OrderTypeID,
            ShippedOrderCount: result.ShippedOrderCount,
            UnshippedOrderCount: result.UnshippedOrderCount,
        });
    }
   
    return report;
}

With this code, the benefits of using IQueryable in the repository are clear over IEnumerable.

If we leave the repository with IEnumerable, the SQL generated will be done in two phases:

  1. The filtering of the “FindOrdersRequest” will be executed as the SQL statement above and the results will be stored into a temporary IEnumerable collection
  2. The Group By operation will operate on this temporary collection. More trips to the database will be taken if any navigation properties are referenced in the GroupBy (there are none in this example)

If we change the repository to use IQueryable, the SQL generated will be done in a single, neat statement. It will filter and perform the group by at once, resulting in much better performance. Another performance benefit is that we are projecting specific columns and not populating an entire Order object with every field. For this trivial example, it doesn’t make much of a difference, but if you’re dealing with tables/objects that have many columns/properties, you will notice. If you’re deploying to a cloud-based environment, you know that compute time and efficiency matter a lot, so following best practices for performance will help you out a lot in that respect.

 

Asp.Net Core appsettings tips

I’ve recently had the opportunity to work on a new project where I was able to use Asp.Net Core for the first time. Well, not completely – I’ve contributed to an open source project that has been using .Net Core for some time, when it was called DNX or ASPNET 5. Anyway, the work I did there really was focused on writing code for the application, not configuring the infrastructure.

A lot has changed, but the changes are largely for the better. There are a few things that tripped me up, so I figured I’d write about them here.

AppSettings have gone JSON

This in and of itself isn’t much of a revelation, but I, for one, am glad to have JSON configuration over XML. In the Startup.cs file, appsettings are configured by default as such:

public Startup(IHostingEnvironment env)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddEnvironmentVariables();
    Configuration = builder.Build();
}

Just like before, when we had Web.config, Web.Release.Config, Web.{EnvironmentName}.config, any environment configuration will be applied on top of the rules defined appsettings.json file. So, if you have an appsettings.json file that looks like:

{
    "MyVariable1": "value1",
    "MyVariable2": "value2"
}

and then you define a file appsettings.production.json that looks like:

{
    "MyVariable1": "productionValue",
}

The production file’s value will be used for MyVariable1 when the application is running in a production environment, as expected.

Accessing appsettings

The easiest way to access a value from your appsettings file is to use Configuration.GetValue:

Configuration.GetValue("MyVariable1", "");

The above will retrieve the value for MyVariable1 or an empty string if there is no key found for MyVariable. The nice thing is you don’t get an exception if a key isn’t found, but this could be an issue if you were expecting a key and get the default value instead.

If your appsettings file has nested objects like this:

{
    "Logging": {
        "IncludeScopes": false,
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
        }
    }
}

you can retrieve values by using Configuration.GetValue(“Logging:LogLevel:Default”);

Personally, I don’t like to use magic strings – I prefer to use a strongly typed configuration.

Strongly Typed appsettings

Rick Strahl has a very good article about Strongly typed appsettings, but I will cover the basics. In a nutshell, you need to do two steps to make this work:

  1. Create a class that has all of the corresponding properties of your appsettings (or just a subsection of your appsettings, as I will show below)
  2. Wire up your class by calling the services.Configure<T> method In the ConfigureServices method of your Startup.cs class

Let’s use the following appsettings.json file as an example:

{
    "MySettings" : {
        "AdminEmail" : "admin@email.com",
        "ErrorPath" : "/Home/Error"
    }
}

All we need to complete step 1 is to have a corresponding class for these settings. Here is the corresponding example:

public class MySettings
{
    public string AdminEmail { get; set; }
    public string ErrorPath { get; set; }
}

Now, in our Startup.cs class, we can add the following to our ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    services.Configure<MySettings>(Configuration.GetSection($"{nameof(MySettings)}"));
}

That’s it. Now, we can simply inject MySettings into our MVC/WebAPI controller constructors and Web API will be able to inject that dependency for us.

Note that in this example we called Configuration.GetSection and gave it the name of our section/class – if you only listed the keys AdminEmail and ErrorPath at the root of the appsettings file (without any nested objects), you could have done the same by calling just services.Configure<MySettings>(Configuration);

Using appsettings in your Startup.cs class

One gotcha that had me stumped for a little while was trying to use some of my appsettings configurations to provide configurations in my Startup.cs class. The trick here is using the Bind method on configuration. Here is a good example of what I mean: a lot of tutorials and examples will show configuring exception handling as:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    app.UseExceptionHandler("/Home/Error");
}

I like to make that route configurable in my appsettings, so here is how to do that:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    var mySettings = new MySettings();
    Configuration.GetSection($"{nameof(MySettings)}").Bind(mySettings);

    app.UseExceptionHandler(mySettings.ErrorPath);
}

Sending Labels to a Thermal Printer using POST Requests

If you’ve ever used a Thermal printer on a website like Fedex.com, you know it can be kind of ugly to get your label to print. As of this writing, FedEx.com still relies on a Java plugin to do the dirty work of sending the data to your locally-connected printer. I have a couple of applications that integrate with FedEx web services and in the past, I too have relied on Java applets (jZebra, which became qzPrint) to do the work of sending a print job via a web browser.

After searching around, I found a better way to do it: use a simple POST request to send to a network printer. This article is what really got me started: Label And Receipt Printing – Printing from Websites part 2

The sample code shows how you can create a simple XmlHttpRequest object and send it the EPL/ZPL you want. Here is a barebones sample:

var zpl = "^XA^PW400^LL200^FO20,20^A0N,30,30^FDTest^FS^XZ"; //some zpl to send to the printer
var zebraPrinterUrl = "http://192.168.0.100/pstprnt"; //ip address of the printer
var request = new XMLHttpRequest();
request.onload = function () {
  //take some action
};
request.onerror = function () {
  //take some action
};
request.open("POST", zebraPrinterUrl, true); 
request.setRequestHeader("Content-Length", zpl.length);
request.send(zpl);

That’s really all there is to it.

A couple of caveats – as the article I linked above notes, CORS can be a bit of a problem. The zebra printer does not return the necessary Access-Control-Allow-Origin header, so I found this to be disruptive when I tried to use the AngularJS (1.0) $http service. Sending the post using $http would result in a response coming back with status 0, which indicates a CORS problem.

Therefore, I ended up using the XMLHttpRequest object in my application, which can significantly impact testability, unless you wrap the instantiation of XMLHttpRequest objects in another component that you inject into your controller or service.

Finally, because I’m using promises and asynchronous requests, I had to make sure that all of the labels being printed complete before resolving or rejecting the promise.

This solution obviously isn’t very scalable, but I do think it provides much greater flexibility than the old way of using java or some browser plugin to do the job.

Changing from Underscore to Lodash

I have read in a few places that lodash is the way to go (over underscore) when it comes to javascript collection manipulation libraries, but I hadn’t gotten around to changing it out on any of my applications using underscore until recently. I also had a conversation with one of the contributors to the excellent moment js library who told me that lodash is the way forward, and there are some posts out there on the internet that suggest this is the way to go.

Doing the upgrade, I did find there are a couple of functions that aren’t supported by lodash that are by underscore. The list I have found is below:

  • pluck (use map instead – it appears this was changed sometime in January with the release of 4.0: http://stackoverflow.com/questions/35136306/what-happened-to-lodash-pluck)
  • any (use some instead – I had used any because it is operates much like the LINQ Any method)

I will add more here as I come across them.

Local Web Development with Adobe Typekit

My bread and butter is really front-end scripting (JavaScript) and server side technologies (c#, WebApi, MVC, etc), so I feel like any time I really venture into the world of design, I learn something new — which is great!

While working with a designer on a new website, she requested we use Adobe Typekit to make better use of some nicer fonts on the web. Using great looking fonts has long been a challenge, but Typekit seems to offer a pretty slick solution to give you better control. One quick aside, I noticed that my Ghostery plugin doesn’t really like Typekit fonts and by default blocks typekit scripts. So be aware that users may not see your glorious fonts anyway if you use Typekit and they have Ghostery installed. If you want to know more about Ghostery and Typekit, here is a resource for you.

Using Typekit on Your Webpage

Now, in general, Adobe has made it pretty simple to use Typekit with a website. The basic steps are as follows:

  1. Create a “Kit”. On typekit.com (after logging in), there is a menu item called “<> Kits” that has an option “+ Create new kit”. A kit is a grouping of fonts you would like to use on your website. Once created, this kit will be assigned a unique ID that will be referenced in the Javascript Adobe provides for you to embed in your site.
  2. Add some fonts to your kit. Navigate to your library and find a font you like. Select that font and you will be given an option to “Add to Kit” (there is a button that reads “Web use: Add to Kit”). Once you have added all the fonts and variants to your kit, you are ready for the next step
  3. Add domain(s) to your kit settings. To do this, navigate back to your kit (using the “<> Kits” dropdown menu in the header) and choose “Kit settings”. From here, you can specify any domains where you will be using this kit (among other options). For development purposes, you can enter “localhost”. In theory this is all you have to do to enable Typekit to work with these domains. I found some obstacles in that, hover. Be sure to save your changes.
  4. Publish your kit. After everything is correct in your settings, go back to your kit page and click the publish button in the lower right corner. It says it can take a couple of minutes for changes to become active, so be a little patient.
  5. Grab the embed code (JavaScript) and put it into your site’s head. Right next to where you updated the settings, you can see a “Embed Code” link that will present a popup with your default code that should look something like this:
    <script src="https://use.typekit.net/myKitId.js"></script>
    <script>try{Typekit.load({ async: true });}catch(e){}</script>

    Where the javascript file “myKitId.js” is replaced with whatever kit ID was assigned to your kit.

In theory, this is where everything should just work. All you need to do is use your fonts in CSS as they are named (all lowercase, replacing spaces with hyphens). An example of a page is as follows:

<html>
    <head>
        <script src="https://use.typekit.net/myKitId.js"></script>
        <script>try{Typekit.load({ async: true });}catch(e){}</script>
        <style type="text/css">
            body { font-family: sans-serif; }
            .myFont { font-family: "futura-pt"; }
        </style>
    </head>
    <body>
        <p>This text should have font-family sans-serif</p>
        <p class="myFont">This text should have font-family futura-pt.</p>
    </body>
</html>

Of course, you have to replace font-family with whatever fonts are defined in your kit (and replace the Kit ID), but you should see those two sentences with different fonts. If you don’t, then something isn’t right.

Figuring Out How Typekit Loads Fonts

I saw the fonts load correctly on my public domain, but they wouldn’t show on my development machine (using localhost).

I noticed there was a “Show Advanced” link¬†on Typekit’s “Embed Code” page, which displayed a self-executing JavaScript function that had the added benefit of asynchronous loading. All this “advanced” code really does is load an external JS file into a script tag inserted into the head and provides some CSS classes to hide some page flicker before the Typekit fonts are displayed. There is a pretty good explanation of the differences between the default and advanced Embed code here: https://helpx.adobe.com/typekit/using/embed-codes.html. Unfortunately, this didn’t have anything to do with the problem at hand.

Now, to really figure out why this wasn’t working, I pulled up Firebug and took a look at the HTTP requests in the “Net” tab. I could see that the request to my Typekit JavaScript file wasn’t completing. While I couldn’t figure out why, I tried pasting the url of the request into my browser, and it loaded the Javascript up for me. The javascript displayed was compressed and pretty hard to interpret. I decided to save it and load it locally to see if it would make a difference.

Unfortunately, still no luck. But what I could see now, is that inline CSS was being added to the head of my document. The CSS looked like the below:

@font-face {
    font-family: "futura-pt";
    font-style: normal;
    font-weight: 400;
    src: url("urlToFont") format("woff2");
}
@font-face {
    font-family: "futura-pt";
    font-style: normal;
    font-weight: 300;
    src: url("urlToFont") format("woff2");
}

Storing the Fonts Locally and Serving them Up

It was at this point that I knew I could work around Typekit’s issues. By navigating to the urls in src attribute of the @font-face selectors, I was presented with a file to download. The file downloaded as just “l” (no extension). I guess the file extension would be .woff since the format listed in the CSS file was “woff2”. After downloading each font file and renaming them to something that represented what the font was (“futura-pt-book.woff” and “futura-pt-light.woff”), I put them in a “fonts” directory in my development folder.

Next, I created a fonts-local.css file that contained the CSS code that was being generated by Typekit (see the code snippet above) and stuffed a reference to that CSS file into the head of my document. I replaced the src: urls in the stylesheet with the path to the woff files in the directory I created above.

Finally, in my site template’s html head section, I removed the script calls to Typekit altogether and just referenced my local files.

This technique isn’t exactly ideal, because any changes I make to the kit won’t be propagated to this page (also, your web server has to be configured to serve woff files). However, it allows you to see your fonts locally during development if Adobe’s recommendations don’t work, and also gives you an option if you’re concerned about the whole Ghostery plugin / Adobe privacy concerns issue.

You might be asking, “Why didn’t you just download the .woff files from Typekit?” and skip all the in between, but the problem is that the .woff files don’t seem to be available for direct download. That’s probably because now I could take these woff files and use them with any site or distribute them (illegally), and I think they’re trying to control that kind of behavior as much as possible.