posts on m0sa.net

Roslyn Analyzer Performance

Fri, 02 Aug 2019 00:00:00 +0000

TL;DR

Do you write Roslyn analyzers? Ever wondered what impact they have on your compilation times? Here’s how you can get the stats!

To get the numbers one can use the ReportAnalyzer MSBuild property, that gets passed in to the C# / VB compilation tasks.It’s the MSBuild equivalent of the -reportanalyzer C# compiler command line switch.

I usually use it in combination with the -bl MSBuild switch, so I can inspect the output with the the excellent MSBuild Binary and Structured Log Viewer.

msbuild Your/Project.csproj /p:ReportAnalyzer=true /bl:out.binlog

After the build is done, you can find the relevant bits right after the Csc task

Backstory

I was looking into increased build times (see image above) after introducing some new analyzers, and I couldn’t find a good reference for this kind of investigation.

This is another Stack Overflow .NET Core migration war story.

As we’re working on porting Stack Overflow to .NET Core, there will be a temporary phase, where some of the applications in our codebase will run on ASP.NET MVC, while others will already be ported to ASP.NET Core (see my previous blog post).For our domain logic and data models this means that they will have to be compiled against both.The most simple way to achieve this is if they don’t depend on ASP.NET at all.We’ve worked hard on this decoupling but we were still left with the occasional IHtmlString reference, etc.Some ASP.NET references had to stay, but we didn’t want them to grow unwieldy.So we created Roslyn analyzers that check for ASP.NET usage, and fail the build if they find an usage that is not explicitly opted in (e.g. by using #pragma warning disable ASPNETUSAGE or via SuppressMessageAttribute).Developers that do feature work get a nice error at build time, and a squiggly in their editor, whenever they accidentally introduce a new ASP.NET reference into the shared library.This prevents them from stepping on the toes of the developers working on the port.

Learning about MSBuild Global Properties - The Hard Way

Mon, 25 Feb 2019 00:00:00 +0000

One of the first steps in the Stack Overflow .NET Core migration was to extract a Common library that would target netstandard.This new project would be referenced in both the old ASP.NET MVC 5 as well as the new ASP.NET Core projects, so that we could start moving the old projects one-by-one.We ended up targeting net462 for ASP.NET MVC, and netcoreapp2.2 for ASP.NET CORE.The new Common project had a few ASP.NET MVC / ASP.NET Core specific regions that were conditionally compiled via #if NET462 <MVC 5 CODE> #else <ASPNETCORE CODE> directives (thanks to the new IHtmlContent interface which replaced the old IHtmlString from System.Web, which we use across many models).The target framework moniker conditional defines are a feature of the new SDK-style builds in the .NET Core ecosystem.(Note, that switching on the TFM works for us because we don’t plan on running ASP.NET CORE on the full framework - it might not work for you…)

Everything built fine and dandy locally, but the fun started once we promoted our branch to a PR and the automatic build checks we had in place in GitHub / TeamCity kicked.No matter what we did, we only ever got into one branch of the conditional compilation; the other one was seemingly disregarded.My team spent around 2 days throwing WTFs around, and trying to figure out what was going on.Ultimately we started digging into the MSBuild code that powers the multi-targeting builds.

The magic code that introduces the target framework defines lives here, and at first glance everything seemed to be in order, and nobody from our team spotted what’s going on.So, I started comparing what our CI build does differently from our local build (which worked fine).Finally I managed to repro what’s going on, by including an additional -p:DefineContants=WHATEVER on the msbuild command-line, locally.Our CI PR verification build was passing a conditional compilation symbol in order to speed up the build a bit (we have one that enables us to localize for English only, which means a lot less IO)Next, I compared the MSBuild binlog (you can get one by running MSBuild with the -bl switch), and used the excellent MSBuild Binary and Structured Log Viewer where I searched for DefineConstants.The failing case turned out to short-circuit on whatever was passed in the command-line switch (e.g. WHATEVER), and never re-evaluated the property.Lucky for me, the binlog contained a hint as to why this is happening; the DefineConstants property was under a “global properties” node in the tree.Once I googled for “MSBuild global properties” I got to this piece of doc fairly quickly.

In the end the TIL was: if a property is a global property, and you pass it to MSBuild via command-line args, the value passed in is used, regardless of what your build scripts do.

I’ve came up with a trivial example to explain to the team, what was going on in MSBuild:

I’ve also opened up an issue in the DotNet SDK, since using a global property for the multi-targeting part makes the whole thing very brittle, and I’d expect some defensive programming around that.

Getting Stack Overflow Localization Tooling Ready for .NET Core

Wed, 21 Nov 2018 00:00:00 +0000

By popular demand, this is my writeup of our latest localization tooling refactoring at Stack Overflow

We're about to change all localization on Stack Overflow (and the entire network) to runtime instead of compile time. Let's hope we did this right.
— Nick Craver (@Nick_Craver) November 20, 2018

History

Lets talk about localization in general a bit first, though, to give you some background on how we ended up where here.

It’s a story full of tears, tech debt, and regret.

1. Careers / Talent

Your usual of-the-mill localization library works by having some sort of indirection (e.g. a key lookup) to get a string to be displayed e.g:

Translate(SomeIDForAString, parameters)

Such an approach already provides you with a place where all the original strings are stored.On the other hand, it’s much less friendly towards the person reading (or writing) the code.Ever since we started localizing products at Stack Overflow (which dates before my time), we’ve been using a different approach.One that is much more developer-friendly, and much easier to read.Let me give you an example:

_s("Hello $name$", new { name = GetCurrentUserName() })

This means that all of the strings that we use are directly in our code.

As a consequence, though, we need tooling that knows how to extract all the string templates (so we know what to give to translators).Historically, this tooling has been using Roslyn (the C# compiler, written in C#), since before it’s RTM.Originally, the tooling only did what I’ve highlighted above;it extracted all string templates used to call our localization library, and verified that all the replacement tokens were passed in as arguments.

Since we use ASP.NET MVC and Razor, it also had to extract strings from there, and in order to do that we had to generate C# from the .cshtml files.This turned out to also be the slowest step inside of aspnet_compiler.exe while precompiling the views (it had to re-do the same work we already did before).

If you’re interested in more background, you can read @mjibson’s Careers Localization series (part 1, part 2, part 3)

2. Stack Overflow

The real fun started when we had to localize Stack Overflow, where we care a bit more about perf, and we tried to avoid the long build times.

We figured out we can hook into Razor and affect what code it generates.This allowed us to attach attributes to the generated C# views.We only needed tooling to inspect attributes from all the classes from the precompiled .dll.What it also allowed us to do, was to generate the localization code (a switch statement that only required the current locale at runtime, everything else was codegen-ed) directly in the C# produced by Razor.For a while, on Stack Overflow, we put everything that we wanted to have localized inside Razor / .cshtml files.This is the original source of “precompile localization” idea.

But then, somewhere in the middle of localizing everything, we gave up.Turns out the perf hit of rendering a Razor view for every string you want to render, ever, was just to big.

We also needed to localize .cs files.Oh, and we also found a bunch of strings that needed new localization features, like multiple pluralization tokens, and the same string being used in different contexts that matter in some languages (e.g. Declension).

Enter Roslyn tooling, again, but with a twist!At that time, csc.exe was still the default C# compiler, and aspnet_compiler.exe was (and still is super slow).So we replaced both of them, with the Roslyn-based StackExchange.Precompilation, and to top it of, we made it extensible so it could handle all of our localization needs (both extracting the strings, as well as precompiling localization in both .cshtml and .cs files).

You can read more about this whole project here.

.NET Core

We currently still run on ASP.NET MVC 5, on the full framework. Our ultimate goal is to be able to run on .NET Core. ASP.NET Core is the first stepping stone towards that.

As we were evaluating ASP.NET vNext (== 5+), we had always hoped to roll with the metaprogramming features that were available in the previews, either there, or what was worked on in Roslyn generators.But the time to move to .NET Core and ASP.NET .NET Core is now, and unfortunately none of those features are around anymore…yet.Luckily, though, .NET tooling introduced some high-impact perf improvements that make the CLI tools much faster (e.g. dotnet build-servers).Additionally, ASP.NET Core Razor view pre-compilation is fast!

We evaluated the new ASP.NET Core localization features, but they seemed very allocate-y (it allocates an array every time, things get boxed, etc), and it doesn’t support some of the syntax edge cases we have.Considering this, and the fact that we’d have to rewrite all of our precious “_s&_m” code, we made the decision to keep rolling with our own localization framework.So we had to future-proof our tooling, while still maintaining backwards compatibility with our old MVC stack.

Present

Our localization needs are still the same:

1. No performance regressions

Our code-gen wasn’t perfect.It relied on the Roslyn SyntaxTree only, we didn’t use the semantic model much.A lot of the generated code was not optimal (e.g. we passed around all of the template properties as objects), but calling it was still way faster that having the entire Razor pipeline on the stack.This gave us some wiggle room for writing an optimized runtime implementation.And we managed to make the runtime implementation as fast as the old (albeit not very optimized) compile-time code-gen-ed localization.Kudos to @marcgravell, @Nick_Craver for helping out here. BenchMarkDotNet was also insanely helpful.

We're currently testing and optimizing our runtime localization approach on Stack Overflow (getting out of compile-time if we can). Luckily, sometimes the performance winners are very, very clear: pic.twitter.com/jDNJacTzcl
— Nick Craver (@Nick_Craver) November 13, 2018

2. Compile-time verification of string templates / parameters

We’ve already written Roslyn analyzers that did this.Running the whole compilation with ASP.NET MVC 5 took ~2 minutes on a beefy machine this was really a show stopper for developers. We wanted to give them instant-feedback inside VS, with squiglies and all - analyzers provide that for us.

3. Tooling to extract strings, but AspNetCore compatible

The only new requirement here is that everything it has to work with the .NET Core CLITurns out the analyzer mentioned above already has all the information we need!AspNetCore already pre-compiles .cshtml -> .cs as part of the build, those .cs files can be analyzer-ed.

The main part of the work there was to port all our builds and build tooling to something that can potentially run on .NET Core CLI in the future.The best candidate for that seemed to be an MSBuild ILogger implementation, which can intercept the diagnostic messages, pin-pointing them to specific locations within a source file, and spit out the artifacts we can send out for translation in the end.This is super application and build configuration specific, so this implementation lives in our Stack Overflow solution.

I’ve run into an interesting edge case with Roslyn vs MSBuild diagnostic levels while doing this.

I’ve skipped on one other important detail here, we have a separate tool that rewrites JavaScript, we just made it generate the same diagnostic messages as the C# tools.This allows us to run both that tool and the main build step in parallel, and get a unified view dump of all used strings in the end.

Other tidbits

Since we care about performance and a simple deployment model, we still don’t want to hit the DB on application spin-up.We were still able to get code-gen all of our existing translations / resources into the .dll into as part of the build.We did that by exposing a partial class in code, and code-gen it’s counterpart during build by executing a .csx file via dotnet-script.I did try to write a native MSBuild Task to do the same, but the fact that we have to hit a DB or HTTP endpoint as part of that made it impossible.I found this blog post by @natemcmaster very helpful while digging into that.

Future

With all of that in place, we now finally have a super-slim MoonSpeak library, that provides:

template syntax processing
- compile-time safe
- design-time feedback
- support multiple pluralization tokens per template (e.g.@_s('User $userName$ has asked #numQuestions# questions and #numAnswers# answers', new { numQuestions = model.QuestionCount, numAnswers = model.AnswerCount })
- built-in Markdown support@_m('hello [world](https://example.org)')
a super-fast runtime base implementation which can work on both MVC5, and ASP.NET Core
- the main issue here was to distinguish between the System.Web.IHtmlString (MVC5) and Microsoft.AspNetCore.Html.IHtmlContent (ASP.NET Core) interfaces, and their implementations (we have separate packages for each of them) for backing the _m implementation
C# string extraction via diagnostics

All the other tooling that generates the translator resources, and provide translated strings to the implementation at runtime, are up to the consuming app.Even internally, we have different ways of doing that (Talent has a different workflow and intermediate formats there vs. Q&A).

We might be finally getting to a point where we can open source MoonSpeak.

So long blogger

Thu, 05 Jul 2018 00:00:00 +0000

It’s been almost 10 years since my first blog posts, and I’ve totally neglected blogging in the past year.I totally blame blogger for that.I really want to blog more, so I’ve decided to move to a simpler setup, that’ll allow me to blog more quickly, with better tools.Nowadays I use VS Code for as many things as possible, and I’d really like to use it for blogging to. So I’ve decided to use a static page generator, that works with markdown.

Chosing a static page generator
In the past I’ve had experience with Jekyll, and I’ve also played around with various node-based tools, which are not very nice (coughnode_modulescough).Although I quite liked VuePress, it didn’t offer any nice blogging templates.In the end I decided to give hugo a try, and it’s was very simple, with a lot of nice themes (you wouldn’t like another designed-by-developer style, would you?).
Setting up HUGO and a theme
```
$> chocolatey install hugo -y$> hugo new site m0sa.github.io$> cd blog$/m0sa.github.io> git init$/m0sa.github.io> cd themes$/m0sa.github.io/themes> git submodule add https://github.com/avianto/hugo-kiera
```
Luckily it’s pretty easy to override stuff in the themes. The one I’ve picked, doesn’t have a rss <link> tag, and the blog post titles into the <title> in the header.So I’ve copied the partials/header from the theme into my layouts folder and tweaked it.
Get markdown content of my blogger posts
Luckilly I didn’t blog all to much (which is hopefully going to change now that I have this new super duper setup), so I didn’t have to batch import anything.I’ve used ATS’s html-to-markdown tool. I added them on the cookie blacklist in order to get more than a single post converted w/o having to register.The bulk of the work is adding markdown front matter, and making sure all the code is in there, formatted correctly, and that everything looks OK.
Set up hosting
Initially I wanted to go with github pages, but I went with netlify instead. It has all the bells and whistles (can force https + hsts with an auto generated let’s encrypt cert, CDN), and is more flexible later on (can easily add formst, AWS functions, etc), if I ever need it to be.
I followed the guide on the hugo homepage.
Point redirect from old blogger URLs to the new ones (not done yet…)
Injecting <script type="text/javascript">window.location = "<NEW-URL>";</script> into single posts should do the trick, as suggested by Bjørn Bråthen on SO.
Since I only pulled some (== non-crappy) blog posts over, I didn’t have to do this for everything.
Profit?

Roslyn Adventures: Metaprogramming with StackExchange.Precompilation

Sun, 24 Jan 2016 00:00:00 +0000

In this article we’ll wrap the[StringBuilderInterpolationOptimizer](https://gist.github.com/m0sa/f086bb0e9f62e05995c6#file-stringbuilderinterpolationoptimizer-cs) from my previousarticle intoan[StackExchange.Precompiltion](https://blog.stackoverflow.com/2015/07/announcing-stackexchange-precompilation/) module, and use it to optimize an existing C#project.

First things first, we start off with an empty console project, to which weadd some sample StringBuilder.Append calls, passing interpolated strings asparameters:

Now, let’s add theStackExchange.Precompilation.Buildpackage to our demo project, and make sure it still builds. This packagereplaces csc.exe with a custom and hookable compiler, which is basically awrapper around the roslyn API. Unfortunately, roslyn doesn’t expose any hooksfor adding custom processing from the command line(yet). Luckily the API let’syou do all kinds of crazy stuff, it even comes with a parser for the csc.execommand line arguments…

After this package is successfully installed, you should see the precompilerexecutable being called instead of csc.exe in the build output window:

Next, we need to build the compilation module that can be loaded into thecompilation. So we create an empty class library, and install theStackExchange.Precompilation.Metaprogrammingpackage (contains all the required base types).

After we’ve pulled in the SyntaxTree rewriter from the last article, we canimplement the hook. The interface we need to implement is ICompileModule. Wewant to implement the BeforeCompilation method, which allows us to inspectand modify our compilation before any IL is emitted. Inside this method wehave full access to all syntax trees in the compilation and their semanticmodel. Note that CSharpCompilation is immutable, so we have to replace theone in the compilation context with the modified one:

All that’s left to do now is to wire up our shiny new compile module, so it’spicked up as by the compiler. This is done in the app.config (or web.configfor ASP.NET project) file of the console application:

The modified build script forwards the project’s config file to the compiler(this is especially useful when it needs to actually precompile razor views,but this is out of scope of this post). Also, be sure to add theprecompilation module as a project reference to the console application.

After everything is set up correctly, a quick build and decompilation canverify that the emitted assembly contains optimized string builder calls:

[](https://lh3.googleusercontent.com/-zjm-pQfLE1s/VqQb_GwLFPI/AAAAAAAAH7I/Af-N2jpH5qk/s1600-h/image%25255B32%25255D.png)

The sources for this demo, along with commits reflecting the described steps,can be found on GitHub.

P.S. a useful trick for debugging your compilation module is to add aDebugger.Launch() call to attach your current VS instance to the currentlyexecuting compilation.

Roslyn Adventures: Optimizing StringBuilder string interpolation

Mon, 19 Oct 2015 00:00:00 +0000

C# string interpolation is awesome. But we can make it even more awesome bymaking it less wasteful. Consider this line of code:

new StringBuilder().Append($"foo {1}");

Currently Roslyn emits the following IL for this call:

IL_0001:  newobj      System.Text.StringBuilder..ctorIL_0006:  ldstr       "foo {0}"IL_000B:  ldc.i4.1IL_000C:  box         System.Int32IL_0011:  call        System.String.FormatIL_0016:  call        System.Text.StringBuilder.Append

You see what it does there? Let’s translate that back to C#, to make it moreobvious:

new StringBuilder().Append(string.Format("foo {0}", 1));

It [allocates](http://blog.marcgravell.com/2013/11/allocaction-allocation-allocation.html) another string and possibly evenanother StringBuilder.The thing is, you wouldn’t be using a StringBuilder if you weren’t concernedabout allocations. My initial idea of how to solve this wasFormattableString. So basically something like this:

class Foo{  public static void Bar(FormattableString str)  {    Console.WriteLine("FORMAT: " + str.Format, str.GetArguments());  }  public static void Bar(string str)  {    Console.WriteLine("STRING: " + str);  }  public static void Main()  {    Bar($"a test {42}");  }}

Unfortunately overload resolution doesn’t work in favor of the methodaccepting the FormattableString whenever there is an overload for the stringparameter. So the example above would write STRING: a test 42 to theconsole. If we could make the overload resolution smarter (e.g. make thecompiler create FormattableString instances wherever there’s an matchingoverload accepting FormattableString instead of a string argument) thesolution would be as easy as creating Append/AppendLine(FormattableString)extension methods for StringBuilder.

Roslyn to the rescue

Luckily we can use Roslyn to do some metaprogramming magic and work aroundthis. Basically we need to rewrite StringBuilder.Append/AppendLine calls toStringBuilder.AppendFormat:

new StringBuilder().AppendFormat("foo {0}", 1);

… in IL speak:

IL_0001:  newobj      System.Text.StringBuilder..ctorIL_0006:  ldstr       "foo {0}"IL_000B:  ldc.i4.1IL_000C:  box         System.Int32IL_0011:  call        System.Text.StringBuilder.AppendFormat

Let’s create a reusable Roslyn-based solution that knows how to do thatoptimization.
We can use the class above to rewrite each SyntaxTree in aCSharpCompilation:

class StringBuilderInterpolationOptimizer : CSharpSyntaxRewriter{  private readonly SemanticModel _model;  public StringBuilderInterpolationOptimizer(SemanticModel model)  {    _model = model;  }  // we use the semantic model to get the type information of the method being called  private static bool CanRewriteSymbol(SymbolInfo symbolInfo, out bool appendNewLine)  {    appendNewLine = false;    IMethodSymbol methodSymbol = symbolInfo.Symbol as IMethodSymbol;    if (methodSymbol == null) return false;    switch (methodSymbol.Name)    {      case "AppendLine":      case "Append":        if (methodSymbol.ContainingType.ToString() == "System.Text.StringBuilder")        {          appendNewLine = methodSymbol.Name == "AppendLine";          return true;        }        break;    }    return false;  }  public override SyntaxNode VisitInvocationExpression(InvocationExpressionSyntax node)  {    var memberAccess = node.Expression as MemberAccessExpressionSyntax;    if (memberAccess != null && node.ArgumentList.Arguments.Count == 1)    {      // check if the single method argument is an interpolated string      var interpolatedStringSyntax = node.ArgumentList.Arguments[0].Expression as olatedStringExpressionSyntax;      if (interpolatedStringSyntax != null)      {        bool appendNewLine; // this distinguishes Append and AppendLine calls        if (CanRewriteSymbol(_model.GetSymbolInfo(memberAccess), out appendNewLine))        {          var formatCount = 0;          var formatString = new StringBuilder();          var formatArgs = new List<ArgumentSyntax>();          // build the format string          foreach (var content in interpolatedStringSyntax.Contents)          {            switch (content.Kind())            {              case SyntaxKind.InterpolatedStringText:                var text = (InterpolatedStringTextSyntax)content;                formatString.Append(text.TextToken.Text);                break;              case SyntaxKind.Interpolation:                var interpolation = (InterpolationSyntax)content;                formatString.Append("{");                formatString.Append(formatCount++);                formatString.Append(interpolation.AlignmentClause);                formatString.Append(interpolation.FormatClause);                formatString.Append("}");                // the interpolations become arguments for the AppendFormat call                formatArgs.Add(SyntaxFactory.Argument(interpolation.Expression));                break;            }          }          if (appendNewLine)          {            formatString.AppendLine();          }          // the first parameter is the format string          formatArgs.Insert(0,            SyntaxFactory.Argument(              SyntaxFactory.LiteralExpression(                SyntaxKind.StringLiteralExpression,                SyntaxFactory.Literal(formatString.ToString()))));                return node                  .WithExpression(memberAccess.WithName(SyntaxFactory.IdentifierName("AppendFormat")))                  .WithArgumentList(SyntaxFactory.ArgumentList(SyntaxFactory.SeparatedList(formatParams)));        }      }    }    return base.VisitInvocationExpression(node);  }}

Originally I wanted the optimization to do the same forTextWriter.Write/WriteLine and Console.Write/WriteLine calls, but it turns outthat they actually callstring.Formatinternally anyway. So, add that to the list of possible optimizations.

Conclusion

As you’ve seen by yourself, it’s not particulairly difficult to optimize theback and forth between string interpolation and StringBuilder. I reallythink optimizations like that should be in Roslyn. As long as that’s notimplemented though, using string interpolation to build big string fragments(like for example HTML…) might be a bit more expensive than you might think.

Stay tuned for my next blog post, where I’ll show you how to plug theoptimization into the metaprogramming infrastructure of DNX and/or[StackExchange.Precompilation](https://blog.stackoverflow.com/2015/07/announcing-stackexchange-precompilation/), if you’re not ready to migrate to vNext yet.

Roslyn Adventures: Walk the #line

Fri, 14 Nov 2014 00:00:00 +0000

With Roslyn, C# developers now have a powerful tool which makes modifying thesource code a breeze. At Stack Exchange we’veinvented our own little set of extension to C# for localization purposes,previously described on Matt Jibson’sblog.The project originally supported ASP.NET MVC views only, but we’ve expanded itto C# source files (.cs) because our projects have strings that simply don’tfit into MVC views and rendering a view for each string is overkill. In thisblog series I’ll try to highlight some of the fun things I’ve learned on this.cshtml ->.cs journey.

Localization Adventures: Walk the `#line`

With code rewriting being the biggest problem, one can quickly forget tosupport other things in the development workflow, like debugging. TakeASP.NET MVC’s .cshtml views for example. Although they are compiled to C#source code (you get CodeDOM from Razor, ASP.NET vNext will change that) youcan still step through breakpoints in the .cshtml file. In C# we have the#line [preprocessor directive](http://msdn.microsoft.com/en-us/library/34dk387t.aspx) that can enables us to do this. As trivial as itmight seem, it’s not so easy to get right. The fun part is having more thanone statement on a single line. Let’s say we have this simple program fragmentthat we need to rewrite:

namespace ConsoleApplication1{    class Program    {        static void Main(string[] args)        {            string foobar = _s("$foo$ bar", new { foo = Foo() }), baz = Baz();        }        static string Foo() { return "Foo"; }        static string Baz() { return "Baz"; }    }}

… and the whole thing get’s (badly) rewritten into:

namespace ConsoleApplication1{    class Program    {        static void Main(string[] args)        {            string _s_123 = string.Concat(Foo(), " bar");            string foobar = _s_123, baz = Baz();        }        static string Foo() { return "Foo"; }        static string Baz() { return "Baz"; }    }}

The question is, how to place the #line directives to hit a breakpoint onthe baz = Baz() statement? A clue lies in the breakpoint itself, whichVisual Studio shows as Program.Original.cs, line 7 character 67. As you cansee though, there is no way to specify character 67 using the #linedirective. To solve the problem we need to be aware of two simple facts theMSDN page forgets tomention:

the same line number can be used by multiple #line directives
whitespace is important when using the #line directive

Which brings us to thesolution:

#line 1 "Program.Original.cs"namespace ConsoleApplication1{    class Program    {        static void Main(string[] args)        {#line hidden            string _s_123 = string.Concat(Foo(), " bar");#line 7            string foobar = _s_123                              ,#line 7                                                                  baz = Baz();        }        static string Foo() { return "Foo"; }        static string Baz() { return "Baz"; }    }}

ZeroMQ XPUB sockets

Wed, 04 Apr 2012 00:00:00 +0000

I was wlaying around with the new XPUB &XSUB ZeroMQ socket typesthat allow subscription forwarding, e.g. the producer (XPUB) can track whatvalues he has to produce, which comes in handy when the value production is anexpensive operation and the producer cannot afford to produce all values.

import threadimport timeimport zmqfrom zmq.core.socket import Socket# global zmg contextcontext = zmq.Context()endpoint = "tcp://*:8888"# the subscriber thread functiondef subscriber(name, address, cnt, subscriptions):    print ("starting worker thread %s subscribing to %s for %s"%(name,address,subscriptions))    sub = Socket(context, zmq.SUB)    sub.connect(address)    for subscription in subscriptions:        sub.setsockopt(zmq.SUBSCRIBE, subscription)    for x in range(0, cnt):        print ("%s received %s" % (name, sub.recv()))    print ("%s closing socket after %d messages" % (name, cnt))    sub.close()def main():    publisher = Socket(context, zmq.XPUB)    publisher.bind(endpoint)    address = "tcp://localhost:8888"    thread.start_new(subscriber, ("s1", address, 10, ["a", "b"]))    thread.start_new(subscriber, ("s2", address, 20, ["b", "c"]))    subscriptions = []    r = 0    while True:        # handle subscription flow first to decide what messages need to be produced        while True:            try:                rc = publisher.recv(zmq.NOBLOCK)                subscription = rc[1:]                status = rc[0]== "\x01"                method = subscriptions.append if status else subscriptions.remove                method(subscription)            except zmq.ZMQError:                break        # produce a value for each existing subscription        for subscription in subscriptions:            print "sending " + subscription            publisher.send("%s %d" % (subscription, r))        time.sleep(0.5)        r += 1if __name__ == "__main__":    main()

C# gotcha

Thu, 14 Apr 2011 00:00:00 +0000

Consider you have the following code:

SomeClass reference = null;  Console.WriteLine(reference.Echo("echo"));

Your first thought might be NullReferenceException. Ha, think again! What ifyou have an class defining an extension method like this:

public static class SomeClassExt{    public static string Echo(this SomeClass x, string str)    {        return str;    }}

You have been warned :)

Async Producer-Consumer with Linq, IEnumerable and Threading.Tasks

Wed, 09 Mar 2011 00:00:00 +0000

Lately I’ve been dealing a lot with IEnumerable classes and the yieldreturn keyword. I found it really useful for stream like processing ofotherwise really memory consuming operations. The workflow looks like this:

\- Retrieve Item 1 _(yield return)_\- Process Item 1 _(GetNext())_\- …\- Retrieve Item N\- Process Item N

The approximate time it takes to process the whole operation is:

N x AvgTime(Retrieve) + N x AvgTime(Process)

So I was thinking how to parallelize the whole operation, preferably in a Linqfashion, so it could fit into my existing code. Here is the result, using the(relatively) new System.Collections.Concurrent namespace and the Task ParallelLibrary:

public static IEnumerable<TResult> SelectAsync<TProduct, TResult>(    this IEnumerable<TProduct> producer,    Func<TProduct, TResult> consumer,    int? productsBoundedCapacity = null,    int? resultsBoundedCapacity = null){    var productsQueue = productsBoundedCapacity.HasValue        ? new BlockingCollection<TProduct>(productsBoundedCapacity.Value)        : new BlockingCollection<TProduct>();    var resultsQueue = resultsBoundedCapacity.HasValue        ? new BlockingCollection<Tuple<int, TResult>>(resultsBoundedCapacity.Value)        : new BlockingCollection<Tuple<int, TResult>>();    Task.Factory.StartNew(() =>    {        foreach (var product in producer)        {            productsQueue.Add(product);        }        productsQueue.CompleteAdding();    });    Task.Factory.StartNew(() =>    {        var tasks = productsQueue            .GetConsumingEnumerable()            .Select((product, ind) => Task.Factory.StartNew(() =>                    resultsQueue.Add(new Tuple<int, TResult>(ind, consumer(product)))))            .ToArray();        Task.WaitAll(tasks);        resultsQueue.CompleteAdding();    });    return resultsQueue        .GetConsumingEnumerable()        .OrderBy(x => x.Item1)        .Select(x => x.Item2);}

Note however, that the producer part parallelizes correctly only with theusage of true enumerables that yield return the long running operation result(Arrays, List etc. already contain all the data). Here is a small proof ofconcept app:

class Program{    private static IEnumerable<int> Producer()    {        for (int i = 0; i < 100; i++)        {            Thread.Sleep(50); // long running operation            yield return i;        }    }    private static int Consumer(int item)    {        Thread.Sleep(100); // long running operation        return item * item;    }    static void Main()    {        var sw = new Stopwatch();        sw.Start();        var resultSync = Producer().Select(Consumer).Sum();        sw.Stop();        Console.WriteLine("Result: {0},  {1}ms", resultSync, sw.ElapsedMilliseconds);        sw.Reset();        sw.Start();        var resultAsync = Producer().SelectAsync(Consumer).Sum();        sw.Stop();        Console.WriteLine("Result: {0},  {1}ms", resultAsync, sw.ElapsedMilliseconds);        Console.ReadLine();    }}

Unit Testing JsonResult Actions

Thu, 10 Feb 2011 00:00:00 +0000

Today I had to write a unit test for a Asp.net MVC controller that returned aJsonResult. The Controller’s code looked somewhat like this:

    [HttpPost]    public JsonResult JsonData(Query query)    {      // ... process query, get total_row_cnt and rows      return Json(new { total_row_cnt = rowCount, rows });    }

I often return anonymous types as JsonResult, just because I don’t want tohave additional return model classes for JavaScript that are used only byone controller action and nowhere else.

And what about unit testing such actions? Sure JsonResult exposes a propertycalled Data (of type object), that contains the argument passed to theController’s Json() call. If the type of the Data is known you can easily castit to the known type and do further processing. But what if you pass ananonymous type as data to the JsonResult? How can you verify the results then?

Well you have the following options:

convert the anonymous type to a class,
directly use reflection on the returned object (magic strings!),
or make use of C# 4 dynamics!

The first two options are trivial, and I’m going to discuss only the lastoption. Simply casting the JsonResult.Data won’t do, since the type objectdoes not implement the necessary infrastructure. I decided to implement awrapper class that implements DynamicObject. Since anonymous types can offeronly readonly properties all I had to do was to override the TryGetMembermethod. So here is the implementation:

using System;using System.Dynamic;using System.Reflection;public sealed class AnonymusDynamicGetWrapper : DynamicObject{  private readonly Type _subjectType;  private readonly object _subject;  public static dynamic Create(object subject)  {    return new AnonymusDynamicGetWrapper(subject);  }  private AnonymusDynamicGetWrapper(object subject)  {    _subject = subject;    _subjectType = subject.GetType();  }  public override bool TryGetMember(GetMemberBinder binder, out object result)  {    result = null;    var propertyInfo = _subjectType.GetProperty(binder.Name);    if (propertyInfo == null) return false;    var getter = propertyInfo.GetGetMethod();    if (getter == null) return false;    result = getter.Invoke(_subject, null);    return true;  }}

And finally the unit test method:

[TestMethod]public void Simple_Data_Test(){  // ... setup the test, controller and query  var data = controller.JsonData(query).Data;  var rowCnt = AnonymusDynamicGetWrapper.Create(data).total_row_cnt  Assert.AreEqual(25, rowCnt);}

Obfuscated JS Hello World CodeGolf

Fri, 04 Feb 2011 00:00:00 +0000

Finally the codegolf Stackexchange site is in beta. So I decided to give it atry, as I foundthisnice codegolf:

[](https://lh6.googleusercontent.com/_-AYvMXY8i7Q/TUtOiKTSVGI/AAAAAAAAAEA/RWA-_dMVaaE/s1600-h/golf%5B3%5D.png)

So…. Behold my[solution](https://codegolf.stackexchange.com/questions/307/obfuscated-hello-world/453#453) with JavaScript:

    javascript:(_=(_=[][(f=!!(_='')+_)[3]+(b=({}+_)[1])+($=(c=(d=!_+_)[1])+d[0])])())[f[1]+'l'+(a=d[3])+$]("H"+a+'ll'+b+' W'+b+c+'ld');

Feel free to try it out. Just paste it in your browser’s address bar and letit do the magic.

Well if you would like to know how it works I suggest you read this blog postfirst (I’ve actually shared it in my Google Reader feed a month back, so youmight have already seen it). The script uses the same principle to acquire thesort() function (and consequently the window object and alert() function) asdescribed there. I also found this nice videothat was very helpful.

jQuery.validation breaks jQuery 1.5 ajax API

Thu, 03 Feb 2011 00:00:00 +0000

Today I found out (the hard way) that the use of the[jQuery.validation](http://bassistance.de/jquery-plugins/jquery-plugin-validation/) plugin breaks thejQuery.ajax API. Since the Asp.net MVC3 uses jQuery.validation for its [unobtrusivevalidation](http://weblogs.asp.net/mikaelsoderstrom/archive/2010/10/06/unobtrusive-validation-in-asp-net-mvc-3.aspx) it is included in the default projecttemplate. You can imagine my surprise when I updated jQuery to v1.5, and myheavy ajaxified MVC page stopped working correctly. Every ajax requestcontained an parseerror with an error message stating that the callbackjQuery[some random string] was not called. I tried to fiddle with the ajaxsettings for JSONP(namely jsonp and jsonpCallback), but no matter how I set them, the errorwas still present.

![error_example](https://lh6.googleusercontent.com/_-AYvMXY8i7Q/TUnzg-ApZ4I/AAAAAAAAAD8/5T_rDsz_0L4/error_example_thumb%5B143%5D.png?imgmax=800)

So I started digging around, trying to replicate the problem. When I ransimple html pages $.ajax worked asexpected. Then I started to gradually test all the jQuery plugins and foundout that the plugin causing the error was[jQuery.validation](http://bassistance.de/jquery-plugins/jquery-plugin-validation/). So here is my reproduction and a simple workaround (the wholeproject is available here):

The controller:

using System.Linq;using System.Web.Mvc;public partial class DefaultController : Controller{  private static readonly int[] numbers = Enumerable.Range(1, 10).ToArray();  public virtual ActionResult Index()  {    return View();  }  [HttpPost]  [ValidateAntiForgeryToken]  public virtual JsonResult SimpleArray(int id)  {    return Json(numbers.Concat(new [] { id }));  }}

The view:

@{ Layout = null; }  <!DOCTYPE html>  <html>  <head>  <title>Index</title>    <script src="/Scripts/jquery-1.5.js" type="text/javascript"></script>    <script src="/Scripts/jquery.validate.js" type="text/javascript"></script>  </head>    <body>    <script type="text/javascript">      $(function () {        $('a').click(function (e) {          var req = $.ajax({              url: '@Url.Action(MVC.Default.SimpleArray())',              type: 'POST',               data: $('form').serializeArray(),              dataType: 'json'           });          req.success(function (response, status, xhr) { alert('Success: ' + response); });          req.error(function (xhr, error, msg) { alert('Error "' + error + '": ' + msg); });        });      });    </script>    <a href="javascript:void(0);">Do request</a>    <form action="#">      <input type="hidden" value="-1" name="id" />      @Html.AntiForgeryToken()    </form>  </body>  </html>

The workaround:

$(function () {  $.ajaxSettings.cache = false;  $.ajaxSettings.jsonp = undefined;  $.ajaxSettings.jsonpCallback = undefined;})

The cause of the problem is this line of JavaScript in jQuery.validate.js,that overrides the settings you pass into the $.ajax call with all the defaultones (and jQuery.ajaxSettings defaults to { jsonp: “callback”, jsonpCallback:function() {…}}):

    // create settings for compatibility with ajaxSetupsettings = $.extend(settings, $.extend({}, $.ajaxSettings, settings));