Introduction to Projection Buffers

I’d like to preface this by thanking Jason Malinowski for his help navigating projection buffers.

One could go a lifetime writing Visual Studio extensions and be forgiven for not understanding or using Visual Studio’s projection buffers. They’re mentioned only briefly on MSDN, and Microsoft has yet to release any samples on how to use them properly.

Projection buffers allow us to create composite editors from different source buffers and are used in ASP .Net’s Razor pages that interlace HTML and C#/VB.Net. They can also be used to subset a buffer, and display only this subset to the user. At Code Connect, we’ve used them to display files on a function-by-function basis.

Projection buffers make cool things like this possible:

Today, we’ll be looking at the steps to embed two editors within a Visual Studio Tool Window, one with an original source file and the other with a projection of the first.

Note: This guide is for Visual Studio 2013 + Roslyn. The C#/VB.Net Language Services were largely re-written and their interaction with projection buffers has evolved and changed. As with all Visual Studio extensions, you’ll also need the Visual Studio SDK.

You can download the complete project from Github: https://github.com/JoshVarty/ProjectionBufferTutorial

Step-by-Step Guide

1. Create a new Visual Studio Package with a Tool Window. I’ve named mine ProjectionBufferTutorial.

2. Let Visual Studio know you’ll define a MEF export by following these steps:

• Open source.extension.vsixmanifest in Solution Explorer
• Click Assets
• Click New
• Set Type to: Microsoft.VisualStudio.MefComponent
• Set Source to: A project in current solution
• Set Project to: ProjectionBufferTutorial (or whatever you’ve named your project)
• Click OK
• Save

3. Right click your project and add references to:Microsoft.VisualStudio.Editor.dll

• Microsoft.VisualStudio.Text.UI.Wpf.dll
• Microsoft.VisualStudio.Text.UI.dll
• Microsoft.VisualStudio.Text.Data.dll
• Microsoft.VisualStudio.Text.Logic.dll
• Microsoft.VisualStudio.CoreUtility.dll
• Microsoft.VisualStudio.ComponentModelHost.dll
• System.ComponentModel.Composition.dll

4. Modify MyControl.xaml to contain the following code. This creates two content controls, one for the entire file and one for a subset of the file.

5. Add the following static class VisualStudioServices.cs to your project. This allows us to interface with a number of services within Visual Studio.

6. Modify ProjectionBufferTutorialPackage.cs (or <YourName>Package.cs) to contain the following. This initializes the various static Visual Studio services for us to use.

7. Add the following code to MyToolWindow.cs. The amount of boilerplate is an absolutely astonishing, but necessary evil. This class creates two WpfTextViewHosts representing the full file. However, it attaches the custom role “CustomProjectionRole” to one. It then adds start position and end position properties to the text buffer. We’ll use these to define the range of text we’d like to project.

Note: Make sure to modify filePath to point to a valid C# file on your machine.

Note: DO NOT OVERWRITE THE GUID AT THE TOP OF YOUR CLASS. This Guid is randomly generated and stored in Guids.cs. These two must match, therefore make sure to use your own.

8. Finally, add a new file to the project called ProjectionTextViewModelProvider.cs.  This class listens for the creation of WpfTextViews with the role “CustomProjectionRole”. It then modifies the visual buffer to display only a subset of the original file.

Important: Point the private string filePath to a valid C# file on your file system. Run the project.

9. Open the solution containing your chosen file. Click View > Other Windows > ProjectionBufferTutorial (or whatever you’ve named your project)

A tool window will open containing two files, the right being a subset of the first. As you make changes to one, the changes are instantly reflected in the other. All Language Services should be working.

Explanation

I’d first like to hedge this section by saying I don’t work at Microsoft. These APIs are largely undocumented with no samples available on how to correctly use them. My usage of ITextViewModelProvider and various other Visual Studio services borders on cargo cult programming.

That being said, this is my understanding of what’s happening.

MyToolWindow.cs

There’s a lot going on here. For starters, our class inherits from IOleCommandTarget. An entire blog post could (and should) be written on this interface and commanding within Visual Studio. Here’s my quick-and-dirty take on this.

Visual Studio uses the command chain design pattern to route commands. Essentially, a linked list is created of different components (all inheriting from IOleCommandTarget) that are interested in listening to commands. Basic commands include arrow key presses, Ctr-Z, and backspace. For just a sampling of the many possible commands see VSConstants.VSStd97CmdID.

When a command is received by a command filter object, it can do one of the following:

• Handle the command and pass the command down the chain.
• Handle the command and do not pass the command down the chain.
• Do not handle the command, and pass it down the chain.
• Ignore the command completely and not pass it down the chain.

The IOleCommandTarget.Query() method is fired before the command is actually passed down the chain. This method simply probes the chain to see if anyone can even handle the command.

The IOleCommandTarget.Exec() method is fired when the command can be handled. The command is passed down the chain, handled and then an error code is returned. If no error was encountered, the value 0 is returned.

MyToolWindow inherits from IOleCommandTarget and routes commands (backspace, arrow keys, Ctr-Space etc.) to the IVsTextView of the editor.

The other major workhorse within MyToolWindow is CreateEditor(). This method creates an IVsInvisibleEditor for a given filepath. This IVsInvisibleEditor takes care of a lot of background work not relevant to this blog post, including registering the file within the Running Document Table.

We then retrieve the IVsTextLines from this IVsInvisibleEditor and use it to create a new IVsCodeWindow. I believe this code window represents the dual-pane editor we use in Visual Studio when working with any code files. (All code windows are dual pane, drag the slider above the vertical scrollbar if you’re unsure what I’m talking about). Below is a screen shot of a dual-pane window:

After setting the content of this dual-pane window, we can set the roles for it. Common roles include “DOCUMENT” and “ZOOMABLE”. Manipulating these roles allows us to change properties of the code window. For example, we can omit “ZOOMABLE” and remove the ability for the user to zoom in and out. We can also add custom roles, which we’ve done here. We’ve added “CustomProjectionRole” to the text buffer. This will allow us to handle this buffer different in the future and distinguish it from other C# buffers that may have been created by Visual Studio or another plugin.

Finally, we convert the IVsTextView to a IWpfTextViewHost, and object we can embed within typical WPF elements.

ProjectionTextViewModelProvider.cs

There are two classes defined here, ProjectionTextViewModelProvider, and the actual ProjectionTextViewModel that we are providing. The provider’s sole responsibility is to wait for C# text buffers to be created with the role “CustomProjectionRole”. When this happens, it creates a custom view of the buffer to be displayed to the user. It does this by creating a projection buffer and a ProjectionTextViewModel that uses this projection buffer as its VisualBuffer. When VisualStudio goes to display a WpfTextView to a user, it uses the contents of the VisualBuffer.

It’s worth noting that we’ve MEF imported the IProjectionBufferFactoryService (what a majestic name…) in order to create our projection buffer. If you’ve never used MEF before, this will look like magic. (It still does to me).

Hopefully this is enough to get most people started with projection buffers. If you feel overwhelmed, don’t worry, that’s completely natural. One of the unfortunate realities of Visual Studio extensions is that there are an ungodly number of moving parts. The only way to ever hope to understand what is going on is to get your hands dirty. Take this project and start taking things apart and breaking it. You’ll gradually grow more comfortable with the many interfaces and how they interact with one another.

Finally, check out how we’re using this stuff in Code Connect and follow me on Twitter.

In Part 2: Analyzing Syntax Trees With LINQ, we explored different approaches to picking apart pieces of the syntax tree. This approach works well when you’re only interested in specific pieces of syntax (methods, classes, throw statement etc.) It’s great for singling out certain parts of the syntax tree for further investigation.

However, sometimes you’d like to operate on all nodes and tokens within a tree. Alternatively, the order in which you visit these nodes might be important. Perhaps you’re trying to convert C# into VB.Net. Or maybe you’d like to analyze a C# file and output a static HTML file with correct colorization. Both of these programs would require us to visit all nodes and tokens within a syntax tree in the correct order.

The abstract class CSharpSyntaxWalker allows us to construct our own syntax walker that can visit all nodes, tokens and trivia. We can simply inherit from CSharpSyntaxWalker and override the Visit() method to visit all nodes within the tree.

This short sample contains an implementation of CSharpSyntaxWalker called CustomWalker. CustomWalker overrides the Visit() method and prints the type of the node being currently visited. It’s important to note that CustomWalker.Visit() also calls the base.Visit(SyntaxNode) method. This allows the CSharpSyntaxWalker to visit all the child nodes of the current node.

The output for this program:

We can clearly see the various nodes of the syntax tree and their relationship with one another. There are two sibling MethodDeclarations who share the same parent ClassDeclaration.

This above example only visits the nodes of a syntax tree, but we can modify CustomWalker to visit tokens and trivia as well. The abstract class CSharpSyntaxWalker has a constructor that allows us to specify the depth with which we want to visit.

We can modify the above sample to print out the nodes and their corresponding tokens at each depth of the syntax tree.

Note: It’s important to pass the appropriate SyntaxWalkerDepth argument to CSharpSyntaxWalker. Otherwise, the overridden VisitToken() method is never called. Personally, I don’t think CSharpSyntaxWalker’s arguments should be optional. It was unclear to me that the most conservative depth would be walked when I was learning how to use this class.

The output when we use this CSharpSyntaxWalker:

The previous sample and this one share the same syntax tree. The output contains the same syntax nodes, but we’ve added the corresponding syntax tokens for each node.

In the above examples, we’ve visited all nodes and all tokens within a syntax tree. However, sometimes we’d only like to visit certain nodes, but in the predefined order that the CSharpSyntaxWalker provides. Thankfully the API allows us to filter the nodes we’d like to visit based on their syntax.

Instead of visiting all nodes as we did in previous samples, the following only visits ClassDeclarationSyntax and MethodDeclarationSyntax nodes. It’s extremely simple, just printing out the concatenation of the class’ name with the method’s name.

This sample simply outputs:
MyClass.MyMethod
MyOtherClass.MyMethod

The CSharpSyntaxWalker acts as a really great API for analyzing syntax trees. It allows one to accomplish a lot without resorting to using the semantic model and forcing a (possibly) expensive compilation. Whenever inspecting syntax trees and order is important, the CSharpSyntaxWalker is usually what you’re looking for.

A few weeks ago, I set out to work a 100 hour work week. Not 100 hours at the office, but 100 hours working on programming work for Code Connect.
Nick Winter and Bethany Soule have both posted about their experiences doing similar things. Nick clocked an astonishing 120 hours and Bethany worked for an impressive 87.

These two strike me as demi-gods of productivity. Nick has self-published a book on motivation and Bethany co-founded Beeminder, a company dedicated to keeping people motivated and on track for their goals.

I, on the other hand, haven’t done much self-examination on motivation beyond the usual “I should probably procrastinate less…”

Also, as I use Windows I had to build my own (much worse) version of Nick Winter’s Telepath Logger. Mine is absolute garbage and randomly breaks down all the time. (I suspect some jabs at Windows might be made, but they should probably be aimed at me, if anything).

Here’s a time-lapse video of my attempt:

In total, I clocked about 95 hours of productive time. I completely fell apart on the final day, when I got frustrated with a task and quit at 6:00 pm at my friends’ encouragement (More on that later). If I had kept going until 2:00 am, I would have met my 100 hour goal.

Here’s a look at my time-per-day.

Some thoughts on the whole experience:

1. The video was a huge motivator.

The idea that someone might see me cheating helped me resist the urge to go off task. I’ve always been a big believe in internal motivation and that it trumps extrinsic motivation ten times out of ten. This experiment changed my opinion on that a little bit. Perhaps certain external motivators can work together with internal motivators. Perhaps, certain external motivators are different than others and research has yet to distinguish between the two?

2. Have a well-defined goal at all points

The work that was easiest to do, was well-defined and relatively straightforward. When I say straightforward, I don’t mean simple and without thought. But at the same time, certain difficult architecture decisions seemed to almost paralyze me and stall productivity.

A big chunk of my time (Wednesday to Sunday) was spent trying to reverse-engineer how Visual Studio’s Intellisense worked. Occasionally I would get blocked, and not know where to look next. This instantly killed any “Flow” I had and left me frustrated. Perhaps tasks like this are not suited to be worked on for long periods of time.

3. Block everything distracting

Like Bethany, I compiled a list of distracting sites (Reddit, HackerNews etc.) and dumped them in my hosts file to redirect to 127.0.0.1. I’ve developed an awful habit, where I’ll open up a new tab, hit “R” or “H” and press enter and instantly be brought to Reddit or HackerNews. Blocking these sites helped prevent this. I knew that I could easily unblock the websites, but that forced it to be a conscious decision on my part, something I could more easily think through.

4. WARNING: YOUR FRIENDS ARE NOT YOUR FRIENDS

The points at which this week was the hardest was when my friends would try to convince me to take a break. Imagine all the excuses you tell yourself, coming at you from text, email and in-person.

“You’ve worked hard enough today, just come out”
“You need to take a break, no one could sustain this pace”
“Just take a break for a couple hours”

Ultimately, they won out and in a bout of frustration and nagging from friends, I gave up early. I hope to do this again, and next time I’ll completely get rid of my phone. I’m typically not distracted by it (I don’t text much and don’t get too much email) but it was absolutely the biggest distraction during this week.

I can’t blame them, though. I’d probably do the same thing to them.

5. It was not the happiest week I’ve had.

Nick Winter mentioned that his 120-hour work week was the happiest he’d been since he began quantifying his happiness. I’ve never quantified mine, but I’m 100% positive it was not my happiest week. There were points in each day at which I was downright miserable and wanted nothing more than to give up. I love programming and do it almost every day, but this was exhausting.

Nick also mentioned that it was easy for him. It was not easy for me. It was extremely hard. I missed exercise, I missed talking to friends and I missed doing things other than programming.

Final Thoughts

It was a pretty good week overall. I felt like I learned a lot about myself, my motivations and how to improve. In terms of technical progress, I knocked off most of my Git issues and now have a pretty intimate knowledge of Visual Studio’s Intellisense.

I need a second screen. I work entirely on a 13 inch ultrabook screen, which makes it a lot more difficult to look at different parts of large systems. Watching the other maniac week videos left me feeling extremely jealous.

If you want to see what I was building check out the demo video at: http://codeconnect.io

P.S. FOR ANY MICROSOFT DEVS: I still haven’t figured out how to embed Intellisense within a Projection Buffer. If you know how, or know someone who might: Please contact me on Twitter @ThisIsJoshVarty.

Syntax trees are made up of three things: Syntax Nodes, Syntax Tokens and Trivia.

The Roslyn documentation describes Syntax Nodes and Syntax Tokens as follows:

Syntax nodes are one of the primary elements of syntax trees. These nodes represent syntactic constructs such as declarations, statements, clauses, and expressions. Each category of syntax nodes is represented by a separate class derived from SyntaxNode.

Syntax tokens are the terminals of the language grammar, representing the smallest syntactic fragments of the code. They are never parents of other nodes or tokens. Syntax tokens consist of keywords, identifiers, literals, and punctuation.

While both definitions are accurate, they don’t give newcomers much insight on the difference between the two.

Let’s take a look at the following class and its Syntax Tree.

Using Roslyn’s Syntax Visualizer, we can take a peek at the syntax tree:

The Syntax Visualizer shows Syntax Nodes in blue and Syntax Tokens in green.

Syntax Nodes:
ClassDeclaration
MethodDeclaration
ParamteterList
Block

Syntax Tokens:
class
SimpleClass
Punctuation
void
SimpleMethod

Syntax Tokens cannot be broken into simpler pieces. They are the atomic units that make up a C# program. They are the leaves of a syntax tree. They always have a parent Syntax Node (as their parent cannot be a Syntax Token).

Syntax Nodes, on the other hand, are combinations of other Syntax Nodes and Syntax Tokens. They can always be broken into smaller pieces. In my experience, you’re most interested in Syntax Nodes when trying to reason about a syntax tree.

Note: I’ve also created a ten-minute video to explore the Syntax Tree API

I won’t spend much time explaining Syntax Trees. There are a number of posts that deal with that including the Roslyn Whitepaper. The main idea is that given a string containing C# code, the compiler creates a tree representation (called a Syntax Tree) of the string. Roslyn’s power is that it allows us to query this Syntax Tree with LINQ.

Here is a sample in which we use Roslyn create a Syntax Tree from a string. We must add references to Microsoft.CodeAnalysis and Microsoft.CodeAnalysis.CSharp. You can do so using Method 1 from Part 1 Installing Roslyn.

We first start by parsing a string containing C# code and getting the root of this syntax tree. From this point it’s extremely easy to retrieve elements we’d like using LINQ. Given the root of the tree, we look at all the descendant objects and filter them by their type. While we’ve only used ClassDeclarationSyntax and MethodDeclarationSyntax there are corresponding pieces of syntax for any C# feature.

Visual Studio’s Intellisense is extremely valuable for exploring the various types of C# syntax we can use.

We can composed more advanced LINQ expressions as one might expect:

Above, we start by finding all methods, and then filtering by those that accept parameters. We then take this method and work our way upwards through the tree with the Ancestors() method, searching for the first type that contains this method.

Hopefully this acts as a base for you to play around and explore the Syntax Tree API. There are some limitations to the kind of information you can discover at a purely syntactical level and to overcome these we must make use of Roslyn’s Semantic Model, which will be the subject of future posts.

Roslyn is deployed as a NuGet package.

Navigate to: Tools > NuGet Package Manager > Package Manager Console

Paste the following: Install-Package Microsoft.CodeAnalysis

A complete walkthrough of installing other Roslyn tools maybe found below:

 

We’ve been working hard on Code Connect and we’re at the point where we just need to finish development of a minimum viable product and release it in beta form. I’ve decided to take a page from the book of Nick Winter and embark on a Maniac Week. Nick Winter clocked 120 hours of programming during his, which works out to approximately 17 hours of work a day. Bethany Soule of Beeminder recently attempted the same feat; to work as much as she could during one week.

One thing Bethany and Nick Winter had going for them is that they were essentially motivation experts. Nick Winter has published an e-book on the topic, and Bethany co-founded Beeminder, a company dedicated to keeping people motivated and on track to meet their goals. 

I have no such qualifications. I procrastinate, have no tools to keep me on track and have no real strategy to stay on track. To help me stay focused, I’ve blocked all websites I frequently waste time on. I’ve also built my own tool for Windows to take screenshots and webcam captures every minutes. Hopefully, I’ll be able to stitch these into a video documenting the experience sometime next week.

It’s been a few months since I introduced the idea behind Code Connect. Code complexity has been increasing for decades, but the tools we use to manage this complexity have largely stagnated. It’s hard for a programmer new to a large project to understand what’s going on in a maze of abstraction and function calls that span hundreds of files, and thousands of lines. Many of us either insert a breakpoint and trace execution using the debugger or resort to writing out function calls by hand.

Why are we using the debugger to understand a codebase? Why are we resorting to pen and paper while sitting in front of a machine that can easily understand our source code? We essentially resort to “playing computer” when we try to understand code execution paths using only our mind. This approach is dated and makes no sense.

Code Connect strives to solve these problems by dynamically visualizing function calls as you develop. For the first time, we’re finally ready to show off our progress and hear your feedback.

If you’d like updates on our progress, sign up at http://codeconnect.io

Microsoft’s new C# compiler Roslyn has been in the pipeline for quite some time. Eric Lippert first put out a call for developers to work on Roslyn back in 2010, when we worked with C# 3.0 and built our own state machines to handle aynchronous function calls. 

So it’s safe to say that Roslyn has been on the team’s mind for well over three years. Roslyn obviously marks a large investment on behalf of Microsoft. So why take the time to rearchitecture the C# and VB compilers? After all, software rewrites have typically been at the top of on the list of “Things You Should Not Do”.

Most people familiar with Roslyn understand that it marks the first real change in compilers in recent memory. For decades, compilers operated as black boxes that took source code in, and spit out binary, ByteCode or IL. Short of building your own compiler, there was no easy way to peek at the syntax tree or reason about the semantics of a codebase. Roslyn opens the compiler up, making syntactical and semantic structure available to average developers who don’t want to write a feature-complete compiler just to create a Visual Studio plugin.

So the questions surrounding Roslyn boil down to: “What can I build now, that I couldn’t before”.

Intelligent Tooling

Roslyn allows developers to reason about the code in an intelligent manner. No longer are extensions forced to deal with a codebase as a set of files containing strings. From the perspective of the extension, a C# solution is made up of pieces of syntax and structure at varying levels of granularity. Need to find all method invocations within a given document? No problem, simply crawl the syntax tree looking for all syntax of type InvocationExpressionSyntax. Need to find that method’s declaration? No problem, simply pass that InvocationExpressionSyntax to the document’s SemanticModel.

We’ve been using this exact approach to build CodeConnect, a Visual Studio extension in which users edit functions rather than files.

Roslyn allows us to quickly find all invoked functions within a given function and lay them out for a user. This kind of project would only have been possible had we built our own parser and lexer and built an internal representation of a given codebase. It would have needed to be fault tolerant and handle temporary errors as users typed their code. It would have had to waste clock cycles compiling code Visual Studio had already compiled internally.

Scripting Engine

I’m equally excited about Roslyn’s Scripting Engine, despite only having toyed with it briefly. The ability to compile, analyze and execute arbitrary code seems incredible. Already, individuals have worked to develop plugins that execute code as you’re developing,  in the style of Brett Victor. While I think it’s a difficult solution to generalize, I would not be surprised if we see a feature-complete, LightTable-esque approach to this problem once individuals become more familiar with the Scripting Engine.

In short, I expect Visual Studio’s tooling to improve dramatically once Roslyn ships. Roslyn has lowered the barrier to entry when it comes to writing intelligent, useful extensions for Visual Studio. I expect we’ll see a number of exciting tools be created that dramatically change the way in which we interact with our code.

I’m looking forward to tools and plugins that go beyond simple renaming refactoring and enforcing coding standards. I’m hoping the C# community can churn out some really cool tools that change the way in which we write code altogether.

Check out http://codeconnect.io for our take on making interesting plugins for Visual Studio.

Interested in playing around with Roslyn? Learn Roslyn Now!

Code Connect has been and idea that a classmate (Amadeus) and I have been talking about for the past year and a half. After seeing demos for Code Bubbles, Debugger Canvas, Light Table and watching a number of Brett Victor talks, we agreed that something had to be done about the state of programming. Dumping ASCII into text files quickly gets cumbersome. While tooling has improved over the years, we haven’t seen a major overhaul of text editors or IDEs.

Code Connect is our attempt to “fix” programming. It’s a plugin we’re developing for Visual Studio that allows you to retain all the benefits you’ve come to enjoy from that IDE. (For example, I love VsVim and Amadeus loves ReSharper). We believe you shouldn’t have to give up years of progress to take advantage of these new features.

We’re working hard to get a demonstrable prototype out by the end of February as part of our fourth year design project at the University of Waterloo.