Visual Studio Paste Special – Paste Xml and JSON as Classes

I caught this great little tip while listening to the fantastic .Net Rocks Podcast – you can use the ‘Paste Special’ command in Visual Studio to create classes from raw Xml and JSON. Seriously, this previously unknown (to me) functionality is amazing and will save me a LOAD of time!

Paste Xml as Classes

In the animated GIF below, you can see just how easy it is to create classes from Xml input – simply copy the Xml, select ‘Paste Special  -> Paste XML as Classes’ from within Visual Studio and BAM! we have a class corresponding to the Xml on the clipboard:

Visual Studio - Paste Special - Paste XML as Classes

The example Xml shown here is a stripped down version of the Books Xml taken from the MSDN website at: https://msdn.microsoft.com/en-us/library/ms762271%28v=vs.85%29.aspx.

Paste JSON as Classes

The same also works for JSON:

Visual Studio - Paste Special - Paste JSON as Classes

In case you’re interested, I created the GIF animations using the fantastic ScreenToGIF tool which is available on Codeplex at https://screentogif.codeplex.com/

Happy class generating!

Creating Personal Information Exchange (.pfx) Files from Separate Public and Private Key Files

This blog post forms part of a larger series of posts looking at setting-up a SFTP Server for integration testing purposes.

Some Certificate Authorities (CAs) use different file formats to store public and private keys. For example, some CA’s store the certificate’s private key in a Private Key (.pvk) file and store the certificate and public key in a .spc or .cer file. The makecert tool will also generate separate .cer (public key) and .pvk (private key) files. Where this is the case, you may need to merge the two files into a Personal Information Exchange (.pfx) file.

Imagine you have created a set of self-signed keys using the makecert command on the VS Developer Command Prompt (Server.cer is the public key and Service.pvk is the private key):

makecert -r -pe -n "CN=Modhul" -sky exchange Server.cer -sv Server.pvk

In order to create a PFX file, we need to merge the .cer (public key) and .pvk (private key) files using the following command, again on the VS Developer Command Prompt:

pvk2pfx.exe -pvk Server.pvk -spc Server.cer -pfx Server.pfx

The Server.pfx file is our newly created Personal Information Exchange (.pfx) file.

Further information about the pvk2pfx tool can be found at: http://msdn.microsoft.com/en-us/library/windows/hardware/ff550672%28v=vs.85%29.aspx.

Installing Redis Cache Locally in a Development Environment

I recently blogged about using the excellent Redis Cache – which is now the preferred Azure caching solution – for a recent CRM integration project.

In my development environment, I’m pointing against the Azure Redis Cache and while performance is fantastic, I recently saw that Chocolatey have an MS Open Tech version of Redis in their repository that I can run locally in my development environment.

I wondered whether I could easily use the Chocolatey version as a direct drop-in replacement for Azure Redis and even better, eek more performance out of a local install, especially to increase the speed of my unit-tests. The answer is ‘yes you can’ on both points.

Installing Redis via Chocolatey

With Chocolatey installed, we can go ahead and install Redis by issuing the extremely simple command from an Administrator Command Prompt:

Chocolatey - Redis Install Command

As we’re using defaults, Chocolatey will install Redis into C:ProgramDatachocolateylibredis-64.2.8.17 (note that the version number might change for you if you try this with later versions) and a shim into the C:ProgramDatachocolateybin directory (the shim is a link that points to the actual .exe in the lib directory when the installation package contains .exe files and not an MSI file):

Chocolatey - Redis Install Screenshot

Configure and Start Redis

Due to Redis’ dependence on the Linux fork() system call, the Windows version has to simulate fork() by moving the Redis heap into a memory mapped file that can be shared with a child process. If no hints are given on startup, Redis will create a default memory mapped file that is equal to the size of physical memory; there must be disk space available for this file in order for Redis to launch.

During fork() operations the total page file commit will max out at around:

(size of physical memory) + (2 * size of maxheap)

For instance, on a machine with 8GB of physical RAM, the max page file commit with the default maxheap size will be (8)+(2*8) GB , or 24GB.

If you don’t give any hints to Redis, you get an error similar to the following:

The Windows version of Redis allocates a large memory mapped file for sharing
the heap with the forked process used in persistence operations. This file
will be created in the current working directory or the directory specified by
the ‘heapdir’ directive in the .conf file. Windows is reporting that there is
insufficient disk space available for this file (Windows error 0x70).

You may fix this problem by either reducing the size of the Redis heap with
the –maxheap flag, or by moving the heap file to a local drive with sufficient
space.
Please see the documentation included with the binary distributions for more
details on the –maxheap and –heapdir flags.

Redis can not continue. Exiting.

To get around this limitation, specify the –maxheap flag when starting Redis, using a value that is relevant to your machine:

redis-server –maxheap 1gb

which will successfully start Redis:

Redis Server - Started

Allowing us to connect to the local Redis server with a connection string similar to the following:

“localhost:6379,ssl=false”

Note that Redis will create the memory mapped file on your file-system at %USERPROFILE%AppDataLocalRedis that is the size you specify with the –maxheap flag.

Redis Memory Mapped File

Shuting down the server (Ctrl+C in the command prompt window where Redis was started) deletes the file.

Performance Testing

So, what is the performance difference between Azure Redis and a local install of Redis? I created a simple console test app that would create 1000 cache items (integers) and then retrieve the same 1000 cache items; the cache is flushed before I execute each test.

The following results are based on the console test app running locally against my development VM.

Executing against a local Redis instance (all times in ms):

Run 1 Run 2 Run 3
Iteration 1 2515 2799 2526
Iteration 1 2380 2285 2380
Iteration 1 2234 2703 2641
Avg: 2495

Executing against an Azure Redis (1Gb Standard Pricing Tier) instance (all times in ms):

Run 1 Run 2 Run 3
Iteration 1 47955 45139 45725
Iteration 1 48549 47773 46422
Iteration 1 45311 49194 46144
Avg: 46912

I was quite shocked at just how slow the same test was against the Azure Redis instance (2.495 seconds vs. 46.912 seconds). So, to investigate whether this issue was network latency, I tried the same test running on a basic A2 Azure VM (Windows Server 2012, 2 cores, 3.5 GB memory) in the same region as the Azure Redis Cache:

Run 1 Run 2 Run 3
Iteration 1 1211 1185 1186
Iteration 1 1439 1257 1245
Iteration 1 1343 1187 1196
Avg: 1249

The results indicate that executing from the Azure platform to an Azure Redis cache executes faster than a simple install on my local dev. environment (2.495 seconds vs. 1.249 seconds). Kudos to Microsoft for such an excellent and performant service!

Capturing Custom Logs from Azure Worker Roles using Azure Diagnostics

In this post I’ll show you how to correctly configure diagnostics in an Azure Worker Role to push custom log files (NLog, Log4Net etc.) to Azure Storage using the in-built Azure Diagnostics Agent.

 

Configuring our Custom Logger – NLog

I’m not a massive fan of the recommended Azure Worker Role logging process, namely using the Trace.WriteLine() method as I don’t feel as though it provides sufficient flexibility for my logging needs and I think it looks crap when my code is liberally scattered with Trace.WriteLine() statements, code is art and all that.

NLog, on the other hand, provides all the flexibility I need including log file layout formatting, log file archiving – with granulatrity down to one minute – and archive file cleanup to name just a few. Having used it as the main logging tool on several projects, I feel completely at home with this particular library and want to use it within my Worker Role implementations.

I won’t go into how to add NLog (or any other logging framework you may use) to your project as there are loads of examples on the interweb, however I will share with you my Azure Worker Role app.config file which shows the configuration I have for my logging:

On line 8 we define our MessageWorker target which instructs NLog to:

  • Write our (current) log file to ‘MessageWorker_Current.log‘ (the fileName property) with a particular log-file layout (the layout property);
  • Create archives every day (the archiveEvery property) with an archive filename of ‘MessageWorker_{#}.log‘ (the archiveFilename property) – the ‘{#}’ in the archive filename is expanded to a date/time when an archive file is created (the archiveNumbering and archiveDateFormat properties);
  • Maintains a rolling 14 day window of archive files, deleting anything older (the maxArchiveFiles property);
  • Not perform concurrent writes (the concurrentWrites property) or to keep files open (the keepFilesOpen property) which I discovered helps the Azure Diagnostics Agent consistently copy log files to Blob Sstorage – your mileage may vary with these settings.

These configuration settings result in a number of files in our log directory – in the screenshot below I am archiving every minute:

Message Worker Logs

 

Define a Local Storage Resource

It is recommended that custom log files are written to Local Storage Resources on the Azure Worker Role VM. Local Storage Resources are reserved directories in the file system of the VM in which the Worker Role instance is running. Further information about Local Storage can be found online at MSDN.

In order to define Local Storage, we add a LocalResources section to the ServiceDefinition.csdef file and define our ‘CustomLogs’ Local Storage, as shown on line 16 below. The cleanOnRoleRecycle tell Azure not to delete the Local Storage if the role is recycled (restarted or scaled up/down) and the sizeInMB tells Azure how much Local Storage to allocate on the VM.

On an Azure Worker Role VM, the Local Resources directory could be found at: C:ResourcesDirectory[GUID].[WorkerRoleName]

On a local development machine, the Local Resources directory can be found at: C:Users[USER]AppDataLocaldftmpResources[GUID]directoryCustomLogs

 

Configure NLog to Write Log Files to the Local Storage Resource

When using Local Storage Resources, we don’t know the actual directory name for the resource (and therefore our log files) until runtime. As a result, we need to update our NLog Target’s fileName and archiveFilename property as the Worker Role starts. To do this, we need a small helper to re-configure the targets:

This helper is passed the NLog Target to be updated and the directory path of the Local Storage Resource that is made available at runtime:

  • On line 11, we retrieve the FileTarget from our NLog configuration based on the supplied Target name.
  • On lines 16 & 17, we retrieve the current log filename and the archive filename (stripping a trailing apostrophe which appears to come from somewhere).
  • On lines 20 & 21, we combine the current log filename and archive filename with the supplied base directory (which contains the Local Storage directory passed to this method).
  • On line 28, we re-configure the existing loggers, activating these changes.

This helper is called within the Worker Role’s OnStart() method:

On line 15, we call the SetLogTargetBaseDirectory() method passing the NLog Target from our app.config file and the directory of the Local Stroage Resource, obtained by calling:

RoleEnvironment.GetLocalResource(“CustomLogs”).RootPath

where “CustomLogs” is the name of the Local Storage defined in the ServiceDefinition.csdef file (see the earlier Define a Local Storage Resource section). RoleEnvironment is a class located in the Microsoft.WindowsAzure.ServiceRuntime namespace.

 

Configuring Azure Diagnostics

Our final step in the process is to configure diagnostics, which instructs the Azure Diagnostics Agent what diagnostics data should be captured and where it should be stored. This configuration can be performed either by configuration (via the diagnostics.wadcfg file), or through code in the OnStart() method of the Worker Role. Be aware that there is an order of precendence for configuration diagnostic data – take a look at Diagnostics Configuration Mechanisms and Order of Precedence for more information.

The important lines for our custom logging solution are found in the <DataSources> element between lines 6 & 10:

  • Line 7 – <DirectoryConfiguration> details the Azure Storage Container where the custom log files should be written; and
  • Line 8 – <LocalResource> details the Local Storage Resource where the log files reside.

 

Bringing it all Together

Lets recap on the various moving parts we needed to get this up and running:

  • Configure the app.config file for your Worker Role, adding the required NLog configuration. Check that this is working locally before trying to push the settings to the cloud.
  • Define a Local Storage Resource in your the ServiceDefinition.csdef file for your Cloud Service Project where log files will be written to – in this blog post we have called our resource ‘CustomLogs’.
  • Add functionality to update the NLog target’s filename and archive filename at runtime by querying the Local Resource’s Root Path in the Worker Role’s OnStart() method.
  • Add a DataSources section to the diagnostics.wadcfg file instructing the Azure Diagnostics Agent to push custom log files from the Local Storage Resource to the specified Azure Blob Container.

With all of the required pieces in place we can deploy to our Cloud Service on Azure as usual, either through Visual Studio or the Management Portal. Give the service time to spin-up and we should hopefully start to see our log files appear in the specified container within Azure Storage – in the screenshot below, log files are being archived once a day and copied to the Storage Container at approx. 2 mins past midnight every morning:

Custom Log Files Copied to Azure  Storage Container
Custom Log Files Copied to Azure Storage Container – Click to enlarge

 

Gotchas

In closing, there is one gotcha that I would like to highlight. When a Cloud Service Project is deployed to a Cloud Service in Azure, the diagnostics configuration (derived from the diagnostics.wadcfg file) is written to a control configuration blob in the wad-control-container container, as specified in the Diagnostics Connection String (setting Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString) from the deployed ServiceConfiguration.Cloud.cscfg configuration file.

Azure wad-control-container
Azure PaaS Control Configuration Blob – Click to enlarge

The purpose of this file is detailed as follows in the MSDN article Diagnostics Configuration Mechanisms and Order of Precedence:

A control configuration blob will be created for each role instance whenever a role without a blob starts. The wad-control-container blob has the highest precedence for controlling behavior and any changes to the blob will take effect the next time the instance polls for changes. The default polling interval is once per minute.

In order for any changes that you make to the diagnostics.wadcfg file to take effect, you will need to delete this control configuration blob and have Azure re-create it when the role re-starts. Otherwise, Azure will continue to use the outdated diagnostics configuration from this control configuration blob and log-files won’t be copied to Azure Storage.

Serializing Custom .Net Types for use with the Azure Redis Cache

In my previous post I looked at a real-world example of using the new Azure Redis Cache. One thing that was missing was the storing of custom .Net Types as cache values, which we’ll look at here.

The RedisValue Type

The Microsoft Azure documentation recommends using the StackExchange.Redis API for accessing the Redis Cache. This API stores cache values within the RedisValue type, which has a number of implcit conversions from primitive .Net types (Int32, Int64, Double, Boolean etc.), but conversions from the RedisValue type to the primitive .Net type need to be made explicitily.

For example, setting an int as the cache value is implicit, while retrieving an int from a cache value needs to be cast (the following screenshot is from the MSDN documentation):

Cache Serializer - Get and Set of Primitive Types

In addition to the implicit/explicit conversion of primitive data types, the RedisValue type can store binary data as the cache value. This means that we can serialize a custom .Net type into a byte array and pass that as the cache value within the StringSet() method; we can then retrieve that cache value using the StringGet() method and deserialize the byte array back to the custom type.

To complete the serialization/deserialization, we need a helper class, which is shown below. This class is inspired by the sample on MSDN (Work with .Net objects in the cache) but is tweaked slightly:

Note that during deserialization, if the byte array is null, a default instance of type T will be returned. This caught me out when I was initially testing, so beware that it is returning the value you expect.

Using these helpers with the StackExchange.Redis API’s StringSet() and StringGet() methods to store custom .Net types makes life really easy:

To retrieve a custom type, we call the StringGet() method, passing the cache Key and deserialize the returned byte array into the .Net Guid type; In order to determine whether the cache item was found, I check whether the returned cacheItem equals Guid.Empty, which is the value returned by default(T) for the Guid type from the Deserialize() method.

To store a custom .Net type (in this case a GUID), call the StringSet() method and instead of passing a primitive type as the cache value, we serialize our cache item and pass the resulting byte array.