Qluent

Qluent is a Fluent Queue Client for Azure storage queues

Qluent is simple fluent API and set of wrapper classes around the Microsoft Azure Storage SDK, allowing you to interact with storage queues using strongly typed objects like in the code snippet below. You can see lots of other ways to use it in the Documentation on Github.

var queue = await Builder
    .CreateAQueueOf<Entity>()
    .UsingStorageQueue("my-entity-queue")
    .BuildAsync();
    
await queue.PushAsync(new Entity());

So why did I build this.

Back in March, some colleagues and I ran into an issue with a legacy project that we’d inherited at work. At random times a queue consumer would just get stuck and stop dequeuing messages. When we went to debug it we discovered the code responsible for dequeueing, processing and deleting the message was buried in an assembly, and the source code was … unavailable :confounded:.

After much hair pulling and assembly decompilations, we eventually tracked down the bug, but it got me thinkings about a couple of things:

  1. Setting up an azure storage queues through the SDK is a little tedious. There’s quite a bit of ceremony involved to create an CloudStorageAccount, CloudQueueClient and CloudQueue, to ensure it exists and to deal with serialization/deserialization.

  2. There are some aspects of the SDK I dislike. Specifying the large majority of settings on the methods (such as message timeout visibility etc…), rather than as configurations on the CloudQueueClient itself seems wrong. It leaves lots of sharp corners for the developer to get caught on, after they fetch a queue from DI and want to interact with it.

  3. There are lots of tricky scenarios to account for even in simple messaging use cases, such as idempotency issues, handling retries and dealing with poison messages.

  4. Developers shouldn’t need to worry about writing consumers/dispatchers. They should just need to worry about getting their message handled.

The Goal: Keep it simple

What I really wanted to provide was a very simple fluent API for creating a CloudQueue and a message consumer around that CloudQueue. Creating a consumer is simply a matter of providing, a type, a queue, a message handler and starting it up.

var consumer = Builder
    .CreateAConsumerFor<Entity>()
    .UsingQueue(queue)
    .ThatHandlesMessagesUsing((msg) => 
        { 
            Console.WriteLine($"Processing {msg.Value.Property}"); 
            return true; 
        })
    .Build();

await consumer.Start()

The library is intentionally meant to simplify things. Often times I’ll find myself having to scaffold something and spending way too long focusing on the infrastructure code to support message queuing when I should be focusing on the actual problem I’m trying to solve. That’s what this is for. It is a simple wrapper around Azure storage queues to make working with them a little easier.

However there are lots of complicated things you may find yourself needing doing in a distributed environment: Complex Retry Policies; complicated routing paths; Pub/Sub models involving topics and queues; the list goes on. If that’s the case, then perhaps you should be looking at a different technology stack (Azure Service Bus, Event Hubs, Event Grid, Kafka, NService Bus, Mulesoft etc…)

Below you can see some of the features and that the library supports.

Features

Creating a Queue

Queues can be created by simply specifying a storage account, queue name and a type for your message payload. You can purge the queue and obtain an approximate count of messages from it. All operations are async awaitable, and all support taking a CancellationToken.

var q = await Builder
    .CreateAQueueOf<Person>()
    .ConnectedToAccount("UseDevelopmentStorage=true")
    .UsingStorageQueue("my-test-queue")
    .BuildAsync();

await q.PurgeAsync(); 

var count = await q.CountAsync() 

Basic Push/Pop Operations

Basic queue operations include push, pop and peek for one or multiple messages.

var person = new Person("Eoin");
await q.PushAsync(person);


var peekedPerson = await q.PeekAsync();
var poppedPerson = await q.PopAsync();
IEnumerable<Person> peekedPeople = await q.PeekAsync(5);
IEnumerable<Person> poppedPeople = await q.PopAsync(5);

Receipted Deletes

You can also control the deletion of messages from the CloudQueue using the Get and Delete overrides. Under the hood this will use PopReceipts to subsequently remove the message or on visibility timeout, the message will reappear on the queue.

var wrappedMessage = await q.GetAsync();

try
{    
    //attempt to process wrappedPerson.Value;
    await q.DeleteAsync(wrappedMessage);
}
catch(Exception ex)
{ 
    //message will reappear on queue after timeout    
}

Queues can also be configured to support

  • Delayed visibility of messages
  • Message TTLs
  • Visibility timeouts for dequeue events
  • Automatic rerouting of poison messages after a number of dequeue & deserialize attempts
  • Customized object serialization

The message consumer provides a simple way to asynchronously poll a queue including. It supports

  • Message Handlers
  • Failed Processing Handlers (Fallback)
  • Exception Handlers
  • Flow control for when exceptions occur (Exit or Continue)
  • Custom Queue Polling Policies
  • Integration with NLog for Logging

And there’s more detailed info in Github Repo README.md.

Get It on Nuget

I’d really appreciate feedback on it so if you want to try it out, you can get it on nuget

Nuget Stable

Recently I had the displeasure of Wordpress screwing up on me yet again. I’ve been paying a small but measurable amount to a colo based in Ireland for about the past 8 years to provide a small VPS running wordpress. At the time I set it up it was great. But times have changes. The VPS is underpowered. I don’t have the need for a remote windows box anymore with Azure MPN/Dev subscriptions giving me all the free .NET hosting I need. And, Wordpress is a slow hunking mess. I really don’t need a big kludge of a CMS running on MySQL for a personal blog with minimal traffic.

Jekyll

After a little bit of research, I decided to give Jekyll a try. Jekyll is a simple, blog-aware, static site generator built with Ruby. It also happens to be the engine behind Github Pages as well. So you can setup your jekyll blog, upload it to your Github pages repo in your github account and Github will autogenerate your site for you.

Ruby Environment

In order to build and run Jekyll locally, you’ll need access to a ruby dev environment. The general consensus from my bit of research was to do this on a linux distro. Thankfully this is much easier for a windows nerd like me with the new Windows Subsystem for Linux. Simply enable the WSL from powershell and then install your preferred distro from the Microsoft Store. I decided to install Ubuntu.

#Enable Windows Subsystem for Linux
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
#Reboot

Once installed I needed to run some package updates and upgrades. This will be slightly different for each distribution.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install ruby-full build-essential 
gem install bundler jekyll

Github Pages

Once you have jekyll and bundle installed, go ahead and create a github pages repo. I created mine at eoincampbell.github.io. The name of this repo is the same as your github account. You can browse the folder structure of my site here to get a sense of the content/folder structure.

Next check out the empty repo to your local machine, and run the jekyll create command in it.

jekyll new your-site-name
bundle exec jekyll serve

There’s tonnes on information Jekyll and how to configure and theme it on their site.

Migrating Wordpress Comments

Before shutting down my wordpress account, I wanted to retain the comment history for my old site. Jekyll has built in comment/discussion support using Disqus.

You can export all your wordpress content by logging into your wordpress admin section, and running the Export tool under the tools menu. You should select to export all content including posts, pages and comments.

Next create an account on Disqus and login to the Disqus Importer. Here you’ll be able to upload your wordpress xml data dump. Comments are stored against your domain name, and post urls. That means so long as your new site has the same domain and same permalinks, your comments will just appear.

Migrating Wordpress Posts

Finally, you’ll want to convert your old wordpress blog posts into markdown files.

Thankfully this a ruby 1 liner. (though you may need to install the gem first)

ruby -rubygems -e 
    'require "jekyll/migrators/wordpressdotcom"; 
    Jekyll::WordpressDotCom.process("/path/to/wordpress.xml")'

Gotchas

The whole process was pretty painless to be honest, but I did get caught out by a couple of issues, mostly due to the fact that it’s a long time since I dabbled with Linux and it kinda has some sharp corners.

There’s a good article here on how to install ruby on ubuntu. Originally I missed adding the system Gems’ to my path which caused some headaches.

Another issue I ran into had to do with some dependencies after changing from the Default Jekyll gem to using the github pages version. It looked for a dependency called nokogiri which was missing a further dependency called Zlib, and I needed to install Zlib manuaully via apt-get.

Finally, although the jekyll markdown format supports html, you’ll probably find that you’ll need to do a pass of your old posts to clean them up a little and get the formatting just right.

Have fun

Eoin C

For the last 6 months I've been working as a software architect for ChannelSight here in Dublin. I'm really enjoying it and lucky in that I have a fairly broad remit here. But what is "Architecture". What do I do? If you want to be architect:

  • What does that career path look like?
  • What skills do you need?
  • What can you expect?

In interviews with senior developers, they'll mentioned that a role as an architect is a career goal for them, but when asked how they're trying to achieve that goal, they flounder.

I've held roles with junior, mid-level & senior development titles. Then I moved into lead & management roles with a mix of architecture, project management and people management. I think this is a natural progression for developers. The opportunities for progression for a senior developer might involve staying in development, moving into architecture, taking  a managerial position, or some combination of the three.

Personally, I wanted to progress along a technical path. Most importantly I enjoy architecture. I like designing things, doing research, coming up with ideas and overseeing the creation of new software.

What's in a Job Title?

My full job title is Senior Technical Architect. That title will mean different things to different people. Some might call it Software/Solution Architect or Technical Lead or something else. Depending on the software/products you build, there might be slight discrepancies in what the job entails. Depending on company size, the responsibilities in the role may differ.

I've only worked in SMEs with 20-60 employees, so the architecture roles I've held have also had tendrils into client engagement, requirements, team leading, project management and hiring. In a larger organisation, an Architect's role might be more focused on designing the software.

So what can you expect as a Software Architect?

You need to be able to design software

Software Architecture

And obvious one, but you need to be able to design software. This might mean just the software itself or the systems and infrastructure that go with it. Ultimately, you need the capability to look at a problem, and think up an appropriate software solution for it.

  • Do you understand the business and domain problem?
  • Do you know the software & architectural patterns that can be applied?
  • Do you have an appreciation for the the function and non-functional requirements you need to satisfy?

Sometimes the answer to these questions will be "No" and you'll start from a position of disadvantage. Part of the challenge is being able to figure out those answers.

You'll need to draw on your experiences in previous projects. You'll need to do research. And ultimately when you've pulled all the various pieces together, you need to assemble them into a design that meets the needs of your client or company.

You are responsible

Not to say you don't already have responsibilities, but as an architect, there comes an additional level of responsibility and the buck stops with you. You are responsible for the designs you put forward. If you propose a solution that won't work, and your team builds that solution, then the responsibility if it fails is yours.

In smaller companies, you may have other responsibilities too. You may have to:

  • ... engage with clients, and thus are responsible for the company's image.
  • ... participate in hiring and are in part responsible for the composition and capability of the development team.
  • ... run training and up-skill workshops.

With more seniority, comes more responsibility and you need to step up.

You need to make time for thinking

Thinking Time

There will always be an amount of straight forward, humdrum project work involving good old n-tier architectures that most developers are pretty comfortable with. But for other projects, I need to put time aside to get my head around the task. I need time to think.

I use mind-maps to brainstorm out my thoughts. I eek out space in my day (sometimes out-of-hours) with a whiteboard or a notepad and scribble things down. You should bring others into this thinking time as well. Some of the best solutions I've come up with in the past weren't a single eureka moment on my own. They happened with two or three smart people in the room with me as we all collaboratively came up with a solution that we could confidently stand over.

It's helpful to have some resources to hand too. Having a copy of the Gang of Four: Design Patterns on your (e-)book shelf is a good idea. Since I work on the .NET stack and Azure Platform, knowing the .NET tools and frameworks, and being able to refer back to Cloud Architecture Patterns is helpful also.

You need to be a good communicator

As an architect, expect to have to talk... a lot. Pre-design, you'll talk to clients, business analysts and senior management to get the information you need. After you've create a design, you need to communicate it to everyone; first to get buy in from the stakeholders and second so that your team understands what they've been tasked to build. You'll attend a lot more meetings (workshops, requirements, reviews etc...) and be expected to engage with a much broader group in your company.

You'll present more often; sometimes a pre-prepared PowerPoint deck to the business, pitching your ideas; other times presenting to tech staff, explaining the solutions there and then on the whiteboard.

You'll need to explain your ideas in varying levels of complexity/detail. While talking with developers, those conversations might be technical, getting into the nitty-gritty of a design. Other times, you may have to take an Explain-It-Like-I'm-5 (ELI5) approach with business colleagues, who don't have the same technical grasp, but need a higher level understanding.

Finally, you need to be comfortable fighting your corner. There is a balance to be struck between a pragmatic architecture (avoiding over-architecture/unneeded complexity) and ensuring it is for purpose (avoiding a sub-par solution due to some other constraints). You must be comfortable having these discussions and be able to make your case with facts and evidence to back it up.

You will write more documents and less code

A big part of that process is getting your designs and ideas across to other people. You'll spend more time writing docs and less time writing code. My role now involves writing specifications, drawing UML diagrams, typing up Jira Backlog Items and filling out Confluence documentation so that others have the information they need. This is at the expense of getting to write code on a day to day basis myself.

As a result, some ring rust has been creeping into my capabilities as a developer. When I do have the luxury to write code, it takes me longer to do things than it did in the past, and because of time constraints and priority juggling, it's very rare I can get into a flow.

You need to be able to write requirements

You should be comfortable capturing & writing requirements. In a larger organisation, the responsibility to write a Business/User Requirements Specification document might be the job of a dedicated business analyst/project manager. Or it might fall upon you. You might contribute to functional requirements specs, or document out use-cases or user-stories. You need to understand how to capture non-functional requirements. On a brown-field project, you may need to research what's there already and conduct a gap analysis.

At later stages in the design phase, you'll need to be comfortable creating a software design document or technical specification that explains the technicalities of what you plan to build and how it will satisfy those business/functional requirements.

Fast Cheap Good

You need to know how to frame the requirements and how to present your proposed solution to the stakeholders. The may be many approaches that could be taken. The design you put forward will affect where in the Good-Fast-Cheap triangle the implementation will land. If there are time, cost or quality considerations to take into account, these need to fit into your thinking and design rationale.

If your architecture role also carries some PM responsibilities, you may need to go back to stakeholders with recommendations, some that might not be popular. Perhaps it's just not possible to meet their requirements within the constraints they've set out and some prioritization approaches like MoSCow (Must-Should-Could-Would) may be necessary to decide what requirements will and won't be met during the (initial) implementation phase.

You need to let go

Something I find challenging is having to take a step back from the implementation and let others get on with it. If you've come from a senior dev background, you've probably contributed to both design and implementation. You may have strong opinions on what the code/implementation should look like. But that might not be your job anymore. You can still be involved, maybe in an oversight capacity, maybe guiding more junior developers on how to do something, or taking part in code-reviews or maybe you can contribute to some small part of the implementation if you're lucky.

For the most part, you won't have the time and bandwidth to embed yourself in the day to day implementation. You need to let go and hand the implementation reins over to the developers. And you need to trust that those developers are capable of delivering your vision for the solution.

This might make you feel like you're becoming a jack-of-all trades and master of none. But that's OK. Your job as architect is to focus on the big picture. To see the overall solution.

You need to be able to self-improve and do research

As a closing thought, you need to find a way to improve as an architect. The software development landscape is constantly evolving. New technologies are constantly introduced. New methodologies will gain and fall out of favor. New ecosystems will emerge (mobile, wearables, iot etc...) and new regulations will appear (GDPR and ePrivacy).

If you're lucky, your organisation may have a way to support you. Perhaps there's an architecture team/board where new ideas can be discussed. Or perhaps there's dedicated time put aside for R&D. But in smaller companies, you may be seen as the "most senior developer" and not have anyone else in your immediate circle of colleagues that you can learn from on a day to day basis.

You'll need to find away to pull yourself up by your own bootstraps. That may involve:

  • reading books & blog posts
  • listening to podcasts
  • finding time in your professional or personal life to try out new things
  • testing and coding with new technologies and frameworks

If you don't have someone in your company that you can go to, then perhaps you can find a mentor outside. An ex-colleague or friend who's further along their career path that can guide you or at least act as a sounding board

There are membership communities like the International Association of Software Architects where you can find further material on software architecture. There are formal training and certification paths like the TOGAF Certification where you can get a recognized qualification as an enterprise architect. And within the Microsoft world, there are MCA (Microsoft Certified Architect) paths that you can take, and several Application/Solution level certifications like Architecting Microsoft Azure Solutions.

Good Luck

 

~Eoin C

 

I've been trying to sell my colleagues on the benefits of PowerShell. Need to interact with Git, do it from PowerShell. Got some repetitive task to perform? PowerShell. I'd like a cup of coffee... seriously write a script to contact your IOT enabled coffee machine and have it ready for you when you get there.

PowerShell All The Things

A buddy was doing some QA testing of a GeoIP lookup service we've built. We store IPs as unsigned ints (for speedy Azure table storage row-key lookup) and he needed a quick way to go from integer to 4-octet (1.2.3.4) and back. So we knocked up some PowerShell functions. And then a train of thought took over.

Everyone should hopefully be in the mindset of automating and scripting common tasks. But the benefit is even more powerful if you can share and distribute these things among your friends and colleagues. Maybe you do already. Maybe you paste scripts into Slack. Maybe you have a Github repo that people can grab them from and add import statements to their PSProfile. Here, we're going to look at bundling our module into a package and deploying it to MyGet so it can be shared and installed by the masses.

Seems like a good idea... right?

Create a PowerShell module

First, I created a new module in my github repo of powershell scripts, creatively named EoinCModule.

The Module directory contains 2 files, the module itself, and a manifest file.

  • EoinCModule
    • EoinCModule.psm1
    • EoinCModule.psd1

A PowerShell module, is a PSM1 file which contains your functions and commandlets. It's a pretty straight forward PowerShell script file with parameters, comments etc...

The manifest file contains meta-data about the module to describe it during the publish process. To create a template manifest file, you can run the following command in your module directory.

New-ModuleManifest -Path EoinCModule.psd1

The manifest file contains the meta data about your module. At a minimum you'll need to update it to include

  • Author
  • Description
  • ModuleVersion
  • RootModule (your PSM1 file)
  • FunctionsToExport

Signup to MyGet

Next you'll need somewhere to publish your package to. MyGet is a public package management site that's free to sign up for.

I signed up with my gmail account, created a public feed and that's it.

Publish your Packages

PowerShell 5.0 has integrated publish support to allow you to publish direct to MyGet.

In MyGet, navigate to your profile page and grab your Publisher API key.

Once you have your API Key, run the following commands from your PowerShell prompt to publish your module. (Don't forget to replace your feed name & api key)

Import-Module PowerShellGet
$PSGalleryPublishUri = 'https://www.myget.org/F/YOUR-FEED-NAME/api/v2/package'
$PSGallerySourceUri = 'https://www.myget.org/F/YOUR-FEED-NAME/api/v2'
$APIKey = 'YOUR-API-KEY'

Register-PSRepository -Name MyGetFeed -SourceLocation $PSGallerySourceUri -PublishLocation $PSGalleryPublishUri
Publish-Module -Path .\YOUR-MODULE-DIRECTORY -NuGetApiKey $APIKey -Repository MyGetFeed -Verbose

That's it. Your package is published.

Importing your package

Once your package is published you (or anyone else) can import it using the following command.

Install-Module -Name "MODULE-NAME" -RequiredVersion "VERSION" -Repository "REPO-NAME"

Other Bits & Pieces

The process outlined above is pretty trivial, and if you have some module scripts already, getting them published shouldn't take any longer than 5-10 minutes. I did run into a couple of little issues though. When you generate your manifest, the default template will contain a version number with only a Major.Minor component. In order to publish, the version number must contain at BUILD component as well.

In order to publish you'll notice a command above to register a PSRepository. This will persist across PowerShell sessions once created. In the above example, I named my repository "MyGetFeed".

To install my module for example, you would run

Install-Module -Name "EoinCModule" -RequiredVersion "1.0.0" -Repository "MyGetFeed"

To install my module on a different machine, you would first need to register the repository on that machine. e.g.

Import-Module PowerShellGet
Register-PSRepository -Name "eoinc" -SourceLocation "https://www.myget.org/F/eoinc/api/v2"
Install-Module -Name "EoinCModule" -RequiredVersion "1.0.0" -Repository "eoinc"

~Eoin C

I've been maintaining/curating a nuget package for combinatorics over on Github for a while, which I decided to upgrade to .NET Standard over the weekend. I'd noticed that lots of other FOSS projects were up and running on a platform called AppVeyor and had nice little build passing checks on them like this [![Build status](https://ci.appveyor.com/api/projects/status/5uwjfc79j458m4ju?svg=true)](https://ci.appveyor.com/project/eoincampbell/qluent) so I decided to suss it out.

Appveyor is a cloud hosted continuous integration & continuous delivery platform. We use various CI and CD tools at work like Jenkins & Octopus which require a little bit of setting up so I wasn't really sure what to expect. I certainly wasn't expecting it to be this easy.

Appveyor Build

Signing up was a matter of creating a free account using my Github credentials (AppVeyor is free for open-source projects/public repos). Then it will prompt you to select the projects you'd like to pull in for CI. I chose the Combinatorics project. Next it figured out from the presence of the Solution (.sln) file, and two unit test projects (MSTest) within, that this was a .NET project and it auto-generated an MSBuild CI job, including automated test runs. Boom! CI Pipeline Done. I'm suitably impressed.

Beyond the vanilla setup it seems quite extensive in its capabilities:

  • You can control which branches/tags and pull requests automatically trigger builds as well as triggering them yourself via web-hook or cron schedule
  • It allows you to customise your entire build pipeline by inserting scripts pre build, pre package, post build, on success, on fail through out for custom jobs execution.
  • You can be notified in a variety of different ways including email, web-hook or slack
  • It has built in transformation processes for AssemblyInfo files so you can tie your build and version info together
  • You can add custom artifacts to include in deployments
  • It supports a whole variety of target deployments for your project artifacts including Azure, FTP, Nuget and more

There's probably more I haven't discovered yet. Next job will be to automate the push to Nuget based on commits back to master after Pull Requests are confirmed.

~Eoin C