Monkey See, Monkey Do; occasionally monkey learn.

This is an Elephant!

Tue, 01 Jul 2025 09:04:00 GMT

First, I am no AI expert. Yeah, I did some neural network stuff at university in the 90's with some Kohonen self-organizing maps and 3-layer back propagation networks using a network of transputers etc but these are nothing compared to the behemoths of todays AI systems. You could easily train and run our neural networks on modern phones in a fraction of the time it took us then.

As a software developer in today's world we are now expected to embrace AI to help us produce code faster etc, but if you are like me you get frustrated, not at what it can do which is impressive at times, but when it screws up and you feel the task has become harder. There is nothing more annoying as a response like "Yes, you are absolutely right" when you correct the AI on a mistake it made. The expletives that would occasionally follow would make sailor blush; or make you laugh maniacally depending on your breaking point.

Being Lazy

One day I was being lazy, as a good developer should be, and instead of writing a bit of code, I decided to ask ChatGPT to do a task. I had a UNIX timestamp and I wanted to know what the time and date it was generated.

Excellent, its recognised it as a UNIX timestamp and seems to know what to do, and look it has even converted it from UTC to local time. Nice...

Hang on a minute, something is not quite right. Today is Wednesday, 11th June 2025 (Australia) yet the response says the date was generated on a Saturday the 10th.

Let's see if we can get the AI to correct its mistake.

Okay that seems better but now I am nervous so I go `old skool` and actually write some code

console.log(new Date(1749633533666));

and the answer

Wed Jun 11 2025 19:18:53 GMT+1000 (Australian Eastern Standard Time)

So even when corrected ChatGPT is still 2+ hours out. Sigh! But what is worrying is that if it had given this answer from the off I probably wouldn't have caught the mistake and may be even relied on the answer for something else, compounding the issue.

Over confidence

But why does a modern AI give us wrong answers(*) even when it "knows" the correct answer when corrected? I don't know but I feel the problem isn't new, we were aware of these issues even back at university. The willingness of our networks to give an answer on incomplete data, was seen as both a strength and a weakness. We were advised at the time "We don't really know how they work, so please don't use them for safety critical control systems". Looking for defects in production lines, yes. Control your nuclear power reactor, no.

Here I'd like to tell you a story that explains the post headline and the accompanying picture. 30 years ago my friend, was studying for his PhD and was using neural networks for images that had been pre-processed, edge detection etc. We knew of bias even then and so he was looking to remove anything from the image that was not the subject he wanted to detect. He decided to use toys as his subject, easy to manipulate, and he had a set of toy soldiers that he trained his network on and he also had a set of plastic animals. He used these items to train his networks, one for soldiers and one for animals.

One day he was asked to demo his progress and so he loaded up a pre-trained model, inserted a toy soldier into the box and snap. The computer whirred, photo taken, image processed, data sent to the neural network on the transputer array and back came the answer, "Elephant". Now this was funny to everyone, except possibly my friend, as it was so obvious he had loaded the wrong network. But it shows that the neural network was willing to give an answer even when it so obviously wrong to the rest of us. Of course if the network was big enough it could've been trained on both sets of toys and then it would have some training on what a soldier is as well as what an elephant is and perhaps then would've got the answer right. No matter how big a network or the number of subjects trained upon, I don't believe it would've been able to respond with "I don't know" when presented with unknown data, the willingness to supply an answer is its goal.

Now modern AIs are obviously much, much bigger and have been trained on nearly the total wealth of available human and now synthesized output(**) but using similar training principles. It needs to supply an answer, in fact that is the business model of many of these companies, no-one wants to spend a credit on asking a question and get a "I don't know" response. So in some respects we have asked for this situation we find our selves in.

(*) I know the modern term is hallucinations but I feel this usage masks a serious problem with modern AIs and the trust we put in them.
(**) model collapse is a major concern for the future of AIs.

Upgrading yarn from 1.x

Sun, 17 Dec 2023 23:20:01 GMT

I needed to update a project from yarn 1.x to the latest version (and maintain node modules) and though I have done this a number of times this year I always look up the instructions from the yarn website. Today I struggled to find them and then eventually realised that the instructions were on an older version of the yarn website (for 3.x) and that they were not included in the latest website (for 4.x).

So for my own sanity I am noting them here [Note this works for 1.22.19 to 4.0.2]

yarn set version berry    
yarn config set nodeLinker node-modules            
yarn install

Then a little testing and deduping doesn't go amiss here

yarn test
yarn dedupe
yarn test

Next, add the following to .gitignore

.yarn/*
!.yarn/releases

And finally, if possible, add it to renovate (updating according the node version) so that deduping happens on every package update "postUpdateOptions": ["yarnDedupeHighest"],

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"],
  "automerge": false,
  "labels": ["renovate"],
  "timezone": "Australia/Melbourne",
  "postUpdateOptions": ["yarnDedupeHighest"],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
      "matchPackagePatterns": ["*"],
      "automerge": true,
    },
    {
      "matchPackagePatterns": ["node"],
      "allowedVersions": "= 18"
    }
  ]
}

Once everything is stable and patched/upgraded then I look if it is possible to switch to yarn with PnP.

Photo by 🇸🇮 Janko Ferlič on Unsplash

CSP and Alpine.js

Sun, 22 Oct 2023 08:54:54 GMT

Whilst attempting to convert my blog to use a Content Security Policy (CSP) I was inundated with CSP violations relating to unsafe-eval that was nailed down to the use of the alpine.js package. Now Alpine.js does have a CSP friendly build but it is yet (at the time of writing) to be published to npm. The instructions require you to build and distribute your own copy of alpine.js which isn't ideal but once that hurdle has been overcome it is possible to convert our usage to be CSP friendly. The instructions on alpine.js are rather light in detail about what this entails and so I will detail some of my own findings.

Starting off

The navbar for this site is one of the controls utilising alpine.js and this is what it looked like before conversion.

  <nav
    x-data="{ isOpen: false }"
    class="..."
    @keydown.escape="isOpen = false"
    @click.away="isOpen = false"
  >
    <!--Toggle button (hidden on large screens)-->
    <button
      @click="isOpen = !isOpen"
      type="button"
      class="..."
      :class="{ 'transition transform-180': isOpen }"
      aria-label="Menu"
    >
      <svg
        class="h-6 w-6 fill-current"
        xmlns="http://www.w3.org/2000/svg"
        viewbox="0 0 24 24"
      >
        <path
          x-show="isOpen"
          fill-rule="evenodd"
          clip-rule="evenodd"
          d="..."
        />
        <path
          x-show="!isOpen"
          fill-rule="evenodd"
          d="..."
        />
      </svg>
    </button>
  ...  
  </nav>

Identification of the `unsafe-inline` violations

The following items cause the unsafe-inline violation to fire and will need moving to external APIs as detailed by alpine.js

{ isOpen: false }
!isOpen
isOpen = false
isOpen = !isOpen
{ 'transition transform-180': isOpen }

Creating and applying the external API

Following the initial advice and some investigation (okay some judicious logging) the following API was created

Alpine.data("navbar", () => ({
  isOpen: false,

  isClosed() {
    return !this.isOpen;
  },

  closeNavbar() {
    this.isOpen = false;
  },

  toggleNavbar() {
    this.isOpen = !this.isOpen;
  },

  generateNavbarClasses() {
    return { transition: this.isOpen, "transform-180": this.isOpen };
  },

  ...
}));

and wired into the html like so

<nav
    x-data="navbar"
    class="..."
    @keydown.escape="closeNavbar"
    @click.away="closeNavbar"
  >
    <!--Toggle button (hidden on large screens)-->
    <button
      @click="toggleNavbar"
      type="button"
      class="..."
      :class="generateNavbarClasses"
      aria-label="Menu"
    >
      <svg
        class="h-6 w-6 fill-current"
        xmlns="http://www.w3.org/2000/svg"
        viewbox="0 0 24 24"
      >
        <path
          x-show="isOpen"
          fill-rule="evenodd"
          clip-rule="evenodd"
          d="..."
        />
        <path
          x-show="isClosed"
          fill-rule="evenodd"
          d="..."
        />
      </svg>
    </button>
  ...
  </nav>

Repeating the above steps for other areas now means the site is CSP friendly and unsafe-eval does not need to be applied to the Content Security Policy for the site to work.

As always, your feedback is appreciated

Photo by Stephan Seeber on Unsplash

Protecting me from myself

Sat, 14 Oct 2023 22:31:05 GMT

I've mentioned in the past that I use renovate and dependabot to manage package updates for my repositories but sometimes though the build is good, the package update breaks my local development environment or doesn't follow my preferred practice. Since we get 2000 GitHub Action minutes for free on a personal account (and I have paid for minutes on some of my organisation accounts) I decide to use some of them for these sanity checks.

Yarn Package Deduplication

I have a preference to dedupe my packages and ensure that I have the highest version of each package that is possible. I do this so that when I am taking security updates I have found it is quicker to adopt these updates when my packages are already deduplicated. Usually renovate does a good job of deduplication (dependabot less so) but occasionally it doesn't dedupe the packages and I have to do this manually, but only if I know/recognise that this is required. Yarn v3 has a command for this yarn dedupe (for Yarn v1 I use the yarn-deduplicate package).

The GitHub Action for this looks like this

jobs:
  yarn:
    name: check yarn
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - name: Setup Nodejs
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: "yarn"
      - run: yarn install --immutable
      - run: yarn dedupe --check

Using this action I can ensure that the package.json and the yarn.lock file are in sync and that there are no packages that can be deduped.

Validate .tool-versions

Just recently I took a renovate update that was to upgrade to the latest version of node 18 and as part of the that update it updated my .tool-versions. However the version of node being upgraded to wasn't yet available via the asdf tooling; the irony of this update breaking my developer experience is not lost on me. Whilst researching how to build my own step/action to install asdf and then install the tools to check their availability, I lucked upon an action that already does what I needed and so I plugged that in.

The GitHub Action for this looks something like this

jobs:
  asdf:
    name: check asdf
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - name: Install asdf & tools
        uses: asdf-vm/actions/install@v3

Once these sanity check steps are wired into the branch any failures of these steps will highlight in the PR and in the case of renovate stop any auto merges that have broken the build.

Though, I have shown the jobs as separate actions for the above examples it is more efficient (reduced GitHub Action minutes) to merge them together and run them as one.

As always, your feedback is welcomed and appreciated.

Photo by Bernard Hermant on Unsplash

Lambda function not picking up new configuration

Fri, 22 Sep 2023 11:31:09 GMT

The other day we encountered a very unusual scenario with one of our production systems, that even after many years of working with AWS Lambdas took us by surprise; well at least me.

Background

We updated the environment variables of a number of Lambdas in our stack due to a deprecation of systems that now had newer replacements. We were really proud with ourselves that we had done this well in advance of the old systems being switched off and the platform worked as expected with the new configuration; we patted ourselves on the back and moved on. Then the legacy system was switched off and one (of many) Lambdas started to bleat. Looking at the logs we could see it was still trying to talk to the old system and yet we could see the new configuration for that Lambda in the AWS Console. What was going on!?

Investigation

As mentioned we could see the new configuration in the AWS Console and yet the Lambda was still using the older configuration. We did the usual knee-jerk actions such as force a redeploy; we weren't hopeful as the Application stack that this Lambda belongs to is often deployed several times a week due to npm and terraform module updates, but hey it is worth a try. We could see the the new stack was deployed as one of our environment variables was the latest build number and that had changed and yet the important environment variable to direct to the new infrastructure was still not being picked up. A review of the code also led to more head scratching as the Lambda was actually very trivial; validate some incoming data and ping the newer service with the result, it used only a couple of libraries for logging purposes so it should really be idiot proof. If any Lambda in this stack was going to cause us issues this one was at the bottom of the list.

It turns out that simplicity of the lambda was its downfall. Despite all the deploys of the stack the generated output code of that particular Lambda never actually changed i.e. the JavaScript was identical and had remained the same for at least the past month. It would appear that a physical difference of uploaded code is needed for a Lambda to pick up the new configuration from the environment variables. So a small tweak to the code, an extra logging line, and after the next deploy we could see the new config was now being used. Crisis averted!

Solution/Workaround

So that we don't run into this situation again, what we need to do was make sure the code changes on every deploy and thus force a refresh of the configuration via the supplied environment variables [these aren't secrets by the way]. We initially thought about generating a file with a build number or date and then pulling that in to each lambda but that felt messy. So then our thoughts turned to our build system and we identified a webpack plugin called BannerPlugin that can be used to inject snippets into the header or footer of the output file i.e. our Lambda code. What we ended up doing was something like the following in our webpack configuration:

import { BannerPlugin, Configuration } from 'webpack';

const buildNumber = process.env.BUILD_NUMBER ?? '0';

const config: Configuration = {
  ...,
  plugins: [
    new BannerPlugin({
      banner: `var __build_number__ = ${buildNumber};`,
      raw: true,
    }),
    ...
  ],
}

modules.exports = config;

Now, each Lambda we produce using this config is stamped with the build number and thus making each incarnation we deploy unique.

Note: If you don't have a build number to hand you could just use Date.now() or some other mechanism to generate a unique enough value.

Feedback

As always, your thoughts are always appreciated.

Photo by Brett Jordan on Unsplash

Some notes on setting up a CSP on a legacy site

Sat, 02 Sep 2023 10:08:15 GMT

Nonces instead of Hashes

Using hashes for small sites is okay when you have a small number of scripts and styles from internal and external sources but quickly becomes unwieldy when working with a large and the better approach is to use nonces. Since a nonce needs to be applied afresh each time a page is served then a placeholder is often used in your code and then replaced by server processing e.g.

<script nonce="**CSP_NONCE_PLACEHOLDER**" src="https://xyz.com/script.js"></script>
<script nonce="**CSP_NONCE_PLACEHOLDER**">
  ...
</script>
<style nonce="**CSP_NONCE_PLACEHOLDER**">
  ...
</style>

NGINX

A lot of sites use nginx and it is simple to replace the placeholder using a sub filter

set $cspNonce "${request_id}";
sub_filter_once off;
sub_filter_types *;
sub_filter **CSP_NONCE_PLACEHOLDER** $cspNonce;

The generated nonce can then be used later on when creating the Content-Security-Policy header.

Nonces and external ...

... scripts

The strict-dynamic keyword can be used with nonces to trust a script and simplify your CSP as you can ditch the domain whitelists (for script-src only).

... code snippets

Over the years your legacy code base has probbaly acquired a lot of code snippets especially by tracking networks that are so useful to help you understand your users e.g. googletagmanager. At the time these scripts were pasted into your pages they were probably ignorant of nonces and so will stop working properly when they now try to create a <script /> element. Since then google have created a nonce aware script that is compatible with the latest CSP version. Comparing the scripts (old vs new) the following line of code was added, to get a nonce from the page and apply it to the <script /> tag being created

var n=d.querySelector('[nonce]');
n&&j.setAttribute('nonce',n.nonce||n.getAttribute('nonce'));

Now a lot of these type of scripts (same era) use the same pattern/technique to inject a <script /> block onto your page and it is possible to tweak the above line for each scenario.

Note: This may be less of an issue nowadays due to cookie consent policies as these snippets are often refactored out to loaded only when the site visitor has consented to their use.

Inline scripts

It is common to execute JavaScript code on user actions e.g.

<select onchange="doSomething();">
  ...
</select>

To avoid supplying hashes for every snippet they can be refactored to use an inline script i.e.

<select id="picker">
  ...
</select>

<script nonce="**CSP_NONCE_PLACEHOLDER**">
document.getElementById("picker").addEventListener('change', () => {
  doSomething();
});
</script>

JavaScript URLs

Using a JavaScript URL on the href of an anchor (<a />) tag was often used to emulate a button but without all the default styling that came with a button and then using a library like JQuery to attach the proper event handler e.g.

<a id="clickme" href="javascript:void(0);" >Click Me!</a>

The recommendation is to convert them into button elements and attach the event via an event listener (as above) which is also better IMO for accessibility. If there are 100s (or 1000s) of these then this might be a mammoth task and so the first instinct maybe to use a hash of the script i.e.

'sha256-rRMdkshZyJlCmDX27XnL7g3zXaxv7ei6Sg+yt4R3svU=' # javascript:void(0)
'sha256-kbHtQyYDQKz4SWMQ8OHVol3EC0t3tHEJFPCSwNG9NxQ=' # javascript:void(0);

And there are articles that indicated that this is (or was) possible but browser compatibility isn't comprehensive. Alternatives such as using href="#!" may work better.

I've personally found that replacing the href attribute with a role with appropriate styling is cleaner and also improves accessibility e.g.

<a id="clickme" role="button" class="btn">Click Me!</a>

What else?

More to be added as and when...

Photo by AbsolutVision on Unsplash

Steps to setting up a Content Security Policy

Sat, 05 Aug 2023 05:11:20 GMT

As part of protecting your site and your users you should really consider applying a Content Security Policy (CSP) to your site. I am going to describe the path I took to applying a basic CSP to my blog and the tools I used to make my journey less painful.

But first what is a CSP and why should you care?

Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross-Site Scripting (XSS) and data injection attacks. These attacks are used for everything from data theft, to site defacement, to malware distribution.

"But I am already using the X-Frame-Options and X-XSS-Protection headers so I am protected right?" Probably not any more, modern browsers have dropped support for X-XSS-Protection and deprecated some of the capabilities of X-Frame-Options in favour of a CSP being supplied instead. A CSP is however extremely configurable/flexible and with that comes additional complexity. To help with that there are two headers that we need to be aware of when writing a CSP.

Content-Security-Policy - this is the header we will eventually be using to enforce the policy
Content-Security-Policy-Report-Only - this is the header we will start with, this header checks the policy but does not enforce it, allowing corrections to be made until all issues are rectified.

Both headers will generate messages in the console (Chrome at least) and through the use of the report-uri or report-to directives can send those reports to a 3rd party reporting service.

Getting started

To start building a CSP for a site we can start with a basic reporting header

Content-Security-Policy-Report-Only: "default-src 'none';"

The directive default-src is the final fallback for all of the other directives that are available in a CSP header, by setting it to none will force every possible violation to be reported and will allow full visibility of what interactions the site actually has.

Once the basic header has been applied, a refresh of the site will produce a very noisy report.

The number of reported violations may look daunting at first but they can be easily reduced by applying directives that refer to interactions reported.

It is not unusual to see default-src set to self rather than none as this can quickly cut the noise with little effort but it will also hide interactions that you may be unaware of; depending on the security posture of a site this may be okay.

Reducing the noise

Rather than set default-src to self, it is sometimes better for visibility to apply the self option to each of the applicable directives. Doing so will allow the browser to trust the host site for each of the interactions reported; in this case the browser will trust images (PNG, GIF, ...), js (JavaScript) and css (Style sheets) from the host site.

Content-Security-Policy-Report-Only: "default-src 'none'; 
  img-src 'self'; script-src-elem 'self'; style-src-elem 'self'"

Now that the self interactions have been dealt with it is now possible to deal with the violations relating to external interactions

This now becomes an iterative process of updating the CSP, applying it and reviewing the results. The report will inform as to which directive is being violated and possible remediation steps.

Content-Security-Policy-Report-Only: "default-src 'none'; img-src 'self'; 
  script-src-elem 'self' cdn.jsdelivr.net netlify-cdp-loader.netlify.app unpkg.com; 
  style-src-elem 'self' cdn.jsdelivr.net; frame-src app.netlify.com"

Unsafe?

Now the only violations being reported are suggesting the unsafe-* keywords as possible solutions. Any use of these keywords should be a temporary measure and plans should be made to replace them with something more secure.

There are several ways to deal with inline scripts and styles as suggested in the report. If there are just a few inline scripts and styles then using the suggested hashes is probably the quickest solution. This however can quickly become unwieldly and perhaps using a nonce is much better, this approach will require some sort of backend processing to ensure nonce(s) are generated and applied on each page impression. For modern static sites without a backend server available to generate a nonce, then extracting these scripts and styles into separate files is probably the better approach.

Content-Security-Policy-Report-Only: "default-src 'none'; img-src 'self'; 
  script-src-elem 'self' 'sha256-bX5JSFYfDe6u21arSDQFhDhQZCpkhyBKkXyrQB1U2rM='
    cdn.jsdelivr.net netlify-cdp-loader.netlify.app unpkg.com; 
  script-src 'unsafe-eval'; style-src-elem 'self' cdn.jsdelivr.net; 
  style-src 'unsafe-hashes' 'sha256-dH+oOZOdDv+MWU0F8bCZOoFHX0jFM4+bwNqOKujbv90='; 
  frame-src app.netlify.com"

Leading to no violation reports being generated. This is just one page and more updates to the CSP will be required as the rest of the site is tested against the applied policy.

Using hashes or nonces is recommended but if the violations are coming from 3rd party scripts then they are much harder to control especially if the styles or scripts are changing regularly. It is not surprising developers get frustrated and resort to just using the unsafe-* keywords and move on.

It is also possible to use the strict-dynamic keyword against the script-src directive, this will usually reduce the number of hashes to be applied against the script-src-elem directive. Unfortunately there does not seem to be an equivalent for style-src.

Note: With browsers that support CSP-2 then unsafe-inline is ignored if a hash or nonce is provided for that directive but it will still allow the site to work with older browsers that only support CSP-1.

Moving from reporting to enforcement

After a number of iterations no more violations should be recorded and the next step is to move to enforcement, but what happens if something is missed and something breaks for visitors to the site. As mentioned earlier there are two reporting directives that can be used to send violation reports to a reporting endpoint. You could create your own if you like but if you needs are light then a 3rd party service will probably suffice.

3rd party services

There are many providers to choose from and all offer similar services and capabilities. Adding a reporting endpoint to use one of these providers is as simple and only requires the use of the report-to or report-uri directives. The report-uri directive, though being deprecated, is still the easiest to configure; if you decide to use this then keep an eye on the browser support.

Content-Security-Policy-Report-Only: "...; report-uri <uri-to-service> ;..."

Here are a couple of suggestions:

Report URI

Report URI has a free offering that will suffice for a single site and get access to the basic reporting features. If a violation is reported it will be viewable hopefully enough information exists in the report to rectify the issue.

There is also a wizard capability where the reporting can be used to dynamically build a CSP. It should be noted that, under the free version at least, that the CSP generated default to using unsafe-* keywords rather than gather the hashes as we see in the console. This is because the hashes are not sent as part of the report.

The hash however is generated by Chrome in the console

Csper

Csper is another service that provides a reporting endpoint that you can use to collate reports. It doesn't, at this point, have a free offering but it does have a trial that may be worth investigating. It does however have a free Chrome extension that when loaded will gather the reports and then build a basic policy.

It is noted they do not grab/generate the hashes but instead suggest that any inline scripts and styles are rewritten instead; this is not bad advice but not something that can be extended to any 3rd party scripts.

The extension is useful to create a basic policy but will need tweaking before it can be used in production.

CSP Evaluator

CSP Evaluator is a handy tool that will analyse your Content-Security-Poilicy headers and check for any issues. It will provide suggestions on making your policy backward compatible for browsers that do not support the latest CSP syntax. There is also a Chrome extension that will check the policy on the current site but does not seem to, as yet, offer the backward compatibility suggestions.

Ready to go live?

Once the reports stop being generated and no more violations are being captured, it should be possible to switch the policy from reporting only to enforcement by changing the header from Content-Security-Policy-Report-Only to Content-Security-Policy. Now any violations from this point may cause site failures i.e. scripts will not run or styles may not be applied when expected. If you have a reporting endpoint then review any violations and decide, after repeating of course, if and how they need to be addressed.

Enough about XSS what about clickjacking?

X-Frame-Options is still the old faithful if you want to prevent your site being wrapped in an IFRAME element but if you would like your site to be wrapped but control who is allowed to do so then you should extend the CSP.

The CSP equivalent of X-Frame-Options is the frame-ancestors directive e.g.

Content-Security-Policy: "...; frame-ancestors 'self' ;..."

Afterthought

I believe one of my first attempts at XSS was when I got involved in the 'Bearding Bob' competition in 2002 on the Code Project website. To my total surprise I won, though not sure if it was for my code or my graphic skills; probably not the latter and not convinced about the former either so probably just that I read and followed the rules. The picture of the prize seems to have gone from the site but I still have it.

As always, your feedback is welcome and appreciated.

Photo by sebastiaan stam on Unsplash

Heroku wants paying, so time to move on...

Fri, 23 Sep 2022 11:03:57 GMT

For over ten years I have been using Heroku to host small servers using their free dyno offering, I have even hosted production servers on their platform because it was so easy to setup and manage. However, Since their announcement to cease their support of free dynos I have to find a new home for my blog, or more specifically the headless Ghost CMS portion.

Even though the end date isn't for another month or so, I find myself quite time poor and so I needed to move quickly. I realised there are a LOT of blog hosting offerings and the choice is just bewildering; I also considered Medium, as I read it a lot and have a subscription, but couldn't find a way to host legacy posts.

I needed a something simple, easy to migrate to, definitely easy to migrate from, and I that could host on my own domain. I already use Netlify to host my Gatsby generated blog and so I initially looked for solutions that would allow migration from one headless CMS to another but use the same site generator. I then realised that Netlify have their own CMS so I decided to give that a look.

A nice thing about Netlify CMS is that the blog content itself is hosted on your own Github account so you always have full control of the data. I also found that the local development experience was excellent in that I could tweak the articles as I migrated them across and instantly review them without having to go through a long build cycle and use up those precious Netlify build minutes.

Migration process

I don't intend to supply the scripts I used as they are a mess, I went fast and dirty here as I knew I would not need these scripts once the job is done. I will however talk about the tooling I used to migrate the data.

First up was to migrate the blog data itself. Ghost CMS has an api and so using the @tryghost/content-api package I was able to quickly grab the content.

import ghostContentAPI from '@tryghost/content-api';

const api = new ghostContentAPI({
  url: ghostURL,
  key: 'xxxxxxxxxxxxxxxxxxxxxxxxxx',
  version: 'v2',
});

const posts = await contentType.browse({
  include: 'tags,authors',
  limit: 'all',
});

The content provided however was HTML and the Netlify CMS, as do a few others, requires it in Markdown format instead.

To convert this to Markdown, I turned to another package turndown to do the conversion. Off the bat, it did a reasonable job but it failed to correctly handle code snippets that were using prismjs. What was needed was to extend turndown using a new rule to recognise these snippets and convert them correctly.

import TurndownService from 'turndown';

...

var turndownService = new TurndownService();
turndownService.addRule('prismCode', {
  filter: function (node) {
    return (
      node.nodeName === 'PRE' &&
      node.childNodes.length === 1 &&
      node.childNodes[0].nodeName === 'CODE' &&
      /^language-/.test(node.childNodes[0].getAttribute('class'))
    );
  },
  replacement: function (content, node) {
    const matches = node.childNodes[0]
      .getAttribute('class')
      .match(/^language-(?<lang>.+)/);
    return (
      '```' +
      matches.groups['lang']
        .replace('cmd', 'bash')
        .replace(',line-numbers', '') +
      '\n' +
      content +
      '\n```'
    );
  },
});

...

const markdown = post.html
    ? turndownService.turndown(post.html)
    : '';

With the posts now converted to markdown it is now possible to generate the post markdown files and map the data to the format expected by the Netlify CMS; the required content for a blog post is customisable and I suggest you give this some thought before committing the next step.

const template = `
---
date: ${post.published_at.slice(0, 10)}
title: ${post.title}
description: ${post.custom_excerpt ? post.custom_excerpt : ''}
featuredImage:
  src: ${post.feature_image ? post.feature_image : '/static/img/not-found-image.jpg'}
... more fields go here
---
${markdown}
`;

Finally, I needed to move the images as they were hosted on Cloudinary but the account was tied to the Heroku account and application which I intended to delete. I used regex to identify the images hosted on Cloudinary and saved the files to disk, and then used regex again to rewrite the new locations in the markdown files.

import fs from 'fs-extra';
import fetch from 'node-fetch';

...

const downloadImage = async ({ url, outputPath }) => {
  const response = await fetch(url);
  const data = await response.buffer();

  await fs.outputFile(outputPath, data);
};

Site generation

I also decided to ditch Gatsby and give 11ty a try instead as it appears to be more lightweight; if it is a disaster then I can always try something else now that I have my content in a convenient format. I also decided to change my comment platform to utterences as it is another github based storage solution and so I have full control of the data there as well.

Site generation is currently < 30s whereas with the Gatsby+Ghost setup this was 3-4 mins; this is important as previews (editorial workflow) use a branch and need building+deploying before they can be seen.

Afterthought

There is still some work to do, tags, search etc..., and I am not 100% convinced the Netlify CMS is the final solution as it has not seen a lot of love of late however I have my data in a format that should be easy to migrate if needs be.

Your feedback, as always, is appreciated!

Simplifying your package management and security

Mon, 19 Sep 2022 08:24:00 GMT

Every day seems to come with more package updates and security fixes and though I, like most I feel, tend to ignore them for personal projects, I do tend to keep an eye on those projects that are actively hosted/published eg my blog.

But how do you stop it becoming a full time job? I used to heavily rely on Github's dependabot but I was getting messages for upgrading almost daily; never mind the regular security patches and evaluating if they actually impacted my site.

I then came across Renovate by Mend (was Whitespace) and realised that this was the path to take if I wanted to get some balance to my life. Setup was simple and I can control which projects renovate was allowed to manage. Once connected, Renovate will create a PR which tells you what it intends to do

Once accepted, Renovate will then create a simple dashboard within your repository

Now the best bit of renovate, the configuration to save you time. If you have plenty of tests and are happy to release if your build+tests pass then you can automate the package management to merge them straight in. You can group like packages together that always update around the same time eg jest or eslint, and for noisy packages, I am looking at you aws-sdk, that seem to update almost daily you can schedule/limit them to just once a week.

I am still not 100% all-in on full automation but I have decided to allow minor-like patches on my devDependancies to be fully automated

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  ...
  "packageRules": [
    ...,
    {
      "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
      "matchPackagePatterns": ["*"],
      "matchDepTypes": ["devDependencies"],
      "automerge": true,
      "automergeType": "branch"
    }
  ]
}

Now all I need to do is manage the major updates and check on packages that make up the production deliverables.

Extending renovate

Unlike dependabot, Renovate supports a lot more tools, eg Docker, BuildKite, ... to name but a few and will also accept extensions from contributors. You may have noticed the .tool-versions (asdf) extension in the above images, I am glad to say that I was involved in writing that extension and though it only supports nodejs for now it should be easy for others to extend for their asdf managed tooling should they need to; I may update it myself but I only intend to do so for tools I need.

Security alerts

Though Mend does have a security module it is not something as a home developer I could afford so I still use dependabot to provide security alerts but as I am more on top of package updates I have found these happening less frequently.

Your feedback, as always, is appreciated.

Photo by FLY:D on Unsplash

Speed up your Docker developer experience on MacOS

Wed, 17 Nov 2021 00:00:00 GMT

Okay, if you are reading this, you probably already know that Docker (and not just Docker Desktop) on a MacOS can be awfully slow, especially for developers when the container is accessing files on the local drive via a volume mount. For the most part, this has not really been noticeable with the applications I develop against and so I have been happy to use the defaults. More recently though I have been working on a PHP application and due to its nature it required that "nearly" the whole development space needed to be mapped to a volume and the impact was very apparent even when using the gRPC FUSE option.

Starting point

Using a fairly standard delegated mapping like the one below, we found that the average time to open our home page locally was ~600ms (with the gRPC fuse setting)

version: '3.5'

services:
  app:
volumes:
  - ./:/var/www/:delegated

Though not the simplest page it wasn't the most complex we have to deal with, can we do better?

Using NFS Mounts

Now a common approach to addressing this issue has been to use an NFS mount instead, setting it up is quite complicated but thankfully it is well documented and scripts are readily available for us to use.

version: '3.5'

services:
  app:
volumes:
  - nfsmount:/var/www/

volumes:
  nfsmount:
driver: local
driver_opts:
  type: nfs
  o: addr=host.docker.internal,rw,nolock,hard,nointr,nfsvers=3
  device: ":${PWD}"

Switching to the above config for mounting the file system and running the same home page test reduced our average load time to ~290ms. This is a pretty good improvement but can we do better?

Using Mutagen

Mutagen is a file synchronization utility that can be used with containers and more recently, with the beta version at least, it can be configured using docker-compose files.

Installing

Installing is quite easy, though it may take some time; there may be a faster approach but I just followed the hints from homebrew.

xcode-select --install
brew install mutagen-io/mutagen/mutagen-compose-beta
mutagen daemon start

Configuring

Since we are using the beta we can define the mounts in our docker-compose files.

version: '3.5'

services:
  app:
volumes:
  - code:/var/www/

volumes:
  code:

x-mutagen:
  sync:
defaults:
  ignore:
    vcs: true
    paths:
      - .DS_Store
      - .git
      - .idea
      - .vscode
  permissions:
    defaultFileMode: 0644
    defaultDirectoryMode: 0755

code:
  alpha: "."
  beta: "volume://code"
  mode: "two-way-resolved"

Note: the permissions section was required as mutagen applies the default permissions of 0600 and 0700 for files and directories respectively however I require the permissions to be 0644 and 0755 so that I could launch the web application (nginx).

Finally, we need to use mutagen-compose instead of docker-compose on the command line (or in scripts). It takes a little longer to start due to the initial build of the docker volume; this can be tweaked by being more selective on what files need to be synchronized between the host and the docker volume.

Running the same test again, against the home page, we managed to reduce the average load time to ~160ms.

Summary

All of the above tests were done for a developer build, performance is much better for production builds which are hosted in the cloud.

method

load time (ms)

default

~600

nfsmount

~290

mutagen

~160

It seems that mutagen is definitely a candidate to mitigate some of the performance issues encountered with Docker on the MacOS and should be considered alongside other utilities like docker-sync (requires ruby) and docker-bg-sync.

As always, I hope you find this useful and your feedback is appreciated.

Night-night OpenCover

Sun, 14 Nov 2021 00:00:00 GMT

Over 10 years ago, I started developing OpenCover, an open-source code coverage tool for the .NET Framework. It started as a failed attempt to upgrade PartCover to be able to support 64-bit processes but ended with me starting from scratch and creating a new code coverage tool that would support 32 and 64-bit processes on the Windows platform. It has been an interesting journey, I have met some wonderful people along the way and I learned a lot in the process of managing an open-source project and supporting the community. However, all good things come to an end and I feel this is the time for OpenCover development to cease whilst the memories are good. I wasn't sure how to finish this post so I decided to try out a Q&A format.

Q. Why now? A. First, .NET Core - for OpenCover to support .NET Core was not going to be a trivial exercise, I felt it would need doing again from scratch and that wasn't a challenge I wanted to take on at this point. Next, Maintenance - whilst there were no new features being added, there was still a constant maintenance cycle that had to be managed due to security patches; each cycle the task got more and more laborious with no real gain eg just recently the build pipeline failed because of an external dependency that moved location and I realised I didn't have the heart to fix it anymore - that is when I knew the time was right.

Q. What are you most proud of? A. I am not sure where to start? I could talk about the little gems I feel are in OpenCover such as the IL parser that is used to instrument the code or solving an awful performance issue by using shared memory buffers. Now I know this is going to sound trite, but actually, the thing I am most proud of is walking into a new workplace and finding they were using OpenCover and didn't have a clue I was the one who kicked the project off; that really made my day and made me realize people were actually using this on real projects.

Q. What would you do differently? A. I had to have a long and hard think on this one, I believe what I should have done was to have a proper roadmap, stick to what I feel was the right thing to do and, not be so eager to please. There were some features that took OpenCover away from its core purpose of providing code coverage for unit testing in a CI/CD pipeline, that only made the project more complicated to manage and probably caused more headaches than just saying no. It was my own time I was spending on solving other people's (usually work) problems and I was not getting paid for it; some people did buy me a coffee though.

However, I didn't create OpenCover to make money but more to keep alive the knowledge I had from working on profilers back in 2005, especially as I felt that knowledge was leaving the community and only existed in the commercial realm.

Q. Would you sell OpenCover? A. No. Nor, do I intend to relinquish control of the name. This may sound harsh but there are too many bad actors out there who would buy access to a product that currently has 3K downloads a day (according to NuGet) and use it for evil. If you want to take over OpenCover then please fork it and give it a new name. Honestly, I won't mind, why should I, it is after all how OpenCover started in the first place. If you do, let me know and I'll happily redirect people to your codebase.

Thanks again for supporting OpenCover over the years - night, night.

Scheduling AWS Lambdas, a 101

Fri, 15 Oct 2021 00:00:00 GMT

I often find myself needing to schedule some automation and in the past I have (ab)used services such as Zapier or IFTTT (and other services of similar ilk) to achieve my goals. More recently I needed to do something for work, using the above services wouldn't cut it and I didn't have ready access to an internal cron-like service to do my bidding so I needed another approach.

Though not immediately obvious, well not initially to me at least, you can use an Eventbridge (or CloudWatch events) trigger with a cron expression and use that to kick off your lambda on a particular schedule.

To configure this in your template.yml it would look something like this

Resources:
  OperationToBeScheduled:
Type: AWS::Serverless::Function
Properties:
	  ...

  OperationToBeScheduledTrigger:
Type: AWS::Events::Rule
Properties:
  Description: trigger our scheduled lambda
  ScheduleExpression: cron(0 17 ? * SUN *)
  Targets:
    - Arn: !GetAtt OperationToBeScheduled.Arn

  OperationToBeScheduledLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
  FunctionName: !GetAtt OperationToBeScheduled.Arn
  Action: lambda:InvokeFunction
  Principal: events.amazonaws.com
  SourceArn: !GetAtt OperationToBeScheduledTrigger.Arn

First, you need to create the trigger and configure the ScheduleExpression. Next, you need to grant the trigger the right to invoke your lambda when it fires.

The cron expression is in UTC only so you'll need to do the mental gymnastics to get the trigger to activate according to your timezone and if necessary take daylight savings into account as well.

The best part here is that I can have as many of these as I like and not have to take on any subscriptions with 3rd party services; though now I will probably need to keep more of an eye on my monthly AWS bill.

The above snippet is for my future self and I hope you find it useful. Your feedback, as always, is appreciated.

Debugging lambdas in VSCode

Wed, 15 Sep 2021 00:00:00 GMT

I've recently been doing some lambda development as part of my involvement with a hackathon known as Random Hack of Kindness. We ended up creating a small API using AWS-SAM-CLI and using typescript + webpack to build each lambda.

The nice thing is that it is easy to run the lambda locally, so you don't need to constantly deploy to your AWS environment to test it. It also means that there is an opportunity to debug the lambda locally should the need arise; there are even plenty of instructions to make this work but none of them worked for me. I suspected the typescript + webpack combination was somehow involved but how to investigate it and if possible find a possible solution. All the applicable samples seemed to be plain javascript + node setups and were configured something like this.


  "version": "0.2.0",
  "configurations": [
{
  "name": "Attach to SAM CLI",
  "type": "node",
  "request": "attach",
  "address": "localhost",
  "port": 5858,
  "localRoot": "${workspaceRoot}/src/<path-to-lambda-function>",
  "remoteRoot": "/var/task",
  "protocol": "inspector",
  "stopOnEntry": false,
  "sourceMaps": true
}
  ]
}

So I obviously tried the following


  "version": "0.2.0",
  "configurations": [
{
  "name": "Attach to SAM CLI",
  "type": "node",
  "request": "attach",
  "address": "localhost",
  "port": 5858,
  "localRoot": "${workspaceRoot}/dist/<path-to-lambda-function>",
  "remoteRoot": "/var/task",
  "protocol": "inspector",
  "stopOnEntry": false,
  "sourceMaps": true
}
  ]
}

It was always possible to attach the debugger but the breakpoints for them were never triggered. I even tried several of the source map options available in webpack before settling on dev-tool: source-map as the one most likely to work.

Could it be how the map files were being built? Inside the generated map files, there were references to webpack:///... andI had seen other debug tutorials using sourceMapPathOverrides in the launch.json to assist the debugger in mapping these paths to the local paths; again tried various options but nothing seemed to help. Whilst reviewing the webpack documentation, I saw that the webpack:///... path appeared to be generated by devtoolModuleFilenameTemplate and this could be overridden in the webpack config file, a common option being [absolute-resource-path]. This produced a map file that had the full pathname and as this only going to be for local development builds I was not concerned about exposing my local drive paths; I can see why we wouldn't want this for files being distributed on websites though.

Bingo - we can now debug. This is what I now have in my webpack.config.ts file.

const config: Configuration = {
 ...
 devtool: isProduction ? false : 'source-map',
 ...
 output: {
   ...
   devtoolModuleFilenameTemplate: '[absolute-resource-path]',
   ...
 }
}

However, I was still not satisfied because with the localRoot setting the way it is, we could only debug one lambda at a time and we were planning on having many lambdas as part of the application we were developing, if we wanted to debug another lambda then we would have to update the launch.json. Changing the path to be something other than the lambda source location threw a message in the debug console that the app.map.js file couldn't be found; note: this message only showed if you happened to have the console active at the time. Would it be possible to use one of the inline source map options instead? After some trial and error, I lucked upon the following combination that allowed me to debug each lambda without having to update the launch.json file.

launch.json

{
  "version": "0.2.0",
  "configurations": [
{
  "name": "Attach to SAM CLI",
  "type": "node",
  "request": "attach",
  "address": "localhost",
  "port": 5858,
  "localRoot": "${workspaceRoot}/",
  "outFiles": [
    "${workspaceFolder}/dist/**/*.js"
  ],
  "remoteRoot": "/var/task",
  "protocol": "inspector",
  "stopOnEntry": false,
  "sourceMaps": true
}
  ]
}

webpack.config.ts

const config: Configuration = {
 ...
 devtool: isProduction ? false : 'eval-cheap-module-source-map',
 ...
 output: {
   ...
   devtoolModuleFilenameTemplate: '[absolute-resource-path]',
   ...
 }
}

I am still trying to determine which source map option is best and if it matters but at least we have a debug option available to us if we need it.

As always these entries are a reminder to my future self but if you have any feedback or tips they'll be appreciated.

AWS Control Tower, making my life easier

Tue, 31 Aug 2021 00:00:00 GMT

I've spent the past few weeks refreshing myself on the latest and greatest in the AWS IAM space and applying what I can to my own personal AWS account that I use for tinkering.

A recommendation is to have one master organisation (org) and then create a separate org to hold all the audit and log data (some guidance I came across also says to have one org for audit and another for logs) and then create other orgs for your production, testing, etc environments. The setup for such was going to take quite some time and probably a bit too much for a simple dev account. However, these types of setups are the sort of environment that I spend my working life in and I have had very little insight into what is happening behind the scenes. Now, I could've followed the rather tedious instructions (and probably make a lot of mistakes along the way) to do this manually or I can use AWS Control Tower to do the heavy lifting and then look at the final outcome and tinker with it, so that is what I did.

Once the process was complete (had to create 3 additional email accounts) I was left with my original org as Root and two additional child orgs.

Security - with two AWS accounts, that I'll refer to as audit and log-archive.
Sandbox - with a single AWS account, that I'll call sandbox.

I did notice that after the AWS Control Tower process was complete I wasn't really sure what to do next or even if there was something I should be doing to make my life a little easier.

Securing

At no point did I have to create any passwords for the three additional accounts and so adding complex passwords (thank heavens for password managers) and securing with 2FA was the next chore. Feeling a bit better, I wanted to see what was going on in the Security org and the resources they were holding. Swapping between accounts quickly became tedious (password + code) and I knew I needed to set up roles in each of the new accounts that would allow my IAM account in the root org to jump between orgs and see the resources within.

Account Aliases and Roles

AWS loves their 12 digit numbers but they are hard to remember and so I make use of account aliases as much as I can. Unfortunately all the good (and obvious/simple) aliases have gone and so you'll probably need to be inventive, but if you're lucky you'll find something you can tolerate.

In each of the three new accounts, I created a role called admin that granted the AdministratorAccess policy to the root account, using the Another AWS account flow; I also took the opportunity to give each account a simple alias. AWS Control tower also creates similar roles but they are extremely wordy (ie long and hard to remember) and also differ in name from account to account.

With these new roles in place, I can now log-in to the original root account as my IAM user (not the root user) and then use the Switch Roles feature to hop between the Root, Security and SandBox orgs quickly. For the account, I could use the account alias rather than the number and so in the UI so rather than admin @ 123456789012, you see admin @ <alias> which is a lot easier to remember.

If I had other users in my account then I would look into creating roles with fewer permissions (ie read-only) in the Security org and allowing others the use of that role instead.

As always this is a note to my future self but your feedback is appreciated.

Upgrading to Ghost v4 on Heroku

Tue, 24 Aug 2021 00:00:00 GMT

If you've read any of my earlier posts you'll know that I have been running the administrative side of my blog (using Ghost) on Heroku. But in this age, things move quickly and Ghost has since updated to v4 but looking at their docs the supported upgrade path is all around the ghost-cli tooling. However, as I didn't use the ghost-cli to create this installation then I'll need to resort to some manual steps.

As I had already trimmed the installation down to the bare bones that I required for headless operation, updating to the latest v3 flavour (as mentioned in the upgrade docs) of Ghost was trivial. Throwing caution to the wind, I forced the ghost version to the latest version and gave it a go. And... nope.

A quick delve and it seems all that that is required is a small change in to how Ghost is launched.

ie change server.js (as used in v3)

var ghost = require('ghost');

// Run a single Ghost process
ghost()
  .then( ghostServer => ghostServer.start() )
  .catch( error => {
console.error(`Ghost server error: ${error.message} ${error.stack}`);
process.exit(1);
  });

to now be

require('ghost');

Try again and the blog part of the site now launches. Now let's try the admin side and have a look at the new features. Uh-oh.

admin error

No solution forthcoming from the only other similar issue raised several months ago (that I could find). That number `3.0.0` just looks suspicious and the only place I could find it was in the package.json (version), a quick tweak and the message changed to match the version. So the final changes to my package.json was

package.json changes

And now I have a working v4 blog that I can push to Heroku.

Well other than that subtle issue with accessing the admin side of things that was relatively painless.

Heroku Pipelines

Up until now, I have been pushing direct to Heroku and then pushing a backup to Github. I am lazy (as well as cheap) and I only want to push once so I have now switched to using a Heroku pipeline and now when any changes are pushed to Github they are picked up by Heroku.

Heroku pipeline

A cute thing that this enables is that I can now see my Heroku environment on Github.

As usual, these posts are to remind my future self of what I was doing at the time, but your feedback is nonetheless still appreciated.

Oh my zsh...

Sat, 24 Jul 2021 00:00:00 GMT

When I moved to using a Mac for development and working on projects involving React.js etc I knew I was going to have to reintroduce myself to the command line and so I started a search around what is considered to be the best terminal emulator and shell to use. During this search, I stumbled across oh-my-zsh and I was an immediate convert. To quote the developers:

Oh My Zsh is a delightful, open source, community-driven framework for managing your Zsh configuration. It comes bundled with thousands of helpful functions, helpers, plugins, themes, and a few things that make you shout...

"Oh My ZSH!"

So after installing iTerm2 (the product of my original search) and switching zsh for my shell, I installed oh-my-zsh and started configuring it.

Themes

What immediately got my attention when I came across oh-my-zsh was the git integration and the ability for the terminal to show the git status of the folder I was currently in. Now all the themes appear to support this but the theme that really caught my eye was powerlevel10k, the clever use of fonts really condensed the information I needed to know; I finally settled on MesloLGS NF as my preferred font (it's a personal choice).

I use VSCode a lot and use the built-in terminal when live sharing during pairing sessions, so I wanted the same experience there, adding the following to the settings in VSCode resolved that.

{
...,
"terminal.external.osxExec": "iterm.app",
"terminal.integrated.fontFamily": "'MesloLGS NF'",
...,
}

Plugins

There are lots of plugins and I haven't even begun to explore them all and I am wary that too many plugins will really impact the shell loading times, though I believe that powerlevel10k has a workaround for that called instant prompt.

git

I think it goes without saying that I have this plugin enabled, it also comes with lots of useful aliases (of which I know about 10% of them).

asdf

I am constantly switching between node versions and I use asdf to manage which node version I should be using when it is important I am not using the default. The oh-my-zsh plugin shows the current version as part of the prompt. Though I only use asdf for managing node installations it supports many other languages such as ruby, elixir, erlang, and countless other community contributions.

zsh-autosuggestions

This plugin provides auto-suggestions as you type and is a great time saver.

zsh-syntax-highlighting

This plugin provides syntax colouring as you type and helps cut down on the obvious typos, alongside `zsh-autosuggestions` this has improved my command line experience.

Other plugins

100s exist, some are novelty ie chuck norris, cow talk etc and I am not so bothered about those long term - good for a laugh but in the end pointless.

Your thoughts are, as usual, appreciated.

Sharing "developer" secrets

Fri, 04 Dec 2020 00:00:00 GMT

Committing secrets to source control is a real problem for developers and I, like many others, have done it at least once. When it happens, the team then has to go through the awful process of updating the affected secrets and, if required, removing those secrets from source control as well (assuming they got pushed to a central location.)

Sometime ago, I came across a feature in .net core that allows you to store secrets in a json file that is part of you user profile and can be loaded by your application. By default it uses a new location for every application but with a little tinkering you can use the same location for several applications. The magic works due to the ability to "layer" configuration from multiple sources when using ConfigurationBuilder

var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddUserSecrets<Program>()
.AddEnvironmentVariables();

and then applying the UserSecretsIdAttribute that you can either set as an assembly attribute

[assembly:UserSecretsId("8d49e500-1a72-4596-bdfa-e56080b10b5f")]

or you can set it directly in the project file, which is more common for .net core projects.

  <PropertyGroup>
<UserSecretsId>8d49e500-1a72-4596-bdfa-e56080b10b5f</UserSecretsId>
  </PropertyGroup>

Once this is set in the project you can then access the secrets.json file from the location in the user profile and start overriding your configuration and not worry about the secrets being committed to source control.

e.g. if in your config you wanted to override a specific field

{
  "object": {
"key": ""
  }
}

then you can override this in the secrets.json like so, take note of the flattening of the json object that happens

{
  "object:key": "secret-data-here"
}

The by using the same identifier, usually a GUID, across multiple "application" projects will allow you to have a shared set of secrets for those applications. Of course you may prefer to use environment variables or even pull the secrets from a secure location e.g. Azure KeyVault, but it is nice to have options.

Like most of my posts these entries are more notes for my future self but from which others may benefit. Your feedback is appreciated.

Using Ghost as a headless CMS (part 2)

Sun, 15 Nov 2020 00:00:00 GMT

So getting around to this took a lot longer than I thought it would but I eventually made the switch and now I am running my blog using a self hosted Ghost using Heroku and I am also using a Gatsby generated blog (uses React) with Netlify as the build and hosting tool. Why so long? Well there were a few integrations I wanted to make sure I could get working properly before I finally switched over from a managed Ghost platform.

For a starting point I used the JAMStack Ghost/Gatsby starter from styxlab and added a few tweaks to the theme that comes with it. I chose this theme as it has a few features built into it that I like e.g. Infinite Scroll, and looks to have an active community in case if I have any issues.

Disqus integration

I've been using Disqus for comment management for a few years and so I wanted to make sure I could hook that into the new theme. There is already a Disqus package for Gatsby and so getting started was relatively easy. What I initially stalled on was working out how to customise the Gatsby theme such that the Disqus component would load and to do this I needed to understand what a theme means in Gatsby; it isn't just CSS and images that I am more familiar with, with Gatsby you can also override the React components.

First I needed to identify the component I needed to shadow and then add the modified component to my own repository in the correct location. Looking at the template page.js in node_modules\gatsby-theme-try-ghost\src\templates there is a Comments component that looked like an ideal candidate and this component can be found in node_modules\gatsby-theme-try-ghost\src\components\common. The Comments.js is just a stub

/**
*
* Placeholder for Comments
*
*/

// The actual component
const Comments = () => (null)

export default Comments

To shadow this I copied this to my repository and placed it in its shadow location i.e. node_modules\gatsby-theme-try-ghost\src\components\common\comments.js to src\gatsby-theme-try-ghost\components\common\comments.js. Finally, I modified the component to bring in the Disqus component

import React from 'react'
import PropTypes from 'prop-types'

import { Disqus } from 'gatsby-plugin-disqus'

const Comments = ({id}) => {
let disqusConfig = {
    identifier: id,
}
return (
    <>
      <Disqus config={disqusConfig} />
    </>
)
}

Comments.propTypes = {
id: PropTypes.string,
}

export default Comments

To test this I updated my Disqus site to allow comments to appear on any site on my domain so I could view this on the development site I had hooked up on Netlify.

I did notice that not all the blog posts had hooked up properly but you can use the management tools within Disqus to move the comments onto the correct pages.

Algolia integration

I've been using Algolia as a search engine on the site for a few years and I didn't want to lose that functionality as I use it myself to find specific articles. Getting started was again easy as I followed the tutorial on the Gatsby site but because I was using Ghost rather than markdown then I needed to make a few tweaks. First I modified GraphQL in the starter algolia-queries.js to fetch the the from Ghost instead

const indexName = `sitePages`

const pageQuery = `
{
  pages: allGhostPost {
edges {
  node {
    id
    updated_at
    slug
    title
    plaintext
    excerpt
    tags {
      name
      description
    }
  }
}
  }
}`

function pageToAlgoliaRecord({ node: { id, ...rest } }) {
  return {
objectID: id,
...rest,
  }
}

const queries = [
  {
query: pageQuery,
transformer: ({ data }) => data.pages.edges.map(pageToAlgoliaRecord),
indexName,
settings: { attributesToSnippet: [`excerpt:20`] },
  },
]

module.exports = queries

Next, once the index was built on Algolia, I made some small changes to the default behaviour in Algolia to make the returned results a little more efficient (by default it returns the whole search document).

With all this in place and the Search component added to the site the results are presented like this

However after each update to the site or adding a new post (i.e. this one) the returned data being returned as part of the query results had reverted to returning everything again. It seemed that all my manual settings had been undone except for the excerpt:20 in the attributesToSnippet section which I was supplying as part of my query above, it seems I am required to supply all these settings rather than set them in the Algolia UI, so...

const queries = [
  {
query: pageQuery,
transformer: ({ data }) => data.pages.edges.map(pageToAlgoliaRecord),
indexName,
settings: {
    attributesToSnippet: [`excerpt:20`],
    searchableAttributes: [`title`, `tags.name, tags.description`, `unordered(plaintext)`],
    attributesToRetrieve: [`title`, `slug`, `excerpt`]
},
  },
]

With the above integrations eventually in place, and sticking, I flipped the switch and moved to being a completely self managed Heroku+Ghost+Netlify+Gatsby blog.

One final advantage of pushing directly to Aloglia during the build process is that I can now deactivate the Zapier+Azure integration I was using when I first added Algolia support to my site.

[Update 18 Nov 2020] Some time back I blogged about creating a tags page so I could see all the tags I had created and find articles by those tags. I decided to repeat the task with the new site by taking inspiration from the starter theme and used a similar approach used to display the posts. Starting with a copy of index.js and customising the GraphQL query to fetch the tag data instead, including the post count. Then I created new components to finally display the tags in the same 1,3,2 pattern layout.

As always your thoughts are appreciated.

SSH tunneling from a Docker container

Sat, 05 Sep 2020 00:00:00 GMT

In a previous post I mentioned the approach I took to secure an RDS server behind a bastion server being hosted on AWS. Using SSH to create a tunnel to access this resources is well documented and so there is no need to labour on that. However I also needed to access these resources from applications that are running on Docker containers running elsewhere. I am sure there is more than one way to achieve this but I will describe the approach I took and has so far proven to be pretty stable.

I dismissed any code based options as they just made the application more complex to develop and test and instead went for an approach that was purely container based. To do this I used two applications

AutoSSH - to set up the SSH tunnel and maintain it,
Supervisor - to execute and run/monitor multiple programs.

AutoSSH is obvious but why did I need Supervisor? Well a container only has a single ENTRYPOINT (or CMD) which can be used to run an application or a script but in this case I needed to run several (at least two) applications at the same time and for both applications to stay running. This could be managed by scripts but I decided to give Supervisor a try instead.

Adding Supervisor and AutoSSH to the container

Initially I was running a dotnet application and launching it like so in my Dockerfile

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "My.Application.dll"]

I updated this section to fetch AutoSSH and Superisor and install them

FROM base AS final
# install additional packages
RUN mkdir -p /var/log/supervisor /run/sshd
RUN apt-get update && apt-get install -y supervisor autossh

# configure supervisor
RUN mkdir -p /var/log/supervisor
COPY ["web/My.Application/supervisord.conf", "/etc/supervisor/conf.d/supervisord.conf"]

# install certificates
COPY ["web/My.Application/certs/rds-ca-2019-root.crt", "/usr/local/share/ca-certificates/"]
RUN chmod 644 /usr/local/share/ca-certificates/rds-ca-2019-root.crt
COPY ["web/My.Application/certs/rds-ca-2019-ap-southeast-2.crt", "/usr/local/share/ca-certificates/"]
RUN chmod 644 /usr/local/share/ca-certificates/rds-ca-2019-ap-southeast-2.crt
RUN update-ca-certificates

# copy application
WORKDIR /app
COPY --from=publish /app/publish .

ENTRYPOINT ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

Finally in the supervisord.conf I configured it to launch AutoSSH and the original dotnet application.

[supervisord]
nodaemon=true
logfile=/share/conf/logs/supervisord/supervisord.log

[program:dotnet]
command=/usr/bin/dotnet /app/My.Application.dll

[program:autossh]
command=autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i %(ENV_SSH_ARGS_KEYPATH)s -N -L %(ENV_SSH_ARGS_MAPPING)s "-o UserKnownHostsFile=/dev/null" "-o StrictHostKeyChecking=no" %(ENV_SSH_ARGS_HOSTNAME)s
stdout_logfile=/share/conf/logs/supervisord/autossh.log
redirect_stderr=true

The /share folder is actually a file mount so that we can preserve the logs for later analysis. The key file is also hosted on a secured file mount and passed in via the environment variable SSH_ARGS_KEYPATH.

It is probably overkill to use Supervisor like this but it was a lot simpler (and I feel easier for others to follow) than writing lots of shell scripts to achieve the same.

As always your feedback is appreciated.

Setting up a bastion server on AWS

Tue, 01 Sep 2020 00:00:00 GMT

I recently had a need to further secure an RDS server so that it was no longer publicly facing but would instead be accessed via SSH. It is easy to mess something like that up if you need to setup all the parts manually but thankfully AWS have a quick start that does the majority of the heavy lifting. Once the deployment was finished I could move the RDS so that it lived on the private subnet but even when I was connected to the bastion server I couldn't connect to the RDS server.

What the quick start doesn't give any guidance on is what to do next once you have this all setup so what was missing? This is probably immediately obvious to anyone who lives and breaths AWS everyday but what was missing was than I needed to update the security group for the private subnet to allow Inbound traffic from the public subnet where the bastion server resides. Being picky I limited the access to just the ports I needed.

Once I could connect to the RDS server from the bastion server (I used netcat to test this), I then updated the Inbound rules of the security group so that only specific IP ranges could connect to the bastion server using SSH. To test the SSH tunnel I needed to enable SSH tunneling by setting AllowTcpForwarding yes in the /etc/ssh/sshd_config, followed by sudo systemctl restart sshd.service to allow the new settings to take effect.

Adding more users

Now that the RDS server was secured I needed to allow others access to the same RDS server via the SSH tunnel but I also wanted to restrict them to SSH tunneling, i.e. no interactive sessions on the bastion server. To achieve this I created a linux user that could be restricted to SSH tunneling only and would be used by the other users of the RDS

First, create a new linux user and switch to it

sudo adduser -m bastion sudo passwd bastion su - bastion

Then, create a .ssh folder and add an authorized keys file to host the public keys of each user

mkdir .ssh chmod 700 .ssh cd .ssh touch authorized_keys chmod 600 authorized_keys nano authorized_keys

Then add each public key (e.g. .ssh\id_rsa.pub) of each user that needs access. Finally as we don't want the "bastion" user to have an interactive session over SSH we add the following entry to the bottom of /etc/ssh/sshd_config and restart the sshd service again.

Match User bastion ForceCommand echo 'This account can only be used for tunnelling'

With this in place, only users with the correct private keys and on the expected networks can connect to the RDS server via the SSH tunnel.

I am sure there are other ways to achieve this.

[Update 18/11/2020] I would also recommend reading up on best practices for securing your bastion server as these change as we learn more about the threats we face.

As always your feedback is important.

Using Ghost as a headless CMS

Sun, 05 Jan 2020 00:00:00 GMT

In a previous post I described how I investigated hosting ghost for free using Heroku and how this will allow me to potentially pull data from Ghost and build it into a static site which I can also get hosted for free (or at least a lot cheaper than what I am currently paying). I did spend some time investigating various site scrapers that would pull the ghost content directly and build a static site and though I could get these (mostly) working locally I couldn't find anyway to run them simply/cheaply in the cloud so that I could automate the process and make the effort of getting the scraper work properly worth it.

Ghost however refers to itself as a headless CMS and so I thought that I'll look into what the community is doing in this space and this is how I came across Gatsby and the deployment/hosting platform called Netlify, as I had a few false starts I'll describe how I got it to work.

Installing and running Gatsby

First download the Gatsby CLI tool using npm and then prepare the starter project that is pre-configured to pull data from Ghost via its API.

npm install -g gatsby-cli
gatsby new gatsby-blog-ghost https://github.com/TryGhost/gatsby-starter-ghost

This will create a folder called gatsby-blog-ghost and inside it will be, after a little configuration, all that is needed to build a static site from the Ghost platfrom.

From the Ghost admin portal create a custom integration
Update the production entry in `.ghost.json` (found in the gatsby-blog-ghost folder) with the supplied API key and API URL
Test the settings work by and navigating to the supplied URL e.g. http://localhost:9000

gatsby build gatsby serve

The downside

The default theme is a not really to my liking and if I want my disqus and algolia integrations then I'll need to do some work on the theme, once I have found one that I like.

Automating the Gatsby build using Netlify

To get Netlify to build and host the blog requires us to push the Gatsby folder created earlier to an online git repository site such as Github and then configure Netlify to build the site from that repository.

Create a private repository on github and add the remote to the git repository in gatsby-blog-ghost and push all changes.
Using Netlify, create a "new site from git" and select the private repository. Netlify will immediately start building your site (and will do so on any change to that repository) and deploy it to the configured netlify.com site.
Automate Netlify to rebuild on ghost site changes by configuring a webhook (Settings/Build & Deploy) and then adding that webhook to the integration created in Ghost to be called when "Site changed (rebuild)".

Now what we have is a static site that will always rebuild whenever the blog is updated whether it be due to changes in content or theme.

Configuring the Netlify site to run as a subdomain

I has some initial issues getting this to work with my Cloudflare DNS as it felt like I had to move my DNS to Netlify and I didn't want to do that, however after some trial and error I managed to get this to work and now the Heroku hosted Ghost site is now being rendered using Gatsby as a subdomain of my main site.

Automating the Gatsby build using Render

This is virtually identical to using Netlify to build and deploy.

Create a private repository on github and add the remote to the git repository in gatsby-blog-ghost and push all changes.
Using Render, create a "New Web Service" and select the private repository. Render will identify the repository as a Gatsby site and set some defaults and then once confirmed will immediately start building your site (and will do so on any change to that repository) and deploy it to the configured render.com site.
Automate Render to rebuild on ghost site changes by configuring a webhook (Settings/Deploy Hook) and then adding that webhook to the integration created in Ghost to be called when "Site changed (rebuild)".

Now what we have is a static site that will always rebuild whenever the blog is updated whether it be due to changes in content or theme.

Configuring the Render site to run as a subdomain

This felt like it was a lot easier than configuring Netlify but this could also because I was now more familiar with the overall DNS configuration in Cloudflare and now the Heroku hosted Ghost site is now being rendered using Gatsby as a subdomain of my main site.

Summary

I think Gatsby may be the way to go here as I think just a little bit of theme-ing will be a lot simpler than getting a static scraper to work and then finding somewhere to host it.

Your feedback is appreciated

Hosting Ghost For Free

Sat, 04 Jan 2020 00:00:00 GMT

I've been running my blog on a paid Ghost platform now for several years now but as I rarely blog I sort of chafe at the hosting cost, especially as this is now the only bit of hosting that is actually costing me. I've toyed with the idea of a static site and as there are a number of static site generators out there that support Ghost, I can potentially use these one of generators to push the site to an AWS S3 bucket for static web site hosting.

If I do go down this route it would still require me to self-host Ghost so that I can access the admin site when I need to, and here is the rub as these usually cost money. I also wanted the simplest hosting I could find and since I am planning on using a static site generator at some point then the I don't need anything that is permanently online/available so I looked into what it would take to host Ghost on heroku.

Getting started

Well it turned it to be simplest thing ever as this is virtually a 1-click deployment and within minutes I have a Ghost (3.X) blog up and running on heroku. This particular installation defaults to using Cloudinary for image upload storage but we can reconfigure it to use S3 if we decide to. Once heroku had done its thing all I had to do was create the admin account and work out how to migrate the content. Luckily Ghost has an Export/Import content section in the "Labs" which deals with the majority of the blog content, unfortunately it doesn't have any mechanism to pull the images across so these had to be manually extracted and uploaded again. One thing I noticed was that during the uploading of images, the integration with Cloudinary did some optimisation on the images and the Ghost platform then referenced those optimised images instead; with the originals still available on Cloudinary but now marked with a _o suffix.

Custom themes

The default installation comes with a number of built in themes which will work fine however I've spent some time customising the Casper theme over the years so that I can have support from disqus (comments) and algolia (search) and I would rather like to keep them. Unfortunately the heroku integration doesn't allow for persistent saving of an uploaded theme due the ephemeral nature of the heroku filesystem, so as long as the heroku dyno stays up the uploaded theme will still work but as soon as it stops/restarts the uploaded theme is gone and the blog is broken.

There is however a way forward but it will require that we maintain our own repository with our custom theme(s) installed and push them to heroku. We can also use the method to keep our copy of ghost up to date should we so wish.

First I download and install the heroku CLI and test that the blog still works

git clone https://github.com/snathjr/ghost-on-heroku
cd ghost-on-heroku

heroku login
heroku git:remote -a YOURAPPNAME
heroku info

git push heroku master

I had to use the the `--force` switch when pushing but I wasn't concerned as I knew I could always use the heroku portal to revert back to the last known good deployment should it get messed up.

Time to add the custom theme

I removed the following entries from the .gitignore file so I could actually add the theme to the repository.

content/themes/*
!content/themes/.gitkeep

2. Next I created a folder for our theme in the /content/themes folder e.g. monkeysee and then added the changes back to the repository and pushed them to heroku.

git add .
git commit -m "Important changes"
git push heroku master

Once the changes are pushed, heroku rebuilt the site and once it was completed we could see the custom theme was available for selection, because the theme is now baked in we are unable to delete it via the admin site.

The result of this little experiment can be found here https://monkey-see-monkey-do-blog.herokuapp.com/ and now allows me to now experiment with the next phase which is introducing a static site generator.

[Update 18/11/2020] The heroku site is now locked as I have now done the move to only hosting ghost as a headless CMS on Heroku and using Gatsby to generate a static site.

As always, your feedback is appreciated.

What is this "waterfall" thing of which you speak?

Mon, 18 Mar 2019 00:00:00 GMT

I'm going to describe my personal views about managing large software developments. I have had various assignments during the past nine years, mostly concerned with the development of software packages for spacecraft mission planning, commanding and post-flight analysis. In these assignments I have experienced different degrees of success with respect to arriving at an operational state, on-time, and within costs. I have become prejudiced by my experiences and I am going to relate some of these prejudices in this presentation.

- Winston W. Royce (Managing the development of large software systems)

The above quote is the start of the paper that to those of us who were taught any form of IT at university in the 80's and early 90's (and perhaps later) has become synonymous with the software development life cycle (SDLC) model known as "Waterfall"; whether this association is correct or not I'll leave to the historians, or in this case Wikipedia. It does seem to me though that anyone who has actually read the paper past page 2, would never make the assertion that a waterfall-like model was something Royce was recommending and what he was describing was actually something rather more familiar to what we try to do today.

Page 2 of the paper shows the classic waterfall model (or something very similar because Royce never used the term waterfall himself) that I recall having to memorise and recall for an exam.

Waterfall SDLC

By the next few pages the model has gone iterative and by the time you reach the last page, you have the following, which I am going to call Royce's SDLC.

Royce's SDLC

Now, if you are one to read the words and not just look at the pretty pictures you will see references to "Prototyping", "Testing" and "Involving the Customer" which I personally don't recall being mentioned during the course and to the current me it also feels a little "agile-y". There is also a long section on documentation (Step 2) of which Royce has very fond of but then he was also dealing with spaceflight and probably peoples' lives. It is interesting to read that even 40 years ago, developers and documentation didn't seem to mix well, actually I think it is people and documentation don't mix and something that we thankfully do less of nowadays and have instead replaced it with living documentation such as wikis and backlogs.

Has anyone actually worked on a waterfall project?

I also had the chance recently to reflect over my resume, something that was long overdue, and that brought back a lot of memories about the projects I worked on, what practices we used then, and whether I would call those projects waterfall or something else. My first software development roles were at firms where hardware was part of the product and those engineers I worked with definitely iterated over their designs so it felt right to me that we did the same with software. All the big projects were somewhat iterative in nature, new features being added with regular releases. The first time I recall seeing a "big" specification was when I was on a UK government project in 2003-2005 (EU emissions trading, Kyoto protocol) but even then the development team was trying to use agile processes under the hood; not perfectly admittedly, we were just learning to do the agile-walk.

Some people claim to have done waterfall as I often get to see resumes with "skills: waterfall, agile, ..." etc but I do wonder if waterfall is an actual real-life process or if it is just used as a placeholder for when the SDLC is heavy-process, not-agile. I say a placeholder but I sometimes wonder if it is a straw man or even a bogeyman as it gives a target for those who dislike the not-agile world. I don't believe I have ever worked on a waterfall project myself, well not one that followed the flow that we would typically attribute to waterfall. I have however been on many-a-project whose process flow would look somewhat like the iterative cascade of Figure 10. I am still surprised that the waterfall model even came into being is it just feels unnatural and not how I, as a mostly self-taught developer, have ever thought would be a good idea. The cynic in me though feels it is an ideal process for the sort of company where change control and charging for changes is a big part of their operating/revenue model; it could be though that the charging model I am critical of was because of the single direction of the waterfall model and that backing up a step was so expensive.

As always, your feedback is appreciated.

Debugging NuGet Packages

Sat, 16 Feb 2019 00:00:00 GMT

Have you ever noticed that now we have embraced the micro-services architecture of doing things, instead of one big monolithic repository of code we have it all broken into smaller repositories of single/reduced responsibility...

To share these little bits of wisdom we package them up into little parcels that we can import into 100s of other projects distributed across our services. Then, something goes wrong and we can't debug what is going on inside these little gems and we wonder why we did this and consider if a monolithic repository was all that bad after all.

Yeah, me too.

So what can we do about it? Well I've picked up a few tricks, forgotten a few, and relearned them again once more. So I am going to note them down now for my future-self.

General debugging

Before we talk about debugging NuGet packages we need to talk about what we need to actually debug our code and how the debugger uses the various assets we have available to help us do that. The following items are pretty well common for most compiled languages on windows and linux platforms.

Our source-code: this is the stuff we write and where we are going to make changes should we find issues;
The compiled execution: this is the thing we build and execute, usually an assembly;
A map-file, this allows us to identify where in our source-code the execution is currently at when debugging.

For .NET this looks like the following

source-code

execution

map-file

C#, F#, VB.NET, ...

DLL, EXE

PDB, MDB

Now when we are debugging locally we have all these things but when we download a NuGet package it is unusual to find a PDB in the package, probably due to size, and it's highly unlikely that we'll see the source code packaged.

Debugging NuGet packages

If you're making your own NuGet packages then please consider generating and including the PDB files with your compiled assets. It will make your life so, so much easier because you'll have better quality stacktraces should an error occur and you can quickly locate where an issue happened, hopefully avoiding the need to debug in the first place.

Having the location of the errors in your stacktrace saves so much time in identifying where to look for the error from which you can then trace the cause.

System.InvalidOperationException: L'opération n'est pas valide en raison de l'état actuel de l'objet. File "C:\projects\opencover\main\OpenCover.Framework\Communication\MessageHandler.cs", line 113, col 21, in StandardMessage Int32 StandardMessage(OpenCover.Framework.Communication.MSG_Type, OpenCover.Framework.Manager.IManagedCommunicationBlock, System.Action2[System.Int32,OpenCover.Framework.Manager.IManagedCommunicationBlock], System.Action1[OpenCover.Framework.Manager.M... File "C:\projects\opencover\main\OpenCover.Framework\Communication\CommunicationManager.cs", line 60, col 13, in HandleCommunicationBlock ...

Before you ask, yes, you can generate PDB files for release builds of your assemblies; also please consider including any intellisense files in your NuGet packages if you have generated them but only if they are suitable for public consumption.

Even with the PDB files available you are not going to be able to debug very well, if at all, unless you have access to the source code. You may be lucky as you may already have it downloaded (or cloned) or you can pull it from the relevant source control system. Even then you will still need to make sure your source code matches the code base at the time the assembly was built and that can be a little tricky at times as you'll need to know when it was built and revert your code back to that point in-time.

Using ReSharper

If you are using .NET and have the right tooling there is a simpler way. When using Visual Studio with ReSharper it will simply allow you to debug any assembly without source code as long as the PDB file exists. You can either step right into the method you want to investigate and ReSharper will automatically create the source code for you by using the IL and creating equivalent source code. You can also use the Go To Implementation shortcut (usually Ctrl+F12) and start drilling into the source code of the 3rd party assembly placing breakpoints as you go before you start your debugging run.

What happens when you don't have a PDB? Well, you can still use the Go To Implementation shortcut. However whilst debugging, when you try to create a breakpoint, it will initially fail but if the PDB can be located i.e. in a symbol store, or it can be generated dynamically, it will prompt you to enable debugging and then you can continue debugging as before; generated PDBs are, by default, placed in %USERPROFILE%\AppData\Local\Temp\SymbolCache.

The one disadvantage of this approach is that the source code you are debugging is not your actual source code but it is a close representation of what it could be i.e. source code that when compiled should produce the same IL.

I also understand not everyone can afford ReSharper but all is not lost because the above functionality is also available as part of DotPeek which is free.

Is there another way?

There does appear to be a solution on the horizon (still in Beta at the time of writing but has strong support) that will allow debugging of packages and it is called "Source Link", it can be used with most git repositories such as GitHub, Bitbucket and Azure Devops etc and what it does is add a special link into your nuget package to your git repository along with the appropriate SHA, with this information and a supporting debugger (i.e. Visual Studio) can pull the appropriate source code resources. As I said it is still beta but if you are okay to update all your packages going forward then this is probably the future.

Over time I can see that the latest packages on NuGet will start to use this method as it makes sense in that it resolves the issue of stepping through the actual code rather than a close representation and by using the SHA it focuses on the files as they were at the time. Here is a demo of Source Link being used in Newtonsoft.Json.

What if I have no access to the source code?

You can still create a PDB for your assembly using tools that come with Visual Studio and you can access from your developer command prompt e.g.

ildasm /out=target_assembly_source.il target_assembly.dll ilasm target_assembly_source.il /dll /pdb

Once you have your new DLL and associated PDB you will now need to recompile using the newly rebuilt assembly, then you'll be able to step into the code, you will be stepping though the generated IL code though, this isn't perfect though and YMMV.

As always, your feedback is appreciated.

Adding a tags page to a Ghost blog

Mon, 11 Feb 2019 00:00:00 GMT

An odd thing about Ghost is though it allows you to tag your posts it doesn't provide an out-of-the-box way to access all the tags as a single view.

It is relatively easy to set one up yourself and give it a reasonably consistent look and feel to the main blog.

To start you need to enable the host public API, you'll find this in the Labs section of your ghost admin. Then, you need to create a static page with a post url of tags.

Now we have to create a special theme file called page-tags.hbs for that tags page. This page will be used to render content for this post instead of the default post.hbs.

{{!< default}}
{{!-- The tag above means - insert everything in this file into the {body} of the default.hbs template --}}

{{!-- The big featured header, it uses blog cover image as a BG if available --}}
{{#post}}
<header class="site-header outer {{#if feature_image}}" style="background-image: url({{feature_image}}){{else}}no-cover{{/if}}">
<div class="inner">
    {{> "site-nav"}}
    <div class="site-header-content">
        <h1 class="site-title">{{title}}</h1>
    </div>
</div>
</header>

{{!-- The main content area --}}
<main id="site-main" class="site-main outer">
<style scoped>
.inner-page-tags {
    margin-top: inherit;
}

@media (min-width: 900px) {
  .inner-page-tags {
    margin-top: -8vw;
  }
}
</style>
<div class="inner inner-page-tags">
    <div class="post-feed">
        {{#get 'tags' limit='all' include='count.posts' order='count.posts desc'}}
        {{#foreach tags}}
            {{!-- The tag below includes the markup for each tag - partials/tag-card.hbs --}}
            {{> "tag-card"}}
        {{/foreach}}
        {{/get}}
    </div>
</div>
</main>
{{/post}}

In the partials folder create another theme file called tag-card.hbs, this file will be used to render each tag card so that it has a similar look and feel to the posts when viewed as a collection.

<article class="post-card {{post_class}}{{#unless feature_image}} no-image{{/unless}}">
{{#if feature_image}}
    <a class="post-card-image-link" href="{{url}}">
        <div class="post-card-image" style="background-image: url({{feature_image}})"></div>
    </a>
{{/if}}
<div class="post-card-content">
    <a class="post-card-content-link" href="{{url}}">
        <header class="post-card-header">
            <h2 class="post-card-title">{{name}}</h2>
        </header>
        <section class="post-card-excerpt">
            <p>{{description}}</p>
            <p>A collection of {{plural count.posts empty='posts' singular='% post' plural='% posts'}}</p>
        </section>
    </a>
</div>
</article>

Finally add a link to your blog navigation so your visitors can find this new handy page.

Not dead yet...

Sun, 10 Feb 2019 00:00:00 GMT

So it has been a while but I am glad to finally say I have released another version of OpenCover - 4.7.922.

The release candidate has been out for a few weeks now with no issues reported so I decided to bite the bullet and get the latest release out; the releases, as always, can be found on Nuget, Chocolatey and Github.

I'd say most of the major issues were addressed and OpenCover should have better support for assemblies built using .net core thanks to the wonderful Mono.Cecil library. I've also hooked in Sentry to collect crash reports, we used to use DrDump with 4.6.519 but I feel that Sentry will help us be more proactive in identifying and fixing any issues. If you do have any problems then please raise them on issues page.

There are a few defects/issues that I'll probably have a go at in the next few weeks/months but I can't see any major features being added in the near future, I will however look at and accept any pull-requests that adds value to the project so feel free to dive in.

I also recently found out that NDepend is also supporting OpenCover coverage files now, so that is a big win to the community that have been asking for that capability over the years.

NDepend v2019.1 has been released with support for VisualStudio solution analysis, support for OpenCover and VS coverage binary format, 9 new security rule and more https://t.co/5cjhEHJgqz #ndepend #opencover #visualstudio #2019 pic.twitter.com/SEiPmza2Ha

— ndepend (@ndepend) January 31, 2019

As always your feedback is appreciated.

A spike on `robust` JSON handling in .NET

Tue, 20 Feb 2018 00:00:00 GMT

When working with JSON formatted data in .NET I have always found it frustrating that I am losing something valuable if I want to use some form of contract to help reason about the code and improve understanding about the entities being worked upon.

What did I just receive?

The first issue I often come across is that after I have deserialized the data I simply can't tell the difference between null and undefined (or missing/absent) e.g. if we had the following entity

public class User
{
  string Name { get; set; }
  int? Age { get; set; }
}

then if we deserialized either of the following JSON objects into it

{
  "Name" : "Arthur Dent",
  "Age" : null
}

or,

{
  "Name" : "Arthur Dent"
}

then in both cases, after deserialization, Age is null. Should we wish to serialize that entity we can either serialize Age as null or we can omit it, but, we can't do both i.e. we can never be 100% sure we are creating the same representation that was received and we could be adding or losing information that may have a detrimental effect on a downstream system that consumes the data.

I don't know about you yet!

Another issue I often encounter is if the JSON object received has more information than expected e.g.

{
  "Name" : "Arthur Dent",
  "Age" : 42,
  "Address" : "ZZ9 Plural Z Alpha"
}

If we deserialize and serialize again, then we will now lose the Address field and this is something we probably want to avoid.

In all these cases, there are ways to handle them but I find them tedious to implement and it probably leaves an ungodly mess behind for someone else to pick up.

Coupling of services

In the last example, where an unknown field is potentially lost, this is often solved by ensuring all services have the same contract. When you add/remove fields from a contract we often use a shared package or file and this leads to laborious upgrading of each service or client that uses that contact and then we have to deal with the subsequent deployment and synchronization issues that then ensue. This just feels wrong as we should be developing (micro)services that can be independently deployed but now we have created a dependency between these services and have effectively created a distributed monolith. This problem is probably better explained in this Channel 9 talk "Data/Contract Coupling in Messaging".

Robustness Principle

We have better things to do with our time. What I needed was a way to take a typed contract e.g. using an interface that describes the entities I want to work with, that I can deserialize JSON objects into but be tolerant of what it reads whilst preserving the original JSON structure unless I have actually modified it; this sounds similar to the robustness principle.

The Robustness Principle is also known as Postel's Law and simply states:

Be conservative in what you do, be liberal in what you accept from others

In the world of services (and now micro-services I suppose) this is sometimes referred to as the Tolerant Reader where we need to be tolerant of what we read so that our contracts can simply evolve without holding us back.

Introducing Hale.NET

Now there was no way I am going to be able to create a better Json.NET library for this purpose but I can use it to do the heavy lifting when handling JSON. I also feel I should try some AOP techniques such as interceptors (e.g. Castle Windsor Dynamic Proxy) or weaving (e.g. Fody) to remove some of the repetitiveness.

The spike

Since my spike is showing potential and that other people have expressed an interest I thought it would be better to share now rather than just tinker myself when time permits. The spike (definitely not production ready) can be found on Github and I have released it under the MIT licence. It currently only handles the most basic of object structures (no arrays or hierarchies) but it does

preserve the underlying data by using a JObject and an interceptor to handle the get_ and set_ of our properties.
has some basic handling for dealing with null vs undefined using exceptions and some extension methods.

Assert.Null(user.GetValueOrDefault(u => u.Age));
Assert.False(user.IsReferenced(u => u.Age));

Next steps

The next step is to handle hierarchies of objects and arrays because without the ability to do this, it is not going to be of any use whatsoever.

A weekend at Random Hacks of Kindness

Mon, 04 Dec 2017 00:00:00 GMT

Random Hacks of Kindness (or RHoK for short) is a hacking event that I've been involved with on and off over the past four or five years. Since we have just had a recent RHoK event in Melbourne (Nov 2017) I thought I should write down what my own personal involvement was this time so that others may get a feel of what it is like to become involved in RHoK or other similar hacking events.

RHoK is not a competitive hacking event with big prizes up for grabs but instead aims to be a collaborative hacking event where some of the hackers will switch teams when they cease to be of use to their current team or feel they can provide more value on another team. Rather than trying to use some company's new API and do something cool with it we instead look at trying to solve real-world problems for the changemakers who usually come from a charity, community group or social-enterprise and who have a real-world problem that they would like help with i.e. usually idea-rich, cash-poor looking for a kick-start. Another feature of RHoK, is the emphasis on continuing involvement in the same project between RHoK events through what has become known as RHolls (Rock 'n' Roll - get it?) that take place every 4-6 weeks; and even using your own spare time if you have any to spare. There is no actual commitment to be involved with a project after a hack event has finished but it is really encouraged and it is up to each individual to decide on their continuing level of involvement.

So who gets involved in RHoK? Well, it isn't just software developers or coders as these projects require people with a wide range of skills and backgrounds so that we can be effective and deliver the right thing. My friend Samara tweeted this picture

This is why you don’t put the computer scientist on hack weekend registration duty @rhokmelb 😉RHoKstar traits (v1.0) #hackathon #rhoksummer pic.twitter.com/3NIwdTBwu3

— Samara (@EvilAngelPixie) November 26, 2017

The hack weekend

Now that the stage has been set I'll continue to describe my involvement with the most recent Melbourne RHoK event.

Day 1.

The Changemaker Pitches

This was the final chance for each of worthy changemakers to make their cause known and to persuade any final non-committed hackers to join their team. In no particular order (mainly because I forgot) the changemakers on the day were.

Berry Street - Berry Street are back with their mission to help children in protection stay connected to their families and other important people in their lives. This changemaker, like several changemakers before them, are proof that you can get a team to engage in several RHoK and RHoLL events to get that project across the line.
Go Jane Give - Hoping that RHoK can help them extend the reach of the grassroots campaign that allows you to turn your talents and interests into a fundraiser for your favourite causes.
Carers Couch - Needs help to provide online support to carers of people with cancer. Martina of Carers Couch has introduced a new term "cabbage salad" that we are sure is to replace the word "spaghetti" when dealing with awful/complex code.
PollyannR - Wants to improve the way that regional creatives can get the funding they need.
Start Broadband - Helping disadvantaged Aussie families get online.
Shifra - Looking to use technology to increase/improve sexual and reproductive health resources for mobile populations, such as refugees.
Care to Compare - An enterprise where consumers will get to compare and choose health products with 100% of the profits going to charity.
Hitnet - Co-creates and distributes health information and services to marginalised communities. They need help in maintaining and managing the content for their 70+ (and growing) information hubs that are provided to local communities across the country.

Team forming and storming

Hackers who had already been to the information night held earlier in the month had already picked their cause and started to form around the respective changemakers, others quickly joined the teams that they felt they could contribute to. I joined up with Carers Couch as I had some previous interaction with them on the run-up to the hack event as I was asked to be their RHoK mentor and to help them prepare for the weekend. The team started off by introducing themselves and getting down to the task of determining what the needs were and what could be tackled in that weekend. As the team was large enough that there was even a possibility that several tasks could be tackled in the same weekend.

During this time another RHoK mentor came over and asked if we had any UI/UX people willing to jump ship to help another team that really needed one. Once we had a lull in our session I went over to the Hitnet team to get more details about what they needed and it looked they had already found a UI person and had now picked their goldilocks problem (not too big or too small, just right) and were stuck with another issue involving XML. After a quick discussion I realised I could probably contribute more to this team rather than to the initial Carers Couch team I was currently involved in, and so after a quick apology to the team, I moved tables.

After another round of introductions, we started to carve up the problem into a potential solution and assign some tasks to ourselves.

The problem

Dan Laush has written a wonderful write-up of the project itself so I'll only summarise here.

The Hitnet team need to update an XML configuration file for each hub on their network to control what content is to be displayed on each and they currently do this by hand for over 70+ hubs each time new content is supplied to them.

The approach

The team identified we had just the right skills to split into three to tackle each part of the intended solution of our simple multi-tier architecture. As I had already decided to jump onto this team to help them tackle the XML issue I decided to concentrate on the worker process that would take XML files that were used for hub configuration and update them based on data pulled from an API that was to be built by other members of the team. By breaking up the project this way kept the concerns separate and allowed individual teams to largely work independently of each other, mocking the data where required until the other team(s) were ready to serve us our data.

The worker process

Hitnet were already using Google Cloud Platform (GCP) to host their servers and so it seemed only right that we put the worker there right next to the servers/files it will need to interact with. An added bonus is that I had never used GCP before so now was a chance for a learning experience as well; a big plus in my opinion.

The worker process had the task of taking an XML configuration file for a hub and updating it with data retrieved from an API call. The data itself was the description of the modules, i.e. content, that are to be displayed on that hub. We quickly fleshed out a number of tasks needed to meet the objectives and check if the worker was a feasible choice.

How easy is it to create a Function on GCP?
How do we read and write files from/to storage?
How will we manipulate the XML files?
How will we make the API calls?

Assumptions

As we did not have direct access to Hitnet's GCP account (and since we are new to this it is right we should be kept well away) we made the assumption that the XML files were available via a GCP storage bucket.

Creating a Function on GCP is really easy and in minutes, I had created an account and got access to $300 of free credits to use over the next year. Node.js appears to be the default language for writing Functions on GCP and Google has also provided a number of examples; these examples had pretty well answered how to do steps (1) and (2) which was a win. The worker process itself is triggered by a simple HTTP(s) request and it seems it could also be triggered by a message placed on a queue but we have no need for that at the moment. There was no mechanism for a timed trigger like with have with Azure Functions but we could use a cron job or a service like Zapier.

Manipulating the XML, proved to be harder than it should have been and I eventually settled on using the xml2js NPM package as it used the xmlbuilder NPM package that would allow us to save the manipulated data back to XML again. We finally got to the stage where we had the ability to pull, manipulate and push back a single XML file. Finished the day by adding code to iterate each file in the target storage bucket and update them.

End of the day

At the end of the day, each member of the Hitnet team had something successful to declare: UI team - had a React app up and running and created their first forms. API team - had spun up an API using nginx and a Postgres database. Worker team - had a worker process that could do all the parts required for it.

Day 2.

Coming in by train

I had a look at the code I helped create the day before whilst heading back to the event and after all the hacking and tweaking the day before it was a mix of callbacks and promises and it felt dirty. However, this is not a difficult thing to resolve and so I used the time to clean up the code.

Putting it all together

After the success of the day before we were all ready to connect everything up and this really put the pressure on the API team to create the APIs we needed. Whilst I waited for the API team to create the many endpoints needed by the front-end team I spent some time adding some error handling, improving the detail of the console logging and, generally tidying up the code.

Once the API team were ready for me, we worked together to build the query that returns the required content modules for each hub. To consume the API I decided to use the node-fetch NPM package to make the API call and, because I dislike building URLs using string concatenation is this always seems to cause problems for someone later down the line, I decided to construct any URLs using the url-assembler NPM package.

The marketplace

At previous RHoK events we stopped at 3pm for presentations held in a lecture theatre but this year the organisers decided to use "The Marketplace". During the time allocated each team can spend time presenting and demoing to the other hackers and the judges and hopefully persuade those with remaining tokens to vote for them for the People's Choice awards (and yes you could vote for yourself if you wanted to but I don't know of anyone that did).

The future

There are still things to do with this function to make it production ready and we hope to deal with these in the next few weeks through the RHoll events that we have already booked up.

Security - we need some, not only to make sure only trusted users can use the new UI but to also make sure the API is not called incorrectly and to finally ensure the worker is not constantly triggered and use up our precious credits on GCP.
Integration - we need to integrate with Hitnet's GCP and see if we can access their files via the cloud storage that we assumed we could.
Process - improve the build, test, deployment process of the worker function. We are looking to hand this back to Hitnet at a later date so they can maintain this going forward.

Summary

I definitely enjoyed the weekend and I definitely learnt something new, even better when it is doing something that would be of actual value to someone. The code for the worker (like everything produced at RHoK) can be found on their github account i.e. hitnet-worker.

I believe everyone in the team tried/learnt something new, I saw this tweet from Liz who also joined the Hitnet team to work with Dan on the front-end.

Finally had a crack at building a usable React app at @rhokmelb for @HITnet_au. Also put on my UX/designer hat for it. So happy with what our team achieved! 😊 pic.twitter.com/CJc0O7JTUD

— Lizarrrrgh (´･ω･`) (@lnoogn) November 26, 2017

and my understanding is that Ankit and Peter, who made up the API team, were also in somewhat personally unexplored territory when building the API using nginx and Postgres.

Can you give something back?

Wed, 17 May 2017 00:00:00 GMT

Why does it seem, to me at least, that the vast body of developers will actively use software that has been contributed through open source efforts but will do nothing to add to that wealth and instead continue to rely on the efforts of others? How can we try to change that behaviour and get more developers contributing in some way?

I wrote about a similar topic some years ago and I referred to it as a type of welfare. Perhaps it was a bit harsh because it tends to infer that these developers are taking advantage of those contributing, whereas instead they are just enjoying the fruits of what those people have contributed and, more importantly, freely given for such purposes. In a way it is a bit like art in the public gallery/domain, in that once the code/software has been written and released freely, anyone can use it without diminishing the pleasure of it for someone else.

So why do some developers not contribute? I've tried to come up with some reasons I think may be the case, based on conversations I have had, and added my responses.

It's all about the money!

You believe that these projects make millions as suggested by this cartoon and you don't want to give your valuable time for free so someone else can make money? https://commons.wikimedia.org/wiki/File:ME_439_OpenSource.png

It is something to consider I suppose and there are some projects where a number of service companies have spun up around supporting the project/product in the commercial space but these projects are rare when considered against the number of open source projects out there, well known perhaps, but still rare. Though we should consider that when we use an open source projects in our own code we are saving millions for ourselves and our employers and clients and that is nearly as good as making millions. According to the open hub network the popular Json.NET package has taken 33 person years of effort with an estimated cost of $1.8m. I personally believe the cost is much higher.

Where do I start?

You do not know where to start or even if your contribution would be accepted? I suppose it can be a little scary but also a little exciting as well when you submit that first pull request, most projects have guidelines that help any new contributers to a project and it is unlikely that a submission is rejected outright. It maybe be pushed back a bit due to things like testing or coding style but that is all part of the learning process and the things that we do every day when we join new teams. There are a few initiatives out there that aim to get beginners involved in contributing to open source projects e.g. first timers only and though there are not many issues using the 'first-timers-only' tag at the moment, it is a start.

I first started to contribute to projects that solve, or nearly solve, my own issues (I would say that is still where the majority of my own efforts go) and just dived in and gave it a go; if I didn't fix it, well, who was to know except my own ego.

Will my bosses let me?

You are unsure about your legal obligations when to comes to contributions to open source projects? This is a tough one and right now I am going to use that irritating phrase that people use when they are about to talk something legal, I am not a lawyer (IANAL) but...

If you have an employment contract read it over and look for relevant clauses - a lot of these contracts have clauses from a bygone era and just haven't changed to keep up with the time, talk it over with your bosses explain them the benefits of contributing e.g.

Fixing an issue that affects your day to day work and would in turn make you more effective or the software you are developing for them better.
Personal and professional development by working on software and concepts that are not normally part of your day to day work, improving those skills will have a beneficial impact on your work.
You'll work on it in your own time, though you may need to get permission to use company resources e.g. laptop/software. You should try to get an agreement about when you use company time vs personal time i.e. if the issue you are addressing is directly related to work then you can use company time, treat it as part of the project and encourage others to contribute.

As long as you are not contributing to a open source product that rivals your company's business, most employers I'd have thought, will agree to some level of contribution - some more pro-arguments for contributing can found in this blog article.

What about my rights?

Some projects have a Community Licence Agreement for contributions and these agreements are more about protecting the project and making sure that your contribution can be used without restriction for both the project and for yourself. Open source projects often don't have the dollars to fight any legal battles and so this way these projects protect themselves from any predatory legal action. It is highly unlikely you will be divulging any trade secrets through any contribution you make and it would/should be pretty obvious to yourself if you are treading the thin line. Just like when you use an open source project, or pull a package from nuget, you should look at the licence that comes with it to make sure that you usage complies with your business activities, again IANAL so you may want to have it checked out if you are unsure.

I just don't have the time!

And this is probably the main crux of it all, it is tough finding time to learn outside of work especially with family obligations and keeping up with those social connections that we do, contrary to popular opinion regarding software engineers, actually have.

So this is my big ask, if you use open source projects in your work or personal projects all I want from you 4 hours a month.

Yes, less than 1 hour a week on average is all I am asking of you to try and contribute, hopefully you are doing more than this anyway for personal development and improving your craft. Perhaps integrate it as part of that development; want to learn a new programming language and hate just working through code samples? Find a project and just pitch in, it'll probably more fun and rewarding than just following trite "hello world" examples. Also, though not an argument you may want to use with your employers, it is useful to show your contributions to prospective future employers as it demonstrates a willingness to learn and grow that go far beyond the stock "I have a Pluralsight subscription" (as good as it is by the way but along as long as you actually use it) responses during interviews. Not all of us have website portfolios to demonstrate our capability and it goes far beyond the code challenges that we are often presented with as we move from role to role, so consider it an investment in the future you.

As always your thoughts are appreciated.

Aw shucks, thank you!

Thu, 20 Apr 2017 00:00:00 GMT

I just received this today and it really made my day and I thought I should share.

Hiya Shaun,

Just wanted to drop a quick email of emphatic thanks for your many years of work with OpenCover. I was tasked with the job just recently of introducing unit + service tests, along with things to ensure quality outcomes are being achieved with this (i.e., measurable outputs)... OpenCover was such a simple drop-in tool.

Having never used it, I was up and going in under 2 hrs, complete with HTML reports, and a wrapping cmd file so it's about as idiot proof as I can get things.

So... no reason to this email other than to make sure you know at least someone appreciates your hard work :-)

Hope life is treating you well!

Cheers,

As I said that really made my day, I am glad that someone has found our work on OpenCover useful and not just at places I/we happen to work at. I must also give a shout out to the other other contributors of this project over the years and give a big thanks to Daniel who wrote the reporting generator tool that makes the output from OpenCover look so good.

The importance of canary builds.

Fri, 14 Apr 2017 00:00:00 GMT

So now that Visual Studio 2017 is officially out I thought I would use the long Easter weekend to upgrade the OpenCover project to this new version and tackle any issues that I normally encounter when this upgrade time comes around. However before I can start this upgrade process I first need to tackle the bit rot that has recently set in.

Bit rot (or software rot) is when your code just starts failing due to lack of maintenance or a change in the build or execution environment.

Fortunately I have a weekly canary build that I previously set-up that amongst other things will alert me when this happens (as well as inform me if I get any new code quality issues), it started bleating, or should that be tweeting, a few weeks back so I knew I had some issues to rectify but not the actual severity. So in the space of a few months (I decided to stop work on OpenCover over the summer as I needed a break and time to look at other technologies) the following things happened:

AppVeyor upgraded its Visual Studio 2015 image to use a release version of .NET Core. This caused a number of issues that I couldn't repeat on my local build environment so I have temporarily resolved the issue by uninstalling it for my server builds.
The local and server builds don't build the same flavour of the .NET Core application so I needed to update the tests to take that into account; I intend to look into this more after completing the upgrade.
Badges were failing and so the landing page was looking a bit daggy.
The build was failing and was not reporting it properly, this was easy to resolve and I was surprised it had escaped unnoticed for all this time.

If it wasn't for the regular weekly build I would not have been aware of some of these issues and makes me glad that I configured that build to trigger regardless of whether I or anyone else had submitted code to the repository. Now that I have a working (and hopefully more reliable) build again, I am now ready to tackle the upgrade and more than likely I will have to update the tooling again.

Title Image: Getty, CC-BY

Keep your development workspace tidy

Sat, 01 Apr 2017 00:00:00 GMT

A big issue I find when working in today's modern development environment is the ever increasing number of git repositories we now have to maintain.

With each repository comes a number of branches that we create, push and pull as we progress the development of each service and over time they just start to build up like driftwood on a beach; not the pretty driftwood of photos and art-pieces but junk useless wood that have long served their purpose. Just recently I counted that in the space of a couple of months I had pulled in over 40 repositories and most of those repositories had multiple branches and in one extreme case I had over 30 branches, all of the branches had either been merged back to master or I had abandoned e.g. spikes.

Now I could have done the simplest thing and just removed each one individually as I went along or just devote sometime every so often and do the same but en-masse. Instead I decided to do neither and instead I wrote a script that I could execute from a bash shell and later refined it into an alias.

alias -p bfg='for d in */; { echo $d; cd $d; { git checkout -q master; git branch | egrep -v "(^\*)|(^\s+(master|dev|hotfix|qa))" | xargs --no-run-if-empty git branch -D ; }; cd ..; };'

I call it BFG because it shows no mercy and I only run it when I just want to tidyblow things up.

Let me describe how it works (and I am doing this as much for my sake as for yours.)

For each subfolder under the current folder
switch to the master branch e.g. git checkout -q master
list all the available branches e.g. git branch ...
filter out those in the 'keep' list e.g. ... | egrep -v "(^\*)|(^\s+(master|dev|hotfix|qa))" ...
if any extra branches found delete them e.g. ... | xargs --no-run-if-empty git branch -D ;

I now try to run this at least once a month, especially when we have delivered on a number of features, and I know that it is safe to do so. If I am feeling nervous about running the script I substitute the -D for -d when executing git branch as this will not remove a branch if it has not yet been merged with its upstream.

Any questions or suggestions for improvements please post below.

Adding search to your [ghost] blog - part 2

Mon, 27 Mar 2017 00:00:00 GMT

Okay this is the next step in adding search functionality to my blog.

The biggest let down of the original integration was that I need to remember to run the search updater every time I made a new, or just updated a, blog post, but what if I was writing the entry remotely or, more than likely, just plain forgot? At the end of the last post I mentioned the possibility of adding a IFTTT snippet (actually they call them applets) to automate the process, so whilst the idea was still fresh I decided to investigate the idea. Unfortunately I could not find a way for IFTTT to execute a node script when my blog changed but I could use it to send a web request to another service which could then execute a script when it received that request. For this second service I decided to use Azure functions because well I have some spare free credit per month and I might as well use it for something useful.

Setting up the Azure function

Creating an Azure function is relatively easy (assuming you have an account) but if you decide to use another service, e.g. AWS Lambda, then the below may provide hints about how you may implement similar functionality.

If you don't already have one, create a "Function App" (Compute).
Once created, Create a simple app that uses Javascript (node) and could be triggered by an API. [I also chose to set the authorization level as Function as I prefer some level of access-control.]

We need to pull in the algolia-webcrawler package, to do so upload the package.json file below and run npm install from the Console. [I upload all the files into the same folder as the function I am creating. I also find the Kudu console works best for this as it displays the output/progress of the command.]

{
  "name": "rss-ghost-algoliasearch",
  "version": "1.0.0",
  "dependencies": {
"algolia-webcrawler": "^1.0.3"
  }
}

Next, upload your configured config.json file that is used to control the algolia-webcrawler.
Now, replace the index.js file with the following snippet. [I did play about a bit here and finally decided that spawn was the method that worked best for me.]

var exec = require('child_process').spawn;

module.exports = function (context, req) {
  context.log('HTTP trigger function processed a request.');

  child = exec('node',
[__dirname + '/node_modules/algolia-webcrawler',
'--config',
__dirname + '/config.json'],
  {});

  child.stdout.on('data', function(data) {
context.log(data.toString());
  });

  child.stderr.on('data', function(data) {
context.log(data.toString());
  });

  child.on('close', function(code) {
context.log('[END] code', code);
  });

  res = {
// status: 200, /* Defaults to 200 */
body: ""
  };

  context.done(null, res);
};

Finally, run the application to test that it works. [You can see the output in the Logs section.]

Setting up IFTTT

Goto IFTTT and create a New Applet
For the "IF" choose RSS (new feed item) and enter the URL to the rss feed of the blog e.g. http://blog.many-monkeys.com/rss/
For the "THAT" choose Maker WebHooks and enter the url to the function e.g. https://functions-many-monkeys.azurewebsites.net/api/UpdateBlogSearchIndex?code=K2mp...
Test the trigger works - this may take some time

Assuming everything above works without issue, then whenever your blog updates the IFTTT applet will be triggered, usually within an hour - docs say 15 mins) and the search index on Algolia updated.

Using Zapier

I am not sure what is going on with IFTTT but I found I couldn't rely on it to run when I updated my posts so I tried a similar service called Zapier instead. Creating a zap is just as easy as creating an applet, RSS->WebHook, and the free plan also states 5 (or 15 mins according to the free plan) between checks.

To make Zapier work though I had to use the option "anything is different" for the RSS trigger, so the issue with IFTTT may be related to how Ghost generates the RSS feed that is being consumed by the trigger.

I would appreciate any feedback and suggestions for further improvements.

Adding search to your [ghost] blog

Sat, 25 Mar 2017 00:00:00 GMT

Recently I wanted to find something in my own blog that I had written about and though I only have a small number of posts compared to some, it still took more time than it really should have.

So I decided to look at adding some search functionality; because I am originally from Yorkshire it should be free, because I am lazy it should be relatively easy. A quick google search (the meta-irony is not lost on me) and a few dead ends, I ended up trying Algolia. Now this looks like it is a really powerful search tool and I am nowhere near going to be using it to its fullest potential for this blog but the demos I have seen look amazing.

Now it was so simple to get going I thought I should share what I did to actually integrate it to my site (and so add to the myriad of other tutorials on the subject). First you've probably already seen the search on the main index page but if not this is what it looks like

and when searching

okay it's not the prettiest set of results at the moment however it is functional and I can tweak it later.

Getting Started

Once signed up and goingskipping through the tutorial you arrive at a dashboard where you can create an application and choose a region. It is explained and if your user-base is largely based in a single region then this is really useful but otherwise I was a little confused so I just choose the place nearest to where the blog is hosted, which is probably the same region.

Once the application was created I was stuck (did I mention there was a tutorial?), I expected to find an "index this site" button or feature but it seems Algolia is API based and no such options existed for mere mortals like me.

Loading the search index

Another quick investigation and I came across an npm package named algolia-webcrawler which can be given a sitemap.xml and crawl the site/blog based on its contents. I took the sample config.json and modified it up to look a bit like so.

{
	"app": "blog.many-monkeys.com",
	"cred": {
		"appid": "<<app-id-goes-here>>",
		"apikey": "<<admin-api-key-goes-here>>"
	},
	"oldentries" : 86400000,
	"index": {
		"name": "siteIndex",
		"settings": {
			"attributesToIndex": ["title", "unordered(description)", "unordered(text)"],
			"attributesForFaceting": ["lang"]
		}
	},
	"sitemaps": [
		{"url": "https://blog.many-monkeys.com/sitemap.xml", "lang": "en"},
		{"url": "https://blog.many-monkeys.com/sitemap-posts.xml", "lang": "en"}
	],
	"http": {
		"auth": ""
	},
	"selectors": {
		"title": "title",
		"image": "meta[property=\"og:image\"]",
		"description": "meta[name=\"description\"]",
		"text": "h1, h2, h3, h4, h5, h6, p, li"
	},
	"formatters": {
		"title": "-"
	},
	"defaults": {

	},
	"blacklist": [

	]
}

For some reason, that I obviously haven't yet investigated, it didn't work with just the main sitemap.xml and so I added the additional sitemap-*.xml files that I thought were valuable at this stage. Once the crawler was executed with the above config, I found that I had an index that I could use for searching my blog.

Adding search to the blog

I did find another tutorial about adding Algolia to a Ghost blog but it was in Japanese and Google translate didn't help me much but it did have a plain HTML snippet at the bottom that I shamelessly took and chopped up as necessary to use with my blog; these changes can be seen in this commit on GitHub.

The example didn't work out of the box as-is for me but it also didn't take long to get it running with a bit of debugging and tweaking.

{{!-- The main content area on the homepage --}}
<main id="content" class="content" role="main">
  <div style="text-align: center;">
<input type="search" placeholder="Search posts" id="search-input" />
  </div>

  {{!-- The tag below includes the post loop - partials/loop.hbs --}}
  {{> "loop"}}

  <script src="https://cdn.jsdelivr.net/algoliasearch/3/algoliasearch.min.js"></script>
  <script src="https://cdn.jsdelivr.net/autocomplete.js/0/autocomplete.min.js"></script>
  <script>
var client = algoliasearch('<<app-id-goes-here>>', '<<search-api-key-goes-here>>')
var index = client.initIndex('siteIndex');
autocomplete('#search-input', { hint: false }, [
  {
    source: autocomplete.sources.hits(index, { hitsPerPage: 7 }),
    templates: {
      suggestion: function(suggestion) {
        return suggestion._highlightResult.title.value.link(suggestion.url);
      }
    }
  }
]).on('autocomplete:selected', function(event, suggestion, dataset) {
  console.log(suggestion, dataset);
});
  </script>
</main>

Pros

It works and is free, okay you have to drop the Algolia image on your site somewhere (I kept it in the search box where it made sense) but nothing really comes for free and it isn't that intrusive.
Easy to setup and integrate, writing this post took longer than installing the search functionality.
Multiple indexes - I can see these becoming useful if I start to tweak my integration so as to try different things out during development without affecting the current live-integration.
There appears to be a strong community to get plenty of help/support; in fact the snippet I used above looks like it may have initially come from here.

Cons

Could be simpler to get started with a built in webcrawler of some form from the dashboard itself - there could be some standard usage scenarios for blogs (they have a documentation indexer already so perhaps they are heading down that path).
You have to run the webcrawler manually each time you publish a new post to the blog, though perhaps a IFTTT snippet might be possible using the generated RSS feed, a post for another day perhaps.

It's astounding, time is fleeting, madness takes its toll, ...

Sat, 21 Jan 2017 00:00:00 GMT

Okay, starting a blog post using a title from the first lines of the lyrics to "The Time Warp" does seem a bit unusual but with what I intend to cover in this it will all make sense, probably.

Time... what is it really? I am not going into some weird arty-pseudo-psycho babble of the ephemeral nature of time but as a software developer I usually have to deal with time, and dates, in some form everyday and I find myself having the same conversations with other developers about how to handle date and time in specific scenarios. What I have found is that even though we, as people, use time every day in our lives we really don't always understand it and even when we feel we have a grasp we often have to remind ourselves about what we are doing. Even now when someone refers to something like "midnight on the 20th", I have to query "Do you mean the midnight between the 19th and 20th or the 20th and 21st?"; people usually mean the latter but for computers it is always the former. Now, the observant amongst you may have noticed that I started off with 'time' but now I have started to use the terms 'date and time' together and that is very deliberate as time on its own is pretty useless for most cases without the date and together they create the when. And now I am going to add a third term to the mix and this is where because date and time we observe in the real-world is also dependant on where you are and I think anyone reading this will have experienced this at some point.

My rules

I am not going to say my grasp on the subject is perfect, as obvious from above, and to help developers like myself there are some wonderful libraries that really take the grunt work out of the handling of date and time but even with these tools we can still mess it up because we forget about what it is we are trying to handle. Personally, when it comes to development I have a few simple rules that pretty well cover 99% of all cases that I have to deal with when handling date and time that I would like to share.

Rule 1: If in doubt, make sure you use UTC...

... and make it someone else's problem to convert it; this is pretty well my default position on most occasions when I need to deal with time.

UTC stands for Coordinated Universal Time and if you are confused about about the abbreviation well it was apparently chosen so that it does not favour any one language.

But regardless of the abbreviation employed it is an awesome thing for us developers as it gives us a common baseline that can be utilised by the languages we use in our daily development; it is unlikely that you will find a modern language that does not support some way of getting the current date and time in UTC e.g.

JavaScript: var when = Date.now();
.NET: var when = DateTime.UtcNow;

UTC is great for logging and auditing purposes, and really anytime we want to store a time-stamp against a specific piece of data or an event. Even better for computers, as long as they are correctly sync'd to a time server, then the UTC time you will get on each computer will be the same (I am ignoring things like clock-drift etc for this article but I acknowledge its existence and the issues it can cause). Finally UTC isn't affected by DST and so you don't get odd behaviour due to time when this local time change occurs.

DST stands for Daylight Savings Time and its application around the world is rather arbitrary with some countries adopting it, others ignoring and some using it, applying it all year round, abandoning it. And in some cases like Australia it depends on the state.

UTC is also great for storing past events because, the past doesn't change and so converting it to local time for a viewer is always predictable; this is where those libraries really come in useful as they can also take care of that oft-forgotten DST offset that sometimes needs to be applied as well as the users timezone (the when and the where aspects) when you need to present the time of the event back to the user.

Learn to read time

Most languages support the basics of dealing with dates and time but it is also good to make sure you also understand how to interpret some common time representations.

"2017-02-05 05:00:00" - 5am on 5th February 2017, however there is no timezone information and so will, more often than not, be interpreted as local time by most languages.
"2017-02-05 05:00:00Z" - 5am on 5th February 2017 UTC, the Z indicates what is known as Zulu time and is another, usually military/aviation, term for UTC.
"2017-02-05 05:00:00+00:00" - is also 5am on 5th February 2017 UTC, this time we have provided the offset in hours and minutes from UTC that the time represents.
"2017-02-05 05:00:00+11:00" - is also 5am on 5th February 2017 but this time in a timezone that is currently 11 hours ahead of UTC; this is equivalent to "2017-02-04 18:00:00+00:00".
"2017-02-05 05:00:00-07:00" - is also 5am on 5th February 2017, in a timezone that is currently 7 hours behind UTC; this is equivalent to "2017-02-05 12:00:00+00:00".
"2017-02-05T05:00:00+1100" - is also 5am on 5th February 2017, in a timezone that is currently 11 hours ahead of UTC; this is a common representation often seen, a T is used to separate the date and time and the offset is still hours and minutes but they are sometimes not separated by a colon (:); in this case however if the the offset is missing then assume UTC(*).

(*) “Your "common representation often seen" is ISO 8601. This is the universal standard. A pox on all those framework designers who bastardised it slightly because reasons...”

- I Shepherd

https://xkcd.com/1179/

If you do see a time stamp with an offset e.g. +11:00 or -07:00, then to convert it back to UTC you subtract the hours and minutes from the time presented, as shown in the above examples; please note that you should treat the +/- sign as you would in any addition operation.

Learn to write and parse time

Though reading time seems pretty straightforward for us humans, for computers it can be quite tricky because us humans are not sensible/consistent and some libraries don't default the way you would think they would e.g. 01/02/2017 is this Jan 2nd or Feb 1st, depending on where you are from will probably determine what you say; if you are US American then you will probably go with the former but the rest of the world will more likely go with the latter (exceptions apply). But what about computer libraries what will they do?

// Wed Feb 01 2017
console.log(new Date(Date.parse('2017/2/1')).toString());

// Mon Jan 02 2017
console.log(new Date(Date.parse('1/2/2017')).toString());

// Invalid Date
console.log(new Date(Date.parse('13/2/2017')).toString());

So we see that default parsing in JavaScript will assume that if you put a year first then it will be followed by month and then date but it will assume US American format if you put the year at the end. So my rule of thumb here is to always format dates as year/month/day and if I have to parse a date format that I didn't write myself to ask about the format used because you may need to work just a little bit harder when you parse it.

Rule 2: For real-life events, store the timezone...

... of where the event is expected to take place. This will then allow you to convert the stored time to another local time in another timezone. This is really important for events that are to happen in the future because the future is fluid and it is not unknown for some countries to change when they may trigger daylight savings with extremely short notice but if you keep your computer patched with the updates then you should be okay i.e. if you are on a Microsoft OS then KB914387 is the one to keep any eye on; you may find it quiet interesting to see what changes have happened around the world.

Now you may have noticed that I haven't said anything about whether the date and time of the event should be stored in UTC or local time and that is because it depends on your scenario.

Example 1 - TV Schedule. The "Local News at 6pm" will, more often than not, be scheduled at 6pm and this will happen day after day regardless of the DST in effect, even if DST usage was suddenly changed. In this scenario storing the local date and time along with the timezone is probably fine. The same would apply for any sort of events that stop and start in the same location e.g. meetings, restaurant bookings etc. You may even find that the timezone is largely unused if the event is just physical but if there is any electronic sharing of the booking to persons or systems outside the location then it is usually important to maintain the timezone component.

Example 2 - Travel. Events like travel where the start time and end time could possibly be in different times zones e.g. Air travel, then storing the date and time in UTC along with the timezone(s) will probably be the best approach. The complexity involved in planning flights etc would not adapt quickly to a sudden change in DST usage and so the people affected may just find that the local time of the flight has changed but not as it relates to the rest of the world.

Which timezone list?

When storing your timezone try to store and use a library that will let you use the timezone list from the tz database, this list usually breaks a timezone into a region and place e.g. Europe/London or Australia/Melbourne, and tends to have more cross adoption between languages than other, usually OS specific, lists.

This list is also more granular and deal with the oddballs of the world e.g. in Australia for instance we have:

NSW (+10:00), VIC (+10:00), SA (+09:30), ACT (+10:00) and TAS (+10:00) all of which use DST but QLD (+10:00), NT (+9:30) and WA (+08:00) currently do not.
a town called Broken Hill (Yancowinna) that is in NSW (+10:00) but instead shares the same timezone as SA (+09:30) aka "Australia/Broken_Hill".
a town in WA (+08:00) called Eucla (+08:45) which is in a timezone that has a +45 minutes offset to the rest of the state aka "Australia/Eucla".
and (finally) and island off NSW (+10:00) call Lord Howe Island (+10:30) that utilises a +30 minute DST during summer.

It's not just the wildlife that makes Australia just one strange place.

Libraries

There are many libraries out there for your chosen language that will help you handle time especially if your chosen language has limited support. I'll only list ones that I have used recently that have done the job asked of them (and support the above timezone list) rather than discuss the merits of each library.

.NET: NodaTime
JavaScript: moment.js and moment-timezone.js

Rule 3: Avoid using "Local Now"...

... and instead use "UTC Now" whenever possible, especially when processing 2 or more times. "Local Now" is subject to the computer settings the code it is executed on and when the wrong "now" is used the defects are really subtle and may not be easily repeatable on a developers machine. If you are in

Europe/Africa: then the odds you will not notice the issues during your working day but your users may report odd things happening late at night.
North/South America: if anything is going to occur it will probably be during the afternoon when the weird stuff happens.
Asia/Australia: the system will probably have odd behaviour during the morning.

The issues usually arise because developers have their machines running in local time but it is not unusual nowadays for servers in the cloud to be running UTC (with no DST applied) and so if a developer uses "Local Now" they should really use "UTC Now" and convert it to the actual timezone. Personally I would ban any local now usage from my code base and I would like that Microsoft make methods/functions like DateTime.Now obsolete.

Testing

Testing your time manipulations and conversions can be very tricky especially if you use the System API to get the current time in your tests. If you can, try and mock out the usage of such APIs; some libraries such as NodaTime supply one for you, with JavaScript something like Sinon.JS may be your best bet. You may also find it worthwhile that when you run your tests that you change the timezone of you development environment occasionally and check that your tests continue to work regardless of the timezone of the computer running the tests. You may also find that you need to check your test data with an external system so I recommend the meeting planner on http://www.timeanddate.com as it allows you to enter your own times and see the results for multiple locations.

...finally

These are the rules I try to abide by but rules can be broken sometimes and there will always be special cases.

If you have further suggestions about what has worked for you that you would care to share, or see or believe that I have made a grave error in my thinking then please comment below, I am always willing to discuss and learn.

Other Resources

Falsehoods programmers believe about time - Infinite Undo!

GMT Standard Time - A Gotcha

Configuring Cloudflare Page Rules

Sat, 14 Jan 2017 00:00:00 GMT

I moved to Cloudflare sometime ago so that I could take advantage of their tools such as free DNS, SSL, analytics, caching etc. to support my ghost based blog.

I wanted to host my blog on https://blog.many-monkeys.com but I didn't want to host http://www.many-monkeys.com nor http://many-monkeys.com anywhere and wanted to keep my options open i.e. if I wanted to use the domain for more than just a blog.

I still wanted to support https://www... as there are a number of old links out there that point to my old site/blog but I didn't want to pay hosting for a single page that would redirect someone to the blog page. When I read up on this I found some DNS providers have a URL record which is used to configure this sort of behaviour. However I couldn't find out how to configure this sort of record in Cloudflare and so I remained stuck on the single page hosting solution.

The other day I was clicking around the Cloudflare site and I stumbled on the Page Rules and after a bit of trial an error I got it to behave the way I wanted and so I thought I would share.

1. Configure the page rules

I decided to point both names at the blog with a temporary redirect as that will allow me to change it in the future.

2. Configure the DNS

I configured the DNS such that all the domains I wanted to support will route through Cloudflare (the orange cloud image) but instead of being forwarded onto the blog hosting site at ghost.io which would/does fail, the Page Rules above kick-in and do the redirect to blog.many-monkeys.com and all works as I wanted.

By the way you get three free Page Rules and I only needed to use two, I wonder if I'll ever find a need for the third.

Only time will tell

Wed, 13 Jul 2016 00:00:00 GMT

Recently, I decided to review all my old articles that have been scattered across the internet over the past decade or two with the idea of either tidying them up or answering questions that I may have neglected etc.

During this review I came across an old article from 2007 hosted on CodeProject that demonstrates a way of creating custom dialogs for Visual Studio setup projects and was not documented by Microsoft. Since I went through the pain of working it all out, I thought I should share what I'd found out in case anyone else was in the same boat and needed to do the same. The problem I now have with this particular article is that people are still asking questions about how to solve something using the technique I provided and all I want to do is shout "No, that was 2007 there is a better way now, there was even a better way then. Don't any of you read the background of why we did this? Please, just stop!". I say this because there are better ways of working with installers and I now feel guilty that I may have sent one too many people down a horrible path, but what do I do? I could delete the article but that would mean that those who have used the article would lose the information I had provided, so instead I decided to add a new section to the top of the article, a sort of a "Caveat Emptor" or "Here be dragons!"

I hope it works but only time will tell.

Death by a thousand cuts

Sun, 22 May 2016 00:00:00 GMT

Every year I decide to spend some time refreshing OpenCover i.e. upgrading to the latest tools such as Visual Studio, upgrading all the packages that OpenCover depends on etc. etc. and it is never a good time for me.

I don't know why I do this to myself, I know it is going to hurt and it's always the tiny things that somehow take ages to remedy. However I need to do this so that I can uninstall old versions of Visual Studio before I move on to addressing some of the latest features and issues within OpenCover.

Thing is, as projects go that I have to deal with on a regular basis, OpenCover is one of the simplest, it has around 20 projects and there are around 30 external packages and the majority of those projects and packages are there for testing purposes. I am quite happy about the level of testing (it is not perfect - but when is it ever?) and the number of serious defects are quite low; as of time of writing, the only defects in the issues are the ones we raised ourselves from the error logs we received and an issue relating to support for the latest Fakes that comes with Visual Studio 2015.

I've decided this time that I am going to list the steps I go through over the next few days (or weeks depending on whatever time I can find) as I do this again for 2016.

Upgrade the project from Visual Studio 2013 to Visual Studio 2015

[Note: that I didn't go for Visual Studio "15" because it isn't supported by AppVeyor yet and I am not a masochist.]
Installer project failed to upgrade - upgrade to latest Wix in the 3.X series and try again.
No errors during upgrade this time.

Building

Fix C++ issues due to std::hash_map no longer being supported.
Some C# projects no longer compile due to some Roslyn related intellisense error, what the ...?
[external package Mono.Gendarme can no longer be found even though it is in the references, another developer also found this and reported something similar on our gitter channel.]
This makes no sense, remove it, add it again - nope, remove it and get ReSharper to add it - nope.
Check the signing process [the package comes unsigned so I have to manually sign it; is it really that hard to add a strong name key everyone?] - no change, still works in Visual Studio 2013 though. Why me?
Upgrade to latest ReSharper [thanks again for the freebie JetBrains.], wait....
Still doesn't find the reference (yes, I know, it was wishful thinking.) It's there, I can see it, why don't you look harder Visual Studio - Roslyn - whatever...!
... have few beers whilst going round in circles and my google-fu is failing me.
New tack, use new ReSharper to decompile library into code and pull the bits I want into a new file, compile, move on - bite me!
Fix installers and other packaging steps due to assembly removal.

Testing

Seems changes to the compiler behaviour mean that some of the tests that use the output of the compiler to test how OpenCover works now fail, at least it is consistent in release and debug builds. Fix them up.
SpecFlow integration tests fail on command line. Have to install new plugin that supports Visual Studio 2015 to run them in Visual Studio using ReSharper.
Find I can't debug (use breakpoints that is) tests that have been built as Release even though full PDBs exist. Get latest updates of Visual Studio 2015 i.e. Service pack 2, wait... [I am sure I installed Visual Studio 2015 quicker than it took to service pack it.]. What have they done to the logo..? Never mind, let's just move on.
Breakpoints are now working and, okay, fix the parsing [how did we get away with that for so long] and the tests are now passing again.

Building

Build on the command line again... Success!
Build release build on the command line... Success!
Push to GitHub and run a test build on AppVeyor... Success!

Upgrading the Packages

First undo the hack from what now seems weeks away as I can now once again reference Mono.Gendarme; It seems the Service Pack fixed many things, I promise I am not bitter about this, much.
I am now on familiar ground, so let's have a look at what packages need upgrading using Visual Studio, okay 9 updates, 8 of them are used only for testing and 6 of them have gone through major release number changes; I am worried about these because if there is going to be a breaking change then it'll be those 6; it's not showing the solution only packages so I'll deal with them manually.
First, upgrade the 3 packages with the potentially least impact i.e. no major version number changes. [By the way, the UI for Nuget package management is far, far better than before]. Build, test... Success!
Now try the other packages one by one [this allows me to rollback or fix issues with just a single package at a time]. My biggest concern is Nunit and Specflow as these are dependent on each other and my "spider-sense" is tingling, so I am going to leave these to last.
Unity (used for testing only). Build, test... Success!
xUnit (used for testing only). Build, test... Fail. Update xunit.runners package, find I also need to update nuget.exe itself [at least that is just a one liner], only to find it is empty except for a readme file informing me to download another package. Download that package. Update scripts, build, test, ... Success!
Specflow (used for testing only). There are a number of packages all interlinked on versions and dependencies. Now running in a strange universe with two versions of NUnit packages in my solution; I know from past experience they will not play nicely with each in the same folder so will need to take the leap on NUnit soon. Run tests using ReSharper, tests fail due to execution folder no longer the same, fix them up and try again... Success!
NUnit (used for testing only). I've heard some war stories about this upgrade... Well that went well only one build issue (obsolete method) and easily corrected. Build, test... Fail. It seems more than just obsoleted types have changed, though not sure if it is a NUnit thing or a ReSharper integration thing. Ah, more tests failing due to execution folder has changed, lets solve them first as we have a solution for that. And, now we are left with a change in how we handle fields used for ValueSource, easy to remedy. Now I need to manually update the runner for the command line build. Build, test... Success! Because I've upgraded the test runner I'll push the build to AppVeyor again just in case something serious has changed.
log4net (used in multiple projects). Build, test... Success!

And we're done, well hopefully, nothing serious has been caught and we are now running on Visual Studio 2015 with the ability to, at last, start using the latest syntax available. Now this upgrade took about 3 days overall with a lot of elapsed time in between whilst I did other things (life) and waiting on stuff to install.

Improving your source code quality

Sat, 23 Jan 2016 00:00:00 GMT

One reason I work on Open Source projects such as OpenCover is so that I can try things out, experiment if you wish, sometimes it's TDD techniques or a new mocking framework, and sometimes it's tooling; some of these experiments were successes and some were successful failures; my experiment in using SpecFlow for unit testing was interesting but I'll never do that again; my knowledge in what I can do in SpecFlow however has greatly improved.

Tools help us produce better software, the better our tools and the more we know how to use those tools help us become better developers. Just because you have the best tools and splashed out many $$$ to access those tools, if you don't learn how to use them properly or at least the basics then you might as well save your money. Thing is, the lack of money in the Open Source world is our problem, we've devoted our time and usurped our work laptops to access those expensive IDEs and additional tooling, but can we really tell our partners we have personally spent thousands on some software or infrastructure to help make better code we are giving away for free. Thankfully we .NET developers now have access to Visual Studio Community Edition and we may be able to get away with the odd $100 here and there but Code Quality tools are probably at the more expensive end of the scale and out of reach for most individuals. There are however a few good Code Quality tools available to us open source developers for free, in most cases, and they are relatively easy to set up.

Coverity

Coverity was the first quality metric tool we integrated into the OpenCover pipeline; it handles C#, C++ and many others. This tool was suggested sometime ago by one of the OpenCover contributors and we addressed the more serious issues he found at the time but it took a little while before we got round to integrating it into the pipeline. Integration took about a day of tinkering locally and then building the scripts so that it would run on AppVeyor; I now ask myself why did we wait? AppVeyor had already preinstalled the Coverity package onto their images and added it to the path so it was quite a simple task. Once we have finally succeeded in uploading our first scan for analysis we got a clean dashboard and the ability to configure code exclusions and manage the issues.

As you can see, a number of issues were found and the team rallied around to fix them. Only a few defects were dismissed as false positives and only under one occasion did fixing one defect introduce another; we are after all only human. We now have it scheduled to run once a week to keep us honest.

You can also run Coverity locally on your machine and upload to their site (open source projects only). I'll provide the steps I used and there are some nuget packages about to help you if you wish to use them but I didn't really feel a need.

Installation steps:

create an account on Coverity and provide some project details for you project. You can't view the results for free anywhere other than the portal so you might as well do this.
download the package for your platform, unblock and unpack.

Running:

this is very simple as you just used the supplied tool cov-build.exe to run your build in our case it is build.bat build-release-platforms-x64 e.g. from the build script

Viewing the results:

you will need to upload your results you can use a nuget package for this but I used curl instead e.g. from my build script
wait... sometimes your code gets analysed really quickly and sometimes it doesn't, there are a few restrictions with open source projects such as frequency of submission and code size.
play...

SonarQube

SonarSource has an Open Source offering called SonarQube and even offers integration into their own online dashboard. This integration is not currently available to those who require a windows build platform so until they have implemented their push feature there is probably going to be some sort of hosting outlay to make your results publicly accessible.

SonarQube is a bit more verbose/pedantic than Coverity and found 13 critical defects. All of these were OWASP related issues due to the Console.WriteLine statements but since OpenCover is a console application they will all be Resolved as 'won't-fix' or 'false-positive'; still trying to work out what is the best approach. In fact it would be easy to dismiss many of the issues found as some of them are a matter of style; SonarQube does allow you to review all the rules and change their usage depending on how the team wishes to treat each rule. In hindsight, before we started fixing the defects found by Coverity, it might have been better to get both products working to compare the output and see if they found the same issues

One rule that I habitually turn off is the "Literal suffixes should be upper case" rule. This is a 'minor' rule that tries to insist that I write my decimals and doubles as 0M and 0F rather than my preferred 0m and 0f; I just think the latter is easier to read. They do at least provide a reason for each rule which in this case is, 'Using upper case literal suffixes removes the potential ambiguity between "1" (digit 1) and "l" (letter el) for declaring literals.' i.e. is it 0l or 01, which is fair enough for that case but to then blanket apply the rule is a bit excessive. There are some alternative developer fonts e.g. top-10-programming-fonts, that you can use in your development environments that make it easier to distinguish the 1 and l if it is such an issue for you and you don't want to apply the rule like myself.

I did find setting up SonarQube a lot trickier than Coverity as I also had to self-host, steps were available but just not in one place that I could find. For initial testing I just used the in-memory database but it has some caveats and I have since experimented with a MySql+SonarQube setup on windows and linux. The install steps I used for my initial windows hosted experiment follows

Installation Steps:

install/update your Java Runtime Environment
download, unblock, and unpack sonarqube
download, unblock, and unpack sonar-csharp-plugin and place in extensions
download, unblock, and unpack MSBuild.SonarQube.Runner

Running:

*change to match your setup

run sonarqube-5.3\bin\windows-x86-32\startsonar.bat
open Visual Studio Developer Prompt 2013/5 and run the following

\Projects\sonarqube-runner\MSBuild.SonarQube.Runner.exe begin /k:"opencover" /n:"opencover" /v:"0.0.0.1" msbuild main\OpenCover.sln /t:rebuild \Projects\sonarqube-runner\MSBuild.SonarQube.Runner.exe end

Access Results:

Access the site on http://localhost:9000 using admin/admin
change password of the setup if publicly accessible.
play...

We are still looking at how we host and integrate SonarQube into our pipeline and we my look into using the c++ community plugin

ReSharper

ReSharper is one of the best productivity tools about for a .NET developer IMO and it also has some built in code quality rules. We've been using ReSharper for some time now and often try to get to the ReSharper "Green tick of approval" on the files, in doing so we probably preemptively reduced the number of Coverity and SonarQube issues detected when we ran those tools.

Update (25/01/2016)

I eventually got round to integrating it into the build pipeline, I used a headless ubuntu vm and followed these instructions; remembering of course to not use the same default password. The results of these efforts can be found here.

Update (17/07/2016)

I have since migrated the information to sonarqube.com; or nemo as it used to be known.

A line in the sand

Mon, 03 Aug 2015 00:00:00 GMT

Just recently I read Rework again on my kindle as this book really resonated with me at the time and I thought it was about time I read it again; the book is from the guys at 37 Signals aimed at people starting a business. When I got to the section titled "Draw a line in the sand" I realised that this book is also appropriate to anyone who is thinking of creating/managing or getting involved in an open-source project/product; I am not talking abut flinging up some source code or sending a pull-request and going ta-da, I mean potentially committing several hours per week over many months if not years.

This particular section in the book talks about drawing a line under what your product does and doesn't do and asks what are you willing to leave out to keep the product true even if it means some of your customers don't get the features they want. We see this far too often with great commercial products that start to become bloated as they try to chase every possible sale with more and more features that fewer and fewer people use and with an open source project I think it is all too easy to fall into the trap of fulfilling each feature request because you may think "great, someone wants to use my project but only if I do X, Y and Z by next week." I've had this scenario many times over the years with OpenCover and I have to decide whether or not to implement a feature (also do I have the time to do it and how will I support it) so I've had to draw a line in the sand many times using the following criteria "do I need it". It really is as simple as that, if I don't need it myself I am not going to spend my time on it. Now that doesn't mean I will reject code from other developers that meets their needs and enhances the product (especially since they have spent the time and effort to share their work, that would just be churlish) but I have to draw a line under what I am willing to commit to in order that I enjoy working on the project and not feel I am working for someone else, for free.

One feature I've never implemented is creating a UI for OpenCover even though PartCover (OpenCover's predecessor) has one, as I felt the UI provided by ReportGenerator was awesome and did the job perfectly that I felt anything I did implement would always be inferior. More recently another UI related feature that I didn't implement due to lack of personal need was "Visual Studio Integration", I use ReSharper as a test runner and I don't actually look at the code coverage until I think I have all the right tests in place; this keeps me honest, I believe, in that it I try to write tests for the code rather than tests for the coverage. However some members of the community wanted one and went and developed an extension themselves, actually there are two extensions currently in development OpenCoverUI and Testify which I think is great.

Another example is a feature that has sat on backlog for over 2 years "Support Coverage of Windows Store Applications on Windows 8". I know how to do it, well I have some guidance from David Broman at Microsoft so I know where to start, but I don't actually need it. When I write a windows store or phone app I put the majority of the code in an assembly separate from the UI and use unit testing to get the coverage. I don't care that much about coverage from integration testing as I find the results I get from unit testing far more valuable. Now I know people sometimes in the pursuit of 100% coverage want to get coverage results from integration testing and have some automated UI tests etc and OpenCover will usually support them. Hooking into windows services was implemented by someone who needed that capability, I tested and accepted their pull-request but I've never had to use the capability. IIS integration is tricky, I always use IISExpress myself if I need to go down this path, but the guidance was provided by someone who needed it themselves and so I shared it. One day I may need to implement this particular feature so I am not going to take it down and perhaps if someone wants it beforehand the guidance is on the issue if they want to implement it; though there has been no request for it though either.

I could list other sections that seemed related and applicable to open source projects e.g. "Be a curator" or "Don't be a hero", but frankly I think I would just be listing the entire book contents, best you get Rework yourself and read it and then read it again; each time I read it I get something new from it.

Image - Mandala, Coney Beach 4 via photopin (license)

Free stuff for Open Source .NET development

Sat, 25 Jul 2015 00:00:00 GMT

For the past few years that I've been been working on OpenCover I've had the opportunity to use a number of tools during its development, a few of those are commercial tools that have been made available for free to developers of Open Source projects or just to the project itself because I asked nicely.

Some of those tools I still use and some just carried the project through part of its journey so I thought it would be nice to give those tools a shout out and perhaps other developers of Open Source projects in this space may find them useful.

Development

Visual Studio - Community Edition - When I first started developing OpenCover I had been previously supporting PartCover which I had upgraded to support .NET4 so that we could use it for work. Then as I started planning/developing OpenCover I was lucky to have been given a free MSDN licence. Since those days however Microsoft have decided to release a Community Edition of Visual Studio that we developers (subject to some restrictions) can use the latest and greatest tooling and really keep ahead.

GitHub - Unless you've been living under a rock for the past few years you probably know what GitHub is; for those rock dwellers it is a source code repository for those using Git

Resharper - created by JetBrains and probably the mainstay of most .NET developer toolsets at work (after Visual Studio of course). If you have used ReSharper (or any of the JetBrains) tools you'll know just how more productive you become when you use their tools.

NDepend - was one of the earliest supporters of OpenCover. NDepend also consumes the coverage output from OpenCover (after converting it to the NCover format) and integrates that with its static analysis of your code to produce a wonderful dashboard of your code.

BitBucket - similar to GitHub it is a source code repository but supports Mercurial as well as Git.

DotPeek - another great tool from JetBrains and is free. It integrates well with ReSharper but is missing some features such as show IL; when I need to go that low level I use ildasm or more often than not I use ILSpy.

Continuous Integration

Bamboo - another tool provided by Atlassian, this was the first build system used to build OpenCover based on commits pushed up to GitHub. It wasn't totally free as I still had to pay the compute usage of the windows image I was running in AWS but it rarely went over $5 a month so I can't really complain. It was quite configurable and as I already had a command line build script it was easy to set up and integrate. It was lacking a few features at the time such as building on pull requests etc but it was useful as it meant that I could check that anything I pushed would build on another developer's machine.

AppVeyor - is awesome, I can't rave enough about how great this CI system is for .NET developers. It's so easy to set up that I use it on some other .NET projects that I am involved with e.g. OpenCover.UI and AccountRight Live API SDK. It works with pull requests and the nicest thing is that it also integrates with Nuget/MyGet and GitHub so I automate the pushing of releases by just pushing the code to a specific branch.

Reporting

ReportGenerator - is a great tool for visualizing the output from OpenCover.

Coveralls.io - I use Coveralls to display online the test coverage of my projects, yes I actually use OpenCover to get coverage statistics of OpenCover. The online nature means that any contributors of pull-requests can get that coverage feedback.

Moving blogging platform to Ghost

Sat, 18 Jul 2015 00:00:00 GMT

Welcome to my new blog. Like it? I hope you do as I know I prefer it to the the Blogger version I was originally using. Moving platforms had its ups and downs so I thought I would detail a bit of the journey.

Getting started was quite easy, I decided to trial it with Ghost where the nice guys there will do the import for you (of course they will, they want you to sign up with them). Most of the transformation went without a hitch but I have quite a few code samples and not all of them converted properly and so I had to do that work myself. Thankfully Ghost uses Markdown so with a bit of abuse of the

tag I managed to get some semblance of order to the posts. But it wasn't perfect as the code samples were rather plain and I rather liked the code-highlighter script I was using on Blogger. To get round this I needed to work out how to create my own theme.

Creating your own theme sounds daunting doesn't it, I know I was already in strange waters so I was initially hesitant to go down this path. I looked at using the Code Injection section but unless the highlighter code I was intending to use had a CDN or I had an alternate hosting location this was going to get messy and very quickly unwieldy. I also wanted to use my little purple monkey icon for the favicon.ico and I couldn't see how I could do that within the management portal. So, I had a deeper investigation into how themes are deployed into Ghost via the portal and quickly realised it was just a zip'd up folder of files. I actually liked the default Casper theme so I used Github to fork the repo and then branched again to make my changes. I know it seems complex but I wanted a way to apply any fixes back (should, in the unlikely event, I find any) but more so I could take fixes from the original and apply them to my branch and even cherry pick changes should I so wish. Eventually I suspect the changes I will probably make will make this a near impossible task but until then...

So applying the favicon.ico change seemed the simplest thing to start with and with a little help from Ghost For Beginners (an excellent resource) I was on my way; you can see my change here on github. Emboldened I decided to try my next change which was add a code highlighter script. It was suggested I try Prism and as it had most of the features I wanted I gave it go; see change here on github.

Both these changes were quite simple and testing them by zipping the folder and uploading via the portal was quite easy. It would be nice to be able to push my changes via Git (as all the cool kids like to do) but for now it would suffice.

Now all I had to do was re-edit my posts and apply the appropriate markup and I was done.

Using GMock with Visual Studio CppUnitTestFramework

Fri, 03 Apr 2015 00:00:00 GMT

One of the things I have been a bit disappointed with myself during the development of OpenCover is the lack of unit testing around the C++ code that makes up the profiler.

I did toy with GTest and got some decent tests around the instrumentation engine but I was never able to actually test the profiler callbacks, also I found the lack of GTest integration with Visual Studio quite irritating; I know I have been spoilt by ReSharper. Recently however, during handling Fakes through OpenCover, I had an opportunity to work out how to load the profiler using registry free loading and realised that perhaps such testing might be within my reach, what I was missing however was a mocking library and one that I could use with Visual Studio tooling.

Frankly GMock was the only candidate, the commercial alternatives being out as this was for an OSS project, but the instructions all seemed to want to build a number of libraries (64/32 bit Debug/Release) that I would have to statically link to and maintain these builds should the source or build options change. I decided to try a different tack that wouldn't involve building libraries and it has worked out reasonably successful, so I thought it would be worth commenting on here.

Step 1

Get the latest GMock (1.7.0) library as a zip file and uncompress it somewhere within your repository.

Step 2

From within Visual Studio update the Additional Include Directories to include the following paths

$(SolutionDir)lib\gmock-1.7.0
$(SolutionDir)lib\gmock-1.7.0\include
$(SolutionDir)lib\gmock-1.7.0\gtest
$(SolutionDir)lib\gmock-1.7.0\gtest\include

Step 3

Add the following to your "stdafx.h"

#include "gmock/gmock.h"
#include "gtest/gtest.h"

Step 4

Add the following to your "stdafx.cpp"

// The following lines pull in the real gmock *.cc files.
#include "src/gmock-cardinalities.cc"
#include "src/gmock-internal-utils.cc"
#include "src/gmock-matchers.cc"
#include "src/gmock-spec-builders.cc"
#include "src/gmock.cc"

// The following lines pull in the real gtest *.cc files.
#include "src/gtest.cc"
#include "src/gtest-death-test.cc"
#include "src/gtest-filepath.cc"
#include "src/gtest-port.cc"
#include "src/gtest-printers.cc"
#include "src/gtest-test-part.cc"
#include "src/gtest-typed-test.cc"

Step 5

Now all you need to do is add initialise GMock and you are ready; as I am using the CppUnitTestFramework I do the following.

TEST_MODULE_INITIALIZE(ModuleInitialize)
{
// enable google mock
::testing::GTEST_FLAG(throw_on_failure) = true;
int argc = 0;
TCHAR **argv = NULL;
::testing::InitGoogleMock(&amp;argc, argv);
}

Now all you need to do is follow the GMock documentation and add some expectations etc you can as I discovered even mock COM objects and have expectations on them e.g.

EXPECT_CALL(*profilerInfo, SetEventMask(EVENT_MASK_WHEN_FAKES))
.Times(1)
.WillRepeatedly(Return(S_OK));

Bonus Round

There were a few little niggles however the first of which is that if an expectation fails, the Visual Studio test runner takes a little too long to close down (I suspect this may be something on my machine related to DrWatson). The second was that if an expectation did fail I could only initially see the result using DebugView - ugh - however I found a solution at http://www.durwella.com/post/96457792632/extending-microsoft-cppunittestframework which involves using some extra macros; I added these to my "stdafx.h" and voila the results are now available in Visual Studio. Finally, I found the mocks were not very lightweight and in fact if I left them hooked in caused performance issues, replacing them with an admittedly less useful stubs I could avoid this when necessary.

Update 20/2/2016 The link to www.durwella.com has stopped working, but a copy of the article can be found on tumblr - http://durwella.tumblr.com/post/96457792632/extending-microsoft-cppunittestframework#96457792632. For completeness however I am posting the macros here as well with all attribution belonging to durwella.com

#define _TEST_METHOD_EX_EXPANDER(_testMethod)\
	_testMethod { try

// Adds support for seeing std::exception in test output. Requires TEST_METHOD_EX_END after test.
// Example:
// TEST_METHOD_EX_BEGIN(MyFailingTest){ throw std::exception("What happened"); } TEST_METHOD_EX_END;
#define TEST_METHOD_EX_BEGIN(_methodName) _TEST_METHOD_EX_EXPANDER(TEST_METHOD(_methodName))

// Use following test declared with TEST_METHOD_EX_BEGIN
#define TEST_METHOD_EX_END\
	catch (::std::exception& ex) \
	{ \
::std::wstringstream ws; ws << "Unhandled Exception:" << ::std::endl << ex.what(); \
::Microsoft::VisualStudio::CppUnitTestFramework::Assert::Fail(ws.str().c_str());\
	} \
}

Happy Birthday OpenCover

Sun, 22 Feb 2015 00:00:00 GMT

Happy Birthday

Today OpenCover is 4 (four) years old, where has the time gone? In that time it has had over 60,000 nuget downloads, been adopted by the SharpDevelop community as the coverage tool for their IDE, and, as I found out the other day, is also being used by the corefx team to supply coverage information on their tests.

Four years ago I started on OpenCover (first commit - not very interesting but a stake in the ground) in order to create a code coverage tool for the .NET platform that could be used by anyone, but especially so that those of us in the open source community could have a tool available to us to help enhance our testing feedback; in the past we have seen some tools go commercial, some just vanish and others just abandoned. I also wanted to share some of the knowledge I had picked up in this area but no longer used in my day-to-day activities and to ensure it remains within the community by making it maintainable and available without restriction.

It took nearly 6 months to get the first beta release and since that time we have added sequence and branch coverage, support for .NET 2 and .NET 4+, 32 and 64 bit support, and even Silverlight. Later features such as coverage by test and hooking into services and IIS support; not everything works as seamlessly as I would like but the community has either lived with it or improved it - which was the outcome I was seeking. Just recently we even added support for Microsoft.Fakes because some people wanted to use OpenCover for coverage with their tests that used Fakes rather than the coverage tool that they already had available; that was an interesting learning exercise, as well due to some very fortuitous googling.

There even seems to be some movement to make a Mono version of OpenCover which was not something I saw coming but is also quite exciting, especially as Visual Studio now has support for Android and iPhone development, we knew about Xamarin/Mono but actual Visual Studio integration? Who 4 years ago would have seen that one coming ...?

Highlights

One of the highlights of the past few years was starting at my current place of work (MYOB) and then overhearing a conversation within the devops/build team who were discussing the coverage results of this free coverage tool they had found on github, imagine my delight when I realised it was OpenCover they were discussing, and, in mostly favourable terms; this was the first place I had seen OpenCover being used and it wasn't even introduced by me. I secretly implemented a feature in response to their comments.

Another highlight is seeing that at least two Visual Studio integrations involving OpenCover are currently in play, both of these have been started independently, and though I am currently partly involved with one of them it will be interesting to see how they both progress.

I'd like to thank everyone who has contributed to OpenCover either through direct contribution, suggestions, free stuff (more please) and just using it. Here's to another 4+ interesting years and I wonder what will happen to OpenCover in that time - suggestions?

Microservices... Where to Start?

Wed, 29 Oct 2014 00:00:00 GMT

Micro-services are becoming a "thing" now and are probably de-facto when someone begins a new project and are thinking about hosting in the cloud but where do you start when you have a brown field project. Now I don't have any hot answers or amazing insights here all I can do is describe what my first "micro-service" was and how it came into being.

Over time the application was getting more use and the number of servers involved started to increase; we were using auto-scaling and the number of servers increased in line with usage but wavered between 8 and 24 instances. This quite rightly caused some consternation so we tinkered with number of core settings for each instance and thresholds for triggers to scale up and down but nothing seemed to alter the number of total cores being used. We actually have a hefty bit of logging and we can control the output through logging levels so we decided to change the logging to try and get more diagnostic information and this is when things got interesting. As this is a production system getting hold of this log information was initially problematic and slow so we had already started forwarding all the messages to SplunkStorm using the available API and all was well (for over a year) and we were very impressed with how we could use that information for ad-hoc queries. However when we changed the logging levels the servers started scaling and we started to get database errors; unusual ones involving SQL connection issues rather than SQL query errors. We quickly reverted the changes and decided to try and replicate the problem in our CI/SIT environments.

What we realized was that it was our own logging that was causing our performance issues and even more awkwardly was also responsible for the SQL connection issues as the logging to SplunkStorm via its API was using up the available TCPIP connections; this was even more pronounced when we changed the logging level. What we needed to do was refactor our logging such that we could get all our data into SplunkStorm (and Splunk as we were also in the process of migrating to SplunkStorm's big brother) with minimum impact to the actual production systems. Thankfully our logging framework used NLog, which we had wrapped in another entity for mocking purposes, so what we decided to do was write a new NLog target that would instead log to a queue (service-bus) and then have another service read messages from that queue and forward them to Splunk and SplunkStorm and thus our first micro-service was born.

The new NLog target took the log messages, batch pushed them to the queue, then a microservice was written that monitors the queue, pulls messages off in batches, and then pushes them to Splunk and SplunkStorm, also in batches. The initial feasibility spike took 1/2 a day with the the final implementation being ready and pushed into production the following week. Because we were using .NET we could also take advantage of multiple threads so we used thead-pools to limit the number of active Splunk/SplunkStorm messages being sent in parallel. What we found after deployment was that we could scale back our main application servers to 4 instances with only a pair of single core services dealing with the logging aspect, we also noticed that the auto scaling never reaches its old thresholds and the instance count has been stable ever since. Another advantage is that the queue can now be used by other services to push messages to Splunk and can even use the same NLog target in their projects to deal with all the complexities.

I hope the above shows that your first micro-service does not have to be something elaborate but instead deal with a mundane but quite essential task and the benefits can be quite astounding.

Excluding code from coverage...

Mon, 13 Oct 2014 00:00:00 GMT

This may (no guarantees) turn into a series of posts on how to refactor your code for testing using simple examples.

This particular example came from a request to add an "Exclude Lines from Coverage" feature to OpenCover. Now there are many ways this could be achieved, none of which I had any appetite for as they were either too clunky and/or could make OpenCover very slow. I am also not a big fan on excluding anything from code coverage; though OpenCover has several exclude options I just thought that this was one step too far in order to achieve that 100% coverage value as it could too easily abused. Even if I did think the feature was useful it still may not get implemented by myself for several days, weeks or months.

But sometimes there are other ways to cover your code without a big refactoring and mocking exercise which can act as a deterrent to doing the right thing.

In this case the user was using EntityFramework and wanted to exclude the code in the catch handlers because they couldn't force EntityFramework to crash on demand - this is quite a common problem in my experience. The user also knew that one approach was to push all that EntityFramework stuff out to another class and could then test their exception handling via mocks but didn't have the time/appetite to go down that path and thus wanted to exclude that code.

I imagined that the user has code that looked something like this:

public void SaveCustomers(ILogger logger)
{
CustomersEntities ctx = CustomersEntities.Context;//)
try
{
    // awsome stuff with EntityFramework
    ctx.SaveChanges();
}
catch(Exception ex)
{
    // do some awesome logging
    logger.Write(ex);
    throw;
}
}

and I could see why this would be hard (but not impossible) to test the exception handling. Now instead of extracting out all the interactions with the EntityFramework so it is possible to throw an exception during testing I suggested the following refactoring:

internal void CallWrapper(Action doSomething, ILogger logger)
{
try
{
    doSomething();
}
catch(Exception ex)
{
    // do some awesome logging
    logger.Write(ex);
    throw;
}
}

which I would then use like this:

public void SaveCustomers(ILogger logger)
{
CustomersEntities ctx = CustomersEntities.Context;//)
CallWrapper(() => {
    // awsome stuff with EntityFramework
    ctx.SaveChanges();
}, logger);
}

My original tests should still continue as before and now I have a new method that I can now test independently.

I know this isn't the only way to tackle this sort of problem and I'd love to hear about other approaches.

A simple TDD example

Mon, 06 Oct 2014 00:00:00 GMT

I recently posted a response to StackOverflow wrt TDD and Coverage and I thought it would be worth re-posting the response here. The example is simple but hopefully shows how writing the right tests using TDD gives you a better suite of tests for your code than you would probably write if you wrote the tests after the code (which may have been re-factored as you developed).

"As the [original] accepted answer has pointed out your actual scenario reduces to collection.Sum() however you will not be able to get away with this every time.

If we use TDD to develop this (overkill I agree but easy to explain) we would [possibly] do the following (I am also using NUnit in this example out of preference).

[Test]
public void Sum_Is_Zero_When_No_Entries()
{
var bomManager = new BomManager();
Assert.AreEqual(0, bomManager.MethodToTest(new Collection&lt;int&gt;()));
}

and then write the following code (note: we write the minimum to meet the current set of tests)

public int MethodToTest(Collection&lt;int&gt; collection)
{
var sum = 0;
return sum;
}

We would then write a new test e.g.

[Test]
[TestCase(new[] { 0 }, 0)]
public void Sum_Is_Calculated_Correctly_When_Entries_Supplied(int[] data, int expected)
{
var bomManager = new BomManager();
Assert.AreEqual(expected, bomManager.MethodToTest(new Collection&lt;int&gt;(data)));
}

If we ran our tests they would all pass (green) so we need a new test(cases)

[TestCase(new[] { 1 }, 1)]
[TestCase(new[] { 1, 2, 3 }, 6)]

In order to satisfy those tests I would need to modify my code e.g.

public int MethodToTest(Collection&lt;int&gt; collection)
{
var sum = 0;
foreach (var value in collection)
{
    sum += value;
}
return sum;
}

Now all my tests work and if I run that through OpenCover I get 100% sequence and branch coverage - Hurrah!.... And I did so without using coverage as my control but writing the right tests to support my code.

BUT there is a 'possible' defect... what if I pass in null? Time for a new test to investigate

[Test]
public void Sum_Is_Zero_When_Null_Collection()
{
var bomManager = new BomManager();
Assert.AreEqual(0, bomManager.MethodToTest(null));
}

The test fails so we need to update our code e.g.

public int MethodToTest(Collection&lt;int&gt; collection)
{
var sum = 0;
if (collection != null)
{
    foreach (var value in collection)
    {
        sum += value;
    }
}
return sum;
}

Now we have tests that support our code rather than tests that test our code i.e. our tests do not care about how we went about writing our code.

Now we have a good set of tests so we can now safely refactor our code e.g.

public int MethodToTest(IEnumerable&lt;int&gt; collection)
{
return (collection ?? new int[0]).Sum();
}

And I did so without affecting any of the existing tests."

The API Journey for AccountRight Live

Tue, 29 Jul 2014 00:00:00 GMT

My talk at ALT.NET on designing the MYOB API.

"Building an API for your product isn’t just about choosing your technology and planning your scaling capabilities when you unleash it upon the world. For nearly 2 years MYOB have been developing an API for our AccountRight Live product and we would like to share with you our journey into making an API that is used by our own products, such as PayDirect, and one that our developer partners can also use productively - “Integrating with the MYOBapi has been so quick for us. For example [when] MYOBapi let us know when the new endpoints were available for payroll we were able to complete our full depth integration over the space of 16 hours."

The full talk can be found here on YouTube.

Customsing New Relic installation during Azure deployments

Thu, 03 Apr 2014 00:00:00 GMT

For about a year we've been running New Relic to monitor our WebRoles running on the Azure platform. Installing has been quite simple by following the instructions initially found on the New Relic site and is now available via Nuget; however two things about this process have been irking me.

First, I wanted to be able to distinguish the CI and Production deployments in the New Relic portal by making them have different names, but the name as it appears in the New relic portal is controlled through a setting in the web.config and cannot be controlled though the Azure portal.

Second, I wanted to be able to control the licence key we used for CI (free licence, limited functionality) and Production (expensive licence, full functionality) deployments, however the key is embedded in the newrelic.cmd and is applied when the New Relic agent is installed; this is not easy to change during/post deployment.

The initial solution to both these problems involved producing two packages, one for the CI environment(s) and one for the Production environment. Instead of the normal Debug and Release build outputs, a 3rd target, Production, was used and the web.config was modified during the build process using a transform that changed the name to what was wanted. The licence key issue was resolved by have two newrelic.cmd items in the project and then packaging the required one with the appropriate build. This was not ideal but it worked in a fashion however the ProdOps guys were keen on having control over the name and licence key used in production.

Changing the Application name

New Relic gets the Application name from a setting in the web.config and so what is necessary is to read a setting in the Azure configuration and update the web.config. There are many ways to resolve this issue but the approach we took was based on the solution to an identical issue raised on GitHub. For completeness I will however reiterate the steps below:

In the ServiceDefinition.csdef file add a setting to the section

<ConfigurationSettings>
  <Setting name="NewRelicApplicationName" />
</ConfigurationSettings>

In the ServiceConfiguration file for your environment add a setting that will be used to set the Application name in New Relic

<ConfigurationSettings>
  <Setting name="NewRelicApplicationName" value="MyApplication" />
</ConfigurationSettings>

In the WebRole.cs file for your application amend your code with the following

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        ConfigureNewRelic();
        return base.OnStart();
    }

    private static void ConfigureNewRelic()
    {
        if (RoleEnvironment.IsAvailable && !RoleEnvironment.IsEmulated)
        {
            string appName;
            try
            {
                appName = RoleEnvironment.GetConfigurationSettingValue("NewRelicApplicationName");
            }
            catch (RoleEnvironmentException)
            {
                /*nothing we can do so just return*/
                return;
            }

            if (string.IsNullOrWhiteSpace(appName))
                return;

            using (var server = new ServerManager())
            {
                // get the site's web configuration
                const string siteNameFromServiceModel = "Web";
                var siteName = string.Format("{0}_{1}", RoleEnvironment.CurrentRoleInstance.Id, siteNameFromServiceModel);
                var siteConfig = server.Sites[siteName].GetWebConfiguration();

                // get the appSettings section
                var appSettings = siteConfig.GetSection("appSettings").GetCollection();
                AddConfigElement(appSettings, "NewRelic.AppName", appName);
                server.CommitChanges();
            }
        }
    }

    private static void AddConfigElement(ConfigurationElementCollection appSettings, string key, string value)
    {
        if (appSettings.Any(t => t.GetAttributeValue("key").ToString() == key))
        {
            appSettings.Remove(appSettings.First(t => t.GetAttributeValue("key").ToString() == key));
        }

        ConfigurationElement addElement = appSettings.CreateElement("add");
        addElement["key"] = key;
        addElement["value"] = value;
        appSettings.Add(addElement);
    }
}

And that should be it.

Changing the New Relic licence key

The New Relic licence key is applied when the New Relic agent is installed on the host so what we is needed is to read the Azure configuration when the newrelic.bat is executed as part of the Startup tasks (defined in the ServiceDefinition.csdef) and apply it when the agent is installed. There does not appear to be way of changing the licence key if your agents have already been installed other than reducing the number of instances to 0 and then scaling back up (I suggest you use the staging slot for this). In the ServiceDefinition.csdef file add a setting to the section

<ConfigurationSettings>
   <Setting name="NewRelicLicenceKey"></Setting>
</ConfigurationSettings>

and add a new Environment variable to the newrelic.cmd startup task that will be set by the new configuration setting

<task commandline="newrelic.cmd" executioncontext="elevated" tasktype="simple">
  <environment>
<variable name="EMULATED">
  <roleinstancevalue xpath="/RoleEnvironment/Deployment/@emulated"></roleinstancevalue>
</variable>
<variable name="NewRelicLicence">
  <!-- http://msdn.microsoft.com/en-us/library/windowsazure/hh404006.aspx -->
  <roleinstancevalue xpath="/RoleEnvironment/CurrentInstance/ConfigurationSettings/ConfigurationSetting[@name=&apos;NewRelicLicenceKey&apos;]/@value">
  </roleinstancevalue>
</variable>
<variable name="IsWorkerRole" value="false"></variable>
  </environment>
</task>

In the ServiceConfiguration file for your environment add a setting that will be used to set the Application name in New Relic

<configurationsettings>
  <setting name="NewRelicLicenceKey" value="<ADD YOUR KEY HERE>"></setting>
</configurationsettings>

Edit your newrelic.cmd to use the Environment variable

:: Update with your license key
SET LICENSE_KEY=%NewRelicLicenceKey%

Now you should be able to control the New Relic licence key during your deployment.

Book Review - Building Mobile Applications Using Kendo UI Mobile and ASP.NET Web API

Sat, 14 Dec 2013 00:00:00 GMT

I've written a book review on 'Building Mobile Applications Using Kendo UI Mobile and ASP.NET Web API' and posted it up on CodeProject. Summary I liked this book and I took a lot from it that I am now using to build that sample application using KendoUI. If you want to learn about ASP.NET Web API then this book isn't for you and you'll learn a lot more from the ASP.NET Web API site.

Getting code coverage from your .NET testing using OpenCover.

Sun, 03 Nov 2013 00:00:00 GMT

Introduction

OpenCover is a free, open-sourced, code coverage tool for .NET 2.0 and above running on the .NET platform. It supports sequence coverage, branch coverage and has a cover by test facility. Though OpenCover is command line only, a rich HTML UI of the results can be visualized using ReportGenerator.

We will aim to demonstrate how you can use this utility to get visibility into your testing coverage.

Background

OpenCover is currently the only actively developed and maintained open-sourced tool of it's type for the .NET platform but it was not the first, some of the others being:

NCover - Probably the most well-known commercial tool for .NET code coverage started life as an open-source project.

Coverage.eye - Originated at Microsoft and was available on gotdotnet, the repository/sample has since gone and no full mirrors appear to exist (The wayback machine only has the text).

PartCover - This appears to have been created in response to NCover going commercial and though actively used it has limitations e.g. 32-bit only. The original repository is no longer being maintained by its original developers and is now being maintained on github.

Using OpenCover

Preparing for OpenCover

The following steps detail how to download OpenCover (and other supporting tools) from NuGet.

OpenCover is available as a zip or msi download via its bitbucket mirror but for the sake of this article we will use the OpenCover nuget package.

Create a new solution
For that solution "Enable NuGet Package Restore"
Using "Manage NuGet Packages for Solution" add the following packages

OpenCover
ReportGenerator
NUnit.Runners

Once completed you should have a solution with a .nuget folder which contains a packages.config that looks something like the following

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="NUnit.Runners" version="2.6.3" />
  <package id="OpenCover" version="4.5.1923" />
  <package id="ReportGenerator" version="1.9.1.0" />
</packages>

Create a C# project (class library) called Sample
Create a C# project (class library) called Sample.Test and reference the Sample project
Use NuGet to add the following package

NUnit

Now to add a simple class and method and a test that will exercise that method.

In the sample project add the following class

public class Target
{
public static void DoSomething()
{
    try
    {
        Console.WriteLine("I ran!");
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}
}

In the Sample.Test project add the following class

using NUnit.Framework;
[TestFixture]
public class TargetTest
{
[Test]
public void DoSomethingTest()
{
    Target.DoSomething();
}
}

Running tests with OpenCover

OpenCover does not directly execute your tests but instead needs to execute another application that executes your tests in this case we are using NUnit.

First lets create a batch file that we can execute our tests from the command line

......\packages\NUnit.Runners.2.6.3\tools\nunit-console.exe sample.test.dll /noshadow

This batch file can be added to the Sample.Test project (with Copy to Output Directory set to Copy Always).

Now we can use OpenCover to execute this batch file, again we can add this as a a batch file to our Sample.Test project so that we can execute it on the command line

......\packages\OpenCover.4.5.1923\OpenCover.Console.exe -target:runtests.bat -register:user -filter:+[Sample]*

The arguments are:

-target: - used to indicate the target process for OpenCover to execute.

-register:user - used to register the COM objects that OpenCover uses to instrument your assemblies.

-filter: - used to control which assemblies OpenCover will gather coverage data for. The filter is capable of including and excluding assemblies and classes and is actually the same filter format that PartCover uses. The filter is one of the more complex features of OpenCover and more detail is provided with the documentation that is installed alongside OpenCover or in the Usage wiki.

When executed, OpenCover will produce an XML file (default results.xml) that contains all the data related to that test run. To visualize that data we can use ReportGenerator to produce some rich HTML output. ReportGenrator can also be run on the command line

......\packages\ReportGenerator.1.9.1.0\reportgenerator.exe -reports:results.xml -targetdir:coverage

If we open the produced coverage output (coverage\index.htm) we can see the visualization of the coverage of our target code. As you can see it is quite simple to understand what code your tests actually cover.

For a real-world example OpenCover is actually used to gather coverage on it's own tests and the results can be seen on the OpenCover build pipeline.

NOTE: This article was originally published on CodeProject Nov 3rd 2013

Application Tracing

Sun, 15 Sep 2013 00:00:00 GMT

So OpenCover is as feature complete as I care to take it at the moment, I may do this one feature involving Windows Store applications should I have a need for it, and I decided to not continue with OpenMutate as I can't really find a need for it other than an exploratory investigation into reJIT.

I do have one more itch to scratch when it comes to profilers and that is application tracing and this may allow me to play with other technologies which I'll list later. This itch started a few months back, perhaps 6+ months ago, when I was trying to integrate some commercial tracing applications to an application I was working on and they both died horribly and I started to look for alternatives and found nothing available. Now I could have started the project then but I decided to pester both vendors until they fixed the problem which they eventually did (within a week or two of each other or so it seemed to me) and I integrated one of the solutions and moved on... but the itch never went away.

So what am I thinking... well a profiler (obviously) with 32/64 support (again given) and making obscene abuse of the COR_PRF_MONITOR_ENTERLEAVE functionality. The problem here is I don't really know what people will want to track (hey that is why there are companies that do this thing with BAs and such like to decide on this) so in the first instance I'll go at tracing everything (which will probably be very slow) and go from there.

This leads to the next problem, data, lots of it, lots and lots of it, and that data is going to need a home but a home I can then use to create reports at some point. For this I am thinking asynchronous, initially a queue and potentially an event source like data store like EventStore or NEventStore. The advantage of an event source would allow the ability to regenerate the views once we know what they are, perhaps something along the lines of Splunk or SplunkStorm would come into play.

So a name... always the hardest part but thankfully we have the internet and online dictionaries so I've gone with OpenDiscover.

Creating a simple NodeJS app with Mongo

Sat, 09 Mar 2013 00:00:00 GMT

Okay, I woke up this morning (6am) with a need to create a simple reporting dashboard to display the coverage results from OpenCover when it dog-foods its own tests. Now that OpenCover has no **_reported _**bugs, I decided to use my spare time to investigate other technologies for a while.

What I needed was simple 'online' storage to capture results from the build system and the ability to extract that data into charts. Normally I'd probably knock up a simple rails app because it is easy to do, however I decided, probably due to the heat, to use the following:

node.js - a technology I haven't used but have meant to for a while; I also like the JavaScript syntax better than ruby (it's a personal thing)
mongodb - a database I am familiar with
google charts - free; as in beer.
heroku - free; well my intended usage will be.

A quick time-boxed search of the web about how to use node with mongodb and create a RESTful API and I settled on the following packages:

mongoose - for interacting with the mongo database
restify - for creating a simple rest server
nodemon - monitors changes in your app and restarts; sounds useful

I'll assume other packages will be added to the mix as challenges present themselves. It's now 7am and time for breakfast and then the fun starts... And a few hours later we have a simple storage system hosted on heroku all we need now is the chartsThe repository can be found on github. I am sure it will evolve over time but it was very simple to get to this stage by leveraging the work of all those who have gone before.

MongoDB, Mongoid, MapReduce and Embedded Documents.

Sat, 25 Aug 2012 00:00:00 GMT

I am using Mongoid to store some data as documents in a MongoDB database and then run some MapReduce queries against the data. Now I have no trouble with mapping data from normal documents and an embedded document but I could not extract data from an embedded collection of documents i.e.

class Foo
  include Mongoid::Document

  #fields
  field :custom_id, :type =&gt; String

  #relations
  embeds_many :bars

end

``````javascript
class Bar
  include Mongoid::Document

  #fields
  field :custom_field, :type =&gt; String

  #relations
  embedded_in :Foo

end

First it looks like that we need to run the map part of the MapReduce against the parent document and not the child i.e. Foo.map_reduce(...) will work find documents but Bar.map_reduce(...) does not, however that is not surprising as it is also not possible to count all Bar documents by doing Bar.all.count in the rails console.

Now a MapReduce query in MongoDB is done as a pair of JavaScript scripts, the first does the map by _emit_ting a mini-document of data and the second that aggregates the data in some manner. So thinking I had a collection (array) my first attempt to map data from the embedded document was this:

MAP:

function() {
  if (this.bars == null) return;
  for (var bar in this.bars){
emit(bar.custom_field, { count: 1 });
  }
}

REDUCE:

function(key, values) {
  var total = 0;
  for ( var i=0; i&lt; values.length; i++ ) {
total += values[i].count;
  }
  return { count: total };
}

This produced an unusual result such that there was only a single aggregated document with a null key and the count was the total number of child documents (summed across all the parents).

Now I could have just broken the child document out and not embedded it but I didn't want to break the model over something so trivial that must, in my eyes, be possible.

After much googling and reading of forum posts, I couldn't find any samples. I eventually observed of some 'unusual' syntax on an unrelated topic which led me to rewrite the map script into this:

function() {
  if (this.bars== null) return;
  for (var bar in this.bars){
emit(this.bars[bar].custom_field, { count: 1 });
  }
}

Which produced the expected results. Okay this was probably obvious to anyone who knows MongoDB+MapReduce well but it took me a while to find out and it still isn't that intuitive, though I think I now know why it is this way, so I thought I'd write it up as a bit of a reference.

The "Pigs" and "Chickens" fable

Fri, 08 Jun 2012 00:00:00 GMT

I think anyone who is anyone who has heard of Agile and Scrum have heard of the Pigs and Chickens story and how it describes those who are committed to the delivery of the project, as "Pigs", and those who are just involved, as "Chickens"; if not click on the image below and learn more about it.

However I was just recently re-reading "Death March" by Edward Yourdon (1st Edition) and I came across this response to the parable, in the context of commitment whilst on a death march.

“I’m not sure you will find any old pigs in development perhaps more chickens. I think that kind of commitment continues until (inevitably?) you get into the first death march project – then there is a rude awakening. Either the pig realises what’s happening, this is the slaughterhouse! RUN!! Or the pig is making bacon…”

- Paul Mason (Death March).

I just found it quite amusing and thought I should share...

Mutation Testing; a use for re-JIT?

Fri, 03 Feb 2012 00:00:00 GMT

Where to start... Mutation testing is described as modifying a program in small amounts and then executing the original 'passing' tests that exercise that code and then watching them fail. It is a way of making sure your tests are actually testing what you believe they are testing.

Setting the stage... So how can we do this with .NET? Well first we need to know what tests execute what code and we can use OpenCover for that when it is using it's tracking by test feature. With that feature we can see which tests execute which sequence points and also see what branches were exercised, it is this later information we can take advantage of when creating a mutation testing utility.

New toys to play with... Now this mutation tester is going to be working at the IL level and as such we could use the JIT (Just-in-time) compilation feature that is used with OpenCover (and PartCover). However that would mean a complicated instrumentation that we would then have to control which path we would want to exercise, or we could have simpler instrumentation but that would require the process under test (e.g. nunit, mstest, ...) to be stop and started each time to allow new code to be exercised. With .NET 4.5 (in preview at the time of writing) there is a re-JIT compilation feature that we could use instead and this would allow us to use simple instrumentation without needing to stop and start the process under test. There are a number of limitations of re-JIT but after reviewing them (several times) I don't think any are actually show stoppers.

However to make the Re-JIT useful we need a way of executing a test or tests repeatedly without having to restart the application under test and this isn't possible with nunit and mstest. However it should be possible to use the test runners from AutoTest.Net if we host them directly or in a separate process that can be communicated with.

A plan... So the flow will be something like this (I wonder how will this will stand up to the test of time) I haven't looked at the latest profiler API in-depth but documentation on MSDN) and David Broman's Blog seem to indicate this should be possible.

Run OpenCover to produce an XML file with a list of what tests exercised what branches
For each branch point =>All it needs is a name...All of this will be hosted on GitHub under OpenMutate. Let the games begin....

Unusual coverage in VB.NET

Sat, 21 Jan 2012 00:00:00 GMT

Recently a user posted on StackOverflow on why he was seeing unusual coverage results in VB.NET with MSTEST and Visual Studio. The the question already had answers that helped the questioner but I decided to delve a little deeper and find out why the solution proposed worked.

The issue was that in his code sample the End Try was not being shown as covered even though he had exercised the Try and the Catch parts of his code.

First I broke his sample down into something simpler and I have highlighted the offending line.

Function Method() As String
  Try
Return ""
  Catch ex As Exception
Return ""
  End Try
End Function

In debug we can extract the following sequence points (I am, obviously, using OpenCover for this.)

<SequencePoints>
  <SequencePoint
    offset="0"
    ordinal="0"
    uspid="261"
    vc="0"
    ec="32"
    el="7"
    sc="5"
    sl="7"
  />
  <SequencePoint
    offset="1"
    ordinal="1"
    uspid="262"
    vc="0"
    ec="12"
    el="8"
    sc="9"
    sl="8"
  />
  <SequencePoint
    offset="2"
    ordinal="2"
    uspid="263"
    vc="0"
    ec="22"
    el="9"
    sc="13"
    sl="9"
  />
  <SequencePoint
    offset="19"
    ordinal="3"
    uspid="264"
    vc="0"
    ec="30"
    el="10"
    sc="9"
    sl="10"
  />
  <SequencePoint
    offset="20"
    ordinal="4"
    uspid="265"
    vc="0"
    ec="22"
    el="11"
    sc="13"
    sl="11"
  />
  <SequencePoint
    offset="40"
    ordinal="5"
    uspid="266"
    vc="0"
    ec="16"
    el="12"
    sc="9"
    sl="12"
  />
  <SequencePoint
    offset="41"
    ordinal="6"
    uspid="267"
    vc="0"
    ec="17"
    el="13"
    sc="5"
    sl="13"
  />
</SequencePoints>

(where sl = start line, el = end line, sc = start column, ec = end column and offset = IL offset in decimal)

However these only make sense when you look at the IL...

.method public static
string Method () cil managed
{
// Method begins at RVA 0x272c
// Code size 43 (0x2b)
.maxstack 2
.locals init (
    [0] string Method,
    [1] class [mscorlib]System.Exception ex
)

IL_0000: nop
IL_0001: nop
.try
{
    IL_0002: ldstr ""
    IL_0007: stloc.0
    IL_0008: leave.s IL_0029

    IL_000a: leave.s IL_0028
} // end .try
catch [mscorlib]System.Exception
{
    IL_000c: dup
    IL_000d: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
    IL_0012: stloc.1
    IL_0013: nop
    IL_0014: ldstr ""
    IL_0019: stloc.0
    IL_001a: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
    IL_001f: leave.s IL_0029

    IL_0021: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
    IL_0026: leave.s IL_0028
} // end handler

IL_0028: nop

IL_0029: ldloc.0
IL_002a: ret
} // end of method Module1::Method

Now as you can see the End Try line that is causing concern would only be marked as hit (assuming they are using similar instrumentation to OpenCover) if the code reached IL instruction at offset 40 (IL_0028) however when one looks at the IL produced it is not possible to see how you would ever reach that instruction due to the odd IL produced (leave.s is a small jump like instruction that is used to exit try/catch/finally blocks) and if you follow the code you see that you will always reach a leave.s that jumps to IL_0029 first.

In release the IL changes to something more like what I was expecting beforehand and it has no unusual extra IL...

.method public static
string Method () cil managed
{
// Method begins at RVA 0x2274
// Code size 30 (0x1e)
.maxstack 2
.locals init (
    [0] string Method,
    [1] class [mscorlib]System.Exception ex
)

.try
{
    IL_0000: ldstr ""
    IL_0005: stloc.0
    IL_0006: leave.s IL_001c
} // end .try
catch [mscorlib]System.Exception
{
    IL_0008: dup
    IL_0009: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
    IL_000e: stloc.1
    IL_000f: ldstr ""
    IL_0014: stloc.0
    IL_0015: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
    IL_001a: leave.s IL_001c
} // end handler

IL_001c: ldloc.0
IL_001d: ret
} // end of method Module1::Method

but so do the sequence points...

<SequencePoints>
  <SequencePoint
    offset="0"
    ordinal="0"
    uspid="33"
    vc="0"
    ec="22"
    el="9"
    sc="13"
    sl="9"
  />
  <SequencePoint
    offset="15"
    ordinal="1"
    uspid="34"
    vc="0"
    ec="22"
    el="11"
    sc="13"
    sl="11"
  />
  <SequencePoint
    offset="28"
    ordinal="2"
    uspid="35"
    vc="0"
    ec="17"
    el="13"
    sc="5"
    sl="13"
  />
</SequencePoints>

So now one will never see your try/catch lines marked covered, so this is not helpful.

So lets try changing your code as suggested and go back to debug (because that is where you will be running coverage from usually.)

Function Method2() As String
Dim x As String
Try
    x = ""
Catch ex As Exception
    x = ""
End Try
Return x
End Function

Again we look at the sequence points...

<SequencePoints>
  <SequencePoint
    offset="0"
    ordinal="0"
    uspid="268"
    vc="0"
    ec="33"
    el="15"
    sc="5"
    sl="15"
  />
  <SequencePoint
    offset="1"
    ordinal="1"
    uspid="269"
    vc="0"
    ec="12"
    el="17"
    sc="9"
    sl="17"
  />
  <SequencePoint
    offset="2"
    ordinal="2"
    uspid="270"
    vc="0"
    ec="19"
    el="18"
    sc="13"
    sl="18"
  />
  <SequencePoint
    offset="17"
    ordinal="3"
    uspid="271"
    vc="0"
    ec="30"
    el="19"
    sc="9"
    sl="19"
  />
  <SequencePoint
    offset="18"
    ordinal="4"
    uspid="272"
    vc="0"
    ec="19"
    el="20"
    sc="13"
    sl="20"
  />
  <SequencePoint
    offset="31"
    ordinal="5"
    uspid="273"
    vc="0"
    ec="16"
    el="21"
    sc="9"
    sl="21"
  />
  <SequencePoint
    offset="32"
    ordinal="6"
    uspid="274"
    vc="0"
    ec="17"
    el="22"
    sc="9"
    sl="22"
  />
  <SequencePoint
    offset="36"
    ordinal="7"
    uspid="275"
    vc="0"
    ec="17"
    el="23"
    sc="5"
    sl="23"
  />
</SequencePoints>

and the IL...

.method public static
string Method2 () cil managed
{
// Method begins at RVA 0x282c
// Code size 38 (0x26)
.maxstack 2
.locals init (
    [0] string Method2,
    [1] string x,
    [2] class [mscorlib]System.Exception ex
)

IL_0000: nop
IL_0001: nop
.try
{
    IL_0002: ldstr ""
    IL_0007: stloc.1
    IL_0008: leave.s IL_001f
} // end .try
catch [mscorlib]System.Exception
{
    IL_000a: dup
    IL_000b: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::SetProjectError(class [mscorlib]System.Exception)
    IL_0010: stloc.2
    IL_0011: nop
    IL_0012: ldstr ""
    IL_0017: stloc.1
    IL_0018: call void [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.ProjectData::ClearProjectError()
    IL_001d: leave.s IL_001f
} // end handler

IL_001f: nop
IL_0020: ldloc.1
IL_0021: stloc.0
IL_0022: br.s IL_0024

IL_0024: ldloc.0
IL_0025: ret
} // end of method Module1::Method2

So for the End Try to be covered we need line 21 to be hit and that is offset 31 (IL_001F) and as it can be seen both leave.s instructions jump to that point so now that line will be marked as covered.

Adding OpenCover to TeamCity

Sun, 02 Oct 2011 00:00:00 GMT

Adding OpenCover to the latest version of TeamCity (6.5) couldn't be easier however if you need help follow these simple steps.

Download and install OpenCover
Download and install ReportGenerator (actually unzip)
Register the OpenCover profiler DLLs using the regsvr32 utility

regsvr32 /s x86\OpenCover.Profiler.dll regsvr32 /s x64\OpenCover.Profiler.dll

Using TeamCity add a new Build Step to your configuration
Choose Command Line as the runner type then choose Custom Script for the Run option.
Now all is needed is to set up the command to run the profiler against your tests e.g. for OpenCover the working directory is set to** main\bin\debug** and so we have

"%env.ProgramFiles(x86)%\opencover\opencover.console.exe" "-target:......\tools\NUnit-2.5.10.11092\bin\net-2.0\nunit-console-x86.exe" -targetargs:"OpenCover.Test.dll /noshadow" -filter:"+[Open*]* -[OpenCover.T*]*" "-output:......\opencovertests.xml"

"%env.ProgramFiles(x86)%\ReportGenerator\bin\ReportGenerator.exe" ......\opencovertests.xml ......\coverage

Finally setup the artifacts so that you can view the results in TeamCity e.g.

%teamcity.build.workingDir%\opencovertests.xml %teamcity.build.workingDir%\coverage\**\*.*

And there you have it, OpenCover running under TeamCity and visual reports provided by ReportGenerator. I am sure you will find ways to improve upon this for your own builds.

The problem with sequence coverage. (part 2)

Sat, 27 Aug 2011 00:00:00 GMT

Previously I mentioned why just relying on sequence coverage is not a good idea as it is possible to have 100% sequence coverage but not 100% code coverage. However I only described a scenario that used a branch that had 2 paths i.e. the most common form of the conditional branches, but there is one other member of the conditional branch family that exists in IL and that is the switch instruction; this instruction can have many paths. This time I am using the code from the Newtonsoft.Json library because a) it has tests and b) it is very well covered at 83% sequence coverage, but only 72% (by my calculations) branch coverage. The subject of this investigation is BsonReader::ReadType(BsonType) this method has a very large switch statement, that actually is defined as a switch statement in IL, with a default and several fall-throughs; a fall-through is where two or more case statements call the same code. The method itself has 98% sequence coverage and 82% branch coverage; the only code that is uncovered is the handler for the default: path.

which is not unexpected as it is a handler for an Enum which should not be set to any value that is not part of the allowed values. Looking at the branch coverage report we have the following results (the switch instruction we are interested in is at IL offset 8.

Now the first path (0) is unvisited, but we knew that, so the next unvisited branch is #14 and the next is #17; luckily for us the enum in question that is used by the switch instruction is well defined. And as such we can thus deduce that the method is never called during testing with the values Symbol and TimeStamp but the code that they would call is covered; in fact we can see from the code that both these enum values are part of the switch/case statement and are part of fall-throughs. So again we see how branch coverage helps identify 'potential' issues and test candidates.

The problem with sequence coverage.

Thu, 25 Aug 2011 00:00:00 GMT

Sequence coverage is probably the simplest coverage metric, the information is packaged in PDB files and can be read using tools like Mono.Cecil, but just because a method has 100% sequence coverage does not mean you have 100% code coverage.

I'll use an example from OpenCover's own dogfood tests to demonstrate what I mean. Here is a method which shows that it has 100% coverage (sequence point that is).

However I see an issue and that is that on line 101 there is a condition, i.e. a branch, and yet if the visit count is 1 then there is no possibility that both paths for that branch could have been tested. We can therefore infer that even if we had 10000 visits there is no guarantee that every path would be covered even in such a simple method.

Looking at the OpenCover results from which the coverage report was generated. We get

We can see that each sequence point has been visited once, however the branch coverage shows that only one of the paths for the condition we have identified had been visited (in this case it is the true path); which is good as that is what we deduced.So if you are using code coverage tools do NOT just rely on sequence point coverage alone to determine how well covered your code is. Luckily OpenCover now, as of 25th Aug 2011, supports branch coverage and ReportGenerator 1.2 displays most of the information to help you identify possible coverage mismatches.

OpenCover Performance Impact (part 2)

Wed, 10 Aug 2011 00:00:00 GMT

I think I now have a handle on why I was getting the results I earlier reported i.e. OpenCover and PartCover were not some magical performance boosters that added Go Faster stripes to your code.

After a heads up by leppie and his investigations of using OpenCover on his IronScheme project I realised that I needed to spend some time on optimizing how I get data from the profiler and aggregate it into the report. In case you are wondering IronScheme test that took just shy of 1 minute to run on my machine took over 60mins when running under the profiler, Ouch!

The problem

First of all I should explain what sort of data OpenCover gathers (and why) and then I can describe what I did to improve performance. OpenCover records each visit to a sequence point and stores these visits into shared memory; I did it this way as I am hoping to be able to use the order of visits for some form of path coverage analysis at a later date. After 8000 visits it informs the host process that there is a block ready for processing. The host takes this block, makes a copy, releases the shared memory back to the profiler and then processes the data. After processing the data the hosts then waits for the next message. It was this latter stage that was the bottleneck as the host was spending too much time aggregating the data that the profiler was already ready with the next 8000 points.

An (interim) solution

I say interim solution as I am not finished with performance improvements yet but decided that what I had implemented so far was okay for release. First I looked at how the results were being aggregated and noticed that a lot of the time was being spent looking up the sequence points so that the visit count could be updated, I switched this to a list and mapped the visit count data to the model at the end of the profiling run. This helped but only by bringing the profiling run down to ~40mins.

I realised that I just had to get the data out of the way quickly and process it at a later date, so I added a processing thread and a ConcurrentQueue. This was an interesting turn of events as the target process now finished in 4 mins but the host took nearly 40 mins to process the data and the memory usage went up to 2.5GB and a backlog of 40K messages. Hmmm....

After some toying, whilst looking for inspiration, I noticed that the marshaling of the structure (2 integers) was where most of the time was spent. I switched this to using BitConvertor, which also meant that I could avoid the memory pinning required by the marshaling. Now the target process still ran in just under 4 mins but the backlog very rarely reached 20 messages and memory usage stayed at a comfortable level (<100MB) I decided this was enough for now and released a version of the profiler.

But what about the earlier results?

Those earlier results though were still are cause for thought. Why should the OpenCover dogfood tests be faster but the ironscheme test be so much slower. Well the IronScheme tests were doing a lot of loops and were running parts of the code many 1000's of times whereas the dogfood tests were unit tests and the code was only being run several times before moving onto the next test fixture and next section of code. I am now thinking that the issue is due to the optimization that is normally performed by the JIT compiler, but is turned off by the profiler i.e. when running the tests (without profiler) the JIT compiler spends time optimizing the code but the time spent is not recovered as the code is not run enough times to get a net gain, compared to when the JIT compiler just compiles the non-optimised modified code that the profiler produces.

So in conclusion you may see some speed improvements if running tests where your code is only visited a few times but if you are doing intensive execution of code then don't be surprised if the performance is degraded.

OpenCover Performance Impact

Sun, 24 Jul 2011 00:00:00 GMT

So how does OpenCover's profiling impact your testing. The best way is to get some figures so that you can judge for yourself.

I decided to use OpenCover's own tests and use the timing value produced by Nunit itself; just like I'd expect any user who is trying to determine impact I suppose. I've also added the results from PartCover for comparison. Before I took any numbers I (warmed) the code by running the code several times beforehand.

Nunit32

Nunit32 (OpenCover)

Nunit32 (PartCover)

Nunit64

Nunit64 (OpenCover)

2.643

2.691

2.639

4.544

3.807

2.629

2.69

2.611

4.426

3.753

2.642

2.638

2.612

4.46

4.036

Average

2.638

2.673

2.621

4.477

3.865

I don't know how to interpret these results as they don't make much sense, OpenCover seemed to add on average 1.3% to the total time (which I'd expect), whereas PartCover appears to make the code go faster by 0.64%. I can't explain why the results for 64 bit seem to show that OpenCover improves performance by 13.6%.

I tried to come up with a number of reasons for the above but the results I keep getting are reasonably consistent, so I decided to post them anyway and perhaps someone else will be able to tell me what is happening.

Questions about open source and liability in the workplace

Sat, 09 Jul 2011 00:00:00 GMT

Last weekend I attended DDDSydney and one of the most interesting sessions was a panel session about Microsoft and Opensource (Open Source & Microsoft Ecosystem); though as these things go, it went quickly off(ish) topic as expected by the panelists whom I'll refer to as the crazy drupal girl and the 3 stooges (honestly no offence folks, it was highly entertaining).

However it got me thinking about the number of projects where I come have across an unusual bit of open source software that has some use (but has not found a niche or has since been surpassed) and I find that this was introduced by a developer as it was their pet open source project. Now the first question is "what is the liability under this scenario?"

Did the developer ask first as they should before using any open source software on a project? If so then the company accepted the situation but what happens if they did not (or what not made aware) are they still liable or is the developer liable? I assume it would be the company as they should be having some sort of oversight but for small overworked teams where process may not be as strong this may get overlooked.

The other issue is what happens if you introduce your pet open source software project and then you leave, who supports it? How do you separate the open source project needs and the day-job, when they are so intermingled? Does the remaining team support it, do they have the skills? What happens if the parting was acrimonious in nature then they, the team, raised a legitimate issue would you fix it, or leave them to stew?

I don't have answers to the above, I did title this "Questions about...", that can be applied universally the answer to most I suppose is "it depends". Each situation will be different I suspect but I think these type of questions should be asked by any company hoping to use open source software and developers wishing to introduce it, whether that are contributors or not.

Personally I have decided to NOT introduce the open source software I develop into my workplace, yes they could use it and find it useful but they can also afford commercial alternatives. If someone else suggested it, I'd have to make sure there was an agreement should an issue arise that affects them, that if they want it fixed quick then I may have to use 'work' time i.e. no guarantees that it would be done that evening or even that week; after all it is supposed to be fun and not stressful.

How do we get Users out of [open source] Welfare?

Sat, 25 Jun 2011 00:00:00 GMT

How do we get Users out of [open source] Welfare?

Okay an odd title but something I've been thinking about for some time and I suppose is the source of much frustration I have been having whilst maintaining PartCover; I am hoping to reverse the situation with OpenCover.

Categorizing open source users

First I'd like to explain that I like to roughly categorize people involved in open source like thus:

Contributors - these are the guys and gals at the pit-face, developing software, writing documentation and generally striving to make an open source product better.

Investors - these individuals use open source software and help to make the product better via feedback and raising, and following up, issues (probably as it is in their interest to do so).

Benefactors - usually companies that give tools to open source developers or sponsor a project in other ways i.e. free licenses or free hosting e.g. NDepend, JetBrains and GitHub.

Angels - these people provide invaluable advice in just managing a open source project and may not be actively involved in the development itself but just keep you sane.

Community - Our main user base, users of open source but don't actively contribute back and hence why sometimes I refer to them as Welfare. Maybe in the case of the this group it is just a failure to engage, the product just works and they have no need to be involved outside of viewing forums and stackoverflow. But I feel that without the involvement of this group a lot of open source software, no matter how good, can fall by the wayside.

But how do we get them involved? Well first we have to find them, in my case with PartCover as the project had been abandoned the users stopped raising issues on the SourceForge forums and tended to ask questions on other outlets such as StackOverflow, SharpDevelop or Gallio forums and mailing lists.

Finding the users

I scoured the internet and compiled a list of popular places that PartCover was mentioned or supported. I was surprised to find that PartCover was used or supported by SharpDevelop, TeamCity and TypeMock amongst others (and yet again I am surprised it was abandoned and not adopted by anyone sooner).

StackOverflow seems to be the main place where people ask questions and to keep track of questions I have subscribed to an RSS feed for the partcover tag; and as soon as an opencover tag becomes available, or I get enough rep to create it, I'll subscribe to that.

Twitter is also quite a common medium nowadays so I have also set up the following search filter "opencover OR partcover -rt -via" to see if anyone mentions either of the projects.

Engaging the users

Now I have found the users, or the majority of, I started notifying these lists, forums and projects that PartCover was alive again (and I have started to do the same to inform them about OpenCover). Hopefully bringing them back or at least notifying them that if they have really big issues there is somewhere to go.

Involving the Community users

This is the big ask and I don't have an answer. If the product works then they don't need to talk to the forums or declare their appreciation of a job well done. I think sites like ohloh are trying to address the balance. Some OS projects have a donate button, but I am not sure we are doing open source for money, though some projects do eventually go commercial, anyone else can pick up the original code and develop it. Maybe the users don't know how to be involved, in the case of my OS projects they are quite specialised and the learning curve may be too much for some. But I don't think you have to just be involved in projects you use a lot.

Possible ways to get involved

If you are good at graphics why not offer to knock up some graphics for use on web-sites and in the application. [I am quite lucky that Danial Palme added support for PartCover and OpenCover to his Report Generator tool and has done a much better job than I would ever do.]

If you are good at installers, or even if you want to learn more about them, offer to manage them on behalf of the project.If there is a project you like, support them on the forums like StackOverflow and help other users.

Perhaps update the wikis and forums, sometimes the users know how a product works or can be used better then the developers.

If your company uses a lot of open source, why not buy some licenses for useful software tools and donate them, geeks love shiny new toys; quite a few vendors such as NDepend will donate licenses to open source projects.

If you have an issue, try to help the developers as much as possible to resolve it by supplying as much information as you can and repeatable samples, remember the developers are international and doing this in their own time (as you probably know trying to repeat a scenario from scant information is very frustrating) and maintain contact whilst it is being resolved and let them know when it is.

Okay that's me done on the subject for now, suggestions anyone?

OpenCover First Beta Release

Sat, 18 Jun 2011 00:00:00 GMT

OpenCover First Beta Release

Okay, the first post on a blog I created many, many months ago and still not got round to starting. Why the delay? Well, just been busy and not a lot to say; actually some would say I have too much to say it's just not publishable.

But now I am happy to announce that the first release of OpenCover is now available on GitHub.

"So what?" I hear you say, "we have NCover, dotCover and PartCover [and probably many others with the word cover in the name,] do we need another code coverage tool?" Well, I think the answer is "Yes!" but before I say why a brief history.

About a year ago I adopted PartCover when I found it lost and abandoned and only supporting .NET2; also PartCover has a large user base, SharpDevelop, Gallio, to name but two, and I felt it was a shame to just let it fall by the wayside. I had also done some work on CoverageEye (another OpenSource tool that was originally hosted on GotDotNet and has since vanished) whilst working for a client in the UK, so I felt I had a fighting chance to do the upgrade to .NET4; I don't know if my changes ever got uploaded to GotDotNet as I was not in charge of that.

The adoption was far from easy for a number of reasons, one of which was I was surprised just how little C++ I could actually remember and it's changed a bit since I last used it in anger. Also due to lack of communication with the original developers meant that I was on my own in working out a) how it worked and b) just what the issues are (a lot of the reported issues had long since been abandoned by the reporters).

At the beginning of the adoption I cloned the SourceForge repository to GitHub, git being the in-thing at the time, and after I was eventually admitted access to SourceForge I attempted to maintain both repositories. Due to the lack of permissions on SourceForge, no matter how many times I asked, I eventually abandoned SourceForge and kept all development to GitHub; I also updated the SourceForge repository with a lot of ReadMe posts to point to GitHub.

So upgrading PartCover progressed and thankfully bloggers such as David Broman had already covered the subject matter about upgrading .NET2 profilers to .NET4 and things to look out for. That, it would turn out, was the easy bit.

PartCover had 3 main issues (other than lack of .NET4 support)

Memory usage
64 bit support
If the target crashed then you got no results.

I'll tackle each of these in turn:

Memory - PartCover builds a model of each assembly/method/instrumented point in memory; though I managed to cut down memory usage by moving some of the data gathering to the profiler host it wasn't enough - PartCover also added 10 IL instructions (23 bytes) for each sequence point identified + 4 bytes allocated memory for the counter.
64 bit support - PartCover used a complex COM + Named Pipe RPC, which thankfully just worked but I couldn't work out how to upgrade it to 64 bit (a few other helpers have offered and then gone incommunicado, I can only assume the pain was too much).
Crashing == no results - this was due to the profiler being shutdown unexpectedly and the runtime not calling the ::Shutdown method and as such all that data not being streamed to the host process; thankfully people were quite happy to fix crashing code so not a major issue but still an annoyance.

All of this would take major rework of substantial portions of the code and the thought was unbearable. I took a few stabs at bits and pieces but got nowhere.

Thankfully I had received some good advice and though I tried to apply it to PartCover I realised the only way was to start again, taking what I had learned from the guys who wrote PartCover and some ideas I had come across from looking at other opensource tools such as CoverageEye and Mono.Cecil.

OpenCover was born.

This time I created a simple COM object supporting the interfaces and then made sure I could compile it in both 32 and 64 bit from day one.

I then decided to make the profiler as simple as possible, so it is maintainable and move as much of the model handling to the profiler host, thank heavens for Mono.Cecil. The only complex thing was deconstructing the IL and reassembling it after it had been instrumented. OpenCover only inserts 3 IL instruction (9/13 bytes depending on 32/64 bit) per instrumented point; it forces a call into the profiler assembly itself and this C++ code then records the 'hit'.

Finally I decided I had to get the data out of the profiler and into the host as soon as possible. I toyed with WCF and WWSAPI but this also meant I had no XP support, but at least I could test other ideas. However if my target/profiler crashed I would loose the last packet of data; not drastic but not ideal. Eventually I bit the bullet and switched to using shared memory.

The switch to shared memory has brought a number of benefits one of which is the ability to handle a number of processes under the same profiling session, both 64 and 32 bit and to aggregate the results as they all use the same shared memory. I have yet to work out how to set this up via configuration files but anyone wishing to experiment can do so via modifying the call to ProfilerManager::RunProcess in the OpenCover.Host::Program.

So this is where we are now, OpenCover has been released (beta obviously) and as of time of writing some people have actually downloaded it. I am now braced for the issues to come flooding/trickling in.

Feel free to download and comment, raise issues on GitHub, get involved; Daniel Palme, he of Report Generator fame, is hopefully going to upgrade his tool to include OpenCover. OpenCover First Beta Release

Monkey See, Monkey Do; occasionally monkey learn.

This is an Elephant!

Being Lazy

Over confidence

Upgrading yarn from 1.x

CSP and Alpine.js

Starting off

Identification of the unsafe-inline violations

Creating and applying the external API

Protecting me from myself

Yarn Package Deduplication

Validate .tool-versions

Lambda function not picking up new configuration

Background

Investigation

Solution/Workaround

Feedback

Some notes on setting up a CSP on a legacy site

Nonces instead of Hashes

NGINX

Nonces and external ...

... scripts

... code snippets

Inline scripts

JavaScript URLs

What else?

Steps to setting up a Content Security Policy

Getting started

Reducing the noise

Unsafe?

Moving from reporting to enforcement

3rd party services

Report URI

Csper

CSP Evaluator

Ready to go live?

Enough about XSS what about clickjacking?

Afterthought

Heroku wants paying, so time to move on...

Migration process

Site generation

Afterthought

Simplifying your package management and security

Extending renovate

Security alerts

Speed up your Docker developer experience on MacOS

Starting point

Using NFS Mounts

Using Mutagen

Installing

Configuring

Summary

Night-night OpenCover

Scheduling AWS Lambdas, a 101

Debugging lambdas in VSCode

AWS Control Tower, making my life easier

Securing

Account Aliases and Roles

Upgrading to Ghost v4 on Heroku

Heroku Pipelines

Oh my zsh...

Themes

Plugins

git

asdf

zsh-autosuggestions

zsh-syntax-highlighting

Other plugins

Sharing "developer" secrets

Using Ghost as a headless CMS (part 2)

Disqus integration

Algolia integration

SSH tunneling from a Docker container

Adding Supervisor and AutoSSH to the container

Setting up a bastion server on AWS

Adding more users

Using Ghost as a headless CMS

Installing and running Gatsby

The downside

Automating the Gatsby build using Netlify

Identification of the `unsafe-inline` violations