Category Archives: Programmation

Tout ce qui touche la programmation

Automatic Recovery & Restart in .Net application

NET%20LogoThe goal here is to improve the reliability of a .net client application by managing cases where things goes wrong and the universe of your product fall on himself, for example due to memory corruption, unhandled exception, Stack/Memory/<place your stuff-Overflow and so on.

The Issue

In the current situation, when you want to know if things goes wrong there’s natively nothing really in place. You can register for unhandled exception, but if for some reason your application hangs more than 60 seconds and come in the famous “Not Responding” state… well there’s not a lot of things to do. When it occurs, we could want to be able to backup some data’s, cleanly closed some connections, alert the backend, etc

The Answer

The answer is provided by the Windows API Codepack. This codepack available through NuGet package work on windows vista and higher (tested on Vista, 7 and 8, haven’t test on 8.1, ut it should too) and expose some P/Invoke calls to kernel32.dll functions (so be carefull too, when used, it’s out of the “nice pink unicorn .net land”). Once you’ve added the NuGet package reference you gain access to two “events”, Recovery and Restart. Here’s a method to register to theses.

protected override void OnStartup(StartupEventArgs e)
{
    RegisterARR();
    base.OnStartup(e);
}

private void RegisterARR()
{
    if (CoreHelpers.RunningOnVista)
    {
        // register for Application Restart
        ApplicationRestartRecoveryManager.RegisterForApplicationRestart(new RestartSettings(string.Empty, RestartRestrictions.None));

        // register for Application Recovery
        ApplicationRestartRecoveryManager.RegisterForApplicationRecovery(new RecoverySettings(new RecoveryData(PerformRecovery, null), 5000));
    }
}

For my part I put the code in the App.cs and call the register method within the OnStartup. If it’s “running on vista” which means “running on higher version than vista”, we first register to the restart event. When registering for restart we provide an instance of RestartSettings. The first parameter it gets is the command line arguments that will be used for the restart, in case we want to define some special parameters. The second parameters is an enum that allows us to restrict the restart in some cases with 5 defined states.

  • None: No restart restrictions
    NotOnCrash: Do not restart the process if it terminates due to an unhandled exception.
  • NotOnHang: Do not restart the process if it terminates due to the application not responding.
  • NotOnPatch: Do not restart the process if it terminates due to the installation of an update.
  • NotOnReboot: Do not restart the process if the computer is restarted as the result of an update.

The second thing we do is register for recovery. This means that if the application will need a restart (from the same reasons as before), what function do we want to run to allow later recovery.

When registering for recovery we supply an instance of RecoverySettings. The first parameter it gets is the RecoveryData object, which wraps a delegate to be called and some state parameter that will be passed (in this example, null). The second parameter is the keep alive interval, which will be explained shortly. The recovery function should obey some rules in order to avoid the application getting stuck (again) in the recovery function. You must call ApplicationRecoveryInProgress every few miliseconds (in the example, KeepAliveInterval = 5000). This tells the ARR mechanism, “I know it takes some time, but don’t worry, I’m still alive and working on the recovery stuff”.

private int PerformRecovery(object parameter)
{
    try
    {
        ApplicationRestartRecoveryManager.ApplicationRecoveryInProgress();
        // Perform recovery here
        ApplicationRestartRecoveryManager.ApplicationRecoveryFinished(true);
    }
    catch
    {
        ApplicationRestartRecoveryManager.ApplicationRecoveryFinished(false);
    }
    return 0;
}

And finally we just need to define an unregister method, called after the Recovery & Restart to end the process and called from the OnExit event handler of the application. (Yeah, sometimes things goes well… sometimes…)

private void UnregisterApplicationRecoveryAndRestart()
{
    if (CoreHelpers.RunningOnVista)
    {
        ApplicationRestartRecoveryManager.UnregisterApplicationRestart();
        ApplicationRestartRecoveryManager.UnregisterApplicationRecovery();
    }
}

Conclusion

That’s it, now you can face the death and tackle the hell of that kind of situation. But I want to precise some things to conclude. This mechanism doesn’t prevent your application to fall in an inconsistent state. So it means, if this mechanism is called, something really bad happened and you can’t guarantee the consistency of the memory, you should always rely on other mechanism to manage the persistence scenario when things goes wrong, work with little transaction, cache datas, provide validity flags on the database records etc. Most of the time I use it to :

1. Send a memory dump to an ftp server for further debugging
2. Report the issue in a tracking system
3. Dump the memory, check the file on reboot and try to recover the lost documents/datas and always ask the user to check the validity if he want to re-inject them in the system or avoid him to re-inject them based on automatic validation.
3. Clean connections, context, etc… We all knows server with badly managed sessions, not responding due to a massive client crash with persistent connection for whatever reason or a faulty batch/transaction process hanging due to the same reasons…

Catch you next time and keep it bug free !

Kent Beck : “I get paid for code that works, not for tests”

Ohhh...Alright...That’s perhaps the most surprising, powerful and reassuring sentence I’ve read on stackoverflow since I write code ! Kent Beck, respected authority, creator of Extreme Programming, TDD and writter of several great reference books, mainly at the great Addison-Wesley edition, have answered the following text to someone asking “how deep do you unit test”. Myself I usually break on this question as I tend to be JEST (Just Enough System Testing) in the JESA spirit (Just Enough System Analysis, great book)  from Jesa Yourden. And as I’ve started to go deep in BDD, the question was even more important as I can really quickly test stuffs but also forget to test specific dark cases that I would probably test in a more classical TDD way. So here’s the inspiring Beck’s answer :

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. I do tend to make sense of test errors, so I’m extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.

So when that kind of ground breaking answers are made by that kind of guy… it’s always good to meditate a bit on it !

Indeed, since this answer, 5years ago, some big improvements have been made, but it’s still a great view from a inspiring person.

The original StackOverflow topic : http://stackoverflow.com/questions/153234/how-deep-are-your-unit-tests/153565#153565

Catch you next time and keep it bug free !

Noninvasive global software mocking with registry

RegEditI recently had to do something unusual for me.

The issue

In production-like environment, I had to “mock” a third party application called by mine. My application call the third party app just like as we do when we run something from command prompt with some arguments. The problem was a little bug in theses arguments, but I can’t update directly my application and had to find a workaround until I can. Also I would like at the same time to debug a bit what’s done by logging the passed arguments. So here’s the initial situation :

exereplace1

The solution

Here’s what I want to do to solve the problem :

exereplace2

Adding an intermediate application that my application will call and then forward the call to the third party app. A kind of software man-in-the-middle. But as I can’t modify my app I just can’t ask her to point to another place and the call target is not soft coded. I finally, after a day of search, found a solution, this good ol’ registry !

If you go to :

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

You can create a registry key with the name of your app, in my case it’s ThirdPartyApplication.exe and then create a string value called Debugger with the name of the application to run instead. Each time we’ll call ThirdPartyApplication.exe, the mentioned application will be executed with the normal target as arguments. I can then call my third party app and patch the transmitted values until I can patch my application and remove the registry key.

Also it can be a good joke to do to some colleague you (don’t) like… Let’s say, replacing explorer.exe by mspaint… or making iexplore to call iexplore, resulting in a beautiful call loop… I’m sure you have enough imagination to do something a bit fun for 2 minutes the monday morning with that 😉 But never forget, the shorter jokes are the better…

Catch you next time and keep it bug free !

Extract data from HTML page with XPath and Linq

As developer, we usually need to extract data from a html web page in our projects, but most of the time we try to find another solution due to the complexity of the task. Indeed, the classical approach would be to use regex or similar and try to find out our informations in the page structure. But there’s something really helpfull, the HTMLAgilityPack. I’ll give you 2 example, the first one for a basic overview of the HTMLAgilityPack and a second one with much more details about the process I follow for that kind of tasks.

So, HTMLAgilityPack allow us to download a webpage and then consult the content with his methods or with the provided XPath API. To install it, just download the NuGet package from VS2010 >= or grab the project from the codeplex page.

Once done, you can use it to parse and extract data from html page.

using HtmlAgilityPack;

Then you can use it to download the page :

string url = "http://www.simpleweb.org/";
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load(url);

Or you can use LoadHtml(string htmlCode) on the HtmlDocument object to work on raw html code contained in memory.

Once done, you can write XPath query to retreive data. For this example, I want to retreive the url of the wiki (second link in the left menu).
According to the google chrome debugger, the quicker and optimzed XPath to this property is : (thank you chrome :p)

//*[@id="top_option"]/div[2]/span/a

For those not familiar with XPath, it’s really simple. This query just mean, from the root (//) all tags (*) with id attribute set to “top option” ([@id=”top_option”]) then select the second div (div[2]) then select span and within the span select link (/span/a). That give us the following Linq query :

var result = from link in htmlDoc.DocumentNode.SelectNodes("//*[@id='top_option']/div[2]/span/a")
                         select link.Attributes["href"].Value;

The intellisense will be once again your best friends as this assembly is really well designed and intuitive. So here I select all the nodes who match the XPath query and then select the href attribute value.

Let’s take another example, I want to get an object tree which represent the whole menu. Just before that, I would like to mention something about XPath. This is a great query language… if you know it ! I mean, you can write really powerful but complex query using it, and I always think about the next one who’ll need to understand and/or update the query. So I usually prefer if I have to share it or in a collaborative context to write multiple steps/subqueries to make it more “developer friendly”. Even some simple query can seems really complex for someone not familiar with it. That said, let’s write the query to rebuild the full menu tree of the same page. This time I’ll also share the steps I follow for that kind of task.

First, analyze the page structure. We can see that the menu is contained within a td (/html/body/table/tbody/tr[3]/td/table/tbody/tr/td[1]). This td contain 7 div, but only the 3 first one interest us. To know that, I use the Chrome debugger, when you place your move hover a tag, Chrome will highlight in the page the corresponding area.

<div style="float:left;" id="top_option" class="aTopLink">
	<div>
		<span><a class="aTopLink" href="/">Home</a></span>
	</div>
	<div>
		<span><a class="aTopLink" href="/wiki/">Wiki</a></span>
	</div>
</div>
<div style="float:left;" id="my_menu" class="sdmenu">
    <div class="collapsed">
        <span>MIBs</span>
		<a href="/ietf/mibs/">Browser</a>
                <a href="/ietf/mibs/validate/">MIB Validation</a>
                <a href="/ietf/mibs/byEncoding?encoding=txt">Plain text</a>
                <a href="/ietf/mibs/byEncoding?encoding=highlighting">Syntax Highlighting</a>
                <a href="/ietf/mibs/byEncoding?encoding=html">HTML encoding</a>
		<a href="/ietf/enterprise/">Vendor MIBs</a>
		<a href="/ietf/mibs/search/">Search</a>
            </div>
	    <div class="collapsed">
		<span>RFCs</span>
		<a href="/ietf/rfcs/rfcbynumber.php" class="current">By Number</a>
		<a href="/ietf/rfcs/rfcbytopic.php">By Topic</a>
		<a href="/ietf/rfcs/rfcbystatus.php">By Status</a>
		<a href="/ietf/rfcs/rfcbymodule.php">By Module</a>
                <a href="/ietf/rfcs/complete/">Complete</a>
		<a href="/ietf/rfcs/">Search</a>
	    </div>
            <div class="collapsed">
                <span>Software</span>
                <a href="/software/select_obj.php?orderBy=package&amp;oldorderBy=package&amp;orderDir=asc&amp;free=free">Freely available</a>
                <a href="/software/select_obj.php?orderBy=package&amp;oldorderBy=package&amp;orderDir=asc&amp;com=com">Commercial</a>
                <a href="/software/">Search</a>
            </div>
            <div class="collapsed">
                <span>CFP/Conferences</span>
                <a href="/wiki/Events">Conferences</a>
                <a href="/wiki/Cfp">Call for Papers</a>
            </div>
            <div class="collapsed">
                <span>Bibliography</span>
                <a href="/wiki/Books">Books</a>
                <a href="/wiki/Journals">Journals</a>
                <a href="/wiki/Theses">Theses</a>
            </div>
            <div class="collapsed">
                <span>Tutorials</span>
                <a href="/wiki/Internet_Management_Tutorials">Slides</a>
                <a href="/wiki/Video_tutorials_on_Internet_management">Podcasts</a>
                <a href="/wiki/Exercises_in_Internet_Management">Exercises</a>
                <a href="/wiki/Other_tutorials">Other tutorials</a>
		<a href="/tutorials/demo/snmp/v1/">SNMPv1 demo</a>
		<a href="/tutorials/demo/snmp/v2c/">SNMPv2c demo</a>
		<a href="/tutorials/demo/snmp/v3/">SNMPv3 demo</a>
    </div>
</div>
<div style="float:left;" id="bottom_option" class="aTopLink">
    <div>
        <span><a class="aTopLink" href="/wiki/Podcasts">Podcasts</a></span>
    </div>
    <div>
        <span><a class="aTopLink" href="http://www.simpleweb.org/wiki/Traces">Traffic Traces</a></span>
    </div>
    <div>
        <span><a class="aTopLink" href="/ifip/">IFIP WG6.6</a></span>
    </div>
    <div>
        <span><a class="aTopLink" href="mailto:simpleweb@simpleweb.org">Contact</a></span>
    </div>
</div>

To represent it we will have the following object model in C# :

public class Node
{
    public List<Node> Childs { get; set; }
    public string Name { get; set; }
    public string Url { get; set; }
}

And the query will be the following :

var result = (from topNode in htmlDoc.DocumentNode.SelectNodes("//div[@id='top_option']/div/span/a")
                select new Node()
                            {
                                Name = topNode.InnerText,
                                Url = topNode.Attributes["href"].Value
                            })
                            .Union(
                from bottomNode in htmlDoc.DocumentNode.SelectNodes("//div[@id='bottom_option']/div/span/a")
                select new Node()
                        {
                            Name = bottomNode.InnerText,
                            Url = bottomNode.Attributes["href"].Value
                        })
                        .Union(
            from groupNode in htmlDoc.DocumentNode.SelectNodes("//*[@id='my_menu']/div[*]")
            let groupChilds = (from nodes in groupNode.SelectNodes("a")
                                select new Node()
                                            {
                                                Name = nodes.InnerText,
                                                Url = nodes.Attributes["href"].Value
                                            })
            select new Node()
                        {
                            Name = groupNode.SelectSingleNode("span").InnerText,
                            Childs = groupChilds.ToList()
                        });
                                    
DisplayMenu(result.ToList());

We use 3 query with union. As said, I tried to decompose a bit the query, the two first are the same, select the links and create nodes with the innerText of the a tag and the href attribute. Then we have the centre of the menu with hierarchy. For this, we select each div representing a group, then we create a first list of node like for the two first request based on the link contained within the div, and we create the node object with the list we’ve just created as child’s object list and the span content as Name.

As you can see, we can easily write complex requests in no time (this one took me about 30minutes to write, test and include in this article). HTMLAgilityPack is a great tool that allow us to use XPath and finally address a great answer to the critical html data parsing issue !

See you next time and keep it bug free ! 🙂

Very simple plugin mechanism in C#

NET%20LogoOnce again I had to face the famous content aggregator problem. I already have done that on several project so I decided to write an article on that for the ones who wonders how to accomplish things like that.

Background

So, the situation is : I have a system, who consume several services, aggregate the results and provide the output to the user. I also want to be able to develop each providers class as separated assemblies (plugins) so the providers can develop their own plugins and distribute them to the software users without having to recompile the application, just by dragging the assembly within a folder. Here’s a diagram that resume the situation.

Plugin1

Technical Analysis

Regarding the requirements, some answers can seem obvious. First, we need to provide an interface to define what’s a plugin. Then, we need something to control the plugin creation and access. This component will also be responsible in this case for the plugin call. Each providers will have his own implementation of the IPlugin interface. In this case, the manager called PluginProvider, will be a singleton instance and instantiate the plugins on creation. Here’s the basic UML diagram of the system with some example providers.

Plugin2

Then, we have a dependency tree like this one

Plugin3

The assemblies that contains plugins will be placed in a folder where the application will look at startup. They just need to know the interface to implement and the data types exchanged if required. The UI layer refer the core, that contain the PluginManager and the shared assembly. No references to the plugins are created.

Implementation

The implementation is pretty simple. Here’s our plugin interface :

public interface IPlugin
{
    string Name { get; }
    void Action();
}

Then we can implement a basic plugin like this :

public sealed class ExternalPlugin : IPlugin
{
    public string Name
    {
        get { return "Exemple Plugin"; }
    }

    public void Action()
    {
        DoSomething();
    }
}

And finally, the most important thing, the PluginManager.

public sealed class PluginProvider
{
    private static PluginProvider instance;
    /// Warning ! Not a thread safe implementation, change it if needed
    public static PluginProvider Instance { get { return instance ?? (instance = new PluginProvider()); } }

    private List pluginInstances;

    private PluginProvider()
    {

    }

    public void Initialize()
    {
        pluginInstances = pluginInstances ??
               (from file in Directory.GetFiles(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"Plugin\"), "*.dll")
                from type in Assembly.LoadFrom(file).GetTypes()
                where type.GetInterfaces().Contains(typeof(IPlugin))
                select (IPlugin)Activator.CreateInstance(type)).ToList();
    }

    public void Action()
    {
        if(pluginInstances != null)
            pluginInstances.ForEach(plugin => plugin.Action());
    }
}

So, first we have a basic not thread safe singleton implementation. When we call the Initialize() method, the Linq query will list all the dll in the Plugin application folder, load each assemblies, find the types that implement IPlugin and then create an instance of them. All the instances are then stored in the pluginInstances private list. When we call PluginProvider.Instance.Action(), we iterate over each instance and call the Action() method. If the method return something, we can easily adapt the code to aggregate the call returns and return it to the PluginProvider caller. And that’s pretty much it !

Conclusion

This architecture can then be improved with for example repositories for query caching, using code contract to improve strength, using a thread safe singleton implementation, using a facade to directly call Providers.Action() or make this generic to be able to work with any interface which inherit from IPlugin… With all these improvements you’ll start to have a kind of little plugin framework in our personal toolbox. Modularity is usually a key factor in software development and that kind of pattern are really usefull for a lot of scenario.

If you have any questions, feel free to ask in the comments below 🙂

See you next time and keep it bug free !

Reload IFrame and manage Same-Origin policies with JQuery

jquery-logoFor a project, I had to dynamicaly reload an IFrame (within a JQuery dialog box) and include cross domain pages.
To load the IFrame and set the source :

$(FrameID).attr("src", URL);

To reload the IFrame you can use :

$(FrameID).contentDocument.location.reload(true);

But if the content come from another domain you will be denied access to the iframe’s contentDocument property due to the same-origin policy.
To force the cross-domain iframe to reload if your code is running on the iframe’s parent page, you can set it’s src attribute to itself.

var iframe = document.getElementById(FrameID);
iframe.src = iframe.src;

But be carefull, this is a hack ! So if you do something like this in production code, comment it, or the next one would probably remove this code who seems totally useless.

Mono and Firebird : IOControl Not Implemented

mp-mono-logo This is a repost from a question I asked on stackoverflow.com that I finally manage by myself…

The issue

I’m actually working on a project where I need to host a .NET webservice on a linux server hosting a firebird database. When I try to run the sample code from http://www.mono-project.com/Firebird_Interbase and call the method, my webservice return me :

System.NotImplementedException: The requested feature is not implemented.
  at System.Net.Sockets.Socket.IOControl (IOControlCode ioControlCode, System.Byte[] optionInValue, System.Byte[] optionOutValue) [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.Common.Extensions.SetKeepAlive (System.Net.Sockets.Socket socket, UInt64 time, UInt64 interval) [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.Client.Managed.Version10.GdsConnection.Connect () [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.FirebirdClient.ClientFactory.CreateManagedDatabase (FirebirdSql.Data.FirebirdClient.FbConnectionString options) [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.FirebirdClient.ClientFactory.CreateDatabase (FirebirdSql.Data.FirebirdClient.FbConnectionString options) [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.FirebirdClient.FbConnectionInternal.Connect () [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) FirebirdSql.Data.FirebirdClient.FbConnectionInternal:Connect ()
  at FirebirdSql.Data.FirebirdClient.FbConnectionPool.Create () [0x00000] in <filename unknown>:0
  at FirebirdSql.Data.FirebirdClient.FbConnectionPool.CheckOut () [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) FirebirdSql.Data.FirebirdClient.FbConnectionPool:CheckOut ()
  at FirebirdSql.Data.FirebirdClient.FbConnection.Open () [0x00000] in <filename unknown>:0

I’m using the NETProvider-3.1.0.0-MONO_LINUX version of the provider (http://www.firebirdsql.org/en/net-provider/). Apparently this method is not yet implemented in Mono, but as I don’t have the source code, do you have any idea of some workaround ? I’m pretty sure there must be one because if not, this provider made for mono is totaly useless and not tested… 😦

The answer

I’ve found a solution by using the previous version (NETProvider-2.6.5-MONO_LINUX) of the drivers based on this article.
What’s funny, is that the driver in question is not listed anymore on the official Firebird website… anyway, if you have issue to, try this one from fb sourceforge : NETProvider-2.6.5-MONO_LINUX