This morning I got a nice little email from a dear Egyptian friend, Ebeid Soliman (@ebeid_soliman) asking the following:
I know this may be something answered by google, but I trust your opinion.
What is the best free automated UI testing framework/tool you used ? or know ?
I actually already have a long draft on the subject showing the framework I use, and how to get basic stuff working on it, since this one is not yet complete, let me for now share my reply to him with you, as raw as possible …
(I have added some titles to make the long reply more readable)
Choosing a framework
Look, I haven’t tried many. Only Watin and Selenium, and even Watin didn’t dig it enough.
The people around me all seem to be using Selenium. This is not only the story though…
Selenium is meant to be cross-browser, while Watin is more of an IE thingy. You should be able to just swap a line of code to change the browser. In reality Selenium works best with Firefox. But this is usually not a problem because usually you don’t use those frameworks to confirm visual look (like screenshots and pixel-by-pixel comparison) or even cross-browser stuff. Usually you use them to ensure on the long run that the system “still” works the way it’s supposed to, the more you develop features and introduce changes, to make sure those didn’t break existing flows (for example, the presence of menu items, going to product page and completing an order, signing up, etc..).
The State of Selenium
This version is more solid, has so many features, so much coupled with Firefox, runs on Java, but is no-longer in active development.
The problem with this version is that it’s still not as feature-rich as the previous one, you can easily run into limitations and things you can’t do straight forward as in previous version. They try to release new versions very quickly (I think few weeks, like 2 or 4, can’t remember), so, it sure is going to rock, but it’s not still there quite yet.
Real World Requirements
Both of them allow you to run one central server that communicates to client machines if you want for example this server to run your tests in parallel on multiple test machines. At the current customer we have this triggered from a CI server (Cruise) which uses the Selenium library to call the Selenium Grid (Server) and then Selenium (1.3 I guess) connects to client test machines to run our tests. Of course running tests in parallel without affecting each other is another story on its own that is hard to get right as well.
So, we have hundreds of tests (now over 1000) running on Selenium 1.x with custom stuff (like for example an extension to allow using jQuery selectors to select elements, and custom code to overcome its JS nature to be able to reference the actual site window object instead of the wrong Selenium window itself), but I also know of another colleagues at another client getting good experience with Selenium 2.0 WebDriver. They also found some nice stuff from the community, and, they also wrote their own custom hacks to overcome some of its gotchas.
So, UI automation is not exactly nice or straight when you start doing real stuff (for small tries, you’ll be super amazed, it feels like magic to see the browser windows spin up and click themselves and go the boring steps to re-run some scenario! Generally speaking the least evil is Selenium, because it has the biggest community support, hence google-ability if you may (finding solutions created by others easier and get more stuff done). Although Selenium 2.0 is not as robust as Selenium 1.x, if you are going to use it for a new project, obviously not using Selenium 2.0 will be simply adding technical debt to your test code base, so, go with Selenium 2.0 and see how the journey feels for you, starting by exploring the Nuget packages available for it of course (Just search for Selenium and read package descriptions).
And I am always available if you hit something with it maybe I can help :)
More on Selenium
Since it was the main mention, you can learn more about Selenium from:
Hopefully I review my old draft and see what from it I can add here or make another “code-full” post from. The best help you can give me to do this faster is trying to learn about it, and throwing me some questions that can later turn into blog posts here. There are a lot of posts in this blog that couldn’t have existed without my friends throwing challenges and various queries in my inbox!