Selenium is just not pulling its weight

I tried. I really tried.

• I built code that manages Selenium sessions so we can run suites of tests in parallel or standalone from within Eclipse
• We built special back-end code that Selenium tests can use for set-up and tear-down.
• I set up a hub and a handful of RCs (we call them runners, 'cause RC is a stupid name)
• I spent hours finding fixes and patching Selenium itself whenever it broke — pretty much every time a new version of Firefox was released.
• I spent hours chasing down issues with running tests in parallel against the same database, despite our careful setup/teardown code.
• I spent even more hours trying to set up some RCs on windows to use a hub running on RHEL.
• At one point, we had to make a bed of nails, running the same test every few seconds until it failed. We got the failure rate down to once every few days, but never did reach 100%.
  

But let's ask What were we trying to achieve?

We were using Selenium to execute storytests, which are complete, end-to-end tests of the whole user story. In the old days, people used FIT for this purpose (and conveniently omitted the UI layer to get it to work.) We tend not to write any new Selenium storytests, so in a way, our existing set of tests are being used as integration tests.

At one point we decided that Selenium is really only useful for exercising pages that make heavy use of Ajax/JavaScript. But that's precisely where Selenium tests seem to be flaky, failing unpredictably. Anyway, I much prefer to microtest Ajax/JavaScript rather than simply leaving it for Selenium to test.

So is Selenium useless? Hardly. When it works, it works remarkably well on a wide range of platforms. But what is its sweet spot? Dunno. If you think of something, let me know.

References:

• Don’t Just Break Software Make Software. A description and case study of using SDD.
• Story Test-Driven Development Flowchart showing how SDD works. (Slightly out of date.)
• Official Selenium site

Selenium won't launch Firefox under Snow Leopard

The latest FireFox (3.5.*) includes a version of sqlite that drives Selenium nuts. The basic problem is that Selenium is allowing FireFox's (whacko) version of sqlite to take precedence over the OS-supplied one.

The error message you'll see in the Selenium server log files looks like this:

dyld: Library not loaded: /usr/lib/libsqlite3.dylib
Referenced from: /System/Library/Frameworks/Security.framework/
Versions/A/Security
Reason: Incompatible library version: Security requires version
9.0.0 or later, but libsqlite3.dylib provides version 1.0.0

The fix is to disavow Selenium of that silly notion, which entails setting the DYLD_LIBRARY_PATH correctly. This is all explicated in the Selenium bug report.

The problem has already been fixed in the Selenium trunk, so the solution is to build your own Selenium. It's quite trivial, so here goes:

First, grab the patch, then execute these commands:

svn co http://svn.openqa.org/svn/selenium-rc/trunk .
patch -p0 < snow-leopard.patch
mvn install
cd $WHEREEVER_YOU_KEEP_SELENIUM
cp selenium-server.jar selenium-server.jar.bak
cp selenium-server/target/selenium-server-1.0.2-SNAPSHOT-standalone.jar selenium-server.jar

That takes care of the prob.