Developer:Mothership
Delivering the myExperiment.org service
By Danius Michaelides, 4th August 2008
The myExperiment.org service is hosted on two servers; a web frontend and a database backend. The frontend consists of the Apache webserver and a cluster of Ruby on Rails processes using the Mongrel Cluster software. Static content such as CSS stylesheets, Javascript files and images are served directly by Apache, whereas for dynamic content (HTML and xml), Apache makes connections to the Ruby on Rails processes using the load balancing and proxy Apache modules.
The database, which is a major component of the Ruby on Rails system, is hosted on the second server in the form of MySQL. This second server also runs the Solr search server, which is a Java implementation of the Lucene search library running as a Java servlet in Tomcat. Both these components of the service benefit from large amounts of memory and since their memory usage is predictable they are configured to use 75% of the physical memory of the server. Conversely, the memory consumption of the Ruby on Rails processes is more dynamic and tends to grow with time, requiring them to be restarted approximately once a fortnight. High memory usage is monitored using the nagios monitoring tools, which also checks on other operating parameters such as high cpu load and disk usage. Nagios also checks for correct and timely response of the entire service by making web requests as if it were an external user.
The development process of myExperiment.org requires frequent updates to be rolled out to the main service. This is aided by maintaining a separate server and myExperiment installation for final testing of code before updating the code on the main service. Maintaining a separate test server also allows the previewing of new features to users that might have an interest, for example if they were involved in the conception. The test server, contains, where possible, a recent snapshot of the data from the live site so that testers have content to work with. It also enables the checking for performance regressions with automated tools. The test service is also provided to developers who are writing applications that make use of the myExperiment API as during development they may impose unexpected loads which would have otherwise impacted the main site.
