class Rating
{
  public static function ratingAverage($positive, $total, $power = '0.05')
  {
    if ($total == 0)
      return 0;
 
    $z = Rating::pnormaldist(1-$power/2,0,1);
    $p = 1.0 * $positive / $total;
    $s = ($p + $z*$z/(2*$total) - $z * sqrt(($p*(1-$p)+$z*$z/(4*$total))/$total))/(1+$z*$z/$total);
    return $s;
  } 
 
  public static function pnormaldist($qn)
  {
    $b = array(
      1.570796288, 0.03706987906, -0.8364353589e-3,
      -0.2250947176e-3, 0.6841218299e-5, 0.5824238515e-5,
      -0.104527497e-5, 0.8360937017e-7, -0.3231081277e-8,
      0.3657763036e-10, 0.6936233982e-12);
 
    if ($qn < 0.0 || 1.0 < $qn)
      return 0.0;
 
    if ($qn == 0.5)
      return 0.0;
 
    $w1 = $qn;
 
    if ($qn > 0.5)
      $w1 = 1.0 - $w1;
 
    $w3 = - log(4.0 * $w1 * (1.0 - $w1));
    $w1 = $b[0];
 
    for ($i = 1;$i <= 10; $i++)
      $w1 += $b[$i] * pow($w3,$i);
 
    if ($qn > 0.5)
      return sqrt($w1 * $w3);
 
    return - sqrt($w1 * $w3);
  }
}
 

The function takes 3 parameters: the positive votes, total votes, and the power. The power can be adjusted, 0.10 to have a 95% chance that your lower bound is correct, 0.05 to have a 97.5% chance, etc.  Sample usage:

sample(1,0);
sample(100,50);
sample(250,100);
sample(1000,500);
 
function sample($p,$n)
{
  echo Rating::ratingAverage($p,$p+$n);
}
 

Output:

When dealing with sites like Reddit, Digg, and the like you have a certain "freshness" element. The above solution might be a working model for the entire span of the site, but for that front page element you will need to implement some form of "gravity". This can be done by taking the raw score and decaying it over time, like so:

 
class Rating
{
  ...
  public static function gravityRating($positive, $total, $time, $power = '0.05')
  {
    if ($total == 0)
      return 0;
    return (Rating::ratingAverage($positive, $total, $power) / pow($time,0.5));
  }
  ...
}
 
sample(100,50,'0.5');
sample(100,50,'1');
sample(100,50,'4');
sample(100,50,'8');
sample(100,50,'24');
 
function sample($p,$n,$time)
{
  echo Rating::gravityRating($p,$p+$n,$time)."\n";
}
 

In the example above, $time represents the age (in hours) and you can see the decay in the output:

0.83141271310867
0.58789756740385
0.29394878370192
0.20785317827717
0.12000408843024

My recommendation would be to "cap" the time to stop decay after a fixed period such as 12 or 24 hours to stop the initial boost of fresh content and let it normalize quickly. The rate of decay of course, can be adjusted as fast or as slow as you want and again the individual weighting you want to apply will vary from site to site. Depending on the volatility of your content, a front page "freshness" that will encompass a week would not merit a 12 hour decay, but rather a week long decay. Hopefully the above code is enough to get started with content rating and making better use of user feedback and can help lead web masters to making a more intelligent calculation of their content beyond the traditional "5-Star Rating".

[ c9maji2tvz ]

API Features

• Supports CURL and fopen HTTP access methods, with autodetection
• PHP arrays for Google Analytics metrics and dimensions
• Account data object mapping - get methods for parameters
• Report data object mapping - get methods for metrics and parameters
• Easy filtering, use a GAPI query language for Google Analytics filters
• Full PHP5 Object Oriented code, ready for use in your PHP application

Google Quotas

• The quota applies to a single web property
• 10,000 requests per 24 hours
• 100 requests in any given 10-second period
• A query is also limited to pagination limits of 10,000 entries per feed, with a default response of 1,000 entries

While the broad set of data they have is extremely useful to pull down to do your own processing on, I find the custom visitor segments with the API a lot more interesting.    If you have a user based site with registration and collect information on your users, you can pull this data back out to determine a lot of information about your user.   A lot of tools are already built into Analytics for reporting but the ability to mash this data up to provide custom reporting for your users (customers) without giving them access to analytics is key to delivering metrics and performance

However, even with this speed improvement you still should consider how you structure your queries. There is no need to do a query across every field if you know you really want to filter everything down with a single filter query. Try moving that filter query (fq) into the actual query (q) as :. You might be surprised by the results...

i.e. For n cores the maximum throughput is roughly 1/n vs. a single index.

With multi-core sharded indexes the underlying assumption is that search performance improves by splitting your index into smaller chunks. These smaller shards are then faster and more efficient to search and index. However, you never get anything for free, the performance increase comes at a cost of higher CPU utilization. By breaking the index into multiple smaller pieces it makes searching and indexing on that smaller subset of the index faster, but you'll need to search each core individually for every query. Where as a single index runs one slightly slower query, a multi-core sharded query runs n queries in parallel and then combines the results.

There is one problem which still needs to be worked out with the multi-core sharded index. There is no distributed IDF (inverse document frequency). This is to say, if your documents are not spread evenly across all shards then you risk a result set that is improperly ordered based on your sorts, query boosts, etc. This happens with a distributed multi-core index because the scoring of the documents takes place within each individual core before the results are combined and the query returned.

Ideally, a multi-core index is great if you need to increase the performance of your queries and can afford to sacrifice some scalability and throughput to see it through.

Below are some charts of benchmarks that I have compiled on the CitySquares SOLR index. The specifications of the machine and indexes are as follows:

Testing machine - Dell r900:

• 4x Quad Core Intel(R) Xeon(R) CPU E7340 @ 2.40GHz (16 physical cores)
• 24GB RAM
• 3x 15k RPM drives in RAID 0
• Gig-Ethernet on a local LAN

Index Stats:

• 14.5 Million Documents
• 13 GB total size
• 56 fields (indexed and/or stored w/ various amounts of processing)
• Fully optimized index

Benchmarks:

• Used Apache Bench for testing purposes from another machine on the same LAN over Gig-E.

 
#!/bin/bash
echo "" > solr_results.log
for C in 2 4 8 16 32 64 128 256 512
do
N=$(($C*1000))
echo "ab -n$N -c$C" >> solr_results.log
ab -n$N -c$C 'http://solr:8080/solr/select?q=<ID>&qf=<FIELD>&fq=<FIELD>:<ID>&start=0&rows=20' >> solr_results.log
done
 

For the trends in red the lower the number the better.
For the trends in blue the higher the number the better.

We can see here in the above graph that there were no results from the 512 concurrency test. This is because there was a deadlock in the Apache Tomcat server. The max number of connections was set to 512 with an overflow of 100.  This is the cause of all the cases where there are no results for the 512 test case. Ironically the Single core without the cache managed to finish but the test with fieldCache on failed.

The higher the better in the following chart.

The lower the better in the following charts.

The above graph shows the only test to finish successfully with 512 concurrent connections was the single index with caching disabled.

This graph is the same as the one before without the last two concurrency levels so you can see whats going on at the beginning of the benchmark. Its still hard to see but the multi-core sharded indexes are a bit lower that the single indexes. Its clear however at the higher concurrencies that the single indexes beat the multi-core ones hands down.

Ive attached a spreadsheet with actual numbers from the benchmarks since some of the charts are hard to read.

So there it is, take it as you will. There are definitely benefits to moving from a single index to a distributed multi-core sharded index. However, whether it works for your dataset and application is up in the air. After these benchmarks we decided that the multi-core index that had served us well on Amazon's EC2 no longer worked well for us on our new managed hosting. We are currently running a single index at CitySquares.

The following scenario I am about to describe is proof positive that you should always take the time to understand the underlying functionality of whatever operating system, programming language or application you are using. Let my oversight and 'quick fix solution' be a lesson to you, it is almost always worth the upfront cost of doing something right the first time so you don't have to keep revisiting the same issue.


Before I delve into the nuances of SOLR let me first give you some background on what took place over the last half year at CitySquares. Back in the fall of last year the CitySquares website began experiencing an exponential growth in traffic. This growth was due to an expansion of its IYP (Internet Yellow Page) services into the New England and Metro New York areas. Prior to and during the beginning of the first wave of traffic growth, every business listing was powered by very large MySQL queries including a couple joins. The queries themselves weren't all that complex but they were big and unwieldy with joins on very large tables and lots of columns in the result sets. In some of the larger cities covered at the time (Manhattan, Bronx, Queens, Boston, etc) there were up to 100,000 rows of data that needed to be sorted before returning a rather small subset (20-40 rows) for each business listing page load. While this wasn't a big deal when CitySquares was still a niche Boston centric destination, it quickly became a huge burden on the MySQL servers. Some of these queries were so big the servers would run out of memory trying to crunch through a 3GB temp table and start thrashing the disks to server a request for Manhattan. We needed a better solution and quick.

Luckily for us we had already implemented a SOLR search engine with all the necessary data indexed from our database initially with the sole intent that search result sets shouldn't have to query the database. This worked to our advantage since it was very easy for us to modify the code base to query SOLR instead of MySQL. Both result sets were formatted as an object with the same field names and all. It was a perfect drop in replacement.

The SOLR solution we implemented utilized SOLR's wild card q.alt=*:* field to select all documents while applying filter query (fq) on that set to get all documents related to our filter. It was a huge win for us at the time. Not only were the queries faster than the MySQL ones, but the SOLR servers could handle more of these queries without even coming close to exhausting the server's resources. This quick and dirty solution was satisfactory for the next few months until CitySquares' next round of expansion began, where again, the queries became a burden. The second time around we didn't have another seemingly quick fix. I spent a couple days trying to figure out a better way to implement the q.alt=*:* field but to no avail I gave up and moved onto other performance optimizations.

Unfortunately, I didn't take the time to understand the code behind the query and I didn't understand exactly how SOLR was implementing the query in its back end process. Since I didn't understand the basis of the problem I couldn't possibly know the query could be easily re-factored. After a few weeks of high loads, 20+ on our 8 core servers, I struck up a conversation with Michael, the developer who wrote the query. We discussed how the query worked and what it needed to do and after five minutes we had discovered a much better way to structure the query. It took me only about a minute or two to re-factor the original query to produce the exact same result set. This new query was incredibly fast! I benchmarked it to be about 100x faster than the previous query and on top of that it was a simple drop in replacement!

From what I've deduced the original query passed a blank query string with a filter query to SOLR which in turn defaulted to the q.alt catch all first and then applied the filter on the catch all query. This is exactly the opposite of what we were expecting SOLR to do. We believed that the filter was applied first and then the q.alt was applied. However, that was not the case. while this misunderstanding wasn't ideal it wasn't too slow either with only 1.4 million documents to parse over. However once CitySquares hit the 14.5 million mark this query became unmanageable. Basically SOLR parsed over every single document in the index before applying the query filter we were using. To rectify this and regain performance and through put on our servers I simply moved the filter query statement to the query statement and specified the query field to be the same as the original filter field.

i.e.

Original query passed a blank query string with a filter query:

• select?q=+&fq=<FIELD>:<ID>

The updated query now passes the id as the query string and specifies the former filter field:

• select?q=<ID>&qf=<FIELD>

Instead of taking advantages of SOLR's and every other search engines strength of O(1) search time we were at the mercy of its worst case scenario O(n) scan time. This simple misunderstanding of how SOLR processes queries in the back end caused massive performance and throughput bottlenecks. These bottlenecks affected our short and long term infrastructure plans, and was the root cause of many performance headaches for our users, customers and IT department.

If this isn't proof positive that you should always take the time to understand the underlying functionality of whatever operating system, programming language or application you are using I don't know what is.

The algorithm simplifies a polyline by removing vertices that do not contribute (sufficiently) to the overall shape. It is a recursive process which finds the most important vertices for every given reduction. First, the most basic reduction is assumed. A single segment connecting the beginning and end of the original polyline. This is when the recursion starts, the most significant vertex (the most distant) for this segment is found and, when the distance from this vertex to the segment exceeds the reduction tolerance, the segment is split into two sub-segments, each inheriting a subset of the original vertex list. Each segment continues to subdivide until none of the vertices in the local list are further away than the tolerance value.

There is a PHP class that does just this: Douglas-Peucker Polyline Simplification in PHP by Anthony Cartmell. Based on the original quality of the data and tolerance level, I was able to achieve a 90-93% reduction in size. This reduction allows me to represent significantly more data at a reasonable performance level to clients. Keep in mind, that this reduction is removing data out of the coordinate array so the quality of your representation will go down with the tolerance and reduction being applied. I highly suggest that you play around with the tolerance until you find a good balance between data size and image quality.

• Price- An m1.xlarge instance will run you ~$600 with data transfer costs. Managed hosting solutions run cheaper especially if you plan on purchasing in bulk. The grid is designed for on-demand computation and not as a cost efficient web services.
• Configuration- There are a limited number of options and you will not be able to tailor the hardware to your application. Databases over 10GB of size will have performance issues since that is the memory cap.
• Network storage- The primary disks offers limited storage, additional volumes will need to be attached across the network and at an additional cost.
• Software - No hardware based solutions for load balancing or custom application servers. The model is software driven so all needs must be met with a software solution. In a managed hosting or collocation solution you will at least have the option of adding additional hardware and having a private network. No dedicated switching, routers, firewall, or load balancers.

EC2 might be right for you if:

• Distribution Awareness- Your application was designed to scale horizontally from the get-go and you can take advantage of grid computing.
• Research and Development- EC2 & Rightscale will allow for you to bring up new servers, test configuration, and scale quickly. If you are not sure what your hardware demands will be or the scope of the project it will allow for some flexibility to get this right before committing to rather lengthy contracts with other hosting options.
• Disaster Recovery- If you need an off-site mirror for your site that you can keep dormant and activate as needed.

Over the past several months I have been doing extensive development using Amazon's EC2 as my hardware infrastructure. I was tasked with taking CitySquares.com from a New England area hyper-local search and business directory to a national site in a few months. Due to the memory limitations of EC2 instances, m1.xlarge only providing 15GB the jump from a 15GB database to anything larger becomes very costly. When everything was able to be contained in two servers in a master/slave environment we were able to provide redundancy, performance, and easy management when working with the database. The final estimations of the national roll out would put our core data at 50GB. Far too large for any one EC2 instance. Going to disk was not an option as everything works off of EBS attached storage and that any disk writes means traveling over the network. An additional overhead which degrades performance even more when switching off RAM. Then, there was also the nature of the data, which means that at any page load, any piece of data could be requested.

With OS overhead, index storage, and ndb overhead, each x1.mlarge instance gave about 12GB of usable storage. Include replication, and it comes down to 6GB of storage per node. To store a ~50GB database that I had planned on requires 8 storage nodes, two management servers, and two mysqld api servers. This is where it became important to understand the advantages of vertical scaling versus horizontal scaling. EC2 provided fast horizontal scaling and configuration. Servers can be launched on demand and their configurations scripted. While I appreciated that aspect of cloud computing and being able to bring that many servers up and configure each one relatively quickly I really just needed two decent boxes with 64GB of RAM in each and a master/slave setup. The operations costs for the cluster was $6000/month, a hefty bill considering I could buy all the hardware needed to run the cluster in just a few months of paying for EC2.

Twelve servers later and a working cluster we were able to successfully roll out our MySQL Cluster with minimal performance loss. A lot of it was due to tweaking every query top to bottom. The site runs on a Drupal core, which meant a lot of the queries were not designed with distribution awareness both from code in house and the core. This was another added growing pain since the network overhead of running 12 servers on shared resources, with mediocre latency, and throughput limitations was amplifying flaws in the database design and every poorly designed query and join degraded performance significantly.

To give credit where credit is due, EC2 did allow us to scale the site up rather quickly. We were able to test server configurations, new applications, and have easy management. It would not have been possible for us to push out the data, handle influx of new traffic, and expand as fast as we had without it. Long term however, it made absolutely no sense that once we were finished scaling up, to stay on EC2. It is a great platform for start-ups to be able to configure and launch servers for their service or application and grow rapidly.

As a database platform, until EC2 offers more configuration options in it's hardware and the ability to increase memory, the cap of 15GB will make EC2 problematic for any database that plans on growing past that. It is important to understand your application and database needs before considering EC2. It is no surprise to me that even with the RightScale interface, and easy management of EC2 web sites are reluctant to switch off their own hardware or managed hosting.

Both Sun and Continuent are pursuing MySQL Clustering on cloud computing. As of this post, Continuent for EC2 was still in closed beta and testing and Sun is doing their own research into offering more support for database clusters on compute clouds. Maybe in the future this can be something to revisit, but it would require more ndb configuration options on the network layer to cope with shared bandwidth and additional hardware configurations by Amazon (someday).

The 6.4 release which is in beta now offers new features which would make MySQL Clustering more attractive on cloud computing architecture:

• Ability to add nodes and node groups online. This will allow the database to scale up without taking the cluster down.
• Data node multithreading support. The m1.xlarge instance comes with 4 cores.

If you are setting up a MySQL cluster, the following resources will help get you up and running quickly:

• http://www.severalnines.com/
• http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-configuration.html

There are quite a few blogs, articles and reviews out there that mention all the positive aspects of EC2 and I have written a few of them myself. However, I think users need to be informed of the negative aspects of a particular platform as well as the positive. I will be brief with this post as my next will focus on designing an architecture around these limitations.

The biggest limitations of Amazon's EC2 at the moment as I have experienced, are the latencies between instances, latencies between instances and storage (local, and EBS), and a lack of powerful instances with more than 15GB of RAM and 4 virtual CPUs.

All the latency issues can all be traced back to the same root cause, a shared LAN with thousands of non localized instances all competing for bandwidth. Normally, one would think a LAN would be quick... and they generally are, especially when the servers are sitting right next to each other with a single switch sitting in between them. However, Amazon's network is much more extensive than most local LANs and chances are your packets are hitting multiple switches and routers on their way from one instance to another. Every extra node added between instances is just another few milliseconds that get added to the packet's round trip time. You can think of Amazon's LAN as a really small Internet. The layout of Amazon's LAN is very similar to that of the Internet, there is no cohesiveness or localization of instances in relation to one another. So lots of data has to go from one end of the LAN to the other, just like on the Internet. This leads to data traveling much farther than it needs to and all the congestion problems that are found on the Internet can be found on Amazon's LAN.

For computationally intensive tasks this really isn't too big a deal but for those who rely on speedy database calls every millisecond added per request really starts adding up if you have lots of requests per page. When the CitySquares site moved from our own local servers to EC2 we noticed a 4-10x increase in query times which we attribute mainly to the high latency of the LAN. Since our servers are no longer within feet of each other, we have to contend with longer distances between instances and congestion on the LAN.

Another thing to take into consideration is the network latency for Amazon's EBS. For applications that move around a lot of data, EBS is probably a god send as it has a high bandwidth capability. However, in CitySquares' case, we wind up doing a lot of small file transfers to and from our NFS server as well as EBS volumes. So while there is a lot of bandwidth available to us, we can't really take advantage of it, especially since we have to contend with the latency and overhead of transferring many small files. Not only are small files an issue for us but we also run our MySQL database off of an EBS volume. Swapping to disk has always been a critical issue for databases but the added overhead of network traffic can wreak havoc on your database load much more than normal disk swapping. You can think of the difference in access times from disk to disk over a network as a book on a bookcase vs a book somewhere down the hall in storage room B. Clearly the second option would take far longer to find what you are looking for and that's what you have to work with if you want to have the piece of mind of persistent storage.

The last and most important limitation for us at CitySquares was the lack of an all powerful machine. The largest instance Amazon has to offer is one with just 15GB of ram and 4 virtual CPUs. In a day and age where you can easily find machines with 64GB of RAM and 16 CPUs, you are definitely limited by Amazon. In our case, it would be much easier for us just to throw hardware at our database to scale up but the only thing we have at our disposal is a paltry 15GB of RAM. How can this be the biggest machine they offer? Instead of dividing one of those machines in quarters just give me the whole thing. It just seems ludicrous to me that the largest machine they offer is something not much more powerful than the computer I'm using right now.

Long story short, just because you start using Amazon's AWS doesn't mean you can scale. Make sure your architecture is tolerant of higher latencies and can scale with lots of little machines because that's all you have to work with.

]]> http://www.derivante.com/2008/10/08/the-limitations-of-scaling-with-ec2/feed/ 2 http://www.derivante.com/2008/09/16/running-your-own-hardware-vs-ec2-and-rightscale-part-2/ http://www.derivante.com/2008/09/16/running-your-own-hardware-vs-ec2-and-rightscale-part-2/#comments Tue, 16 Sep 2008 14:33:24 +0000 Justin Leider http://justinleider.com/?p=40 This week I've been reminded of a very important lesson... No matter how abstracted you are from your hardware, you still inherently rely on its smooth and consistent operation.

This past week CitySquares' NFS server went down for the count and was completely unresponsive to any type of communication. In fact, the EC2 instance was so FUBAR we couldn't even terminate it from our RightScale dashboard. A post on Amazon's EC2 board was required to terminate it. Turns out the actual hardware our instance was running on had a catastrophic failure of some sort. Otherwise, at least so I'm told, server images are usually migrated off of machines running in a degraded state automatically.

Needless to say, the very reasons for deciding against running our own hardware have come back to plague us. Granted we weren't responsible for replacing the hardware but we were still affected by the troublesome machine. We weren't just slightly affected by the loss of our NFS server either. Since we are running off of a heavily modified Drupal CMS our web servers depend on having a writable files directory. As it turned out Apache just spun waiting for a response from the file system, our web services ground to a halt waiting on a machine that was never going to respond... ever. Talk about a single point of failure! A non critical component, serving mainly images and photos managed to take down our entire production deployment.

This event has prompted us to move forward with a rewrite of Drupal's core file handling functionality. The rewrite will include automatically directing file uploads to a separate domain name like csimg.com or something similar. Yahoo goes into more detail with their performance best practices. However, editing the Drupal core is generally frowned upon and heavily discouraged since it usually conflicts with the upgrade path and maintainability of the Drupal core becomes much more difficult. While we haven't stayed out of the Drupal core entirely, the changes we have made are minor and only for performance improvements. I believe it is possible to stay out of the core file handling by hooking into it with the nodeapi but it seems like more trouble than its worth.

The idea behind the file handling rewrite is to serve our images and photos directly from our Co-Location while keeping a local files directory on each EC2 instance for non user committed things like CSS and JS aggregation caching among other simple cache related items coming from the Drupal core. This rewrite will allow us to run one less EC2 instance, saving us some money as well as remove our dependence on a catastrophic single point of failure.

For the time being we have set up another NFS server. This time based on Amazon's new EBS product. I spoke about this in a previous post. One of the issues we had when the last NFS server went down was the loss of user generated content. Once the instance went down all the storage associated with that instance went down with it. There was no way to recover from the loss, it was just gone. This is just one of the many possible problems you can run into with the cloud. While on the pro side, you don't have to worry about owning your own hardware, the con side is you cant recover from failures like you can with your own hardware. This is a very distinct difference and should be seriously considered before dumping your current architecture for the cloud.

]]> http://www.derivante.com/2008/09/16/running-your-own-hardware-vs-ec2-and-rightscale-part-2/feed/ 2 http://www.derivante.com/2008/09/05/nuances-of-ec2-and-rightscale/ http://www.derivante.com/2008/09/05/nuances-of-ec2-and-rightscale/#comments Fri, 05 Sep 2008 15:25:07 +0000 Justin Leider http://justinleider.wordpress.com/?p=36 So here it is, about two weeks have passed since CitySquares officially migrated its server infrastructure over to EC2 and RightScale. All in all, everything went relatively well. There were a few hiccups on the cut over day that left users with some error pages. Most of these issues were related to the DNS changeover and a little confusion over whether to set up the DNS records with Amazon's internal IPs or the elastic external IPs. Common sense said to set the DNS to the external IPs but turns out we were supposed to use the internal IPs (10.0.0.0/8 and not the elastic IPs 75.0.0.0/8) when referencing machines that are within the Amazon networks. Oops.

Other than that, Ive spent the last couple weeks smoothing everything out and getting things working at 100%. There were a few bugs that cropped up at first, mainly IT stuff, Apache configs, htaccess issues, HAProxy issues, making sure MySQL and our NFS server was backing up correctly. All these things took precedence but lately Ive been working on trying to increase performance. At this moment I'm not entirely sure why but, our MySQL database is running queries extremely slowly. At this point it could be anything from network latency, to slow machines, to an improperly tuned config. However, MySQL performance tuning is out of the scope of this post and will be the topic of a future entry. (If a MySQL DBA is reading this and would like the opportunity to play around with EC2 and RightScale, please get in touch with me.)

In preparation for the tuning, not only for the MySQL server but the Apache servers as well, I have been setting up a separate development environment that is exactly identical to our production. With RightScale's clone feature I was able to easily duplicate everything from one deployment to the other. That said, let me make it clear that it will copy Everything. After changing all the necessary script inputs for the dev deployment I figured I was ready to start launching the new servers... WRONG. After booting the dev master DB server as well as our dev load balancer and dev NFS server I realized that they had stolen all the IPs from our production deployment! Bad News! Needless to say, CitySquares was down for the count for the few minutes it took me to figure out what had happened, fix the mistake and then wait for Amazon to reassign the elastic IPs. So here is a friendly reminder, check the server info tab before launching and make sure it isn't going to clobber your existing elastic IPs.

Another somewhat annoying issue I ran into while trying to copy over our MySQL S3 backup from the production bucket to the development bucket was the lack of a decent copy function. RightScale has provided copy and move functionality on a somewhat basic level. You can move or copy files either one or many at a time. However, there is a limitation to this. Each file you copy will append its location into the URL and each directory path its somewhat long. Eventually you reach the maximum URL string limit and all the effort you put into selecting the files is for nothing. Not only do you have to select every file you want to copy, you have to manually assign it to the new location. This means lots of copy and pasting. If you have a directory that has hundreds of files in it, good luck. You are better off just uploading it to a new bucket. Either way, this could have been easily solved by having a copy bucket or directory option. Problem solved.

While these few things are annoying, they aren't show stoppers, but they are definitely things to keep in mind when using these services. I'd like to end on a positive note so Ill mention the exceptional monitoring services that are installed and configured by default on every server image we have used so far. I am extremely impressed with the out of the box functionality of the graphs and they definitely make up for the other shortcomings. They have everything I could ever want to look at and then some. From standard CPU load to the number of I/Os p/s as well as yearly, quarterly, monthly, daily and hourly time frames in three sizes, small, medium and large. All browsable via up to date thumbnail previews.

If you are considering cloud computing, I would recommend taking a look at RightScale and Amazon's web services.

]]> http://www.derivante.com/2008/09/05/nuances-of-ec2-and-rightscale/feed/ 1 http://www.derivante.com/2008/08/21/amazons-ebs-elastic-block-store/ http://www.derivante.com/2008/08/21/amazons-ebs-elastic-block-store/#comments Thu, 21 Aug 2008 14:48:55 +0000 Justin Leider http://justinleider.wordpress.com/?p=28 I wrote just yesterday about running your own hardware vs. using EC2 and RightScale and one of the major issues I found with EC2 was the lack of a persistent storage medium. Well, I knew the folks over at Amazon were hard at work on a new service that would allow persistent storage and turns out I received this email in my mailbox this morning:

Dear AWS Developer,

We are pleased to announce the release of a significant new Amazon EC2 feature, Amazon Elastic Block Store (EBS), which provides persistent storage for your Amazon EC2 instances. With Amazon EBS, storage volumes can be programmatically created, attached to Amazon EC2 instances, and if even more durability is desired, can be backed with a snapshot to the Amazon Simple Storage Service (Amazon S3).

Prior to Amazon EBS, block storage within an Amazon EC2 instance was tied to the instance itself so that when the instance was terminated, the data within the instance was lost. Now with Amazon EBS, users can chose to allocate storage volumes that persist reliably and independently from Amazon EC2 instances. Amazon EBS volumes can be created in any size between 1 GB and 1 TB, and multiple volumes can be attached to a single instance. Additionally, for even more durable backups and an easy way to create new volumes, Amazon EBS provides the ability to create point-in-time, consistent snapshots of volumes that are then stored to Amazon S3.

Amazon EBS is well suited for databases, as well as many other applications that require running a file system or access to raw block-level storage. As Amazon EC2 instances are started and stopped, the information saved in your database or application is preserved in much the same way it is with traditional physical servers. Amazon EBS can be accessed through the latest Amazon EC2 APIs, and is now available in public beta.

We hope you enjoy this new feature and we look forward to your feedback.

Sincerely,

The Amazon EC2 team

So this is indeed good news and removes the biggest con I mention about the EC2 platform!

From the very beginning our contract IT guy set us up with a extremely robust and flexible IT infrastructure. It consists of a few machines running Xen Hypervisors with Gentoo as the main host OS. Running Gentoo allows us to be as efficient as possible by specifically optimizing and compiling only the things we need. While this is a good step, it is Xen that really makes the big difference. It allows us to trade around resources as we see fit, more memory here, more virtual CPUs there, all can be done on the fly. For a startup or any company with limited resources this is rather essential. You never know where you are going to need to allocate resources in the months to come.

While this is all well and good, we are still limited when it comes to scaling with increasing traffic or adding additional resource intensive features. We have a set amount of available hardware and adding more is an expensive upfront capital investment. Not only that but in order for us to really begin to take advantage of Xen and use it to its full potential we were presented with an expensive option, it required the purchase of a SAN and more servers. For those in the industry I don't think I need to mention that these get expensive in a hurry. This would have been a huge upfront cost for us, one we didn't want to budget for. The second option, which is the one we eventually went with was to drop our current hardware solution and make the plunge into cloud computing with Amazon's EC2.

Here I am now. A couple of weeks into the switch with a lot of lessons learned. There are definitely pros and cons for each platform, either going with EC2 or rolling your own architecture. Before I get into the details I want to make clear that there are many factors involved in choosing a technology platform. I am only going to scratch the surface, touching upon the major pros and cons with respect to my own opinions with best interest for CitySquares in mind.

Let me begin by starting with the pros for running your own hardware:

• The biggest pro is most definitely persistence across reboots. I can not stress the importance of this one. You really take for granted the ability to edit a file and expect it to be there the next time the machine is restarted.

  • You only need to configure the software once. Once its running you don't really care what you did to make it work. It just works, every time you reboot.

  • UPDATE 8/21/08: Amazon releases persistent storage.

• Complete and utter control over everything that is running. This extends from the OS to the amount of RAM, CPU specs, hard drive specs, NICs, etc. The ability to have a economy or performance server is all up to you.

• Rather stable and unchanging architecture. Server host keys stay the same, the same number of servers are running today as there were yesterday and as there will be tomorrow.

• Reboot times. For those times when something is just AFU you can hit the reset button and be back up and running in a few minutes.

• You can physically touch it... Its not just in the cloud somewhere.

Some cons for running your own hardware:

• Companies with limited resources usually end up with architectures that exhibit single points of failure.

  • As an aside, you can be plagued by hardware failures at any time. This usually is accompanied by angry emails, texts and calls at 3am on Saturday morning.

• Limited scalability options. For a rapidly expanding and growing website, the couple weeks it takes to order and install new hardware can be detrimental to your potential traffic and revenue stream.

• Management of physical pieces of hardware. Its a royal pain to have to go to a co-location to upgrade or fix anything that might need maintenance. Not to mention the potential down time.

  • Also, there are many hidden costs associated with IT maintenance.

• Up front capital expenditures can be quite costly. This is especially true from a cash flow perspective.

• Servers and other supporting hardware are rendered obsolete every few years requiring the purchase of new equipment.

These pros and cons for running your own hardware are pretty straight forward. Some people might mention managed hosting solutions which would mostly eliminate some of the cons related to server maintenance and hardware failures. However, this added service comes with an added price tag for the hosting. Whether it is right for you or your company is something to look into. We decided to skip this intermediary solution and go straight to the latest and greatest solution which is cloud computing. To be specific we sided with Amazon's EC2 (Elastic Compute Cloud) using RightScale as our management tool.

Some of the pros for using EC2 in conjunction with the RightScale dashboard are as follows:

• Near infinite resources (Server instances, Amazon's S3 Storage, etc) available nearly instantaneously. No more Slashdot DoS attacks if everything is properly configured and set to introduce more servers automatically. (RightScale Benefit)

• No upfront costs, everything is usage based. In the middle of the night if you are only utilizing one server thats all you pay for. Likewise, if during peak hours you're running twenty servers you pay for those twenty servers. (Amazon Benefit, RightScale is a monthly service)

• No hardware to think of. If fifty servers go down at Amazon we wont even know about it. No more angry calls at 3am. (Amazon Benefit)

• Multiple availability zones. This allows us to run our master database in one zone which is completely separate from our slave database. So if there is an actual fire or power outage in one zone the others will theoretically be unaffected. The single points of failure mentioned before are a thing of the past and this is just one example. (Amazon Benefit)

• Ability to clone whole deployments to create testing and development environments that exactly mirror the current production when you need them. (RightScale Benefit)

• Security updates are taken care of for the most part. RightScale provides base server images which are customized upon boot with the latest software updates. (RightScale Benefit)

• Monitoring and alerting tools are very good and highly customizable. (RightScale Benefit)

Some of the cons for using EC2 and RightScale:

• No persistence after reboot. I can't stress this one enough! All local changes will be wiped and you'll start with a blank slate!

  • All user contributed changes must be backed up to a persistent storage medium or they will be lost! We back up incrementally every 15 minutes with a full backup every night.

  • UPDATE 8/21/08: Amazon releases persistent storage.

• Writing scripts to configure everything upon boot is a time consuming and tedious process requiring a lot of trial and error.

• Every reboot takes approximately 10-20 minutes depending on the number and complexity of packages installed on boot. Making the previous bullet point even that much more painful.

• A few of the pre-configured scripts are written quite well. The one for MySQL is as good as they get. You upload a config file complete with special tags for easy on the fly regular expression customization. The Apache scripts on the other hand are about as bad as they get. Everything must be configured after the fact.

  • With Apache however, you'll be writing regular expressions to match other regular expressions. Needless to say is a royal pain and you usually end up with unreadable gibberish.

So there you have it, take it as you wish. For CitySquares, EC2 and RightScale were the best options. It allows us to scale nearly effortlessly once configured. It is also a much cheaper option up front where as owning your own hardware is generally cheaper in the long run. We did trade a lot of the pros of owning your own hardware to get the scalability and hardware abstraction of EC2. It was a tough decision for us to switch away from our current architecture but in the end it will most likely be the best decision we've made. The flexibility and scalability of the EC2 and RightScale platform are by far the biggest advantages to switching and in the end its what CitySquares needs.

]]> http://www.derivante.com/2008/08/20/running-your-own-hardware-vs-ec2-and-rightscale/feed/ 10 http://www.derivante.com/2008/08/13/digging-into-haproxy/ http://www.derivante.com/2008/08/13/digging-into-haproxy/#comments Wed, 13 Aug 2008 22:59:08 +0000 Justin Leider http://justinleider.wordpress.com/?p=19 Well its been a few weeks since my last posting here and there is certainly a good reason for that. Every once in a while I just need to completely unplug from technology. So it only made sense for me to go away on vacation to the middle of no where up in Maine's great north woods for a couple of weeks. No computers, no cellphones, no towns, no people, just dirt logging roads, lakes, rivers, wildlife and trees. Now that I'm back and caught up I will begin to start posting regularly again.

Getting back to reality, as the title states, this post will focus on the reasons behind using HAProxy as well as a little bit on CitySquare's implementation of the load balancer. Let me start by quoting a description of HAProxy from their website:

“HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. “

While the high availability aspect of HAProxy is all well and good, everything is expected to be high availability these days. Any sort of downtime has become unacceptable even in the middle of the night. This is especially true when relying on search engine driven traffic. I've noticed that search engines like Google and Yahoo to name a couple, really ramp up their crawl rate in the wee hours of the morning. The crawl rate is boosted more so on weekend nights when even fewer people are searching the web and the search engines can allocate more of its resources towards web crawls. CitySquares has certainly been subject to DoS attacks by GoogleBot on Friday nights.

This is where the load balancing aspect of HAProxy comes into play, it is one of the main reasons for choosing it as our front facing service. With just a couple HAProxy servers we can maintain redundancy while having a nearly unlimited pool of Apache web servers to hand off requests to. We don't need any special front facing, load balancing hardware to act as a single point of failure. We can also keep some money in our pocket at the same time by utilizing a software solution. Luckily, HAProxy is open source and free to the world, licensed under the GPL v2.

Not only does HAProxy handle our load balancing but it also serves as a central access point for DNS purposes. This solution is certainly much better than our current DNS round robin which is limited in its own right. Is this common sense? Probably, but I figured it was worth pointing out.

Lastly, security is always a concern for heavily trafficked and high profile sites. The developer behind HAProxy has been very proactive with the program architecture and coding practices and as such HAProxy can claim it's never had a single known vulnerability in over five years. Since all front facing applications are subject to attacks from so many different sources these days, having a stable and secure application is a godsend when it comes to any sort of security related IT maintenance.

As far as implementation goes, I suspect that eventually we might need to move the HAProxy instances onto their own dedicated servers as traffic increases. In the meantime, with EC2, we are running them in parallel with Apache on the same servers. This is purely a cost savings measure as every server instance started with EC2 results in more cash out the door. As it is, HAProxy is incredibly fast and lean and really doesn't consume much in the way of system resources, either CPU load or memory utilization.

There are certainly other reasons for choosing HAProxy but they are past of the scope of this post. I encourage everyone to take a serious look at HAProxy when spec'ing out a load balancer or proxy.

]]> http://www.derivante.com/2008/08/13/digging-into-haproxy/feed/ 2 http://www.derivante.com/2008/07/24/an-architecture-overview-apache-mysql-memcached-sqlite/ http://www.derivante.com/2008/07/24/an-architecture-overview-apache-mysql-memcached-sqlite/#comments Thu, 24 Jul 2008 19:56:41 +0000 Justin Leider http://justinleider.wordpress.com/?p=11

In my last post I mentioned the numerous technologies which were on tap for the upcoming version of CitySquares. This installment will continue to define an overview of the underlying architecture and begin to dig a little deeper into the actual implementation of the technologies. The idea and focus of this new architecture is aimed at creating a much more stable and scalable platform for us to work with. Before I get into the details you'll see Ive provided a graphic representation of how the architecture will be laid out.

A visual representation of a horizontal web architecture.

Bear with me as I explain the work flow behind this graphic as it is not 100% clear from the visual representation. First off, I run Ubuntu Linux which is great for just about everything I need, except for creating any sort of graphics, so I apologize in advance for the lackluster graphic. As you can see, there are a few different layers: users, HA Proxy, Apache, Memcached, SQLite and finally MySQL labeled as databases.

First and foremost are our beloved users, which whom without we would have no need for a website. Starting from the beginning, the users request a page from CitySquares, from there their request is passed through one of two HA Proxy servers. The sole purpose of these two machines is to load balance the incoming requests among all our Apache web servers and serve as a failsafe for one another. Once the user's request has been accepted and forwarded along to Apache we actually begin to process the request.

The Apache servers run PHP and XCache modules. The PHP part I feel is fairly straight forward and out of the scope of this post so I will skip that part of the architecture. XCache however, is used in conjunction with and is an enhancement to PHP. More specifically XCache is an opcode optimizer and cache. It works by removing the compilation time of PHP scripts by caching the compiled and optimized state of the PHP scripts directly in the shared memory of the Apache server. This compiled version can increase page generation times by up to 500%, speeding up overall response time and reducing server load.

Just as with all dynamic websites most if not all the actual data is stored in databases. Gone are the days of flat files with near zero processing required. Databases are the new workhorses of the web world and as such usually become the bottle neck of the overall system. CitySquares is in a somewhat unique position, nearly all our page loads have quite a bit of location and distance based processing and nearly all of this is done in our MySQL database. So while our Apache servers are sitting idle waiting for responses from their queries, the DB is preforming the brunt of the work calculating distances between objects and the like.

We can reduce this bottleneck in a couple of different ways, the first of which is object caching. We will use Memcached to cache objects returned from the database. Say for example, we know the distance between two businesses. We know with a fair amount of certainty that those two businesses are going to be in the same place they were an hour ago, just as they were a week ago and as they will be a day from now. So we can cache this information with an expiration time of a couple days, thus saving ourselves the expense of calculating the distance between them on every page load. Of course if a user comes by and changes the location of one of these businesses, we can expire the object in cache and replace it with a newly calculated object straight from the database on the subsequent page load. These expensive queries require large table scans and mathematical formulas calculations on every row. These query results can be cached to free up the database and allow it to do what it does best. Store and retrieve data.

In the case where we cant find the data in Memcached, either because it doesn't yet exist or has expired we will turn to our databases. We must first query a SQLite instance which is the gate keeper between Apache and the numerous databases we have. By having a separate lookup table we can essentially divide and parcel out our data sets on a table by table basis even down to an entry by entry basis. Depending on the type of data we are requesting SQLite will provide us with the location of one database or another to query for our data.

One could argue that this just adds another layer of latency and they would be correct. However, as scalability becomes an issue you will find that adding database replication generally results in diminishing returns. As new servers are brought online the overhead associated with replicating writes across all the replicated servers becomes choking and creates its own bottleneck. On the other hand, with a lookup table and a horizontal database architecture we don't have to worry about database replication nearly as much. You can just as easily divide your data sets into different databases. Now how you go about this varies greatly depending on your data. For CitySquares the solution turns out to be rather simple. Everything we do is location specific so it only makes sense that each data set is only as big as its parent city. Theoretically every city and all the data related to said city could reside in its own database. As you can probably guess we are only performance limited by the biggest cities, Manhattan, Brooklyn, etc. In these few cases we can always fall back to bigger and better servers and or replication if necessary.

Just as our database has become a bottleneck in our current site, our search engine is also one as well, just to a lesser extent. We can take the lessons learned from our horizontal database architecture and apply it to the search engine architecture as well. By dividing our data sets into logical partitions we can keep our data from getting too large and unwieldy; And with these smaller data sets we can reduce or remove all together the overhead associated with replicating data over multiple machines.

While this solution sounds great, it won't be worth the effort if every time a programmer wanted to access some data they would be required to check Memcached, then SQLite and then finally MySQL for every query. In order for this to be feasible from a programmers standpoint the programmer should never have to think about this underlying architecture. This of course I will discuss in greater detail in the upcoming installments. Stay Tuned.

]]> http://www.derivante.com/2008/07/24/an-architecture-overview-apache-mysql-memcached-sqlite/feed/ 3 http://www.derivante.com/2008/07/21/a-technology-overview/ http://www.derivante.com/2008/07/21/a-technology-overview/#comments Mon, 21 Jul 2008 20:21:28 +0000 Justin Leider http://justinleider.wordpress.com/?p=5

This will be the first post in a multi-part series, each of the following installations will detail the technologies and implementations of the upcoming CitySquares revision. I hope to cover the entire page generation process, starting with the user's first request to the resulting dynamically generated HTML, CSS, JS, etc. Before I dive too deep into the technical aspect of things I would like to give a brief overview of what is to come.

For starters, CitySquares currently owns and operates its own servers in a co-location not far from our headquarters. This will be the first thing to go as we switch to Amazon's EC2 and S3 in conjunction with RightScale. By switching off of our own hardware we will absolve ourselves of this oft troublesome and physically limiting layer. By using RightScale's server templates and management scripts we can control the precise number of servers in operation. Coping with increased or decreased load will be handled autonomously throught the RightScale interface, no more DoS by SlashDot and more more wasted cycles during off peak hours. Our server deployment will contain a few different types, each one specially tuned and selected for its specific purpose. Without getting into too much detail here, our deployment will consist of the following:

• HA Proxy for load balancing

• Apache with PHP and XCache

• Memcached

• MySQL master/slave configuration

• File server with automated revisioning, concatenation and minimization of css, js, etc

• Tomcat with SOLR search engine

Once setup, most of the overhead associated with operating our own IT infrastructure will be removed from the equation.

Not only will our IT situation improve but our coding environment will change dramatically as we move away from Drupal's more primitive procedural style coding practice and towards Symfony's OOP style. Symfony is a PHP based MVC (Model, View Controller) framework. It is loosely based on RoR's (Ruby on Rails) best practices for codability and maintainability. We will be using it with the Doctrine ORM (Object Relational Mapping) and Smarty templating engine. These architectural and IT changes will work to promote a cleaner, more efficient and maintainable coding practice. In the end all these disruptive changes will be justified, allowing us to focus on what we do best, provide users with hyper-local search results.

]]> http://www.derivante.com/2008/07/21/a-technology-overview/feed/ 1