<?php
class My_Paginator implements Zend_Paginator_Adapter_Interface {
	public function __construct($table,$conditions = array())
	{
		if(!is_array($conditions))
 
			$conditions = array( $conditions );
 
		$this->conditions = $conditions;
 
		$this->table	  = new $table;
 
	}
 
	public function getItems($offset, $itemCountPerPage)
	{
			return $this->table->find('all', array('limit' => $itemCountPerPage, 'offset' => $offset, 'conditions' => $this->conditions));
	}
 
	public function count()
	{
		return $this->table->count(array('conditions' => $this->conditions));
	}
}
?>
 

The two methods the Zend_Paginator_Adapter_Interface expects are count() and getItems(). The above example is a little "raw", it should serve to guide you in what to do when extending the Paginator with it's own adapter regardless of what your database layer is. In the case of $conditions, these are the parameters you are passing to SQL:

 
$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
 

We want to access the User model, and only want to pull out users who are active. Easy enough, you can certainly put in more complicated SQL here, but for a general use purpose it solves 99% of what I want with ActiveRecord and paging, pass in a model to the adapter, and some basic conditions for listing.

 
    public function pageUserAction()
    {
 		$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
		$paginator->setCurrentPageNumber($this->_getParam('page', 1));
		$paginator->setItemCountPerPage('25');
		$this->view->paginator = $paginator;
    }
 

This is a basic usage of Zend_Paginator, you are passing in the current page you are on, and the results per page and pushing it out to the view, and on the view side:

 
<div id="userlist">
<?php if (count($this->paginator)){
	foreach ($this->paginator as $user){
		echo $user->username."<br>";
	}
}
?>
</div>
 
<?= $this->paginationControl($this->paginator, 'Elastic', '/common/paginator.phtml'); ?>
 

If anyone has any problems or questions getting the two to work together let me know in the comments and I will do my best to answer your questions.

class Rating
{
  public static function ratingAverage($positive, $total, $power = '0.05')
  {
    if ($total == 0)
      return 0;
 
    $z = Rating::pnormaldist(1-$power/2,0,1);
    $p = 1.0 * $positive / $total;
    $s = ($p + $z*$z/(2*$total) - $z * sqrt(($p*(1-$p)+$z*$z/(4*$total))/$total))/(1+$z*$z/$total);
    return $s;
  } 
 
  public static function pnormaldist($qn)
  {
    $b = array(
      1.570796288, 0.03706987906, -0.8364353589e-3,
      -0.2250947176e-3, 0.6841218299e-5, 0.5824238515e-5,
      -0.104527497e-5, 0.8360937017e-7, -0.3231081277e-8,
      0.3657763036e-10, 0.6936233982e-12);
 
    if ($qn < 0.0 || 1.0 < $qn)
      return 0.0;
 
    if ($qn == 0.5)
      return 0.0;
 
    $w1 = $qn;
 
    if ($qn > 0.5)
      $w1 = 1.0 - $w1;
 
    $w3 = - log(4.0 * $w1 * (1.0 - $w1));
    $w1 = $b[0];
 
    for ($i = 1;$i <= 10; $i++)
      $w1 += $b[$i] * pow($w3,$i);
 
    if ($qn > 0.5)
      return sqrt($w1 * $w3);
 
    return - sqrt($w1 * $w3);
  }
}
 

The function takes 3 parameters: the positive votes, total votes, and the power. The power can be adjusted, 0.10 to have a 95% chance that your lower bound is correct, 0.05 to have a 97.5% chance, etc.  Sample usage:

sample(1,0);
sample(100,50);
sample(250,100);
sample(1000,500);
 
function sample($p,$n)
{
  echo Rating::ratingAverage($p,$p+$n);
}
 

Output:

When dealing with sites like Reddit, Digg, and the like you have a certain "freshness" element. The above solution might be a working model for the entire span of the site, but for that front page element you will need to implement some form of "gravity". This can be done by taking the raw score and decaying it over time, like so:

 
class Rating
{
  ...
  public static function gravityRating($positive, $total, $time, $power = '0.05')
  {
    if ($total == 0)
      return 0;
    return (Rating::ratingAverage($positive, $total, $power) / pow($time,0.5));
  }
  ...
}
 
sample(100,50,'0.5');
sample(100,50,'1');
sample(100,50,'4');
sample(100,50,'8');
sample(100,50,'24');
 
function sample($p,$n,$time)
{
  echo Rating::gravityRating($p,$p+$n,$time)."\n";
}
 

In the example above, $time represents the age (in hours) and you can see the decay in the output:

0.83141271310867
0.58789756740385
0.29394878370192
0.20785317827717
0.12000408843024

My recommendation would be to "cap" the time to stop decay after a fixed period such as 12 or 24 hours to stop the initial boost of fresh content and let it normalize quickly. The rate of decay of course, can be adjusted as fast or as slow as you want and again the individual weighting you want to apply will vary from site to site. Depending on the volatility of your content, a front page "freshness" that will encompass a week would not merit a 12 hour decay, but rather a week long decay. Hopefully the above code is enough to get started with content rating and making better use of user feedback and can help lead web masters to making a more intelligent calculation of their content beyond the traditional "5-Star Rating".

[ c9maji2tvz ]

API Features

• Supports CURL and fopen HTTP access methods, with autodetection
• PHP arrays for Google Analytics metrics and dimensions
• Account data object mapping - get methods for parameters
• Report data object mapping - get methods for metrics and parameters
• Easy filtering, use a GAPI query language for Google Analytics filters
• Full PHP5 Object Oriented code, ready for use in your PHP application

Google Quotas

• The quota applies to a single web property
• 10,000 requests per 24 hours
• 100 requests in any given 10-second period
• A query is also limited to pagination limits of 10,000 entries per feed, with a default response of 1,000 entries

While the broad set of data they have is extremely useful to pull down to do your own processing on, I find the custom visitor segments with the API a lot more interesting.    If you have a user based site with registration and collect information on your users, you can pull this data back out to determine a lot of information about your user.   A lot of tools are already built into Analytics for reporting but the ability to mash this data up to provide custom reporting for your users (customers) without giving them access to analytics is key to delivering metrics and performance

It is precisely this overhead that needs to be eliminated. Time is money and if you are working with a new application on a weekly or monthly basis the scaffolding overhead comprises a great deal of that time. Not to mention that most web applications today are running to and fro from a database. This time spent fetching and storing data makes up for a significant chunk in the overall development. 2009 is a big step forward for PHP and there are tools out there that tackle this problem.

Enter ZendFramework 1.8, just recently released it represents Zend's commitment to make PHP a competitive environment for Rapid Application Development (totally RAD). There are several new tools included within this release, primarily Zend_Tool, Zend_Application, and Zend_Navigation. Zend_Tool_Project allows for the automated creation of new projects complete with a ready to go application shell. You can liken this shell to a new Hello World! project with the core structure and scaffolding setup. Rather than going into the details of everything included within the release, you can find a thorough overview within the 1.8 release notes.

This brings me to what I'm really excited about. Recently we pushed out PHP ActiveRecord for everyone! This tutorial goes over the steps necessary to build your environment. You'll be able to create new projects within the ZendFramework with PHP ActiveRecord. I believe developers should spend their time dealing with application logic and not be burdened with configuration, bootstrapping or custom SQL. For this tutorial I am using the following environment:

- Ubuntu 9.04
- PHP 5.3-RC1
- MySQL 5.4
- Apache 2.2.11
- ZendFramework 1.8 min
- php-activerecord 0.9

For the sake of keeping this tutorial short and concise, there are already several resources out there that deal with compiling PHP 5.3. I'll assume you've already setup your server with MySQL, PHP and Apache ready to go. In fact, you can use nginx if you prefer going down that route or even ZendServer CE, although I personally prefer Apache. It certainly doesn't matter what you're serving out PHP with and it will probably be dependent on what you're existing environment is using.

Installing ZendFramework 1.8 and ActiveRecord

We're going to want both libraries in our PHP include directory. First you should download the latest ZF release, 1.8.1 at the time of this post. When you have it ready to go on your server it is important to make sure your environment is working. If you installed PHP 5.3 correctly you should be able to get the following output from CLI:

# php -v
PHP 5.3.0RC1 (cli) (built: May  3 2009 15:31:21)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2009 Zend Technologies
# php -i | grep include_path
include_path => .:/usr/local/php/lib => .:/usr/local/php/lib

Don't worry if your include path is elsewhere, just make a note of where it is since we'll be dumping the Zend stuff there. Now to extract and install Zend:

# tar xzfv ZendFramework-1.8.1-minimal.tar.gz
# mv ZendFramework-1.8.1-minimal/library/Zend /usr/local/php/lib/
# mv ZendFramework-1.8.1-minimal/bin/zf.* /usr/local/php/bin/
# ln -s /usr/local/php/bin/zf.sh /usr/bin/zf
# zf show version
Zend Framework Version: 1.8.1

I opted to put the zf.sh into my PHP bin directory and symlink it from my bin dir. The next step is to download the source from github and unpack it into your include path to a directory called php-activerecord. Our dependencies are there and we can move on to our first project.

Create Project

Now that everything is installed, we want to test out zf create project. The following example is a bare bones project ready for us to extend:

# zf create project /var/www/newproject
Creating project at /var/www/newproject
# ls /var/www/newproject/
application  library  public  tests

That is it, you are ready for the web. Configure your web server to start serving out pages from the location you set for your new project. The docroot path should be /var/www/newproject/public. Load up your web browser and you should see your page.

Loading up ActiveRecord

Inside your application/Bootstrap.php you might notice the code rather bare. The Zend Bootstrapper will load anything within the bootstrap function starting with _init*. To tack on ActiveRecord, we simply add the following:

class Bootstrap extends Zend_Application_Bootstrap_Bootstrap
{
    protected function _initActiveRecord()
    {
        require_once("php-activerecord/ActiveRecord.php");
        ActiveRecord\Config::initialize(function($cfg)
        {
            $cfg->set_model_directory(APPLICATION_PATH . '/models' );
            $cfg->set_connections(array(
                'development' => 'mysql://user:password@localhost/newproject'));
        });
    }
}

You will need to adjust your SQL connect strings, and at this point if you want to restructure your model directory differently you may do so. There are only two configuration points, both of which you could pass from an XML or INI config. Even though ActiveRecord is loaded, we still need to test out that everything is working. In order to do this we will need to create some test data:

CREATE DATABASE newproject;
USE newproject;
CREATE TABLE books(
    id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
    name varchar(50),
    author varchar(50)
);
INSERT INTO books(id,name,author) VALUES(1,'How to be Angry','Jax');

The next step is to create a model for the books table to be used from within your application. In your model directory that you set within the configuration above, create a Book.php with the following:

class Book extends ActiveRecord\Model { }

And finally to test out that everything is working, within your IndexController.php for indexAction add the following:

public function indexAction()
{
    $this->view->book = Book::find(1);
}

And so we can actually see the output, edit your /views/scripts/index/index.phtml with the following:

Today I read <?=book->name?> by <?=book->author?> and it was terrible.

As you can see, you can pull the variable set by the action assigned to the view or you can query right from the view itself if you please. If you get the following output, everything is good to go:

Today I read How to be Angry by Jax and it was terrible.

Next Steps

Going forward, there is plenty of room to extend this setup. Unfortunately, the Zend_Tool_Project abstraction documents are still being completed but there are examples out there on how to add new commands to it. One new command I am working on is the automated creation of AR models by passing a database connect string to simplify the process even further.

The algorithm simplifies a polyline by removing vertices that do not contribute (sufficiently) to the overall shape. It is a recursive process which finds the most important vertices for every given reduction. First, the most basic reduction is assumed. A single segment connecting the beginning and end of the original polyline. This is when the recursion starts, the most significant vertex (the most distant) for this segment is found and, when the distance from this vertex to the segment exceeds the reduction tolerance, the segment is split into two sub-segments, each inheriting a subset of the original vertex list. Each segment continues to subdivide until none of the vertices in the local list are further away than the tolerance value.

There is a PHP class that does just this: Douglas-Peucker Polyline Simplification in PHP by Anthony Cartmell. Based on the original quality of the data and tolerance level, I was able to achieve a 90-93% reduction in size. This reduction allows me to represent significantly more data at a reasonable performance level to clients. Keep in mind, that this reduction is removing data out of the coordinate array so the quality of your representation will go down with the tolerance and reduction being applied. I highly suggest that you play around with the tolerance until you find a good balance between data size and image quality.

pnPoly - Used to determine if a coordinate falls inside a polygon.
Centroid - Find the center of a polygon..
Area - Calculate the area of a polygon.
googleGeoCoder - Extracts GIS information from Google Maps from an address.
PolylineEncoder - Takes a set of coordinates and encodes it for Google Maps.

If you ran into the problem I did, which is that a lot of the data is coming in the form of shp/dbf files and needs to be parsed out to something friendlier either KML or CSV, there are a couple of solutions for that. You can parse out the data with shp2text if your source coordinate format is already in lat/lng or if you have different coordinate system and use ArcGIS, you can try the plugin Export to KML 2.5.3 to help with the exporting of data with the ESRI suite of products.

Once your data is in SQL, the following query is an example of distance sorting with SQL. You can grab a copy of the zip_codes database here and play around with it.

SELECT *,
sqrt((69.1 * ("37.6" - latitude)) * (69.1 * ("37.6" - latitude)) +
(53.0 * ("-77.6" - longitude)) * (53.0 * ("-77.6" - longitude)))
AS distance
FROM `zip_codes`
HAVING distance < 10
ORDER BY distance ASC

There is a whole suite of tools that are able to break down content to help improve the filter by supplementing it not only with a database of words to categories, but also sets of N-gram derived from the text. There are several scripts out there that will help with this extraction and it offers a few more layers of depth for Bayesian filtering. One such tool is, Ngram Statistics Package (NSP) which is easy to install and run.

I ran a very basic test against an older post to see how it does with bigram extraction.

# perl bin/count.pl --ngram 2 test.cnt test.txt
# perl statistic.pl --ngram 2 dice test.res test.cnt

Sample bigrams found:

cloud computing, master slave, groups online, Back Again, made absolutely, very costly, extensive development, hefty bill, start ups, distribution awareness

Rather than running a probability that the set of words above would fit into one category in this case, "Technology" we can now compound the score with the probability that those terms fall into the category as well. For another layer of scoring, trigrams can be extracted, 4-grams, etc. In the financial sector the terminology is thick and analysis will be almost impossible without N-gram extraction. "Filed for bankruptcy" and "avoided bankruptcy" could not be further apart. With traditional filtering, the word "bankruptcy" would be meaningless because it really is not an indicator as to the probability that the article is favorable or not because there is no context. In this case by extracting the phrases the filter can understand and score appropriate the difference between the two terms.

Paul Graham has been working on improving the Bayesian filter to deal with spam by splitting the data into categories. Text is classified not only as legitimate or spam based on the context of the message, but the likely hood of tokens appearing in various parts of the message. N-gram filtering in this case wouldn't work as well for spam as the amount of grammar mistakes, misspellings, and word ordering would make any benefit worthless. Spammers are adjusting their content to beat such filters all the time. When the source data is reliable, the N-gram addition to the filter will boost categorization accuracy.

Integration to traditional Bayesian filtering is very easy. Google has been using text processing for a while now. This is a huge area of study in linguistics, language processing and machine learning. With so much data out there and more being collected on a daily basis, deriving context from text will allow for applications to behave smarter.

Every developer will grow their own set of tools, but the core components are available for free. Google offers analytics to take control of your traffic performance, sources, and patterns. There is also Adwords Keyword Tool, which will help you target search phrases, volume, and competition. Based on these factors and a list of similar keywords you will be able to identify good opportunities to compete for relevant traffic. There is also the Webmaster guidelines published by Google that will give you a general best practice for search engines.

This process requires a lot of patience. It takes time for changes to take shape and results are delivered. When making changes to any site or even designing a new site with SEO built in, user traffic is not going to happen right away. Seeing the results come in will trigger an OCD to check Analytics and forever make improvements and indentify new markets and opportunities. The vast majority of web sites are there for user consumption. SEO became big business when a lot of people all at once figured out that users translated to consumers.

Google is the search leader, therefore they offer the highest return. They control the flow of traffic on the internet. Luckily, they also published a search engine optimization starter guide in pdf format! This is the 101 of SEO and it will be pointless to try to chase down every obscure reference and tip on the countless SEO sites out there when the components to their content analysis is available all in one place. The document is a general overview but offers some very important best practice rules that are easy to implement:

Title Tags

- Choose a title that effectively communicates the topic of the page's content.
- Create unique title tags for each page
- Use brief, but descriptive titles (limit of 66 characters or 12 keywords)

Description Tags

- Accurately summarize the page's content
- Use unique descriptions for each page
- Avoid filling the description with only keywords
- Avoid copy and pasting the entire content of the document into the description meta tag

URL structure

- Use words in URLs
- Create a simple directory structure
- Provide one version of a URL to reach a document
- Many users expect lower-case URLs and remember them better)

Site Navigation

- Create a naturally flowing hierarchy
- Use mostly text for navigation
- Use "breadcrumb" navigation
- Put an HTML sitemap page on your site, and use an XML Sitemap file
- Consider what happens when a user removes part of your URL
- Have a useful 404 page

Anchor Text (Links)

- Choose descriptive text
- Write concise text
- Format links so they're easy to spot

Heading Text

- There are six sizes of heading tags, beginning with <h1>, the most important, and ending with <h6>, the least important.
- Imagine you're writing an outline
- Use headings sparingly across the page
- Avoid using heading tags only for styling text and not presenting structure
- Avoid excessively using heading tags throughout the page

Other Confirmed Ranking Factors

- Keyword in URL
- Keyword in Domain name
- Freshness of Pages
- Freshness - Amount of Content Change
- Freshness of Links
- Site Age
- Anchor text of inbound link
- Hilltop Algorithm
- Domain Registration Time

There is a lot of helpful content in the document but it does not go deep into the inner mechanics like other sites attempt to do. There are several sites out there that try to go beyond what has been published and into the details for generating traffic, you would just need to google "Google Ranking Factors". A lot of information came from when google released US Patent Application #20050071741.

Use the above as a baseline of the steps to get your site more traffic. This is a topic that is constantly being updated as search improves and requires a lot of time and research to do efficiently. Overhauling existing projects to meet the standards of today's crawlers is tedious, boring, and offers no immediate results. It has been something I avoided in the past, but for a web site to stay competitive and more importantly, be seen it has to be found. I find having some good rules in place for how to deal with SEO makes new projects going forward much easier to deal with.

We intend on predicting the movement of stocks based on real time content analysis. This requires a good deal of machine learning and historical data, but even good content analysis is not enough. Using Bayesian Filtering with noise word reduction we plan on processing historical data and assigning the content to one of three categories: moveup, movedown, nomove. In order to train the filters, past press releases will be inserted into the filter mashed up with the stock data to track how the markets reacted to the context of the content. Over time, the software will be able to recognize keywords that trigger positive versus negative emotion in the market that would drive the price one way or the other. A score can be applied much like spam scores are applied and this number can be used as part of a greater overall algorithm to determine an action.

Just to bring a few readers up to speed on exactly how this will be applied, take the following formula:

Rather than training it to recognize the probability of spam we train it to recognize the probability that the word will trigger positive stock movement:

• p is the probability that the content will result in positive movement.
• p1 is the probability p(S | W1) that it is positive knowing it contains a first word (for example "capital");
• p2 is the probability p(S | W2) that it is positive knowing it contains a second word (for example "boosted");
• etc...

The entire body of the content will be processed against a known database of words and the market reaction to the presence of those words. The basic Bayesian filtering will need to be extended to deal with phrase recognition but overall a solid proven technology for machine learning to build from.

This information by itself, is nothing revolutionary but with strong pattern analysis like candlestick pattern recognition and other market indicators it can be used to create an accurate trading platform for marginal gains which over time can offer pretty high returns. There is certainly a lot of potential for this if it works, but it heavily depends on working accurately and there will be a lot of trial and error in the process.

For more reading on the concepts and components behind this idea, check out:

• Naive Bayes Classifier
• Candlestick Patterns
• Candlestick Charting
• Better Bayesian Filtering
• TD Ameritrade API

The nice part is all the historical data is out there around the internet which makes back-testing and scoring very easy to do and there will need to be a lot of testing.

With that said, we are about to surpass several sites on the way of traffic including reddit.com, fark.com, mcdonalds.com, and ibm.com to name a few. As a developer, seeing the metrics come back helps motivate and encourage the work that I've done. Even now we are still dealing with speed bumps along the way. None of which are noticeable as far as traffic is concerned but this maintained scalability is certainly a huge task. Using Drupal as a back end has proven that there are several challenges with how we proceed going forward. We've decided to scrap the MySQL Master/Master replication due to Drupal's sequences tables and duplicate key problems. An issue easily fixed if only auto increment was used... but alas without rewriting a good chunk of the code base going forward we must adapt to Master/Slave Read/Write splitting. It seems a week does not go by without encountering a scaling/replication pitfall. Drupal's general compatibility attitude torwards their framework makes it very difficult to leverage any perticular technology like MySQL to it's maximum because the database layer is written with several database backends in mind. A word of caution going for other developers that when they plan on creating a high traffic web site, there is a point where an up front investment in the infrastructure and backend will pay off huge. I believe we're reaching that point.

The unfortunate part with rapid growth is if the team is capable of adjusting at the same pace. While there is only but so much that can be planned ahead, now more than ever it is important that issues are indetified long before the become customer facing because the stakes are so much higher. Despite a successful launch, there is still a lot more ahead. How much time do we invest into new features, maintenance, and re-writes? What takes a higher priority, growth or consumer experience? Do we have the resources to invest in research and development?

At the end of every milestone, I find it necessary everyone pats themselves on the back, take deep breath, regroup as a team, and the cycle begins all over again. The gaps in between the end of one project and a start of another is the most important time for management and development to be in step with each other so everyone can move forward rowing in the same direction. Revisit company values, mission statements, and have meaningfull follow up discussions on what went well and what didn't. If as a team there is no time allocated for dialogue, despite accomplishing the task at hand, the same problems will occur over and over again. Not all problems in development are technical-- process and communication are consistent issues that seems to always manifest one way or another when working in a collaborative enviroment and it's important to determine what works well in the current situation. What may have worked in the past on a project, at a previous job, or for one person might not work now.

Congratulations on a job well done, let's open the dialogue and relish in the reflection time... that went well, what now?

I realized that I was over-thinking the problem so I did a little test to just see how much overhead Drupal is adding. I can execute ten times the number of queries natively through PHP and mysql_connect than if I were to execute those queries from within Drupal with just bootstrap included. Apache answered requests and processed these several million round trips to the database much faster. I understand that as a framework there is a lot being loaded, but a whole decimal point off when dealing with requests/sec is a huge overhead.

The better part of the last few days was spent gutting the inner workings of Drupal and removing as much of it as possible to lover overhead, reduce response time, and hopefully let us scale a little further on the hardware that we have. As it is, traffic is growing a steady 40-50% month over month for a while now and we are already dealing with several million unique visitors a month. That number is growing rapidly with no ceiling in the near future and it's unfortunate that at this point the only way to make this work is to break the upgrade path of Drupal.

Going through the code base, it's easy to see where so much of the bloat comes from. With comments like: "TODO: remove this when we require at least PHP 4.4.0" there is a lot of backwards compatibility that is required to support such a large community. I am seeing more and more Frameworks replace legacy CMS systems that provide a bare bone set of tools. The agility, performance, and maintenance of a ground up site using frameworks like Symfony or Rails leads to longer development cycles and increased costs but there is a point where you stop putting a round peg in a square hole and realize that the cost of maintenance and development down the road will far exceed the up front investment.

I'm glad Drupal has been able to take us this far and I'm very grateful for the community behind it. It's a pleasant surprise that everything is still working and hopefully we can get a few more miles out of it before we are ready to do the next iteration of the site.

/*
SQL script to grab the worst performing indexes
in the whole server
*/
SELECT
t.TABLE_SCHEMA AS `db`
, t.TABLE_NAME AS `table`
, s.INDEX_NAME AS `inde name`
, s.COLUMN_NAME AS `field name`
, s.SEQ_IN_INDEX `seq in index`
, s2.max_columns AS `# cols`
, s.CARDINALITY AS `card`
, t.TABLE_ROWS AS `est rows`
, ROUND(((s.CARDINALITY / IFNULL(t.TABLE_ROWS, 0.01)) * 100), 2) AS `sel %`
FROM INFORMATION_SCHEMA.STATISTICS s
INNER JOIN INFORMATION_SCHEMA.TABLES t
ON s.TABLE_SCHEMA = t.TABLE_SCHEMA
AND s.TABLE_NAME = t.TABLE_NAME
INNER JOIN (
SELECT
TABLE_SCHEMA
, TABLE_NAME
, INDEX_NAME
, MAX(SEQ_IN_INDEX) AS max_columns
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA != 'mysql'
GROUP BY TABLE_SCHEMA, TABLE_NAME, INDEX_NAME
) AS s2
ON s.TABLE_SCHEMA = s2.TABLE_SCHEMA
AND s.TABLE_NAME = s2.TABLE_NAME
AND s.INDEX_NAME = s2.INDEX_NAME
WHERE t.TABLE_SCHEMA != 'mysql'                         /* Filter out the mysql system DB */
AND t.TABLE_ROWS > 10                                   /* Only tables with some rows */
AND s.CARDINALITY IS NOT NULL                           /* Need at least one non-NULL value in the field */
AND (s.CARDINALITY / IFNULL(t.TABLE_ROWS, 0.01)) < 1.00 /* Selectivity < 1.0 b/c unique indexes are perfect anyway */
ORDER BY `sel %`, s.TABLE_SCHEMA, s.TABLE_NAME          /* Switch to `sel %` DESC for best non-unique indexes */
LIMIT 10;

To audit just one database if you are running on a server with several different databases, just adjust the where clause to WHERE t.TABLE_SCHEMA = 'mytable'. This would have been very useful when working with the cluster to recover memory/space from indexes that aren't being used and to optimize queries to hit indexes that are meaningful.

1. Download the latest version at http://dev.mysql.com/downloads/mysql-proxy/.
2. yum install glib2-devel.x86_64 ncurses-devel.x86_64 libevent-devel.x86_64 mysql-devel.x86_64
3. ./configure -without-lua (for a straight proxy, you will not need LUA and this will save you having to install additional dependencies).
4. make && make install

Easy enough to get it installed in the system, but I noticed that the default parameters and as with most software is rather wide open so be sure to lock down the ip addresses you are listening on.

mysql-proxy --proxy-backend-addresses=192.168.1.101:3306 --proxy-backend-addresses=192.168.1.102:3306 --admin-address=127.0.0.1:4041 --proxy-address=127.0.0.1:4040 --proxy-skip-profiling --daemon

The admin address and proxy address will be accessible by anyone hitting your server unless you lock it down to your local host. Now configure your web servers to use localhost:4040 to connect to MySQL and all should be working fine. You can find additional configuration parameters, admin commands, and LUA scripts at http://forge.mysql.com/wiki/MySQL_Proxy.

• Price- An m1.xlarge instance will run you ~$600 with data transfer costs. Managed hosting solutions run cheaper especially if you plan on purchasing in bulk. The grid is designed for on-demand computation and not as a cost efficient web services.
• Configuration- There are a limited number of options and you will not be able to tailor the hardware to your application. Databases over 10GB of size will have performance issues since that is the memory cap.
• Network storage- The primary disks offers limited storage, additional volumes will need to be attached across the network and at an additional cost.
• Software - No hardware based solutions for load balancing or custom application servers. The model is software driven so all needs must be met with a software solution. In a managed hosting or collocation solution you will at least have the option of adding additional hardware and having a private network. No dedicated switching, routers, firewall, or load balancers.

EC2 might be right for you if:

• Distribution Awareness- Your application was designed to scale horizontally from the get-go and you can take advantage of grid computing.
• Research and Development- EC2 & Rightscale will allow for you to bring up new servers, test configuration, and scale quickly. If you are not sure what your hardware demands will be or the scope of the project it will allow for some flexibility to get this right before committing to rather lengthy contracts with other hosting options.
• Disaster Recovery- If you need an off-site mirror for your site that you can keep dormant and activate as needed.

Over the past several months I have been doing extensive development using Amazon's EC2 as my hardware infrastructure. I was tasked with taking CitySquares.com from a New England area hyper-local search and business directory to a national site in a few months. Due to the memory limitations of EC2 instances, m1.xlarge only providing 15GB the jump from a 15GB database to anything larger becomes very costly. When everything was able to be contained in two servers in a master/slave environment we were able to provide redundancy, performance, and easy management when working with the database. The final estimations of the national roll out would put our core data at 50GB. Far too large for any one EC2 instance. Going to disk was not an option as everything works off of EBS attached storage and that any disk writes means traveling over the network. An additional overhead which degrades performance even more when switching off RAM. Then, there was also the nature of the data, which means that at any page load, any piece of data could be requested.

With OS overhead, index storage, and ndb overhead, each x1.mlarge instance gave about 12GB of usable storage. Include replication, and it comes down to 6GB of storage per node. To store a ~50GB database that I had planned on requires 8 storage nodes, two management servers, and two mysqld api servers. This is where it became important to understand the advantages of vertical scaling versus horizontal scaling. EC2 provided fast horizontal scaling and configuration. Servers can be launched on demand and their configurations scripted. While I appreciated that aspect of cloud computing and being able to bring that many servers up and configure each one relatively quickly I really just needed two decent boxes with 64GB of RAM in each and a master/slave setup. The operations costs for the cluster was $6000/month, a hefty bill considering I could buy all the hardware needed to run the cluster in just a few months of paying for EC2.

Twelve servers later and a working cluster we were able to successfully roll out our MySQL Cluster with minimal performance loss. A lot of it was due to tweaking every query top to bottom. The site runs on a Drupal core, which meant a lot of the queries were not designed with distribution awareness both from code in house and the core. This was another added growing pain since the network overhead of running 12 servers on shared resources, with mediocre latency, and throughput limitations was amplifying flaws in the database design and every poorly designed query and join degraded performance significantly.

To give credit where credit is due, EC2 did allow us to scale the site up rather quickly. We were able to test server configurations, new applications, and have easy management. It would not have been possible for us to push out the data, handle influx of new traffic, and expand as fast as we had without it. Long term however, it made absolutely no sense that once we were finished scaling up, to stay on EC2. It is a great platform for start-ups to be able to configure and launch servers for their service or application and grow rapidly.

As a database platform, until EC2 offers more configuration options in it's hardware and the ability to increase memory, the cap of 15GB will make EC2 problematic for any database that plans on growing past that. It is important to understand your application and database needs before considering EC2. It is no surprise to me that even with the RightScale interface, and easy management of EC2 web sites are reluctant to switch off their own hardware or managed hosting.

Both Sun and Continuent are pursuing MySQL Clustering on cloud computing. As of this post, Continuent for EC2 was still in closed beta and testing and Sun is doing their own research into offering more support for database clusters on compute clouds. Maybe in the future this can be something to revisit, but it would require more ndb configuration options on the network layer to cope with shared bandwidth and additional hardware configurations by Amazon (someday).

The 6.4 release which is in beta now offers new features which would make MySQL Clustering more attractive on cloud computing architecture:

• Ability to add nodes and node groups online. This will allow the database to scale up without taking the cluster down.
• Data node multithreading support. The m1.xlarge instance comes with 4 cores.

If you are setting up a MySQL cluster, the following resources will help get you up and running quickly:

• http://www.severalnines.com/
• http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-configuration.html