<?php
class My_Paginator implements Zend_Paginator_Adapter_Interface {
	public function __construct($table,$conditions = array())
	{
		if(!is_array($conditions))
 
			$conditions = array( $conditions );
 
		$this->conditions = $conditions;
 
		$this->table	  = new $table;
 
	}
 
	public function getItems($offset, $itemCountPerPage)
	{
			return $this->table->find('all', array('limit' => $itemCountPerPage, 'offset' => $offset, 'conditions' => $this->conditions));
	}
 
	public function count()
	{
		return $this->table->count(array('conditions' => $this->conditions));
	}
}
?>
 

The two methods the Zend_Paginator_Adapter_Interface expects are count() and getItems(). The above example is a little "raw", it should serve to guide you in what to do when extending the Paginator with it's own adapter regardless of what your database layer is. In the case of $conditions, these are the parameters you are passing to SQL:

 
$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
 

We want to access the User model, and only want to pull out users who are active. Easy enough, you can certainly put in more complicated SQL here, but for a general use purpose it solves 99% of what I want with ActiveRecord and paging, pass in a model to the adapter, and some basic conditions for listing.

 
    public function pageUserAction()
    {
 		$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
		$paginator->setCurrentPageNumber($this->_getParam('page', 1));
		$paginator->setItemCountPerPage('25');
		$this->view->paginator = $paginator;
    }
 

This is a basic usage of Zend_Paginator, you are passing in the current page you are on, and the results per page and pushing it out to the view, and on the view side:

 
<div id="userlist">
<?php if (count($this->paginator)){
	foreach ($this->paginator as $user){
		echo $user->username."<br>";
	}
}
?>
</div>
 
<?= $this->paginationControl($this->paginator, 'Elastic', '/common/paginator.phtml'); ?>
 

If anyone has any problems or questions getting the two to work together let me know in the comments and I will do my best to answer your questions.

There is a whole suite of tools that are able to break down content to help improve the filter by supplementing it not only with a database of words to categories, but also sets of N-gram derived from the text. There are several scripts out there that will help with this extraction and it offers a few more layers of depth for Bayesian filtering. One such tool is, Ngram Statistics Package (NSP) which is easy to install and run.

I ran a very basic test against an older post to see how it does with bigram extraction.

# perl bin/count.pl --ngram 2 test.cnt test.txt
# perl statistic.pl --ngram 2 dice test.res test.cnt

Sample bigrams found:

cloud computing, master slave, groups online, Back Again, made absolutely, very costly, extensive development, hefty bill, start ups, distribution awareness

Rather than running a probability that the set of words above would fit into one category in this case, "Technology" we can now compound the score with the probability that those terms fall into the category as well. For another layer of scoring, trigrams can be extracted, 4-grams, etc. In the financial sector the terminology is thick and analysis will be almost impossible without N-gram extraction. "Filed for bankruptcy" and "avoided bankruptcy" could not be further apart. With traditional filtering, the word "bankruptcy" would be meaningless because it really is not an indicator as to the probability that the article is favorable or not because there is no context. In this case by extracting the phrases the filter can understand and score appropriate the difference between the two terms.

Paul Graham has been working on improving the Bayesian filter to deal with spam by splitting the data into categories. Text is classified not only as legitimate or spam based on the context of the message, but the likely hood of tokens appearing in various parts of the message. N-gram filtering in this case wouldn't work as well for spam as the amount of grammar mistakes, misspellings, and word ordering would make any benefit worthless. Spammers are adjusting their content to beat such filters all the time. When the source data is reliable, the N-gram addition to the filter will boost categorization accuracy.

Integration to traditional Bayesian filtering is very easy. Google has been using text processing for a while now. This is a huge area of study in linguistics, language processing and machine learning. With so much data out there and more being collected on a daily basis, deriving context from text will allow for applications to behave smarter.

We intend on predicting the movement of stocks based on real time content analysis. This requires a good deal of machine learning and historical data, but even good content analysis is not enough. Using Bayesian Filtering with noise word reduction we plan on processing historical data and assigning the content to one of three categories: moveup, movedown, nomove. In order to train the filters, past press releases will be inserted into the filter mashed up with the stock data to track how the markets reacted to the context of the content. Over time, the software will be able to recognize keywords that trigger positive versus negative emotion in the market that would drive the price one way or the other. A score can be applied much like spam scores are applied and this number can be used as part of a greater overall algorithm to determine an action.

Just to bring a few readers up to speed on exactly how this will be applied, take the following formula:

Rather than training it to recognize the probability of spam we train it to recognize the probability that the word will trigger positive stock movement:

• p is the probability that the content will result in positive movement.
• p1 is the probability p(S | W1) that it is positive knowing it contains a first word (for example "capital");
• p2 is the probability p(S | W2) that it is positive knowing it contains a second word (for example "boosted");
• etc...

The entire body of the content will be processed against a known database of words and the market reaction to the presence of those words. The basic Bayesian filtering will need to be extended to deal with phrase recognition but overall a solid proven technology for machine learning to build from.

This information by itself, is nothing revolutionary but with strong pattern analysis like candlestick pattern recognition and other market indicators it can be used to create an accurate trading platform for marginal gains which over time can offer pretty high returns. There is certainly a lot of potential for this if it works, but it heavily depends on working accurately and there will be a lot of trial and error in the process.

For more reading on the concepts and components behind this idea, check out:

• Naive Bayes Classifier
• Candlestick Patterns
• Candlestick Charting
• Better Bayesian Filtering
• TD Ameritrade API

The nice part is all the historical data is out there around the internet which makes back-testing and scoring very easy to do and there will need to be a lot of testing.

I've been asked this and similar questions quite a bit lately. But before I delve into the answer to this I want to lay the foundation and ask you a question. This one question should play a large part in your final assessment to go with EC2 or not. The question you should ask yourself is:

How quickly do you actually need to scale either up or down?

The answer to this will likely influence the correct solution to your problems. The following bullet point list is how I classify levels of scalability, each one comes with its own pros and cons but generally the quicker you need something the more expensive it is going to be.

• Immediate - within minutes - EC2 or other cloud computing networks
• Fast - within days to a week - Managed Hosting, Rackspace, The Planet, etc
• Average - within weeks to a month - Own your own hardware, Dell, HP, IBM, etc
• Corporate - within months/years - Good Luck

With this in mind, everyone hears the hype of EC2, with its scalability, fully managed hardware and virtualization but there really aren't that many people out there describing their experiences with it. When we made the decision to go with EC2 we did our research and due diligence before making the switch. There wasn't much to go on but the few articles and blog posts we did read were all positive. I guess we all got caught up in the hype here as well.

Even after all our research it turns out that going with EC2 was one of the poorer IT decisions we have made. EC2 has turned out to be more expensive, more difficult to implement and with poorer performance than we had ever expected even with our worst case estimations. To top it all off, we didn't fully utilize the benefits of going with EC2 which was immediate scalability. Our traffic is relatively predictable and grows or shrinks in manageable percentages and can be scaled up within days instead of minutes. We never have any massive spikes in our traffic either up or down. Even if we did have spikes we are limited by our MySQL cluster.

While we had to rethink a lot of our architecture to create a more horizontal platform instead of the traditional vertical scaling, MySQL was by far our biggest bottleneck. The source of the problem is rooted in Amazon's preset machine size. While they have done an adequate job of offering different types of instances with more memory in one line and more computational power in the other you are still limited to what they are offering. With the large database we have and the latencies between the instances and their permanent storage we were forced to keep as much of our database cached in RAM. Now this shouldn't have been too big a deal. Just get a machine with a ton of RAM. Well, unfortunately Amazon's biggest instance only offered us a maximum of 15GB. Needless to say this was not sufficient and forced us to adopt a cluster solution. This in and of itself is not ideal especially when you should be able to run off a single box with 32GB of RAM and access to fast local disks. However, it took us twelve (12) m1.xlarge instances to reach the level of performance and availability we desired. Not to mention the network IO latency between node and disk storage and node to node adding insult to injury.

While the speed and size of the cluster was not desirable, it worked. However, we had to completely forfeit any sort of scalability to achieve a working database. To my knowledge there is no way to quickly and easily boot up more instances of MySQL to supplement a live cluster. In order for us to add more capacity we would have to perform a rolling reboot of every machine in the cluster. Its unfortunate that databases were not designed with EC2 in mind.

However, there are companies who are trying to tap into this pain point. We were looking very intently at a company called Continuent who produces a MySQL cluster monitoring and management tool. Unfortunately, as of Jan 2009 the product was still in private beta and was unavailable to us. This tool would have allowed us to add nodes to the cluster on the fly without having to take it down in the process. Although, even then with this extra tool, which wasn't cheap, you still couldn't scale down the cluster without taking it off-line. As far as I am concerned, if you are already using the largest instance available to you (an m1.xlarge or c1.xlarge), there is no way to vertically scale up a database with EC2. Instead you are forced into a less than ideal environment for hosting a horizontal architecture which could have serious consequences for your code base and SQL queries.

To be honest, EC2 offers a lot of benefits that are hard to come by with other solutions. EC2 is great for companies doing lots of non-real-time activities such as batch and queued processing. Companies who have a small database that can be cached in RAM and replicated easily will also benefit from EC2, just boot up a bunch of instances and go to town. However, the bottom line is if you have fairly consistent usage patterns and your applications are performance sensitive then there are much faster and more cost effective ways of abstracting your hardware requirements. We at citysquares are in the process of moving off of EC2 and onto a managed hosting platform. We still enjoy the benefits of leased hardware like we had with EC2 and the ability to quickly add new hardware. Granted, more servers aren't available to us at the drop of a hat but a couple days lead time to get another box up and running is more than sufficient for us. Not only that but we also have a whole team of IT people working with us to help alleviate our burden of supporting the entire hardware/software stack. We can now focus on what we do best which is our application.

Keep in mind that there is no concrete answer as to whether EC2 or cloud computing in general will work for you or not. You need to determine if the capacity and latencies of the pre-determined instance sizes will meet your growing infrastructure needs. For us the bitter answer was a resounding no. We were able to spec out a solution in a fully managed hosting environment for about half the monthly cost of EC2 while increasing the performance of our application significantly.

So, is Amazon's EC2 right for you?

]]> http://www.derivante.com/2009/01/26/is-amazons-ec2-right-for-you/feed/ 4