I present to you the Extensible PHP Caching Library (Hosted at GitHub) - This collection of classes makes it easy to customize key or column names to your needs as well as switch from one data store to another. This extensibility is baked into the core of the library via an abstract class. This abstract Cache class takes an associative array of keys/columns and determines how to use those keys based on the type of cache back end you are using. For example: Suppose you are using a RDBMS such as MySQL. In this case the associative array will be parsed and the query built such that the key is the column name and the value is what you want to query on. However, if you chose to use a NoSQL key/value store such as Memcache then the associative array is sorted and imploded to create a single string as the key.

Since we always specify the key as an associative array, switching between different data stores is as simple as changing the Class name from say MCache to SQLCache in your code. Nothing else is required to change data stores. Keys, expiration dates, and data processing all stay the same and all functions are called with the same arguments. This functionality is attributed to the abstract base Cache class which guides and regulates the inheriting classes.

Lets get to some examples:

In the following example we create a new SQLCache object and pass it our associative array of keys and values. Check if there is cached data, if there isnt then do something to generate the data and then cache it. Notice how you dont have to pass the key in again when setting the cache. The key is stored within the object so you can get and set as many times as you need without having to set the key every time. Lastly we delete the cache.

// Get Cache
$cache = new SQLCache(array('column1' => 123, 'column2' => 'blah', … ));
$output = $cache->getCache();
 
if(!$output) {
  // Do something to generate data
  $output = 'some datas';
  // Set Cache
  if($cacheNow) {
    // Force a write to cache now
    $cache->setCache($output, '+1 day', true);
  } else {
    // Setting no time defaults to time() + 86400, one day from now.
    $cache->setCache($output);
  }
}
 
print $output;
 
// Delete the cache
$cache->deleteCache();
 

If ever you needed to update this caching strategy to include Memcached the only change would be to change SQLCache to Mcache:

$cache = new SQLCache(array('column1' => 123, 'column2' => 'blah', … ));
to
$cache = new MCache(array('key1' => 123, 'key2' => 'blah', … ));
 

The column and key names are arbitrary and may be set to anything you want to name it. For SQL caches make sure you create a cache table that has the corresponding column names and that they are indexed optimally.

While I have only created classes for SQL (MySQL - since there is a LIMIT 1), Memcached and File based caching, the base class can be extended to include any key/value store or any database with columns (MongoDB, CouchDB, Tokyo, Postgress, Oracle, etc). Just update the back end calls and you are good to go.

For anyone who updates or adds functionality please let me know so I can give credit where credit is due. This library is available under the LGPLv3.

 
<?php
class My_Paginator implements Zend_Paginator_Adapter_Interface {
	public function __construct($table,$conditions = array())
	{
		if(!is_array($conditions))
 
			$conditions = array( $conditions );
 
		$this->conditions = $conditions;
 
		$this->table	  = new $table;
 
	}
 
	public function getItems($offset, $itemCountPerPage)
	{
			return $this->table->find('all', array('limit' => $itemCountPerPage, 'offset' => $offset, 'conditions' => $this->conditions));
	}
 
	public function count()
	{
		return $this->table->count(array('conditions' => $this->conditions));
	}
}
?>
 

The two methods the Zend_Paginator_Adapter_Interface expects are count() and getItems(). The above example is a little "raw", it should serve to guide you in what to do when extending the Paginator with it's own adapter regardless of what your database layer is. In the case of $conditions, these are the parameters you are passing to SQL:

 
$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
 

We want to access the User model, and only want to pull out users who are active. Easy enough, you can certainly put in more complicated SQL here, but for a general use purpose it solves 99% of what I want with ActiveRecord and paging, pass in a model to the adapter, and some basic conditions for listing.

 
    public function pageUserAction()
    {
 		$paginator = new Zend_Paginator(new My_Paginator('User',' active = "Y" '));
		$paginator->setCurrentPageNumber($this->_getParam('page', 1));
		$paginator->setItemCountPerPage('25');
		$this->view->paginator = $paginator;
    }
 

This is a basic usage of Zend_Paginator, you are passing in the current page you are on, and the results per page and pushing it out to the view, and on the view side:

 
<div id="userlist">
<?php if (count($this->paginator)){
	foreach ($this->paginator as $user){
		echo $user->username."<br>";
	}
}
?>
</div>
 
<?= $this->paginationControl($this->paginator, 'Elastic', '/common/paginator.phtml'); ?>
 

If anyone has any problems or questions getting the two to work together let me know in the comments and I will do my best to answer your questions.

class Rating
{
  public static function ratingAverage($positive, $total, $power = '0.05')
  {
    if ($total == 0)
      return 0;
 
    $z = Rating::pnormaldist(1-$power/2,0,1);
    $p = 1.0 * $positive / $total;
    $s = ($p + $z*$z/(2*$total) - $z * sqrt(($p*(1-$p)+$z*$z/(4*$total))/$total))/(1+$z*$z/$total);
    return $s;
  } 
 
  public static function pnormaldist($qn)
  {
    $b = array(
      1.570796288, 0.03706987906, -0.8364353589e-3,
      -0.2250947176e-3, 0.6841218299e-5, 0.5824238515e-5,
      -0.104527497e-5, 0.8360937017e-7, -0.3231081277e-8,
      0.3657763036e-10, 0.6936233982e-12);
 
    if ($qn < 0.0 || 1.0 < $qn)
      return 0.0;
 
    if ($qn == 0.5)
      return 0.0;
 
    $w1 = $qn;
 
    if ($qn > 0.5)
      $w1 = 1.0 - $w1;
 
    $w3 = - log(4.0 * $w1 * (1.0 - $w1));
    $w1 = $b[0];
 
    for ($i = 1;$i <= 10; $i++)
      $w1 += $b[$i] * pow($w3,$i);
 
    if ($qn > 0.5)
      return sqrt($w1 * $w3);
 
    return - sqrt($w1 * $w3);
  }
}
 

The function takes 3 parameters: the positive votes, total votes, and the power. The power can be adjusted, 0.10 to have a 95% chance that your lower bound is correct, 0.05 to have a 97.5% chance, etc.  Sample usage:

sample(1,0);
sample(100,50);
sample(250,100);
sample(1000,500);
 
function sample($p,$n)
{
  echo Rating::ratingAverage($p,$p+$n);
}
 

Output:

When dealing with sites like Reddit, Digg, and the like you have a certain "freshness" element. The above solution might be a working model for the entire span of the site, but for that front page element you will need to implement some form of "gravity". This can be done by taking the raw score and decaying it over time, like so:

 
class Rating
{
  ...
  public static function gravityRating($positive, $total, $time, $power = '0.05')
  {
    if ($total == 0)
      return 0;
    return (Rating::ratingAverage($positive, $total, $power) / pow($time,0.5));
  }
  ...
}
 
sample(100,50,'0.5');
sample(100,50,'1');
sample(100,50,'4');
sample(100,50,'8');
sample(100,50,'24');
 
function sample($p,$n,$time)
{
  echo Rating::gravityRating($p,$p+$n,$time)."\n";
}
 

In the example above, $time represents the age (in hours) and you can see the decay in the output:

0.83141271310867
0.58789756740385
0.29394878370192
0.20785317827717
0.12000408843024

My recommendation would be to "cap" the time to stop decay after a fixed period such as 12 or 24 hours to stop the initial boost of fresh content and let it normalize quickly. The rate of decay of course, can be adjusted as fast or as slow as you want and again the individual weighting you want to apply will vary from site to site. Depending on the volatility of your content, a front page "freshness" that will encompass a week would not merit a 12 hour decay, but rather a week long decay. Hopefully the above code is enough to get started with content rating and making better use of user feedback and can help lead web masters to making a more intelligent calculation of their content beyond the traditional "5-Star Rating".

[ c9maji2tvz ]

API Features

• Supports CURL and fopen HTTP access methods, with autodetection
• PHP arrays for Google Analytics metrics and dimensions
• Account data object mapping - get methods for parameters
• Report data object mapping - get methods for metrics and parameters
• Easy filtering, use a GAPI query language for Google Analytics filters
• Full PHP5 Object Oriented code, ready for use in your PHP application

Google Quotas

• The quota applies to a single web property
• 10,000 requests per 24 hours
• 100 requests in any given 10-second period
• A query is also limited to pagination limits of 10,000 entries per feed, with a default response of 1,000 entries

While the broad set of data they have is extremely useful to pull down to do your own processing on, I find the custom visitor segments with the API a lot more interesting.    If you have a user based site with registration and collect information on your users, you can pull this data back out to determine a lot of information about your user.   A lot of tools are already built into Analytics for reporting but the ability to mash this data up to provide custom reporting for your users (customers) without giving them access to analytics is key to delivering metrics and performance

However, even with this speed improvement you still should consider how you structure your queries. There is no need to do a query across every field if you know you really want to filter everything down with a single filter query. Try moving that filter query (fq) into the actual query (q) as :. You might be surprised by the results...

It is precisely this overhead that needs to be eliminated. Time is money and if you are working with a new application on a weekly or monthly basis the scaffolding overhead comprises a great deal of that time. Not to mention that most web applications today are running to and fro from a database. This time spent fetching and storing data makes up for a significant chunk in the overall development. 2009 is a big step forward for PHP and there are tools out there that tackle this problem.

Enter ZendFramework 1.8, just recently released it represents Zend's commitment to make PHP a competitive environment for Rapid Application Development (totally RAD). There are several new tools included within this release, primarily Zend_Tool, Zend_Application, and Zend_Navigation. Zend_Tool_Project allows for the automated creation of new projects complete with a ready to go application shell. You can liken this shell to a new Hello World! project with the core structure and scaffolding setup. Rather than going into the details of everything included within the release, you can find a thorough overview within the 1.8 release notes.

This brings me to what I'm really excited about. Recently we pushed out PHP ActiveRecord for everyone! This tutorial goes over the steps necessary to build your environment. You'll be able to create new projects within the ZendFramework with PHP ActiveRecord. I believe developers should spend their time dealing with application logic and not be burdened with configuration, bootstrapping or custom SQL. For this tutorial I am using the following environment:

- Ubuntu 9.04
- PHP 5.3-RC1
- MySQL 5.4
- Apache 2.2.11
- ZendFramework 1.8 min
- php-activerecord 0.9

For the sake of keeping this tutorial short and concise, there are already several resources out there that deal with compiling PHP 5.3. I'll assume you've already setup your server with MySQL, PHP and Apache ready to go. In fact, you can use nginx if you prefer going down that route or even ZendServer CE, although I personally prefer Apache. It certainly doesn't matter what you're serving out PHP with and it will probably be dependent on what you're existing environment is using.

Installing ZendFramework 1.8 and ActiveRecord

We're going to want both libraries in our PHP include directory. First you should download the latest ZF release, 1.8.1 at the time of this post. When you have it ready to go on your server it is important to make sure your environment is working. If you installed PHP 5.3 correctly you should be able to get the following output from CLI:

# php -v
PHP 5.3.0RC1 (cli) (built: May  3 2009 15:31:21)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2009 Zend Technologies
# php -i | grep include_path
include_path => .:/usr/local/php/lib => .:/usr/local/php/lib

Don't worry if your include path is elsewhere, just make a note of where it is since we'll be dumping the Zend stuff there. Now to extract and install Zend:

# tar xzfv ZendFramework-1.8.1-minimal.tar.gz
# mv ZendFramework-1.8.1-minimal/library/Zend /usr/local/php/lib/
# mv ZendFramework-1.8.1-minimal/bin/zf.* /usr/local/php/bin/
# ln -s /usr/local/php/bin/zf.sh /usr/bin/zf
# zf show version
Zend Framework Version: 1.8.1

I opted to put the zf.sh into my PHP bin directory and symlink it from my bin dir. The next step is to download the source from github and unpack it into your include path to a directory called php-activerecord. Our dependencies are there and we can move on to our first project.

Create Project

Now that everything is installed, we want to test out zf create project. The following example is a bare bones project ready for us to extend:

# zf create project /var/www/newproject
Creating project at /var/www/newproject
# ls /var/www/newproject/
application  library  public  tests

That is it, you are ready for the web. Configure your web server to start serving out pages from the location you set for your new project. The docroot path should be /var/www/newproject/public. Load up your web browser and you should see your page.

Loading up ActiveRecord

Inside your application/Bootstrap.php you might notice the code rather bare. The Zend Bootstrapper will load anything within the bootstrap function starting with _init*. To tack on ActiveRecord, we simply add the following:

class Bootstrap extends Zend_Application_Bootstrap_Bootstrap
{
    protected function _initActiveRecord()
    {
        require_once("php-activerecord/ActiveRecord.php");
        ActiveRecord\Config::initialize(function($cfg)
        {
            $cfg->set_model_directory(APPLICATION_PATH . '/models' );
            $cfg->set_connections(array(
                'development' => 'mysql://user:password@localhost/newproject'));
        });
    }
}

You will need to adjust your SQL connect strings, and at this point if you want to restructure your model directory differently you may do so. There are only two configuration points, both of which you could pass from an XML or INI config. Even though ActiveRecord is loaded, we still need to test out that everything is working. In order to do this we will need to create some test data:

CREATE DATABASE newproject;
USE newproject;
CREATE TABLE books(
    id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
    name varchar(50),
    author varchar(50)
);
INSERT INTO books(id,name,author) VALUES(1,'How to be Angry','Jax');

The next step is to create a model for the books table to be used from within your application. In your model directory that you set within the configuration above, create a Book.php with the following:

class Book extends ActiveRecord\Model { }

And finally to test out that everything is working, within your IndexController.php for indexAction add the following:

public function indexAction()
{
    $this->view->book = Book::find(1);
}

And so we can actually see the output, edit your /views/scripts/index/index.phtml with the following:

Today I read <?=book->name?> by <?=book->author?> and it was terrible.

As you can see, you can pull the variable set by the action assigned to the view or you can query right from the view itself if you please. If you get the following output, everything is good to go:

Today I read How to be Angry by Jax and it was terrible.

Next Steps

Going forward, there is plenty of room to extend this setup. Unfortunately, the Zend_Tool_Project abstraction documents are still being completed but there are examples out there on how to add new commands to it. One new command I am working on is the automated creation of AR models by passing a database connect string to simplify the process even further.

Quick Start

We'll start first with a bare bones example to show how little you need to get up and running. There's very little to configure. We've adhered to the convention over configuration philosophy so there are no code generators you need to run and no xml/yaml mapping files to maintain.

<?
// make sure the ActiveRecord project is in your current directory
// or your include_path
require_once 'ActiveRecord/ActiveRecord.php';
 
// assumes a table name of "books" with a primay key named "id"
class Book extends ActiveRecord\Model { }
 
// initialize ActiveRecord
ActiveRecord\Config::initialize(function($cfg)
{
    $cfg->set_model_directory('.');
    $cfg->set_connections(array(
        'development' => 'mysql://username:password@localhost/database_name'));
});
 
print_r(Book::first()->attributes());
?>

That's it! The code for this is located in the examples/simple directory. You can run it with "php examples/simple/simple.php". Make sure to modify the connection string to suit your system and run the simple.sql script appropriately.

Serious Business Time

Now for a not so completely trivial example. This example will simulate a very dumb ordering and payment model. See the examples/orders directory for the source. You can run this sample by executing "php examples/orders/orders.php". Again, make sure you modify the connection string to suit your system and be sure the schema in orders.sql has been created in your test database.

First, let's look at the models:

class Person extends ActiveRecord\Model
{
    // a person can have many orders and payments
    static $has_many = array(
        array('orders'),
        array('payments'));
 
    // must have a name and a state which has a custom error message
    static $validates_presence_of = array(
        array('name'), array('state', 'message' => 'Where do you live then?'));
}

class Order extends ActiveRecord\Model
{
    // order belongs to a person
    static $belongs_to = array(
        array('person'),
        array('order'));
 
    // order can have many payments by many people
    // the conditions is just there as an example as it makes no logical sense
    static $has_many = array(
        array('payments'),
        array('people',
            'through'    => 'payments',
            'select'     => 'people.*, payments.amount',
            'conditions' => 'payments.amount < 200'));
 
    // order must have a price and tax > 0
    static $validates_numericality_of = array(
        array('price', 'greater_than' => 0),
        array('tax',   'greater_than' => 0));
 
    // setup a callback to automatically apply a tax
    static $before_validation_on_create = array('apply_tax');
 
    public function apply_tax()
    {
        if ($this->person->state == 'VA')
            $tax = 0.045;
        elseif ($this->person->state == 'CA')
            $tax = 0.10;
        else
            $tax = 0.02;
 
        $this->tax = $this->price * $tax;
    }
}

class Payment extends ActiveRecord\Model
{
    // payment belongs to a person
    static $belongs_to = array(
        array('person'));
}

And here's the code that does everything:

<?
require_once dirname(__FILE__) . '/../../ActiveRecord.php';
 
// initialize ActiveRecord
ActiveRecord\Config::initialize(function($cfg)
{
    $cfg->set_model_directory(dirname(__FILE__) . '/models');
    $cfg->set_connections(array(
        'development' => 'mysql://test:test@127.0.0.1/orders_test'));
 
    // you can change the default connection with the below
    //$cfg->set_default_connection('production');
});
 
// create some people
$jax = new Person(array('name' => 'Jax', 'state' => 'CA'));
$jax->save();
 
// compact way to create and save a model
$tito = Person::create(array('name' => 'Tito', 'state' => 'VA'));
 
// place orders. tax is automatically applied in a callback
// create_orders will automatically place the created model into $tito->orders
// even if it failed validation
$pokemon = $tito->create_orders(array('item_name' => 'Live Pokemon', 'price' => 6999.99));
$coal    = $tito->create_orders(array('item_name' => 'Lump of Coal', 'price' => 100.00));
$freebie = $tito->create_orders(array('item_name' => 'Freebie', 'price' => -100.99));
 
if (count($freebie->errors) > 0)
{
    echo "[FAILED] saving order $freebie->item_name: " .
        join(',',$freebie->errors->full_messages()) . "\n\n";
}
 
// payments
$pokemon->create_payments(array('amount' => 1.99, 'person_id' => $tito->id));
$pokemon->create_payments(array('amount' => 4999.50, 'person_id' => $tito->id));
$pokemon->create_payments(array('amount' => 2.50, 'person_id' => $jax->id));
 
// reload since we don't want the freebie to show up (because it failed validation)
$tito->reload();
 
echo "$tito->name has " . count($tito->orders) . " orders for: " .
    join(', ',ActiveRecord\collect($tito->orders,'item_name')) . "\n\n";
 
// get all orders placed by Tito
foreach (Order::find_all_by_person_id($tito->id) as $order)
{
    echo "Order #$order->id for $order->item_name " .
        "($$order->price + $$order->tax tax) " .
        "ordered by " . $order->person->name . "\n";
 
    if (count($order->payments) > 0)
    {
        // display each payment for this order
        foreach ($order->payments as $payment)
        {
            echo "  payment #$payment->id of $$payment->amount by " .
            $payment->person->name . "\n";
        }
    }
    else
        echo "  no payments\n";
 
    echo "\n";
}
 
// display summary of all payments made by Tito and Jax
$conditions = array(
    'conditions' => array('id IN(?)',array($tito->id,$jax->id)),
    'order'      => 'name desc');
 
foreach (Person::all($conditions) as $person)
{
    $n = count($person->payments);
    $total = array_sum(ActiveRecord\collect($person->payments,'amount'));
    echo "$person->name made $n payments for a total of $$total\n\n";
}
 
// using order has_many people through payments with options
//
// array('people',
//    'through' => 'payments',
//    'select' => 'people.*, payments.amount',
//    'conditions' => 'payments.amount < 200'));
//
// this means our people in the loop below also has the payment information since
// it is part of an inner join we will only see 2 of the people instead of 3
// because 1 of the payments is greater than 200
$order = Order::find($pokemon->id);
echo "Order #$order->id for $order->item_name ($$order->price + $$order->tax tax)\n";
 
foreach ($order->people as $person)
    echo "  payment of $$person->amount by " . $person->name . "\n";
?>

The orders example should produce the following output:

[FAILED] saving order Freebie: Price must be greater than 0, Tax must be greater than 0

Tito has 2 orders for: Live Pokemon, Lump of Coal

Order #3 for Live Pokemon ($6999.99 + $315 tax) ordered by Tito
  payment #4 of $1.99 by Tito
  payment #5 of $4999.5 by Tito
  payment #6 of $2.5 by Jax

Order #4 for Lump of Coal ($100 + $4.5 tax) ordered by Tito
  no payments

Tito made 2 payments for a total of $5001.49

Jax made 1 payments for a total of $2.5

Order #3 for Live Pokemon ($6999.99 + $315 tax)
  payment of $2.50 by Jax
  payment of $1.99 by Tito

EXTENDED CONFIGURATION

Here's one more example that is basically the same as the first, but the point here is to display some of the ways to extend configuration (while remaining close to convention).

class Book extends ActiveRecord\Model
{
    // explicit table name since our table is not "books"
    static $table_name = 'simple_book';
 
    // explicit pk since our pk is not "id"
    static $primary_key = 'book_id';
 
    // explicit connection name since we always want production with this model
    static $connection = 'production';
 
    // explicit database name will generate sql like so => db.table_name
    static $db = 'test';
}
 
// ActiveRecord allows the use of multiple connections
$connections = array(
    'development' => 'mysql://test:test@127.0.0.1/development',
    'production' => 'mysql://test:test@127.0.0.1/production'
);
 
// initialize ActiveRecord
ActiveRecord\Config::initialize(function($cfg) use ($connections)
{
    $cfg->set_model_directory('.');
    $cfg->set_connections($connections);
});
 
print_r(Book::first()->attributes());
 

Simplicity

From the configuration, all the way to a serious-business example, ActiveRecord takes care of the heavy-lifting and the gritty details for you. This allows you, the developer, to focus more on business logic and complex code instead of composing custom SQL queries or designing ways to handle your data. Because we have embraced a convention over configuration philosophy, using our library is not painful. The conventions are easy to remember which will also contribute to stream-lining your productivity as a developer.

Where can you find ActiveRecord?

Again, the code is hosted on http://github.com/kla/php-activerecord/. We also have a website which will be available in the near future. We hope to have all the necessary content such as: tutorials, screencasts, and documentation.

Find the latest here.

PHP 5.3 gets ActiveRecord!

A quick search to find an implementation of active record for php on google is discouraging when one considers the state of ActiveRecord for Ruby on Rails. The reader will notice that the top results are from very old posts and the rest of the results preview minimial implementations. Of course, eventually, PHP will see a robust active record similar to RoR. Fortunately, that time is now, thanks to PHP 5.3 and the beneficial new features: closures, late static binding, and namespaces.

My friend Kien and I have improved upon an earlier version of an ORM that he had written prior to PHP 5.3. The ActiveRecord we have created is inspired by Ruby on Rails and we have tried to maintain their conventions while deviating mainly because of convenience or necessity. Our main goal for this project has been to allow PHP developers tackle larger projects with greater agility. However, we also hope that use of this resource will push the PHP community further by learning the wonderful benefits of the Ruby on Rails stack. Enough with the rambling, let's get to the interesting piece!

Overview

Allow me to reiterate the fact that we have tried to maintain similarity between our implementation and rails' ActiveRecord to avoid headaches and increase programmer bandwidth. Keeping this similarity in mind, we have tried to re-produce many of the features. Here is a list of those features:

• Finder methods
• Dynamic finder methods
• Writer methods
• Relationships
• Validations
• Callbacks
• Serializations
• Support for multiple adapters
• Miscellaneous options

There are other features such as named scopes, additional adapters, transactions (something we want sooner than later), and a few others we hope to add in the future, but we believe this is a great start. We are hoping to have the code hosted on launchpad or the like within a week or two and a website with documentation. Be sure to check back here shortly for updates!

Configuration

Setup is very easy and straight-forward. There are essentially only two configuration points you must concern yourself with:

• Setting the model auto_load directory.
• Configuring your database connections

Examples:

 
ActiveRecord\Config::initialize(function($cfg)
{
    $cfg->set_model_directory('/path/to/your/model_directory');
    $cfg->set_connections(array('development' =>
       'mysql://username:password@localhost/database_name'));
});
 
#Alternatively (w/o the 5.3 closure):
$cfg = ActiveRecord\Config::instance();
$cfg->set_model_directory('/path/to/your/model_directory');
$cfg->set_connections(array('development' =>
    'mysql://username:password@localhost/database_name'));
 

Once you have configured these two settings you are done. ActiveRecord takes care of the rest for you. It does not require that you map your table schema to yaml/xml files. It will query the database for this information and cache it so that it does not make multiple calls to the database for a single schema.

Finder methods

ActiveRecord supports a number of methods by which you can find records either by primary key or you can construct your own complex conditions array with other options such as: order, limit, select, group.

 
#find by primary key
Author::find(3);
 
#same as above but expecting multiple results
Author::find(1,2,3);
 
#find the first record with limit
Author::first(); 
 
#find last record by order and limit
Author::last(); 
 
Author::all(); 
 
#this may be evil - but you can pass your raw sql to this method
Author::find_by_sql();
 
## you can also pass many options to finder methods
 
#sql => ORDER BY name
Author::find(3,array('order' => 'name'));
 
#sql => WHERE author_id IN (1,2,3)
#find all with conditions as array
Author::find('all',
  array('conditions' => array('author_id IN(?)',array(1,2,3))));
 
#sql => WHERE author_id = 3
#find with conditions as string
Author::find('first',array('conditions' => 'author_id=3'));
 
#sql => GROUP BY name
Author::find(3, array('group' => 'name'));
 
#sql => LIMIT 0,3
Author::find('all', array('limit' => 3));
 
#sql => select * from `author` INNER JOIN etc...
Author::find('all', array('joins' =>
  'INNER JOIN book on (book.author_id = author.id)'));
 
#sql => SELECT name FROM table
Author::first(array('select' => 'name'));
 
#these methods do not return records
#return true/false
Author::exists(1);
 
#return integer
Author::count(array('conditions' => array('name = ?', 'John')));
 
# access is self-evident
$book = Book::first();
echo $book->title;
echo $book->author_id;
 
#returns an array
$books = Book::all();
echo $books[0]->title;
 

Dynamic finder methods

ActiveRecord within rails makes clever use of finders by allowing you to dynamically create methods based on the attribute names. This means you can easily make a "find_by_attribute_name" query without having to explicitly define it in the class. We also make use of this feature by using a new PHP 5.3 magic method: __callStatic().

 
Author::find_by_name('George Bush');
#you can make use of and/or
Author::find_by_name_or_author_id('George Bush', 2);
Person::find_by_first_name_and_last_name('Obama', 'Mama');
#also have find_all_by
Author::find_all_by_name('Tito');
Author::find_all_by_name(array('Tito','Bill Clinton'));
 

Writer methods

What good is it to have an object that encapsulates a record from the database if you can't do anything with it?

 
#call to SQL insert since it knows that it is a new record
$book = new Artist(array('name' => 'Tito'));
$book->save();
 
## updates
#only update 'dirty' attributes meaning the sql would only update
#fields that have been changed
$book = Book::find(1);
$book->title = 'new title!';
$book->save();
 
#this will automatically call save
$book = Book::find(1);
$book->update_attributes(array('title' => 'new title!',
  'price' => 5.00));
 
#will also make call to save
$book = Book::find(1);
$book->update_attribute('title', 'some new title');
 
$book = new Book;
$book->title = 'new title!';
$book->author_id = 5;
$book->save();
$book->created_at # we also support created_at/updated_at timestamps where applicable
echo $book->id; #id is also set to the auto increment value from the db
 
#delete
$author = Author::find(4);
$author->delete();
 
#you can pass readonly on the find so that you cannot save a model
$book = Book::first(array('readonly' => true));
$book->title = 'new'; # or you could set it here by doing $book->readonly(true);
#this will throw an ActiveRecord\ReadonlyException
$book->save();
 

Relationships

Associations are the complex piece of ActiveRecord. They accept many of the same options as with rails.

 
#relationships are declared with a static var
class Book extends ActiveRecord\Model {
  static $belongs_to = array(
    array('publisher'),
    array('author', 'readonly' => true, 'select' => 'name', 'conditions' => "name != 'jax'"),
    array('another', 'class_name' => 'SomeModel')
  );
  #has_many accepts select/conditions/limit/readonly/group/primary_key
  #has_many also takes a through option which you can use with source
  #to clarify the class
  static $has_many = array(
    array('revisions'),
    array('editors', 'through' => 'revisions')
  );
 
  static $has_one = array(
    array('something')
  );
}
 
#the 0 index declaration array of each association is the "attribute_name"
#which you use to access on the model like so:
$book = Book::first();
echo $book->publisher->name;
echo $book->author->name; # we only have name as an attribute b/c of the select opt
#below will throw a readonlyException due to the option -- see the writer
#methods section
$book->author->save();
 
#has_many
echo $book->revisions[0]->id;
#has_many through
echo $book->editors[0]->name;
 

Validations

This is rather self-explanatory. Before save/update/insert, your validations will be called for each declaration you have made and will save the model if all validations have passed. Otherwise, you will have access to an errors attribute which you can use to get back validation error messages.

 
class Book extends ActiveRecord\Model
{
  static $validates_format_of = array(
    array('title', 'with' => '/^[a-zW]*$/', 'allow_blank' => true)
  );
  static $validates_exclusion_of = array(
    array('title', 'in' => array('blah', 'alpha', 'bravo'))
  );
  static $validates_inclusion_of = array(
    array('title', 'within' => array('tragedy of dubya', 'sharks wit laserz'))
  );
  static $validates_length_of = array(
    array('title', 'within' => array(1, 5)),
    array('attribute2', 'in' => array(1,2)),
    array('attribute3', 'is' => 4, 'allow_null' => true)
  );
  # same as above since it is just an alias
  static $validates_size_of = array();
  static $validates_numericality_of = array(
    array('title')
  );
  static $validates_presence_of = array(
    array('title')
  );
};
 
$book = new Book;
$book->title = 'this is not part of the inclusion';
if (!$book->save())
  print_r($book->errors->on('title'));
 

Callbacks

Callbacks allow you to take command of your model before/after certain events during its lifecycle. You can define methods in your model that will occur as callbacks before or after other methods are invoked on the model. Unfortunately, even though PHP 5.3 has closures, you cannot use them in a static var declaration so you must define/use methods.

 
#below are the possible declarations that you can make
#if your callback returns false for a before_* then it will cancel the
#action and the rest of the callbacks
class Book extends ActiveRecordModel{
  static $after_construct;
  static $before_save = array('do_something_before_save');
  static $after_save;
  static $before_create;
  static $after_create;
  static $before_update;
  static $after_update;
  static $before_validation;
  static $after_validation;
  static $before_validation_on_create;
  static $after_validation_on_create;
  static $before_validation_on_update;
  static $after_validation_on_update;
  static $before_destroy;
  static $after_destroy;
 
  #this will be called directly before save()
  public function do_something_before_save() {}
}
 

Serializations

 
class Book extends ActiveRecord\Model{
  public function upper_title(){
    return strtoupper($this->title);
  }
}
 
#produces: {title: 'sharks wit lazers', author_id: 2}
$book = Book::find(1);
$book->to_json(); 
 
#produces: {title: 'sharks wit lazers'}
$book->to_json(array('except' => 'author_id'));
 
#produces:  {upper_title: 'SHARKS WIT LAZERS'}
#make methods an array of methods and it will call them all
$book->to_json(array('methods' => 'upper_title', 'only' => 'upper_title')); 
 
#produces {title: 'sharks wit lazers', author_id: 2}
$book->to_json(array('include' => array('author')));
 
#also support xml w/ the same options but need more tests =)
$book->to_xml();
 

Support for multiple adapters

Currently, there exists support only for MySQL (also through mysqli) and sqlite3. Right now there are only two contributors to the code base; however, we hope to attract more coders to the project that can help support additional adapters such as postgres and oracle. The connection/adapter piece of the code has been sufficiently abstracted so that it should not be difficult to create more adapters when the time comes.

Miscellaneous Options

When declaring a model you can also specify the primary_key and table_name. Protected/accessible declarations are available so that you can avoid mass assignment problems. Attributes can be aliased so that you may access them via different names.

 
class Book extends ActiveRecord\Model{
  static $primary_key = 'book_id';
  static $table_name = 'book_table';
  static $attr_accessible = array('author_id');
  static $attr_protected = array('book_id');
  static $alias_attribute = array(
    'new_alias' => 'actual_attribute',
    'new_alias_two' => 'other_actual_attribute'
  );
}
 

The Future

As I stated previously, very shortly the code will be available on launchpad. We are also in the works of putting up a website to host tutorials and documentation for the code. I will make additional posts once those milestones have been reached. Thanks for reading.

i.e. For n cores the maximum throughput is roughly 1/n vs. a single index.

With multi-core sharded indexes the underlying assumption is that search performance improves by splitting your index into smaller chunks. These smaller shards are then faster and more efficient to search and index. However, you never get anything for free, the performance increase comes at a cost of higher CPU utilization. By breaking the index into multiple smaller pieces it makes searching and indexing on that smaller subset of the index faster, but you'll need to search each core individually for every query. Where as a single index runs one slightly slower query, a multi-core sharded query runs n queries in parallel and then combines the results.

There is one problem which still needs to be worked out with the multi-core sharded index. There is no distributed IDF (inverse document frequency). This is to say, if your documents are not spread evenly across all shards then you risk a result set that is improperly ordered based on your sorts, query boosts, etc. This happens with a distributed multi-core index because the scoring of the documents takes place within each individual core before the results are combined and the query returned.

Ideally, a multi-core index is great if you need to increase the performance of your queries and can afford to sacrifice some scalability and throughput to see it through.

Below are some charts of benchmarks that I have compiled on the CitySquares SOLR index. The specifications of the machine and indexes are as follows:

Testing machine - Dell r900:

• 4x Quad Core Intel(R) Xeon(R) CPU E7340 @ 2.40GHz (16 physical cores)
• 24GB RAM
• 3x 15k RPM drives in RAID 0
• Gig-Ethernet on a local LAN

Index Stats:

• 14.5 Million Documents
• 13 GB total size
• 56 fields (indexed and/or stored w/ various amounts of processing)
• Fully optimized index

Benchmarks:

• Used Apache Bench for testing purposes from another machine on the same LAN over Gig-E.

 
#!/bin/bash
echo "" > solr_results.log
for C in 2 4 8 16 32 64 128 256 512
do
N=$(($C*1000))
echo "ab -n$N -c$C" >> solr_results.log
ab -n$N -c$C 'http://solr:8080/solr/select?q=<ID>&qf=<FIELD>&fq=<FIELD>:<ID>&start=0&rows=20' >> solr_results.log
done
 

For the trends in red the lower the number the better.
For the trends in blue the higher the number the better.

We can see here in the above graph that there were no results from the 512 concurrency test. This is because there was a deadlock in the Apache Tomcat server. The max number of connections was set to 512 with an overflow of 100.  This is the cause of all the cases where there are no results for the 512 test case. Ironically the Single core without the cache managed to finish but the test with fieldCache on failed.

The higher the better in the following chart.

The lower the better in the following charts.

The above graph shows the only test to finish successfully with 512 concurrent connections was the single index with caching disabled.

This graph is the same as the one before without the last two concurrency levels so you can see whats going on at the beginning of the benchmark. Its still hard to see but the multi-core sharded indexes are a bit lower that the single indexes. Its clear however at the higher concurrencies that the single indexes beat the multi-core ones hands down.

Ive attached a spreadsheet with actual numbers from the benchmarks since some of the charts are hard to read.

So there it is, take it as you will. There are definitely benefits to moving from a single index to a distributed multi-core sharded index. However, whether it works for your dataset and application is up in the air. After these benchmarks we decided that the multi-core index that had served us well on Amazon's EC2 no longer worked well for us on our new managed hosting. We are currently running a single index at CitySquares.

The following scenario I am about to describe is proof positive that you should always take the time to understand the underlying functionality of whatever operating system, programming language or application you are using. Let my oversight and 'quick fix solution' be a lesson to you, it is almost always worth the upfront cost of doing something right the first time so you don't have to keep revisiting the same issue.


Before I delve into the nuances of SOLR let me first give you some background on what took place over the last half year at CitySquares. Back in the fall of last year the CitySquares website began experiencing an exponential growth in traffic. This growth was due to an expansion of its IYP (Internet Yellow Page) services into the New England and Metro New York areas. Prior to and during the beginning of the first wave of traffic growth, every business listing was powered by very large MySQL queries including a couple joins. The queries themselves weren't all that complex but they were big and unwieldy with joins on very large tables and lots of columns in the result sets. In some of the larger cities covered at the time (Manhattan, Bronx, Queens, Boston, etc) there were up to 100,000 rows of data that needed to be sorted before returning a rather small subset (20-40 rows) for each business listing page load. While this wasn't a big deal when CitySquares was still a niche Boston centric destination, it quickly became a huge burden on the MySQL servers. Some of these queries were so big the servers would run out of memory trying to crunch through a 3GB temp table and start thrashing the disks to server a request for Manhattan. We needed a better solution and quick.

Luckily for us we had already implemented a SOLR search engine with all the necessary data indexed from our database initially with the sole intent that search result sets shouldn't have to query the database. This worked to our advantage since it was very easy for us to modify the code base to query SOLR instead of MySQL. Both result sets were formatted as an object with the same field names and all. It was a perfect drop in replacement.

The SOLR solution we implemented utilized SOLR's wild card q.alt=*:* field to select all documents while applying filter query (fq) on that set to get all documents related to our filter. It was a huge win for us at the time. Not only were the queries faster than the MySQL ones, but the SOLR servers could handle more of these queries without even coming close to exhausting the server's resources. This quick and dirty solution was satisfactory for the next few months until CitySquares' next round of expansion began, where again, the queries became a burden. The second time around we didn't have another seemingly quick fix. I spent a couple days trying to figure out a better way to implement the q.alt=*:* field but to no avail I gave up and moved onto other performance optimizations.

Unfortunately, I didn't take the time to understand the code behind the query and I didn't understand exactly how SOLR was implementing the query in its back end process. Since I didn't understand the basis of the problem I couldn't possibly know the query could be easily re-factored. After a few weeks of high loads, 20+ on our 8 core servers, I struck up a conversation with Michael, the developer who wrote the query. We discussed how the query worked and what it needed to do and after five minutes we had discovered a much better way to structure the query. It took me only about a minute or two to re-factor the original query to produce the exact same result set. This new query was incredibly fast! I benchmarked it to be about 100x faster than the previous query and on top of that it was a simple drop in replacement!

From what I've deduced the original query passed a blank query string with a filter query to SOLR which in turn defaulted to the q.alt catch all first and then applied the filter on the catch all query. This is exactly the opposite of what we were expecting SOLR to do. We believed that the filter was applied first and then the q.alt was applied. However, that was not the case. while this misunderstanding wasn't ideal it wasn't too slow either with only 1.4 million documents to parse over. However once CitySquares hit the 14.5 million mark this query became unmanageable. Basically SOLR parsed over every single document in the index before applying the query filter we were using. To rectify this and regain performance and through put on our servers I simply moved the filter query statement to the query statement and specified the query field to be the same as the original filter field.

i.e.

Original query passed a blank query string with a filter query:

• select?q=+&fq=<FIELD>:<ID>

The updated query now passes the id as the query string and specifies the former filter field:

• select?q=<ID>&qf=<FIELD>

Instead of taking advantages of SOLR's and every other search engines strength of O(1) search time we were at the mercy of its worst case scenario O(n) scan time. This simple misunderstanding of how SOLR processes queries in the back end caused massive performance and throughput bottlenecks. These bottlenecks affected our short and long term infrastructure plans, and was the root cause of many performance headaches for our users, customers and IT department.

If this isn't proof positive that you should always take the time to understand the underlying functionality of whatever operating system, programming language or application you are using I don't know what is.

The algorithm simplifies a polyline by removing vertices that do not contribute (sufficiently) to the overall shape. It is a recursive process which finds the most important vertices for every given reduction. First, the most basic reduction is assumed. A single segment connecting the beginning and end of the original polyline. This is when the recursion starts, the most significant vertex (the most distant) for this segment is found and, when the distance from this vertex to the segment exceeds the reduction tolerance, the segment is split into two sub-segments, each inheriting a subset of the original vertex list. Each segment continues to subdivide until none of the vertices in the local list are further away than the tolerance value.

There is a PHP class that does just this: Douglas-Peucker Polyline Simplification in PHP by Anthony Cartmell. Based on the original quality of the data and tolerance level, I was able to achieve a 90-93% reduction in size. This reduction allows me to represent significantly more data at a reasonable performance level to clients. Keep in mind, that this reduction is removing data out of the coordinate array so the quality of your representation will go down with the tolerance and reduction being applied. I highly suggest that you play around with the tolerance until you find a good balance between data size and image quality.

pnPoly - Used to determine if a coordinate falls inside a polygon.
Centroid - Find the center of a polygon..
Area - Calculate the area of a polygon.
googleGeoCoder - Extracts GIS information from Google Maps from an address.
PolylineEncoder - Takes a set of coordinates and encodes it for Google Maps.

If you ran into the problem I did, which is that a lot of the data is coming in the form of shp/dbf files and needs to be parsed out to something friendlier either KML or CSV, there are a couple of solutions for that. You can parse out the data with shp2text if your source coordinate format is already in lat/lng or if you have different coordinate system and use ArcGIS, you can try the plugin Export to KML 2.5.3 to help with the exporting of data with the ESRI suite of products.

Once your data is in SQL, the following query is an example of distance sorting with SQL. You can grab a copy of the zip_codes database here and play around with it.

SELECT *,
sqrt((69.1 * ("37.6" - latitude)) * (69.1 * ("37.6" - latitude)) +
(53.0 * ("-77.6" - longitude)) * (53.0 * ("-77.6" - longitude)))
AS distance
FROM `zip_codes`
HAVING distance < 10
ORDER BY distance ASC

There is a whole suite of tools that are able to break down content to help improve the filter by supplementing it not only with a database of words to categories, but also sets of N-gram derived from the text. There are several scripts out there that will help with this extraction and it offers a few more layers of depth for Bayesian filtering. One such tool is, Ngram Statistics Package (NSP) which is easy to install and run.

I ran a very basic test against an older post to see how it does with bigram extraction.

# perl bin/count.pl --ngram 2 test.cnt test.txt
# perl statistic.pl --ngram 2 dice test.res test.cnt

Sample bigrams found:

cloud computing, master slave, groups online, Back Again, made absolutely, very costly, extensive development, hefty bill, start ups, distribution awareness

Rather than running a probability that the set of words above would fit into one category in this case, "Technology" we can now compound the score with the probability that those terms fall into the category as well. For another layer of scoring, trigrams can be extracted, 4-grams, etc. In the financial sector the terminology is thick and analysis will be almost impossible without N-gram extraction. "Filed for bankruptcy" and "avoided bankruptcy" could not be further apart. With traditional filtering, the word "bankruptcy" would be meaningless because it really is not an indicator as to the probability that the article is favorable or not because there is no context. In this case by extracting the phrases the filter can understand and score appropriate the difference between the two terms.

Paul Graham has been working on improving the Bayesian filter to deal with spam by splitting the data into categories. Text is classified not only as legitimate or spam based on the context of the message, but the likely hood of tokens appearing in various parts of the message. N-gram filtering in this case wouldn't work as well for spam as the amount of grammar mistakes, misspellings, and word ordering would make any benefit worthless. Spammers are adjusting their content to beat such filters all the time. When the source data is reliable, the N-gram addition to the filter will boost categorization accuracy.

Integration to traditional Bayesian filtering is very easy. Google has been using text processing for a while now. This is a huge area of study in linguistics, language processing and machine learning. With so much data out there and more being collected on a daily basis, deriving context from text will allow for applications to behave smarter.

Every developer will grow their own set of tools, but the core components are available for free. Google offers analytics to take control of your traffic performance, sources, and patterns. There is also Adwords Keyword Tool, which will help you target search phrases, volume, and competition. Based on these factors and a list of similar keywords you will be able to identify good opportunities to compete for relevant traffic. There is also the Webmaster guidelines published by Google that will give you a general best practice for search engines.

This process requires a lot of patience. It takes time for changes to take shape and results are delivered. When making changes to any site or even designing a new site with SEO built in, user traffic is not going to happen right away. Seeing the results come in will trigger an OCD to check Analytics and forever make improvements and indentify new markets and opportunities. The vast majority of web sites are there for user consumption. SEO became big business when a lot of people all at once figured out that users translated to consumers.

Google is the search leader, therefore they offer the highest return. They control the flow of traffic on the internet. Luckily, they also published a search engine optimization starter guide in pdf format! This is the 101 of SEO and it will be pointless to try to chase down every obscure reference and tip on the countless SEO sites out there when the components to their content analysis is available all in one place. The document is a general overview but offers some very important best practice rules that are easy to implement:

Title Tags

- Choose a title that effectively communicates the topic of the page's content.
- Create unique title tags for each page
- Use brief, but descriptive titles (limit of 66 characters or 12 keywords)

Description Tags

- Accurately summarize the page's content
- Use unique descriptions for each page
- Avoid filling the description with only keywords
- Avoid copy and pasting the entire content of the document into the description meta tag

URL structure

- Use words in URLs
- Create a simple directory structure
- Provide one version of a URL to reach a document
- Many users expect lower-case URLs and remember them better)

Site Navigation

- Create a naturally flowing hierarchy
- Use mostly text for navigation
- Use "breadcrumb" navigation
- Put an HTML sitemap page on your site, and use an XML Sitemap file
- Consider what happens when a user removes part of your URL
- Have a useful 404 page

Anchor Text (Links)

- Choose descriptive text
- Write concise text
- Format links so they're easy to spot

Heading Text

- There are six sizes of heading tags, beginning with <h1>, the most important, and ending with <h6>, the least important.
- Imagine you're writing an outline
- Use headings sparingly across the page
- Avoid using heading tags only for styling text and not presenting structure
- Avoid excessively using heading tags throughout the page

Other Confirmed Ranking Factors

- Keyword in URL
- Keyword in Domain name
- Freshness of Pages
- Freshness - Amount of Content Change
- Freshness of Links
- Site Age
- Anchor text of inbound link
- Hilltop Algorithm
- Domain Registration Time

There is a lot of helpful content in the document but it does not go deep into the inner mechanics like other sites attempt to do. There are several sites out there that try to go beyond what has been published and into the details for generating traffic, you would just need to google "Google Ranking Factors". A lot of information came from when google released US Patent Application #20050071741.

Use the above as a baseline of the steps to get your site more traffic. This is a topic that is constantly being updated as search improves and requires a lot of time and research to do efficiently. Overhauling existing projects to meet the standards of today's crawlers is tedious, boring, and offers no immediate results. It has been something I avoided in the past, but for a web site to stay competitive and more importantly, be seen it has to be found. I find having some good rules in place for how to deal with SEO makes new projects going forward much easier to deal with.

We intend on predicting the movement of stocks based on real time content analysis. This requires a good deal of machine learning and historical data, but even good content analysis is not enough. Using Bayesian Filtering with noise word reduction we plan on processing historical data and assigning the content to one of three categories: moveup, movedown, nomove. In order to train the filters, past press releases will be inserted into the filter mashed up with the stock data to track how the markets reacted to the context of the content. Over time, the software will be able to recognize keywords that trigger positive versus negative emotion in the market that would drive the price one way or the other. A score can be applied much like spam scores are applied and this number can be used as part of a greater overall algorithm to determine an action.

Just to bring a few readers up to speed on exactly how this will be applied, take the following formula:

Rather than training it to recognize the probability of spam we train it to recognize the probability that the word will trigger positive stock movement:

• p is the probability that the content will result in positive movement.
• p1 is the probability p(S | W1) that it is positive knowing it contains a first word (for example "capital");
• p2 is the probability p(S | W2) that it is positive knowing it contains a second word (for example "boosted");
• etc...

The entire body of the content will be processed against a known database of words and the market reaction to the presence of those words. The basic Bayesian filtering will need to be extended to deal with phrase recognition but overall a solid proven technology for machine learning to build from.

This information by itself, is nothing revolutionary but with strong pattern analysis like candlestick pattern recognition and other market indicators it can be used to create an accurate trading platform for marginal gains which over time can offer pretty high returns. There is certainly a lot of potential for this if it works, but it heavily depends on working accurately and there will be a lot of trial and error in the process.

For more reading on the concepts and components behind this idea, check out:

• Naive Bayes Classifier
• Candlestick Patterns
• Candlestick Charting
• Better Bayesian Filtering
• TD Ameritrade API

The nice part is all the historical data is out there around the internet which makes back-testing and scoring very easy to do and there will need to be a lot of testing.