Some Doctrine 2 Best Practices

Doctrine 2 has a very powerful database abstraction layer and features an Object Relational Mapper (ORM) that makes it very easy to manage relationships between the data in your application and provides tools that aid rapid application development (RAD).

With Doctrine also comes DQL, a powerful language for querying your object models,  which is similar to SQL.

Use of Doctrine abstracts away the database concepts of tables and columns, rather you think in terms of objects, entities, properties and relationships. While this is very useful, it is very important for the developer to have an understanding of the underlying SQL that is generated from using the entity manager/DQL and querying your objects/entities.

Without an understanding of the code that Doctrine generates, it is relatively easy to build an inefficient and slow application, here are a few pitfalls that I have found are easy to fall into, and some tips that may be useful for people starting with Doctrine, or if you are trying to get a little more performance out of an existing application.

For the rest of this post I will use the cookie cutter example of a Blog website with article, user and comment object models (entities). In terms of object mapping we will assume all mapping is specified with the default (lazy) fetch method, and MySQL as our DB.

1. Always write a DQL statement for querying your object models

Doctrine repositories provide some ready made methods for fetching your objects such as “->findAll()” and “->findBy()”. These provide some convenience and brevity when initially developing your application but use of these functions is less flexible when further developing your application and can lead to some of the pitfalls below.

By writing the DQL yourself, you will expand your understanding of Doctrine, you have more control of the data you are trying to query and people reading your code will have a better understanding of what you are trying to do (and will find it easier to update and extend your code).

2. Beware of lazy loading when querying entities with associations

If you do not specify an ‘eager’ fetch in your object mapping, or do not explicitly fetch your relationships in your DQL statements (or if you query your objects using the Doctrine methods mentioned in point 1) Doctrine will not fetch those associations in the SQL it generates. This is helpful as you do not want to be loading resources you do not need, however when you call a related object, Doctrine needs a method of fetching these objects and it does this by lazy loading through proxies.

For example if you have an article entity, with associated comments and you want to display a list of all your articles with number of comments on the front page of your blog, you may do something like this (this code is purely for demonstration).

$articles = $article_repository->findAll();

foreach($articles as $article)
{
    echo $article->getTitle();
    echo count($article->getComments());
}

Using this code, Doctrine will initially generate a query for all the rows in the article table. As we have not ‘fetched’ the comment association, Doctrine will lazy load in the comment object for each article via an additional SQL statement. For a table containing 10 rows, this will result in 11 database queries.

//$em is the entity manager
$qb = $em->createQueryBuilder();

$qb->select('Article', 'Comment')
    ->from('Entity\Article', 'Article')
    ->leftJoin('Article.comment', 'Comment');

$query = $qb->getQuery();

$result = $query->getResult();

return $result;

Instead, by using DQL and specifying our relationships, we can use the above code to fetch the same data with 1 query vs the  11 (unnecessary) database queries generated previously. This problem grows with each additional row in a given database.

3. Use array hydration for read only actions

The Doctrine documentation recommends the use of array hydration for read only actions, this is in part due to the extra overhead (increased memory needed from loading proxies and the like) associated with hydrating objects.

If you are just displaying lists of items, or lists of entity properties for a single item (for example a product page, a list of recent blog articles, or indeed most front end operations!) you can use:

$query->getResult(Query::HYDRATE_ARRAY);

This approach can also prevent any accidental lazy-loading problems that may arise as discussed previously (due to the lack of proxies!)

4. By default Doctrine will fetch all of the properties for a given entity

This doesn’t exactly fall under a best practice, indeed making any changes here too early in the development cycle would fall under premature optimisation. Nonetheless consider the following:

$qb->select('Article')
->from('Entity\Article');

It is somewhat wasteful in terms of resources to fetch data from your database that you aren’t going to use. The above code snippet would fetch all the properties for the article entity.

Rather than selecting the whole object, in DQL you can specify specific properties to return in your select statement:

$qb->select('Article.title')
->from('Entity\Article');

However this will always return a flat array, regardless of relationships between your entities. Instead you can select and hydrate partial objects in your DQL, which will give you an object or a nested array representing your object graph (depending on the hydration selected)

$query = $em->createQuery("SELECT partial Article.{title} Entity\Article Article");

As stated in the Doctrine documents it is important to note here that this optimisation can lead to more fragile code, it may not always be clear (especially if working as part of a team, and working code written by others) that object properties have not been fetched.

5. Use prepared statements

Using prepared statements helps to prevent SQL injection attacks that have proven problematic for developers and businesses alike – Sony notably exposed a lot of user data in Summer 2011. See below for a quick example of how to use prepared statements in Doctrine.

$qb = $em->createQueryBuilder();

$qb->select('Article', 'Comment')
    ->from('Entity\Article', 'Article')
    ->leftJoin('Article.comment', 'Comment')
    ->where($qb->expr()->eq('Article.title', ':filter_title')
    ->setParameter('filter_title', $some_user_input);

$query = $qb->getQuery();

$result = $query->getResult();

return $result;

The Doctrine documentation provides a more in depth explanation for achieving this.

6. Be aware of what Doctrine is up to

If you are using Symfony 2, you can use the in-built profiler for monitoring the number of queries generated by doctrine. If you are using Zend Framework you can refer to my previous blog post covering some methods for profiling doctrine 1 or 2.

 

I hope these tips prove helpful if you are just getting started with Doctrine, or if you have been using it for a little while. If you have any tips of your own let us know on twitter @uv_d or leave a comment

Native apps Vs mobile web apps: confused?

I’ve recently been discussing a mobile web app module we have in development for our CMS with a number of existing clients and potential new ones and I’ve learnt a few things of late about peoples understanding of mobile applications; that is to say, as soon as I mention ‘native apps’ in the same breath as ‘mobile web apps’ I confuse at least 80% of my audience. It has been an interesting learning curve for me establishing just what level of understanding people have on these two distinct (but similar… confused already?) methods of reaching an audience and delivering content so I thought I’d write a bit more to help clarify a few things in an attempt to aid an informed choice of which direction to take.

What is a Native App?

Generally speaking, if a customer comes to us with a requirement for a ‘mobile app’ most (99%) will be thinking about a Native app. By Native I mean an application built for a specific mobile handset (they will be picturing a shiny iPhone), purchased from and delivered by a central store (they will be picturing Apple’s App Store), installed on the mobile device with an icon on the homescreen with updates managed through the store from which it was downloaded.

Do your clients Actually Want a Native App?

When you delve into what it is the client actually needs, it is not always apparent that a native app is the most suitable or appropriate tool for the job, especially if it is one being developed in a specific language (Objective C for instance in the case of an iPhone) which then needs porting to other target devices (for example BlackBerry, Android and Nokia).

As is often the case, what a lot of people think they need is down to a preconception (often established through marketing, advertising, trends and fads) that one way of doing things is the only (and best) way of doing things. The alternatives are not explored because they only appear on the periphery whereas particularly with the iPhone, iPad and the App Store people have perceived understanding of the end–to-end delivery and usage of a native app.

What is a Mobile Web App?

On the other hand, when I discuss mobile web apps it is clear from an early point in the conversation that most people think I am discussing a native app whereas what I am actually describing is an application developed on the web specifically for mobile devices (at present many sites use the ‘m’ subdomain to host their mobile web apps); to all intents and purpose this website (mobile web app) might look, behave and feel much like a native app but it is accessed through a browser and doesn’t have to be downloaded via an app store. In a lot of circumstances mobile web apps are actually advantageous:

  1. You can better manage development across multiple platforms
  2. They can look and behave like a native app (even having an icon on the homescreen) so UI/UX can retain a ‘familiarity’ across both types
  3. Other than fetching the initial data / framework the app does not have to be online to run (a common misnomer) as content can be cached and we can use HTML5’s offline mode
  4. You don’t have to go through the App Store acceptance procedure and updates / enhancements can be released as and when required (see point one)
  5. There is no reliance on someone else’s business model (eg Apple) to monetise your idea
  6. You get to keep 100% of the revenue you generate
  7. As you will see in the final paragraph, mobile web apps can be packaged as native apps for multiple platforms (eg iOS, Android, Symbian, Blackberry) using mobile frameworks such as PhoneGap and Appcelerator Titanium
  8. Applications can be developed more rapidly (see all above)

Native is Still Best (sometimes)

That being said, there is no denying some advantages that native apps (particularly in the case of Apple) have:

  1. End users currently better understand the concept (of the store – application – handset relationship)
  2. Businesses currently better understand the business model because it is quite simple
  3. People feel confident in the application vetting process and therefore have confidence they are not being scammed or having malware/viruses delivered to their handset via the app
  4. You can access all the handset features and hardware (eg: the camera, geo locations, graphics acceleration, file system and so on)
  5. Monetisation of the app itself – if your idea is to produce an app which makes money just because it sells 1 million copies at 59p then this is the only way you can go

There is a Third Way

Our mobile web application development uses the Sencha Touch framework (a HTML5 Mobile Web App Framework for touch screen mobile devices). We can expose various methods of our content management system (for getting content or defining functionality) via webservices which take parameters and data and return data in json format (or XML if you desire) to the Sencha Touch application. The framework allows development of mobile web apps and provides a local storage proxy for saving data offline.

The Sencha Touch framework runs on Webkit which is shipped with all the major mobile devices these days (and Google Chrome) so when browsing the mobile web app via a web browser on the mobile handset the user experience is fairly consistent and as previously discussed can very much be like a native app.

However, if you want to deliver your mobile web app as a native app this is now possible using frameworks such as PhoneGap or Appcelerator. In our case we use PhoneGap which allows us to provide native functionality such as access to geo location, camera, storage, contacts and so on (list of PhoneGap supported features).

The steps are fairly easy (paraphrased here):

  1. Develop your mobile web app in Sencha Touch (or other)
  2. Add any native functionality your require (and is supported by your framework)
  3. Package this up for your target device in PhoneGap (or other)
  4. Send native app to appropriate store for acceptance testing
  5. Sell / provide your app via the app store

See, sometimes you can have the best of both worlds!

Further reading:

http://www.phonegap.com
http://www.appcelerator.com
http://www.sencha.com
http://www.lukew.com/ff/entry.asp?1193
http://www.lukew.com/ff/entry.asp?1264
http://www.uxbooth.com/blog/its-about-people-not-devices

Flixel 2.0 – Flash game development

beatbarni-blogimg

We have finished an enjoyable project for SPAR International, which involved creating an athletics game based around the button bashing classic, Daley Thomson’s Decathlon. When creating the graphics and style for this game we wanted to utilise the nostalgic bitmap effect which you see quite often in Flash games and in the graphic style of Pixel Art. From a development perspective this was very convenient because I have been looking for an excuse to try the Actionscript 3 game framework, Flixel, which I have seen lorded around the Internet as perfect for this purpose.

It has it’s own global render cycle which simplifies the process of adding bitmap graphics to the stage and creates a predictable fps independent game loop. Forum posts on the Flixel site state that this rendering process is more efficient than Flash player alone due to the fact that is purely geared toward low resolution bitmapped graphics. There where many elements which increased the speed of development such as the inclusion of the FlxObjects which are already set-up with simple motion facilities such as acceleration and velocity on both x and y axis which allow for the simulation of horizontal character movement and also vertical forces (i.e. gravity).

There are many other noteworthy features of the Flixel framework such as Sprites for Animation and the global/static FlxG control class which have been very well designed to make game development in Flash easier. I will, however, not go into detail about these because a concise post about Flixel from the Inside RIA blog has covered these. For those (developers mainly, I suspect) interested in understanding the affordances of using this framework over creating a game from scratch, reading this and also exploring the host of games and tutorials on the Flixel site will be worthwhile. Canabalt is a particular favourite game due to the intense screen shaking and accompanying music.

The game we created is called Beat Barni and is online now. Due to time constraints we created a pentathlon (5 events) rather than a decathlon, so feel free to challenge Barni to any of the following events; 100m, 400m Hurdles, Javelin, Pole-vault and Long Jump.

Action Message Format (AMF)

Traitors Flash game title image

We have recently built a Flash Game – Traitors – for the Top Gear website:

The game, sponsored by Nissan, requires the user to speed through 3 levels controlling their speed by repeatedly pressing X-Y buttons as fast as they can and using their booster. For the purpose of data submission and receiving data while interacting with the webserver we decided to use AMFPHP which is open-source implementation of the Action Message Format (AMF) and allows binary serialisation of Action Script (AS2 and AS3) native types and objects to be sent directly to server side services. Phew…

The crux of this is the ease of implementing data transfers to and from the webserver (less code on both client application and on the webserver) and the efficiency of the resultant transfer (the data being transferred is binary format which is also compressed further in AMF3 so the amount of data being transferred is smaller). This is brilliant for rapid development (which we do a lot of) and performance of the finished article.

For instance, on the webserver you don’t have to loop through a database recordset, compose some XML and return that XML to Flash which would then load and parse the XML: you can essentially return the recordset directly from php to Flash and the query is automatically converted to the corresponding ActionScript type. There are several processes being skipped and the data being transferred is far smaller (being compressed binary as opposed to XML).

Sneak peek :: Physics

Physics Hero

We’re so close; I can smell it! Our new site is careering dangerously towards completion. One element of the homepage is a frivolous physics environment which I was kindly allowed the time to make in Flash using the Box2DFlash physics engine.

The engine has been ported from Eric Catto’s powerful C++ physics engine (Box2D). It took a little time to adjust to some of the coding conventions that have also found their way from the C++ version but after that initial steep learning curve the engine is fantastic to work with.

I’ve released this today before the site launch so that people can play with it and inevitably find the bugs which I’ve lovingly engineered into it. Feel free to put it through its paces!

edit: This is now on our homepage

AIR app review: Doc? and Polaris

Adobe AIR app icon

I have recently been researching small scale desktop application design for a video project we are producing in Flash. The obvious place to start in this instance was with showcases of Adobe AIR applications. Due to the ease with which applications can now be deployed using the AIR runtime, it already seems to have encouraged the development of applications with greater focus spent on the usability and aesthetics of the user interface and experience. Critics of this low threshold may say that this will lead to a lower standard of overall application functionality and perhaps to dirty code powering inefficient and buggy applications. I’m sure there will be many examples of this but considering the small scale and usually simple nature of AIR apps, I think this will not be the case in the long term.

In any case the main reason I wrote this post was in order to make note of two applications that I found during my research, which are notable more for their utility than their design.

Doc? Air Local LiveDocs
As an Actionscript developer without a photographic memory I am forced to regularly visit LiveDocs for Actionscript3 and Flex component references. If I’m unfortunate enough to be coding in the Flash IDE then I use the built in help docs which are slow and clunky and in Flex Builder 3 they are not much better. Doc? enables you to access LiveDocs for Flash CS3/CS4 and Flex all in the one application providing you have those applications installed (you have to point Doc? to the location on your hard disk where the reference files are installed – instructions for this are on the site FAQ). You can also create bookmarks for the pages which you use regularly.

Polaris
Polaris is quite simply a desktop application for viewing you Google Analytics accounts on your desktop. It’s incredibly compact and easy to use plus you still get all the lovely graphs and visuals which you’ll be used to from the Google Analytics web interface.

These are currently my two favourite AIR apps but I’m sure there’s a plethora of others which I’m yet to find. Feel free to present us with any further contenders.

Creative Tools for e-assessment

One of our most recent projects has involved developing a set of creative tools in Flex / AS3 which are part of a larger academically funded project in which digital tools are used in Design and Technology to encourage creativity and teamwork. This is part of broader discourse in education as to the value of the pen and paper, exam structured methods of assessment which we are all familiar with.

The main point of contention between the traditionalists and those wishing to push the bar on technology in education is whether or not traditional exam assessment is indicative of a students abilities in a certain subject areas. Throughout modern education ICT is used to support and enhance learning yet when it comes to the all important exam assessments students are forced into sweaty school halls and, in isolation, encouraged to recall facts and figures under pressured circumstances. I think I may be showing my opinion in this matter however this is not an academic paper so never mind!!!

The project which we have developed these tools for enables students to be assessed for their Design and Technology course work through various different means; photographs, voice recordings, drawings, written work and mind-mapping. These tools are all controlled through a parent/client application which leads the students through predefined questions or requests which require them to use the creative tools to evidence parts of their work. e.g. Sketch your initial designs for new type of pill holder / Take a photo of your prototype / Mind-map some ideas for new features.

Whilst the work is being created it is being immediately stored in the individual students online portfolio. The evidence and work collected facilitates a more subjective approach to student assessment called ‘Comparitive Pairs’.

Although this system is currently being piloted for Design and Technology only, I hope that further down the line that this type of assessment could be used more widely and potentially offer an alternative to archaic pen and paper type assessment.

The creative tools we have developed at Ultraviolet can be viewed here. All the tools were created in Flex / AS3. As concepts they are in no way groundbreaking, in fact at some points during development it seemed a little fruitless re-inventing the wheel, but the functionality for each tool is prescriptive to the project and clients needs.

Feel free to play and let us know what you think. Or if you find any bugs!! *Gulp*

Issues using multiple swcs in Flex Builder 3.0

As most of the Flash development here is now pure object-oriented in AS3 my IDE of choice is now Flex Builder 3.0 for all ActionScript projects. We left behind Flash IDE for the same reasons as most: it’s text editor is inadequate and the code visualisation for pure object-oriented projects is not sufficient; it also enables us to bring in line our development tools with other languages such as Java. Let’s face it, the Flash timeline IDE was never going to cut it as a serious development model!

An aspect of not developing in Flash that I had to adjust to was coping without a convenient library of graphical assets to drag into my application. There are various ways of importing or loading graphical assets into a 100% code application which nearly all involve and temporary return to Flash IDE. My preferred method is to create my graphical assets in Flash IDE, giving each of them linkage names relevant to the class/package structure of my application. I then publish a swc and import this into my ActionScript project within Flex Builder (project Properties > ActionScript Build Path > Library path > AddSWC).

So far everything works like a dream. All of my graphical assets are appearing within code hinting in Flex and are totally accessible. During a large scale project I decided to try to segregate my graphical assets into different swcs to help me organise my assets. This alone did not present any problems however I occasionally needed to go back into Flash to change or add a graphic to a swc. On exporting the amended swc, I returned to Flex Builder and found that all of the graphics contained in this swc where missing from the project. I tried renaming the swc and re-exporting it in Flash and also removing it in Flex Builder and adding it again. The only solution to this problem was to create and entirely new Flash file and copy all the graphical assets from the old (broken) fla file into the new one and re-export it again.

In a later project the same issue occurred again and this time it only began when I added more than one swc. That is essentially the solution I came to however it does not by any means explain the problem in the first place. Anyone who can fill in the gaps please feel free.

Switching from Webtrends to Google Analytics

We have been working with a client on a website (www.modernselling.com) for sales focused users for several years and have built a system to turn dynamically driven urls (such as news.apx?pageid=xx) into SEO friendly urls (such as news/sales-news-headlines/insider-trading-arrests.aspx).

The client has spent hundreds of hours (and therefore thousands of pounds) struggling with the behemoth otherwise known as Webtrends 8a in order to filter out the chaff traffic (robots and the like) from the valued traffic (actual users, referrals and actual advertising click rates). I won’t bore you with too much information but Webtrends is ultimately an enterprise piece of software and for smaller sites/start-ups and those with less time/budget (most of us) it’s analogous with using a sledge hammer to crack a nut! One example being that the installation documentation and support staff (US based) stress the application must run on its own server and has a recommended RAM size of 4GB. This might be OK for some large businesses but usually one dedicated server (hosting the website) is stretching the budget far enough for the aforementioned clientele without the associated costs of managing, maintaining and purchasing/renting a further server simple to do a bit of traffic analysis!*

So for the past 6 months we have been assessing and testing what the FREE Google Analytics (www.google.com/analytics/) has to offer. I have to say that I am impressed! Instead of tediously running pre-set up profiles and reports (hours of setting up and hours of laborious database churning) AKA the Webtrends Model we can report on everything we need by utilising the JavaScript functions available within the latest Google Analytics script (ga.js). By using pre-defined rules for advert impressions, advert click-throughs and editorial outbound links and applying them through pageTracker._trackPageview() we can record all the statistics we need.

The good news is that the client can now simply search for a string in Google Analytics (such as ‘advert-name-click’) and it will return all the statistics related to click-throughs for that advert. He can then calculate his conversion rates, see which section of his website is most effective for a particular advert and use the system to bill his clients who are all happy with the statistics and reports because they come through Google.

Remember if you are using pageTracker._trackPageview() multiple times on the same page that you must apply a filter to Google Analytics to ensure that your overall page impression statistics are not skewed – you must filter out anything other than the page impression.

More to follow on the detail…

*Note: in order for the client to return the required statistics though Webtrends we first had to write an asp.net desktop application to parse the raw log files to remove anything that looked like suspicious/robot traffic, he then set up some filters and rules in Webtrends (one for each advertising campaign) and ran the parsed logs through

Search