1. Up until March 1, 2012, the data Google collected on you when you used YouTube was carefully cabined away from your other Google products. So, in effect, Google could use data they collected on YouTube to improve and customize the users’ YouTube experience, but couldn’t use the data to customize and improve user experience on, say, Google+.
2. The same siloing took place for your search history. Previously, Google search data was kept separate from other products. Even when users were logged in, Google promised not to share the information they gathered about you from your Google search history when customizing their other products. Considering how uniquely sensitive user search history can be (indicating vital facts about your location, interests, age, sexual orientation, religion, health concerns, and much more), this was an important privacy protection.

(Electronic Frontier Foundation, What Actually Changed in Google’s Privacy Policy)

It looks like the new Google privacy policy does make a substantial—and very detrimental—change: it took questions from Congress and this belated EFF article for me to realise the extent to which Google’s invasion of its users’ privacy has just got substantially worse. I now use Duck Duck Go, combined with the !g bang added to the search, to search Google securely and anonymously, without being tracked, even if I'm actually logged in.

∞

When Google shipped its Search, plus Your World update earlier this month, it turned out better than expected. Google left users the ability to click back and forth between personal and global modes or opt out altogether. [...]

Google does make some effort to identify content from other networks. But some SPYW features only highlight Google+ material, even when other services are more relevant. If Google favors its own product over a better result, users get the short end of the stick. [...] But what about Google+ users? For them, Google+ results are the better result. Arguably, Google should cater to them, as users of its service.

(ReadWriteWeb, It's Not Wrong for Google to Focus on Its Own Users)

I've been feeling from the beginning that criticism of Google suddenly turning evil, of which this post is a good example, is misplaced: you can opt out of Google Search, plus Your World; and if you have a Google+ account, it unquestionably makes your search results better. I've actually found Google Search results have improved very significantly since I complained about them just under a year ago.

∞

No person shall be held to answer for a capital, or otherwise infamous crime, unless on a presentment or indictment of a Grand Jury, except in cases arising in the land or naval forces, or in the Militia, when in actual service in time of War or public danger.

(Fifth Amendment to the United States Constitution)

It was of course a mere coincidence that the closure of the Megaupload file locker site, and the arrest of some of its alleged leaders, occurred on the day following SOPA strike day ^[1]. It so happened that a fortnight before, on January 5, seven individuals and two corporations – Megaupload Limited and Vestor Limited – were indicted by a Federal grand jury in the Eastern District of Virginia, and charged with engaging in a racketeering conspiracy, conspiring to commit copyright infringement, conspiring to commit money laundering and two substantive counts of criminal copyright infringement.

What this means is that at least twelve out of twenty-three US citizens, selected from a pool consisting of names culled from various databases, such as national voter lists, motor vehicle license lists and public utilities lists ^[2], and meeting outside the presence of a judge, decided that there was sufficient evidence to warrant indicting the delightfully-named Herr Kim Dotcom, his business partners and two corporations they control, for engaging in the pretty serious crimes described in the Justice Department’s press release quoted above. Those offenses of course go well beyond mere copyright infringement, although this is also alleged to have occurred on a massive scale.

Megaupload is based in Hong Kong, but some of the alleged pirated content was hosted in the US on leased servers in Ashburn, Virginia, which gave US authorities jurisdiction, the indictment said.

That is not to say this ancient institution, which the United States is alone among common-law countries in retaining, provides a foolproof safeguard. The Cato Institute, among other recent commentators, pointed out in 2003, at the height of the Bush administration’s ‘war on terrorism’, that:

Like its more famous relative, the trial jury, the grand jury consists of laypeople who are summoned to the courthouse to fulfill a civic duty. However, the work of the grand jury takes place well before any trial. The primary function of the grand jury is to inquire into the commission of crimes within its jurisdiction and then determine whether an indictment should issue against any particular person. But, in sharp contrast to the trial setting, the jurors hear only one side of the story and there is no judge overseeing the process. With no judge or opposing counsel in the room, grand jurors naturally defer to the prosecutor since he is the most knowledgeable official on the scene. Indeed, the single most important fact to appreciate about the grand jury system is that it is the prosecutor who calls the shots and dominates the entire process. The grand jurors have become little more than window dressing.

This is of course potentially true. But grand juries are no substitute for criminal proceedings. They are a necessary preliminary stage for them to be initiated, where a decision is taken, not on actual guilt, but on whether there is a sufficient case to be made for indicting someone in the first place. Whatever the pressures on them, the role of the grand jury is only to determine probable cause—not guilt. And in the case at hand, while I have not of course had access to the evidence presented, I believe it is not unreasonable to plead that the above gentlemen had, at least, a case to answer.

In addition, since the private individuals named in the case were not physically present in the US at the time of their indictment, it was necessary for the American administration to contact the governments of the countries where they were resident—in the case of Herr Dotcom and some of his business partners, HM Government in New Zealand—to obtain their agreement to physically apprehend them, in a scene worthy of a gangster movie (‘It was definitely not as simple as knocking at the front door,’ said one of the New Zealand police officers involved in the operation).

Because of this circumstance, the US Government will face the additional hurdle of establishing before the New Zealand extradition court that a reasonable case can also be made that the alleged activities of Herr Dotcom and his associates were also illegal under New Zealand law (no proceedings are being instituted against them in New Zealand). And they are apparently sufficiently rich to afford to defend themselves as vigorously as Wikileaks’s Mr Julian Assange.

Unless and until they are found guilty in a US court, Herr Dotcom and his associates, will thus be protected by due process and a standard of justice that meets the world’s highest standards. The allegations against them are serious ones. I cannot agree with the Electronic Foundation, which said in a statement that the arrests set ‘a terrifying precedent. If the United States can seize a Dutch citizen in New Zealand over a copyright claim, what is next?’ There is no equivalent in New Zealand to the controversial Extradition Act 2003, a pretty scandalous piece of legislation enacted by the Blair government, which allows the US to extradite UK citizens and others for offences committed against US law, even though the alleged offence may have been committed in the UK by a person living and working in the UK ^[3]. And the allegations against the defendants go well beyond mere copyright violations.

What the Megaupload affair illustrates, therefore, is that additional legislation of the type of that proposed under SOPA and PIPA is hardly necessary to enforce copyright law. The global, libertarian environment, largely free of government interference, which made the Internet possible is something that must be preserved, and attempts to curtail it, whether they come from members of the US Congress, the French President’s wife, or a weak Spanish government, need to be resisted. But there is a distinction to be made between libertarianism and anarchy—an environment in which there are no enforceable standards at all. Freedom and creativity aren’t protected when there aren’t any rules at all or when those on the statute book can all be flouted despite being proportionate and reasonable.

I believe the existing body of US legislation and case law governing the digital economy to meet those standards of fairness and proportionality. This site, which is governed by New York law, strictly conforms with these standards, sharing its content via a Creative Commons License, while affirming its copyright on the underlying computer code. Any citations made are done under the well-established practice of Fair Use legal, non-licensed citation or incorporation of copyrighted material. Any incidence to the contrary can be reported on the site’s DMCA page. It’s possible for every actor of the Internet operating in the United States to operate freely under the same provisions. I believe that environment offers the right balance between the insane repression sought by the entertainment industry that was temporarily ended in the US on Wednesday—and will, one hopes, be ended in France later this year after the Presidential elections—and the lawless one on which Herr Dotcom and his associates are alleged to have thrived. I’m certain they will get more than a fair hearing, and wish them good luck for the future: the clothes, swagger, helicopter and the panic room certainly provided entertainment to the world—surely a mitigating circumstance that the Motion Picture Association of America will fully appreciate.

1. It appears that Herr Dotcom was planning to hold an extravagant birthday party at his New Zealand house on January 21,and that all his associates had travelled to the country to attend it, providing authorities with the opportunity to arrest them all together.
2. Unlike potential jurors in regular trials, grand jurors are not screened for biases or other improper factors.
3. The Coalition is considering removing the more controversial aspects of the Blairist extradition legislation.

Direct censorship involving manipulation of the domain name system which lies at the heart of the Internet is what is normally associated with the world’s most illiberal regimes—Iran and China foremost among them—not the United States. Yet that is precisely what SOPA and PIPA are setting out to do:

The entertainment industry, meanwhile, is defending its fast-melting cash cow by putting forward a fundamentally flawed argument. The following series of tweets from Mr Rupert Murdoch, the Chairman of News International, whom I have already discussed in this site, is revealing of the elementary misconception underlying the repressive lobby’s case.

His initial tweet was in reaction the recent news that the Obama administration was distancing itself from the provisions of the SOPA bill as it then stood tabled:

He then went on to state that he that he considers even aggregating or linking to stories to be piracy:

Google rapidly reacted, stating:

This is just nonsense. Last year we took down 5 million infringing web pages from our search results and invested more than $60 million in the fight against bad ads… We fight pirates and counterfeiters every day.

Yet the subtleties of this were lost on Mr Murdoch. In his next tweet, he seemed to equate streaming with links simply appearing in a Google search:

The issues underlying yesterday’s SOPA strike are not new, but the blackout, which was fairly widely reported in the press, has brought them to the fore in an unprecedented way. The tech sector has proved pretty unanimous in its opposition, not just to the detail of the measures contained in SOPA and PIPA, but also to the idea of allowing censorship of the web in any form.

It is often forgotten, however, that the issue is a global one. As the Electronic Frontier Foundation points out:

SOPA and PIPA are really only the tip of the iceberg. The same forces behind these domestic U.S. laws have continued to both push for other states to pass similar domestic laws, as well as to secretly negotiate international trade agreements that would force signatory nations to conform to the same legal standards. The Anti-Counterfeiting Trade Agreement (ACTA), Trans-Pacific Partnership (TPP), Ley Doring (Mexico), Ley Sinde (Spain), Ley Hadopi (France) are only a few examples.

Ultimately, the issue is this: the emergence of the Internet has made the business model on which the entertainment industry made billions of dollars for years obsolete—and it is prepared to destroy it to postpone the consequences for as long as possible. Yet censorship, in any circumstances, should be available only as a last resort, as part of a proportionate procedure and subject to due process—a basic principle incompatible with delegating it to a private entity.

Rarely has a battle of titans implied such high stakes: in the fifteen years or so since it became ubiquitous, the Internet has empowered individual citizens in an unprecedented way, giving them instantaneous access to a range and depth of information in a hitherto unthinkable manner. Yesterday’s blackout went long way to demonstrating that new-won freedoms can never be taken for granted, even in the world’s richest nations and greatest democracies. SOPA and PIPA’s sponsored have not laid down their arms. EFF makes the point succinctly:

Unfortunately, PIPA and SOPA are still very much alive. According to Open Congress, there are still 33 co-sponsors of PIPA. With the Senate bringing PIPA to the floor next week, we cannot expect the content groups to give up without a fight. The entertainment industry is already threatening to cut off campaign donations to President Obama’s re-election campaign. Chief lobbyist of the MPAA, Chris Dodd, also lashed out at the blackout yesterday, saying it was a “gimmick,” and even “dangerous.”

The Anti-Counterfeiting Trade Agreement (ACTA) is well on the way to coming into force. The Trans-Pacific Partnership (TPP), Ley Sinde (Spain) and Hadopi (France) are all in force and all, in different ways, directly threaten the Internet as a free, neutral way of communicating—that should not be compromised neither for mercantile profit nor for political gain. Should France’s President Sarkozy gain re-election in France’s presidential elections in May, or should Mr Obama fail to be re-elected at the year end, these pressures can be expected to gain strength. Now is hardly the time to fall into complacency.

Apple, for all the shit they get for being “closed” and “evil”, has actually done far more to wrestle control back from the carriers and put it into the hands of consumers. Google set off to help in this goal, then stabbed us all in the back and went the complete other way, to the side of the carriers.

This article by MG Siegler is an absolute must-read: it sums up more clearly than anything I’ve read in the past two years why Google’s strategy with Android—which it had to turn on its head after just a few months because the original one was killed off by the carriers—is disingenuous in the extreme.

∞

[…] public G+ posts count as websites now, as far as Google search is concerned. Furthermore, Google search now obscures natural Web results whenever possible, giving Google+ results from users' networks instead. And it hit me: True Believers of the Holy + might see this re-shared Google+ post of an almost-unattributed rip of my story instead of the original. Google+ hates the Internet!

Although I'm not seeing this for my posts yet, I think ReadWriteWeb is right: it's increasingly looking like Google+ is a mess, not just in itself, but also in its implications for Google's core business, search. Is Google+ messing up the Internet? Looks like it's going in that direction at any rate.

∞

When looking at these issues, one must also remember that France’s Constitution puts the President of the Republic at the apex of its institutional system—and while that principle has been considerably diluted since General de Gaulle created the function, next year’s presidential election, together with the attendant National Assembly general election due one month later, will determine the essential direction the country will take for the following five years.

Public policy continues to play a greater role in France than in other countries: the 2012 presidential elections will be a major watershed in a number of issues

M Sarkozy can be expected to seek re-election to the office entrusted to him in 2007 and, unless something unexpected happens, the second ballot in May will offer the French people a choice between the outgoing incumbent and M Hollande, the moderate, liberal-leaning candidate designated by the Opposition Socialist Party in a nationwide primary last autumn.

In most other countries, public policy tends to focus exclusively on economic or foreign-policy issues, but this is unlikely to be the case in France. Debate so far in the French press has mainly covered the two candidates’ personalities and their precise proposals for dealing with the rather dire situation in which the once proud French economy is left, after five years of M Sarkozy’s administration, goes curiously unmentioned. Even more curiously, to non-French eyes, will be the place traditionally accorded to cultural policy and, in recent years, to tech-related issues, by the media when covering French politics. It’s a subject I’ve covered before and, all in all, an important one both in a French and an international context.

M Sarkozy and the ruling UMP party have adopted a singularly illiberal stance on the Internet and on tech

M Sarkozy’s campaign for the French Presidency, in 2006-2007, had little to say about the Internet. He has never shown the slightest interest in information technology and does not use a computer—although he has belatedly realized the damage his total tech illiteracy was doing to his image and now (December 2011) claims to be using an iPad.

His presidency, however, has been marked by an unprecedented onslaught on the tech sector, instituted largely in a desperate attempt to shore up the entertainment industry, to which M Sarkozy, his pop-singer third wife, Culture Minister Frédéric Mitterrand (a former television chat-show director) and the UMP party are all extremely close, via the HADOPI three-strikes and you’re out law adopted in 2009.

Despite the determination with which the Sarkozy administration has attempted to police the Internet, its actions were totally ineffective, as the measures it enacted proved impossible to enforce from the start, a fact that was widely pointed out and led to pretty universal ridicule. The government’s utter ignorance about even the most elementary technical facts underpinning how the Internet works included, famously, a long-held delusion that illegal streaming can be effectively policed.

M Sarkozy, who never shies away from a policy volte-face if he perceives it to be in his interest, famously expressed doubts about his HADOPI brainchild in April this year:

Je prends d’ailleurs ma part de l’erreur. (…) L’intuition que j’avais, c’est qu’on ne pouvait pas (abandonner) les créateurs. Peut-être que la maladresse a été de donner le sentiment que vous étiez attaqués. ^[2]

(M Sarkozy at the Conseil national du numérique, Le Monde, April 27, 2011)

The President’s office, however, swiftly issued a press release retracting those comments and his administration has since stuck to its previous uncompromising stance on the issue: in November 2011, he announced the government’s intention to tax ISPs to finance a new National Music Centre and his desire to extend the existing anti-piracy provisions to streaming.

The current French administration’s repressive stance has not been confined to policing the Internet, extending to to data protection

The Sarkozy administration has also been prepared to scale back significantly the protections enshrined in France’s 1978 Data Protection Act.

It is now forgotten that there was an earlier attempt, called SAFARI, by a French government to create a centralized database of personal data. On March 21, 1974, an article in the newspaper Le Monde, “SAFARI ou la chasse aux Français” (SAFARI; or, Hunting Frenchmen) brought public attention to the project. France’s data protection watch dog, CNIL, was instituted following the scandal this created and as a result of which it was given extensive powers to enforce individual citizens’ rights of access and privacy—at subject still extremely sensitive at that time, given the Vichy régime’s 1941 attempt to set up a centralized nationwide file detailing, among other things, individual citizens’ religious and political affiliations. This database was then used by the collaborationist police to arrest and deport thousands of French and foreign Jews. A report by M Joinet, its first director in 1978-1981, which was circulated by CNIL just two years prior to M Sarkozy’s election shows how sensitive the subject remained until that time.

The Sarkozy government, on the other hand, has felt constrained by no such scruples. Immediately on gaining office, it embarked on creating a series of databases recording an astonishingly wide range of information about private individuals, in most cases people without criminal records.

In July 2008, the Sarkozy government created a database called Edvige, which on investigation turned out to be so injurious to civil liberties (the government was, for instance, empowered to included anyone’s political affiliation or sexual preference in it, even on the basis of hearsay) that the Prime Minister was compelled to descend it altogether the following November; it was replaced with a database from which the most objectionable aspects had been removed.

The French government recently went a step further in the same direction when a bill was adopted by the National Assembly at the government’s request, aimed at creating a fichier des gens honnêtes (a database of honest people). The Sarkozy administration is thus embarking on the same path as the Bush administration with the Patriot Act, and the Blair government with its compulsory identity card scheme—since withdrawn by the present Coalition government). A recent article on French investigation site OWNI goes into some of the possible reasons for M Sarkozy’s relentless drive to keep track of every detail of his compatriots lives.

All in all, thus, M Sarkozy’s obsessive intrusion into individual citizens’ privacy in the name of copyright protection or terrorist prevention has unquestionably exposed him to both anger and ridicule—given the technically inept way in which it has been carried out. The same can be said of the half-baked attempt to set up a government data base, Etalab, of which only 1 per cent is available in an open format, despite initial promises to the contrary.

Will the Sarkozy administration’s appalling record in tech and data protection have any effect on the forthcoming presidential election?

While M Hollande, the designated opposition candidate, has said little as yet on these subjects, he can expect his support to be significantly boosted by the government’s repressive stance on tech issues and on data protection.

So far, however, he and his advisers appear to have chosen to say as little as possible on policy issues until the election campaign actually begins—which can be expected to happen in February. His designated spokesman on tech issues, Madame Fleur Pellerin, a young member of the Court of Auditors, has already begun meeting members of the tech sector—in which resentment of M Sarkozy’s policies failures is strong—and it can be expected that this will encourage the opposition to express positions that will improve the quality of debate in this area.

Tech and data protection are, of course, only one among many issues likely to be debated in the French presidential election. Yet this issue matters, arguably more than others: the Internet is one of the main contributors to growth in the US, the UK and a rising number of North European economies. France lags behind, in sharp and surprising contrast to the leading place it still claimed to to occupy in the technological field as late as the 1960s. OWNI’s remarkable report shows that despite all the obstacles the authorities have put in its way, the Internet accounts for c. €30bn in GDP and 1.15 million jobs or 4.2 per cent of the working population. Even more significantly, it accounted for 20 per cent of French GDP growth between 2004 and 2009. So the Internet unquestionably has the potential to become a much bigger contributor to economic growth in one of Europe’s most deficient and stagnant economies if recent mistakes are reversed: it can only be hoped that France’s next government will understand this better than its current one.

1. Incredible as it may seem, copyright in France continues to be governed essentially by legislation enacted in 1957.
2. I’m quite prepared to admit it: I was party to the mistake […] My intuition was that we couldn’t leave creators on the wayside. It may have been wrong to give the impression that you were being targeted.

In this post, I’ll recount how I set up my new (ve) server, combining it with one of Media Temple’s ProCDN accounts so that tasks are clearly apportioned between the two:

• the (ve) server will be optimized to do what it’s best at, i.e. fetching content from the database and building the dynamic content displayed on my WordPress blog;
• The CDN will host all static content and serve it extremely efficiently from the closest possible location to the user, which in turn is what it’s best at.

I’ve been very happily hosted by Media Temple since 2006 and, since 2008, I’ve been using one of their (dv) servers. I almost invariably get 100 per cent uptime from it, it’s fast and, of course, when needed Media Temple provide live, 24-hour support 365 days a year: I’ve always found their experts reliable, very smart and often happy to provide support—on the rare occasions when I’ve needed it—well beyond the bounds of what is required.

Media Temple recently started offering a service called SiteMovers—a team comprising the company’s sharpest support experts, dedicated to carrying out WordPress site migrations. I knew I probably didn’t need help with the migration, but curiosity got the better of me and I contacted the SiteMovers team to ask if they were interested in helping with migrating my very heavily customized WordPress-powered blog. After just a few minutes the team called back to say they’d be delighted to do so—and although I ended up doing most of the migration myself, they provided me with very useful suggestions throughout: the MySQL optimization I set up was entirely the result of their suggestions, and I’d highly recommend using them for anyone who feels uncomfortable about the process described in this post ^[1].

Whatever you do, back up all your files before editing, deleting or moving anything. I cannot be responsible for anything that happens to your setup as this is only an account of what worked for me. It may not necessarily work for you.

Using a CDN

It’s useful to have a CDN to host your static content so it can be served to your users faster. This applies particularly to me, with roughly half my readers being in the US, and the other half elsewhere, predominantly in Europe.

Since last year, I’ve been using Amazon Cloudfront to provide ridiculously cheap and scalable fast access to certain static resources: images, video, css and javascript files, essentially. Because Cloudfront doesn’t support serving gripped content, however, I had to resort to rather complicated hacks to optimize the rise of the files while still enjoying the benefit of a CDN.

In recent months, I’ve noticed that Cloudfront wasn’t actually that quick at delivering the content hosted on it: testing my site using Yslow, Google Page Speed and Pingdom showed that objects stored on Cloudfront were those that were loading slowest: moving them back to my Media Temple (dv) server actually resulted in a gain in speed.

My reasons for switching from (dv) to (ve) on Media Temple

While I wasn’t unhappy with my (dv) server’s performance, I knew it was running at a fraction of its potential. Yet in the (dv) setup, while fine-tuning is possible, because you have full root access, it isn’t the most convenient thing to do, as the server’s administration passes through a Plesk control panel that is installed by default when the server is configured: and Plesk is one of the most pathetically outdated, bloated and generally dysfunctional pieces of software I have ever handled—I only really put up with it because I don’t spend that much time on server maintenance tasks. But I’d been taking a close look at the (ve) server that Media Temple started offering last year, which is targeted specifically at users who want more control over their setup.

I knew exactly what I wanted: a lean server running only the OS, software and modules I needed, with no bloat, and my resources allocated in a totally optimal way.

Since I wasn’t planning to move away from a WordPress environment, I wanted to continue to separate static content from dynamic content, as I had done with the (dv) and Cloudfront. My choice was to switch all my dynamic content to the (ve) server, and as much as I could of my static content to Edgecast, which Media Temple has chosen as its CDN partner.

How I switched from (dv) to (ve)

Media Temple doesn’t encourage you to switch to a (ve) unless you know what you’re doing: they provision it with your choice of Linux OS—Ubuntu, CentOS, Debian, or Fedora, plus root access. Then you do the rest. No bloat or anything you don’t need—meaning a much more effective server in the end. As Media Temple puts it:

(ve) Server is designed for users who have significant experience with Linux, or for those who are interested in learning. Our goal is to give customers complete control of their Virtual Environment, therefore servers are delivered with SSH only. You will control and install all software.

The process is actually simpler than it sounds. It involved four stages: (1) basic server configuration; (2) installing LAMP; (3) tweaking your server and (4) migrating WordPress. Media Temple has excellent knowledge base guides for each of these. My comments on how I implemented them follow.

Basic Linux server configuration.

Media Temple knowledge base article: Getting started with Ubuntu

This was perhaps where the advantage of (ve) over (dv) is most apparent. You can create just one user and give him control over all the essential administration of your server, by giving him the desired permissions and including him in the appropriate group(s). This takes a few seconds using the command line, with the command root@ve: adduser jsmith and adding jsmith ALL=(ALL) ALL at the appropriate place in the /etc/sudoers file (this is detailed in the knowledge base article). This basically turns the user you’ve just created into a superuser, able to carry out most routine administrative tasks.

I found, however, that in addition to this, it’s useful to decide how you want to organize your permissions, for both groups and individual users: it can make sense to have a single user and group (such as the existing www-data group which is created on setup) to own all resources in /var/www, which is where all the data relating to websites you host on your server will be stored, while leaving the ownership of all other directories unchanged. You may want to invest a little time in understanding Linux directory structure before you take any decisions on this, as it’s not always ideal having to change your system later on.

It is also well worth following the Media Temple suggestions to secure your server, though these are optional: changing your ssh port, disabling root access and using ip-tables. I found I needed to add to the default suggested by Media Temple, however, because of my fondness for using desktop, third-party client for everything: I need to access MySQL remotely from my Mac (my client of choice being the excellent Sequel Pro), as well as ssh from Coda, which I use for all my coding and file transfers (and which you can also use to safely edit text files ^[2] ). If you install webmin—and Media Temple provide a knowledge base article for this too—, you’ll also need to add -A INPUT -p tcp -m state --state NEW -m tcp --dport 10000 -j ACCEPT to your /etc/iptables.rules file in order to access the webmin interface.

At this point in the migration I found I could gain substantial time by creating TextExpander shortcuts for my ssh logins, for the webmin URL (https://YOUR_DOMAIN_IP:10000) and for restarting Apache (sudo /etc/init.d/apache2 restart): considering the number of times one needs to type them in, the combination of the command-line and TextExpander shortcuts is a huge time saver (make sure your Mac is secured by a really secure password and Filevault, if you do this, obviously).

Installing LAMP

Media Temple knowledge base article: (ve):Install LAMP on Ubuntu 9.10

This, I found, was the most straightforward part of the process: I slavishly followed the Media Temple knowledge base article, adding a file with a <VirtualHost> section for each of my sites in the /etc/apache2/sites-available folder, then creating:

• the corresponding paths :

# sudo mkdir -p /var/www/ve-server{1,2}.com/{html,logs}

• files:

# sudo touch /var/www/ve-server1.com/html/index.html

• and content:

# sudo echo -e '<html>\n<head>\n<title>Welcome to apache!</title>\n</head>\n<body bgcolor="white" text="black">\n<center><h1>ve-server1.com is working!</h1></center>\n</body>\n</html>' > /var/www/ve-server1.com/html/index.html

for each site.

Tweaking your server

This, not surprisingly, is at once the most enjoyable and the most frustrating stage in the process. Each user will have different requirements, obviously, so my customizations are just one example—but here’s what I needed to set up—not necessarily before I installed WordPress, as I only discovered the issues while I was testing the new install, but I’m including them here before, for convenience.

Make sure mod_rewrite is enabled

Firstly, I found Rewrite is not enabled by default on my server. As a result, any .htaccess files in my existing WordPress configuration were being ignored and the custom URLs set in my WordPresss were not working ^[3]. All you need to do to solve this is to ensure that mod_rewrite is enabled, as follows:

# sudo a2enmod rewrite

Then restart Apache:

# sudo /etc/init.d/apache2 restart

Configure the server so WordPress can access your FTP credentials

Secondly, I found that, when I tried to download any plugins, I was unable to update FTP details in the WordPress administration panel, in order to download plugins ^[4]: instead I’d get a error message requesting that I update details—and the corresponding fields would be frozen and not accept updates.

The reason for this—which I discovered thanks to this blog post—is that in current versions of WordPress, it checks if the userid it is running under, is the same as the owner of the file or folder it tries to edit: the issue arises at line 876 in the method get_filesystem_method in wordpress/wp-admin/includes/file.php:


if ( getmyuid() == @fileowner($temp_file_name) )
$method = 'direct';
@fclose($temp_handle);
@unlink($temp_file_name);

You can solve this issue by creating an extra entry in your mydomain.com VirtualHosts:80 entry:

<IfModule mpm_itk_module>
AssignUserId jsmith www-data
</IfModule>

Sending email

Servers send email in a variety of ways that you can configure more or less as you like. In all cases, however, they need to have a Mail Transfer Agent in order to send (and, if desired, receive) mail.

SendMail, a a general purpose Internet mail routing facility that supports many kinds of mail-transfer and -delivery methods, is installed by default on a (dv) server—and is what PHP’s Mail(http://via.dj/rQmnjq) hooks into to send emails from your contact forms, for instance, unless you’ve set another MTA in your server. I found that SendMail hadn’t been installed, and that as a result my contact form was not working. Since I only needed to send emails, and was using my Google Apps address as the outgoing mail address, I was actually looking for a lightweight alternative that would only handle sending, but would enable my emails to be routed and authentified via gmail’s SMTP server, ensuring that they would not get flagged as spam.

I did it in the following way. First, I installed sSMTP ^[5] :

# sudo apt-get install ssmtp

Next, I configured sSMTP:

# cd /etc/ssmtp
# vim /etc/ssmtp/ssmtp.conf

Here’s a possible configuration for sending via gmail’s SMTP server:

MailHub=smtp.gmail.com:587             # SMTP server hostname and port
UseTLS=YES                             # Gmail requires a secure connection (SSL/TLS)
FromLineOverride=YES                   # Force the From: line
Hostname=mail._YOUR_DOMAIN             # The name of your host
RewriteDomain=mail._YOUR_DOMAIN        # The host the mail appears to be coming from
Root=YOUR_ACCOUNT_NAME                 # Redirect mail for root@ to postmaster@
AuthUser=YOUR_ACCOUNT_NAME@gmail.com   # Your gmail mail account
AuthPass=YOUR_GMAIL_PASSWORD           # The password for the mail account

You’ll want to add some revaliases to /etc/ssmtp/revaliases from which to send email:

# Format: local_account:outgoing_address:mailhub
root:alerts@YOUR_DOMAIN:smtp.gmail.com:587
Postmaster:alerts@YOUR_DOMAIN:smtp.gmail.com:587
You:YOUR_ACCOUNT_NAME@YOUR_DOMAIN:smtp.gmail.com:587

Optionally, you might want to change the ‘From’ display name on your outgoing emails: to do this, edit the /etc/passwd and add or modify the appropriate user alias  

alerts:Alerts:507:507::/home/alerts:/bin/bash
YOUR_USER_NAME:YOUR_USER_NAME:505:506::/home/YOUR_USER_NAME:/bin/bash
postmaster:POSTMASTER:505:506::/home/postmaster:/bin/bash

To ensure sSMTP is used by PHP to send email, you need to edit your php.ini file in /etc/php5/apache2/php.ini. In your php.ini, find the sendmail_path= line, and edit it to:

# sendmail_path = /usr/sbin/ssmtp -t

PHP requires the -t flag.

Save your php.ini and restart Apache. Your mail() function should now work, using sSMTP.

If you’re installing sSMTP on a server that does have Sendmail installed, you’ll additionally need to remove and disable it:

# service sendmail stop
# sudo chkconfig –levels 2345 sendmail off

You could also move SendMail and create a symbolic link so any processes that are written send mail using SendMail (which many do) use sSMTP instead.

To do this, move or replace the sendmail and then create a symbolic link for sSMTP to sendmail with authentication:

# mv /usr/sbin/sendmail/usr/sbin/sendmail.orig
# ln -s /usr/sbin/ssmtp/usr/sbin/sendmail

Migrating WordPress to the new server

Updating server path references

This side of things is extremely simple. You just need to download your theme files and save a copy of your database, and update any paths in them to reflect the (ve) directory structure, which is difference from that in the (dv): the (dv) places your WordPress files in /var/www/vhosts/YOUR_DOMAIN_NAME/httpdocs, whereas the (ve) stores them in /var/www:YOUR_DOMAIN_NAME/html.

There are several ways of doing this: use that which suits you best. Whichever way you choose, of course, make full backups of everything before you change anything ^[6] You should also compact and repair your database to keep it lean, and remove any tables that are no longer needed (such as those created by old plugins you’ve uninstalled).

I ran a search-and-replace on the saved version of my database in TextWrangler, a robust code editor for the Mac that can carry out regex operations on even the most gigantic of files. Then I ran the same operation in Coda for the theme files.

Installing WordPress on the new server

I made a fresh WordPress install on the new server, reinstalled all my plugins, uploaded the theme files, activated my chosen theme, and added the following to my config.php file:

//remove this before flipping your DNS
define('WP_HOME','http://YOUR_NEW_IP_ADDRESS');
define('WP_SITEURL','http://YOUR_NEW_IP_ADDRESS');

Improving database performance to take advantage of the (ve) server’s power

Media Temple knowledge base article: Using MySQLTuner

Now’s the time, if you want to to, to tweak your database settings to take advantage of your customized (ve) server’s power. I used mysqltuner.pl, a script written in Perl that allows you to review a MySQL installation quickly and make adjustments to increase performance and stability. Media Temple has a knowledge base article about it, which can be used, appropriately adapted, to your (ve) database. Based on the information it provided, I tuned /etc/mysql/my.cnf to raise several of my MySQL limits to utilize available resources on the server (again, make backups of all files you edit beforehand—and bear in mind your requirements will probably differ from mine as your WordPress database is unlikely to be identical):

# * Fine Tuning
#
key_buffer      = 16M
max_allowed_packet  = 16M
thread_stack        = 192K
thread_cache_size       = 64
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
max_connections        = 100
wait_timeout           = 60
table_cache            = 128
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_type    = 1
query_cache_limit   = 2M
query_cache_size        = 32M
tmp_table_size      = 32M
max_heap_table_size     = 32M 
join_buffer_size    = 32M
key_buffer_size     = 16M

I found this induced a noticeable increase in speed in my server.

Moving static content to the CDN

Media Temple knowledge base article: Get started with ProCDN

The final adjustment was moving all static content (CSS, javascript and images, in my case) to the CDN. I chose to store CSS and javascript, which are dependent on your WordPress theme, in the traditional place in a subdirectory of the active theme. Images, however, I decided to store in in a subdirectory of /var/www/YOUR_DOMAIN_NAME/html, since I don’t use the WordPress media management feature, preferring to keep control of my files.

If you want to take advantage of this feature, you first need to add ProCDN to your Media Temple account, though: it costs $20/month, which in my care comes on top of the $50 monthly charge for my (ve) server. But I think the speed improvement justifies it. Setting it up only takes a few minutes and is detailed in a knowledge base article. Once you’ve done this, you’ll have a super-fast mirror of your site, replicated in Edgecast’s data centers around the world, ready within the time it takes for the settings to propagate (about an hour after you add the necessary CName to your DNS). You then need to change any references to static files in your theme to reference the CDN (using, say, cdn.yourdomain.com instead of yourdomain.com, if that is your CName for the CDN), and you’re ready to go.

Flipping the DNS

Once you’re all set up, the final moment is when you’re certain your new server is totally functional and you flip the DNS for your web site from the (dv) server to the (ve) server. This is childishly simple. Media Temple provides a function in your Account Center which allows you to point your existing DNS zones to the new ip (ip addresses are tied to the particular server cluster on which your domain is hosted at Media Temple, so you’ll necessarily have to change the ip address associated with your domain when you switch from (dv) to (ve)). Make sure you lower ythe Time To Live for each of your domains about twelve hours, at least, before you switch, from the default 43,200 seconds to 300 seconds, in the Account Center, and the change in site will propagate within about five minutes of flipping. You can keep the TTL low for a few hours afterwards, too, so you can flip back if you discover something has gone wrong—though obviously in extreme cases only.

1. SiteMovers costs $150.
2. Coda seems to have an issue with setting file permissions, however. I’ve take up the matter with Panic, its editor, but as of writing this post, the issue hasn’t been resolved as far as I know, and I find it safer to set and change permissions via the command line only.
3. I actually chose to set any Ovverride parameters in the httpd.conf file called by apache2.conf which contains all the default configurations: this has the advantage of being faster, because the server no longer has to check for individual .htaccess files at each level the directory structure, and simpler to maintain because each directory’s configuration is located in a single, centralized file.
4. In former WordPress editions, you could include FTP credentials in your wp-config.php file, but this no longer works.
5. sSMTP is a program that replaces sendmail on workstations that should send their mail via the departmental mailhub from which they pick up their mail (via pop, imap, rsmtp, pop_fetch, NFS… or the like). This program accepts mail and sends it to the mailhub, optionally replacing the domain in the From: line with a different one.
6. One very reliable WordPress database management utility, which takes the form of a plugin, is WP-Database-Manager, by Lester Chan, which, although it’s no longer very actively maintained, still works absolutely flawlessly.

One of the big drawbacks of WordPress is the incredible number of pages it generates, matching all sorts of combinations including tags, categories, months and years. While it’s convenient to have the ability to generate these pages on-the-fly when the corresponding queries are entered, it’s not always realised that these pages will normally also get indexed by Google in the same way as individual posts and pages.

This matters more than one would spontaneously think, for two reasons:

• firstly, it means the index of one’s site will end up being rather messy, with a lot of duplicate and overlapping content, making it rather difficult for someone who’s looking for an item to find it;
• secondly, that duplicate content has long been one of the factors in Google’s page-ranking algorithm, and a rather negative one—if your site’s cluttered up with pages basically recycling identical content, you’l be doing your page rank no favors.

The drawbacks of plugins

As with everything in WordPress, there are plugins—plugins for everything under the sun—that claim to help you solve this issue, and plugins are the best solution for many people. The trouble with plugins, however, is that by installing them you’re effectively giving up control of one aspect of your site setup to a third party. Because a plugin will never be custom built with your needs in mind and because they’re often written to cater for every possible requirement, you’ll almost almost always end up with unnecessary bloat.

One of the most irritating things about plugins is the habit they have of hooking in to one’s html code, inserting inline styles that pollute one’s code and slow down one’s loading time.

Because of this, it’s good practice to regularly run down one’s list of installed WordPress plugins, and check on which ones can be dispensed with. Simple tasks can usually be accomplished as well, if not better, and certainly more cleanly, by googling the issue and then writing a few lines of your own code, armed with what you have found.

How to use conditional statements in WordPress

WordPress makes abundant use of PHP conditional statements, which in their most basic form look like this:

<?php if ( ) { ?>......<?php  } ?>

Using WordPress-specific conditional statements, you can (and probably do in your blog) target certain parts of your site to apply formatting or produce certain events. Because the site is dynamic and everything is displayed on the fly, the conditional statements fetch information from the database, and display it after applying the conditions stated in the tag. Popular statements include is_page(), is_category(), is_single() and is_page(). The WordPress Codex provides a full list of conditional tags.

Conditional tags are often misused in WordPress. Database queries are very resource-intensive and ought to be reserved for content that changes constantly and cannot be known in advance. Using conditional statements to display your name, for instance, when you’re the only author on your blog, is a bit absurd, yet occurs quite a lot. I prefer to hard-code as much of my theme as possible, using html in preference to PHP, which speeds the site up considerably.

PHP, however, comes into its own for one purpose: fine-tuning what HTML gets displayed in your site’s source code: I use this extensively in my own header, allowing me to call different css and scripts depending on which page is displayed:

<!-- Stylesheets  -->
<?php
if ( is_front_page() ) :{ ?>
<link rel="stylesheet" href="/home.css" /><?php }
else :{ ?><link rel="stylesheet" /style.css" /><?php }
endif;
?>

With this in your header.php file, you do not even need to have a separate robots.txt

Fine-tuning which of your content gets indexed by Google using PHP conditional tags

You can use exactly the same structure to determine which content is indexed by Google: and there’s certainly no harm in having a robots.txt file. But the trouble with them is that not all robots obey the robots rules: even Google has been reported to ignore certain robots rules.

<!-- Meta tags -->
<?php if (is_front_page() || is_page('about') || is_single() ) :{ ?>
<meta name="robots" content="follow,index" /><?php }
else :{ ?>
<meta name="robots" content="noindex, nofollow">
<?php }
endif;?>

The above code will tell robots to index the front page (which WordPress allows you to have as a separate page, like it is in this site), About page and individual blog posts. Everything else is given a <meta name="robots" content="noindex, nofollow"> tag, meaning your duplicate content won’t be indexed. Your page rank won’t be wasted on pages whose purpose is to help internal navigation and not external indexation. And only your fresh, relevant content will be indexed.

Targeting only part of each page’s content for indexing with googleon and googleoff tags

But indexing only certain pages may not be enough. On each page, only part of the content is actually relevant for indexation purposes. Navigation text, sidebar items about similar posts, tweets, and other tangential (or, to use the html5 phrase, asides) serves no useful purposes being indexed on the same page as an article featuring original content about one subject. If you let these parts of your blog post pages get indexed, they will show up in searches even though they aren’t actually relevant to the main article in that page.

An article by Perishable Press, Tell Google to Not Index Certain Parts of Your Page, alerted me to the fact that there’s an accepted way of fine-tuning the content you designate for indexing even within one page: googleon and googleoff tags ^[1].

<p>This is normal (X)HTML content that will be indexed by Google.</p>
<!--googleoff: index-->
<p>This (X)HTML content will NOT be indexed by Google.</p>
<!--googleon: index>

By applying these tags to my single.php, front_page.php and about.php pages (but you could target any content you wanted), and applying googleon and googleoff tags to tell robust only to index the actual title and text of the blog post within the template, I’m certain that only relevant content will show up in searches ^[2].

1. There are four types of googleon/googleoff tags: (1) index — content surrounded by “googleoff: index” will not be indexed by Google; (2) anchor — anchor text for any links within a “googleoff: anchor” area will not be associated with the target page; (3) snippet — content surrounded by “googleoff: snippet” will not be used to create snippets for search results; (4) all — content surrounded by “googleoff: all” are treated with all attributes: index, anchor, and snippet
2. These tags are specific to Google, so other search engines will continue to crawl your pages from top to bottom.

Regular visitors to this site may or may not have noticed that its domain has changed. This certainly doesn’t reflect any decision on my part to turn it into a commercial venture. It’s simply the result of my purchase, after it took me three years to convince its previous owner to sell it, of the rather more generic donaldjenkins.com, which will now be the actual platform for the site. A 301 Redirect will effortlessly propel anyone using the old address to the new one, and Google have been notified of the change, although you might want to update any links accordingly.

Switching domains actually turned out to be childishly simple: since the directory structure of the new site is exactly the same as the old’s, all that was needed was applying a few regular expressions to remove any trace of the old url in my database and php files, and I was up and running.

New layout: the Belgravia theme

I’ve taken the advantage of the move to also switch to a new layout. I wasn’t too happy with the Chelsea theme that I’d designed when I switched to html5 this spring: It moved away from the rather more subdued layout of the earlier Astorg theme and I was never quite comfortable with it.

The Belgravia theme whose arrival coincides with the .com switch was initially intended as an upgrade to the Chelsea theme: but I ended up moving away from it sufficiently to justify a name change. I’ve retained much of the html code and attention to semantics introduced in the spring—and html5 document outlines still show up meaningfully. I’ve trimmed down the markup whenever possible to ensure faster loading speeds. I find that a custom version of Paul Irish’s excellent Modernizr is all I need ^[1] to ensure the site displays reasonably well in all broswers, so I was able dispense with jQuery altogether, except on the contact form, which I designed from scratch to provide server and client validation.

Sticking with WordPress—but increasingly tempted by Jekyll

I’ll try and write posts over the next few days to cover some of the more interesting components I implemented in the site. One idea with which I toyed before ruling it to was switching away from WordPress to a completely static site such as Jekyll. Octopress, a beautifully well thought-out platform designed for Jekyll by Brandon Mathis, would be my top candidate if I did. It would probably minimize what to me is the biggest drawback of Jekyll: Jekyll is a static site builder, so naturally it has to rebuild your site any time anything changes. One would hardly want to wait forever to publish every single post: with WordPress and MarsEdit, I’ve got a workflow that’s pretty unbeatable for comfort and speed, and one I’m happy to stick with for the time being. Also, WordPress is sufficiently well-established to be pretty much future-proof, which isn’t the case,obviously, for Jekyll.

In the meantime, though, I’m sticking with a WordPress platform in which I’ve hard-coded he vast majority of the content that doesn’t change (the contact form is entirely hard-coded to overcome the frequent inconsistencies between jQuery and WordPress’s internally-generated code), to keep database requests to queries that actually serve a useful purpose: I make systematic use of PHP conditional statements to determine which resources are served depending on the page or type of content displayed—this works really well and has speeded up loading time to well under a second for the most part. Next time round, though, I may well choose to make the switch.

1. Modernizr really works superbly at what it sets out to do, as this screencast shows.