Last autumn, I was talking about how I would like to see pdo_pgsql for PHP to be improved.

Over the last few days I had time to seriously start looking into making sure I get my wish. Even though my C is very rusty and I have next to no experience in dealing with the PHP/Zend API, I made quite a bit of progress over the last few days.

First, JSON support

screenshot showing off json support

If you have the json extension enabled in your PHP install (it's enabled by default), then any column of data type json will be automatically parsed and returned to you as an array.

No need to constantly repeat yourself with json_parse(). This works, of course, with directly selected json columns or with any expression that returns json (like array_to_json or the direct typecast shown in the screenshot).

This is off by default and can be enabled on a per-connection or a per- statement level as to not break backwards compatibility (I'll need it off until I get a chance to clean up PopScan for example).

Next, array support:

screenshot showing off array support

Just like with JSON, this will automatically turn any array expression (of the built-in array types) into an array to use from PHP.

As I'm writing this blog entry here, this only works for text[] and it's always enabled.

Once I have an elegant way to deal with the various types of arrays and convert them into the correct PHP types, I'll work on making this turnoffable (technical term) too.

I'll probably combine this and the automatic JSON parsing into just one setting which will include various extended data types both Postgres and PHP know about.

Once I've done that, I'll look into more points on my wishlist (better error reporting with 9.3 and later and a way to quote identifiers comes to mind) and then I'll probably try to write a proper RFC and propose this for inclusion into PHP itself (though don't get your hopes up - they are a conservative bunch).

If you want to follow along with my work, have a look at my pdo_pgsql-improvements branch on github (tracks to PHP-5.5)

Read on →

In the summer of 2012, I had the great oportunity to clean up our hosting infrastructure. Instead of running many differently configured VMs, mostly one per customer, we started building a real redundant infrastructure with two really beefy physical database machines (yay) and quite many (22) virtual machines for caching, web app servers, file servers and so on.

All components are fully redundant, every box can fail without anybody really needing to do anything (one exception is the database - that's also redundant, but we fail over manually due to the huge cost in time to failback).

Of course you don't manage ~20 machines manually any more: Aside of the fact that it would be really painful to do for those that have to be configured in an identical way (the app servers come to mind), you also want to be able to quickly bring a new box online which means you don't have time to manually go through the hassle of configuring it.

So, In the summer of 2012, when we started working on this, we decided to go with puppet. We also considered Chef but their server was really complicated to set up and install and there was zero incentive for them to improve because that would, after all, disincentivse people from becoming customers of their hosted solutions (the joys of open-core).

Puppet is also commerically backed, but everything they do is available as open source and their approach for the central server is much more «batteries included» than what Chef has provided.

And finally, after playing around a bit with both Chef and puppet, we noticed that puppet was way more bitchy and less tolerant of quick hacks around issues which felt like a good thing for people dabbling with HA configuration of a multi machine cluster for the first time.

Fast forward one year: Last autumn I found out about ansible (linking to their github page - their website reads like a competition in buzzword-bingo) and after reading their documentation, I immediately was convinced:

  • No need to install an agent on managed machines
  • Trivial to bootstrap machines (due to above point)
  • Contributors don't need to sign a CLA (thank you so much, ansibleworks!)
  • No need to manually define dependencies of tasks: Tasks are run requentially
  • Built-in support for cowsay by default
  • Many often-used modules included by default, no hunting for, say, a sysctl module on github
  • Very nice support for rolling updates
  • Also providing a means to quickly do one-off tasks
  • Very easy to make configuration entries based on the host inventory (which requires puppetdb and an external database in the case of puppet)

Because ansible connects to each machine individually via SSH, running it against a full cluster of machines is going to take a bit longer than with puppet, but our cluster is small, so that wasn't that much of a deterrent.

So last Sunday evening I started working on porting our configuration over from puppet to Ansible and after getting used to the YAML syntax of the playbooks, I made very quick progress.

progress

Again, I'd like to point out the excellent, built-in, on-by-default support for cowsay as one of the killer-features that made me seriously consider starting the porting effort.

Unfortunately though, after a very promising start, I had to come to the conclusion that we will be sticking with puppet for the time being because there's one single feature that Ansible doesn't have and that I really, really want a configuration management system to have:

I'ts not possible in Ansible to tell it to keep a directory clean of files not managed by Ansible in some way

There are, of course, workarounds, but they come at a price too high for me to be willing to pay.

  • You could first clean a directory completely using a shell command, but this will lead to ansible detecting a change to that folder every time it runs which will cause server restarts, even when they are not needed.

  • You could do something like this stack overflow question but this has the disadvantage that it forces you into a configuration file specific playbook design instead of a role specific one.

What I mean is that using the second workaround, you can only have one playbook touching that folder. But imagine for example a case where you want to work with /etc/sysctl.d: A generic role would put some stuff there, but then your firewall role might put more stuff there (to enable ip forwarding) and your database role might want to add other stuff (like tweaking shmmax and shmall, though that's thankfully not needed any more in current Postgres releases).

So suddenly your /etc/sysctl.d role needs to know about firewalls and databases which totally violates the really nice separation of concerns between roles. Instead of having a firewall and a database role both doing something to /etc/sysctl.d, you know need a sysctl-role which does different things depending on what other roles a machine has.

Or, of course, you just don't care that stray files never get removed, but honestly: Do you really want to live with the fact that your /etc/sysctl.d, or worse, /etc/sudoers.d can contain files not managed by ansible and likely not intended to be there? Both sysctl.d and sudoers.d are more than capable of doing immense damage to your boxes and this sneakily behind the watching eye of your configuration management system?

For me that's inacceptable.

So despite all the nice advantages (like cowsay), this one feature is something that I really need and can't have right now and which, thus, forces me to stay away from Ansible for now.

It's a shame.

Some people tell me that implementing my feature would require puppet's feature of building a full state of a machine before doing anything (which is error- prone and frustrating for users at times), but that's not really true.

If ansible modules could talk to each other - maybe loosly coupled by firing some events as they do stuff, you could just name the task that makes sure the directory exists first and then have that task register some kind of event handler to be notified as other tasks touch the directory.

Then, at the end, remove everything you didn't get an event for.

Yes. This would probably (I don't know how Ansible is implemented internally) mess with the decouplling of modules a bit, but it would be so far removed from re-implementing puppet.

Which is why I'm posting this here - maybe, just maybe, somebody reads my plight and can bring up a discussion and maybe even a solution for this. Trust me: I'd so much rather use Ansible than puppet, it's crazy, but I also want to make sure that no stray file in /etc/sysctl.d will bring down a machine.

Yeah. This is probably the most words I've ever used for a feature request, but this one is really, really important for me which is why I'm so passionate about this. Ansible got so f'ing much right. It's such a shame to still be left unable to really use it.

Is this a case of xkcd1172? Maybe, but to me, my request seems reasonable. It's not? Enlighten me! It is? Great! Let's work on fixing this.

Read on →

Like many others, I couldn't wait for Apple to finally upgrade their MacPro and like many others, when they could finally be ordered, I queued up to get mine.

Last Monday, after two months of wait, the package finally arrived and I could start playing with it. I have to say: The thing is very impressive.

The hardware itself is very lightweight and compact. Compared to the old aluminium MacPro it was replacing, it felt even smaller than it is. Also, the box is nearly silent - so silent in fact, that now the hum of the dimmed background light in my old 30" Cinema Display is louder than the machine itself.

Speaking of that 30" display: It's using a dual-link DVI port. That means a special adapter is required to connect it to the new Thunderbolt ports - at least if you want to use a higher resolution than 1280x800 (which you definitely do).

The adapter is kinda difficult to get, especially as I totally forgot about it and I reall wanted to migrate to the new machine, so I had to look through local retail (only the one from Apple even remotely available) as opposed to Amazon (three other models available, some cheaper).

The device is huge by the way. I'm sure there's some electronics in there (especially when you consider that you have to plug it into a USB port for power), probably to split the full 2560x1600 pixels sent over Thunderbolt into two images of 1280x800, only to be reassembled in the display I guess.

The fact that there obviously is processing going on leaves a bit of a bad taste as it's one more component that could now break and, of course, there might be display lag or quality degradation.

At some time, there was for sure, if the adapters reviews are to be believed, but so far, I wasn't able to notice bad quality nor lag, but the fact that now there's one more active component involved in bringing me a picture makes me just a tad bit nervous.

Anyways - let's talk about some more pleasant things.

One is the WiFi: With the old MacPro I had peak transfer of about 3 MBytes/s which was just barely good enough for me to not wanting to go through the trouble of laying cable, even though it really pissed me off at times.

On the new Pro, I reached 18 MBytes/s over the exact same WiFi yesterday which removes any need for ever considering installing a physial cable. Very handy. Remember: It's not a file server, it doesn't run a torrent client, it doesn't serve movies to my home network. The really large bulk transfers it does are mainly caused by Steam which clearly is the bottleneck here (it never manages to saturate my 150MBit/s downstream).

Another thing that really surprises me is the sleeping behavior of the box. Well, actually, the waking up behavior: When asleep, the thing wakes up instantly (less than a second) - never in my live have I seen such a quick waking up from sleep in a computer.

Yes. I'm waiting for the fan to spin down and all audible noise to go away, but still. Hit any key on the keyboard and the machine's back. We're talking "waking an iphone from sleep" speeds here.

It might be that the machine has multiple levels of sleep states, but the instant wake-up also happens after sleeping for over 12 hours at which point a deeper sleep would totally make sense if there was any.

What is strange though: I seem to be able to wake the machine by pinging it. Yes. I know about the bonjour proxy, but in this case, I'm pinging it directly by IP and it wakes up (the first ping has a roundtrip time for 500ish ms - yes. it wakes THAT quickly).

This leads me to believe that the machine might not actually be sleeping for real though because waking from a direct ping requires quite a bit more technology than waking from a WOL packet.

Somdeday, I'll play with tcpdump to learn what's going on here.

Performance-wise, I haven't done that much testing, but replaying a test Postgres database dump that takes 5ish minutes on a 2012 retina MacBook Pro completes in 1:12 minutes on the pro - pretty impressive.

And one last thing: When you get a machine as powerful as this, there's of course also the wish of playing a game or two on it. As I had one SSD dedicated to Bootcamp in the old Pro, I was curious whether I might be able to keep this setup: The built-in flash drive dedicated to MacOS and Windows on its own (the old one) dedicated SSD.

Now that we don't have internal drive bays any more, this might seem tricky, but yesterday, I managed to install Windows 8 nicely on that SSD after connecting it via Thunderbolt using this adapter (no affiliate code - I got the link straight from google).

I guess the fact that it's using Thunderbolt makes Windows think it's a built-in hard drive which is what makes this work: You're not allowed to install Windows on a portable drive due to licensing issues.

The adapter is not actually intended for use with arbitrary drives (it's an accessory to some Seagate portable drives), but it works totally well and is (physically) stable enough. I'll have to do a bit of benchmarking to see how much performance I lose compared to the old built-in solution, but it certainly doesn't feel any slower.

Overall, I'm really happy with my new toy. Yes, it's probably overpowered for my needs, but it's also cool has hell, it is the first MacPro I own where sleep works reliably (though I'm inclined to say that it works suspiciously well - it might be cheating) and the fact that bootcamp still works with a dedicated external drive makes me really happy too.

Read on →

Sensational AG is the company I founded together with a collegue back in 2000. Ever since then, we had a very nice combination of fun, interesting work and a very successful business.

We're a very small team - just four programmers, one business guy and a product designer. Me personally, I would love to keep the team as small and tightly-knit as possible as that brings huge advantages: No politics, a lot of freedoms for everybody and mind-blowing productivity.

I'm still amazed to see what we manage to do with our small team time and time again and yet still manage to keep the job fun. It's not just the stuff we do outside of immediate work, like UT2004 matches, Cola Double Blind Tests, sometimes hosting JSZurich and much more - it's also the work itself that we try to make as fun as possible for everybody.

Sure - sometimes, work just has to be done, but we try as much as possible to distribute the fun parts of the work between everybody. Nobody has to be a pure code monkey; nobody constanly pulls the "change this logo there for the customer" card (though, that card certainly exists to be pulled - we just try to distribute it).

Most of the work we do flows into our big eCommerce project: Whenever you order food in a restaurant here in Switzerland, if the restaurant is big enough for them to get the food delivered to them, the stuff you eat will have been ordered using the product of ours.

Whenever you visit a dentist, the things they put in your mouth likely have been ordered using the product of ours.

The work we do helps countless people daily to get their job done more quickly allowing them to go home earlier. The work we do is essential for the operations of many, many companies here in Switzerland, in Germany and in Austria.

From a technical perspective, the work we do is very interesting too: While the main part of the application is a web application, there are many components around it: Barcode Scanners, native Smartphone applications and our very own highly available cluster (real, physical hardware) that hosts the application for the majority of our customers.

When you work with us, you will have to deal with any of

  • The huge home-grown PHP (5.5)-application (its history goes back to 2004 and it has changed version controlling systems three times so far - from CVS to SVN to git)
  • Backend Jobs written in PHP, Ruby and JavaScript
  • Frontend-Code written in JavaScript and Coffee Script (heck, we were using XmlHttpRequest before using it was called AJAX)
  • Software for Barcode-Scanners written in C#, C and soon whatever we'd like to use on Android
  • Software to read data from USB barcode Scanners written in Objective-C and Delphi of all things
  • Puppet to manage our cluster of 5 physical and about 25 virtual machines (though if only I knew about Ansible when we started this)
  • git where all our code lives, some on github, some on our own server.
  • PostgreSQL which is our database of choice (constantly updated to be able to play with the latest and gratest toys^Wfeatures) and I love it so much that no day goes past where I don't try to convert people over :-)
  • Ubuntu Linux which is the underlying OS of our whole server infrastructure.

The platform we use to develop on is everybodys own choice. Everybody here uses Macs, but whatever you are most productive with is what you use.

All of the code that we work with daily is home-grown (minus some libraries, of course). We control all of it and we get to play with all the components of the big thing. No component can't be changed, though we certainly prefer changing some over the others :-)

Between the Cola tests and the technical versatility and challenges described above: If I can interest you, dear reader, to join our crazy productive team in order to improve one hell of a suite of applications, then now is your chance to join up: We need more people to join our team of developers.

You'd have to be able to live with the huge chunk of PHP code though as that's too big to migrate away from, no matter how much we'd all love to, but aside of that, chose your battles in any of the above list of technologies.

Also, if your particular problem is better solved in $LANGUAGE of your choice, feel free to just do it. Part of the secret behind our productivity is that we know our tools and know when to use them. Good code is readable in any language (though I'd hve to brush up my lisp if you chose to go that route).

Interested? I would love to hear from you at phofstetter@sensational.ch.

Read on →