In the summer of 2012, I had the great oportunity to clean up our hosting infrastructure. Instead of running many differently configured VMs, mostly one per customer, we started building a real redundant infrastructure with two really beefy physical database machines (yay) and quite many (22) virtual machines for caching, web app servers, file servers and so on.
All components are fully redundant, every box can fail without anybody really needing to do anything (one exception is the database - that's also redundant, but we fail over manually due to the huge cost in time to failback).
Of course you don't manage ~20 machines manually any more: Aside of the fact that it would be really painful to do for those that have to be configured in an identical way (the app servers come to mind), you also want to be able to quickly bring a new box online which means you don't have time to manually go through the hassle of configuring it.
So, In the summer of 2012, when we started working on this, we decided to go with puppet. We also considered Chef but their server was really complicated to set up and install and there was zero incentive for them to improve because that would, after all, disincentivse people from becoming customers of their hosted solutions (the joys of open-core).
Puppet is also commerically backed, but everything they do is available as open source and their approach for the central server is much more «batteries included» than what Chef has provided.
And finally, after playing around a bit with both Chef and puppet, we noticed that puppet was way more bitchy and less tolerant of quick hacks around issues which felt like a good thing for people dabbling with HA configuration of a multi machine cluster for the first time.
Fast forward one year: Last autumn I found out about ansible (linking to their github page - their website reads like a competition in buzzword-bingo) and after reading their documentation, I immediately was convinced:
- No need to install an agent on managed machines
- Trivial to bootstrap machines (due to above point)
- Contributors don't need to sign a CLA (thank you so much, ansibleworks!)
- No need to manually define dependencies of tasks: Tasks are run requentially
- Built-in support for cowsay by default
- Many often-used modules included by default, no hunting for, say, a
sysctlmodule on github
- Very nice support for rolling updates
- Also providing a means to quickly do one-off tasks
- Very easy to make configuration entries based on the host inventory (which requires puppetdb and an external database in the case of puppet)
Because ansible connects to each machine individually via SSH, running it against a full cluster of machines is going to take a bit longer than with puppet, but our cluster is small, so that wasn't that much of a deterrent.
So last Sunday evening I started working on porting our configuration over from puppet to Ansible and after getting used to the YAML syntax of the playbooks, I made very quick progress.
Again, I'd like to point out the excellent, built-in, on-by-default support for cowsay as one of the killer-features that made me seriously consider starting the porting effort.
Unfortunately though, after a very promising start, I had to come to the conclusion that we will be sticking with puppet for the time being because there's one single feature that Ansible doesn't have and that I really, really want a configuration management system to have:
I'ts not possible in Ansible to tell it to keep a directory clean of files not managed by Ansible in some way
There are, of course, workarounds, but they come at a price too high for me to be willing to pay.
You could first clean a directory completely using a shell command, but this will lead to ansible detecting a change to that folder every time it runs which will cause server restarts, even when they are not needed.
You could do something like this stack overflow question but this has the disadvantage that it forces you into a configuration file specific playbook design instead of a role specific one.
What I mean is that using the second workaround, you can only have one playbook
touching that folder. But imagine for example a case where you want to work with
/etc/sysctl.d: A generic role would put some stuff there, but then your
firewall role might put more stuff there (to enable ip forwarding) and your
database role might want to add other stuff (like tweaking shmmax and shmall,
though that's thankfully not needed any more in current Postgres releases).
So suddenly your
/etc/sysctl.d role needs to know about firewalls and
databases which totally violates the really nice separation of concerns between
roles. Instead of having a firewall and a database role both doing something to
/etc/sysctl.d, you know need a sysctl-role which does different things
depending on what other roles a machine has.
Or, of course, you just don't care that stray files never get removed, but
honestly: Do you really want to live with the fact that your
/etc/sudoers.d can contain files not managed by ansible and likely not
intended to be there? Both sysctl.d and sudoers.d are more than capable of doing
immense damage to your boxes and this sneakily behind the watching eye of your
configuration management system?
For me that's inacceptable.
So despite all the nice advantages (like cowsay), this one feature is something that I really need and can't have right now and which, thus, forces me to stay away from Ansible for now.
It's a shame.
Some people tell me that implementing my feature would require puppet's feature of building a full state of a machine before doing anything (which is error- prone and frustrating for users at times), but that's not really true.
If ansible modules could talk to each other - maybe loosly coupled by firing some events as they do stuff, you could just name the task that makes sure the directory exists first and then have that task register some kind of event handler to be notified as other tasks touch the directory.
Then, at the end, remove everything you didn't get an event for.
Yes. This would probably (I don't know how Ansible is implemented internally) mess with the decouplling of modules a bit, but it would be so far removed from re-implementing puppet.
Which is why I'm posting this here - maybe, just maybe, somebody reads my plight
and can bring up a discussion and maybe even a solution for this. Trust me: I'd
so much rather use Ansible than puppet, it's crazy, but I also want to make sure
that no stray file in
/etc/sysctl.d will bring down a machine.
Yeah. This is probably the most words I've ever used for a feature request, but this one is really, really important for me which is why I'm so passionate about this. Ansible got so f'ing much right. It's such a shame to still be left unable to really use it.
Is this a case of xkcd1172? Maybe, but to me, my request seems reasonable. It's not? Enlighten me! It is? Great! Let's work on fixing this.Read on →
the new (2013) MacPro
Like many others, I couldn't wait for Apple to finally upgrade their MacPro and like many others, when they could finally be ordered, I queued up to get mine.
Last Monday, after two months of wait, the package finally arrived and I could start playing with it. I have to say: The thing is very impressive.
The hardware itself is very lightweight and compact. Compared to the old aluminium MacPro it was replacing, it felt even smaller than it is. Also, the box is nearly silent - so silent in fact, that now the hum of the dimmed background light in my old 30" Cinema Display is louder than the machine itself.
Speaking of that 30" display: It's using a dual-link DVI port. That means a special adapter is required to connect it to the new Thunderbolt ports - at least if you want to use a higher resolution than 1280x800 (which you definitely do).
The adapter is kinda difficult to get, especially as I totally forgot about it and I reall wanted to migrate to the new machine, so I had to look through local retail (only the one from Apple even remotely available) as opposed to Amazon (three other models available, some cheaper).
The device is huge by the way. I'm sure there's some electronics in there (especially when you consider that you have to plug it into a USB port for power), probably to split the full 2560x1600 pixels sent over Thunderbolt into two images of 1280x800, only to be reassembled in the display I guess.
The fact that there obviously is processing going on leaves a bit of a bad taste as it's one more component that could now break and, of course, there might be display lag or quality degradation.
At some time, there was for sure, if the adapters reviews are to be believed, but so far, I wasn't able to notice bad quality nor lag, but the fact that now there's one more active component involved in bringing me a picture makes me just a tad bit nervous.
Anyways - let's talk about some more pleasant things.
One is the WiFi: With the old MacPro I had peak transfer of about 3 MBytes/s which was just barely good enough for me to not wanting to go through the trouble of laying cable, even though it really pissed me off at times.
On the new Pro, I reached 18 MBytes/s over the exact same WiFi yesterday which removes any need for ever considering installing a physial cable. Very handy. Remember: It's not a file server, it doesn't run a torrent client, it doesn't serve movies to my home network. The really large bulk transfers it does are mainly caused by Steam which clearly is the bottleneck here (it never manages to saturate my 150MBit/s downstream).
Another thing that really surprises me is the sleeping behavior of the box. Well, actually, the waking up behavior: When asleep, the thing wakes up instantly (less than a second) - never in my live have I seen such a quick waking up from sleep in a computer.
Yes. I'm waiting for the fan to spin down and all audible noise to go away, but still. Hit any key on the keyboard and the machine's back. We're talking "waking an iphone from sleep" speeds here.
It might be that the machine has multiple levels of sleep states, but the instant wake-up also happens after sleeping for over 12 hours at which point a deeper sleep would totally make sense if there was any.
What is strange though: I seem to be able to wake the machine by pinging it. Yes. I know about the bonjour proxy, but in this case, I'm pinging it directly by IP and it wakes up (the first ping has a roundtrip time for 500ish ms - yes. it wakes THAT quickly).
This leads me to believe that the machine might not actually be sleeping for real though because waking from a direct ping requires quite a bit more technology than waking from a WOL packet.
Somdeday, I'll play with
tcpdump to learn what's going on here.
Performance-wise, I haven't done that much testing, but replaying a test Postgres database dump that takes 5ish minutes on a 2012 retina MacBook Pro completes in 1:12 minutes on the pro - pretty impressive.
And one last thing: When you get a machine as powerful as this, there's of course also the wish of playing a game or two on it. As I had one SSD dedicated to Bootcamp in the old Pro, I was curious whether I might be able to keep this setup: The built-in flash drive dedicated to MacOS and Windows on its own (the old one) dedicated SSD.
Now that we don't have internal drive bays any more, this might seem tricky, but yesterday, I managed to install Windows 8 nicely on that SSD after connecting it via Thunderbolt using this adapter (no affiliate code - I got the link straight from google).
I guess the fact that it's using Thunderbolt makes Windows think it's a built-in hard drive which is what makes this work: You're not allowed to install Windows on a portable drive due to licensing issues.
The adapter is not actually intended for use with arbitrary drives (it's an accessory to some Seagate portable drives), but it works totally well and is (physically) stable enough. I'll have to do a bit of benchmarking to see how much performance I lose compared to the old built-in solution, but it certainly doesn't feel any slower.
Overall, I'm really happy with my new toy. Yes, it's
probably overpowered for my needs, but it's also cool has hell, it is the first MacPro I own where sleep works reliably (though I'm inclined to say that it works suspiciously well - it might be cheating) and the fact that bootcamp still works with a dedicated external drive makes me really happy too.
sensational ag is hiring
Sensational AG is the company I founded together with a collegue back in 2000. Ever since then, we had a very nice combination of fun, interesting work and a very successful business.
We're a very small team - just four programmers, one business guy and a product designer. Me personally, I would love to keep the team as small and tightly-knit as possible as that brings huge advantages: No politics, a lot of freedoms for everybody and mind-blowing productivity.
I'm still amazed to see what we manage to do with our small team time and time again and yet still manage to keep the job fun. It's not just the stuff we do outside of immediate work, like UT2004 matches, Cola Double Blind Tests, sometimes hosting JSZurich and much more - it's also the work itself that we try to make as fun as possible for everybody.
Sure - sometimes, work just has to be done, but we try as much as possible to distribute the fun parts of the work between everybody. Nobody has to be a pure code monkey; nobody constanly pulls the "change this logo there for the customer" card (though, that card certainly exists to be pulled - we just try to distribute it).
Most of the work we do flows into our big eCommerce project: Whenever you order food in a restaurant here in Switzerland, if the restaurant is big enough for them to get the food delivered to them, the stuff you eat will have been ordered using the product of ours.
Whenever you visit a dentist, the things they put in your mouth likely have been ordered using the product of ours.
The work we do helps countless people daily to get their job done more quickly allowing them to go home earlier. The work we do is essential for the operations of many, many companies here in Switzerland, in Germany and in Austria.
From a technical perspective, the work we do is very interesting too: While the main part of the application is a web application, there are many components around it: Barcode Scanners, native Smartphone applications and our very own highly available cluster (real, physical hardware) that hosts the application for the majority of our customers.
When you work with us, you will have to deal with any of
- The huge home-grown PHP (5.5)-application (its history goes back to 2004 and it has changed version controlling systems three times so far - from CVS to SVN to git)
- Software for Barcode-Scanners written in C#, C and soon whatever we'd like to use on Android
- Software to read data from USB barcode Scanners written in Objective-C and Delphi of all things
- Puppet to manage our cluster of 5 physical and about 25 virtual machines (though if only I knew about Ansible when we started this)
- git where all our code lives, some on github, some on our own server.
- PostgreSQL which is our database of choice (constantly updated to be able to play with the latest and gratest toys^Wfeatures) and I love it so much that no day goes past where I don't try to convert people over :-)
- Ubuntu Linux which is the underlying OS of our whole server infrastructure.
The platform we use to develop on is everybodys own choice. Everybody here uses Macs, but whatever you are most productive with is what you use.
All of the code that we work with daily is home-grown (minus some libraries, of course). We control all of it and we get to play with all the components of the big thing. No component can't be changed, though we certainly prefer changing some over the others :-)
Between the Cola tests and the technical versatility and challenges described above: If I can interest you, dear reader, to join our crazy productive team in order to improve one hell of a suite of applications, then now is your chance to join up: We need more people to join our team of developers.
You'd have to be able to live with the huge chunk of PHP code though as that's too big to migrate away from, no matter how much we'd all love to, but aside of that, chose your battles in any of the above list of technologies.
Also, if your particular problem is better solved in $LANGUAGE of your choice, feel free to just do it. Part of the secret behind our productivity is that we know our tools and know when to use them. Good code is readable in any language (though I'd hve to brush up my lisp if you chose to go that route).
Interested? I would love to hear from you at firstname.lastname@example.org.Read on →
pdo_pgsql needs some love
Today, PostgreSQL 9.3 was released. September is always the month of PostgreSQL as every September a new Major Release with awesome new feature is released and every September I have to fight the urgue to run and immediately update the production systems to the new version of my favorite toy
As every year, I want to talk the awesome guys (and girls I hope) that make PostgreSQL one of my favorite pieces of software overall and for certain my most favorite database system.
That said, there's another aspect of PostgreSQL that needs some serious love: While back in the days PHP was known for its robust database client libraries, over time other language environments have caught up and long since surpassed what's possible in PHP.
To be honest, the PostgreSQL client libraries as they are currently available in PHP are in serious need of some love.
Both solutions are, unfortunately, quite inadequate solutions that fail to expose most of the awesomeness that is PostgreSQL to the user:
On the positive side, being a small wrapper around libpq, the pgsql
extension knows quite a bit about Postgres' internals: It has excellent
support for COPY, it knows about a result sets data types (but doesn't
use that knowledge as you'll see below), it has
to correctly quote identifiers, it support asynchronous queries and it
But, while pgsql knows a lot about Postgres' specifics, to this day,
pg_fetch_* functions convert all columns into strings. Numeric
types? String. Dates? String. Booleans? Yes. String too ('t' or 'f',
both trueish values to PHP).
To this day, while the extension supports prepared statements, their use is terribly inconvenient, forcing you to name your statements and to manually free them.
To this day, the
pg_fetch_* functions load the whole result set into
an internal buffer, making it impossible to stream results out to the
client using an iterator pattern. Well. Of course it's still possible,
but you waste the memory for that internal buffer, forcing you to
manually play with DECLARE CURSOR and friends.
There is zero support for advanced data types in Postgres and the library doesn't help at all with todays best practices for accessing a database (prepared statements).
There are other things that make the extension unpractical for me, but they are not the extensions fault, so I won't spend any time explaining them here (like the lack of support by newrelic - but, as I said, that's not the extensions fault)
pdo_pgsql gets a lot of stuff right that the pgsql extension doesn't: It doesn't read the whole result set into memory, it knows a bit about data types, preserving numbers and booleans and, being a PDO driver, it follows the generic PDO paradigms, giving a unified API with other PDO modules.
It also has good support for prepared statements (not perfect, but that's PDOs fault).
But it also has some warts:
- There's no way to safely quote an identifier. Yes. That's a PDO shortcoming, but still. It should be there.
- While it knows about numbers and booleans, it doesn't know about any of the other more advanced data types.
- Getting metadata about a query result actually makes it query the
database - once per column, even though the information is right there
in libpq, available to use (look at the
PDOStatement::getColumnMeta). This makes it impossible to fix above issue in userland.
- It has zero support for COPY
Imagine the joy of having a pdo_pgsql that actually cares about Postgres. Imagine how selecting a JSON column would give you its data already decoded, ready to use in userland (or at least an option to).
Imagine how selecting dates would at least give you the option of
getting them as a
DateTime (there's loss of precision though -
TIMESTAMP has more precision than
Imagine how selecting an array type in postgres would actually give you
back an array in PHP. The string that you have to deal with now is
notoriously hard to parse. Yes. There now is
Postgres, but hat shouldn't be needed.
Imagine how selecting a HSTORE would give you an associative array.
Imagine using COPY with pdo_pgsql for very quickly moving bulk data.
Imagine the new features of
PGResult being exposed to userland.
Giving applications the ability to detect what constraint was just
violated (very handy to detect whether it's safe to retry).
Wouldn't that be fun? Wouldn't that save us from having to type so much boilerplate all day?
Honestly, what I think should happen is somebody should create a
pdo_pgsql2 that breaks backwards compatibility and adds all these
getColumnMeta just return the OID instead of querying the
database. Have a
quoteIdentifier method (yes. That should be in PDO
itself, but let's fix it where we can).
fetch() return Arrays or Objects for JSON columns. Have it
return Arrays for arrays and HSTOREs. Have it optionally return
DateTimes instead of strings.
Wouldn't that be great?
Unfortunately, while I can write some C, I'm not nearly good enough to produce something that I could live with other people using, so any progress I can achieve will be slow.
I'm also unsure of whether this would ever have a chance to land in PHP itself. Internals are very adverse to adding new features to stuff that already "works" and no matter how good the proposal, you need a very thick skin if you want to ever get something merged, no matter whether you can actually offer patches or not.
Would people be using an external pdo_pgsql2? Would it have a chance as a pecl extension? Do other people see a need for this? Is somebody willing to help me? I really think something needs to be done and I'm willing to get my hands dirty - I just have my doubts about the quality of the result I'm capable of producing. But I can certainly try.
And I will.Read on →