If I had to make a list of attributes I would like the ISP of my dream to have, then, I could write quite the list:

  • I would really like to have native IPv6 support. Yes. IPv4 will be sufficient for a very long time, but unless pepole start having access to IPv6, it'll never see the wide deployment it needs if we want the internet to continue to grow. An internet where addresses are only available to people with a lot of money is not an internet we all want to be subjected to (see my post «asking for permission»)
  • I would want my ISP to accept or even support network neutrality. For this to be possible, the ISP of my dreams would need to be nothing but an ISP so their motivations (provide better service) align with mine (getting better service). ISPs who also sell content have all the motivation to provide crappy Internet service in order to better sell their (higher-margin) content.
  • If I have technical issues, I want to be treated as somebody who obviously has a certain level of technical knowledge. I'm by no means an expert in networking technology, but I do know about powering it off and on again. If I have to say «shibboleet» to get to a real technicial, so be it, but if that's not needed, that's even better.
  • The networking technology involved in getting me the connectivity I want should be widely available and thus easily replacable if something breaks.
  • The networking technology involved should be as simple as possible: The more complex the hardware involved, the more stuff can break, especially when you combine cost-pressure for end-users with the need for high complexity.
  • The network equipment I'm installing at my home and which has thus access to my LAN needs to be equipment I own and I control fully. I do not accept leased equipment to which I do not have full access to.
  • And last but not least, I would really like to have as much bandwidth as possible

I'm sure I'm not alone with these wishes, even though, for «normal people» they might seem strange.

But honestly: They just don't know it, but they too have the same interests. Nobody wants an internet that works like TV where you pay for access to a curated small list of "approved" sites (see network neutrality and IPv6 support).

Nobody wants to get up and reboot their modem here and then because it crashed. Nobody wants to be charged with downloading illegal content because their Wifi equipment was suddenly repurposed as an open access point for other customers of an ISP.

Most of the wishes I list above are the basis needed for these horror scenarios never coming to pass, however unlikely the might seem now (though getting up and rebooting the modem/router is something we already have to deal with today).

So yes. While it's getting rarer and rarer to get all the points of my list fulfilled, to the point where I though this to be impossible to get all of it, I'm happy to say that here in Switzerland, there is at least one ISP that does all of this and more.

I'm talking about Init7 and especially their awesome FTTH offering Fiber7 which very recently became available in my area.

Let's deal with the technology aspect first as this really isn't the important point of this post: What you get from them is pure 1Gbit/s Ethernet. Yes, they do sell you a router box if you want one, but you can just as well just get a simple media converter, or just an SFP module to plug into any (managed) switch (with SFP port).

If you have your own routing equipment, be it a linux router like my shion or be it any Wifi Router, there's no need to add any kind of additional complexity to your setup.

No additional component that can crash, no software running in your home to which you don't have your password to and certainly no sneakily opened public WLANs (I'm looking at you, cablecom).

Of course you get native IPv6 (a /48 which incidentally is room for 281474976710656 whole internets in your apartment) too.

But what's really remarkable about Init7 isn't the technical aspect (though, again, it's bloody amazing), but everything else:

  • Init7 was one of the first ISPs in Switzerland to offer IPv6 to end users.
  • Init7 doesn't just support network neutrality. They actively fight for it
  • They explicitly state that they are not selling content and they don't intend to start doing so. They are just an ISP and as such their motivations totally align with mine.

There are a lot of geeky soft factors too:

  • Their press releases are written in Open Office (check the PDF properties of this one for example)
  • I got an email from a technical person on their end that was written using f'ing Claws Mail on Linux
  • Judging from the Recieved headers of their Email, they are using IPv6 in their internal LAN - down to the desktop workstations. And related to that:
  • The machines in their LAN respond to ICMPv6 pings which is utterly crazy cool. Yes. They are firewalled (cough I had to try. Sorry.), but they let ICMP through. For the not as technical readers here: This is as good an internet citizen as you will ever see and it's extremely unexpected these days.

If you are a geek like me and if your ideals align with the ones I listed above, there is no question: You have to support them. If you can have their Fiber offering in your area, this is a no-brainer. You can't get synchronous 1GBit/s for CHF 64ish per month anywhere else and even if you did, it wouldn't be plain Ethernet either.

If you can't have their fiber offering, it's still worth considering their other offers. They do have some DSL based plans which of course are technically inferior to plain ethernet over fiber, but you would still support one of the few remaining pure ISPs.

It doesn't have to be Init7 either. For all I know there are many others, maybe even here in Switzerland. Init7 is what I decided to go with initially because of the Gbit, but the more I leared about their philosophy, the less important the bandwith got.

We need to support companies like these because companies like these are what ensures that the internet of the future will be as awesome as the internet is today.

Read on →

A few months ago, the awesome provider Init7 has released their awesome FTTH offering Fiber7 which provides synchronous 1GBit/s access for a very fair price. Actually, they are by far the cheapest provider for this kind of bandwith.

Only cablecom comes close at matching them bandwidth wise with their 250Mbits package, but that's 4 times less bandwith for nearly double the price. Init7 also is one of the only providers who officially states that their triple-play strategy is that they don't do it. Huge-ass kudos for that.

Also, their technical support is using Claws Mail on GNU/Linux - to give you some indication of the geek-heaven you get when signing up with them.

But what's really exciting about Init7 is their support for IPv6. In-fact, Init7 was one of the first (if not the first) providers to offer IPv6 for end users. Also, we're talking about a real, non-tunneled, no strings attached plain /48.

In case that doesn't ring a bell, a /48 will allow for 216 networks consisting of 264 hosts each. Yes. That's that many hosts.

In eager anticipation of getting this at home natively (of course I ordered Fiber7 the moment I could at my place), I decided to play with IPv6 as far as I could with my current provider, which apparently lives in the stone-age and still doesn't provide native v6 support.

After getting abysmal pings using 6to4 about a year ago, this time I decided to go with tunnelbroker which these days also provides a nice dyndns-alike API for updating the public tunnel endpoint.

Let me tell you: Setting this up is trivial.

Tunnelbroker provides you with all the information you need for your tunnel and with the prefix of the /64 you get from them and setting up for your own network is trivial using radvd.

The only thing that's different from your old v4 config: All your hosts will immediately be accessible from the public internet, so you might want to configure a firewall from the get-go - but see later for some thoughts in that matter.

But this isn't any different from the NAT solutions we have currently. Instead of configuring port forwarding, you just open ports on your router, but the process is more or less the same.

If you need direct connectivity however, you can now have it. No strings attached.

So far, I've used devices running iOS 7 and 8, Mac OS X 10.9 and 10.10, Windows XP, 7 and 8 and none of them had any trouble reaching the v6 internet. Also, I would argue that configuring radvd is easier than configuring DHCP. There's less thought involved for assigning addresses because autoconfiguration will just deal with that.

For me, I had to adjust how I'm thinking about my network for a bit and I'm posting here in order to explain what change you'll get with v6 and how some paradigms change. Once you've accepted these changes, using v6 is trivial and totally something you can get used to.

  • Multi-homing (multiple adresses per interface) was something you've rarely done in v4. Now in v6, you do that all the time. Your OSes go as far as to grab a new random one every few connections in order to provide a means of privacy.
  • The addresses are so long and hex-y - you probably will never remember them. But that's ok. In general, there are much fewer cases where you worry about the address.
    • Because of multi-homing every machine has a guaranteed static address (built from the MAC address of the interface) by default, so there's no need to statically assign addresses in many cases.
    • If you want to assign static addresses, just pick any in your /64. Unless you manually hand out the same address to two machines, autoconfiguration will make sure no two machines pick the same address. In order to remember them, feel free to use cute names - finally you got some letters and leetspeak to play with.
    • To assign a static address, just do it on the host in question. Again, autoconfig will make sure no other machine gets the same address.
  • And with Zeroconf (avahi / bonjour), you have fewer and fewer oportunities to deal with anything that's not a host-name anyways.
  • You will need a firewall because suddenly all your machines will be accessible for the whole internet. You might get away with just the local personal firewall, but you probably should have one on your gateway.
  • While that sounds like higher complexity, I would argue that the complexity is lower because if you were a responsible sysadmin, you were dealing with both NAT and a firewall whereas with v6, a firewall is all you need.
  • Tools like nat-pmp or upnp don't support v6 yet as far as I can see, so applications in the trusted network can't yet punch holes in the firewall (what is the equivalent thing to forwarding ports in the v4 days).

Overall, getting v6 running is really simple and once you adjust your mindset a bit, while stuff is unusual and taking some getting-used-to, I really don't see v6 as being more complicated. Quite to the contrary actually.

As I'm thinking about firewalls and opening ports, actually, as hosts get wiser about v6, you actually really might get away without a strict firewall as hosts could grab a new random v6 address for every connection they want to use and then they would just bind their servers to that address.

Services binding to all addresses would never bind to these temporary addresses.

That way none of the services brought up by default (you know - all those ports open on your machine when it runs) would be reachable from the outside. What would be reachable is the temporary addresses grabbed by specific services running on your machine.

Yes. An attacker could port-scan your /64 and try to find the non-temporary address, but keep in mind that finding that one address out of 264 addresses would mean that you have to port-scan 4 billion traditional v4 internets per attack target (good luck) or randomly guessing with an average chance of 1:263 (also good luck).

Even then a personal firewall could block all unsolicited packets from non-local prefixes to provide even more security.

As such, we really might get away without actually needing a firewall at the gateway to begin with which will actually go great lengths at providing the ubiquitous configuration-free p2p connectivity that would be ever-so-cool and which we have lost over the last few decades.

Me personally, I'm really happy to see how simple v6 actually is to get implemented and I'm really looking forward to my very own native /48 which I'm probably going to get somehwere in September/October-ish.

Until then, I'll gladly play with my tunneled /64 (for now still firewalled, but I'll investigate into how OS X and Windows deal with the temporary addresses they use which might allow me to actually turn the firewall off).

Read on →

Last autumn, I was talking about how I would like to see pdo_pgsql for PHP to be improved.

Over the last few days I had time to seriously start looking into making sure I get my wish. Even though my C is very rusty and I have next to no experience in dealing with the PHP/Zend API, I made quite a bit of progress over the last few days.

First, JSON support

screenshot showing off json support

If you have the json extension enabled in your PHP install (it's enabled by default), then any column of data type json will be automatically parsed and returned to you as an array.

No need to constantly repeat yourself with json_parse(). This works, of course, with directly selected json columns or with any expression that returns json (like array_to_json or the direct typecast shown in the screenshot).

This is off by default and can be enabled on a per-connection or a per- statement level as to not break backwards compatibility (I'll need it off until I get a chance to clean up PopScan for example).

Next, array support:

screenshot showing off array support

Just like with JSON, this will automatically turn any array expression (of the built-in array types) into an array to use from PHP.

As I'm writing this blog entry here, this only works for text[] and it's always enabled.

Once I have an elegant way to deal with the various types of arrays and convert them into the correct PHP types, I'll work on making this turnoffable (technical term) too.

I'll probably combine this and the automatic JSON parsing into just one setting which will include various extended data types both Postgres and PHP know about.

Once I've done that, I'll look into more points on my wishlist (better error reporting with 9.3 and later and a way to quote identifiers comes to mind) and then I'll probably try to write a proper RFC and propose this for inclusion into PHP itself (though don't get your hopes up - they are a conservative bunch).

If you want to follow along with my work, have a look at my pdo_pgsql-improvements branch on github (tracks to PHP-5.5)

Read on →

In the summer of 2012, I had the great oportunity to clean up our hosting infrastructure. Instead of running many differently configured VMs, mostly one per customer, we started building a real redundant infrastructure with two really beefy physical database machines (yay) and quite many (22) virtual machines for caching, web app servers, file servers and so on.

All components are fully redundant, every box can fail without anybody really needing to do anything (one exception is the database - that's also redundant, but we fail over manually due to the huge cost in time to failback).

Of course you don't manage ~20 machines manually any more: Aside of the fact that it would be really painful to do for those that have to be configured in an identical way (the app servers come to mind), you also want to be able to quickly bring a new box online which means you don't have time to manually go through the hassle of configuring it.

So, In the summer of 2012, when we started working on this, we decided to go with puppet. We also considered Chef but their server was really complicated to set up and install and there was zero incentive for them to improve because that would, after all, disincentivse people from becoming customers of their hosted solutions (the joys of open-core).

Puppet is also commerically backed, but everything they do is available as open source and their approach for the central server is much more «batteries included» than what Chef has provided.

And finally, after playing around a bit with both Chef and puppet, we noticed that puppet was way more bitchy and less tolerant of quick hacks around issues which felt like a good thing for people dabbling with HA configuration of a multi machine cluster for the first time.

Fast forward one year: Last autumn I found out about ansible (linking to their github page - their website reads like a competition in buzzword-bingo) and after reading their documentation, I immediately was convinced:

  • No need to install an agent on managed machines
  • Trivial to bootstrap machines (due to above point)
  • Contributors don't need to sign a CLA (thank you so much, ansibleworks!)
  • No need to manually define dependencies of tasks: Tasks are run requentially
  • Built-in support for cowsay by default
  • Many often-used modules included by default, no hunting for, say, a sysctl module on github
  • Very nice support for rolling updates
  • Also providing a means to quickly do one-off tasks
  • Very easy to make configuration entries based on the host inventory (which requires puppetdb and an external database in the case of puppet)

Because ansible connects to each machine individually via SSH, running it against a full cluster of machines is going to take a bit longer than with puppet, but our cluster is small, so that wasn't that much of a deterrent.

So last Sunday evening I started working on porting our configuration over from puppet to Ansible and after getting used to the YAML syntax of the playbooks, I made very quick progress.

progress

Again, I'd like to point out the excellent, built-in, on-by-default support for cowsay as one of the killer-features that made me seriously consider starting the porting effort.

Unfortunately though, after a very promising start, I had to come to the conclusion that we will be sticking with puppet for the time being because there's one single feature that Ansible doesn't have and that I really, really want a configuration management system to have:

I'ts not possible in Ansible to tell it to keep a directory clean of files not managed by Ansible in some way

There are, of course, workarounds, but they come at a price too high for me to be willing to pay.

  • You could first clean a directory completely using a shell command, but this will lead to ansible detecting a change to that folder every time it runs which will cause server restarts, even when they are not needed.

  • You could do something like this stack overflow question but this has the disadvantage that it forces you into a configuration file specific playbook design instead of a role specific one.

What I mean is that using the second workaround, you can only have one playbook touching that folder. But imagine for example a case where you want to work with /etc/sysctl.d: A generic role would put some stuff there, but then your firewall role might put more stuff there (to enable ip forwarding) and your database role might want to add other stuff (like tweaking shmmax and shmall, though that's thankfully not needed any more in current Postgres releases).

So suddenly your /etc/sysctl.d role needs to know about firewalls and databases which totally violates the really nice separation of concerns between roles. Instead of having a firewall and a database role both doing something to /etc/sysctl.d, you know need a sysctl-role which does different things depending on what other roles a machine has.

Or, of course, you just don't care that stray files never get removed, but honestly: Do you really want to live with the fact that your /etc/sysctl.d, or worse, /etc/sudoers.d can contain files not managed by ansible and likely not intended to be there? Both sysctl.d and sudoers.d are more than capable of doing immense damage to your boxes and this sneakily behind the watching eye of your configuration management system?

For me that's inacceptable.

So despite all the nice advantages (like cowsay), this one feature is something that I really need and can't have right now and which, thus, forces me to stay away from Ansible for now.

It's a shame.

Some people tell me that implementing my feature would require puppet's feature of building a full state of a machine before doing anything (which is error- prone and frustrating for users at times), but that's not really true.

If ansible modules could talk to each other - maybe loosly coupled by firing some events as they do stuff, you could just name the task that makes sure the directory exists first and then have that task register some kind of event handler to be notified as other tasks touch the directory.

Then, at the end, remove everything you didn't get an event for.

Yes. This would probably (I don't know how Ansible is implemented internally) mess with the decouplling of modules a bit, but it would be so far removed from re-implementing puppet.

Which is why I'm posting this here - maybe, just maybe, somebody reads my plight and can bring up a discussion and maybe even a solution for this. Trust me: I'd so much rather use Ansible than puppet, it's crazy, but I also want to make sure that no stray file in /etc/sysctl.d will bring down a machine.

Yeah. This is probably the most words I've ever used for a feature request, but this one is really, really important for me which is why I'm so passionate about this. Ansible got so f'ing much right. It's such a shame to still be left unable to really use it.

Is this a case of xkcd1172? Maybe, but to me, my request seems reasonable. It's not? Enlighten me! It is? Great! Let's work on fixing this.

Read on →