sensational ag is hiring
Sensational AG is the company I founded together with a collegue back in 2000. Ever since then, we had a very nice combination of fun, interesting work and a very successful business.
We're a very small team - just four programmers, one business guy and a product designer. Me personally, I would love to keep the team as small and tightly-knit as possible as that brings huge advantages: No politics, a lot of freedoms for everybody and mind-blowing productivity.
I'm still amazed to see what we manage to do with our small team time and time again and yet still manage to keep the job fun. It's not just the stuff we do outside of immediate work, like UT2004 matches, Cola Double Blind Tests, sometimes hosting JSZurich and much more - it's also the work itself that we try to make as fun as possible for everybody.
Sure - sometimes, work just has to be done, but we try as much as possible to distribute the fun parts of the work between everybody. Nobody has to be a pure code monkey; nobody constanly pulls the "change this logo there for the customer" card (though, that card certainly exists to be pulled - we just try to distribute it).
Most of the work we do flows into our big eCommerce project: Whenever you order food in a restaurant here in Switzerland, if the restaurant is big enough for them to get the food delivered to them, the stuff you eat will have been ordered using the product of ours.
Whenever you visit a dentist, the things they put in your mouth likely have been ordered using the product of ours.
The work we do helps countless people daily to get their job done more quickly allowing them to go home earlier. The work we do is essential for the operations of many, many companies here in Switzerland, in Germany and in Austria.
From a technical perspective, the work we do is very interesting too: While the main part of the application is a web application, there are many components around it: Barcode Scanners, native Smartphone applications and our very own highly available cluster (real, physical hardware) that hosts the application for the majority of our customers.
When you work with us, you will have to deal with any of
- The huge home-grown PHP (5.5)-application (its history goes back to 2004 and it has changed version controlling systems three times so far - from CVS to SVN to git)
- Software for Barcode-Scanners written in C#, C and soon whatever we'd like to use on Android
- Software to read data from USB barcode Scanners written in Objective-C and Delphi of all things
- Puppet to manage our cluster of 5 physical and about 25 virtual machines (though if only I knew about Ansible when we started this)
- git where all our code lives, some on github, some on our own server.
- PostgreSQL which is our database of choice (constantly updated to be able to play with the latest and gratest toys^Wfeatures) and I love it so much that no day goes past where I don't try to convert people over :-)
- Ubuntu Linux which is the underlying OS of our whole server infrastructure.
The platform we use to develop on is everybodys own choice. Everybody here uses Macs, but whatever you are most productive with is what you use.
All of the code that we work with daily is home-grown (minus some libraries, of course). We control all of it and we get to play with all the components of the big thing. No component can't be changed, though we certainly prefer changing some over the others :-)
Between the Cola tests and the technical versatility and challenges described above: If I can interest you, dear reader, to join our crazy productive team in order to improve one hell of a suite of applications, then now is your chance to join up: We need more people to join our team of developers.
You'd have to be able to live with the huge chunk of PHP code though as that's too big to migrate away from, no matter how much we'd all love to, but aside of that, chose your battles in any of the above list of technologies.
Also, if your particular problem is better solved in $LANGUAGE of your choice, feel free to just do it. Part of the secret behind our productivity is that we know our tools and know when to use them. Good code is readable in any language (though I'd hve to brush up my lisp if you chose to go that route).
Interested? I would love to hear from you at firstname.lastname@example.org.Read on →
pdo_pgsql needs some love
Today, PostgreSQL 9.3 was released. September is always the month of PostgreSQL as every September a new Major Release with awesome new feature is released and every September I have to fight the urgue to run and immediately update the production systems to the new version of my favorite toy
As every year, I want to talk the awesome guys (and girls I hope) that make PostgreSQL one of my favorite pieces of software overall and for certain my most favorite database system.
That said, there's another aspect of PostgreSQL that needs some serious love: While back in the days PHP was known for its robust database client libraries, over time other language environments have caught up and long since surpassed what's possible in PHP.
To be honest, the PostgreSQL client libraries as they are currently available in PHP are in serious need of some love.
Both solutions are, unfortunately, quite inadequate solutions that fail to expose most of the awesomeness that is PostgreSQL to the user:
On the positive side, being a small wrapper around libpq, the pgsql
extension knows quite a bit about Postgres' internals: It has excellent
support for COPY, it knows about a result sets data types (but doesn't
use that knowledge as you'll see below), it has
to correctly quote identifiers, it support asynchronous queries and it
But, while pgsql knows a lot about Postgres' specifics, to this day,
pg_fetch_* functions convert all columns into strings. Numeric
types? String. Dates? String. Booleans? Yes. String too ('t' or 'f',
both trueish values to PHP).
To this day, while the extension supports prepared statements, their use is terribly inconvenient, forcing you to name your statements and to manually free them.
To this day, the
pg_fetch_* functions load the whole result set into
an internal buffer, making it impossible to stream results out to the
client using an iterator pattern. Well. Of course it's still possible,
but you waste the memory for that internal buffer, forcing you to
manually play with DECLARE CURSOR and friends.
There is zero support for advanced data types in Postgres and the library doesn't help at all with todays best practices for accessing a database (prepared statements).
There are other things that make the extension unpractical for me, but they are not the extensions fault, so I won't spend any time explaining them here (like the lack of support by newrelic - but, as I said, that's not the extensions fault)
pdo_pgsql gets a lot of stuff right that the pgsql extension doesn't: It doesn't read the whole result set into memory, it knows a bit about data types, preserving numbers and booleans and, being a PDO driver, it follows the generic PDO paradigms, giving a unified API with other PDO modules.
It also has good support for prepared statements (not perfect, but that's PDOs fault).
But it also has some warts:
- There's no way to safely quote an identifier. Yes. That's a PDO shortcoming, but still. It should be there.
- While it knows about numbers and booleans, it doesn't know about any of the other more advanced data types.
- Getting metadata about a query result actually makes it query the
database - once per column, even though the information is right there
in libpq, available to use (look at the
PDOStatement::getColumnMeta). This makes it impossible to fix above issue in userland.
- It has zero support for COPY
Imagine the joy of having a pdo_pgsql that actually cares about Postgres. Imagine how selecting a JSON column would give you its data already decoded, ready to use in userland (or at least an option to).
Imagine how selecting dates would at least give you the option of
getting them as a
DateTime (there's loss of precision though -
TIMESTAMP has more precision than
Imagine how selecting an array type in postgres would actually give you
back an array in PHP. The string that you have to deal with now is
notoriously hard to parse. Yes. There now is
Postgres, but hat shouldn't be needed.
Imagine how selecting a HSTORE would give you an associative array.
Imagine using COPY with pdo_pgsql for very quickly moving bulk data.
Imagine the new features of
PGResult being exposed to userland.
Giving applications the ability to detect what constraint was just
violated (very handy to detect whether it's safe to retry).
Wouldn't that be fun? Wouldn't that save us from having to type so much boilerplate all day?
Honestly, what I think should happen is somebody should create a
pdo_pgsql2 that breaks backwards compatibility and adds all these
getColumnMeta just return the OID instead of querying the
database. Have a
quoteIdentifier method (yes. That should be in PDO
itself, but let's fix it where we can).
fetch() return Arrays or Objects for JSON columns. Have it
return Arrays for arrays and HSTOREs. Have it optionally return
DateTimes instead of strings.
Wouldn't that be great?
Unfortunately, while I can write some C, I'm not nearly good enough to produce something that I could live with other people using, so any progress I can achieve will be slow.
I'm also unsure of whether this would ever have a chance to land in PHP itself. Internals are very adverse to adding new features to stuff that already "works" and no matter how good the proposal, you need a very thick skin if you want to ever get something merged, no matter whether you can actually offer patches or not.
Would people be using an external pdo_pgsql2? Would it have a chance as a pecl extension? Do other people see a need for this? Is somebody willing to help me? I really think something needs to be done and I'm willing to get my hands dirty - I just have my doubts about the quality of the result I'm capable of producing. But I can certainly try.
And I will.Read on →
when in doubt - SSL
Since 2006, as part of our product, we are offering barcode scanners with GSM support to either send orders directly to the vendor or to transmit products into the web frontend where you can further edit them.
Even though the devices (Windows Mobile. Crappy. In progress of updating) do support WiFi, we really only support GSM because that means we don't have to share the end users infrastructure.
This is a huge plus because it means that no matter how locked-down the customer's infrastructure, no matter how crappy the proxy, no matter the IDS in use, we'll always be able to communicate with our server.
Until, of course, the mobile carrier most used by our customers decides to add a "transparent" (do note the quotes) proxy to the mix.
We were quite stomped last week when we got reports of an HTTP error 408 to be reported by the mobile devices, especially because we were not seeing error 408 in our logs.
tcpdump has clearly shown how we were getting a RST
packet from the client, sometimes before sending data, sometimes while
Strange: Client is showing 408, server is seeing a RST from the client. Doesnt' make sense.
Tethering my Mac using the iPhones personal hotspot feature and a SIM card of the mobile provider in question made it clear: No longer are we talking directly to our server. No. What the client receives is a 408 HTML formatted error message by a proxy server.
Do note the "DELETE THIS LINE" and "your organization here" comments. What a nice touch. Somebody was really spending alot of time getting this up and running.
Now granted, taking 20 seconds before being able to produce a response is a bit on the longer side, but unfortunately, some versions of the scanner software require gzip compression and gzip compression needs to know the full size of the body to compress, so we have to prepare the full response (40 megs uncompressed) before being able to send anything - that just takes a while.
But consider long-polling or server sent events - receiving a 408 after just 20 seconds? That's annoying, wasting resources and probably not something you're prepared for.
Worse, nobody was notified of this change. For 7 years, the clients were able to connect directly to our server. Then one day it changes and now they aren't. No communication, no time to prepare and certainly too strict limits in order to not affect anything (not just us - see my remark about long polling).
The solution in the end is, like so often, to use SSL. SSL connections are opaque to any reverse proxy. A proxy can't decrypt the data without the client noticing. An SSL connection can't be inspected and an SSL connection can't be messed with.
Sure enough: The exact same request that fails with that 408 over HTTP goes through nicely using HTTPS.
This trick works every time when somebody is messing with your connection. Something f'ing up your WebSocket connection? Use SSL! Something messing with your long-polling? Use SSL. Something decompressing your response but not stripping off the Content-Encoding header (yes. that happend to me once)? Use SSL. Something replacing arbitrary numbers in your response with asterisks (yepp. happened too)? You guessed it: Use SSL.
Of course, there are three things to keep in mind:
Due to the lack of SNI in the world's most used OS and Browser combination (any IE under Windows XP), every SSL site you host requires one dedicated IP address. Which is bad considering that we are running out of addresses.
All of the bigger mobile carriers have their CA in the browsers trusted list. Aside of ethics, there is no reason what so ever for them to not start doing all the crap I described and just re-encrypting the connection, faking a certificate using their trusted ones.
failing that, they still might just block SSL at some point, but as more and more sites are going SSL only (partially for above reasons no doubt), outright blocking SSL is going to be more and more unlikely to happen.
So. Yes. When in doubt: Use SSL. Not only does that help your users privacy, it also fixes a ton of technical issues created by practically non-authorized third-party messing with you.Read on →
how to accept SSL client certificates
Yesterday I was asked on twitter how you would use client certificates on a web server in order to do user authentication.
Client certificates are very handy in a controlled environment and they work really well to authenticate API requests. They are, however, completely unusable for normal people.
Getting meaningful information from client side certificates is something that's happening as part of the SSL connection setup, so it must be happening on whatever piece of your stack that terminates the client's SSL connection.
In this article I'm going to look into doing this with nginx and Apache (both traditional frontend web servers) and in node.js which you might be using in a setup where clients talk directly to your application.
In all cases, what you will need is a means for signing certificates in order to ensure that only client certificates you signed get access to your server.
In my use cases, I'm usually using openssl which comes with some subcommands and helper script to run as a certificate authority. On the Mac if you prefer a GUI, you can use Keychain Access which has all you need in the "Certificate Assistant" submenu of the application menu.
Next, you will need the public key of your users. You can have them
send in a traditional CSR and sign that on the command line (use
openssl req to create the CSR, use
openssl ca to sign it), or you
can have them submit an HTML form using the
<keygen> tag (yes. that
exists. Read up on it on MDN
You absolutely never ever in your lifetime want the private key of the user. Do not generate a keypair for the user. Have them generate a key and a CSR, but never ever have them send the key to you. You only need their CSR (which contains their public key, signed by their private key) in order to sign their public key.
Ok. So let's assume you got that out of your way. What you have now is your CAs certificate (usually self-signed) and a few users which now own certificates you have signed for them.
Now let's make use of this (I'm assuming you know reasonably well how to configure these web servers in general. I'm only going into the client certificate details).
For nginx, make sure you have enabled SSL using the usual steps. In
addition to these, set
to the path of your CA's certificate. nginx will only accept client
certificates that have been signed by whatever
you have configured.
on. Now only requests that provide a client certificate signed by
above CA will be allowed to access your server.
When doing so, nginx will set a few additional variables for you to
use, most importantly
$ssl_client_cert (full certificate),
$ssl_client_s_dn (the subject name of the client certificate),
$ssl_client_serial (the serial number your CA has issued for their
certificate) and most importantly
$ssl_client_verify which you should
add_header to pass these variables through to
your application (in the case of add_header make sure that it was
really nginx who set it and not a client faking it).
I'll talk about what you do with these variables a bit later on.
As with nginx, ensure that SSL is enabled. Then set
SSLCACertificateFile to the path to your CA's certificate. Then set
Apache will also set many variables for you to use in your application.
SSL_CLIENT_S_DN (the subject of the client
SSL_CLIENT_M_SERIAL (the serial number your CA has
issued). The full certificate is in
If you want to handle the whole SSL stuff on your own, here's an
example in node.js. When you call
pass in some options. One is
requestCert which you would set to true.
The other is is
ca which you should set to an array of strings in PEM
format which is your CA's certificate.
Then you can check whether the certificate check was successful by
looking at the
client.authorized property of your
If you want to get more info about the certificate, use
Once you have the information about the client certificate (via fastcgi, reverse proxy headers or apache variables in your module), then the question is what you are going to do with that information.
Generally, you'd probably couple the certificate's subject and its serial number with some user account and then use the subject and serial as a key to look up the user data.
As people get new certificates issued (because they might expire), the subject name will stay the same, but the serial number will change, so depending on your use-case use one or both.
There are a couple of things to keep in mind though:
- Due to a flaw in the SSL protocol which was discovered in 2009, you cannot safely have only parts of your site require a certificate. With most client libraries, this is an all-or-nothing deal. There is a secure renegotiation, but I don't think it's widely supported at the moment.
- There is no notion of signing out. The clients have to present their certificate, so your clients will always be signed on (which might be a good thing for your use-case)
- The UI in traditional browsers to handle this kind of thing is absolutely horrendous. I would recommend using this only for APIs or with managed devices where the client certificate can be preinstalled silently.
You do however gain a very good method for uniquely identifying connecting clients without a lot of additional protocol overhead. The SSL negotiation isn't much different whether the client is presenting a certificate or not. There's no additional application level code needed. Your web server can do everything that's needed.
Also, there's no need for you to store any sensitive information. No more leaked passwords, no more fear of leaking passwords. You just store whatever information you need from the certificate and make sure they are properly signed by your CA.
As long as you don't lose your CAs private key, you can absolutely trust your clients and no matter how much data they get when they break into your web server, they won't get passwords, not the ability to log in as any user.
Conversely though, make sure that you keep your CA private key absolutely safe. Once you lose it, you will have to invalidate all client certificates and your users will have to go through the process of generating new CSRs, sending them to you and so on. Terribly inconvenient.
In the same vein: Don't have your CA certificate expire too soon. If it does expire, you'll have the same issue at hand as if you lost your private key. Very annoying. I learned that the hard way back in 2001ish and that was only for internal use.
If you need to revoke a users access, either blacklist his serial number in your application or, much better, set up a proper CRL for your certificate authority and have your web server check that.
So. Client certificates can be useful tool in some situations. It's your job to know when, but at least now you have some hints to get you going.
Me personally, I was using this once around 2009ish for a REST API, but I have since replaced that with oAuth because that's what most of the users knew best (read: "at all"). Depending on the audience, client certificates might be totally foreign to them.
But if it works for you, perfect.Read on →