Category: Developer

Google Guava – sweet and succulent

2016-March-21

I have a bit of java code that handles JWT. It generates a MACVerifier and then uses that to verify a signature. Someone commented that it was taking more time than they expected. I didn’t see a ton of opportunity for optimization, but I thought I might wrap the generation of the MACVerifier in a cache.

At first I tried EHCache. EHCache is the gold standard as far as Java caching. There are sooo many options, and there is sooo much flexibility. Write through caches, read-through caches, caches with persistence that is configurable in ways you had not imagined you needed. Java Attributes to add caching to servlets or JAX-RS. EHCache has it all.

Do One Thing Well

So I figured it would be a safe choice. But after a little bit of fiddling with it, I decided EHCache was too much. To me, EHCache violates the “do one thing well” principle of design, or if you like, the Single responsibility principle (As applied to the module, if not a particular class), or, just unsatisfying documentation which is a common problem even among “successful” open source projects.

Why is there a CacheManager? What if I create a Cache and don’t register it with a CacheManager – what happens? What do I lose? Why do I want a CacheManager? Why are there names for both managers and caches? What would happen if I registered a Cache with multiple managers? What if I don’t want persistence? What if the Cache itself goes out of scope – will it be garbage collected?

I couldn’t find ready answers to these questions and the whole experience left me lacking confidence whether the cache would do the right thing for me. In the end I concluded that EHCache was more, much more than I needed, and would require more time than I wanted to invest, to get a cache. I just wanted a simple in-memory Cache in Java with TTL support (where TTL also implies time-since-last-access or time-to-idle). And what do you know! Google Guava provides that!

Guava

Goooooooooogle

At first it was unclear how to best exploit it. But a little reading showed me that Guava has a clever design that allows the cache itself to load items into it. I don’t need to write MY code to check for existence, and then create the thing, and then put it into the cache. Guava has a LoadingCache that does all this for me. I just call cache.get() and if the item is present, it is dispensed. If it is not in the cache, then the cache loads it and gives it to me. Read-Through cache loveliness. So simple and easy.

This is my code to create the cache:

And to use the cache, I just call cache.get(). Really slick. Thanks, Google!

restclient.el – sending API Requests directly from within Emacs

2016-March-8

Hey, something new! (to me!) the restclient.el library, for emacs. I tried it. I like it. I recommend it.

What does it do? Allows you to send REST requests (really just http requests) right from emacs, interactively. And then pretty-prints the results if possible (if XML or JSON or image). It includes a simple text mode that allows you to define a set of requests and some variables that can be used in those requests. The whole thing is simple, easy, handy.
Activate it with C-c C-c

Separately, I have a library that reads .netrc files from within elisp. It’s a natural complement to restclient.el , for API endpoints that require HTTP Basic authentication. That covers lots of API endpoints, including OAuth token dispensaries that require the client_id and client_secret to be passed in as an HTTP Basic authentication header. Here’s a simple example use:

Really nice. How did I not know about this elisp library?

One problem I had when using it: The restclient.el helpfully uses a function json-pretty-print-buffer to pretty-print the buffer containing the response, if the content-type of the response is application/json.

I don’t know that function, and it wasn’t defined on my emacs. This led to a runtime error, and a json buffer that was hard for me to visually parse.

But my emacs does have the similarly named json-prettify-buffer. So I used the following gist to get the restclient to succeed in its pretty-printing efforts.

The restclient.el module is not a huge thing, but it’s nice for us emacs people. I know about Postman, and use it. I know about Paw (but don’t use it). I know and use Fiddler. I am a big fan of curl, and someitmes curlish. This is a nice additional tool for the toolbox.  Really handy.

Thanks, Jake McCrary, for writing up your experience with Emacs and restclient.el; your blog post is how I discovered it.  And thanks of course to Pavel Kurnosov, the original author of the restclient.el library. Thanks for sharing.

EDIT – I made a change in restclient.el to fix an issue that causes an extra unintended newline to be appended to the last form parameter. This issue cost me about 90 minutes of debugging my JWT verification code, bummer! My change just trims trailing newlines from the entity being sent. This will be a problem for you if you want to send an entity that ends in several newlines. Find my fixed restclient.el here .

letsencrypt and NearlyFreeSpeech

2016-February-2

I’ve been running this site on nearlyfreespeech for some time now.

Last week I created a cert using the tools and service made available by letsencrypt.org, and then configured my NFS server to use it. It was pretty easy, but not documented. I’ll share here what I did to make it work.

I am able to SSH into the nearlyfreespeech server. I can also perform a git clone from that server to get the letsencrypt tools. But when I ran the letsencrypt-auto tool from the server, it didn’t do what I wanted it to do. This was my first time with the tool, and I’m unfamiliar with the options, so maybe it was just pilot error.

In any case, I solved it by running the tool on my Mac OSX machine and transferring the generated PEM files to the server.

  1. I ran git clone on my local workstation (Mac OSX)
  2. from there, I ran the letsencrypt tool with these options: 
    ./letsencrypt-auto certonly  --manual  \
   -d www.dinochiesa.net -d dinochiesa.net \
   --email dpchiesa@hotmail.com
  3. follow the instructions. I needed to create endpoints on my NFS server that responded with specific values.
  4. when that completed, I had the cert and keys in PEM format. I then copied them to /home/protected/ssl on the NFS server
  5. opened a service ticket on NFS as per This FAQ
  6. a couple hours later, the NFS people had completed the SSL config for me

Maybe this will help someone else.

It’s possible that I could have used the –manual option on the NFS Server, and avoided the need to transfer files. Not sure. If anyone else has done this, I’d like to know. I will need to renew my certs every couple months.

I’m really pleased about the letsencrypt service. I hope it gets used widely.

Update, 2017 December 7: I’ve updated my certs 3 or 4 times since I made this post. Now, this is what I do: 

   sudo certbot certonly  \
     --authenticator manual  \
     --domain www.dinochiesa.net \
     --domain dinochiesa.net \
     --email dpchiesa@hotmail.com \
     --rsa-key-size 4096

I’ve automated the other parts – creating the right endpoints on the NFS server, and then copying the generated certs when they’re sent. Also NFS no longer requires a service ticket; it will automatically install certs when I update them. The change takes a minute or less. Super easy.

Use PHP code to make WordPress redirect to secure site

2016-February-1

Lots of people use the .htaccess redirect rules to force their wordpress sites to load with the secure option.

It looks like this:

But if you have a hoster that does not provide you the ability to modify the .htaccess file, that won’t work. These hosters typically set up your server behind their load balancer which means the wordpress code sometimes cannot directly infer whether HTTPS is in use. In other words, the $_SERVER[‘HTTPS’] is not correct.

It is possible to introduce code into your theme that will do what you need. This is the PHP code:

Insert that in your theme header.php file. Or maybe the functions.php file. Invoke the maybe_redirect_to_ssl_site() function in the theme header before emitting any HTML.

Pre-request script for Postman, to calculate HttpSignature

2016-January-27

If you do REST, you probably have a favorite REST client testing tool.
Mine is Postman, a Google Chrome app.

Postman has a nifty feature called Pre-request scripts, which allows you to write some Javascript code that performs a calculation and then reads and writes the “environment” object for the request. This means you can calculate … hashes or digests or signatures or anything you like.

Here’s an example of a script that calculates an HMAC-SHA256 HttpSignature, using the keyId and secret-key embedded in the environment. It also computes a digest on the message payload. Postman helpfully includes CrytoJS in the JS sandbox, so it’s easy to do HMAC and SHA.

In this particular case, the HttpSignature verification on the server requires 2 headers (date and digest) plus the ‘(request-target)’ value. The digest is a SHA-256 of the payload, which is then base64 encoded.

Anyone can start with this and modify it to do other variations.
Good luck!

Addendum

I should have mentioned this: Postman, even the latest Chrome app version, uses XmlHTTPRequest to send out requests. XHR is purposefully limited, in the specification, to restrict some headers from being set explicitly on outbound requests. The list of restricted headers includes: Origin, Date, Cookie, Via, and others. The reason for this restriction: it is desired that the user-agent be fully in control of such request headers.

My example above uses an HttpSignature that signs the Date header. This means the code must also SET the Date header to a known value; in my case I am using a value generated by the pre-request script.

postman-set-headers

The value corresponds to “now”, the current moment. But the point is the value has to be absolutely known.

This is not possible in standard Postman, because of the XHR restriction. The effect you will see is that the intended Date header silently does not get sent with the request.

This may confound you if you don’t have an ability to trace the request on the receiving (server) side. In the context of a request that uses HttpSignature, the server will throw an error saying “Missing Date header”.

But! in Postman v0.9.6 and above, it is possible to configure Postman with something the Postman people call “the Intercptor plugin”, which then allows the lifting of this restriction. The Date header gets sent, and everything works.

If you don’t want to rely on the Interceptor plugin, and you want the HttpSignature to include the date value, then you’ll have to use a differently named header to hold the date. Use X-Date or anything other than “Date”. You need to change the client as well as the server of course, to make everything hold together.

Online calculator for SHA and HMAC-SHA

2016-January-26

Here’s a thing I built. It’s just a webpage that calculates SHA-(1,224,256,384,512) and HMAC with the same algorithms.

I was using this to help with building a system that relies on HttpSignature. Developers need some help in constructing and validating their HMACs and SHAs.

Naïve Data analysis leads to incorrect conclusions – WWII Bomber Plane edition

2015-July-8
Raid by the 8th Air Force

Here’s a good story showing us how focusing on the data we have in front of us may lead us to incorrect conclusions.

The summary is this: In World War II, Allied bomber command undertook an analysis effort to determine how to optimally reinforce bomber planes to protect them from German anti-aircraft guns. They studied the bullet hole patterns in planes after they returned from missions, and the first impression was to re-inforce the planes where the bullet holes appeared most commonly.

But a smart guy named Abraham Wald pointed out that if the planes were returning with those bullet holes, then the bullets that caused those holes were not critical to avoid. The planes that didn’t return had likely sustained the bullet holes that demanded mitigation.

Are we talking about Bomber Planes or Software here?

Focusing on the data we have is a common approach that supports iterative improvement efforts. The Lesson from the Bomber Plane story is: there may be “invisible data” that needs consideration.

Clayton Christensen World Economic Forum 2013

Mr Christensen looking stern and wise.

This is the central problem explored by Clayton Christensen, in his classic book, The_Innovator’s Dilemma. What Christensen found was that companies can tend to focus on feedback they get from existing customers, while ignoring the needs of prospective customers. And this is dangerous.

We have a project going on right now in which we’re looking at the UI usage data for a particular tool in a web app. The web app tool has for some time now been stable. It was never very complete, and it hasn’t been extended actively. You could say it was stagnant. We know the reasons for that – we had other priorities and this web tool just needed to stay in the backlog for a while. Also, there were, and there still are, other ways to accomplish what was exposed by this web app tool; there are public APIs that expose the underlying function and people have the option to build tactical apps on those APIs. Now we want to enhance the web tool, and the web tool has recently been enhanced to at least collect data on how people are currently using it. How much should we rely on this usage data?

With the bomber planes, the data they had were the patterns of bullet holes in the returned planes. The data they didn’t have were the patterns of bullet holes in bomber planes that didn’t return.

With the web tool, the data we have will be the UI usage patterns of current users. The data we don’t have will be the usage patterns of people who abandoned the stagnant tool some time ago, and are relying on the underlying APIs and independent tools to accomplish their goals. The former set of users will have a relatively higher proportion of novice users, while the latter will have a higher proportion of experts.

So what’s the right path forward? Should we invest in improvements in the tool, solely based on the data we have? Should we guess at the data we don’t have? In the bomber plane example, the data we don’t have was clear by deduction – those planes were most likely shot down. In the web tool example, the data we don’t have are truly unknown. We cannot deduce what, exactly, people are doing outside the tool.

I think the only way to resolve this is to combine three factors: analysis of the API usage, analysis of the UI usage, and finally, perspective on the behavior of the expert users who have abandoned the tool, perhaps formed by direct observation or by intelligent inference. API analytics, UI analytics, and some intelligence.

Then we’ll need to apply some sort of priority to the different scenarios – do we need to help the experts more with their expert problems? or do we need to invest more in making the simple things even easier for the more novice users? Which is the more important audience, the novices or the experts? Which one is more important to serve, at this time in the product’s lifecycle?

Interesting questions! There’s no one right answer for every situation. I know which option I favor in our current situation.

Upgrade your nodejs on Mac OS X to v0.12.6

2015-July-6

Per medium.com, there are critical vulnerabilities in nodejs and iojs. People running nodejs should upgrade to v0.12.6.

On Mac OSX, this takes just a few moments. Thanks to Stackoverflow for the tip.


~/ $ node -v
v0.10.22
~/ $ sudo npm cache clean -f
Password:
npm WARN using --force I sure hope you know what you are doing.
~/ $ sudo npm install -g n
npm http GET https://registry.npmjs.org/n
npm http 200 https://registry.npmjs.org/n
npm http GET https://registry.npmjs.org/n/-/n-1.3.0.tgz
npm http 200 https://registry.npmjs.org/n/-/n-1.3.0.tgz
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
n@1.3.0 /usr/local/lib/node_modules/n
~/ $ sudo n stable

install : node-v0.12.6
mkdir : /usr/local/n/versions/node/0.12.6
fetch : https://nodejs.org/dist/v0.12.6/node-v0.12.6-darwin-x64.tar.gz
installed : v0.12.6

~/ $ node -v
v0.12.6
~/ $

Web vs Native apps, part 12763

2015-June-2

From Hacker News recently, Another posting, this one by Peter-Paul Koch, about the web-vs-native debate, dramatically entitled “Let’s concede defeat”!!

This is the latest installment in a long line of commentary on the topic. In short, the topic is: is it better to build and offer a native, platform-specific app, or to invest in building a web-based user experience? We’ve been having this debate for years, and the discussion just seems to spin round and round on the same points. readwrite.com is also chiming in on the “morass”. Everyone has a viewpoint. Web is better. Native is better. Hybrid is the future.

Lots of drama. A range of viewpoints. Many of the strongest opinions come from people with something to sell: they sell the vision of web as write-once-run-anywhere, or they sell a product that helps produce hybrid apps.

What’s the real story?

In my opinion, it’s not that complicated. Here you go: Web apps are super convenient to deploy, update, and access. There’s no “app store”, there’s no worry about if and when users will update. But, they’re not optimal for a crisp device-optimized experience. There are many web frameworks and choosing the right one is challenging. Attempting to “emulate” native app behavior can get you into trouble when trying to support N different device platforms.

The native app, on the other hand, is crisp and device optimized, and users like it. It’s easier to do offline-capable apps, and it works with system notifications. But, using native means you’re basically writing many apps, one version for each device. Also, the way users access the app is inconvenient, and there’s no guarantee users will update their apps (although there are mitigations for that).

But most apps will be hybrids, combining some web-app capability with some native capability.

Dzone has a nice table listing decision criteria between the three options. Kinlan has a nice, day-in-the-life journal that gives color to the tradeoffs, although looking for a web-based alarm clock to supplant the device-supplied one seems like tilting at windmills. No one needs a web-based alarm clock when every phone has several native apps to choose from. He made that point very effectively, though I don’t think he intended to do so.

We, as an industry, haven’t settled this issue yet. There’s still more talking we need to do, apparently.

Though there’s been much talk, there are still seemingly common perspectives that are just wrong.

  1. from ignorethecode, Native apps mainly benefit Apple and Google, not their users. It’s not in anyone’s interest to be locked into a specific platform, except for the platform’s owner.
    Wrong. This just completely disregards the platform-specific stuff developers can do with accelerometers, GPS, cameras, and screens. Baloney. Yes, you can get access to some of those things, some of the time, in web apps, but not in a reliably cross-platform way.
  2. from dzone, Native apps often cost more to develop and distribute because of the distinct language and tooling ecosystems,
    Wrong. This assumes the main cost is in the licensed tools, like Xcode and a deployment license from Apple. Sheesh. That is the tiniest component of app development and deployment cost.. C’mon now. Developers (people) and their time are expensive. Licenses are cheap.

I like what Mr Koch had to say in the quirksmode blog. Basically: different solutions will better suit different sets requirements, for commerce, news sites, and so on. Pragmatic. Realistic. Helpful.

Regardless whether people choose web apps, native apps, or hybrid apps, the backend is always APIs. Everybody needs APIs! And they need to be simple, easy to consume, well documented, secure, and managed and measured properly.