Joe Conley Tagged playframework

JSONPath Library for Play

Tue, 15 Apr 2014 00:00:00 +0000

I’ve been working on a platform that transforms, composes, and serves data. As part of this effort, I’ve developed a library for Play that performs a JSONPath query on a Play JsValue. You can learn about JSONPath by reading Stefan Goessner’s blog post on the subject.

I use Gatling’s jsonpath library to parse the JSONPath expression. I then fold over the tokens, performing a pattern match on each to construct the apporpriate JsValue. This parser supports all queries except for queries that rely on expressions of the underlying language like $..book[(@.length-1)]. However, there’s usually a ready workaround as you can execute the same query using $..book[-1:].

Example

Here’s a scala worksheet which traces the examples on Stefan’s post:

Deviation from JSONPath

One conscious choice I made as far as deviating from JSONPath is to always flatten the results of a recursive query. Using the bookstore example, typically a query of $..book will return an array with one element, the array of books. If there was another book array somewhere in the document, then $..book will return an array with two elements, both arrays of books. However, if you were to query $..book[2] for our example, you would get the second book in the first array, which assumes that the $..book result has been flattened. In order to make recursion easier and the code simpler, I always flatten the result of recursive queries regardless of the context.

If you have any questions, comments, or suggestions please let me know. I hope to be introducing an early iteration of my data platform shortly so stay tuned!

Building better APIs with Play!

Tue, 04 Mar 2014 00:00:00 +0000

This is the technical companion to my Point.io post, Building better APIs with Play!. Herein lies coding examples galore!

RESTful Architecture - Routing

The routes file of a Play app allows you to define the HTTP verb, the route path/pattern, and the corresponding method from the controller. In addition to denoting basic path variables, you can also use regular expressions to match on specific routes (i.e. xml or html formats for example). What’s great about this approach is this file is compiled along with the source code, so any mistakes like an incorrect controller method or invalid HTTP verb will be caught and thrown at compile time.

Action composition

We define two types of custom actions: atomic and composed. Atomic actions can be used stand-alone or as building blocks to be composed with other actions. We use the following pattern for building an atomic action.

The object allows us to use the action by itself, and the case class allows us to compose this action with other actions.

A composed action is strictly syntactic sugar, making it more convenient to combine behaviors and keeping the controller code more concise. We define composed actions using just an object.

Filters

Filters are handy for cross-cutting concerns. We’ve had one use case where it was necessary to modify every JSON response with links to metadata. We achieved this using a filter and the Play Enumeratee library

JSON - Global messaging

Building an effective API requires being responsive to users in a comprehensive manner. All concievable events should be handled appropriately, such as incorrect requests from the client or internal server errors. Creating a Global object allows you to generically craft responses to handle these situations. We define methods to handle events like internal errors, route not found, or a bad request.

Conclusion

Play is well-equipped to handle the nuances of API development and maintenance. We’re pleased with the stability and performance we’ve seen thus far and are looking forward to continuing down this path of reactive goodness.

(Triz)Swagging out at the Philly Codefest

Tue, 25 Feb 2014 00:00:00 +0000

This past weekend I teamed up with some buddies from Point.io (Angel, Jon, and Dylan) to participate in my first hackathon, Philly Codefest. We spent the weekend bringing Angel’s dream to life: a platform called TrizSwagger to analyze and track the use of “swag” (i.e. T-shirts, office supplies, and other marketing mathoms). We leveraged social media and geolocation to give companies real-time visibility to their marketing campaigns. Feel free to check out the app here.

Angel demoing Point.io's apiDoc service

Lessons Learned

Good programming is always concerned with simplicity and efficiency, whether it’s using efficient data structures and algorithms, conciseness in your codebase, or even naming variables properly. Building an app in 24 hours, however, throws the need for simplicity and efficiency into sharp relief. Here are a few takeways from my experience.

Coast-to-Coast JSON

I’ve always been a big fan of Domain-driven Design. Writing POJOs in Java and case classes in Scala can provide a clear crystallization of the main actors of your application. However, models may not always be necessary, especially if your app is backed by a service/database which gives you JSON (TrizSwagger is backed in MongoDB and served by Flask). The extra layer of complexity in marshalling/unmarshalling between JSON and your model can hinder performance and readability of your code, especially if you’re using heavy ORM frameworks like Hibernate. During a hackathon, if you’re rapidly making changes to the model you’ll surely get slowed down. For a more thorough treatment of JSON Coast-to-Coast, check out the Mandubian Blog

Knockout.js Mapping plugin

In order to implement coast-to-coast design effectively, it’s important to have a front-end framework that manages JSON well. One such framework I’m fond of is Knockout.js, more specifically their mapping plugin. This plugin automatically maps a JSON message into a Javascript observable object. You can then code the front-end directly against object properties without having to pre-define a viewmodel. You can also customize how objects are mapped by either modifying or enhancing the created object. This strategy proved quite helpful during the hackathon as any changes to our back-end API literally only had to be changed in one place (the front-end).

One caveat is the creation of a new object using this plugin. Since the plugin requires a JSON object to build out the observable, I wrote a basic method in Play to take an expected JSON object and “empty” it, setting default values that would be used in the new object form.

Understanding your tools

TrizSwagger integrates with both Twitter and Facebook. Understanding and setting up those integrations, however, occupied a lot of our time. We also ran into issues with a server on OpenShift, slowing us down further. Ultimately I think simpler is better, and every choice in technology needs to be well thought-out and well-suited to its use case.

Conclusion

Overall it was an awesome experience. Even though we didn’t win (or place, or show for that matter), we learned a lot and we still took the time to mentor other teams who were using the Point.io API. Great job Team TrizSwagger!

Jon, Dylan, and Joe doing some last-minute coding

Hacking NPR's Sunday Puzzle

Mon, 10 Feb 2014 00:00:00 +0000

I’m a big fan of puzzles. I’ll often start my day attempting the Philadelphia Inquirer’s jumble and crossword, with varying degress of success. One puzzle I never miss is NPR’s Weekend Edition Puzzle featuring New York Times puzzle editor Will Shortz. At the end of each segment, he poses a question to the audience, and occasionally these questions can be solved with the help of programming. To that end, I’ve built an app to help non-programmers solve these puzzles. I’ve also added common puzzle utilities like an Anagram checker and a Scrabble solution generator.

You can find a running version of the puzzle solver here. The Scala library of the puzzle utilities can be found here. This project also has an npr package which shows examples of programs written to solve past NPR puzzles.

Puzzle Solver

This single-page app searches through a specified list of words searching for one of three things: anagrams, potential Scrabble solutions, or most powerfully, a regular expression. We’ll use this last mode to solve a recent NPR puzzle.

Puzzle Modes

Anagrams

This mode will search for all potential anagrams of the input word. Helpful for solving the jumble commonly found in your newspaper. For example, here are today’s four jumble clues from the Inquirer:

GREEV WORNC KNITSY KRUTYE

Setting the app controls to Mode = Anagram and Word List Source = Scrabble (a good list source for most purposes), we set Input once for each jumble and after hitting Submit, we get one proper anagram for each jumble.

Scrabble

This mode will search for all possible valid Scrabble words based on the letters (i.e. your Scrabble hand) provided. You can also specify how many wild cards (i.e. “blanks”) are in your hand. This is useful not only to help find solutions but to verify solutions as well (faster than leafing through a Scrabble dictionary).

Regular expressions

This is the most powerful mode. This will return all words matching a valid Java regular expression. You can find a good tutorial about Java regular expressions here.

Word Lists

I’ve gathered two common word lists, a list of valid Scrabble words mentioned here and the UNIX word list. I’ve also added a space to add a custom list of words to search.

Technology

I’ve built this app using Play Scala as the backend. After importing my puzzles library, here’s the relevant controller code:

I probably could have handled the JSON a bit safer by using a Reads[T] object to handle the parsing, but as this app was fairly simple I used the unsafe conversion JsPath.as. Please don’t think less of me!

I’ve also employed Knockout.js to manage the front-end functionality. Knockout.js is a lightweight MVVM framework which manages DOM updates automatically and succinctly, ensuring that your front-end code is not a monolith of jQuery calls. Here’s the code for the front-end:

And that’s it! A good future exercise would be to stream the solutions reactively, especially when dealing with a long word list. This would be done using Play’s Enumeratee library. If anyone’s interested in that I can post a follow-up detailing that solution.

Puzzle Solution - Double S (Non-Programmers)

Now our tool is ready to help us solve a recent puzzle. Here’s the question, reprinted from NPR’s website:

What word, containing two consecutive S's, becomes its own synonym if you drop those S's?

Using Regex mode and our Scrabble word list, we can define the regular expression .ss. to return all words with a double S:

From there, we use Excel to store this list in the first column. We can then copy this list in the second column and do a Find/Replace to remove instances of “SS”. We could inspect each row for synonyms but as we have 1976 results, that’s a lot of words to inspect. To further narrow down our choices, I used Excel to transpose the second column into one row, copy that into a text file and do another Find/Replace by highlighting the space between each word and replacing with a vertical bar. This gives us another regular expression that should look like this:

abbe|aby|abyes|...|zestle|ziple|zonele

Using that regular expression, we can use our Puzzle Solver to determine which words in the list are valid words and which aren’t (we only care about words that are valid). This second search takes significantly longer, but we’ll eventually get a list back of 156 valid words. Plugging this list back into Excel, we can use a VLOOKUP to identify the pairs of words which are valid. After sorting to see valid word pairs first, we can inspect to see which pairs are synonyms. Luckily, the answer appears fairly early in the list (spreadsheet can be downloaded here):

Puzzle Solution - Double S (Programmers)

The previous non-coding solution might have seemed a bit convoluted. A much simpler method would be to use my Scala library directly to find the solutions, which can be done with as little as four lines of code:

Conclusion

If you enjoy NPR’s Sunday Puzzle, I’d highly recommend Blaine’s Puzzle Blog as an excellent companion resource. This blog community offers tantalizing, interesting hints for the solution of the puzzle and often digress into other challenging puzzles as well.

Happy puzzling!

Roll Your Own Notification Service

Mon, 27 Jan 2014 00:00:00 +0000

Have you ever wished you could receive customized updates whenever your favorite websites update their content? Most sites offer the means to get notified when a new blog post hits the wire or new products are added to their catalog (RSS, social media, e-mail, etc.). But what if the site doesn’t use any of these services? Or what if you only want specific updates (i.e. blog posts from author X, new products containing the name Y)? Then you’re left with only one course of action: build your own notification service!

Armed with the mighty powers of HTML scraping, the Scala programming language, and a recurring scheduling mechanism (in this case Play’s Akka scheduler), you have all the tools you need to setup your custom notification.

My New EBook Notification Service

Let’s create a notification service which let’s us know when new ebooks are available at my local digital library, Delaware County Library System. At the time of this writing, no such notification service exists. As I’d prefer not to miss any notifications, I’d like to setup an RSS feed. Specifically, we’ll write a process which periodically checks the digital library site for new ebooks and updates an RSS feed accordingly.

Scala/SBT

We’ll start out by creating a basic Scala application using SBT (you can checkout a skeleton project here). Let’s add the HTMLUnit and Scala IO libraries to our project. We’ll use HTMLUnit to parse the HTML code of the library’s website, and we’ll use Scala IO to write our XML to file. Your build file should now look like this (assuming you named your project “ebook”):

Scala XML

Let’s start by building an abstraction for an RSS feed (you can read about the basics of RSS here). We’ll start with an Item case class which holds the basic properties of an RSS item and a method to generate xml. Similarly, we define the basic properties of a Feed using a trait. We’ll make this abstract in the anticipation of re-using this abstraction for other feeds.

Screen Scraping with HTMLUnit

Let’s build a NewEBookFeed which implements Feed. When we implement the items method, we’ll use HTMLUnit to parse the HTML code from Delaware County Library System to find out the newest items. This requires digging around the source HTML a bit to understand the structure and find useful patterns. Basic knowledge of XPath is required to leverage those patterns. After inspecting the source code and following the appropriate links, we can view the New Ebook page source and parse out the new titles, authors, and image URLs.

That’s it! You can find my complete code as part of my scrape library, specifically the com.josephpconley.books and com.josephpconley.rss packages. We can test the code by running the following:

Deploy using Play

Now that we have a way to generate an up-to-date RSS feed, we need a way to update our feed periodically and make it publically available to an RSS Reader like feedly (my personal favorite). We could handle this a few different ways (i.e. schedule a CRON job to push a file to our Dropbox folder), however I’d like to demonstrate how to handle both the scheduling and file writing/serving using the Play Framework.

Start a new Play Scala project, and either package our ebook project as a jar and copy to the lib folder, or just copy and paste the source code into the new Play project (I’ve done the former).

Akka Scheduler

To hook into Play’s Akka scheduler, we create a Global object in the app folder and override the onStart method, which allows us to run code once the application starts. The Akka system scheduler allows you to schedule a recurring process for a given Duration. In our case, since the site doesn’t update that frequently and we want to be respectful by not overloading the site with requests, we’ll set the duration to 12 hours.

From there, it’s simply a matter of building out a controller with some routes to host the updated file (a straightforward exercise I’d leave to the reader). I personally included this code and hosted the RSS feeds in my own Play app running on Heroku.

Drawbacks

One drawback you might have noticed from this specific example is the possibility of the target site’s source code changing. We relied on very specific HTML tags, text and class attributes to query the information we needed, and should the site be re-written significantly, it’s possible that we would have to re-write our scraping code to accommodate.

Conclusion

Managing the daily flow of information can be a challenge. With a little bit of coding, however, we can gain finer control over the information we consume, helping us be more productive in our everyday life.