Spigot

Spigot is an attempt to bring some sanity to consuming external API data. Removing the need for manual ad hoc attribute mapping throughout your code.

Without spigot, you may have some creation code such as this.

You're usually pulling specific data out of the response, not using half of what you get back, digging through keys and values and checking presence all over the place. These problems get multiplied when you have multiple external sources for your model.

You end up with a ton of code just to get the data to fit our model.

if params[:data].present?
  data = params[:data]
  record = User.where(external_id: data[:id]).first

  if record.nil?
    url = "https://github.com/#{data[:login]}"

    user = User.new({
      name: data[:first_name],
      email: data[:email_address],
      url: url
    })

    if data[:profile].present?
      user.bio = data[:profile][:text]
    end

    user.save!
  end
  # gross
end

# ./config/initializers/spigot.rb
Spigot.resource :user do
  first_name    :name
  email_address :email
  profile do
    text :bio
  end

  url :login do |value|
    "https://github.com/#{value}"
  end
end

While using Spigot, you only need to define your map using simple ruby in an initializer and the task of wrangling this data gets much more manageable.

Spigot is able to do the translation for you and put the API data into a language your implementation understands.

This leaves you with a simple statement to accomplish the same task:

User.find_or_create_by_api(params[:data])

Much better.

Examples

Basic Implementation

# Our Model
class User < ActiveRecord::Base
  include Spigot::Base
end

# Api Data Received
data = JSON.parse("{\"full_name\":\"Dean Martin\",\"login\":\"dino@amore.io\",\"token\":\"abc123\"}")

# Spigot definition to map the data
Spigot.define do
  service :github do
    resource :user do
      full_name :name
      login     :email
      token     :auth
    end
  end
end

# Usage
User.create_by_api({github: data}).inspect
#=> User:0x007f976 id: 1, name: "Dean Martin", email: "dino@amore.io", auth: "abc123"

Authentication

Spigot.define do
  service :github do
    resource :user do
      login :username
      name :name
      image(:gravatar_id){|id| "https://2.gravatar.com/avatar/#{id}" }
    end
  end
end

# Using Omniauth
class CallbacksController < Devise::OmniauthCallbacksController

  def github
    user = User.create_or_update_by_api({omniauth: request.env['omniauth.auth']})
    sign_in(:user, user)
    redirect_to dashboard_path
  end

end

Coordinating Sources

# ./config/initializers/spigot.rb
Spigot.define do
  service :github do
    resource :user do
      name  :name
      login :username
      gravatar_id(:image_url){|id| "https://2.gravatar.com/avatar/#{id}" }
    end
  end

  service :twitter do
    resource :user do
      name        :name
      description :bio
      screen_name :username
      profile_image_url :image_url
    end
  end
end

# ./app/models/message.rb
class Message
  after_create :fill_author_data

  def api_data
    # API Calls ...
  end

  def fill_author_data
    user = User.find_or_create_by_api({self.network => api_data})
    update_attribute(:user_id, user.id)
  end
end

# ./app/controllers/messages_controller.rb
class MessagesController < ApplicationController

  def create
    Message.create(params[:message])
  end

end

Documentation

Syntax

We define the map used to translate api data by passing a block to the Spigot.define method.

Spigot.define do
  service :github do
    resource :user do
      name  :name
      login :username
    end
  end
end

This map defines the data attributes for a User received from Github. The data received has the keys :name and :login. The values at those keys are mapped to the columns :name and :username respectively.

Methods

Each ActiveRecord method has a similar signature. Each accepts a hash of data, if you want to scope your data to a specific service, pass the data in as a value keyed to the service name {'username' => 'dino'} vs. { github: {'username' => 'dino'} }.

You specify the attribute used in your query by assigning the primary_key option in your Spigot definition.

find_by_api
find_all_by_api
create_by_api
update_by_api
find_or_create_by_api
create_or_update_by_api

# Spigot Definition
Spigot.resource :user do
  full_name :name
  login     :email
  token     :auth
  options do
    primary_key :email
  end
end

# API Data
data = {"full_name":"Dean","login":"dino@amore.io","token":"bcd456"}

User.find_by_api(data)
User Load (0.1ms)  SELECT "users".* FROM "users" WHERE "users"."email" = 'dino@amore.io' ORDER BY "users"."id" ASC LIMIT 1
#=> User id: 1, name: "Dean Martin", email: "dino@amore.io", token: "abc123"

Services

Spigot definitions have a service keyword. You can pass it a name and then a block of resource definitions. Each service denotes a different source of API data, such as Twitter or Github. You can also call service on the `Spigot` object without the define method.

Spigot.define do
  service :twitter do
    ...
  end
end

Spigot.service :yelp do
  resource :review do
    ...
  end
end

Resources

Spigot definitions have a resource keyword. This corresponds to the name of the class that is implementing this map. Spigot uses the resource name to look up which map to use for the class you're currently mixing Spigot into.

In this code block, we are defining a service for the Twitter API. When we pass twitter data to a spigot method used on the User class, it will use the mapping defined in the :user resource.

If you define a resource without enclosing it in a service, it will use this map whenever you don't specify which service to use as you pass the data in.

Spigot.define do
  service :twitter do
    resource :user do
      ...
    end
  end
end

Spigot.resource(:tweet) do
  ...
end

Nested Data

The data received from an API is often nested. We can convey this using a nested block within a resource definition.

In this code block, we are mapping data that has a contact key that points to a nested hash containing data with keys login and phone. We will drill down into the contact hash and map those corresponding values to the appropriate attributes in our formatted hash.

Spigot.resource :user do
  full_name :name
  contact do
    login :email
    phone :telephone
  end
end

Evaluated Ruby

On any attribute map, you're able to pass it a block which will be used during the assignment of the value. This let's you modify the value before assignment to the attribute in the formatted hash.

In this code block, we are taking the value that is present in the API data under the username key and interpolating the value into a static url to build our database's value for the user's URL

Spigot.resource(:user) do
  full_name :name
  username :url do |value|
    "https://twitter.com/#{value}"
  end
end

Options

There are a couple options available to help define how Spigot accomplishes various tasks. Right now this is limited to database queries. Each Spigot option set is defined per resource.

primary_key: The attribute used to execute a select statement when querying this resource's table

Spigot.resource(:user) do
  full_name :name
  login :email
  options do
    primary_key :email
  end
end