Looking at Ruby’s Net::HTTP Library

Every so often, we find ourselves in need of accessing REST APIs. There are some great libraries out there for this, but what about the library behind these libraries? In this article, we’re going to take a look at the Ruby’s Net::HTTP library. We’ll go through simple REST API principals like:

• GET
• POST
• PUT
• DELETE

I’ll also cover more advanced topics like:

• HTTP Blocks
• 301 Redirects
• Headers
• Basic Auth
• File Downloads

What is REST and Why Should You Care?

REST stands for REpresentational State Transfer. It was theorized by Roy Fielding in his doctoral dissertation: Architectural Styles and the Design of Network-based Software Architectures, then later implemented by the World Wide Web Consortium (W3C) for HTTP 1.1. It’s basically a software (mainly web) architecture style that allows for easy API access, along with some other helpful goals.

REST APIs focuses on four main verbs: GET, POST, PUT, and DELETE. We can use Ruby’s Net::HTTP library to access REST API’s. In this article, we’ll learn how to do this, as well as some other goodies. If you want full definitions of all HTTP 1.1 methods, then you should visit this W3C page.

Excited? Let’s get started.

Simple: Testing

Let’s use Sinatra as our little API server. If you’re unfamiliar with Sinatra, you can learn the basics in 10 minutes. Here’s our server:

# file: rest-server.rb
require 'sinatra'

# all status codes given are based on W3C standards

get '/' do
  'Here is some data.'
end

post '/' do
  puts "Data: '#{params[:foo]}' recieved, creating object."
  status 201
end

put '/' do
  puts "Data: '#{params[:foo]}' recieved, updating object."
end

delete '/' do
  puts 'Deleting data.'
end

Now to start it up, just open a new terminal tab:

$ ruby rest-server.rb
== Sinatra/1.4.4 has taken the stage on 4567 for development with backup from Thin
Thin web server (v1.6.1 codename Death Proof) # might be WEBrick instead
Maximum connections set to 1024
Listening on localhost:4567, CTRL+C to stop

You now have a functional REST API running on your computer.

Simple: GET Request

A GET request gets, or reads information from a server. This request is quite common. When you visited this page, your browser made a GET request which returned a hypertext document from the server. You can do this same thing in Ruby. In our example, we’re going to be getting data from our server.

require 'net/http'
require 'uri'

url = 'http://localhost:4567/' # trailing slash is important
uri = URI.parse(url)

Net::HTTP.get(uri) # GET request
# => "Here is some data."

This simple example pulls data from our server and returns it. Elegant, right? Here it is in ugly one-liner form:

require 'net/http' # URI is required by Net::HTTP by default

Net::HTTP.get(URI 'http://localhost:4567/')
# => "Here is some data."

As an intelligent SitePoint Ruby reader like you can obviously see, this code has some issues. Don’t worry, we’ll get to most of these later in the advanced Net::HTTP bits.

Simple: POST Request

A POST request posts, or sends data/parameters to a server. APIs use this for recieving data from a user. This request is also quite common. Every time you send a tweet, you’re sending a POST request to Twitter with your tweet as parameters. In this example, we’re going to be send our server a POST request.

require 'net/http'
require 'uri'

url = 'http://localhost:4567/'
uri = URI.parse(url)

params = {foo: "bar"}

Net::HTTP.post_form(uri, params)
# => #<Net::HTTPCreated 201 Created readbody=true>

And of course, here is its ugly one-liner equivalent:

require 'net/http'

# ugly, right?
Net::HTTP.post_form(URI('http://localhost:4567/'), {foo:"bar"})
# => #<Net::HTTPCreated 201 Created readbody=true>

Simple: PUT Request

A PUT request puts, or updates something on a server. APIs use this mainly when updating data on the server. For example, if you edit a pull request on GitHub, you’re sending a PUT request with your data. Unfortunately, PUT is not very well documented in the Ruby Net::HTTP library, so there is no elegant solution to it.

require 'net/http'
require 'uri'

url = 'http://localhost:4567/'
uri = URI.parse(url)

params = {foo: "change"}

http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Put.new(uri.path)
request.set_form_data(params)
http.request(request) # the actual PUT request
# => #<Net::HTTPOK 200 OK readbody=true>

I’m not going to show this as a one liner because it would be over 80 characters, which would not only be unreadable, but also goes against the Ruby Style Guide.

Simple: DELETE Request

A DELETE request deletes something from the server. If you were to delete a tweet, you would be sending a delete request to Twitter. This example is very similar to the last.

require 'net/http'
require 'uri'

http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Delete.new(uri.path)
http.request(request)
# => #<Net::HTTPOK 200 OK readbody=true>

Great. We’ve gone through the basics and REST verbs using Ruby. Keep in mind, there are multiple ways of doing most of these HTTP requests. We’ll go through some of these in the next section.

Advanced: Testing

For the advanced section, our server is going to have to be more complicated in order to return and recieve data like the API we are trying to mimic in each example. So, the server will need to be changed in each example.

From now on, we will be focussing on the GET and POST HTTP methods. If you need to apply the techniques below to the other methods, just mix the simple version of the request with the advanced technique.

Advanced: HTTP Blocks

In the principle of DRY, HTTP blocks are important. HTTP blocks are simply Net::HTTP blocks. This may seem a little confusing, but it’s quite a simple concept.

First we create a new method. We’ll call it http_request and give it one argument, uri.

require 'net/http'

def http_request(uri)

end

Now for the fun part. Call Net::HTTP.new and yield it to a block.

require 'net/http'

def http_request(uri)
  Net::HTTP.start(uri.host, uri.port) do |http|
    yield(http, uri)
  end
end

This means that we can do really cool things like this: (using the simple server we made)

# ... http block code above ...

uri = URI 'http://localhost:4567/'

http_request(uri) do |http, uri|
  http.get(uri.path).body
end
# => "Here is some data."

We’ll be using these in the next examples, so get familiar. Just keep in mind that http_request is the same thing as Net::HTTP.start, it’s just a shortcut.

Advanced: 301 Redirects

Oh no! Our server just took a step up and now has the ability to redirect requests.

require 'sinatra'

get '/foo/?' do
  redirect '/bar'
end

get '/bar/?' do
  'Hello!'
end

If we try the simple GET method from earlier, let’s see what happens now…

require 'net/http'

uri = URI 'http://localhost:4567/foo'
Net::HTTP.get(uri)
# => ""

Hmm… that gave us nothing. But, being the smart people that we are, we try a second simple method.

require 'net/http'

def http_request(uri)
  Net::HTTP.start(uri.host, uri.port) do |http|
    yield(http, uri)
  end
end

uri = URI 'http://localhost:4567/foo'
http_request uri do |http, uri|
  http.get(uri.path)
end
# => #<Net::HTTPFound 302 Moved Temporarily readbody=true>

That gave us more information, but still no 200 OK. What can we do to follow these redirects without getting lost? There are two methods.

Method one uses Net::HTTP.get_response to get information from the server, not just the body.

require 'net/http'

uri = URI 'http://localhost:4567/foo'

def get_follow_redirects(uri, request_max = 5)
  raise "Max number of redirects reached" if request_max <= 0

  response = Net::HTTP.get_response(uri)
  case response.code.to_i
  when 200
    response.body
  when 301..303
    get_follow_redirects(URI(response['location']), request_max - 1)
  else
    response.code
  end
end

get_follow_redirects(uri)
# => "Hello!"

Method two involves using HTTP blocks in a similar method.

require 'net/http'

uri = URI 'http://localhost:4567/foo'

def http_request(uri)
  Net::HTTP.start(uri.host, uri.port) do |http|
    yield(http, uri)
  end
end

def get_follow_redirects(uri, request_max = 5)
  http_request uri do |http, uri|
    response = http.head(uri.path)
    case response.code.to_i
    when 200
      http.get(uri.path).body
    when 301..303
      get_follow_redirects(URI(response['location']), request_max - 1)
    else
      response.code
    end
  end
end

get_follow_redirects(uri)
# => "Hello!"

As you can see, this method is very similar to the other one. It all depends on preference and how much control you want over the request.

Advanced: Headers

In the previous examples, we used headers to find the 301/302 redirect location. In the next example, we’ll isolate this ability so we can access any part of the headers.

First, let’s make a request and see what headers we get. We’ll be using the first, simple server in this example.

require 'net/http'

uri = URI 'http://localhost:4567/'

response = Net::HTTP.get_response(uri)
response.header.to_hash.inspect
# => "{\"content-type\"=>[\"text/html;charset=utf-8\"], \"content-length\"=>[\"18\"], \"x-xss-protection\"=>[\"1; mode=block\"], \"x-content-type-options\"=>[\"nosniff\"], \"x-frame-options\"=>[\"SAMEORIGIN\"], \"connection\"=>[\"keep-alive\"], \"server\"=>[\"thin 1.6.1 codename Death Proof\"]}"

Pretty sweet, huh? If you don’t need the all the headers, you can just use response['something'] as a shortcut.

Of course, there is also an HTTP block method version of this…

require 'net/http'

# ... http block code ...

uri = URI 'http://localhost:4567/'

http_request(uri) do |http, uri|
  response = http.head(uri.path)
  response.to_hash.inspect
end
# => "{\"content-type\"=>[\"text/html;charset=utf-8\"], \"content-length\"=>[\"18\"], \"x-xss-protection\"=>[\"1; mode=block\"], \"x-content-type-options\"=>[\"nosniff\"], \"x-frame-options\"=>[\"SAMEORIGIN\"], \"connection\"=>[\"keep-alive\"], \"server\"=>[\"thin 1.6.1 codename Death Proof\"]}"

HTTP headers are quite useful. You can get all sorts of information about the data source and leverage this information to your advantage.

Advanced: Basic Auth

Here’s a less common situation, but equally as useful as the last one. Sometimes APIs offer basic http authentication. Let’s simulate that in our server:

require 'sinatra'

# source: http://recipes.sinatrarb.com/p/middleware/rack_auth_basic_and_digest#label-HTTP+Basic+Authentication
use Rack::Auth::Basic, "Protected Area" do |username, password|
  username == 'foo' && password == 'bar'
end

get '/' do
  'Classified data.'
end

Now we have to find a way to access that classified data. Let’s do this.

require 'net/http'

def http_request(uri)
  Net::HTTP.start(uri.host, uri.port) do |http|
    yield(http, uri)
  end
end

uri = URI 'http://localhost:4567/'

req = Net::HTTP::Get.new(uri)
req.basic_auth 'foo', 'bar'

http_request uri do |http, uri|
  response = http.request(req)
  response.body
end

# => "Classified data."

Of course, this type of authentication is not very common with OAuth being in existance, but it’s always good to know.

Advanced: File Downloads

Sometimes as developers, we need a programmatic way to download files. This is as easy as making a GET request and piping each line into a new file. For this example, we’re going to use a real-world file from the internet. There’s no point in rewriting our server for this situation.

For our first example, we’ll modify the simple GET request that we made earlier.

require 'net/http'

content = Net::HTTP.get(URI 'http://www.google.com/humans.txt') # get the contents of the file

File.open('humans.txt', 'w+') do |file|
  file.write(content) # write the contents
end

Simple right? But we’re savvy programmers. What about live streaming? Let’s have some fun.

require 'net/http'

uri = URI 'http://www.google.com/humans.txt'

def http_request(uri)
  Net::HTTP.start(uri.host, uri.port) do |http|
    yield(http, uri)
  end
end

http_request uri do |http, uri|
  request = Net::HTTP::Get.new(uri)

  http.request(request) do |response|
    open('humans.txt', 'w+') do |file|
      response.read_body do |chunk|
        file.write(chunk)
      end
    end
  end
end

With small files, this second method is just several lines of code longer, but on larger files, this can speed up downloads greatly.

Advanced: SSL

SSL is very common in APIs and other web services. Don’t worry, Net::HTTP has us covered.

For this next example, we’ll be using Google because it uses SSL for its connection. We won’t be using Sinatra because it’s out of the scope of this tutorial to configure it.

require 'net/https' # notice the change to https

uri = URI 'https://www.google.com/'

http = Net::HTTP.new(uri.host, 443) # hardcoded for forced SSL
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

req = Net::HTTP::Get.new(uri.path) # or any other request you might like
resp = http.request(req)
resp.body
# => ... very long response body (Google's homepage) ...

Conclusion

You can do so many things with the Net::HTTP library. It is extensible, but also simple to use. I suggest that you also check out the HTTParty, Rest-Client, and Curb gems. They provide a very simple way to access these same methods in a better way. Whatever you choose, remember that anything your browser can do Ruby can do as well.

Further Reading

• Architectural Styles and the Design of Network-based Software Architectures
• Ruby Net::HTTP Documentation
• HTTP Code Definitions
• HTTP Method Definitions
• Ruby IO::File Mode Definitions

Frequently Asked Questions (FAQs) about Ruby Net::HTTP Library