HTTP client

(:require [org.httpkit.client :as http])
  

Like the Server, the client uses an event-driven, non-blocking I/O model. It's lightweight and efficient.

  • Concurrency made easy by asynchronicity and promises
  • Keep-alive makes quite a difference in performance, and can be disabled with {:keepalive -1}
  • Timeout per request
  • HTTPS supported, ~100k of RAM for issuing a HTTPS request. TCP connection and SSLEngine get reused if possible
  • Easy-to-use API, modeled after clj-http

asynchronous with promise, callback

;fire and forget, returns immediately[1], returned promise is ignored
(http/get "http://host.com/path")

(def options {:timeout 200             ; ms
              :basic-auth ["user" "pass"]
              :query-params {:param "value" :param2 ["value1" "value2"]}
              :user-agent "User-Agent-string"
              :headers {"X-Header" "Value"}})
(http/get "http://host.com/path" options
          (fn [{:keys [status headers body error]}] ;; asynchronous response handling
            (if error
              (println "Failed, exception is " error)
              (println "Async HTTP GET: " status))))
 ; [1] may not always true, since DNS lookup maybe slow
  

synchronous with @promise

Synchronous programming is easy to understand, just like clj-http:

;; synchronous
(let [{:keys [status headers body error] :as resp} @(http/get "http://host.com/path")]
  (if error
    (println "Failed, exception: " error)
    (println "HTTP GET success: " status)))
;; Form params
(let [options {:form-params {:name "http-kit" :features ["async" "client" "server"]}}
      {:keys [status error]} @(http/post "http://host.com/path1" options)]
  (if error
    (println "Failed, exception is " error)
    (println "Async HTTP POST: " status)))
  

Combined, concurrent requests, handle results synchronously

Send request concurrently, with half the waiting time

(let [resp1 (http/get "http://http-kit.org/")
      resp2 (http/get "http://clojure.org/")]
  (println "Response 1's status: " (:status @resp1)) ; wait as necessary
  (println "Response 2's status: " (:status @resp2)))
(let [urls ["http://server.com/api/1" "http://server.com/api/2"
            "http://server.com/api/3"]
      ;; send the request concurrently (asynchronously)
      futures (doall (map http/get urls))]
  (doseq [resp futures]
    ;; wait for server response synchronously
    (println (-> @resp :opts :url) " status: " (:status @resp))
    )

Persistent connection

HTTP persistent connection, also called HTTP keep-alive, or HTTP connection reuse, is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair

HTTPS persistent connection is also supported.

By default, http-kit keeps idle connections for 120s. That can be configured by the keepalive option:

 ; keepalive for 30s
@(http/get "http://http-kit.org" {:keepalive 30000})
 ; will reuse the previous TCP connection
@(http/get "http://http-kit.org" {:keepalive 30000})
 ; disable keepalive for this request
@(http/get "http://http-kit.org" {:keepalive -1})

Pass state to asynchronous callback

Sometimes, it's handy to pass some state to callback. You can do it this way:

(defn callback [{:keys [status headers body error opts]}]
  ;; opts contains :url :method :header + user defined key(s)
  (let [{:keys [method start-time url]} opts]
    (println method  url "status" status "takes time"
             (- (System/currentTimeMillis) start-time) "ms")))

;;; save state for callback, useful for async request
(let [opts {:start-time (System/currentTimeMillis)}]
  (http/get "http://http-kit.org" opts callback))
  

Output coercion

;; Return the body as a byte stream
(http/get "http://site.com/favicon.ico" {:as :stream}
        (fn [{:keys [status headers body error opts]}]
          ;; body is a java.io.InputStream
          ))
;; Coerce as a byte-array
(http/get "http://site.com/favicon.ico" {:as :byte-array}
          (fn [{:keys [status headers body error opts]}]
            ;; body is a byte[]
            ))
;; return the body as a string body
(http/get "http://site.com/string.txt" {:as :text}
          (fn [{:keys [status headers body error opts]}]
            ;; body is a java.lang.String
            ))
;; Try to automatically coerce the output based on the content-type header, currently supports :text :stream, (with automatic charset detection)
(http/get "http://site.com/string.txt" {:as :auto})

Nested params

httpkit has some supports for nested params, like clj-http does.

{:query-params {:a {:b {:c 5} :e {:f 6}}}} => "a[e][f]=6&a[b][c]=5"
httpkit and clj-http do so by encoding the nested data structure, expect server understand the encoding, and do the proper decoding. This is not robust. Recommanded usage is do the encoding and decoding explicitly, using your favorite encoding.

Take JSON for example:

(require '[clojure.data.json :as json])

(http/post "http://your-server/api"
           ;; using json to encode the nested params before pass to httpkit
           {:query-params {:a (json/write-str {:b {:c 5} :e {:f 6}})}})

;; on the server side, get the param, decode it using json/read-str.
;; json supports more data structure and is easier to understand and reason about.
;; It's also cross language, has minimum requirement for your server's implementation
 

Various options

(http/request {:url "http://http-kit.org/"
               :method :get             ; :post :put :head or other
               :user-agent "User-Agent string"
               :oauth-token "your-token"
               :headers {"X-header" "value"
                         "X-Api-Version" "2"}
               :query-params {"q" "foo, bar"} ;"Nested" query parameters are also supported
               :form-params {"q" "foo, bar"} ; just like query-params, except sent in the body
               :body (json/encode {"key" "value"}) ; use this for content-type json
               :basic-auth ["user" "pass"]
               :keepalive 3000          ; Keep the TCP connection for 3000ms
               :timeout 1000      ; connection timeout and reading timeout 1000ms
               :filter (http/max-body-filter (* 1024 100)) ; reject if body is more than 100k
               :insecure? true ; Need to contact a server with an untrusted SSL cert?

               ;; File upload. :content can be a java.io.File, java.io.InputStream, String
               ;; It read the whole content before send them to server:
               ;; should be used when the file is small, say, a few megabytes
               :multipart [{:name "comment" :content "httpkit's project.clj"}
                           {:name "file" :content (clojure.java.io/file "project.clj") :filename "project.clj"}]

               :max-redirects 10 ; Max redirects to follow
                ;; whether follow 301/302 redirects automatically, default to true
                ;; :trace-redirects will contain the chain of the redirections followed.
               :follow-redirects false
               })

http/get http/post etc, are build on top of http/request, so, these options apply to them too

Faking Responses in Tests

Often in tests you want to prevent requests from being sent over HTTP by stubbing out calls to the client. This can be done with http-kit-fake.

(with-fake-http ["http://http-kit.org/" "a fake response"]
  (http/get {:url "http://http-kit.org/"})) ; promise wrapping the faked response

See the http-kit-fake README for a full set of options.