Uploading/Sharing data with a JSON Web Token in R

The amount of code in this post is going to be embarrassingly small. Just, really tiny. But I’ve spent a few days trying to piece together how to do this, and so thought it would be wise to put all that I’ve gathered into one place so other folks don’t need to do the same.

Wanna jump to the code? Like a recipe? I got you.

What I’ve been trying to do: Upload data to an international database using a JSON Web Token (JWT) with my username and password. If I wanted to, I could upload each data file one by one using their web interface. However, I have ~100 files to upload, so that doesn’t seem like a good use of my time. Instead, we’re going to use web tokens and the R programming language. I’m doing this in R because a) I like it best and b) there seems to be fewer guides online for this using R. You can also use Python, C++, JS, Java, probably Matlab, and a whole host of other languages. Check out the JWT website for more info.

You can read more about JWTs here, but briefly is is an open, secure method for two servers to talk to each other through an application programming interface (API). Most guides are show you how to either create your own API that uses JWTs (which, why is that first, what I need is sooo much simpler) or GET data from an API that uses javascript data objects, or JSON (object notation). The latter makes sense, and probably what most people are searching to do. You can use versions of this code to access data from the New York Times, Twitter, or any other titan on this list.

However, that’s likely not why you’re here. If you need to share data across institutions with a JWT, this post will help you.

You’ll need the libraries httr and jsonlite. My code also has some set variables, like usr (username), pw (password) and url (root site I am trying to access), as well as strings to append to the root url to create a token, access the database, and upload data. Those will be called Token, getListFileScreening, and UploadFile, respectively.

OK, first, let’s get that token.

r <- httr::POST(url = paste0(url,Token), 
            body = list(
                UserName = usr,
                Password = pw,
                #This varible is optional, check with the host of the API you are trying to access.
                #See comments in text.
                encode = "form"
             ), verbose())

So, here we are POSTing to the Token generating URL. You need to use POST here, rather than GET, which does seem counter-intuitive because you want to get a token, right? However, it’s better to think that you are POSTing your username and password to the token generating url, and it is returning a JWT.

The ‘encode’ variable will change here depending on the type of API you are trying to access, and how this data is sent to the server. Again, I was interacting with a form on a site initially, so I need to encode the POST to ‘form’ here. There are three other choices, but for this step you will likely use form or ‘json.’ ‘json’ takes the data you are POSTing, and unboxes it so it can become a JSON. To upload a file or a string, you would encode the post as ‘multipart.’ Encoding your POST to ‘raw’ would require additional format settings not covered in this blog. Best check with the team behind the API you need to access if you aren’t sure which option to choose (I certainly did…).

The verbose() function will give you a print out after your POST your request, including what type of error you are getting if it didn’t go through. This is critical for troubleshooting!

If this still doesn’t work for you, look into ‘grant_type.’ This may be necessary depending on the type of resource owner password credentials grant the API you are trying to access is using.

OK, your username and password are working? Great. Now, let’s look at that token!

jsonlite::prettify(httr::content(r, "text"))
{
 "result": {
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJodHRwOi8vc2NoZW1hcy54bWxzb2FwLm9yZy93cy8yMDA1LzA1L2lkZW50aXR5L2NsYWltcy9uYW1lIjoiZ3JpZmZpdGhzZSIsImp0aSI6IjZjYTFiYTgwLWQ2NjQtNGE1ZC1hOWU4LWU3YzEwN2NhOGFjYSIsImh0dHA6Ly9zY2hlbWFzLnhtbHNvYXAub3JnL3dzLzIwMDUvMDUvaWRlbnRpdHkvY2xhaW1zL2VtYWlsYWRkcmVzcyI6ImVtaWx5dGdyaWZmaXRoc0BiaW9zLmF1LmRrIiwiZXhwIjoxNTk4Njk4NDkzLCJpc3MiOiJ2bXNhcHAiLCJhdWQiOiJ2bXN1c2VycyJ9.QA9rD8AIZfEy3WZSCwF4QZBnMuYrFkpSCYu_mFKiP3U",
    "expiration": "29 August 2020 12:54:53"
},
"id": 3,
"exception": null,
"status": 5,
"isCanceled": false,
"isCompleted": true,
"isCompletedSuccessfully": true,
"creationOptions": 0,
"asyncState": null,
"isFaulted": false

}

Here’s we’re using the prettify() function n the jsonlite library to look at the text of our request. Lo’ and behold! A web token. A great way to quickly check your web token is to head to the JWT website and plug your token into their DeBugger tool. If your username data doesn’t pop up in their Payload section, your token did not work!

Now, let’s upload some data.

#Reformat the data so it is easier to parse.
tkn=jsonlite::fromJSON(content(r, "text"), simplifyVector = FALSE)

r2=POST(url=paste0(url,UploadFile), body = list(fileToUpload = httr::upload_file('O:/IGotUrDataRightHere.h5')),
         httr::add_headers('Authorization' = paste("Bearer", tkn$result$token, sep = " ")), verbose())

These few lines have a bit of redundancy to make the code more readable, and to make sure that your R console is pulling functions from the correct library. 'upload_file’ is a common function, so here we are making sure that it is the httr library that function is coming from. If you want to read more about this, take a look at this page explaining namespaces.

Same logic with the ‘add_headers’ function; make sure that it is pulling from the httr library. Your header is were your token information is sent to the ‘UploadFile’ url alongside the file that you want to upload. There were two major ways I saw to do this while I was researching this code, the way listed above and this:

httr::add_headers("x-auth-token"=tkn$result$token),encode = 'multipart', verbose())

See that encode = ‘multiform’? How your header is formatted has to do with, you guessed it, the API you are trying to upload to. However, using ‘x-auth-token’ is not standard language, just common language. It could also be ‘X-Auth-Token’ or any combination of uppers and lowers, or different syntax all together.

Using ‘Authorization’ is authenticating you, the bearer of the token, the authority to upload a file to the API.

You’ll notice that if you use the verbose() function here, your R console will start speaking in tongues.

>> ?'æ}R@n¡UÂë]S@hgé HR@J±=`«R@Ë`S@
>> Š&ŠÒR@Ü8`MlS@fé„v×R@eQ$T@Žì¸ü‘S@²£²pS@½;¸RS@L4g«0S@h6ð—WS@¢–Jl×½K@lGÜºª"L@

This is OK, but if you need to upload a large number of files, you may want to remove the verbose() function after you are confident that your code is working properly.

After the Wingdings and LONG series of numbers, what you are looking for after uploading the file is the following:

<- HTTP/1.1 200 OK
<- Content-Type: text/plain; charset=utf-8
<- Server: Microsoft-IIS/10.0
<- Strict-Transport-Security: max-age=2592000
<- X-Powered-By: ASP.NET
<- Date: Wed, 02 Sep 2020 15:23:28 GMT
<- Content-Length: 188

You did it! If the API has a verification site, you can access it with the following:

res <-httr::GET(paste0(url,getListFileScreenings),
httr::add_headers(Authorization = paste("Bearer", tkn$result$token))
# return list of submissions
httr::content(res)

Use the GET command this time, rather than the POST since you are only trying to GET something from the site.

Full Code:

library(jsonlite)
library(httr)

# Set username, password, and urls.
usr<-'username'
pw<-'password'
url <- "https://coolbeans/api/"
Token='Token'
getListFileScreenings='getListFileScreenings'
UploadFile='UploadFile'

#Get Token
r <- httr::POST(paste0(url,Token), 
                body = list(UserName = usr, password =pw,grant_type = "password"),
            encode = "form", verbose())

jsonlite::prettify(httr::content(r, "text"))
#Reformat your token.
tkn=jsonlite::fromJSON(content(r, "text"), simplifyVector = FALSE)

#Post a file to your API
r2=POST(url=paste0(url,UploadFile), body = list(fileToUpload = httr::upload_file('O:/allthesinglefiles.h5')),
         httr::add_headers('Authorization' = paste("Bearer", tkn$result$token, sep = " ")), verbose())

#Get a list of the files you have submitted and their status.
#This will only work if the API you are trying to communicate with has it set up.
res <-httr::GET(paste0(url,getListFileScreenings),
httr::add_headers(Authorization = paste("Bearer", tkn$result$token))
# return list of submissions
httr::content(res)

BONUS:

A good tool to idiot proof what you are doing is Postman. Download that app, open it up, and open a new tab.

In there, open the Body -> form-data tab. Enter your URL, set the action to POST, and provide your username and password.

After that, hit send and you should get a JWT back. If you don’t, make sure you have all of the data entered correctly. If you do, contact the server you are trying to connect with to make sure your username and password is correct. It will tell you if you have an error, like 401 or 400.