Skip to content

OFFLINE API

Jump to bottom
Pierre Depaz edited this page Jun 13, 2017 · 2 revisions

How to create your own API

Steps

1. SCRAPING

  • Running a Node script
  • Serving a webpage
  • Fetching a webpage
  • Parsing content

2. SAVING

  • Introducing JSON
  • Writing to a file
  • Appending to a file

3. SERVING

  • Catching an endpoint
  • Parsing a request
  • Sending back the appropriate data

SCRAPING

Running a Node script

  • Node is just a way to write JavaScript in your terminal, and write server-side code. Server-side code allows you to do things such as choosing what content to display to the user (with things like user profiles, databases, newsletter subscriptions, etc.)

  • First things first, then, we need to install NodeJS, by downloading it here. Once that's done, you should be able to open up your Terminal (OSX/Unix) or your console (Windows), and type which node, and have something along the lines of /usr/bin/node show up. If that's not the case, then something went wrong with your installation :/

  • You can now write Javascript and run that code directly in your console by typing node *name_of_your_script.js*

Serving a webpage

  • Node is mostly used to create web applications, which means that, at some point, we are going to need to write some code to serve some content. It's possible to write this in pure Node code, but that would be very tedious, and smarter people than us have made it easier for the rest of the world to do so, by distributing packages. * Packages are pieces of code that are put up online and that anybody can re-use. In our case, we are going to use a package called express, which allows us to build web servers.
  • To install express, we need to be in the same directory as our server code, and we type npm install express in our console. We should see a bunch of output, concluded by Successfully installed Express.
  • Now that we have downloaded the Express code, we need to include it in our code. You can see a very basic web server code with express here.
  • If you run that code by typing node b_basic_server.js, and then go to localhost:2046 in your browser to see what we're serving!

Fetching a webpage

  • If we're going to serve content, we first need to find, save and organize that content. Doing that automatically on the web is called scraping.
  • Node has a package for requesting called request. To install it, we do the same thing we did for express, type npm install request.
  • We can then ask our script to request a web page, in order to then go through it and extract the elements we want. You can find that code here.

Parsing a webpage

  • Now that we can request a webpage, we need to parse it; that is, we need to load the HTML, make sense of it, and only extract the content that we want. To do that, we start by installing cheerio by typing npm install cheerio
  • Inside the callback function for our request, we load the body that we get back into cheerio. We can then ask to get the content inside any HTML element, by tag name, by class, or by id.
  • For example, we can inspect a webpage by opening the web console (Window > Developer > Developer Tools on Chrome), look for the particular class of the item that we're interested in and specify that to cheerio. If our page includes something like that <span class="name">Erik Prince</span>, in order to extract that data, we would need to write:
var $ = cheerio.load(body);
var name = $('.name').text();

SAVING

Introducing JSON

  • JSON stands for JavaScript Object Notation, and is a very convenient way to save data. A typical object would look like this:
var person = {
    'first_name': 'Pierre',
    'last_name': 'Depaz',
    'nationalities' : ['FRA', USA']
};

console.log(person.first_name); //outputs 'Pierre'
console.log(person.nationalities); //outputs '['FRA', 'USA']'
console.log(person.nationalities[1]); //outputs 'USA'

Writing to a file

  • If we want to make our data more permanent, we need to write that data to an external file -in our case a file ending in .json.
  • In order to do that, we must first include a package that allows us to access the filesystem. That package is called fs, and we don't need to install it through npm, it already comes with Node.
  • fs has a function called writeFile(*path*, *data*, *callback*) which writes (1) to a file (that's the path you need to provide, something like data/results.json), (2) some data (that's the data we want to save, and it needs to be of type string, not JSON, or HTML, or numbers, or images.
var fs = require('fs');
var myData = 'lorem ipsum and so on';
fs.writeFile('snippet.txt', myData, function(){
    console.log('successfully written to file!');
});
  • If you run the above code, and assuming you already have a file called snippet.txt, then you should find the sentence lorem ipsum and so on written inside snippet.txt, inside the same folder as your script. You can take a look at this script for a working example.

Appending to a file

  • The problem is that fs.writeFile() always re-writes the whole file. In order to avoid that we need to (1) read the existing file, (2) add our content to that file, and (3) write the updated content to the same file.
  • To look at a working example, take a look at this script.

SERVING

Serving an endpoint

  • Web applications can be separated by endpoints, which will allow you to categorize your content and only serve what is relevant to the person who requests it.
  • Endpoints follow the format protocol://address:port/endpoint. So the endpoint to serve the About section of say, my website, would be https://pierredepaz.net/about
  • All you need to do to create an endpoint is to write some code in your server script that will respond something if a user asks for that specific endpoint. The code for the above example of serving the About page on a website would look like this.
app.get('/about', function(request, response, error){
    response.serveFile(__dirname + '/about.html');
});
  • However, you can respond with any type of data, such as JSON! In that case, it would look like this:
app.get('/data', function(request, response, error){
        console.log('requested the /data endpoint');

        var myJSON = {
                'hello': 'darkness',
                'my': 'old friend'
        };

        response.json(myJSON);
});
  • For a working example, you can take a look at this script.

Parsing a request

  • One of the strong points of APIs is that we don't know exactly what the user wants, but we want to be able to serve them that. To do that, we can look for specific keywords in the endpoint requested, and use those as filters.
  • These keywords are called queries, and they always have a field and a value. So if the user requests http://mywebsite.com/data?name=pierre&last=depaz, the request.query element in our server code will be:
{
    'name':'pierre',
    'last':'depaz'
}
  • So if we can both serve JSON data and see what the user is asking for, we can process their request in order to only give them what they need.

Sending back the appropriate data

  • If we look at the queries for a particular endpoint, we can send only the data that the user needs. Let's say we have some data about the 5 largest companies by revenue:
var data = {
    companies: [
        {
            'name':'Wal-Mart',
            'country':'USA',
            'revenue':'485870'
        },{
            'name':'State Grid',
            'country':'CH',
            'revenue':'329601'
        },{
            'name':'China National Petroleum',
            'country':'CH',
            'revenue':'299271'
        },{
            'name':'Sinopec Group',
            'country':'CH',
            'revenue':'294344'
        },{
            'name':'Royal Dutch Shell',
            'country':'UK/NE',
            'revenue':'272156'
        }
    ]
};
  • Then, if our user only wants to see data related to a specific country, she can request that specific endpoint: http://mywebsite/data?country=CH.
  • To see a working example, take a look at this script.

Clone this wiki locally