One of the concern of the NIST/MEP project is that how would we crawl the different web pages of collaborative partners to get the data. Different institution might have different structure for their web page. Besides, an institution can change the web page structure. So writing a web crawler to get the data might not be a good idea as the maintenance cost will be huge. Hence we are considering a RESTful approach to get the necessary information from the partners.
We will use a RESTful architecture to get the data from the collaborative partners. Some high level idea about the structure is as follows:
- A REST service will be at our side which will get the data from collaborators and store in the local database.
- Each of the collaborative partners will have a client at their end which will provide web service.
- The web service at the client side will interact with the REST service residing at server side.
- There will be a CRON JOB running at the server side which will run at a regular interval and pull data from the web service of individual client.
- REST service will get the data from CRON JOB and store the data in the local database
- Whenever there will be a change in the client’s database, the web service will push the update to the server’s REST service. In this way, the server will always hold up-to-date information about the clients.
- There will be authentication mechanism implemented at REST service end of the server and web service end of the client so that no intruder can interfere. OAUTH can be used to implement the authentication process.
The web service at the client side will act as a translator which will convert the data from the client’s database to our pre-defined standard and vice versa. There are some options regarding this web service:
- If the client already has some sort of web service at their end, they can provide us our required data in a standard format
- If the client does not have a web service but has a development team we can provide them an interface which they will implement to meet the needs
- If the client does not have any development team we can develop the web service for them
We will also have to consider developing some adapter for connecting to the different databases of the client.