Mindoo Blog - Cutting edge technologies - About Java, Lotus Notes and iPhone

  • XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Karsten Lehmann  27 April 2012 18:59:38
    This article presents the first demo of my session about NoSQL databases at the German Entwicklercamp conference in March 2012. It demonstrates how the document-oriented NoSQL database MongoDB can be used in an XPages web application for the IBM Lotus Domino server.

    The rise of location based services

    Location based services have become quite popular in the past years: Most of the smartphones carry a GPS sensor, and there are a lot of popular apps out there (e.g. Foursquare, Yelp) that use cloud services to find people, restaurants and places nearby.

    For a number of reasons (mostly scalability/performance related), the NSF database of Lotus Notes/Domino is not the best choice to implement these kind of applications/services. MongoDB is much more suited for this use case and the article shows how you can integrate MongoDB into Notes/Domino to combine both worlds and get the best from both of them.

    The Dojo based web application that I am going to discuss in detail uses MongoDB's geospatial indexing feature to easily find the nearest points of interest for a mobile web user.

    Here is how our application looks like:



    The "Administration" tab provides a user interface to add/remove places to/from the MongoDB database. A place is stored as a document in a MongoDB document collection and consists of a name, a type (e.g. shop/gas station) and the position information as [longitude, latitude] value pair.

    The database can be queried on the "Search" tab by entering an address, a distance in kilometers and an optional type. The address will automatically get converted to [longitude, latitude] coordinates by using the Google Geocoding API, which is then used to find places within the specified distance.

    Our MongoDB document collection is indexed in a way so that we can quickly query the database for places that surround the current user's position and sort them by ascending distance between both points.



    Since geospatial indexing is a built-in feature of MongoDB, the solution is very easy to implement.

    Limitations of NSF

    While it is not impossible to build this application in pure Lotus Notes/Domino technology with an NSF database, the missing geo index and limited scalability of NSF would make it difficult to keep response times low with large data sets, e.g. millions or even billions of places worldwide. MongoDB instead provides a sharding feature to distribute the data evenly to several servers and reduce server load.

    Additional indexer like Apache Lucene in combination with the OSGi tasklet service plugin's extension hooks might help to improve Domino's limited indexing support (we already used a Lucene index with Domino data in a big customer project) , but using MongoDB for this use case is just so much easier.

    Architecture

    The application consists of client-side and server-side code:
    Client-side code is written in JavaScript language as a Dojo class using the class declaration system of Dojo 1.6. This Dojo class handles all user interface operations.

    Server-side code is written in Java as a servlet and is exposed to the client-side via REST API's (here: GET/POST requests sent via dojo.xhrGet / dojo.xhrPost).

    We developed two servlets for this application. The first servlet handles init/add/remove/query operations in MongoDB. The second servlet simply acts as a proxy to access the Google Geocoding API from client-side Javascript in order to convert between addresses and geo coordinates (longitude/latitude).

    Since the REST protocol is completely stateless, the server-side code is very easy to test. With a bit of refactoring, it's even possible to run the whole application in alternative servlet containers like Jetty, Tomcat or directly within the Eclipse IDE instead of the Domino server.

    Personally, I really like this REST API based approach because it's clean, transparent and keeps technologies separated.
    It enables you to replace the server environment, the database system and the client-side UI toolkit at any time. You can even build multiple UIs (e.g. based on Dojo 1.6, 1.7 and Sencha's ExtJS) or mobile clients that work with the same REST API.

    Diving into the code: server side
    Our sample comes as an Eclipse plugin "com.mindoo.mongo.test" to be run on Domino's OSGi framework (tested with Domino 8.5.3 GA without any fixpacks or XPages extension library):



    As you can see above, I added the Mongo API classes to the plugin's classpath.
    Adding the Mongo API classes to an NSF and using them directly from SSJS code might also be possible, but I prefer the plugin solution, because the MongoDB access is supposed to be a central service on the Domino server. In addition, by using a plugin we don't have any issues with Domino's restricted security manager that prevents operations to run properly in an XPages context (e.g. a lot of drivers are using Log4J for logging, which does not run well within an XPages application).

    Our two servlets are defined in the plugin.xml file of the plugin:

    <?xml version="1.0" encoding="UTF-8"?>
    <?eclipse version="3.4"?>
    <plugin>
            <extension point="org.eclipse.equinox.http.registry.servlets">
                    <servlet alias="/mongotest" class="com.mindoo.mongo.test.servlet.MongoTestServlet"
                            load-on-startup="true">
                    </servlet>
            </extension>
            <extension point="org.eclipse.equinox.http.registry.servlets">
                    <servlet alias="/mongogeo" class="com.mindoo.mongo.test.servlet.GeoQueryServlet"
                            load-on-startup="true">
                    </servlet>
            </extension>
    </plugin>


    That's all you need to declare your own servlet on Domino.
    After that, the servlet can either be accessed on the URL base level (http://server/mongogeo) or in the context of a Domino database (http://server/path/to/db.nsf/mongotest).

    If called in the context of a database, the Domino HTTP server first makes sure that the current user is allowed to access the database and if not, it displays a login prompt. A utility class (com.ibm.domino.osgi.core.context.ContextInfo) can be used to get session/database objects for the current HTTP request.

    In our sample, we use the first format for our Geo API query, because it's an open service of the server without access restriction. Anyone can use it and the URL is simple to remember.

    The second URL format (/path/to/db.nsf/mongotest) is used to access MongoDB data. This enables us to check the user against the Notes database ACL to see if he is allowed to read or write MongoDB data.

    Here is the method of our MongoDB access servlet that handles POST requests:

    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
            Session session=ContextInfo.getUserSession();
            Database dbContext=ContextInfo.getUserDatabase();
            if (dbContext==null) {
                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                            "Servlet must be run in the context of a Domino database", null);
                    return;
            }

            try {
                    String pathInfo=req.getPathInfo();
                    if ("/deleteplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doDeleteGridData(session, dbContext, req, resp);
                    }
                    else if ("/addplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doAddGridData(session, dbContext, req, resp);
                    }
                    else if ("/queryplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Read, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to read data", null);
                                    return;
                            }
                            doQueryPlaces(session, dbContext, req, resp);
                    }
                    else if ("/initdb".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doInitDbWithValues(session, dbContext, req, resp);
                    }
                    else if ("/cleardb".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doClearDb(session, dbContext, req, resp);
                    }
                    else {
                            ServletUtils.sendHttpError(resp, 500, "Unsupported command "+pathInfo, null);
                    }
            }
            catch (Throwable e1) {
                    e1.printStackTrace();
                    //avoid posting the full stacktrace to the user (for security reasons)
                    ServletUtils.sendHttpError(resp, 500, "An error occurred processing the request. The error has been logged. Please refer to the server log files/console for details", null);
                    return;
            }
    }


    The code is pretty simple: We check if the user is allowed to call the specific service (read operations require the role [MongoDBRead], write operations the role [MongoDBWrite] ) and then forward the request to helper methods.

    Note that I am not going into full detail in this blog article how we actually access data in MongoDB. You can find that in the provided download archive and there is a great tutorial at the MongoDB website about using the Java API.

    Believe me, this stuff is really easy to use!


    Click here for part 2 of this article!
    Comments

    1Mark Barton  01.05.2012 10:45:10  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Karsten,

    This is great stuff thanks for this.

    I am also a big fan of the REST architecture with strict separation of the UI from the backend.

    Also thanks for the heads up about using an eclipse plugin.

    Mark

    2Naveen  27.08.2012 8:23:50  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    I am trying to use you example but I am getting error at importing "com.ibm.domino.osgi.core.context.ContextInfo" that the imporcannot be resolved. What additionally I should do so that I can get it to work?

    3Karsten Lehmann  27.08.2012 10:48:44  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Naveen, I am currently on vacation and can take a closer look next week. Have you looked at the plug-in dependencies of the download archive? ContextInfo is only available on the Domino server, afaik, not on the client.

    4Naveen  28.08.2012 18:47:56  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Yes Karsten I am using it on local Notes client.

    But I discovered a way to get it on local machine in Domino Designer. I went in Package Explorer, in the NSF database I opened the plugin.xml file, in Dependencies tab added "com.ibm.domino.osgi.core.context" in Required plugins. I was then able to use the class "com.ibm.domino.osgi.core.context.ContextInfo" in my Java code inside NSF database.

    Any ideas where "com.ibm.domino.osgi.core" package is stored? Because even after importing Notes.jar in my Eclipse IDE it does not show up while trying to add in my plugin.xml.

    5Karsten Lehmann  29.08.2012 16:19:11  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    It should be part of a file com.ibm.domino.osgi.core_<versionnr>.jar somewhere in the Domino server (and maybe Notes Client) installation.

    6Genachka  21.02.2013 20:47:40  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    I am trying to use Couchbase as my NoSQL db but ran in to the issues you described in another blog entry about the permissions. I would like to learn how to create a Domino OSGi Plugin to bypass the issue. Any chance you would create a step by step for that and/or share the code for you MongoDB to learn from?

    7John Curtis  08.03.2013 18:49:17  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Hi Karsten,

    We are trying to reach you for a BP meeting with IBM dev engineers. Please e-mail me so I can schedule you in.

    Thanks,

    John Curtis

    8Karsten Lehmann  08.03.2013 19:41:55  XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Hi John,

    sure, I dropped you an email.

    Best regards,

    Karsten