UCombinator

Startup Submission 07: Adding a Database

Your startup is near the end of a long and challenging road. You tangoed with HTML and CSS, struggled with React and JavaScript, and wrestled with HTTP routes, requests, and responses. Now, it is time for your startup’s final task: Adding a database to your product.

Your product will still assume a single logged-in user. Originally, we were going to have your startup add user login to your product in an 8th startup submission. Due to time constraints, we will not ask you to do that; instead, we will release a final workshop before the end of the class in which you add proper authentication and encryption to Facebook to give you a taste on how that works.

Startup Submission

The overall goal of this submission is to migrate your server from using the mock database to using MongoDB.

If your product is in good shape, you should only have to make small changes to your client to accomodate MongoDB’s ObjectIDs, with the rest of the changes happening in the server.

Update Document IDs

As mentioned in the workshop, MongoDB uses ObjectIDs to identify non-embedded documents within the database, which are sent to the client as strings. Our mock database used numbers, which the client saw as numbers.

You will have to update both the client, server, and mock objects to expect these ObjectIDs. In particular, you will need to:

  • Convert mock objects’ mock database IDs to MongoDB ObjectIDs, like you did in the workshop.
    • This includes references to other Object’s IDs. For example, a comment’s author for Facebook contains a user’s ID. We converted that reference to an ObjectID in the workshop.
  • Update any hardcoded database IDs in the client to be the string version of MongoDB ObjectIDs, like you did in the workshop
    • This includes the user ID in the unencrypted JSON token!
  • Update server routes to expect the client to send database IDs as strings rather than numbers, which you did in the workshop.

Add resetdatabase.js, Insert Mock Objects, Hook it Up to “Reset DB” Button

To populate MongoDB with your mock objects, you will need to put your mock objects into a reset database script like you did in the workshop. Remember to change all IDs and references to IDs into ObjectIDs!

In the “Update Git Repository” section below, we link to where you can get an empty resetdatabase.js to put into server/src/resetdatabase.js.

Once resetdatabase.js contains your mock objects with IDs converted to ObjectIDs, you can run it from the command line to reset MongoDB’s server. Also, you should update the “Reset DB” HTTP route to use resetdatabase.js, like you did in the workshop. Once you make that change, clicking the “Reset DB” button in the web browser will reset the database.

Add mongo-express For Your Own Sanity

When things are going wrong, and your database is returning weird stuff, you will want to take a look at what you are storing inside it. For your own sanity, we request that you add mongo-express to your product, like you did in the workshop. Please see the “Update Git Repository” section below for information on how to add mongo-express as a dependency of your server.

Update Server Routes to Use Database

Your product’s server routes currently use the mock database to read and update material. Now, you need to change those routes to use MongoDB.

The database workshop covers converting the server to connect to the database, as well as how to convert many types of HTTP routes to use the database. You will need to make similar changes.

When a route updates data, you should be careful of atomicity issues. We briefly touch upon atomicity in the workshop, The main idea is that your database can change in any number of ways between the last time you read data and the next time you write to it.

So, if you currently have code that:

  • Reads a user object from the database.
  • Changes a few fields in the user.
  • Writes the user object to the database.

…other updates to the same user may happen between the time you read the user object and write it back to the database.

MongoDB supports update queries that let you change objects in the database using many types of operators. You should use these queries when you are changing data in the database.

Students may note that atomicity issues can also happen when reading from the database to resolve references. We are not concerned with these issues in this course, as it is not a destructive error – it does not erroneously delete database state. Instead, the user sees something weird, and refreshes the application to fix it. There are ways to deal with these issues, too, which I am happy to discuss with interested students in office hours.

Small Note: Embedded Arrays

I want to briefly mention a subject not mentioned in the workshop: Updating a particular element in an embedded array. MongoDB treats an array index just like the name of a field.

For example, in this object:

{
  "_id": new ObjectID("000000000000000000000007"),
  "planet": {
    "name": "Mars",
    "coords": [1212.1231, 2321.12312, 5.34334]
  }
}

I would reference the "name" of the "planet" as "planet.name" in an update’s filter or update document. I would reference the second "coords" of the "planet" (2321.12312) as "planet.coords.1" in an update’s filter or update document.

Why is this important? Many of your products contain database documents with embedded arrays. For example, Facebook status updates have a comments array:

{
  "_id": new ObjectID("000000000000000000000001"),
  "likeCounter": [
    new ObjectID("000000000000000000000002"), new ObjectID("000000000000000000000003")
  ],
  "type": "statusUpdate",
  "contents": {
    "author": new ObjectID("000000000000000000000001"),
    "postDate": 1453668480000,
    "location": "Austin, TX",
    "contents": "ugh."
  },
  "comments": [
    {
      "author": new ObjectID("000000000000000000000002"),
      "contents": "hope everything is ok!",
      "postDate": 1453690800000,
      "likeCounter": []
    },
    {
      "author": new ObjectID("000000000000000000000003"),
      "contents": "sending hugs your way",
      "postDate": 1453690800000,
      "likeCounter": []
    }
  ]
}

Let’s say that I wanted to update the contents of the first comment to say “hope you are OK”. This is the comment at index 0 in the comments array.

I would use the following query to make the change without any atomicity issues:

// Assumption: userId is the authenticated user's ID as an `ObjectID`
// The filter document makes sure that the comment at index 0 is authored by the authenticated user.
db.collection('feedItems').updateOne({
  _id: new ObjectID("000000000000000000000001"),
  "comments.0.author": userId
}, {
  $set: {
    "comments.0.contents": "hope you are OK"
  }
}, function(err, results) {

});

Constructing the above query when you have the comment index in a variable is a little bit tricky. JavaScript does not let you construct objects like this: { "comments." + commentIndex + ".author": userId }

Instead, you will have to do the following:

var query = {  _id: new ObjectID("000000000000000000000001") };
// Compute the string path to the author field of the target comment.
query["comments." + commentIndex + ".author"] = userId;
var update = { $set: {} };
// Compute the string path to the author field of the target comment
update.$set["comments." + commentIndex + ".contents"] = "hope you are OK";

db.collection('feedItems').updateOne(query, update, function(err, results) {

});

Now that you are using a proper database, you should finish implementing search/filters/etc. MongoDB queries should give you the flexibility you need to implement these features. In the workshop, we cover how to implement basic text search on the contents of status updates.

Note that a document collection can have at most one $text index. If you want to search multiple document types in a single document collection, you may want to consider putting them into their own document collection!

For example, if we wanted Facebook search to support searching comments, we would have a problem due to our database structure! Comments are embedded into feed items, and we already put a $text index on the status update text in feed items. We would need to create a new document collection for comments, change status updates to reference comments by ID instead of embedding them, and then add an index on the text of the comments.

Expectations for Third Party API Integration and Other Features

Your startup should continue to work on integrating with any planned third party APIs or web services. As in previous submissions, we understand the difficulty of working with new APIs, and will understand if your final submission has some rough edges. Please document any limitations.

If integration with a third party API proves too challenging, you can change the design of your product to avoid the integration. For example, if you planned to integrate with Amazon to grab a user’s purchase history, you may change your product so users enter their purchases manually into your app. Document any changes like this in your report.

Report

Include a report in reports/submission_07.pdf in your client repository with the following information.

Special Server Setup Procedure

As in the previous submission, if your server has any advanced features that require additional setup besides npm install and node src/server.js, include a section in the report that describes how to set up these features. Include details on how to tell if the feature is working properly.

Individual Contributions

Include a section that describes each startup founder’s contribution. Each startup founder should be responsible for at least one product feature and its HTTP route(s). Name the feature and the HTTP routes that the founder updated to use the database.

Each founder must be responsible for converting at least one HTTP route.

Lingering Bugs / Issues / Dropped Features

If your application has any lingering bugs, issues, or missing/dropped features, include a section listing these.

Similarly, if any particular features or components are not using the database, please mention those features here.

Grading Rubric

  • 10% Report clearly describes what feature(s) and HTTP routes each startup founder was responsible for updating to use the database
  • 10% HTTP routes continue to use unencrypted JSON token to check if a user has access to perform an operation
  • 30% Server uses MongoDB instead of mock database
  • 5% “Reset DB” button now resets MongoDB using resetdatabase.js
  • 5% Server uses mongo-express so database objects can be seen at http://localhost:8080/mongo_express (or whatever port you decided to use for your server if you didn’t use 8080)
  • 10% Updates to the database avoid atomicity issues with proper update queries
  • 30% [Individual] Founder’s product feature(s) is/are working properly and its HTTP route(s) use the database properly

Update Git Repository

Have one founder of your startup perform the following steps to update your repository for this submission.

  1. Download the resetdatabase.js module here, and add it to your project as server/src/resetdatabase.js.
  2. Open a terminal to the product’s Git repository folder, cd into the server folder, and run npm install --save mongo-express mongodb
    • Don’t forget the --save argument – that will update the server’s package.json with the new dependencies!
  3. add server/src/resetdatabase.js to the repository, commit and push to GitHub.
  4. Have the other members of your startup run npm install in the server directory to install the mongo-express and mongodb Node packages.

Submission

Please submit the URL to your team respository to Moodle by the assigned due date.