One of the technologies Engineering.com said they currently use is Elasticsearch, so we needed to set out to learn what that was and how to integrate it into what we are creating. At first I thought it was just some Amazon proprietary technology. Not that this mean's it's bad, just that I get more excited about open source technologies, because if I learn how to use them once, I can use them freely in anything else I create, be it open source or other projects at work.
I was pleasantly surprised to learn that it was actually a pretty stable, useful, open source framework. It's basically a program you run that abstracts out a lot of the complex data analysis and machine learning type stuff, so that you can do queries against it similar to how you'd do queries against a database. "Show me articles that are similar to this one..." etc. It uses its own data store to perform these complex queries and filters from. So some people will just use the Elasticsearch data store as their main database, or they just need to figure out how to sync their database with it if they've already got something started and they want to add Elasticsearch to it. This is our situation. Engineering.com has a primary data store in their CMS containing their articles, etc. They already sync it with Elasticsearch, and we'll need to make sure we integrate our information that way too if that's applicable.
I spent some time today learning about Elasticsearch as an open source framework, that way when it comes time to use Amazon Elasticsearch, Amazon's implementation of it that Engineering.com already uses in production, I'll better understand it. They have an an official book which is available online for free, and I was able to find some nice YouTube videos which show you how to dive in. So far, my team mate and I imagine our system being a combination of querying the data we collect and the stats crunching Elasticsearch can do, to return really useful information for the client. I'll be focusing a lot on continuing to learn and practice using Elasticsearch over the next little while.
Note: This was originally posted on the blog I used for my co-op term while at Seneca College (mswelke.wordpress.com) before being imported here.