Putting Solr into the kitchen
Posted by Ian Holsman
One of the things I’m responsible for at AOL is their use of Solr in their upcoming web developments.
a task that we keep on finding ourselves doing is taking a input feed (be it CSV, XML, or DB table) and transforming that into a Solr Index. (we call it injestion), it’s a boring and thankless task, but it is critical to get it done correctly. Especially when you need to deal with real time and batch updates.
This led me to have a reason to try out Kettle, which is a open source ETL engine to do these kind of things. But out of the box it had no support for Solr :-(
So I created this proof of concept plugin to show how easy it could be to just shove a data stream into solr, and am trying to get a demo going showing how easy it is to take some input data and make it into a Solr search engine (as well as other things at the same time).
It works well enough for me to do a proof of concept with a couple of different feeds and show the channel development teams how easy life could be.
disclaimer: before you go and start using it in production, please be aware that it needs alot more work when it comes to setting options and stability.
So if your interested in this type of thing.. feel free to ping me and I’ll add you to the project. (with the aim that either Solr or Kettle take this and make it part of their standard packages)