The first 2 weeks of October were rough. As many of you noticed, search was requiring a refresh or two to work – and sometimes didn’t work at all for a while. Search is a big part of customer support: you need to know if other people have had similar issues to the one you are responding to, find other discussions from the same user, point users back to older discussions or KB articles solving their problems. In short: search is essential.
So when our search engine started misbehaving, we immediately started investigating what was going on to fix it as fast as possible. The machines in the search cluster were running out of memory, requiring regular restart of the search engine. This caused troubles because the search engine was not available while it was restarting and re-initializing, but also responding poorly as it was reaching the limits of the machine. We added more machines and more memory almost immediately to at least alleviate the problem and keep search generally usable, but no matter how big the machine, the server would always run out of memory. It thus became clear that something else needed to be done.
The reason of this sudden memory hunger was a corruption in the index. We repaired the index a few times, but this didn’t seem to fix it, and so we had to create a brand new index, on new servers, to come back to a stable state. In the process, we made several changes to the way our index is structured: where previously all sites were part of one big bucket, we isolated all sites on the new index, which means that search and indexing are much faster. Also, if issues arise on one site, other sites are less likely to be affected, and we can work on that site specifically.
During this whole time, we worked almost 24/7 to keep things running and to build the new infrastructure. Tender is now running on this new index, on very powerful machines, and search is working properly and fast. We still have to iron out a few things, and work will continue during the next few days, but you shouldn’t notice any problems and we’ll do the maximum to make those changes without any downtime. If you notice anything wrong (search unavailable, missing discussions or KB articles), let us know as soon as you can so we can jump on it.
Right now everything is looking good and we’ll keep improving things, but if you have any question or problem, drop us a line at help@tenderapp.com, and we’ll get back to you ASAP.
Thanks for your support and your patience. You are the greatest customers one can hope for. We love Tender, and we’ll continue to do our best to bring you the best customer experience possible, both as staff and clients.
Cheers.
