Hi everyone. Unfortunately, yesterday evening the server responsible for our User database suffered a hard drive failure. Database fail-safes kicked in and kept all accounts online and accessible. However, custom themes which are stored as separate meta data needed to be restored and imported separately, making themes appear reset until the data was restored this morning.
While we were thrilled that the site survived the crash, we were very disappointed with the time needed to restore custom themes. As engineers who take a tremendous amount of pride in our work, we are meticulous in ensuring the reliability of our products. This particular delay was the result of our misunderstanding how the user database storage engine (InnoDB) interacted with our incremental back-ups. What should have been a 30-minute recovery kept all of us up until 4AM. Huge thanks to brilliant team at Rackspace (Will Kelly is our hero), and the impeccable Marco Arment.
As always, it breaks our hearts to inconvenience our users. And we’re putting more measures in place to ensure the reliability of our service.
Thank you for your continued support and understanding while we continue to expand our infrastructure. April will be bringing some very cool updates to Tumblr.
29 responses so far ↓
Mini ZuD // April 14, 2007 at 10:05 am
Your focus on users is impressive and appreciated!
Glad to be a part of this fine community.
Nick // April 14, 2007 at 10:24 am
I second Mini ZuD’s comment. A lot of companies still neglect to keep their user base adequately informed of problems, even in this age of “Web 2.0″.
Fortunately I had a personal backup of my edited theme so wasn’t greatly inconvenienced by the hard drive crash.
Keep up the good work. Looking forward to the updates.
Daniel // April 14, 2007 at 10:52 am
My posts on the subject pretty much sums up my take on all of this:
http://blogish.ohdanny.net/post/897160
http://blogish.ohdanny.net/post/912637
Long story short: Kudos for the speedy info and speedy recovery. I look forward to the new features!
Get some sleep.
Marc LaFountain // April 14, 2007 at 11:08 am
Hey, you guys stayed up untl 4 a.m. supporting a free product. How mad at you can we really be? ;-P
Seriously, thanks for getting things straightened out quickly. And, thanks for Tumblr. For me, it has made blogging fun again!
Marc :-)
null // April 14, 2007 at 11:19 am
Champion! I’m happy to say that I, for one, appreciate all of your efforts! :-)
Bill I // April 14, 2007 at 12:59 pm
Your dedication and the proof that your users come first is very reassuring. Thank you for you hard work.
Jason // April 14, 2007 at 1:12 pm
Wooo! Sorry you had to go through that, great to hear you’re back up and running! Keep up the good work! Hearts and bunnies!
Okay, maybe not hearts and bunnies, but we appreciate it! :D
lloydnebres // April 14, 2007 at 1:26 pm
It was startling to see all the Tumblr logs I visit (and my own) looking exactly alike. ::chuckle:: But hardly inconveniencing. I didn’t skip a bit at all.
A huge mahalo to you guys for your brilliant product and extraordinary, user-centric service.
It’s clear you love what you do. :-)
GreySheepDuo // April 14, 2007 at 1:43 pm
Aye. I come from Second Life, where Linden Lab routinely has issues like these they don’t seem to know how to fix. It’s nice to see a company that listens to their users, is so prompt to fix things, and keeps them up to date.
doug // April 14, 2007 at 1:46 pm
As someone who was up until 4AM this week bringing an entire data center back online after both a power and UPS failure, I have great empathy for you! You guys did a great job in keeping our logs up and running and getting things squared away fast. You can bet I’ll be signing up for a pro account once they become available.
maique // April 14, 2007 at 4:38 pm
thanks a lot for the effort.
it’s great you were able to fix it, although it’s a shame you had to work so late… thanks
Pat // April 14, 2007 at 8:03 pm
Great post! I appreciate it as both a Tumblr and as someone who’s interested in major hosting.
Just curious… what kind of package/deal/setup do you have with Rackspace?
Thanks!
jquig99 // April 14, 2007 at 10:52 pm
Get some sleep - and all your work is appreciated. Take care.
c0wb0yz // April 15, 2007 at 12:12 am
Thank you for your impressive work.
You’re terrific !
Hyperreality // April 15, 2007 at 5:39 am
Thank you guys for keeping us informed and for your hard work in restoring everything! Much appreciated! :)
coleslaw // April 15, 2007 at 3:33 pm
for some reason, shortly after the restore, all of my non-link text has gone italicized? Me or not me?
thoughtvoid // April 17, 2007 at 7:52 am
A belated thanks for the time you guys put in to getting it all back up and running.
The only things I lost were a couple of minor CSS changes I had obviously made since the last back-up.
Much appreciated, and for a free service, this was excellent… :)
topherchris // April 17, 2007 at 9:45 am
No problem, I wanted a new design anyway. :) I really did actually.
Thanks to the entire Tumblr team for their hard work.
Sarah Lane // April 17, 2007 at 6:14 pm
Are we having database issues again, by chance? My custom theme just re-arranged itself, and not in a good way! Yikes!
David Karp // April 17, 2007 at 6:22 pm
Hey Sarah! It looks like you have a stray <div> tag in your post: http://sarahlane.tumblr.com/post/1007822
That seems to be breaking the formatting.
Our HTML filtering is supposed to be protecting against that automatically, so we’ll check out what’s up.
Sarah Lane // April 17, 2007 at 6:23 pm
Hmmm, it happened when I posted a video from Dailymotion. Deleted the video, reposted, and everything seems to be ok now. Nothing to see here.
Marius // April 19, 2007 at 2:34 pm
Thanks for building & supporting tumblr :) I trully love it! What new features are brewing? (Please say support for visitor comments!)
Marius
Joel // April 20, 2007 at 11:05 pm
Honestly, I gave up and left Tumblr when everything broke down. I was actually in the middle of editing!
After calming down, I realized Tumblr is currently the best, fastest, and most convenient way of “tumblng” with our thoughts.
Thanks for continually taking care of us, your loyal tumblrs.
And more power to you!
iván s. // April 21, 2007 at 6:05 pm
Request: upload images to Tumblr when a user adds one to a regular post.
Kartik Agaram // April 23, 2007 at 5:39 pm
Request: please add a link to a feedback from from the tumblr dashboard.
Kartik Agaram // April 23, 2007 at 5:40 pm
Request: next/prev navigation links at each post/tumble permalink page.
Heath // April 26, 2007 at 4:58 am
I don’t really see anywhere to leave suggestions, but I just wanted to make a couple of them.
1- I love tumblr. I dislike writing blog posts because I am an obsessive editor, but with tumblr I feel okay writing a couple sentences about every link I submit. My first suggestion is that you make a way to integrate tumblr into blogs. I don’t want to convert my whole blog to a tumblr address, but I would like a WordPress plugin that checked if tumblr posts were available and used them as entries.
2- Make the submission process easier. Easier? Isn’t it extremely easy? Yes, but what might be nice would be a toolbar of some kind (I use flock). I use tumblr so often that it would be nice to be able to add a comment in a toolbar and select the type of comment and away you go.
Thanks for the great tool.
Doug // April 26, 2007 at 6:14 am
Big fan!
I like simple and effective tools.
Tumblr is a near perfect match for what I’ve been looking for.
I only have one item to add to the growing “wishlist”:
Is it possible to add a navigation element that can filter posts by content-type?
i.e. “Home (all) | Posts | Quotes | Videos | Pictures | Links | Conversations”
The “occasional blogger’s” posts get lost in the other tumbles - it’d be great to lasso them as either a reader or author.
Thanks.
yo man // September 1, 2007 at 9:10 am
Leave a reply
Leave a Comment