A Letter from DASH's Chief Engineer

June 15th 2015

My name is Bill, and I'm the fellow that runs DASH. I certainly appreciate that you may have experienced some issues with our site over the past several weeks. I will tell you that it has also been personally and professionally awful for us, and I am sorry for the frustration and inconvenience that this may have caused you and many other users. 

I wanted to give you a summary of the situation. We have been live with the DASH Marketplace for about 2 years now. We have had 4 interrupted auction days in total over that period, but 3 of those were unfortunately in the last 3 weeks. That is 4 too many interruptions, of course, but we're doing the best we can.

The fact of the matter is that we have grown very strongly over these past two years, and this growth caught up to us. Starting 3 weeks ago, we started working to improve our capacity to handle the load. We were not able to roll out our first improvement in time to prevent our first auction interruption. Since then, we rolled it out and discovered that we needed to do another enhancement. While we were working on that, our DNS provider (completely unrelated to the load issues) decided that we needed to re-work how users reached our servers via their ISPs. That caused some "downtime" even though our systems were fully functional.

Since we implemented that second enhancement for handling laod. it has resulted in a 300% performance improvement for many of our users while at the same time reducing our server load. Or so we thought...

An unintended by-product of our enhancement was to put additional load on our servers when a listing closed. This past Sunday evening, we were closing a listing about every 2.5 seconds for an hour.  So, after about 45 minutes of this, the actual load on our servers was astronomical - the fault being in the code of the very thing that should have made everyone's experience so much better. Two steps forward, but a frustrating step backward.

All I can say is we have now corrected this issue and that we will be "all hands on deck" to ensure that everyone's experience is seamless this week. As you can imagine, it is extremely difficult to simulate these circumstances. Though it may not seem like it, the team at DASH is very accomplished and has managed systems for literally millions of users.

Thanks to everyone for your patience and I hope to see you online throughout the week!

New Comment

Login to post comments.

If you don't already have an account, register for free to get started.

Comments

  • 2 years, 10 months ago

    Your company sells software. Your product is software. That is all you do. Yet, you cannot simulate the loads that you have been experiencing every week while doing revision upgrades and/or bug fixes?
    Having been a software project manager with developers scattered over 3 continents, the simple fact is you guys are in over your heads. Promised too much and are now under-delivering. Maybe if I wasn't intimately familiar with what you have trying to accomplish, I wouldn't be so critical.
    FYI: I would be much happier with slow service than no service as many of us have experienced in the last several weeks. You guys are killing what should be a fun and exciting activity. I shouldn't be this unhappy with my buying experience with Trainz.

    See jakesbabyboomertoyz's collection
    Reply
    Link

    Login to post comments.

    If you don't already have an account, register for free to get started.

    • 2 years, 10 months ago

      I appreciate your frustration, and I think these issues are behind us. The proof of that will come from us providing the same good experience we have delivered over the past 18 months (last several weeks notwithstanding). We are 100% focused on that.

      Everyone can draw their own conclusions, but I assure you that we are not "over our heads." Facing a steep 40% increase in load, we worked on some optimizations, and quite frankly, we made a mistake in one of the optimizations. That error was exacerbated by the load. That is behind us, and we will have a number of engineers online to ensure we deliver a good experience for you and the rest of our loyal users.

      Again, we all regret the issue and have worked very hard to improve the experience and be transparent about the progress over the last several weeks. I hope you will have a good experience on DASH from now on, and I hope you will enjoy some of the enhancements we are investing in for the good of the model train community over the coming months.

      See Bill's collection
      Reply
      Link

      Login to post comments.

      If you don't already have an account, register for free to get started.

  • 2 years, 10 months ago

    I was actively involved in a bidding during the period trouble was being experienced. As a "collector" always on the lookout for that "needle in the haystack" you stay persistent, at any price or difficulty to win/acquire your prospective prize.On that particular weekend your system difficulty added to the frustration but also the excitement of being close to the prize. Bottomline: you did not lose auction bid data, I hung in there and walked away with a prize acquisition for my collection.I think DASH is doing a fine job.As a professional systems engineer, there's always going to be a glitch somewhere in a growing program. Well Done, and I will be back.

    See skipschweizer's collection
    Reply
    Link

    Login to post comments.

    If you don't already have an account, register for free to get started.

  • 2 years, 10 months ago

    Like TRUMP says, get real, sometimes thing just don't go as we want....j

    See Uintxjay's collection
    Reply
    Link

    Login to post comments.

    If you don't already have an account, register for free to get started.