I’m Mitch Trale: a Technologist at PucaTrade.com. I’m here to reflect on the transition we’ve made from PucaTrade’s Revised version to the Future Site version, which you are now using. This is Part 1 of a series of posts on the state of the site. I’ll write the next bit once we’re out of the rapids.
I often think of PucaTrade as a boat. Everyone who has ever used the site is still on the boat. Anyone who has active trades, stored value, or fond memories of Puca is aboard. We’re all out here every day, trading, exploring, and building something we’ve never seen built before. With the launch of Future Site, we have sailed the ship into uncharted territory.
We went to work at 11pm on August 20th, a Saturday night about three weeks ago. There were six of us there -- two engineers and four Ops guys. We had a plan, plenty of La Croix, and some positive nervous energy. We threw the Revised site into maintenance mode at midnight, migrated the database for six hours, transmuted data for another two hours, and ran some tests on our new production hardware.
More teammates arrived around 6am while we jammed some last minute fixes into the Magic Online bots with our partners at MTGOSS. Our community manager and site admins were in Discord, hyping folks up, answering questions on social media, getting everyone ready for the re-opening of the site. We brought PucaTrade up right around noon...
...And we took it back down within 30 minutes. The boat had rocked hard. We got hammered by thousands of eager members rushing the front door, ready to setup their profiles, stoked to link their MTGO accounts to their PucaTrade account, hoping to import 200,000 card CSVs to their collections, trying to load every page, and generally using the site in all of the new ways we’d hoped they would. We were happy to see so much enthusiasm, but dismayed to find that our infrastructure could not handle the load of launch day.
We assessed the situation. Our boat had leaks. Trades were going through, but there were some issues. Someone completed a 100 card package at our MTGO bot, which we were not prepared for, and it blew out our API. We saw socket errors and card pricer problems, our master database spiked, our replica lagged to a minute. We were taking on too much load, so we had to drop anchor.
We had patches in mind for the issues we were seeing but little energy to apply them. We continued to work through Sunday. A few of us left the office at 5pm -- 18 hours after arriving, tired and pissed off that the site wasn’t running smoothly already. I went home and had dinner with my kids and was generally miserable to be around until I took a nap and woke up at midnight to get back to work.
I remember how strange it felt to have PucaTrade offline that night. Midnight marked essentially 24 hours of downtime, which may be more darkness than we’ve experienced in sum total since we launched the Revised version in 2013. The longest downtime I can remember was about three hours in October 2015, when a third-party service knocked us out in a mysterious way. We’ve otherwise been proud of our uptime, and were anxious to get the site back online as quickly as possible.
I worked until 3 or 4 in the morning of the 22nd, got some sleep, woke up with my family, and saw them off to their first day of school. My wife teaches English at Oakland High School, and my girl Io is in 2nd grade. She rode the bus across town to her new school, and I met her there. Then I went downtown and got back to work.
The PucaTrade team came together that morning, recognizing that everyone needed to do everything in their power to steady the ship. We made a plan, prioritized problems, and worked all day. We brought the site back up Monday evening, about 40 hours after taking it down for migration Sunday morning, and we have stayed up since.
And so it has been for the last three weeks. We meet every morning to understand and decide to work on what matters most. We monitor the site constantly. We’re here day in and day out, resolving the most pressing problems first. We have made considerable progress, and we will continue to work until the ship is righted.
Many people have written in with messages of support, as well as constructive and destructive criticism. We welcome and encourage all feedback, and we appreciate those who take the time to correspond with us amicably. We understand that it is frustrating when technology does not work in a flawless way. We know there are things we could have done to make the transition smoother. And we recognize that Future Site has brought changes which have disrupted many people’s daily trading habits. We regret that these things have been necessary in order for us to progress the platform.
But this is our reality, and we are committed to pushing it forward. In a future post, I will explain in more detail why we made the technical and design changes we made, but for now, we are focused on the present.
Package trades are here, and we have begun to hear about new delight in using our enhanced auto-matching algorithm. We now support cards in every condition and language with a dynamic, first-of-its-kind pricing process in place to support them. We have seen 20,000+ digital card trades completed in under two weeks of our Magic Online bots being fully operational.
And a note on the new look and feel of Future Site: some people are crazy about the way the site looks now, and some people are driven crazy by it. If you fall into the latter camp, know that we have heard you and have near-term work in place to address your concerns: Day Theme will bring some new light, clean lines, and sharp images back to PucaTrade. We'll share more information and snapshots of this upcoming visual theme with you soon.
Not all of these things matter to all people, but PucaTrade must still carry the weight of expectation in all areas at once. Whether you care most about our classic paper trading, our new digital trading, historic trend and pricing data, our search tools, or are just here for the articles, we will continue to work to meet you where you are, and to give you good reasons to hang out on this site.
This is our commitment and it is our job.
We appreciate your patience and perseverance as we navigate these waters together.
Thanks for reading. Part 2 of this series will be published in the coming weeks.