Technic Blog

Technic Tuesday: Shorter and Slightly Early

Well, folks, I'm not gonna sugar coat it.  Here, I'll pad with some new stats to make it look slightly better.  From Tuesday, February 10 through Monday, February 16:

  • Official Solder Uptime: 100%
  • Official Solder Estimated Satisfaction (T=0.1 seconds): 92%
  • Official Solder Usage:  Over 62 million requests (Note: Last week we reported 18 million solder requests, but that was incorrect by a factor of 3)
  • Platform Uptime: 99.72%
  • Modpack Runs: Over 5 million
  • Fresh modpack installs: Over 1 million
  • Launcher API Requests: Over 94 million

Well that doesn't seem so bad.

  • Launcher API Estimated Satisfaction (T=0.2 seconds): 85%

Oh, well you can

  • Launcher API Uptime: 92.9%


So, this week we worked on some side projects and packs we'll be able to announce later, but mainly we were working on this.  As a result, I don't have anything that I can discuss this week except the Launcher API.  I know that you want to hear that we'll fix it, so here's the good news: we fixed it three hours ago.  Here's what happened.

Basically, sct Rewrote the API

We deployed the rewrite a few hours ago.  It's working great, better than we imagined.  It did break some stuff for about 20 minutes after we deployed it.  Your launcher might have cached some bad data during that period of time, so if your launcher is acting weird, select the pack that's giving you guff and give it a quick moment to update from the API.  I'm going to be investigating this week what I can do to make the launcher finish at least trying to refresh a pack before letting you play it.  But otherwise things are good, really good.  Here's all the ways the new API is better:

  • Previously we cached pack data for between 1 and 10 minutes (depending on the type of pack data) in the API after which we'd pull replacement data from the Platform.  Pulling replacement data is pretty costly, so we wanted to not do it very often.  But we also wanted to not annoy pack authors by having long cache times which would make it harder to see and test their changes.  It turns out, different cache times for different data failed at both goals.
  • Instead, we now have long cache times, 60 minutes, but changing your pack on the platform kicks off a job that updates the API cache after just a few moments.  Now you'll see your changes in the API instantly, but with huge cache times, our database can rest easy.
  • Another problem is that doing a cross-country data pull in the middle of a web request turned out to be a really great way to generate slow responses.  For packs with a lot of traffic, a ton of DB queries could be generated between the time the cache expired and the time the first request re-filled it.  Now if your pack receives a request in the 10 minutes before the cache expires, a job is kicked off to update the cache before it expires.  This means that our heaviest-trafficked packs are also our fastest to serve: there is no time when useful data isn't in the cache.
  • This is the important part: the jobs that I mentioned are running on separate hardware from the API, meaning that when you're trying to use the launcher, you don't have to share time with the work being done to maintain the cache.  With the exception of rarely-used packs, the web half of the API is basically a web service pulling text from cache and spitting it at you.  If your pack is in the cache, your response will be basically instant.
  • We were trying to do a lot with the old version- different parts of the pack data had different cache times.  We were trying to give one cache time to the stuff that had to be updated often while the less-updated stuff was cached for longer.  The result was we had to check several different caches when building a response- even a fully-cached pack required several cache gets to build a response.  Since the Platform is now proactive in rebuilding the cache, we can give everything the same cache length and cache a single, pre-constructed response.  This means that even fully-cached pack data is much faster.

So what effect has this had on our service?

  • Well, the servers stopped crashing.  So, there's that.
  • The response speed is so fast now, it's hard to explain.  I'm looking at a graph of our response times over the last half hour- our slowest 1% of traffic peaked at an average response time of 0.35 seconds.  Yeah.  It was usually faster than that.
  • Our Estimated Satisfaction was so high that I had to reduce the expected response time from 0.2 seconds to 0.1 seconds just so that I can see a difference between good and "bad" traffic.
  • Previously, our traffic numbers were so spiky that we couldn't use autoscalers reliably and had to manually scale our hardware.  That is no longer the case.  We now have robots watching the shop for us, which means that we don't have to have someone wake up at 7AM on weekends to manually restart in case of an outage or scale in case of traffic surge.
  • Usage on the DB shared by the Platform and API is a third of what it was.
  • Oh, also our hosting bills are down.  Adding separate worker hardware saved us money.  Weird, right?

So obviously it's early days yet- it went live on a school night at the end of a 3-day weekend.  But this early data is incredibly positive- it's night and day, even compared to this time last week.  I want to thank all of you for sticking with it- I promise that these past few weeks haven't been in vain, they've been vital for us gathering data and building a plan.  I understand that it doesn't make much sense for it to take this long, but it's tough to gather data on what's going wrong when you're handling traffic during the week without breaking a sweat and getting murdered on the weekends.  It's hard to see what works and what doesn't.  The good news is we feel like this solution will end major problems with the API.  I'm sure we'll find other problems to correct, in the same way we still find problems with the Platform.  But we're now at the point where you can expect the Launcher API to be working on weekends.  We look forward to serving you launcher data this Saturday!  And I look forward to sleeping in on Sunday.

As always, stop by IRC and reddit to tell us what you're playing, and thanks for reading!

Technic Tuesday: February 10, 2015

Public Service Announcement

Before we dive in to this week's Technic news, I just wanted to remind everyone that if you're having problems logging in, using all or most official packs, using the platform or launcher, or creating or administrating a Platform pack, that the best place to get help is  There are some articles you can read up on and if they don't help, "Make a Request" will let you open a support ticket and get direct help from a Technic staff member.  A lot of people have been reporting packs or users as a way to get ahold of us, and that isn't a great way to do it.  We don't have a good way of responding to you when you use a report, but on we have a whole ticket system where we can see when you get back to us and you can see when we get back to you.  Your tickets will also help us refine the available articles so future users will get help even easier.  Please use the new support site, it's really great and it's been tragically underused!

Technic Tuesday!

So with that out of the way, let's talk Technic Tuesday.  This week, our new launcher build is 4.272 and while there will be a full changelog at the end of the post, I don't want to bury the lede: there is a major fix for Mac users that will drastically increase the number of packs that work on many Macs.  This weekend, we were alerted to an issue with the changeover from Java 6 to Java 7 on Macs that has been negatively affecting our users- when Java 7 came out, the required file extension for JNI libraries changed.  That sounds confusing, but the upshot of it is that Mac users on old versions of Java can't play new versions of Minecraft, and Mac users on new versions of Java can't play old versions of Minecraft.  Dealing with this was a simple hack, but it's going to make a huge difference if you're playing on a Mac!

Let's Talk Uptime

Here's the stats for Technic's servers this past week, from Tuesday, February 3 through Monday, February 9-

  • Platform Uptime: 99.95%
  • Launcher API Uptime: 97.0%
  • Launcher API Estimated Satisfaction (T=0.2 seconds): 93%
  • Launcher API Usage: Over 46 Million Requests
  • Official Solder Uptime: 100%
  • Official Solder Estimated Satisfaction (T=0.1 seconds): 92%
  • Official Solder Usage: Over 18 Million Requests

So if you've got a trained eye, you probably notice that that Launcher API Uptime is a lot lower than last week- 97% is not a very impressive number.  I've got a lot to say about this, so if all you want to hear is "we're working on it", we are, so feel free to skip to the next section.

A large part of the low uptime is due to a large outage we had on Saturday- in fact, when you remove Saturday from the equation, our uptime for the week was 99.6%, slightly better than last week.  What it comes down to is that even though we've now dealt with the most obvious issues in the API, there still seem to be some problems that cause our API workers to gradually slow down when under load and stop responding.  Since they are not technically crashed, our robots don't reboot them, we have to be around to do it manually.

This weekend, we added an awful lot of logging and diagnostics to the API to allow us to investigate, and it's already yielded some results- it told us that an issue we were previously aware of is much more serious than we thought.  That is the issue of unupgraded launchers.  

As you're probably aware, we had some issues during the rollout of the new site and launcher, the API had some bugs in how it handled traffic, but just as importantly, the launcher had bugs in how it used the API.  While the site has long since been fixed, there are more than a few users who are using launchers over a month old and they are using far more than their fair share of our resources, sometimes in sharp spikes that make it harder for us to find a cost-effective way to provision our servers.  To deal with this, we're going to be adding a simple parameter to all API requests where the launcher will pass in its build number, and requests that are missing that parameter (or include a rather old build number) will be rejected.  It won't stop someone who's determined to bust our chops, but this is intended as a defense against well-meaning users accidentally hurting us.  It should go live on the API in the next week or two, once we feel confident that almost everyone has the change.  If you are a pack author who is using the Platform API to show pack data on an offsite page, please be aware of this change and realize that you can get the correct build number to use for your requests from this API request.  Do the right thing and cache your data.

It's a little extra work, but it's going to go a long way toward allowing us to keep our traffic stable and provide a good experience for everybody.  In addition to this change, we're going to be adding a feature that will cause launchers that have been open for 24 hours to check for a new build and alert the user to restart, just like Chrome does.  We believe that this combination will protect our service from old, buggy launchers while supporting people with unusual usage patterns better.

Reports and The Modpocalypse

When the new Platform first launched, it included a new feature allowing you to report people, comments, modpacks, and other objects.  For the first time, our users could collaborate with us on keeping garbage off the Platform and improving the overall quality of your pack search results.  It only had one problem- the administration interface for handling reports was a real pain!  Last Monday, we rolled out a new report administration interface that allows us to process user reports much more quickly.  At the time it rolled out, we had over 1500 reports waiting in our queue!  On Friday night, that count hit 0, and it's been kept near 0 ever since.  So now we can safely say that we are listening!  Your reports are getting processed and garbage is being removed from the Platform as it is pointed out to us.

The big reason why we had to roll out the new reporting interface is that this week is the MODPOCALYPSE.  A few weeks ago, I posted a news item saying that starting on February 9th, any pack which is not compatible with Java 8 will be disabled as non-functioning.  That's already begun.  If you've waited too long and had your pack disabled for Java 8 incompatibility, all is not lost!  Update to Forge 1230 or include LegacyJavaFixer in your pack and report your pack requesting to be reinstated.  We will field your report in a day or two, verify that the pack is functioning again, and reinstate you.

Launcher Changelog v4.272

This week was heavy on news but light on updates.  The only major change to the Platform is that we fixed a bug that would allow users to access packs that had been disabled, causing some confusion.  In the launcher, the big headlines are the Mac fix previously mentioned and a small feature: if you'd like, you may now pull up a pack by pasting its Platform URL instead of its API link. The launcher changes for this week are the following:

  • Mac users will now be able to launch LWJGL on different java versions than what LWJGL was built with.
  • Pasting a pack's Platform URL will now pull up the pack the same as if you'd pasted the API link.
  • Added Polish language support.
  • Your launcher's build number will now be sent to the API when attempting to query. We will soon be rejecting API requests from old launcher builds.
  • Adding java versions from a file will no longer fail on windows.
  • Unzip failures will now have additional information logged to the console where necessary to diagnose issues.
  • Changed the color of the selected modpack.
  • Improved the look of the search and filter bar.
  • Limited the search bar input to 90 characters.

Thanks for reading, and thanks for playing!  If you're interested, come join us on IRC or reddit, we'd love to hear what you're playing!

Technic Tuesday: Launcher v4.265 is Stable!

Launcher v4.265 has just been pushed live, and it's time for Technic Tuesday!  Here's the changelogs for this week's Launcher update and all the latest Technic news!

Trending Recalculation is Fixed!

I made a newspost about this earlier in the week, but the Trending Modpacks list is updating once again, and now it's doing it without hurting the platform!  Last week, I posted that we had turned off the Trending recalculation because while it ran each night, it was damaging your ability to use the Platform and the API.  This was creating about 15-20 minutes of downtime each night and is now fixed.  We also fixed the last remaining stability issue with the Launcher API.    So how is the new and improved Platform working?  Here are the stats we have for the past week, Wednesday January 28 through Tuesday, February 3:

  • Platform Uptime: 99.97%
  • Launcher API Uptime: 99.54%
  • Launcher API Estimated Satisfaction (T=0.2 seconds): 93%
  • Launcher API Usage: Over 47 Million Requests

We're really proud of these numbers!!  There's still lots of room for improvement, though.  If you spend a lot of time on the platform, you'll know that there's occasionally random error pages that pop up and go away just as quickly.  We've got a plan for taking care of those this week, we'll see how it goes!  In the meantime, we hope you enjoy the new speed and stability on the weekends.

A Platform Facelift (Important Info for Pack Authors!)

In case you haven't noticed, the Platform received a minor facelift this week.  A few visual improvements to pages, including awesome artwork in the header of each page, some improvements to the way the pages handle the navbar, long titles and usernames, and better-looking page titles.  One important fact: page titles no longer have a drop shadow! 

If you're a pack author whose pack has a bright background, you may notice that your pack's title is now harder to read!  The good news is that we've added an option to your "Edit Modpack" menu that allows you to make the pack title dark-colored instead, to allow users to read your title against a whiter background.  Please check to see whether your pack title needs to be color-inverted in order to be seen.  A good-looking pack page attracts more users!

Get Help On Our New Support Page!

Another thing we've been quiet about is a gradual rollout of our new support site:!  A link to the new support site is now available from your Dashboard page, and a search bar for our help system is available at the bottom right of each page.  If you need assistance getting your launcher or the platform to work, feel free to visit and open a ticket.  Please don't open a ticket about a modpack- the place to get help for a modpack is still from the pack author.  If we're the pack author, you can get help on the forums tracker.

Changelog: Launcher 4.265

  • The available RAM settings in the launcher's Java Options is now based on the selected Java version instead of the launcher's Java version.
  • A warning will now show in Java Options if the RAM settings have been restricted due to 32-bit java being selected.  The warning will point out if a 64-bit version of java is available to be used.
  • The launcher now correctly handles non-Roman characters in modpack titles, descriptions, and other data.
  • The language selection dropdown box in the Login & Install menus will now use the same custom scrollbar as other UI elements.
  • If a file fails to unzip during pack installation, the resulting error message will now contain the file name.
  • The launcher now supports Hungarian.
  • The launcher's Czech translation has been updated.
  • Minecraft will now use the incremental garbage collector by default.

This week, we will be adding improved Maven support for version.json libraries and looking at what improvements we can make to the pack installation process to assist users.

Thanks for playing, and feel free to come join us in IRC or on /r/technicplatform.  We look forward to seeing you there!

Trending Recalculation is Back!

We wanted to write a brief news post to remind you that a few days ago we shut off the trending recalculation until we could make some improvements to keep it from hurting the site.  Well, last night at 4am US Eastern time, we ran the trending recalculation once again.  It took about 30 minutes and both the site and launcher API were completely acessible during that time.  Our launcher API and Platform uptime for the last 24 hours are both 100%.  We wanted to let you know that this is happening, not just to let you know that Trending packs are at 100% again, but also to point out one of the things we've been doing this month to fix the site's reliability.  We hope you enjoy the slightly more stable website!

Technic Tuesday: Launcher v4.256 Goes Stable

Hello, I've just pushed Launcher version 4.256 live.  4.255 was pushed live this weekend as part of a hotfix to reduce load on Solder.  New in this version is Java version selection and some bug fixes to the update process.  We also have a little bit of news.

Platform Changes

We're currently chasing down an issue that causes the Trending Pack recalculation to hurt the site as it's recalculated.  This has been the primary source of outages during the week since January 1.  We've temporarily turned off the nightly recalculation for a couple days while we fix this issue, but it should be back up in a day or two!

This Weekend

This weekend, we had some rolling outages.  There were a few issues that combined together to give you guys a bad time, but the main one was a Real Deal Serious Time DDoS that caused our host to temporarily null-route the Platform site's IP.  We were accidentally leaking some information about our site via our DNS records, and that was giving the attackers ammunition.  Once we found the leak and plugged it, things went back to normal pretty quickly.  We also had some unrelated trouble with the official pack Solder which has now been fixed.  The good news is that we operated for over a day with the DDoS shield up without any real damage to the launcher experience, and we're now very confident in our ability to fend off DDoS attacks, so we don't expect similar problems in the future!  We're looking into ways that we can keep the Discover page active during an attack so that in the future, you guys won't even notice the difference.

During the attack, we put together a launcher hotfix to alleviate pressure and that caused us to push Launcher version 4.255 early.  Tonight we've pushed version 4.256.  We apologize for the double update, but it was important for getting the site running as well as possible.

Changelog: Launcher 4.256

  • There is now the capability to select the java version which Minecraft will run with in the Java tab of Launcher Options. By default, Minecraft will run with the same version of java which is running the launcher, but you may configure it to select an individual version of java installed on your system, or to use the highest-versioned 64-bit version of java known to the launcher.
  • Modpack installs on the Platform are now being properly incremented again.
  • In situations where the local environment prevents the Launcher from setting java properties the way it wants, the Launcher will no longer restart itself endlessly in attempts to fix the java properties.
  • The launcher will now be usable after launching minecraft if the launch action "Stay Open" is chosen in Launcher Options.
  • The launcher will no longer fail to update when placed in a directory whose path includes + symbols.
  • Made some improvements to launcher behavior when overwriting the existing launcher during an update, in order to allow us to better diagnose problems that users encounter.
  • Improved some logging when the launcher fails to pull version info from a pack's linked solder.

This week, we'll be working to improve our Java selection by tying the available RAM options to the selected Java version instead of the launcher's Java version.  We'll also be improving our Maven support for pack authors who choose to include custom libraries in the version.json file.

Thanks for playing, and feel free to come join us in IRC or on /r/technicplatform.  We look forward to seeing you there!