This week, we made a couple minor changes to the platform that will be important for new pack releases. First, we re-added the ability to have teaserpages, which are always a big hit, and are once again hosting the old teaser pages for existing packs. The second thing we did is a follow-up to a change we made a couple months ago. In March, we removed some older packs from the list of default packs in the launcher. However, those packs' pages still said "Included in the Technic Launcher"! You could still search for these packs from the launcher, but it was frustrating if you were trying to install Tekkit Classic on a new launcher. All official packs that are not included with the launcher now have a Platform API Link on their Platform page.
Additionally, this week we will be reinstating stats in the launcher. The path to get us there has been a long journey from sct but we're just about ready to unveil that, so look forward to it!
A Word About Hack Slash Mine
This week, we removed Hack Slash Mine as an official Technic pack and we wanted to drop a quick word about that. We really like HSM and have been playing it for years, so it's not a decision we came to lightly. Unfortunately, recent Java updates have broken Hack Slash Mine and Frizzil has not been reachable. Frizzil is now a professional game developer who's worked on some really impressive titles, and doesn't usually have time to work on HSM. As a result, we understand why it hasn't been updated, but unfortunately, we can't endorse the pack now as we could a few months ago. We also have no means to fix the problem ourselves.
Users can get it running by installing Java 6 or Java 7 and selecting that java version from Launcher Options, so it is runnable. We are not going to remove it from the Platform as we do with other packs that aren't Java 8 compatible, and we will continue to host and maintain the pack the best we can. However, it will no longer automatically appear at the top of relevant search results and will no longer appear in the official modpacks list on the Platform. We apologize to fans of the pack and hope that Hack Slash Mine will return again someday.
That's Pretty Much It!
It was a quiet week, but a lot is going on just under the surface- you'll see soon! Come visit us on IRC, our forums, or reddit, and tell us what you'd like to see!
It's been awhile since you've heard from us, but we haven't been idle! We've been working hard on a lot of fronts to make 2015 the Summer of Technic! You're going to be hearing from us a lot more often over the next 3 months, but let's round up all the great things we've been up to!
Welcome To Florida
Over the past 6 months, CanVox and Technic's artist, TheHave, have been undertaking the cross-country move necessary to put them in Florida, home of KakerMix and sct. This has taken up a lot of our time, but we're now closer than ever, and getting more done than we thought possible. This move has been really stressful, but you're going to start to see some real benefits!
Welcome, Zehn!
No, he's not yet another Pack Welder. Zehn is a new Technic guy who's going to help us out with your support questions and also work Modpack reports so we can respond more consistently to problem modpacks, and restore modpacks when they've been fixed. We know that this has been a problem area for us, so it's Zehn to the rescue!
Are You Ready for New Packs?
This is going to be the summer of brand new 1.7.10 packs for Technic! We've been working hard on packing, testing, and providing exclusive Technic content for all of our packs! The first one will be released by the end of June and we think you're going to be pretty impressed. This summer, you can look forward to a new and improved Official Technic version of Blightfall, Tekkit, and Hexxit. And by the end of the year, a new, super-secret pack that we can't even talk about!
In the meantime, enjoy these teasers for some of the great new stuff coming in Hexxit II!
Boots of Ultimate Parkour:
A brand new mod allowing magical abilities to be added to the game!
Platform, Platform, Platform!
Some important improvements have been made to the Platform lately to really improve the speed and scalability of the website. You may have noticed that in-launcher stats display and the website's personal Feed have been shut off. This is a temporary measure- soon, both features will be back up and faster than ever, allowing for great performance on the weekends once again. Let's go over what we've done so far:
First, stats tracking. Stats have been a major scaling problem with the platform for some months. We wanted to store the stats in a way that would allow pack authors to produce reporting and see trends. While the initial data structure supported that, it was too granular, and the database bloomed out of control.
We've fixed this problem for new data and old data is being processed to be less granular. Unfortunately, the processing is taking much longer than we anticipated. We're currently working on a solution to allow the website's Redis-based stats display to be used on the API, so that stats display can return to the launcher before the database table is ready to be used.
Secondly, new Solder linkages with the Platform. It used to be really easy for a badly-configured Solder server, when linked to the Platform, to break your pack's platform page with very little information. We've fixed this issue, but it's kind of squirrely so please let us know if you're bitten by this in IRC.
Next, database load. Most site slowness is caused by load on the database from lots of people using the site. We've re-examined the way we're retrieving certain types of data (mainly, ratings and whether you've +1'd an item you're looking at). We'll be reorganizing the queries for these over the coming weeks to reduce load and allow us to bring back the dashboard feed and even expand functionality without hurting performance. If we need to, we're not unable to or opposed to upsizing the database, but we'd like to try optimization first. Please bear with us while we explore solutions.
Lastly, we had to add a throttle to the launcher API. We didn't want to, but unfortunately someone chose to execute a DOS attack against us in April that targetted us by fishing for cache misses to place load on the database. We were only able to defeat the attack by placing an API command limit with a cooldown. We're very sensitive to legitimate users being bitten by this limit, so please make us aware if it's a problem for you.
We're also tracking an issue with the platform currently where sometimes when a pack owner or contributor saves the pack, it fails to hand off the new data to the API. If you've saved your pack, restarted the launcher, and the new data isn't visible in the launcher. Try resaving the pack on the platform and restarting the launcher again. This generally fixes the issue, and we will be implementing a possible fix this week.
Launcher Updates: v4.301
4.301 was released some time ago, but here are the release notes for that version.
v4.301:
Updated Czech translation
Updated French translation
The library seek & download method has changed. It now uses the Maven library "Aether" to build a real dependency list for select libraries in version.json instead of merely building a URL from the artifact data and downloading it. This allows version.json to support things such as SNAPSHOT builds, which is vital for experimental libraries such as Mixin.
When the modpack's version.json overrides a dependency library version in minecraft's version.json, we will no longer preference minecraft's version during the launcher problem. This fixes an issue which was occurring with newer version of forge in which packs were crashing on launch because forge wanted a newer version of a library and we were not correctly providing it.
And Much, Much More!
When we say this is going to be the Summer of Technic, we mean it! More great stuff for players, more great stuff for pack authors, server owners, this is going to be nuts! Keep checking back here and keep playing!
If you were keeping up with Technic Tuesday over the past month, you'll know that we've been uncharacteristically silent these past two weeks. There's a couple reasons for that- for one, the site and API have been behaving, meaning that a lot of people care a little bit less about what we're up to these days. The second reason is we are deep into crunch on a pack that will be released very, very soon. We'll tell you more when we're ready to get you hype.
Welcome, Talonos!
If you haven't played Blightfall yet, just go ahead and stop what you're doing and go play it. It's a perfect example of the sort of pack we'd love to see more of on the Platform, and we're very excited to have it here. What we're even more excited about is that Blightfall's author, Talonos, has joined Technic as a Pack Welder! We have him working on our best, most super secret project and we've already seen some really incredible progress! We know it's going to be a great pack and we can't wait to tell you more... soon! (TM)
So Let's Talk Uptime
What are the numbers these past two weeks? Well, we're definitely proud of them, let's take a look: here are the statistics for Tuesday, February 17 through Wednesday, March 4.
Platform Uptime: 99.69%
Fresh Modpack Installs: 2.6 million
Modpack Runs: Over 12 million
Launcher API Uptime: 99.96%
Launcher API Estimated Satisfaction (T=0.1 seconds): 97%
Launcher API Requests: About 300 million
Official Solder Uptime: 97.05%
Official Solder Estimated Satisfaction (T=0.1 seconds): 91%
Official Solder Requests: About 140 million
We've been having some occasional solder outages on the weekend, culminating into a a pretty major one this past Saturday. We're currently working on porting the API half of solder to NodeJS so that we can give it the same treatment we gave to the Launcher API. This alternative NodeJS implementation will be open source for anyone who wants it, but our belief is that only that official Technic solder receives a level of traffic that justifies its use.
Some Pretty Great Platform Changes
These past couple weeks, while I've been getting this new pack in order, sct has been working on some truly excellent changes to the platform. Pack pages are now better-laid out for pack authors to communicate with their users and drive discussion. He's also been hunting down some of the remaining issues and sprucing up the admin site for us. Here's the changes that matter to you guys:
We made changes to the way the Platform's Redis cache is configured. You should now experience far less 500 and 520 errors while navigating the Platform.
Packs now have a Discussion tab which allows all users to post status messages to the modpack.
The Overview of a modpack now contains the data which used to be contained in the About tab.
The About tab has been replaced with an Updates tab, which now contains the pack news which used to be in the Overview tab.
The "announcement" widget in the right gutter, which is usually the latest Technic news, will be a modpack's latest update when navigating a modpack page.
Made a few improvements to the way the site acts in maintenance mode.
Speaking of Maintenance
Our host for everything except the Launcher API has been performing rolling maintenance on most of our boxes over the course of this week. This is a mandatory maintenance for us, and we do not have the power to stop or reschedule it. You may have noticed a brief outage a few nights ago as the search box had maintenance performed on it. There may be momentary glitches throughout the weekend as various web servers and nodes have maintenance performed, but our balancers should pull the serviced nodes out of the rotation almost immediately after the service begins. There are two key maintenance windows you should be aware of. Sunday evening, there will be some brief website downtime as the site's cache machine is serviced. On Saturday evening, there will be downtime on both the website and the API as the database is serviced. This will mean a worldwide launcher outage during which you will not be able to install or update packs. These outages will both last less than 20 minutes and the services will be restarted just as soon as we get the OK from our hosts.
Older Packs And You
If you don't play our older packs (Tekkit Classic, Tekkit Lite, Yogbox, Big Dig, Hack Slash Mine, and Voltz) you may notice them missing from the launcher now. Don't worry, they're still here! You can pull them up and install them from the Search or Add Packs bar just like any other platform pack. Additionally, users who have these packs installed will still see them in their rightful position. Some of these packs, such as Hack Slash Mine or Voltz, could make a return as default packs in the launcher if they receive an update from the individuals who they now belong to. But for now, we wanted to hand over more launcher real estate to your packs, so it's easier to play a lot of Platform packs without doing a lot of scrolling. These older packs were taking up a lot of space for people who aren't too interested in them, so now only currently-maintained official packs have a slot in the launcher by default.
Currently, the only way to add these hidden packs are by search in the launcher. We're currently working on adding the ability to copy platform API links from these official packs' pages just like any other Platform pack. In the meantime, just search!
Launcher Updates: v4.283
These are all the launcher changes since v4.272: v4.282 has been live since February 24. v4.283 went live tonight. These changes are not significant, as I have been spending my time working on packs these past few weeks.
v4.283
Updated German launcher translation.
Updated Chinese (Taiwan) launcher translation.
Some behavior fixes for official packs that have been removed as default packs.
v4.282
The currently-detected OS is now printed in the log at launcher start.
Minecraft now correctly receives its logging configuration from FML instead of the Mojang launchwrapper. This will allow Minecraft to format its logs correctly when launched from the Technic Launcher.
The statistics for a modpack will now round to the nearest value instead of always rounding down. This should cause the statistics shown in the launcher to more closely track those shown on the Platform. There will still be occasions, due to API caching, when the launcher will show slightly lower values than the Platform, however.
The statistics for a modpack will now always show at least two significant figures at all times. So "5K" will become "5.2K", etc.
The incremental garbage collector has been turned off in Minecraft once again. We chose to experiment with forcing Minecraft to run with it because of the sheer number of sources advising us to use it. However, it made performance significantly worse in a number of situations.
Errors when attempting to authenticate with Mojang will no longer show an error to the user identifying the failed connection as "auth.minecraft.net". The target server will be correctly identified in error messages as "authserver.mojang.com".
As always, thank you for reading, and thank you for playing Technic! Feel free to stop by IRC and reddit and let us know what you've built lately. And don't forget to play Blightfall and tell Talonos what you think!
Well, folks, I'm not gonna sugar coat it. Here, I'll pad with some new stats to make it look slightly better. From Tuesday, February 10 through Monday, February 16:
Official Solder Uptime: 100%
Official Solder Estimated Satisfaction (T=0.1 seconds): 92%
Official Solder Usage: Over 62 million requests (Note: Last week we reported 18 million solder requests, but that was incorrect by a factor of 3)
Platform Uptime: 99.72%
Modpack Runs: Over 5 million
Fresh modpack installs: Over 1 million
Launcher API Requests: Over 94 million
Well that doesn't seem so bad.
Launcher API Estimated Satisfaction (T=0.2 seconds): 85%
Oh, well you can
Launcher API Uptime: 92.9%
Oh.
So, this week we worked on some side projects and packs we'll be able to announce later, but mainly we were working on this. As a result, I don't have anything that I can discuss this week except the Launcher API. I know that you want to hear that we'll fix it, so here's the good news: we fixed it three hours ago. Here's what happened.
Basically, sct Rewrote the API
We deployed the rewrite a few hours ago. It's working great, better than we imagined. It did break some stuff for about 20 minutes after we deployed it. Your launcher might have cached some bad data during that period of time, so if your launcher is acting weird, select the pack that's giving you guff and give it a quick moment to update from the API. I'm going to be investigating this week what I can do to make the launcher finish at least trying to refresh a pack before letting you play it. But otherwise things are good, really good. Here's all the ways the new API is better:
Previously we cached pack data for between 1 and 10 minutes (depending on the type of pack data) in the API after which we'd pull replacement data from the Platform. Pulling replacement data is pretty costly, so we wanted to not do it very often. But we also wanted to not annoy pack authors by having long cache times which would make it harder to see and test their changes. It turns out, different cache times for different data failed at both goals.
Instead, we now have long cache times, 60 minutes, but changing your pack on the platform kicks off a job that updates the API cache after just a few moments. Now you'll see your changes in the API instantly, but with huge cache times, our database can rest easy.
Another problem is that doing a cross-country data pull in the middle of a web request turned out to be a really great way to generate slow responses. For packs with a lot of traffic, a ton of DB queries could be generated between the time the cache expired and the time the first request re-filled it. Now if your pack receives a request in the 10 minutes before the cache expires, a job is kicked off to update the cache before it expires. This means that our heaviest-trafficked packs are also our fastest to serve: there is no time when useful data isn't in the cache.
This is the important part: the jobs that I mentioned are running on separate hardware from the API, meaning that when you're trying to use the launcher, you don't have to share time with the work being done to maintain the cache. With the exception of rarely-used packs, the web half of the API is basically a web service pulling text from cache and spitting it at you. If your pack is in the cache, your response will be basically instant.
We were trying to do a lot with the old version- different parts of the pack data had different cache times. We were trying to give one cache time to the stuff that had to be updated often while the less-updated stuff was cached for longer. The result was we had to check several different caches when building a response- even a fully-cached pack required several cache gets to build a response. Since the Platform is now proactive in rebuilding the cache, we can give everything the same cache length and cache a single, pre-constructed response. This means that even fully-cached pack data is much faster.
So what effect has this had on our service?
Well, the servers stopped crashing. So, there's that.
The response speed is so fast now, it's hard to explain. I'm looking at a graph of our response times over the last half hour- our slowest 1% of traffic peaked at an average response time of 0.35 seconds. Yeah. It was usually faster than that.
Our Estimated Satisfaction was so high that I had to reduce the expected response time from 0.2 seconds to 0.1 seconds just so that I can see a difference between good and "bad" traffic.
Previously, our traffic numbers were so spiky that we couldn't use autoscalers reliably and had to manually scale our hardware. That is no longer the case. We now have robots watching the shop for us, which means that we don't have to have someone wake up at 7AM on weekends to manually restart in case of an outage or scale in case of traffic surge.
Usage on the DB shared by the Platform and API is a third of what it was.
Oh, also our hosting bills are down. Adding separate worker hardware saved us money. Weird, right?
So obviously it's early days yet- it went live on a school night at the end of a 3-day weekend. But this early data is incredibly positive- it's night and day, even compared to this time last week. I want to thank all of you for sticking with it- I promise that these past few weeks haven't been in vain, they've been vital for us gathering data and building a plan. I understand that it doesn't make much sense for it to take this long, but it's tough to gather data on what's going wrong when you're handling traffic during the week without breaking a sweat and getting murdered on the weekends. It's hard to see what works and what doesn't. The good news is we feel like this solution will end major problems with the API. I'm sure we'll find other problems to correct, in the same way we still find problems with the Platform. But we're now at the point where you can expect the Launcher API to be working on weekends. We look forward to serving you launcher data this Saturday! And I look forward to sleeping in on Sunday.
As always, stop by IRC and reddit to tell us what you're playing, and thanks for reading!
Before we dive in to this week's Technic news, I just wanted to remind everyone that if you're having problems logging in, using all or most official packs, using the platform or launcher, or creating or administrating a Platform pack, that the best place to get help is http://support.technicpack.net/ There are some articles you can read up on and if they don't help, "Make a Request" will let you open a support ticket and get direct help from a Technic staff member. A lot of people have been reporting packs or users as a way to get ahold of us, and that isn't a great way to do it. We don't have a good way of responding to you when you use a report, but on http://support.technicpack.net/ we have a whole ticket system where we can see when you get back to us and you can see when we get back to you. Your tickets will also help us refine the available articles so future users will get help even easier. Please use the new support site, it's really great and it's been tragically underused!
Technic Tuesday!
So with that out of the way, let's talk Technic Tuesday. This week, our new launcher build is 4.272 and while there will be a full changelog at the end of the post, I don't want to bury the lede: there is a major fix for Mac users that will drastically increase the number of packs that work on many Macs. This weekend, we were alerted to an issue with the changeover from Java 6 to Java 7 on Macs that has been negatively affecting our users- when Java 7 came out, the required file extension for JNI libraries changed. That sounds confusing, but the upshot of it is that Mac users on old versions of Java can't play new versions of Minecraft, and Mac users on new versions of Java can't play old versions of Minecraft. Dealing with this was a simple hack, but it's going to make a huge difference if you're playing on a Mac!
Let's Talk Uptime
Here's the stats for Technic's servers this past week, from Tuesday, February 3 through Monday, February 9-
Platform Uptime: 99.95%
Launcher API Uptime: 97.0%
Launcher API Estimated Satisfaction (T=0.2 seconds): 93%
Launcher API Usage: Over 46 Million Requests
Official Solder Uptime: 100%
Official Solder Estimated Satisfaction (T=0.1 seconds): 92%
Official Solder Usage: Over 18 Million Requests
So if you've got a trained eye, you probably notice that that Launcher API Uptime is a lot lower than last week- 97% is not a very impressive number. I've got a lot to say about this, so if all you want to hear is "we're working on it", we are, so feel free to skip to the next section.
A large part of the low uptime is due to a large outage we had on Saturday- in fact, when you remove Saturday from the equation, our uptime for the week was 99.6%, slightly better than last week. What it comes down to is that even though we've now dealt with the most obvious issues in the API, there still seem to be some problems that cause our API workers to gradually slow down when under load and stop responding. Since they are not technically crashed, our robots don't reboot them, we have to be around to do it manually.
This weekend, we added an awful lot of logging and diagnostics to the API to allow us to investigate, and it's already yielded some results- it told us that an issue we were previously aware of is much more serious than we thought. That is the issue of unupgraded launchers.
As you're probably aware, we had some issues during the rollout of the new site and launcher, the API had some bugs in how it handled traffic, but just as importantly, the launcher had bugs in how it used the API. While the site has long since been fixed, there are more than a few users who are using launchers over a month old and they are using far more than their fair share of our resources, sometimes in sharp spikes that make it harder for us to find a cost-effective way to provision our servers. To deal with this, we're going to be adding a simple parameter to all API requests where the launcher will pass in its build number, and requests that are missing that parameter (or include a rather old build number) will be rejected. It won't stop someone who's determined to bust our chops, but this is intended as a defense against well-meaning users accidentally hurting us. It should go live on the API in the next week or two, once we feel confident that almost everyone has the change. If you are a pack author who is using the Platform API to show pack data on an offsite page, please be aware of this change and realize that you can get the correct build number to use for your requests from this API request. Do the right thing and cache your data.
It's a little extra work, but it's going to go a long way toward allowing us to keep our traffic stable and provide a good experience for everybody. In addition to this change, we're going to be adding a feature that will cause launchers that have been open for 24 hours to check for a new build and alert the user to restart, just like Chrome does. We believe that this combination will protect our service from old, buggy launchers while supporting people with unusual usage patterns better.
Reports and The Modpocalypse
When the new Platform first launched, it included a new feature allowing you to report people, comments, modpacks, and other objects. For the first time, our users could collaborate with us on keeping garbage off the Platform and improving the overall quality of your pack search results. It only had one problem- the administration interface for handling reports was a real pain! Last Monday, we rolled out a new report administration interface that allows us to process user reports much more quickly. At the time it rolled out, we had over 1500 reports waiting in our queue! On Friday night, that count hit 0, and it's been kept near 0 ever since. So now we can safely say that we are listening! Your reports are getting processed and garbage is being removed from the Platform as it is pointed out to us.
The big reason why we had to roll out the new reporting interface is that this week is the MODPOCALYPSE. A few weeks ago, I posted a news item saying that starting on February 9th, any pack which is not compatible with Java 8 will be disabled as non-functioning. That's already begun. If you've waited too long and had your pack disabled for Java 8 incompatibility, all is not lost! Update to Forge 1230 or include LegacyJavaFixer in your pack and report your pack requesting to be reinstated. We will field your report in a day or two, verify that the pack is functioning again, and reinstate you.
Launcher Changelog v4.272
This week was heavy on news but light on updates. The only major change to the Platform is that we fixed a bug that would allow users to access packs that had been disabled, causing some confusion. In the launcher, the big headlines are the Mac fix previously mentioned and a small feature: if you'd like, you may now pull up a pack by pasting its Platform URL instead of its API link. The launcher changes for this week are the following:
Mac users will now be able to launch LWJGL on different java versions than what LWJGL was built with.
Pasting a pack's Platform URL will now pull up the pack the same as if you'd pasted the API link.
Added Polish language support.
Your launcher's build number will now be sent to the API when attempting to query. We will soon be rejecting API requests from old launcher builds.
Adding java versions from a file will no longer fail on windows.
Unzip failures will now have additional information logged to the console where necessary to diagnose issues.
Changed the color of the selected modpack.
Improved the look of the search and filter bar.
Limited the search bar input to 90 characters.
Thanks for reading, and thanks for playing! If you're interested, come join us on IRC or reddit, we'd love to hear what you're playing!
