Jekyll2021-12-17T15:36:41+00:00https://www.erichartzog.com/feed.xmlEric HartzogThe projects and blog of John Eric Hartzog, including the homepage for StickWars, the hit iOS game.J. Eric HartzogFabriq & Reals2021-09-01T17:00:00+00:002021-09-01T17:00:00+00:00https://www.erichartzog.com/projects/fabriq-reals<h1 id="reals">Reals</h1>
<p>A private social messenger to encourage authentic sharing in a safe space.</p>
<p>Link: <a href="https://www.reals.app">https://www.reals.app</a></p>
<p>My work included:</p>
<ul>
<li>Designed client architecture to support a social messaging application.</li>
<li>Lead the team in the full stack solution for capturing and sharing videos between clients.</li>
<li>Shaped styling & component design methods for front end team.</li>
<li>Built CI/CD pipeline to test and deploy apps all the way to App & Play store.</li>
<li>Established guidelines and processes for ensuring unit test coverage of front end business logic.</li>
</ul>
<h1 id="fabriq">Fabriq</h1>
<p>A delightfully visual way to remember to stay in touch with those you care about.</p>
<p>Link: <a href="https://www.ourfabriq.com">https://www.ourfabriq.com</a></p>
<p>Some of my work on the app:</p>
<ul>
<li>Reduced TTI from 8 seconds to <3.5 seconds.</li>
<li>Revamped the styles and component design to allow for reusability and style consistency between all components.</li>
<li>Integrated third party library to provide advanced push notification features (actions & media) on both platforms.</li>
<li>Built a headless plugin to allow reliable actions based on incoming Android notifications.</li>
</ul>J. Eric HartzogRealsNerd Fitness Coaching2019-02-01T17:00:00+00:002019-02-01T17:00:00+00:00https://www.erichartzog.com/projects/nerd-fitness-coaching<p>The Nerd Fitness Coaching apps are a suite of apps designed to allow trainers to manage and communicate with dozens of clients.</p>
<p>Separate Apps:</p>
<ul>
<li>A client facing iOS/Android React Native app</li>
<li>An extensive React web app for trainers to manage their clients</li>
<li>A client facing React web app</li>
</ul>
<p>Features Include:</p>
<ul>
<li>Workout designs</li>
<li>Tracking workout progress</li>
<li>Calendar scheduling</li>
<li>Dietary journalizing</li>
<li>Messaging</li>
</ul>
<p>Native apps are available on the <a href="https://apps.apple.com/us/app/nerd-fitness-coaching/id1393166838">App Store</a> and <a href="https://play.google.com/store/apps/details?id=com.nfreactnativecoachingcustomerclientapp&hl=en_US&gl=US">Play Store</a> but require a membership to fully explore.</p>J. Eric HartzogThe Nerd Fitness Coaching apps are a suite of apps designed to allow trainers to manage and communicate with dozens of clients.Code module sharing for React Native and Web2018-08-22T17:00:00+00:002018-08-22T17:00:00+00:00https://www.erichartzog.com/blog/module-code-sharing-for-react-native-and-web<p><strong>Edit (11/1/2018)</strong>: After I finished this work, I found <a href="https://github.com/wix/wml">wix/wml</a> which solves the file syncing issue with minimal pain. While the approach in this post is still generally applicable, I’d recommend starting with <code class="language-plaintext highlighter-rouge">wml</code> rather than using custom scripts to sync the module files.</p>
<hr />
<p>When reading about various options for native app development, a huge pitch for using React Native (RN) is that you don’t need to switch context & languages, just use what you already know from web apps and it will mostly work!</p>
<p>While this is definitely true for almost all projects, if you’re assembling a product with distinct but similar web and native releases, it’s not just enough to be able work in the same general language, but you want to be able to share common code and functionality between your projects.</p>
<h2 id="the-smaller-code-ecosystem-of-react-native">The smaller code ecosystem of React Native</h2>
<p>Without getting to deep into the weeds, React Native operates differently enough from the typical browser based DOM environment that only a fraction of third party code written from React web will operate seamlessly when pulled into a RN project. This is especially true for anything UI related, as anytime a project throws a <code class="language-plaintext highlighter-rouge"><div></code> into its output, it’s not longer good to go for RN.</p>
<p>After building a moderately complicated web app, I was used to pulling in well written UI libraries to handle a large amount of simple UI aspects, such as <a href="http://allenfang.github.io/react-bootstrap-table/">react-bootstrap-table</a>, <a href="https://github.com/JedWatson/react-select">react-select</a>, and <a href="https://github.com/vazco/uniforms">uniforms</a>. These libraries are not perfect, but tend to save you dev time while giving most of the features and UI that you need.</p>
<p>Later on when starting out with React Native, I was surprised to find the ecosystem a lot more limited than I initially expected. There was not a single go-to solution for styling/theming an app, with popular solutions like <a href="https://nativebase.io/">NativeBase</a> being rather poorly documented and buggy implementation on basic features. I don’t mean to bash their work, but it’s a result of a working with a ‘bleeding-edge’ framework with a smaller dev pool to work with the projects.</p>
<p>As a result of the lesser amount of high-quality libraries, we ended up using a larger amount of custom components than initially expected. The good thing about doing this is React components are actually pretty quick and stable to create, if you do it right, and you can expand them to fit exactly what you need.</p>
<h2 id="sharing-react-native-and-web-code">Sharing React Native and Web code</h2>
<p>After trying a number of methods and hitting issue after issue, the our final solution ended up being relatively simple:</p>
<ul>
<li>
<p>Maintain all module code in a <a href="https://github.com/jehartzog/rn-web-shared-modules">top level directory</a></p>
</li>
<li>
<p>Use a simple <a href="https://github.com/jehartzog/rn-web-shared-modules/blob/master/native-project/package.json#L15">build script</a> to copy over all the module code into each project directory</p>
</li>
<li>
<p>When needed to separate RN vs Web specific code, create <a href="https://github.com/jehartzog/rn-web-shared-modules/tree/master/modules/src/fancy-text/src">two copies</a> of a file to be imported with the RN version using the extension <code class="language-plaintext highlighter-rouge">.native.js</code>.</p>
</li>
<li>
<p>Add the copied files to <a href="https://github.com/jehartzog/rn-web-shared-modules/blob/master/native-project/.gitignore#L19"><code class="language-plaintext highlighter-rouge">.gitignore</code></a> and optionally <a href="https://github.com/jehartzog/rn-web-shared-modules#code-editor-configuration">prevent them</a> from showing up in your code editor search results & file tree.</p>
</li>
</ul>
<p>We use <code class="language-plaintext highlighter-rouge">rsync</code> during development, so it only updates files that are changed, and both CRNA and CRA build tools pick up on these changes and refresh the project right after you hit save. This ends up with a process that gives up single-digit second iteration speed when editing modules, along with confidence our web and native module code is always in sync.</p>
<p>Although not done in the example project, this also allows you to import modules from your projects using absolute path for both RN and Web, for example <code class="language-plaintext highlighter-rouge">import FancyText from 'src/modules/fancy-text'</code>, avoiding the need to remember deal with dropping in large amounts of <code class="language-plaintext highlighter-rouge">../../..</code> in larger projects.</p>
<h2 id="the-cleaner-methods-that-didnt-quite-work">The ‘cleaner’ methods that didn’t quite work</h2>
<p>There are a number of far more ‘correct’ ways to do this that simply didn’t pan out. Here are a few of them along with why they didn’t work.</p>
<ul>
<li>
<h3 id="use-local-npm-packages">Use local npm packages</h3>
<ul>
<li>
<p>This seemed by far the best way to manage <a href="https://docs.npmjs.com/getting-started/installing-npm-packages-locally">local packages</a>, but after npm v5 this is done via <a href="https://stackoverflow.com/questions/44624636/npm-5-install-folder-without-using-symlink">symlinks</a>. Unfortunately React Native build tools <a href="https://github.com/facebook/metro/issues/1">do not support symlinks</a>, and it has been this way for over a year.</p>
</li>
<li>
<p>This also means using <code class="language-plaintext highlighter-rouge">npm link</code> will not work out.</p>
</li>
</ul>
</li>
<li>
<h3 id="use-remote-npm-packages">Use remote npm packages</h3>
<ul>
<li>
<p>This can be done via private repository or using github to host the module. While this works better than local packages, they new issue that came up is the huge difference in how the CRNA and CRA were set up to build. Basically React Native compiles everything with Babel, where Webpack in CRA excludes everything in node_modules. This will cause web build errors in all your modules, as Webpack will not be able to properly process anything ES6/JSX if Babel hasn’t processed it yet.</p>
</li>
<li>
<p>For our Web app, we hadn’t yet ejected CRA and I had no desire to do so just so I can spend a ton of time tweaking Webpack config to make this work. Other people have <a href="https://pickering.org/using-react-native-react-native-web-and-react-navigation-in-a-single-project-cfd4bcca16d0">had success</a> doing it this way, but tweaking config and ‘Monkey Patching’ individual modules seemed more work than it was worth.</p>
</li>
</ul>
</li>
<li>
<h3 id="use-npm-pack--npm-install">Use <code class="language-plaintext highlighter-rouge">npm pack</code> + <code class="language-plaintext highlighter-rouge">npm install</code></h3>
<ul>
<li>The answer to the above SO post, this solution did work but the actual pack operation takes 20+ seconds, even with a small number of package, which does not allow for acceptable iteration speed when trying to work on modules. It also still required additional dev/build scripts to keep the modules in sync.</li>
</ul>
</li>
</ul>
<h2 id="try-it-out">Try it out</h2>
<p>If you want to see an example of this up and running, I’ve created a <a href="https://github.com/jehartzog/rn-web-shared-modules">github project</a> which uses CRA and CRNA and sets up a few basic examples of using shared components between the two.</p>J. Eric HartzogEdit (11/1/2018): After I finished this work, I found wix/wml which solves the file syncing issue with minimal pain. While the approach in this post is still generally applicable, I’d recommend starting with wml rather than using custom scripts to sync the module files.AWS vs Galaxy for Meteor Hosting2018-02-15T17:00:00+00:002018-02-15T17:00:00+00:00https://www.erichartzog.com/blog/aws-vs-galaxy-for-meteor-hosting<p>After 6 months of hosting on Galaxy with multiple <a href="/blog/meteor-galaxy-not-production-ready">host caused outages</a>, I was willing to put in the effort of transitioning to AWS. Nothing is more stressful for a DevOps provider than to have his apps drop offline for no good reason, and not have any alerts to warn of issues before Pingdom tells you that your website is timing out.</p>
<h2 id="time-for-a-change">Time for a change</h2>
<p>After deciding I couldn’t stick around and wait for Galaxy to fix itself, I started looking for alternatives. I was intrigued by <a href="https://www.nodechef.com/">NodeChef</a>, but after the negative experience with Galaxy I was wary of trying a ‘specialty’ hosting provider over a battle-tested solution. I had experience with AWS in the past for hosting LAMP stack application, so I started the process of using <a href="http://meteor-up.com/">MUP</a> to get my app up and running on AWS EC2 instances.</p>
<h2 id="performance-performance-performance">Performance, Performance, Performance!!!</h2>
<p>I know that the Galaxy group invested a lot of effort into building a scalable Meteor hosting solution, but <strong>it is way too slow</strong>. I was surprised when I first fired up APM on Galaxy and saw 400ms pub/sub response times, which are acceptable but not great. When you add that delay to the typical Meteor delay of downloading/processing the initial bundle, you’re talking about a good bit of time before a useful render.</p>
<p>After I while I figured this was just a Meteor thing and I should accept that. <strong>I was wrong</strong>, and AWS showed me how performant Meteor can truly be.</p>
<p>I had some Galaxy performance data from my <a href="/blog/scaling-with-meteor">scaling with Meteor</a> post, and I was able to compare it to my new AWS setup under a nearly identical customer load.</p>
<h3 id="app-on-galaxy">App on Galaxy</h3>
<p><img src="/images/skoolers-peak-post-oplog.png" alt="skoolers-peak-galaxy" title="Skoolers Peak on Galaxy" /></p>
<ul>
<li>3x compact containers (0.5 ECU, 512 MB each).</li>
<li>~$120/month.</li>
</ul>
<h3 id="app-on-aws">App on AWS</h3>
<p><img src="/images/skoolers-peak-aws.png" alt="skoolers-peak-aws" title="Skoolers Peak on AWS" /></p>
<ul>
<li>2x t2.micro containers (1 vCPU, 1 GB each).</li>
<li>~$50/month (2x t2.micro + network load balancer).</li>
</ul>
<h3 id="how-aws-dominated-galaxy">How AWS dominated Galaxy</h3>
<ul>
<li>Speed.
<ul>
<li>I’m getting 10x faster response times on AWS than on Galaxy while the webservers are at similar CPU/memory conditions.</li>
</ul>
</li>
<li>Simplicity.
<ul>
<li>I don’t need to auto-scale. Because of how Galaxy is built, Compact containers are hard-capped at ~15% CPU as shown in APM. This is a serious problem if something eats up CPU usage suddenly as delayed methods/publications cause a cascading effect due to how Meteor waits to ensure operations are completed in order.</li>
<li>On the other hand, AWS allows t2 instances to burst above baseline for a sustained amount of time, allowing the app to stay responsive under unexpected loads.</li>
</ul>
</li>
<li>Metrics.
<ul>
<li>Galaxy has zero reporting or alerting capabilities. Enough said there.</li>
</ul>
</li>
<li>Stability.
<ul>
<li>While I’m less than 2 months into AWS, already I’ve had a much more stable hosting experience. I don’t get the constant maintenance restarts that sometimes cause latency and other unexpected issues that came with Galaxy hosting.</li>
</ul>
</li>
<li>Cost.
<ul>
<li>I’m paying less than 50% for AWS than what I did for Galaxy.</li>
<li>AWS does have a number of small charges that are hard to estimate, but overall it’s still far cheaper than Galaxy.</li>
</ul>
</li>
</ul>
<h2 id="what-i-miss-about-galaxy">What I miss about Galaxy</h2>
<p>Overall I estimate that I spent roughly 40 hours to patch/fix/build things that Galaxy provided for me. While the paypack period for this work was longer than it’s probably worth, I now have a piece of mind allowing me to relax knowing AWS has my app up and running strong.</p>
<p>Here is a list of all the major features and compromises I had to deal with when leaving Galaxy:</p>
<h3 id="apm">APM</h3>
<p>There are a number of posts online about how to deploy your own APM server. NodeChef even provides a standalone <a href="https://www.nodechef.com/docs/node/meteor-apm">APM service</a>. I tried that out and was generally satisfied, but it didn’t offer any alerts and only kept ~3 days of data at $10/month/server. I had some time to try out other solutions and ended up going with <a href="https://github.com/lmachens/meteor-apm-server">Imachens/meteor-apm-server</a> self-hosted on AWS. It was a bit painful to get up and running, but the end solution gave me everything I wanted at reasonable hosting costs.</p>
<h3 id="seo">SEO</h3>
<p>This one kinda hurt for a bit. Right after switching from Galaxy, pages started to drop out of the Google index. I then created a <a href="https://prerender.io">prerender.io</a> token and integrated <a href="https://github.com/prerender/prerender-node">prerender-node</a>, which was a quick integration, but I’m still experiencing difficulty getting all my pages back in the Google index.</p>
<p>In January Google released their new search console which gave me better information on why my pages were not being included in the index. Google reported ‘Submitted URL not selected as canonical’, but I’m still trying to figure out why Google is rejected the pages I provide in sitemap as non-canonical. More to do there.</p>
<h3 id="zero-downtime-updates">Zero-downtime Updates</h3>
<p>The one super nice thing about Meteor was pushing and updating your app. You just pushed your app, and it handled updating the containers one by one with rolling restarts.</p>
<p>With my current setup I have two webservers running behind a load balancer. When I push an update, MUP updates server 1, then server 2. In order to prevent crashing from excessive RAM usage, <a href="https://github.com/zodern/meteor-up/issues/827#issuecomment-366001159">MUP</a> stops the running container, starts a new one, runs npm install and starts up Meteor. This takes between 20-40 seconds, during which users connected to that server will see errors. I could write a script to take those servers out of the load balancer and add them in after MUP is finished, but this very brief downtime during updates is acceptable for now.</p>
<h3 id="timezones">Timezones</h3>
<p>This one was unexpectedly a pain and kept messing up my scheduled cron jobs. Galaxy reads the environment variable <code class="language-plaintext highlighter-rouge">TZ</code> and sets the timezone appropriately. MUP doesn’t address how to adjust server timezones, so it’s up to you to do. In the end I ended up creating a <code class="language-plaintext highlighter-rouge">post.setup</code> hook in my mup config:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">...</span>
<span class="na">meteor</span><span class="p">:</span> <span class="p">{</span>
<span class="p">...</span>
<span class="na">hooks</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">'</span><span class="s1">post.setup</span><span class="dl">'</span><span class="p">:</span> <span class="p">{</span>
<span class="na">remoteCommand</span><span class="p">:</span> <span class="dl">'</span><span class="s1">sudo timedatectl set-timezone EST</span><span class="dl">'</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">};</span>
</code></pre></div></div>
<h3 id="meteor-app-unique-naming">Meteor App Unique Naming</h3>
<p>Another task that Galaxy performed for me that I wasn’t aware of is ensuring that my Meteor containers all had unique app names. When I first set up MUP with my two webservers, the apps were both named in the MUP config like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">...</span>
<span class="na">meteor</span><span class="p">:</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">webserver-production</span><span class="dl">'</span><span class="p">,</span>
<span class="p">...</span>
<span class="p">},</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Turns out that is a very bad idea, as Meteor works with MongoDB and the oplog in a way that depends on unique app names. I thought the server was running fine, but it turns out updates from one of my two webservers would show in the client UI but not be saved to the DB. No errors/warnings were thrown to alert me to any issues. I didn’t catch this error in initial testing, but clients noticed it quickly and I was able to resolve it in an hour by ensuring each webserver had a unique name (‘<code class="language-plaintext highlighter-rouge">webserver-production-one</code>’, ‘<code class="language-plaintext highlighter-rouge">webserver-production-two</code>’, etc…).</p>
<p>This wasn’t listed anywhere in the <a href="https://guide.meteor.com/deployment.html">Meteor documentation</a> or called out clearly in <a href="http://meteor-up.com/getting-started.html">MUP</a>. This was the single largest ‘gotcha’ that I experienced switching from Galaxy to AWS/MUP that negatively affected production operations. Luckily it was relatively minor and I was able to fix it quickly.</p>
<h2 id="final-thoughts">Final Thoughts</h2>
<p>I’ve held off posting my negative experience with Galaxy for a long time, because as a developer I’ve enjoyed working with Meteor and I intend on doing so again in the future. They’ve continue to push fantastic updates to Meteor and been focused on improving the platform while keeping in mind changes in the community as they go. But after putting in the time to create a Meteor app and transform it into a working production application, I feel strongly that Meteor is absolutely production ready, and Galaxy is not even close.</p>
<p>I’ve offered detailed feedback to the Galaxy support team and given all I can to try to help them improve what they offer, but while always polite they have always seem uninterested in major improvements.</p>
<p>While I’ve never spoken to anybody at AWS, it’s because I’ve never needed to; their solutions just work, are incredibly reliable, and if AWS goes down, so does most of the internet so your clients won’t really blame you. For my future projects I’ll definitely be biased towards using the stronger ‘general-purpose’ cloud providers (AWS, DO, GCP, Azure) rather than specialty providers who wrap up the service with their own opaque framework.</p>J. Eric HartzogAfter 6 months of hosting on Galaxy with multiple host caused outages, I was willing to put in the effort of transitioning to AWS. Nothing is more stressful for a DevOps provider than to have his apps drop offline for no good reason, and not have any alerts to warn of issues before Pingdom tells you that your website is timing out.Goodbye iOS, Hello Android2017-12-13T17:00:00+00:002017-12-13T17:00:00+00:00https://www.erichartzog.com/blog/goodbye-ios-hello-android<h2 id="a-possible-apple-fanboy">A possible Apple Fanboy</h2>
<p>While I don’t think of myself as an <a href="https://www.urbandictionary.com/define.php?term=Apple%20Fanboy">Apple Fanboy</a>, I can’t deny that at a minimum, I come very close.</p>
<p>I’ve lined up at Apple stores for a few launches of new products, starting with the iPhone 3G. Since 2008 I purchased at least 10 iPhone/iPod Touches, multiple MacBook’s, an iMac, and a few iPad’s here and there. I’ve worn an Apple watch for over a year now. A lot of those were for <a href="/projects/stickwars">StickWars</a> development, and I tended to quickly sell the extra/older models. Still though, I had my arms full of Apple products over the past decade.</p>
<h2 id="it-was-all-justified-perhaps">It was all justified, perhaps?</h2>
<p>Back in 2008 when I was shopping for a new higher-end laptop, the newfangled <a href="http://apple-history.com/mb_late_08">Unibody MacBook</a> was an easy choice as the price fell right around an equally configured Dell, and the Apple had a better build quality. I had never used Mac OS before, but I’d heard good things and figured it would be fine. Turns out it was better than fine. I enjoyed the OS far more than any windows machine I’d ever used, and the MacBook Pro I ended up upgrading to would last me over 5 years of development use.</p>
<p>Switching to the iPhone 3G was an easy choice, and was easily the top choice of phone for a number of years. I later upgraded to an iPhone 4, and when Apple finally made a larger screen sized on the iPhone 6 Plus I picked that baby up and used it for a number of years.</p>
<h2 id="apples-stalled-innovation">Apple’s stalled innovation</h2>
<p>The iPhone 6 Plus came out in Fall 2014, and since then there have been no major changes to software or hardware. The notification system has stayed the same, the configuration menus have been shaken up a bit but generally unchanged, and even as the processing capability grew, the ability to multitask apps has stayed the same.</p>
<p>That is not to say they haven’t done anything. They <a href="/blog/stickwars-ninth-life">kicked out 32-bit apps</a>, added <a href="https://www.igeeksblog.com/how-to-add-screen-effects-to-imessages-in-ios-10/">fun effects to iMessage</a>, and tried to provide some of the features every Android phone already had by adding <a href="https://www.iphonefaq.org/archives/975459">breadcrumbs</a> to speed app switching. All of these were free enhancements, but nothing that really changed the interface of the phone, instead stacking more features around the existing UI.</p>
<h2 id="ios-11-the-killer-blow">iOS 11, the killer blow</h2>
<p>After 3 years I was still quite happy with my iPhone 6 Plus, given the fact that the later phones didn’t really add anything new besides a newer processor, camera, and screen. This was all true until Apple released the hell that is iOS 11 upon the world.</p>
<p>As a UI developer, I take notice of certain things and can qualify them in a specific way, more so than saying “my phone seems slow”. Before iOS 11, my phone was perfectly acceptable, not super fast but rarely lagging enough for me to notice. After iOS 11, almost every single UI related event (keyboard typing, touch, menu navigation) had an additional 300-2000ms worth of latency. And if 2 seconds of latency for a keypress seems insane, yes I thought so to. <a href="https://discussions.apple.com/message/32351004#message32351004">This post</a> sums up the issue rather well– basically, every single use of the phone is about half a second slower than it was on iOS 10.</p>
<p>I also had battery drain issues– it went from 36 hours of normal usage to 8 hours, but I fixed this by resetting all settings and spending the time re-setting up my iPhone. Yay the hours spent on this.</p>
<p>I wasn’t the only one to <a href="https://www.tomsguide.com/us/iphone-ios-11.0.1-update-bugs,news-25913.html">notice this</a>, and many others with older devices reporting iOS 11 making their phones nearly unusable.</p>
<p>I understand that an update can get pushed with some problems, so I gave it time, and then some more. After 2 months I realized that no iOS 11 update will ever come out to fix the UI latency issues introduced, and while it will never be publicly admitted by Apple, they likely designed this update as a method of forced retirement of older devices.</p>
<h2 id="but-what-about-the-iphone-x">But what about the iPhone X?</h2>
<p>I wanted to like the first OLED phone that Apple made, the first to try to improve the actual hardware design in over 3 years. But then I saw the release and was astounded. This is 100% a matter of personal taste, but to take a spectacular OLED screen and cut a notch out of it for the purposes of marketing should be a crime. With phones becoming more bezel-free these days, Apple played the game that people who could afford to spend a grand on a phone would want the people standing around them to know they spent a grand on their phone, and a distinctive cut-out from the screen is a great way to do that. But that’s not the reason I want to buy a nice phone, no thanks.</p>
<h2 id="a-time-for-change">A time for change</h2>
<p>I remember researching Android phones years ago and hearing about so many of the basic things which just “don’t work”. Years later, the bulk of those have been smoothed out and Android is a much more solid system than it used to be. With Samsung, Google, and other manufacturers offering great phones at a discount to the iPhone, there are a number of good choices to pick from.</p>
<p>While the display and overall feel of the Samsung Note 8 blew me away, I decided on the Google Pixel 2 XL. The deciding factor was the common theme among reviewers who said something like “the phone is stunning, but the Samsung branded software you can’t uninstall kinda sucks”. No thanks. I remember having a giant folder of iOS apps that I couldn’t remove from my iPhone until a later update, and I didn’t want to go back to that. After all, with the bezel-less phones looking more and more alike these days (unless you cut out a portion of the display for an ugly ass notch), the thing which makes or breaks the phone is the software. And Google does software very well.</p>
<p>I’ve been rocking the Pixel 2 XL for about a week now, and it’s an overwhelmingly better experience. There is not a single software or hardware feature where I find myself missing the way it used to work on my iPhone, and I’ve already gotten so used to some of the better features on Android that I know I won’t be going back to iOS for a long time. The mind blowing improvements come with the combination of the better notification system and the Smart Lock system which keeps your phone unlocked when at your home and/or car. Now I can actually handle 90% of the notifications from the lock screen itself, without having to fingerprint, load an app, and navigate around in order to clear the notification. If you are a heavy Gmail user, you will absolutely love how the notifications work. If you are one of those people who are still happy using Outlook, you may not really notice a big difference.</p>
<p>So as I said at the start, goodbye iOS, hello Android. I’m here to stay, and will recommend to any who ask that they do the same.</p>J. Eric HartzogA possible Apple FanboyMeteor Galaxy is not Production Ready2017-10-04T17:00:00+00:002017-10-04T17:00:00+00:00https://www.erichartzog.com/blog/meteor-galaxy-not-production-ready<p>Despite the <a href="/blog/meteor-galaxy-autoscale">10x cost premium</a>, I wanted to enjoy my hosting experience with Meteor Galaxy, I really did. I knew my <a href="/projects/Skoolers">app</a> was a perfect fit for them, with something around 1000 daily users max, I could use the benefits of their managed hosting without worrying about staggering costs as I tried to scale</p>
<p>I started off with a small audience from May to Aug 2017, giving me time to optimize my app and remove all <a href="/blog/scaling-with-meteor">bottlenecks caused by app code</a>. The full audience hit the app in September, allowing me to see how it did against 1000+ daily active users, and I was glad to see the app handling everything as well as could be expected, managing to support that load with about 1 CPU and 1 GB of memory.</p>
<h3 id="meteor-galaxy-support-was-responsive-polite-and-professional">Meteor Galaxy support was responsive, polite, and professional</h3>
<p>Before getting into all the negatives, I do want to share that the Meteor Galaxy support service was generally very professional, responding within 6 hours for all initial tickets, and letting me know when more time was needed for a more detailed response.</p>
<h2 id="six-months-of-hosting-on-meteor-galaxy">Six months of hosting on Meteor Galaxy</h2>
<p>Unfortunately these past six months have also exposed my application to three different Galaxy caused outages in my app, causing a serious lack of confidence in the seriousness as a production hosting provider. Through all of the outage, the response was consistently unacceptable as a professional hosting company for these basic reasons:</p>
<ol>
<li>
<p>The <a href="http://status.meteor.com/">Galaxy Status</a> never changed from green, nor sent an alert to subscribers.</p>
</li>
<li>
<p>In some cases, Galaxy caused an outage or period of extreme latency that was never reported on forums or their status page, merely acknowledged privately back to me after continued questions through their support system.</p>
</li>
<li>
<p>The response time to acknowledge these smaller scale outages took <strong>13 days</strong> to provide a confirmation of a Galaxy-caused outage.</p>
</li>
</ol>
<p>For all the times listed, I am referring to US East Coast.</p>
<h3 id="outage-1---27-june-6am---10am">Outage #1 - 27 June, 6am - 10am</h3>
<p>This was a huge one, approx 4 hours of outage until MDG finally responded with a solution.</p>
<p>Pingdom sent an alert that the site was unreachable, and after I woke I saw <a href="https://forums.meteor.com/t/contain-health-checks-fail-on-galaxy-deploy/37403/29?u=avariodev">others were having the same issue</a>. Multiple hours after the outage began, Galaxy status still indicated all green with the only posts being from affected customers. Finally a forum post went up on the above thread, but no update went out via the official Galaxy Status system.</p>
<p>They ended up writing a detailed <a href="http://status.meteor.com/incidents/tf630kbt1x2n">after action report</a> which was the right thing to do in this case. They listed the steps they took to prevent this from recurrence, so while overall their response in-situ was extremely delayed, they eventually did the right thing.</p>
<p>I would have hoped they would address lack of any updates on the Galaxy Status page until the day after, but thought that may be part of their fixes to their maintenance schedule.</p>
<h3 id="outage-2---5-september-3pm---330pm">Outage #2 - 5 September, 3pm - 3:30pm</h3>
<p>This one was a shorter time duration but still very uncomfortable. They ran a maintenance update at 3pm which caused 30 minutes of disruption to all my running containers. Latency for methods, pubs, and connections rose to 10+ seconds, and in many cases the containers were completely unreachable.</p>
<p>APM graphs coverage the outage
<img src="/images/galaxy-outage-5-sept.png" alt="galaxy outage 5 sept" title="Galaxy Outage 5 Sept" /></p>
<p>This being prime-time for my app, clients noticed right away and contacted me, and I started desperately spinning up additional containers and up-sizing them while trying to understand what was going on. After about 30 minutes latency fell back to normal and I was left wondering what happened, worried my app code was flawed in some way I didn’t understand.</p>
<p>After six hours, I got a response on the ticket I opened saying that a normal container maintennace restart occured during that time. I pointed out the service logs were blank for the entire time period in question. I got a response 24 hours later from another rep saying they were taking this to their engineering team to take a look and would get back to me.</p>
<p><strong>13 days later</strong>, on 19 September, I received the following response:</p>
<blockquote>
<p>Thanks for your patience here; I received a full debrief from our engineering team about this. In short, the increased latency you observed was our fault, and we apologize for this.</p>
<p>What happened was that Galaxy changed the EC2 instance type we use for hosting application containers. Because of this change, Galaxy relaunched every running container, for every user.</p>
<p>The mass container relaunch temporarily overloaded the EC2 instances that host application containers, resulting in the increased latency you observed.</p>
<p>In hindsight, it would be have been better to complete this work during a scheduled maintenance window; testing in our staging environment, as well as smaller regions, made us believe the impact would be negligible - but we were mistaken, and the latency you observed was the result. Again, we apologize for the inconvenience. I hope this explains what happened, but please let me know if there’s anything else I can answer.</p>
</blockquote>
<p>While I was very satisfied with receiving a response, I was bothered by a number of ways in which this situation was handled.</p>
<ul>
<li>
<p>No public acknolwedgement of this outage. Unless this truly only affected one customer (unlikely), it should have been posted to their status page along with an announcement.</p>
</li>
<li>
<p>14 days from opening of ticket to confirmation of outage. I get them I’m a small customer, but I’m also a small customer with AWS, and at least when their customers get unexplained multi-second latency, they will be sure to pay attention to it.</p>
</li>
</ul>
<h3 id="outage-3---3-october-622pm---635pm">Outage #3 - 3 October, 6:22pm - 6:35pm</h3>
<p>With this outage being recent, I don’t yet have confirmation that this was actually a Galaxy caused outage, so it’s just my suspicion for now.</p>
<p>At 6:21pm, Meteor Galaxy replaced my running container as part of a maintenance event, this is common and happens without incident multiple times per day. However, the container that started up to replace it (<code class="language-plaintext highlighter-rouge">qzffh</code>) was deeply flawed, CPU pegged at max right away. Unfortunately it was responsive enough to pass the health checks, so Galaxy killed the old container and the new one was left serving web traffic.</p>
<p>Latency quickly rose to 5+ seconds for all subscriptions and method calls, and while Pingdom alerts me on a complete outage, something like the above latency requires better monitoring and alerts to let me know that something is going wrong.</p>
<p>After 8 minutes my client called to ask about general slowness. In response I spun up new containers and killed the flawed one, which restored the app to normal.</p>
<p>Galaxy service logs for the time in question</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>m28br 2017-10-03 18:15:44-04:00 The container is being stopped to scale down the number of running containers.
m28br 2017-10-03 18:16:17-04:00 Application exited with signal: killed
qzffh 2017-10-03 18:21:18-04:00 Application process starting, version 189
t24kk 2017-10-03 18:21:56-04:00 The container is being stopped because Galaxy is replacing the machine it's running on.
t24kk 2017-10-03 18:22:29-04:00 Application exited with signal: killed
0w5em 2017-10-03 18:32:10-04:00 Application process starting, version 189
3qdq2 2017-10-03 18:32:22-04:00 Application process starting, version 189
1fcej 2017-10-03 18:34:56-04:00 Application process starting, version 189
qzffh 2017-10-03 18:36:30-04:00 The container is being stopped due to a user kill request.
qzffh 2017-10-03 18:37:02-04:00 Application exited with signal: killed
</code></pre></div></div>
<p>APM graphs
<img src="/images/galaxy-outage-3-oct.png" alt="galaxy outage 2 oct" title="Galaxy Outage 3 Oct" /></p>
<p>While I freely admit it’s possible my app code somehow caused this flawed container, I consider that unlikely as I’ve never seen this type of behavior before, and my past experience with Galaxy maintenance restarts having unintended side effects. I’ve moved all cron and batch processing jobs to a cheaper AWS EC2 instance, so the only thing the webserver should be doing is client connections, delivering assets, pub/sub of collections and methods.</p>
<p>Another possible issue is database performance, but all stats there were normal, no excessive page faults or query latency.</p>
<p>The deeper issue is the lack of any sort of alert system that can let me know when average method and pub/sub response time rises above 500ms, a clear indicator of something going wrong. It’s trivial to send this alert to my high priority slack channel, which can send me a push notification and let me try to fix things before my client has to give me a ring.</p>
<h2 id="shaken-confidence">Shaken Confidence</h2>
<p>After this six-month run with Meteor Galaxy, it looks like it’s unusable as a long-term provider for our app. While I’m a huge fan of the deep Meteor integration and managed app deployments, the lack of sustained low-latency availability and performance alert notifications mean I can never be sure if the application is really up and running as it should be.</p>J. Eric HartzogDespite the 10x cost premium, I wanted to enjoy my hosting experience with Meteor Galaxy, I really did. I knew my app was a perfect fit for them, with something around 1000 daily users max, I could use the benefits of their managed hosting without worrying about staggering costs as I tried to scaleScaling with Meteor2017-10-03T17:00:00+00:002017-10-03T17:00:00+00:00https://www.erichartzog.com/blog/scaling-with-meteor<p>After months monitoring and tuning my Meteor app for <a href="/projects/skoolers">Skoolers Tutoring</a>, I’ve finally hit peak traffic expected for the app at over 1000 daily active users (DAL). Since there is a limited number of students enrolled at any one time, I can plan on being prepared for this level of traffic, and any expansion plans will probably come with sub-domains running completely independent versions of the app.</p>
<p>With the countless forum posts online asking <a href="https://forums.meteor.com/t/how-many-simultaneous-users-does-the-biggest-current-meteor-app-support/10421/7">how does Meteor scale?</a>, I thought I share the results I saw in a real-world app.</p>
<h1 id="app-background">App background</h1>
<p>Many answers to the often asked ‘how many concurrent users can my server support?’ come back with ‘it depends’, with good reason. I will try to break down my app as an example along with results of real-world testing to help present a baseline of what to expect.</p>
<p>If you don’t feel like reading all the <a href="/blog/skoolers-app-details">details about the app</a>, then just read this summary. I consider my app to be on the heavy side of what a Meteor app can be expected to provide. It uses a large number of publications, some of which can transfer hundreds of KB of data, for each student, and has a number of features which result in constant Meteor method calls.</p>
<p>While I rely entirely on third party services (AWS S3, Vimeo, Wistia) to deliver all the heavy static assets (PDF’s, hosted videos), but the Meteor server is still heavily used to provide real-time data to the clients and record nearly all client interactions via method calls.</p>
<h1 id="does-it-scale">Does it scale?</h1>
<p>Why yes, it sure does. I’m currently supporting <strong>over 1000 DAL with 1-3 <a href="https://www.meteor.com/pricing">Galaxy compact containers</a></strong>, with container numbers managed automatically by my <a href="/blog/meteor-galaxy-autoscale">galaxy-autoscale</a> package.</p>
<p>While I had to do a lot of performance tuning of inefficient pubs/methods as I went, there were three major steps which dramatically improved the app performance, allowing me to serve more users with significantly less resources.</p>
<h2 id="initial-public-launch">Initial public launch</h2>
<p>I was fortunate that I initially launched the app against less than 1/10th of my actual peak expected audience, which gave me months to observe the app in real-world use and ensure it was ready for peak load. I released the first version at the start of the summer semester, with very few students enrolled in courses.</p>
<p>If I didn’t have this luxury of gradually increasing usage, I would have needed to do more load testing, which I think is a tricky thing to do with Meteor given the async nature of it’s publication loading and method calls.</p>
<h2 id="performance-boost-1---mongodb-indexes">Performance boost #1 - MongoDB Indexes</h2>
<p>This one is obvious, but it’s one of those tunings you either need experience to know how to do right, or you wait until your app is getting some usage and take a look at the ‘slow queries’ tools provided by your MongoDB provider and add the indexes it recommends.</p>
<p>I started off without adding indexes besides those included by default, and the graphs showed the database working pretty hard when it shouldn’t have been. This graph covers the period start at first public launch to when I started adding in indexes, and it’s easy to see the transition.</p>
<p><img src="/images/skoolers-indexes-bad.png" alt="skoolers indexes bad" title="Skoolers MongoDB Indexes Bad" /></p>
<p>That peak in the middle is the database scanning through over 6,000 documents per second on average, at a time when my DAL was just over 200. That purple line that looks like a zero is actually the number of documents found via indexes, it’s just too low to be clearly visible. Definitely some easy optimization to do here.</p>
<p>Again this step was easier than it sounds, just go through the ‘slow queries’ tool in your MongoDB provider, evaluate which indexes it recommends and add most of them. The only ones I didn’t end were queries that were inefficient due to third-party packages I was using where I couldn’t easily edit them to take advantage of the indexes.</p>
<p>The easiest way to evaluate the effectiveness of this is to add the indexes, observe the difference over 24 hours to verify the improvement you were looking for, and keep going until you are satisfied with the coverage of your indexes. This is what the same graph looks like now.</p>
<p><img src="/images/skoolers-indexes-good.png" alt="skoolers indexes good" title="Skoolers MongoDB Indexes Good" /></p>
<h2 id="performance-boost-2---oplog-tailing">Performance boost #2 - Oplog Tailing</h2>
<p>After first setting up my production/staging environments and doing some limited tested, I tested enabling <a href="https://blog.meteor.com/tuning-meteor-mongo-livedata-for-scalability-13fe9deb8908">oplog tailing</a> and saw that it led to noticeably higher idle CPU usage. This was before I had done any significant optimization work, and I’m not sure why I saw these results, but either way I decided at that time to disable oplog tailing.</p>
<p>This turned out to be the wrong choice, as I saw when I later re-enabled oplog tailing much later. The good part about this test it I was able to collect some real data on exactly how much better Meteor performs with oplog tailing enabled.</p>
<h3 id="before-oplog-tailing">Before oplog tailing</h3>
<p><img src="/images/skoolers-peak-pre-oplog.png" alt="skoolers-peak-pre-oplog" title="Skoolers Peak Log Pre-Oplog Tailing" />
Note about CPU usage graph: the % doesn’t take into account container size. With compact size, the CPU maxes out at 15%.</p>
<p>Raw stats:</p>
<ul>
<li>6 compact containers (0.5 ECU, 512 MB each)</li>
<li>~200 unique users, with ~350 total connections</li>
<li>Average CPU at 40%, average memory at 235 MB (45%)</li>
</ul>
<p>Calculated stats:</p>
<ul>
<li>1.4 total ECU used, 1.4 GB memory used</li>
<li>0.7 ECU, 700 MB memory used for every 100 unique users</li>
</ul>
<p>Since I always wanted to have excess resources available, I can look at the numbers to come up with some decent estimates on how many resources I’d want to allocate based on concurrent user load.</p>
<hr />
<p>Pre oplog tailing resources required <strong>for every 100 unique users</strong>:</p>
<ul>
<li><strong>1.5 ECU</strong></li>
<li><strong>1.5 GB of memory</strong></li>
</ul>
<hr />
<h3 id="after-oplog-tailing">After oplog tailing</h3>
<p>These were taken the day after the above data set, with no other major changes made the app or database. Luckily user traffic was almost identical, allowing for a great comparison.</p>
<p><img src="/images/skoolers-peak-post-oplog.png" alt="skoolers-peak-post-oplog" title="Skoolers Peak Log Post-Oplog Tailing" /></p>
<p>Numbers:</p>
<p>Raw stats:</p>
<ul>
<li>3 compact containers (0.5 ECU, 512 MB each)</li>
<li>~200 unique users, with ~340 total connections</li>
<li>Average CPU at 25%, memory at 273 MB (53%)</li>
</ul>
<p>Calculated stats:</p>
<ul>
<li>0.38 total ECU used, 820 MB memory used</li>
<li>0.19 ECU, 410 MB memory used for every 100 unique users</li>
</ul>
<p>Looking at the steady state load, I adjust the margins I’m comfortable with and come out with new resource requirements.</p>
<hr />
<p>Post oplog tailing resources required <strong>for every 100 unique users</strong>:</p>
<ul>
<li><strong>0.5 ECU</strong></li>
<li><strong>512 MB of memory</strong></li>
</ul>
<hr />
<p>I’m going closer to the memory limit here, but in general memory usage is very stable at this point.</p>
<h3 id="effects-on-mongodb">Effects on MongoDB</h3>
<p>While my database provider never slowed down my app, it was clear that the pre oplog tailing was putting an enormous, and unnecessary, burden on the database. Looking at a few of the MongoDB graphs covering the same period before/after the transition to oplog tailing, you can see the dramatic difference in the amount of documents requested by Meteor.</p>
<p><img src="/images/skoolers-db-oplog.png" alt="skoolers-db-oplog" title="Skoolers MongoDB Scan Before/After Oplog Tailing" /></p>
<p>Before oplog tailing was enabled, my cluster of Meteor containers were requesting and processing <strong>1000 documents per second and 18 Mbps of data</strong> from my MongoDB cluster to support only 200 unique users.</p>
<p>After oplog tailing, those figures dropped to <strong>80 documents per second and 400 Kbps</strong> of outbound data. That’s a lot less unnecessary work processing all that duplicate data</p>
<h2 id="performance-boost-3---offload-cron-jobs">Performance Boost #3 - Offload cron jobs</h2>
<p>I initially used <a href="https://github.com/percolatestudio/meteor-synced-cron">synced-cron</a> to manage a number of maintenance scripts. After traffic picked up and my containers were pushed closer to their limits, the scripts would sometimes bump up memory usage of a container enough to kill itself.</p>
<p>Rather than run these scripts on client facing servers, I shifted this to a cheap AWS EC2 server, deployed using <a href="https://www.npmjs.com/package/mup">MUP</a>, and disabled synced-cron on the client facing servers. This led to extremely smooth CPU and memory graphs, allowing me to run closer to the limits without getting occasional memory kills or CPU pegs.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>The app now runs very smoothly on 1-3 compact containers, using my <a href="/blog/meteor-galaxy-autoscale">galaxy-autoscale</a> package to bump up or down container numbers with shifting load. While it took a bit of work, Meteor ended up scaling to exactly the level I needed within my low budget of $100/month.</p>
<p>Given the ease the entire development process was from start to finish, and the efficiency of the current application, I plan on continuing to use Meteor for future client projects :).</p>J. Eric HartzogAfter months monitoring and tuning my Meteor app for Skoolers Tutoring, I’ve finally hit peak traffic expected for the app at over 1000 daily active users (DAL). Since there is a limited number of students enrolled at any one time, I can plan on being prepared for this level of traffic, and any expansion plans will probably come with sub-domains running completely independent versions of the app.Skoolers App Details2017-10-02T17:00:00+00:002017-10-02T17:00:00+00:00https://www.erichartzog.com/blog/skoolers-app-details<p>This is a portion of my <a href="/blog/scaling-with-meteor">Scaling with Meteor</a> post broken out for readability.</p>
<p>For other developers helping to plan out whether Meteor can support their apps, I lay out the generic requirements of the <a href="/projects/skoolers">Skoolers Tutoring</a> app that I created, along with some hosting details.</p>
<h2 id="meteor-version-and-front-end">Meteor version and front-end</h2>
<p>My app was originally started with Meteor 1.3 and has been updated to keep in sync with the latest released version. I use React as the front end, and do not perform any server-side rendering.</p>
<h2 id="hosting-provider">Hosting provider</h2>
<p>I used <a href="https://www.meteor.com/hosting">Meteor Galaxy</a> for my webserver and a <a href="https://mlab.com/">mLab</a> shared MongoDB cluster. Galaxy was very easy to get set up and running, and I would recommend for anyone starting out with their first Meteor product but I had some issues over a few months of using them. mLab was awesome, their support was fantastic the single time I ever needed it, and the price was just right.</p>
<h2 id="pubs">Pubs</h2>
<p>With about 10 publications to every student depending on their current page, all of which are important to be real-time with low-latency. While I carefully limit the number of documents provided in these publications, since they are user-provided posts/questions/etc, the overall data transferred to each client can be over 100 KB. A non-exhaustive list of some of the data includes:</p>
<ul>
<li>All the current course offerings for their school</li>
<li>A list of all the courses each student is enrolled in</li>
<li>A list of the paginated wall posts for a course they are looking at</li>
<li>A list of all the videos available for a course</li>
<li>A list of all the scheduled events for a course</li>
<li>A list of all the files available for download for a course</li>
<li>A list of all the quizzes available for a course</li>
<li>A list of all the questions available when taking a quiz</li>
</ul>
<h2 id="methods">Methods</h2>
<p>The app also has a number of Meteor method calls which are called very often to keep track of user progress. At peak load I end up getting a method call which results in some sort of database write every ~2 seconds.</p>
<h2 id="bundle-size">Bundle size</h2>
<p>When Meteor 1.5 released I used their new dynamic imports feature along with React Loadable to <a href="/blog/code-splitting-with-meteor-dynamic-imports-and-react-loadable">trim the bundle size</a>, resulting in a gzipped main bundle size of ~470KB. Not as trim as it could be, but considering my traffic is primarily returning clients and I don’t run constant updates to the site, the time investment to further trim the bundle wasn’t worth it.</p>
<h2 id="traffic-schedule">Traffic schedule</h2>
<p>All my traffic is student based at local universities, all in a single time zone and with regular schedules. The first traffic starts up around 8am, rising to a plateau that lasts from 12pm to 12am, with traffic dropping to zero around 3am.</p>
<p>The traffic is also strongly correlated with upcoming exams, which can double the total DAL (daily active users) from one day to the next. I don’t desire to follow this exam schedule closely, and wanted to build a system that can automatically adjust for this large variance in traffic.</p>
<h2 id="interactive-wall">Interactive wall</h2>
<p>This is one of the data-heavy features we use, receiving hundreds of posts per day which are then published to the entire audience. It is features like this where Meteor really shines, allowing real-time, efficient updates.</p>
<p>The posts themselves are entered using a <a href="https://www.froala.com/">Froala</a> RTE, and saved as HTML. The server also handles things like providing temporary S3 tokens for users to post images/files/videos to S3 and automatically link to them inside the RTE.</p>
<p><img src="/images/skoolers-wall.png" alt="skoolers wall" title="Skoolers Wall" /></p>
<h2 id="practice-quizzes">Practice quizzes</h2>
<p>This allows the tutors to create tailored quizzes using multiple/single choice or fill-in-the-blank questions. Students can then practice these quizzes and review their results. As a student is taking a quiz, each answer they provide as type type/click is submitted via Meteor method to ensure their progress is saved.</p>
<p>These quizzes turned out to be much more popular than anticipated, but Meteor handled the unexpected load just fine.</p>
<h2 id="enrollments">Enrollments</h2>
<p>As a paid product, the app is responsible for managing which students have been enrolled in specific products, and providing the correct publication data and routing.</p>
<h2 id="logs">Logs</h2>
<p>I care about tracking how users are interactive with various features on the site, so I want to keep track of which products they are downloading, and what videos they are actually watching. These videos are the primary product of the site, so accurately recording this information is key to evaluating which videos are successful.</p>
<h1 id="more-about-skoolers-scaling">More about Skoolers scaling</h1>
<p>To read more about how this app performed under real production load, continue with the <a href="/blog/scaling-with-meteor">Scaling with Meteor</a> post.</p>J. Eric HartzogThis is a portion of my Scaling with Meteor post broken out for readability.Meteor Galaxy AutoScale2017-09-21T17:00:00+00:002017-09-21T17:00:00+00:00https://www.erichartzog.com/blog/meteor-galaxy-autoscale<p>I’ve released the public version of <a href="https://atmospherejs.com/avariodev/galaxy-autoscale">galaxy-autoscale</a> which helps to auto-scale and provide cost savings for hosted Meteor apps.</p>
<p>If you’re shopping around on how to host your upcoming Meteor app, the one host you’ll sure to know about is <a href="https://www.meteor.com/hosting">Meteor Galaxy</a>, created by <a href="https://www.meteor.io/">MDG</a>, the group that actually created Meteor.</p>
<h1 id="premium-cost-for-devops-time-savings">Premium cost for DevOps time savings</h1>
<p>At <a href="https://www.meteor.com/pricing">~$40/month</a>, you are paying almost 10x the cost of a <a href="https://aws.amazon.com/ec2/instance-types/t2/">AWS t2.nano</a> instance which have comparable performance. The T2 instances are actually superior in most cases as they offer <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html">burst-able performance</a> over a 24 hour period, which is incredibly helpful if most of your customers are in the same time zone.</p>
<h2 id="what-meteor-galaxy-provides">What Meteor Galaxy provides</h2>
<p>What are you getting in return for this premium price on server resources?</p>
<ol>
<li>
<p>Automated deployments of updated app versions with zero-downtime rolling restarts.</p>
</li>
<li>
<p>Integrated <a href="https://blog.meteor.com/introducing-galaxy-professional-built-in-apm-5e063839a4aa">APM</a>, the essential monitoring suite for Meteor apps.</p>
</li>
</ol>
<h2 id="what-meteor-galaxy-takes-away">What Meteor Galaxy takes away</h2>
<p>Here are the essential features you may assume would be part of the hosting service but you actually <strong>lose</strong> by using Meteor Galaxy.</p>
<ol>
<li>
<p>Any type of alert notification, including container health, maintenance restarts, CPU/memory usage.</p>
</li>
<li>
<p>Auto-scaling number/size of containers based on configurable rules.</p>
</li>
<li>
<p>Stable uptime of your app containers (more to follow on this, maybe…).</p>
</li>
</ol>
<h1 id="band-aid-where-its-easy">Band-aid where it’s easy</h1>
<p>During the process of seeing a <a href="/blog/scaling-with-meteor">full production load</a> for the first time, I initially had to add quite a few containers to handle 200+ concurrent users. At one point I was running 6x compact pro containers just to ensure everything was far away from limits. Left at that level, our hosting cost would have been a solid $198/month, not something sustainable for such a small customer base.</p>
<p>The first place to help with this is just took look at our traffic, all of our users are on the US east coast, so our entire traffic happens during the same time period each day. It also varies moderately from day to day, so we want some easy to way have the right number of containers running without having to keep a human eye on these graphs 24/7.</p>
<p><img src="/images/hourly-users-graph.png" alt="hourly users graph" title="Hourly Users Graph" /></p>
<p>After <a href="https://forums.meteor.com/t/galaxy-auto-scaling/22221">searching</a> and <a href="https://forums.meteor.com/t/auto-scaling-on-galaxy/33676">searching</a> and finding nothing but feature requests followed by +1 comments, I realized was going to have to roll my own solution.</p>
<p>It took me less than a days work to write a <a href="https://github.com/jehartzog/galaxy-phantomjs-autoscale">simple script</a> that runs on NodeJS that uses PhantomJS and WebdriverIO to scrape the needed figures from the Galaxy app dashboard, and click ‘up’ or ‘down’ to adjust the number of running containers. Definitely not my proudest bit of work, but it got the job done and didn’t suck up a lot of time.</p>
<p>I set it up to run on a t2.nano EC2 instance with crontab and voila, I had the number of containers smartly adjusting based on the number of connections coming to my app.</p>
<p><img src="/images/autoscale-graph.png" alt="autoscale graph" title="AutoScale Graph" /></p>
<h1 id="sharing-a-used-band-aid">Sharing a used band-aid?</h1>
<p>Since I saw so many people looking for a Galaxy auto-scaling solution, I’d thought I package my little script up in case anybody else wanted to use it.</p>
<p>I repackaged it as a <a href="https://atmospherejs.com/avariodev/galaxy-autoscale">Meteor package</a> and <a href="https://forums.meteor.com/t/meteor-galaxy-auto-scaling-package/39400">posted it</a> to the forums. The response was as expected, some people half-smiling at such a brittle and janky approach, but most acknowledged it at least was a solution given the current lack of other options.</p>
<p>While I’m hoping adding features make that package entirely pointless sometime soon, at least it’s a solution for now, or until it’s time to move away from Meteor Galaxy all together.</p>J. Eric HartzogI’ve released the public version of galaxy-autoscale which helps to auto-scale and provide cost savings for hosted Meteor apps.Google Font Loading in Meteor2017-08-27T17:00:00+00:002017-08-27T17:00:00+00:00https://www.erichartzog.com/blog/google-font-loading-in-meteor<p>In my search to load up some google fonts in Meteor, I found nothing in the official Meteor documentation about best practices, so search around community forums to figure out the best way to do to it right.</p>
<p>What I found was wrong information, bad information, and janky information. None of it terribly useful for my end solution, so I ended up going with the usual course when uncertainty strikes; trust google.</p>
<p>In this case, it was easy as they co-developed the <a href="https://github.com/typekit/webfontloader">webfontloader</a> npm package and provided some clean documentation on how to use it.</p>
<h2 id="posted-solutions-didnt-quite-pan-out">Posted solutions didn’t quite pan out</h2>
<p>I wanted a solution that didn’t involve a ton of <a href="https://forums.meteor.com/t/how-to-include-fonts/16702">monkey-patching Meteor</a> (the top google result) or using <code class="language-plaintext highlighter-rouge">@import</code> and <a href="https://www.webucator.com/blog/2016/10/load-web-fonts-asynchronously-avoid-render-blocking-css/">blocking all rendering</a> while waiting for the fonts to load.</p>
<p>The best post I found was <a href="https://forums.meteor.com/t/adding-google-fonts/1095/3">this</a> which led me to webfontloader, but the method they recommended would wait until your main package was delivered, than attached a script element to the DOM that would start downloading webfontloader, which would then start loading your desired fonts. All the while ensuring you get a nice long FOUT (flash of unstyled text) to wait for this chain of requests to complete.</p>
<h2 id="the-balancing-act-of-font-loading">The balancing act of font loading</h2>
<p>After reading through <a href="https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/webfont-optimization#webfonts_and_the_critical_rendering_path">google’s tips</a>, I settled on the solution below for a few reasons:</p>
<ul>
<li>
<p>The compact webfontloader JS is included in the main package, no need to wait for a second round trip before the browser even begins to download fonts. The webfontloader package is a very small addition to your main bundle, well worth the extra speed in getting the font downloaded.</p>
</li>
<li>
<p>No render blocking at all, thanks to webfontloader replacing any <code class="language-plaintext highlighter-rouge">@import</code> css rules.</p>
</li>
<li>
<p>Control over FOUT using a few simple CSS rules that work with webfontloader. I decided to hide the text until the fonts were loaded.</p>
</li>
<li>
<p>Automatic <a href="https://github.com/typekit/webfontloader#events">fallback</a> if google fonts fail to load.</p>
</li>
</ul>
<h2 id="kiss-shows-its-truth-yet-again">KISS shows its truth yet again</h2>
<p>After we run <code class="language-plaintext highlighter-rouge">npm install -E webfontloader</code>, go ahead and tell the client to use it when it loads up. Notice I don’t put this in a <code class="language-plaintext highlighter-rouge">Meteor.startup()</code> call, as I want it to be processed right away, not when the DOM is ready.</p>
<p><code class="language-plaintext highlighter-rouge">/imports/client/load-fonts.js</code></p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">WebFont</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">webfontloader</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">WebFont</span><span class="p">.</span><span class="nx">load</span><span class="p">({</span>
<span class="na">google</span><span class="p">:</span> <span class="p">{</span>
<span class="na">families</span><span class="p">:</span> <span class="p">[</span>
<span class="dl">'</span><span class="s1">Nunito:regular,bold</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Lato:regular,bold,italic</span><span class="dl">'</span><span class="p">,</span>
<span class="p">],</span>
<span class="p">},</span>
<span class="p">});</span>
</code></pre></div></div>
<p>This SCSS is optional, it prevents the FOUT by hiding the text, but falls back to showing it if the google font fails to load. I don’t write a lot of CSS code, so keep that in mind :).</p>
<p><code class="language-plaintext highlighter-rouge">/client/styles.scss</code></p>
<div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">h1</span><span class="o">,</span><span class="nt">h2</span><span class="o">,</span><span class="nt">h3</span><span class="o">,</span><span class="nt">h4</span><span class="o">,</span><span class="nt">h5</span><span class="o">,</span><span class="nt">h6</span><span class="o">,</span><span class="nt">p</span><span class="o">,</span><span class="nt">a</span> <span class="p">{</span>
<span class="nl">visibility</span><span class="p">:</span> <span class="nb">hidden</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.wf-active</span><span class="o">,</span> <span class="nc">.wf-inactive</span> <span class="p">{</span>
<span class="nt">h1</span><span class="o">,</span><span class="nt">h2</span><span class="o">,</span><span class="nt">h3</span><span class="o">,</span><span class="nt">h4</span><span class="o">,</span><span class="nt">h5</span><span class="o">,</span><span class="nt">h6</span><span class="o">,</span><span class="nt">p</span><span class="o">,</span><span class="nt">a</span> <span class="p">{</span>
<span class="nl">visibility</span><span class="p">:</span> <span class="nb">visible</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After looking at the screenshots on a fresh render, this gives me exactly what I was looking for, quick page load that shows a layout that matches the end result, with the actual text hidden until the font is loaded within a second.</p>
<p>While something this common probably should be laid out in official documentation somewhere, it was still worth the effort as the end result ended up shaving almost a second off the initial load time where the website displays nothing but a white page while it waits for render blocking resources.</p>J. Eric HartzogIn my search to load up some google fonts in Meteor, I found nothing in the official Meteor documentation about best practices, so search around community forums to figure out the best way to do to it right.