<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://cashapp.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://cashapp.github.io/" rel="alternate" type="text/html" /><updated>2024-10-03T15:14:57+00:00</updated><id>https://cashapp.github.io/feed.xml</id><title type="html">Cash App Code Blog</title><subtitle>Cash App Code Blog</subtitle><entry><title type="html">We moved! Update your RSS feed!</title><link href="https://cashapp.github.io/2021-04-15/" rel="alternate" type="text/html" title="We moved! Update your RSS feed!" /><published>2021-04-15T00:00:00+00:00</published><updated>2021-04-15T00:00:00+00:00</updated><id>https://cashapp.github.io/</id><content type="html" xml:base="https://cashapp.github.io/2021-04-15/"><![CDATA[<p>Hello reader of the Cash App code blog RSS feed!
We have moved to a shiny new domain at <a href="https://code.cash.app">code.cash.app</a>.</p>

<p>Please update your RSS reader to point at the new feed.
The new feed URL is https://code.cash.app/feed.xml.</p>

<p>If you were using a category-based feed their new URLs are:</p>
<ul>
  <li>Android: https://code.cash.app/feed/android.xml</li>
  <li>iOS: https://code.cash.app/feed/ios.xml</li>
  <li>Server: https://code.cash.app/feed/server.xml</li>
</ul>

<p>A new post hot off the presses awaits you after the switch.</p>

<p>Thanks for reading!</p>]]></content><author><name>Cash App</name></author><category term="[&quot;android&quot;, &quot;server&quot;, &quot;ios&quot;]" /><summary type="html"><![CDATA[Hello reader of the Cash App code blog RSS feed! We have moved to a shiny new domain at code.cash.app.]]></summary></entry><entry><title type="html">Rx to Coroutines Concepts, Part 1</title><link href="https://cashapp.github.io/2021-04-13/rx-to-coroutines-concepts" rel="alternate" type="text/html" title="Rx to Coroutines Concepts, Part 1" /><published>2021-04-13T00:00:00+00:00</published><updated>2021-04-13T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-04-13/rx-to-coroutines-concepts</id><content type="html" xml:base="https://cashapp.github.io/2021-04-13/rx-to-coroutines-concepts"><![CDATA[<p>Our Android codebase at Cash App is thoroughly invested in the strange and wonderful land of RxJava. Almost all of our presenter layer and a fair amount of code outside of it is written entirely in Observables and compositions of Observables.</p>

<p>By and large, this is great. Writing code in a reactive paradigm has given us superpowers that we can’t give up. Whether we’re wiring up a toggle in our debug drawer, a changing user interface element, a value written to SharedPreferences or a value pulled from a customer’s sync’d profile, for the most part everything Just Works™ without us doing any additional work to make it so or worrying about what thread a value was emitted on.</p>

<p>This is an incredible superpower. If you haven’t experienced it already, you’re missing out. And if it’s part of your day-to-day work, no doubt you’d rather not go back to the days when it wasn’t. Nothing beats a tool that gives you useful functionality for free, and that’s what the reactive paradigm does.</p>

<p>There is a cost, though. The price we paid was that we had to stop writing natural code, and switch to expressing ourselves in Rx-ese.</p>

<h2 id="how-things-used-to-work">How things used to work</h2>

<p>Let me show you what I mean. Here is a simple example of some straightline logic adapted from our codebase:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">viewModel</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">viewModel</span><span class="p">.</span><span class="n">value</span><span class="o">!!</span><span class="p">.</span><span class="nf">copy</span><span class="p">(</span><span class="n">isLoading</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="n">isError</span> <span class="p">=</span> <span class="k">false</span><span class="p">))</span>
<span class="kd">val</span> <span class="py">result</span> <span class="p">=</span> <span class="n">aliasRegistrar</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">registerArgs</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="k">is</span> <span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Successful</span><span class="p">)</span> <span class="p">{</span>
  <span class="kd">val</span> <span class="py">sms</span> <span class="p">=</span> <span class="n">alias</span><span class="p">.</span><span class="nf">redact</span><span class="p">()</span>
  <span class="kd">val</span> <span class="py">nextData</span> <span class="p">=</span> <span class="n">args</span><span class="p">.</span><span class="n">blockersData</span>
    <span class="p">.</span><span class="nf">copy</span><span class="p">(</span>
      <span class="n">flowToken</span> <span class="p">=</span> <span class="n">flowToken</span><span class="p">,</span>
      <span class="n">smsNumber</span> <span class="p">=</span> <span class="n">sms</span>
    <span class="p">)</span>
    <span class="p">.</span><span class="nf">updateFromResponseContext</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="n">responseContext</span><span class="o">!!</span><span class="p">)</span>
  <span class="n">goTo</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">blockersNavigator</span><span class="p">.</span><span class="nf">getNext</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">nextData</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This is straightline code, just as we would have written when we first learned to program imperatively. There are no callbacks or threads, just blocking requests.</p>

<p>This is also why this code isn’t written this way in Cash App’s codebase, nor can it be written this way in any UX. The call to <code class="language-plaintext highlighter-rouge">viewModel.accept</code> drives an immutable <code class="language-plaintext highlighter-rouge">viewModel</code> instance into the <code class="language-plaintext highlighter-rouge">View</code>, updating the user interface, while the call to <code class="language-plaintext highlighter-rouge">aliasRegistrar.register(registerArgs)</code> is a blocking call to a web service. Running both of these on the same thread isn’t feasible: we’ll either update the UI from the background thread and explode, or run a web request on the UI thread and hang.</p>

<p>This is such old news that I hate recapitulating it, but it’s worth saying: if this worked, we’d still write code this way.</p>

<p>But it doesn’t. This is why we started write code in callbacks instead.</p>

<h2 id="preemptibility-the-heros-call">Preemptibility: The Hero’s Call</h2>

<p>This failure of straightline code to even work is our call to write code that is <em>preemptible</em> instead of <em>greedy</em> with respect to computational resources.</p>

<p>Here’s what I mean: The old way of writing code is <em>greedy</em> because when you run it, it uses all of the thread you run it in to do its job. It starts at the start, and it continues on to the end, no matter what.</p>

<p><em>Preemptible</em> code tries to be the opposite: instead of using all of the thread you give it, it tries to get out of the way so that you can use the thread for other things. That’s what the humble callback does:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">viewModel</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">viewModel</span><span class="p">.</span><span class="n">value</span><span class="o">!!</span><span class="p">.</span><span class="nf">copy</span><span class="p">(</span><span class="n">isLoading</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="n">isError</span> <span class="p">=</span> <span class="k">false</span><span class="p">))</span>
<span class="n">aliasRegistrar</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">registerArgs</span><span class="p">)</span> <span class="p">{</span> <span class="n">result</span> <span class="p">-&gt;</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="k">is</span> <span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Successful</span><span class="p">)</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">sms</span> <span class="p">=</span> <span class="n">alias</span><span class="p">.</span><span class="nf">redact</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">nextData</span> <span class="p">=</span> <span class="n">args</span><span class="p">.</span><span class="n">blockersData</span>
      <span class="p">.</span><span class="nf">copy</span><span class="p">(</span>
        <span class="n">flowToken</span> <span class="p">=</span> <span class="n">flowToken</span><span class="p">,</span>
        <span class="n">smsNumber</span> <span class="p">=</span> <span class="n">sms</span>
      <span class="p">)</span>
      <span class="p">.</span><span class="nf">updateFromResponseContext</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="n">responseContext</span><span class="o">!!</span><span class="p">)</span>
    <span class="n">goTo</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">blockersNavigator</span><span class="p">.</span><span class="nf">getNext</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">nextData</span><span class="p">))</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The callback works because it gives the programmer two important capabilities:</p>

<p>You can choose which threading resources to use. The humble callback doesn’t make those resources explicit or parameterizable, but it does let you say, “Hey, run this registration on your background thread and get back to me on my current thread.”
You can split up your code into pieces that can be run one at a time.</p>

<p>It solves the problem, but it’s not pretty, especially when chaining multiple callbacks together. And each callback API has to independently expose how to schedule the work and assign it to a thread.</p>

<h2 id="translating-to-rx-the-high-elvish-of-callbacks">Translating to Rx, the High Elvish of Callbacks</h2>

<p>RxJava and other reactive frameworks emerged out of the functional programming world, but in the Android context they operate more like a sort of uber-callback tool. Once the solution is expressed as a reactive component like an <code class="language-plaintext highlighter-rouge">Observable</code> or <code class="language-plaintext highlighter-rouge">Single</code>…</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">aliasRegistrar</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">registerArgs</span><span class="p">).</span><span class="nf">subscribe</span> <span class="p">{</span> <span class="n">result</span> <span class="p">-&gt;</span> 
  <span class="o">..</span><span class="p">.</span>
<span class="p">}</span>
</code></pre></div></div>

<p>…one can take this uber-callback and chain on operators or other callbacks to do all kinds of things. The data flowing through the callback can be transformed, side effects can be triggered, additional callbacks can be triggered — it’s a whole language, a data transformation High Elvish that we can translate the original example into:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">disposables</span> <span class="p">+=</span> <span class="n">aliasRegistrar</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">registerArgs</span><span class="p">)</span>
  <span class="p">.</span><span class="nf">maybeUntil</span><span class="p">(</span><span class="n">signOut</span><span class="p">)</span>
  <span class="p">.</span><span class="nf">subscribeOn</span><span class="p">(</span><span class="n">backgroundScheduler</span><span class="p">)</span>
  <span class="p">.</span><span class="nf">toObservable</span><span class="p">()</span>
  <span class="p">.</span><span class="nf">startWith</span><span class="p">(</span><span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Loading</span><span class="p">)</span>
  <span class="p">.</span><span class="nf">errorHandlingSubscribe</span> <span class="p">{</span> <span class="n">result</span> <span class="p">-&gt;</span>
    <span class="k">when</span> <span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">is</span> <span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Loading</span> <span class="p">-&gt;</span> <span class="p">{</span>
        <span class="n">viewModel</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">viewModel</span><span class="p">.</span><span class="n">value</span><span class="o">!!</span><span class="p">.</span><span class="nf">copy</span><span class="p">(</span><span class="n">isLoading</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="n">isError</span> <span class="p">=</span> <span class="k">false</span><span class="p">))</span>
      <span class="p">}</span>
      <span class="k">is</span> <span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Successful</span> <span class="p">-&gt;</span> <span class="p">{</span>
        <span class="kd">val</span> <span class="py">sms</span> <span class="p">=</span> <span class="n">alias</span><span class="p">.</span><span class="nf">redact</span><span class="p">()</span>
        <span class="kd">val</span> <span class="py">nextData</span> <span class="p">=</span> <span class="n">args</span><span class="p">.</span><span class="n">blockersData</span>
          <span class="p">.</span><span class="nf">copy</span><span class="p">(</span>
            <span class="n">flowToken</span> <span class="p">=</span> <span class="n">flowToken</span><span class="p">,</span>
            <span class="n">smsNumber</span> <span class="p">=</span> <span class="n">sms</span>
          <span class="p">)</span>
          <span class="p">.</span><span class="nf">updateFromResponseContext</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="n">responseContext</span><span class="o">!!</span><span class="p">)</span>
        <span class="n">goTo</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">blockersNavigator</span><span class="p">.</span><span class="nf">getNext</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">nextData</span><span class="p">))</span>
      <span class="p">}</span>
      <span class="k">else</span> <span class="p">-&gt;</span> <span class="p">{</span>

      <span class="p">}</span>
    <span class="p">}</span>
  <span class="p">}</span>
</code></pre></div></div>

<p>The grammar of this example is different: the subject is usually not, say, the response, but a stream of data containing the response. To the uninitiated it may seem unnatural, but the strange powers it grants are irresistable. (And fun.) Namely, it does the two things callbacks let you do: it lets you split up your code into independently schedulable morsels, and it lets you assign those morsels to a particular thread. Delicious.</p>

<p>Work for a long time in RxJava, and eventually one starts to think in this grammar. It’s easy to forget that it doesn’t correspond to how any of us would describe the operation of the machine to a lay person, or to most programmers. If someone asks what the linear control flow of your reactive presenter is, you are better off encouraging them to rethink their question in terms of reactive streams than giving a straight answer. Because if you do, that answer might look like this:</p>

<p><img src="/assets/2021-04/reactive-control-flow.png" alt="" /></p>

<p>Don’t think about Rx this way! It’s a bad idea. Unfortunately, this way of thinking is the best fit for many real world tasks</p>

<h2 id="returning-to-the-common-tongue-with-coroutines">Returning to the Common Tongue With Coroutines</h2>

<p>RxJava converts like me often zero in on Kotlins’ reactive <code class="language-plaintext highlighter-rouge">Flow</code> tools, which allow one to go on speaking foreign grammar. But coroutines allow you to discard it and write what is essentially straight up Kotlin code. We could replicate the original example like this:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">scope</span> <span class="p">=</span> <span class="nc">CoroutineScope</span><span class="p">(</span><span class="n">dispatchers</span><span class="p">.</span><span class="n">io</span><span class="p">)</span>
<span class="n">scope</span><span class="p">.</span><span class="nf">launch</span> <span class="p">{</span>
  <span class="n">viewModel</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">viewModel</span><span class="p">.</span><span class="n">value</span><span class="o">!!</span><span class="p">.</span><span class="nf">copy</span><span class="p">(</span><span class="n">isLoading</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="n">isError</span> <span class="p">=</span> <span class="k">false</span><span class="p">))</span>
  <span class="kd">val</span> <span class="py">result</span> <span class="p">=</span> <span class="n">aliasRegistrar</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">registerArgs</span><span class="p">)</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="k">is</span> <span class="nc">AliasRegistrar</span><span class="p">.</span><span class="nc">Result</span><span class="p">.</span><span class="nc">Successful</span><span class="p">)</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">sms</span> <span class="p">=</span> <span class="n">alias</span><span class="p">.</span><span class="nf">redact</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">nextData</span> <span class="p">=</span> <span class="n">args</span><span class="p">.</span><span class="n">blockersData</span>
      <span class="p">.</span><span class="nf">copy</span><span class="p">(</span>
        <span class="n">flowToken</span> <span class="p">=</span> <span class="n">flowToken</span><span class="p">,</span>
        <span class="n">smsNumber</span> <span class="p">=</span> <span class="n">sms</span>
      <span class="p">)</span>
      <span class="p">.</span><span class="nf">updateFromResponseContext</span><span class="p">(</span><span class="n">result</span><span class="p">.</span><span class="n">responseContext</span><span class="o">!!</span><span class="p">)</span>
    <span class="n">goTo</span><span class="p">.</span><span class="nf">accept</span><span class="p">(</span><span class="n">blockersNavigator</span><span class="p">.</span><span class="nf">getNext</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">nextData</span><span class="p">))</span>
  <span class="p">}</span>
<span class="p">}</span>
<span class="n">disposables</span> <span class="p">+=</span> <span class="n">signOut</span><span class="p">.</span><span class="nf">errorHandlingSubscribe</span> <span class="p">{</span> <span class="n">scope</span><span class="p">.</span><span class="nf">cancel</span><span class="p">()</span> <span class="p">}</span>
</code></pre></div></div>

<p>We retain the same grammar as the original example, and while we could use <code class="language-plaintext highlighter-rouge">Flow</code> to express this reactively we are not <em>required</em> to as we were in RxJava. We don’t have to give up our superpowers to do it, either: we can assign work to different threads (with different <code class="language-plaintext highlighter-rouge">CoroutineDispatchers</code>, or wrapping a section of work in <code class="language-plaintext highlighter-rouge">withContext</code>), and we can split up our work into little morsels of – hold up, where are our morsels?</p>

<p>This is why coroutines had to be a language feature rather than a library like RxJava: the compiler morselizes your code based on the methods it calls. <code class="language-plaintext highlighter-rouge">aliasRegistrar.register(registerArgs)</code> is a <code class="language-plaintext highlighter-rouge">suspend fun</code> in the example above. Being <code class="language-plaintext highlighter-rouge">suspend</code> signals to the compiler that it should split your method before and after that method call, creating a morsel on either side. (See Coroutines: Suspending State Machines for all the details on how that happens.)</p>

<h2 id="how-big-a-deal-is-it-really">How big a deal is it, really?</h2>

<p>The wins laid out here are mostly in clear expression and comprehension. If nothing else, never having to explain the difference between <code class="language-plaintext highlighter-rouge">subscribeOn</code> and <code class="language-plaintext highlighter-rouge">observeOn</code> would save a lot of heartache in the future. But the ability to clearly express how a system works in code is invaluable, and RxJava denies it to us by forcing us into a reactive paradigm.</p>

<p>There are other benefits to coroutines, too. But that’s for a later post.</p>]]></content><author><name>Bill Phillips</name></author><category term="[&quot;android&quot;]" /><summary type="html"><![CDATA[The reactive paradigm is powerful, but can also be obscure. Coroutines provide an alternative paradigm that is better in every way.]]></summary></entry><entry><title type="html">Great teams merge fast</title><link href="https://cashapp.github.io/2021-04-08/great-teams-merge-fast" rel="alternate" type="text/html" title="Great teams merge fast" /><published>2021-04-08T00:00:00+00:00</published><updated>2021-04-08T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-04-08/great-teams-merge-fast</id><content type="html" xml:base="https://cashapp.github.io/2021-04-08/great-teams-merge-fast"><![CDATA[<p><img src="/assets/2021-03/great_teams_merge_fast_cover.jpg" alt="Great teams merge fast" /></p>

<p>One of the many things I was constantly amazed about during my first few weeks at <a href="https://square.com/">Square</a>/<a href="https://cash.app/">Cash App</a> was seeing how fast this team shipped code. All members of the Android team (and likely other platforms as well) were merging to the main git branch multiple times every day.</p>

<p>This was in stark contrast to all my previous jobs where it was common for us to raise large PRs after a couple days of work. Getting them merged-in wasn’t fast either. Reviewing these PRs was one of my least favorite parts of my work. In all fairness, I was always the first person to object that we can’t merge faster. Boy was I wrong.</p>

<p>To other teams who feel they can be merging faster, here are a few things I’ve noticed I’m doing differently at Square:</p>

<h2 id="trust-your-team-members">Trust your team members</h2>

<p>When reviewing PRs, we approve them immediately after leaving our feedback instead of holding it to ransom until our requested changes are made. This unblocks people from merging their PRs once they’ve resolved all feedback instead of waiting for another round of reviews. We trust our team members that they’re also trying to do their best for the project.</p>

<p>If someone merges code that shouldn’t have been merged, that’s okay. A broken main branch can be quickly fixed. Bad commits can be easily reverted because rolling release windows are very forgiving. Nobody expects internal staging builds to be fully stable.</p>

<h2 id="automate-nitpicking-and-bikeshedding">Automate nitpicking and bikeshedding</h2>

<p><img src="/assets/2021-03/deny_listed_api_detector.png" alt="DenyListedApiDetector" /></p>

<p>Code suggestions that often come up in PR reviews are bad signs (<a href="https://blog.danlew.net/2021/02/23/stop-nitpicking-in-code-reviews/">Dan Lew agrees</a>). One of my favorite parts of our stack of static checks is a <code class="language-plaintext highlighter-rouge">DenyListedApiDetector</code> linter that discourages developers from using APIs we dislike. It’s like Kotlin’s <code class="language-plaintext highlighter-rouge">@Deprecated</code> annotation, but for code we don’t own — like the Android SDK or third-party libraries. For example, we block usages of <code class="language-plaintext highlighter-rouge">Observable#test()</code> in favor of an internal implementation that’s like <a href="https://github.com/cashapp/turbine">cashapp/turbine</a> but for RxJava.</p>

<p>Here’s another example/question for you: how many of these APIs does your Android app use for reading XML drawables?</p>

<div class="language-markdown highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">1.</span> Context#getDrawable()
<span class="p">2.</span> ContextCompat.getDrawable()
<span class="p">3.</span> AppCompatResources.getDrawable()
<span class="p">4.</span> ResourcesCompat.getDrawable()
</code></pre></div></div>

<p>Only one of them is universally correct. We’ve deny-listed the rest of them.</p>

<h2 id="stack-prs">Stack PRs</h2>

<p>PRs that introduce 100 lines of changes or lower are the best. Anything more than that and the likelihood of reviewers skimming through crucial changes increases with every line, especially if they own Logitech MX Master that can scroll billions of lines with a single flick. For related work, we stack our PRs on top of each other so that they can be reviewed super quickly. GitHub automatically handles re-targeting of stacked PRs when their base is merged.</p>

<p>I like to think that by making it easy to review PRs, we’re essentially showing empathy towards our customers. An unsuspecting bad piece of code can potentially result in people being unable to use their money and that’ll be a terrible thing to do.</p>

<h2 id="ship-work-in-progress-code-to-production">Ship work-in-progress code to production</h2>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">featureFlags</span><span class="p">[</span><span class="nc">SuperSecretFeature</span><span class="p">].</span><span class="n">isEnabled</span><span class="p">)</span> <span class="p">{</span> 
  <span class="n">navigator</span><span class="p">.</span><span class="nf">goTo</span><span class="p">(</span><span class="nc">SuperSecretScreen</span><span class="p">())</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> 
  <span class="n">navigator</span><span class="p">.</span><span class="nf">goTo</span><span class="p">(</span><span class="nc">BoringOldScreen</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Merging code daily effectively translates to shipping work-in-progress code in production builds. To keep customers from running new features or incomplete refactorings of old code, we use feature flags.</p>

<p>While this may sound obvious to some, I’ve worked on teams that instead held off merging their work until it’s ready. We were able to make it work, but it had huge downsides. Feature branches isolate your team, preventing other developers from seeing what you’re working on until it’s done. In large teams, this often leads to people organizing merge-conflict resolving parties. They’re not fun.</p>

<p>Feature flags are also great for undoing mistakes. At Cash App’s scale, refactoring code that’s used by millions of customers is sometimes scary, but I feel comfortable knowing that I can push a button and remotely switch all our customers from <code class="language-plaintext highlighter-rouge">BottomSheetV2</code> to <code class="language-plaintext highlighter-rouge">BottomSheetV1</code> if an obscure bug is found in v2.</p>

<hr />

<p>Wrapping up, I like to think that when an engineering team is empowered to merge fast, it both demonstrates and cultivates trust in each other. Barriers to merging code are best used for high-level design discussions and pain points in the codebase, and not as a substitute for pairing sessions.</p>

<p>Cover photo by <a href="https://unsplash.com/@sawyerbengtson">Sawyer Bengtson</a>.</p>]]></content><author><name>Saket Narayan</name></author><summary type="html"><![CDATA[]]></summary></entry><entry><title type="html">AssistedInject is dead, long live AssistedInject!</title><link href="https://cashapp.github.io/2021-03-31/assisted-inject-is-dead-long-live-assisted-inject" rel="alternate" type="text/html" title="AssistedInject is dead, long live AssistedInject!" /><published>2021-03-31T00:00:00+00:00</published><updated>2021-03-31T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-03-31/assisted-inject-is-dead-long-live-assisted-inject</id><content type="html" xml:base="https://cashapp.github.io/2021-03-31/assisted-inject-is-dead-long-live-assisted-inject"><![CDATA[<p>After ~5 years of existing and ~4 years as an open source project, our AssistedInject library has been deleted. Mourn not, however, for the same functionality is <a href="https://dagger.dev/dev-guide/assisted-injection.html">now available</a> directly in Dagger and InflationInject <a href="https://github.com/cashapp/InflationInject">got its own repo</a>.</p>

<p><em>(Psst. Know Dagger but don’t know what assisted or inflation injection is? That’s okay. Check out <a href="https://jakewharton.com/helping-dagger-help-you/">this introductory talk</a>. Otherwise, this post won’t make much sense!)</em></p>

<p>This post will cover the steps required for migrating from Square AssistedInject to Dagger AssistedInject, minor differences between the two, and the status of InflationInject.</p>

<h2 id="migration-and-differences">Migration and differences</h2>

<p>Our migration to Dagger’s AssistedInject was done in a few discrete steps after failing to migrate in a single change. It is possible to migrate in a single change, but multiple changes help ensure the differences are handled correctly.</p>

<p>These steps assume you are using AssistedInject 0.6.0 which was the latest stable version for the past six months.</p>

<h3 id="step-1-upgrade-to-07">Step 1: Upgrade to 0.7</h3>

<p><a href="https://github.com/cashapp/InflationInject/blob/trunk/CHANGELOG.md#070---2021-03-21">Version 0.7.0</a> contains changes from the last 6 months which will make future changes simpler.</p>

<p>Most notably, this version will require specifying qualifier annotations for assisted parameters of the same type.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class Example @AssistedInject constructor(
   graphValue: GraphValue,
<span class="gd">-  @Assisted assistedValue1: String,
</span><span class="gi">+  @Assisted @Named("one") assistedValue1: String,
</span><span class="gd">-  @Assisted assistedValue2: String,
</span><span class="gi">+  @Assisted @Named("two") assistedValue2: String,
</span> ) {
   // ...
   
   @AssistedInject.Factory
   interface Factory {
     fun create(
<span class="gd">-      assistedValue1: String,
</span><span class="gi">+      @Named("one") assistedValue1: String,
</span><span class="gd">-      assistedValue2: String,
</span><span class="gi">+      @Named("two") assistedValue2: String,
</span>     ): Example
   }
 }
</code></pre></div></div>

<p>While this seems like boilerplate, the previous behavior of relying on parameter names was not safe as they are not always available (Dagger will also require this).</p>

<p>This version also includes a bug fix that allows using Dagger’s <code class="language-plaintext highlighter-rouge">@AssistedInject</code>-annotated types
inside a Square <code class="language-plaintext highlighter-rouge">@AssistedInject</code> or <code class="language-plaintext highlighter-rouge">@InflationInject</code>-annotated type. If you are migrating incrementally rather than all modules at once this fix is essential for interoperability between the two libraries.</p>

<h3 id="step-2-upgrade-to-08">Step 2: Upgrade to 0.8</h3>

<p>If you are using inflation injection, <a href="https://github.com/cashapp/InflationInject/blob/trunk/CHANGELOG.md#081---2021-03-22">version 0.8.1</a> changes the annotation used in the constructor from <code class="language-plaintext highlighter-rouge">@Assisted</code> to <code class="language-plaintext highlighter-rouge">@Inflated</code>.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-import com.squareup.inject.assisted.Assisted
</span><span class="gi">+import com.squareup.inject.inflation.Inflated
</span> import com.squareup.inject.inflation.InflationInject

 class ExampleView @InjectionInject constructor(
<span class="gd">-  @Assisted context: Context,
</span><span class="gi">+  @Inflated context: Context,
</span><span class="gd">-  @Assisted attrs: AttributeSet,
</span><span class="gi">+  @Inflated attrs: AttributeSet,
</span>   picasso: Picasso,
 ) : View {
   // ...
 }
</code></pre></div></div>

<p>Since Dagger validates that all usages of its <code class="language-plaintext highlighter-rouge">@Assisted</code> are inside constructors annotated with its <code class="language-plaintext highlighter-rouge">@AssistedInject</code>, this change is required before migrating. Additionally, since the layout inflater provides the values instead of the user, the new name is also more accurate.</p>

<h3 id="step-3-switch-to-dagger">Step 3: Switch to Dagger</h3>

<p>At this point we can start migrating to Dagger’s version. While it has been released for a few versions, the latest (at time of writing) is 2.33 which has some important bug fixes.</p>

<p>Aside from simple import changes, Dagger uses <code class="language-plaintext highlighter-rouge">@AssistedFactory</code> instead of <code class="language-plaintext highlighter-rouge">@AssistedInject.Factory</code> and disambiguation for assisted parameters of the same type goes directly on the <code class="language-plaintext highlighter-rouge">@Assisted</code> annotation rather than through a qualifier annotation.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-import com.squareup.inject.assisted.Assisted
</span><span class="gi">+import dagger.assisted.Assisted
+import dagger.assisted.AssistedFactory
</span><span class="gd">-import com.squareup.inject.asissted.AssistedInject
</span><span class="gi">+import dagger.assisted.AssistedInject
</span><span class="gd">-import javax.inject.Named
</span>
 class Example @AssistedInject constructor(
   graphValue: GraphValue,
<span class="gd">-  @Assisted @Named("one") assistedValue1: String,
</span><span class="gi">+  @Assisted("one") assistedValue1: String,
</span><span class="gd">-  @Assisted @Named("two") assistedValue2: String,
</span><span class="gi">+  @Assisted("two") assistedValue2: String,
</span> ) {
   // ...
   
<span class="gd">-  @AssistedInject.Factory
</span><span class="gi">+  @AssistedFactory
</span>   interface Factory {
     fun create(
<span class="gd">-      @Named("one") assistedValue1: String,
</span><span class="gi">+      @Assisted("one") assistedValue1: String,
</span><span class="gd">-      @Named("two") assistedValue2: String,
</span><span class="gi">+      @Assisted("two") assistedValue2: String,
</span>     ): Example
   }
 }
</code></pre></div></div>

<p>With assisted injection built-in to Dagger there is no need to have something like <code class="language-plaintext highlighter-rouge">@AssistedModule</code>. Delete it!</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-@AssistedModule
-@Module(includes = AssistedInject_ExampleModule.class)
</span><span class="gi">+@Module
</span> class ExampleModule {
   // ...
 }
</code></pre></div></div>

<p>There is one pattern to watch out for in this migration. Dagger’s assisted injection requires that the factory return type matches the type which is being created. If one of your factories was returning a supertype of the enclosing class, you will have to pull out a second factory interface and use <code class="language-plaintext highlighter-rouge">@Binds</code> to support it.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> class TestPresenter @AssistedInject constructor(
   // ...
 ) : ObservableTransformer&lt;Event, Model&gt; {
   // ...
  
<span class="gi">+  @AssistedFactory
+  interface DaggerFactory : Factory {
+    override fun create(/*..*/): TestPresenter
+  }
</span><span class="gd">-  @AssistedInject.Factory
</span>   interface Factory {
     fun create(/*..*/): ObservableTransformer&lt;Event, Model&gt;
   }
 }
</code></pre></div></div>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> // In a module somewhere…
<span class="gi">+@Binds
+abstract fun daggerFactoryTestPresenter(
+  factory: TestPresenter.DaggerFactory,
+): TestPresenter.Factory
</span></code></pre></div></div>

<h3 id="step-4-remove-dependencies-or-upgrade-to-09">Step 4: Remove dependencies or upgrade to 0.9</h3>

<p>At this point you can remove the Square AssistedInject dependencies. And if you’re not using inflation injection then you’re done!</p>

<p>If you are using inflation injection, upgrade to <a href="https://github.com/cashapp/InflationInject/blob/trunk/CHANGELOG.md#091---2021-03-26">version 0.9.1</a> which completely removes the assisted injection parts. You’re <em>almost</em> done…</p>

<h2 id="inflationinject-10">InflationInject 1.0!</h2>

<p>With assisted injection now excised from the project, <a href="https://github.com/cashapp/InflationInject">the GitHub repo</a> was renamed to InflationInject, moved into the ‘cashapp’ organization, and given a quick makeover for its 1.0.0 release.</p>

<p>The Maven coordinates now use the groupId of ‘app.cash.inject’.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-implementation 'com.squareup.inject:inflation-inject:0.9.1'
</span><span class="gi">+implementation 'app.cash.inject:inflation-inject:1.0.0'
</span><span class="gd">-annotationProcessor 'com.squareup.inject:inflation-inject-processor:0.9.1'
</span><span class="gi">+annotationProcessor 'app.cash.inject:inflation-inject-processor:1.0.0'
</span></code></pre></div></div>

<p>Imports were changed to also use <code class="language-plaintext highlighter-rouge">app.cash.inject</code> as the base package.</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-import com.squareup.inject.inflation.Inflated
</span><span class="gi">+import app.cash.inject.inflation.Inflated
</span><span class="gd">-import com.squareup.inject.inflation.InflationInject
</span><span class="gi">+import app.cash.inject.inflation.InflationInject
</span>
 class ExampleView @InjectionInject constructor(
   // ...
 ) : View {
   // ...
 }
</code></pre></div></div>

<p>Inflation injection serves a niche need, and with Jetpack Compose UI becoming viable the project is unlikely to see much more development. With the primary goal of landing assisted injection upstream into Dagger fulfilled, it’s still nice to have the rest of the project reach 1.0.</p>]]></content><author><name>Jake Wharton</name></author><category term="[&quot;android&quot;]" /><summary type="html"><![CDATA[After ~5 years of existing and ~4 years as an open source project, our AssistedInject library has been deleted. Mourn not, however, for the same functionality is now available directly in Dagger and InflationInject got its own repo.]]></summary></entry><entry><title type="html">protoc-gen-grpc-gateway-ts - Clean, Idiomatic TypeScript for grpc-gateway</title><link href="https://cashapp.github.io/2021-03-26/protoc-gen-grpc-gateway-ts-clean-idiomatic-typescript-for-grpc-gateway" rel="alternate" type="text/html" title="protoc-gen-grpc-gateway-ts - Clean, Idiomatic TypeScript for grpc-gateway" /><published>2021-03-26T00:00:00+00:00</published><updated>2021-03-26T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-03-26/protoc-gen-grpc-gateway-ts-clean-idiomatic-typescript-for-grpc-gateway</id><content type="html" xml:base="https://cashapp.github.io/2021-03-26/protoc-gen-grpc-gateway-ts-clean-idiomatic-typescript-for-grpc-gateway"><![CDATA[<p><a href="https://github.com/grpc-ecosystem/protoc-gen-grpc-gateway-ts"><code class="language-plaintext highlighter-rouge">protoc-gen-grpc-gateway-ts</code></a> is a TypeScript client generator for the <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a> project. 
It generates idiomatic TypeScript clients that connect the web frontend and golang gRPC backend fronted by <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a>.</p>

<h2 id="background">Background</h2>

<p>In Cash App, we prefer gRPC as the way of communication. We also use <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a> to expose our gRPC services to legacy and web clients over HTTP/1.</p>

<p>This worked well until the day we needed to start integrating <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a> with Web clients. To build a Web client for a gRPC enabled service, there are two options.</p>

<ol>
  <li>
    <p>Use <a href="https://github.com/grpc-ecosystem/grpc-gateway/tree/master/protoc-gen-openapiv2"><code class="language-plaintext highlighter-rouge">protoc-gen-openapiv2</code></a> (previously <code class="language-plaintext highlighter-rouge">protoc-gen-swagger</code>) to generate an OpenAPI schema then run an OpenAPI generator to get the Web client</p>
  </li>
  <li>
    <p>Use <a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a> with envoy grpc-web filter to pipe the request through.</p>
  </li>
</ol>

<p>Although both are valid options and have reasonable numbers of users, after some investigation both options have their own pain points.</p>

<h2 id="pain-points">Pain points</h2>

<h3 id="protoc-gen-openapiv2"><a href="https://github.com/grpc-ecosystem/grpc-gateway/tree/master/protoc-gen-openapiv2"><code class="language-plaintext highlighter-rouge">protoc-gen-openapiv2</code></a></h3>

<p><a href="https://github.com/OAI/OpenAPI-Specification">OpenAPI</a> has both popularity and maturity. However, going down this path means</p>

<ul>
  <li>
    <p>proto -&gt; OpenAPI -&gt; Typescript</p>
  </li>
  <li>
    <p>Generated clients only support unary gRPC calls, where <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a> can do server side streaming.</p>
  </li>
</ul>

<h3 id="grpc-web"><a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a></h3>

<p><a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a> is a Javascript implementation of gPRC for browsers. To use it, we found:</p>

<ul>
  <li><a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a> client requires either a proxy, or envoy with a filter in between to do the translation.
    <ul>
      <li>The proxy is an extra layer for our service while we have <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a> already</li>
      <li>For the envoy + filter option it was not easy to set up.</li>
    </ul>
  </li>
  <li>The generated <a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a> client is verbose and hard to use.
    <ul>
      <li>Setter is required for every desired field and there is no way to simplify it.</li>
    </ul>
  </li>
  <li>TypeScript support
    <ul>
      <li>It is experimental</li>
      <li>3 assets files(1 <code class="language-plaintext highlighter-rouge">.js</code> &amp; 2 <code class="language-plaintext highlighter-rouge">.ts</code> files) generated for a single proto file. Here is <a href="https://github.com/improbable-eng/grpc-web/tree/master/integration_test/ts/_proto/improbable/grpcweb/test">an example</a> of the <a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a> proto generation output.</li>
    </ul>
  </li>
</ul>

<h2 id="solution">Solution</h2>

<p>To solve the pain points found in both <a href="https://github.com/grpc-ecosystem/grpc-gateway/tree/master/protoc-gen-openapiv2"><code class="language-plaintext highlighter-rouge">protoc-gen-openapiv2</code></a> &amp; <a href="https://github.com/grpc/grpc-web"><code class="language-plaintext highlighter-rouge">grpc-web</code></a>, we build <a href="https://github.com/grpc-ecosystem/protoc-gen-grpc-gateway-ts"><code class="language-plaintext highlighter-rouge">protoc-gen-grpc-gateway-ts</code></a>.
It comes with the following features:</p>

<ol>
  <li>Idiomatic TypeScript clients and messages</li>
  <li>Single step - proto -&gt; TypeScript</li>
  <li>Supports both unary calls and server side streaming</li>
  <li>POJO message construction guarded by message type definitions.</li>
  <li>Standard protoc plugin that can be run alongside any other plugins</li>
</ol>

<p>After the creation of <a href="https://github.com/grpc-ecosystem/protoc-gen-grpc-gateway-ts"><code class="language-plaintext highlighter-rouge">protoc-gen-grpc-gateway-ts</code></a>, we have</p>

<ul>
  <li>Battle tested against 1000s of proto files in Cash App &amp; Square for proto generation.</li>
  <li>Used the generated code extensively for our internal web admin modules, giving the following benefits:
    <ul>
      <li>POJO message construction greatly simplifies the process to make a request, a simple request can be one liner.</li>
      <li>TypeScript is widely supported in main-stream IDEs, code completion, documentation flows naturally to help out daily development</li>
      <li>The ability to do server side streaming means the Web client is feature parity with all other clients.</li>
    </ul>
  </li>
  <li>Is gaining more and more traction across internal teams</li>
</ul>

<p>Here is an example of a simple counter service</p>

<p>The proto file:</p>
<div class="language-proto highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// file: counter.proto</span>
<span class="kd">message</span> <span class="nc">Request</span> <span class="p">{</span>
  <span class="kt">int32</span> <span class="na">counter</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>

<span class="kd">message</span> <span class="nc">Response</span> <span class="p">{</span>
  <span class="kt">int32</span> <span class="na">result</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>

<span class="kd">service</span> <span class="n">CounterService</span> <span class="p">{</span>
  <span class="k">rpc</span> <span class="n">Increment</span><span class="p">(</span><span class="n">Request</span><span class="p">)</span> <span class="k">returns</span> <span class="p">(</span><span class="n">Response</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p><strong>Generated TypeScript file:</strong></p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// file: counter.pb.ts</span>
<span class="k">import</span> <span class="o">*</span> <span class="k">as</span> <span class="nx">fm</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">./fetch.pb</span><span class="dl">"</span>

<span class="k">export</span> <span class="kd">type</span> <span class="nx">Request</span> <span class="o">=</span> <span class="p">{</span>
  <span class="nx">counter</span><span class="p">?:</span> <span class="kr">number</span>
<span class="p">}</span>

<span class="k">export</span> <span class="kd">type</span> <span class="nx">Response</span> <span class="o">=</span> <span class="p">{</span>
  <span class="nx">result</span><span class="p">?:</span> <span class="kr">number</span>
<span class="p">}</span>

<span class="k">export</span> <span class="kd">class</span> <span class="nx">CounterService</span> <span class="p">{</span>
  <span class="k">static</span> <span class="nx">Increment</span><span class="p">(</span><span class="nx">req</span><span class="p">:</span> <span class="nx">UnaryRequest</span><span class="p">,</span> <span class="nx">initReq</span><span class="p">?:</span> <span class="nx">fm</span><span class="p">.</span><span class="nx">InitReq</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="nx">UnaryResponse</span><span class="o">&gt;</span> <span class="p">{</span>
    <span class="k">return</span> <span class="nx">fm</span><span class="p">.</span><span class="nx">fetchReq</span><span class="o">&lt;</span><span class="nx">UnaryRequest</span><span class="p">,</span> <span class="nx">UnaryResponse</span><span class="o">&gt;</span><span class="p">(</span><span class="s2">`/main.CounterService/Increment`</span><span class="p">,</span> <span class="p">{...</span><span class="nx">initReq</span><span class="p">,</span> <span class="na">method</span><span class="p">:</span> <span class="dl">"</span><span class="s2">POST</span><span class="dl">"</span><span class="p">,</span> <span class="na">body</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">req</span><span class="p">)})</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Example usage of the generated <code class="language-plaintext highlighter-rouge">counter.pb.ts</code></p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span><span class="nx">CounterService</span><span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./counter.pb</span><span class="dl">'</span>

<span class="c1">// increase the given number once  </span>
<span class="k">async</span> <span class="kd">function</span> <span class="nx">increase</span><span class="p">(</span><span class="nx">base</span><span class="p">:</span> <span class="kr">number</span><span class="p">):</span> <span class="nb">Promise</span><span class="o">&lt;</span><span class="kr">number</span><span class="o">&gt;</span> <span class="p">{</span>
  <span class="kd">const</span> <span class="nx">resp</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">CounterService</span><span class="p">.</span><span class="nx">Increase</span><span class="p">({</span><span class="na">counter</span><span class="p">:</span> <span class="nx">base</span><span class="p">})</span>
  <span class="k">return</span> <span class="nx">resp</span><span class="p">.</span><span class="nx">result</span>
<span class="p">}</span> 
</code></pre></div></div>

<h2 id="open-sourcing">Open Sourcing</h2>
<p>It was originally planned to be open-sourced via <a href="https://github.com/square">Square</a>/<a href="https://github.com/cashapp">Cash App</a>’s public Github account. 
However, with the help from Google Engineers, it now lives in an ideal location: <a href="https://github.com/grpc-ecosystem/"><code class="language-plaintext highlighter-rouge">grpc-ecosystem</code></a>, which is next to <a href="https://github.com/grpc-ecosystem/grpc-gateway"><code class="language-plaintext highlighter-rouge">grpc-gateway</code></a>.</p>

<h2 id="end-note">End Note</h2>
<p>It’s a fun and resourceful journey to create <a href="https://github.com/grpc-ecosystem/protoc-gen-grpc-gateway-ts"><code class="language-plaintext highlighter-rouge">protoc-gen-grpc-gateway-ts</code></a>. Hope it soon will start getting traction and contributions around the globe.</p>]]></content><author><name>Yun Lai</name></author><category term="[&quot;server&quot;, &quot;protobuf&quot;, &quot;typescript&quot;, &quot;frontend&quot;, &quot;grpc&quot;]" /><summary type="html"><![CDATA[protoc-gen-grpc-gateway-ts is a TypeScript client generator for the grpc-gateway project. It generates idiomatic TypeScript clients that connect the web frontend and golang gRPC backend fronted by grpc-gateway.]]></summary></entry><entry><title type="html">Rolling Out Org-Wide Changes</title><link href="https://cashapp.github.io/2021-03-25/rolling-out-org-wide-changes" rel="alternate" type="text/html" title="Rolling Out Org-Wide Changes" /><published>2021-03-25T00:00:00+00:00</published><updated>2021-03-25T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-03-25/rolling-out-org-wide-changes</id><content type="html" xml:base="https://cashapp.github.io/2021-03-25/rolling-out-org-wide-changes"><![CDATA[<p>Cash App has grown into a complex engineering organization. Our team of hundreds of engineers deploys dozens of services to AWS multiple times a day. Our teams span several time zones and locations, and each team has its own availability and velocity requirements.</p>

<p>Unfortunately our custom Kubernetes management tooling wasn’t able to keep up with our organizational complexity or able to provide the service management features which we required. Service teams were also not able to dictate deployment strategies for their custom services and we wanted infrastructure to support <a href="https://en.wikipedia.org/wiki/Continuous_delivery">Continuous Delivery</a>.</p>

<p>Changing this was a big challenge. We needed to change both the people process and software tools without interrupting our teammates ability to deploy new code. We also needed to release onboarding docs to provide training for all the new things we’d be rolling out.</p>

<p>I’m happy with the process we used and really proud of its result. This post explains how we made this change. Hopefully it’ll help you to make your own organization-wide changes!</p>

<p><img src="/assets/2021-03/SpinnakerAndCash.png" alt="" /></p>

<h2 id="step-1-requirements-list">Step 1. Requirements List</h2>

<p>We started by listing all the features we wanted out of a tool that handles Cash App service management. We did not want to invest in our own homegrown tooling at that point in time, so searched for an open source, industry standard tool that met our requirements. We made a shortlist, analyzed how well each tool met our requirements, and ranked them in terms of ease of use and integration with our environment.</p>

<p>The requirements, possible tools, and analysis on each were outlined in a document which was distributed to both the Platform team and service teams which would be impacted by the change.</p>

<h2 id="step-2-proof-of-concept">Step 2. Proof of Concept</h2>

<p>From the shortlist, we chose <a href="https://spinnaker.io/">Spinnaker</a> —  an open source continuous delivery tool that claimed to meet almost all of our requirements. We set up a local deployment to verify the features we were interested in, and to understand what we needed to do to configure Spinnaker for our purposes. We tested the main integration points with our existing environment, as well as authentication and authorization to ensure it fit with our existing security model.</p>

<p>We demoed the proof of concept to the team at our end of sprint meeting.</p>

<h2 id="step-3-design-doc">Step 3. Design Doc</h2>

<p>Next we created a design doc to encapsulate our design for integrating Spinnaker into our system. Our design docs are a high level description of the problem space and solution, and can be broken down into the following:</p>
<ul>
  <li>Overview of the problem statement, and the proposed solution</li>
  <li>Goals and Anti-Goals of the project</li>
  <li>Design — this can be quite detailed, as it can include architecture diagrams, security overview (integration points with the system, PII, authn/authz, additional components required), and the description and breakdown of the solution</li>
  <li>Timeline</li>
  <li>Open Questions</li>
</ul>

<p>Our design docs are sometimes expanded to include more, but almost always include the above items. We share the design doc to spread awareness — especially to teams who will be impacted by our change.</p>

<h2 id="step-4-security">Step 4. Security</h2>

<p>At this point, we wanted to ensure that the integration of Spinnaker into our current system was not going to make us vulnerable from a security perspective. We had multiple meetings with the security team and they closely vetted the integration and the system, to make sure points of potential attack were secure and there was minimal risk. All communication between Spinnaker and our system was secured and left clear audit trails. We also used the opportunity to improve our security tooling and infrastructure.</p>

<p>This is recommended for any system you introduce to your organization, as an attack can compromise integrity of user data, cause significant downtime, or leak PII. It is also an opportunity to improve the security tooling in your organization.</p>

<h2 id="step-5-build">Step 5. Build</h2>

<p>At this point, we’d made all the decisions and were ready to build our easy-to-deploy and manage Staging and Production Spinnaker to manage Cash App services. The major unknowns we were dealing with at the beginning when determining what tool to use were:</p>
<ul>
  <li>Secured interaction with our clusters</li>
  <li>Integration with our security model</li>
  <li>Secured communication between Spinnaker and Cash App services</li>
</ul>

<p>These points were answered, but building still took time. We naturally came across issues when integrating within a large system. We didn’t want to compromise on code quality, or the additional security tooling we built for secure Spinnaker communication, and broke things down into subproblems to ensure all components worked as expected.</p>

<h2 id="step-6-early-adopters">Step 6. Early Adopters</h2>

<p>We finished building and were ready to start testing real world scenarios. We sent out a survey for early adopters and received a response from ~20 teams. We tested by doing the following:</p>
<ul>
  <li>Onboarding 4 of our own test services</li>
  <li>Onboarding 4 Platform test services</li>
  <li>Onboarding ~8 early adopters</li>
  <li>Using what we learned we created an onboarding guide, making Continuous Delivery self-serve</li>
  <li>Monitored the onboarding of the remaining services to see what issues people ran into</li>
  <li>Setup interviews with 5 teams to learn about difficulties they encountered, or improvements they were interested in after a few weeks of use</li>
</ul>

<p>Each step allowed us to understand what kind of information would be useful for onboarding, and the interviews helped to iteratively improve the guide. The interviews and observations of how teams were using our features also informed our future planning.</p>

<h2 id="step-7-general-availability">Step 7. General Availability</h2>

<p>We finished testing with our early adopters and ironing out the kinks. We stress-tested the system with our Production clusters and launched our Spinnaker Platform to general availability.</p>

<p>This is when we celebrated!</p>

<h2 id="future-work">Future Work</h2>

<p>Launching is only the beginning. Rolling out an org-wide process/tool change takes time, but there is more to do still — new features to work on, feedback from teams on how the tool is performing or meeting their needs, and further expansion. The journey is not yet over, but a major hurdle is overcome, and the path is paved to work on more exciting features in the future.</p>]]></content><author><name>Saman Alvi</name></author><category term="[&quot;spinnaker&quot;, &quot;process&quot;, &quot;platform&quot;]" /><summary type="html"><![CDATA[Cash App has grown into a complex engineering organization. Our team of hundreds of engineers deploys dozens of services to AWS multiple times a day. Our teams span several time zones and locations, and each team has its own availability and velocity requirements.]]></summary></entry><entry><title type="html">Investigating Crashes with Aardvark</title><link href="https://cashapp.github.io/2021-03-16/investigating-crashes-with-aardvark" rel="alternate" type="text/html" title="Investigating Crashes with Aardvark" /><published>2021-03-16T00:00:00+00:00</published><updated>2021-03-16T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-03-16/investigating-crashes-with-aardvark</id><content type="html" xml:base="https://cashapp.github.io/2021-03-16/investigating-crashes-with-aardvark"><![CDATA[<p>Crashes can be one of the most straightforward types of issues to debug. There are many different categories of crashes that can happen on iOS - everything from bad memory access to preconditions that crash when the app enters an unexpected state - with a large variety of causes. But one thing these crashes all have in common is that they come with debugging data in the form of a crash report, which contains information about the cause of the crash and a backtrace of each thread in the process.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Incident Identifier: 5C6A7EC9-E763-4BE6-80B9-A0EC949A2B62
Hardware Model:      iPhone10,6
Process:         AardvarkCrashRep [6720]
Path:            /private/var/containers/Bundle/Application/F2DA0515-B31F-478A-BB77-DDB18765FFD0/AardvarkCrashReporterDemo.app/AardvarkCrashReporterDemo
Identifier:      com.squareup.AardvarkCrashReporterDemo
Version:         1.0 (1)
Code Type:       ARM-64
Parent Process:   [1]

Date/Time:       2021-03-02 04:21:28 +0000
OS Version:      iPhone OS 13.3 (17C54)
Report Version:  104

Exception Type:  SIGTRAP
Exception Codes: #0 at 0x1a305f168
Crashed Thread:  0

Thread 0 Crashed:
0   libswiftCore.dylib                  0x00000001a305f168 &lt;redacted&gt; + 380
1   libswiftCore.dylib                  0x00000001a2e62210 $ss12_ArrayBufferV37_checkInoutAndNativeTypeCheckedBounds_03wasfgH0ySi_SbtF + 196
2   libswiftCore.dylib                  0x00000001a2e64fa4 $sSayxSicig + 84
3   AardvarkCrashReporterDemo           0x0000000104f45978 $s25AardvarkCrashReporterDemo14ViewControllerC07triggerB033_C7A9EC1A6962A633C365EE02C1845969LLyyF + 72
4   AardvarkCrashReporterDemo           0x0000000104f459ac $s25AardvarkCrashReporterDemo14ViewControllerC07triggerB033_C7A9EC1A6962A633C365EE02C1845969LLyyFTo + 32
5   UIKitCore                           0x0000000199c32e50 &lt;redacted&gt; + 96
...
19  CoreFoundation                      0x0000000195b3723c &lt;redacted&gt; + 1080
20  CoreFoundation                      0x0000000195b36adc CFRunLoopRunSpecific + 464
21  GraphicsServices                    0x000000019fabc328 GSEventRunModal + 104
22  UIKitCore                           0x0000000199c31ae0 UIApplicationMain + 1936
23  AardvarkCrashReporterDemo           0x0000000104f4b288 main + 88
24  libdyld.dylib                       0x00000001959c0360 &lt;redacted&gt; + 4

Thread 1:
0   libsystem_kernel.dylib              0x00000001959b6a7c __workq_kernreturn + 8
1   libsystem_pthread.dylib             0x00000001958dd760 start_wqthread + 8

...
</code></pre></div></div>
<p><em>A sample of a crash report on iOS. The report has been truncated for readability.</em></p>

<p>Many teams send these crash reports to a central source to be aggregated, making it easier to triage reports and determine which crashes are happening the most often. They can also be <a href="https://developer.apple.com/documentation/xcode/diagnosing_issues_using_crash_reports_and_device_logs/acquiring_crash_reports_and_diagnostic_logs">collected from devices directly</a>, making it easier to debug crashes that you experience while testing the app locally, although sending the reports to a crash reporting service is often preferred since it can happen automatically on the next app launch.</p>

<p>Apple provides <a href="https://developer.apple.com/documentation/xcode/diagnosing_issues_using_crash_reports_and_device_logs/identifying_the_cause_of_common_crashes">extensive documentation</a> for how to interpret these crash reports. For many types of crashes, these reports can clearly highlight the problem area and guide engineers to a solution quickly. But sometimes the crash report alone doesn’t provide enough context to reproduce and debug the issue locally.</p>

<h2 id="connecting-crash-reports-and-bug-reports">Connecting Crash Reports and Bug Reports</h2>

<p>When crash reports aren’t enough, bug reports can save the day. A bug report that contains detailed reproduction steps and other information about the application state can make it clear how the app got to the point where it crashed. The combination of the bug report and the crash report paints a more complete picture of what happened.</p>

<p><a href="https://github.com/square/Aardvark">Aardvark</a>, our open source bug reporting framework for iOS, makes it dead simple to file actionable bug reports. By attaching additional metadata such as a device identifier and timestamps to both reports (which is supported by Aardvark and many popular crash reporting services), you can search for the crash report associated with a given bug report.</p>

<p>Sometimes, though, you get a bug report saying that the app crashed, but it’s unclear how these bug reports map to the crash reports you’ve collected. This is especially common if the crash isn’t a result of a direct user action, such as when the crash is caused by code executing on a background thread. You can often determine which crash report corresponds to the bug report through process of elimination - reading through each crash report and determining whether the root cause could be related to the described bug - but this can be a very time consuming process if there are a large number of crashes.</p>

<p>Now there’s a better way.</p>

<h2 id="bringing-crash-reports-into-aardvark">Bringing Crash Reports into Aardvark</h2>

<p>Introducing <a href="https://github.com/cashapp/AardvarkCrashReport">AardvarkCrashReport</a>, a new open source framework that provides utilities to embed a crash report directly in your Aardvark bug report. The time of manually tracking down the corresponding crash report is over. AardvarkCrashReport makes it straightforward to add a check when your app launches to determine whether the previous session terminated in a crash and, if so, prompt the user to file a bug report.</p>

<p><img src="/assets/2021-03/AardvarkCrashDetected.png" alt="" />
<em>A screenshot from an internal Cash App build showing a prompt to file a bug report for a detected crash.</em></p>

<p>AardvarkCrashReport also supports attaching live reports to aid in debugging other types of bugs, for example those related to performance issues caused by background threads. Now you’ll have access to even more information about the state of your app when the bug report was filed, making the already-powerful bug reports provided by Aardvark even easier to investigate.</p>

<p>If you already use Aardvark in your app, it’s simple to integrate AardvarkCrashReport’s lightweight utilities into your existing bug reporting setup. And if you don’t already have a bug reporting tool built into your app, there’s never been a better time to adopt Aardvark.</p>]]></content><author><name>Nick Entin</name></author><category term="[&quot;ios&quot;]" /><summary type="html"><![CDATA[Crashes can be one of the most straightforward types of issues to debug. There are many different categories of crashes that can happen on iOS - everything from bad memory access to preconditions that crash when the app enters an unexpected state - with a large variety of causes. But one thing these crashes all have in common is that they come with debugging data in the form of a crash report, which contains information about the cause of the crash and a backtrace of each thread in the process.]]></summary></entry><entry><title type="html">Debugging UI Issues with AardvarkReveal</title><link href="https://cashapp.github.io/2021-03-03/debugging-ui-issues-with-aardvarkreveal" rel="alternate" type="text/html" title="Debugging UI Issues with AardvarkReveal" /><published>2021-03-03T00:00:00+00:00</published><updated>2021-03-03T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-03-03/debugging-ui-issues-with-aardvarkreveal</id><content type="html" xml:base="https://cashapp.github.io/2021-03-03/debugging-ui-issues-with-aardvarkreveal"><![CDATA[<p>Reports of UI issues can be difficult to debug when the issue is not easily reproducible. Bug reports created through <a href="https://github.com/square/Aardvark">Aardvark</a> include a screenshot of the app at the time the report was filed by default. This screenshot shows engineers exactly what the user was seeing when they filed the report, making it much easier to understand what behavior they encountered. But this still may not be enough to diagnose <em>why</em> they saw that behavior.</p>

<p>For example, say you get a report of an image view not being visible. From the screenshot, you can confirm that nothing is showing up where the image view should be. But why was this the case? Was the view not installed? Does it have an alpha of zero? Is it hidden? Did it fail to load the image? There are many potential root causes for this problem and unfortunately it’s impossible to tell which it is from the screenshot alone.</p>

<p>When debugging the app locally in Xcode, engineers can use the view debugger to visually inspect the view hierarchy and look at properties of the views and layers on the screen. <a href="https://revealapp.com">Reveal</a> takes this concept even further, providing invaluable insights into your app’s UI, and the Reveal framework can even be embedded in an app to do live inspection over the network. The internal version of Cash App that the team uses for our daily testing ships with Reveal, giving engineers the ability to inspect the app any time an issue arises. If only there were a way to see this same information in our bug reports…</p>

<h2 id="aardvark--reveal--aardvarkreveal">Aardvark + Reveal = AardvarkReveal</h2>

<p>Say goodbye to difficult-to-diagnose UI issues and hello to <a href="https://github.com/cashapp/AardvarkReveal">AardvarkReveal</a>, a new open source framework that brings the power of Reveal to Aardvark. The framework provides a simple utility for generating an Aardvark bug report attachment containing a Reveal snapshot, which can later be opened in the Reveal app to see the exact state of your app’s UI when the bug was filed.</p>

<p><img src="/assets/2021-03/RevealScreenshot.png" alt="" />
<em>A screenshot of the Reveal app showing a snapshot of a receipt in Cash App captured using AardvarkReveal.</em></p>

<p>AardvarkReveal is built on top of Aardvark 4.0, which takes Aardvark’s already-powerful bug reporting capabilities and <a href="https://cashapp.github.io/2021-02-24/taking-bug-reporting-further">makes them even more flexible</a>. Aardvark’s one-line setup makes it dead simple to set up a basic bug reporter, and the components added in the latest release give you the power to take your bug reporter to the next level, fitting it to your app’s exact needs. But whether you’re using the basic email-based bug reporter or something totally custom, AardvarkReveal will fit right in.</p>

<p><img src="/assets/2021-03/AardvarkRevealFlow.png" alt="" />
<em>Screenshots of the custom bug reporting flow used in internal Cash App builds to send a bug report email with the Reveal bundle as an attachment.</em></p>

<p>We’ve created a <a href="https://github.com/cashapp/AardvarkReveal#getting-started">step-by-step guide</a> to getting AardvarkReveal set up in your app. It’s never been easier to get high quality, information rich data for debugging UI issues in the wild.</p>]]></content><author><name>Nick Entin</name></author><category term="[&quot;ios&quot;]" /><summary type="html"><![CDATA[Reports of UI issues can be difficult to debug when the issue is not easily reproducible. Bug reports created through Aardvark include a screenshot of the app at the time the report was filed by default. This screenshot shows engineers exactly what the user was seeing when they filed the report, making it much easier to understand what behavior they encountered. But this still may not be enough to diagnose why they saw that behavior.]]></summary></entry><entry><title type="html">Taking Bug Reporting Further</title><link href="https://cashapp.github.io/2021-02-24/taking-bug-reporting-further" rel="alternate" type="text/html" title="Taking Bug Reporting Further" /><published>2021-02-24T00:00:00+00:00</published><updated>2021-02-24T00:00:00+00:00</updated><id>https://cashapp.github.io/2021-02-24/taking-bug-reporting-further</id><content type="html" xml:base="https://cashapp.github.io/2021-02-24/taking-bug-reporting-further"><![CDATA[<p>This week we released a new major update for <a href="https://github.com/square/Aardvark">Aardvark</a>, our open source bug reporting framework for iOS. Aardvark makes it dead simple to create actionable bug reports from right inside your app. This latest release makes it even easier to build out powerful bug reporting flows that capture a greater amount of information than ever before and provide increased flexibility in how your bug reports are submitted.</p>

<h2 id="modularizing-the-components">Modularizing the Components</h2>

<p>In 2016, Aardvark was split into two frameworks: Aardvark and CoreAardvark. The basic mechanisms for creating, distributing, and storing log messages lived in CoreAardvark, which could be used across the application itself and app extensions, while everything else lived in Aardvark. As the amount and variety of functionality grew over time, this two framework system led to a lot of complexity within Aardvark and an unclear boundary for exactly what the framework did.</p>

<p>In the latest release, we split the components down even further, bringing the project to a total of four frameworks. <strong>CoreAardvark</strong> still contains the basic structures for logging that are safe to be used in app extensions. <strong>Aardvark</strong> now contains only the core components for building a bug report, as well as some extensions for logging around UIKit. The email-based bug reporting flow has been moved to a new framework, <strong>AardvarkMailUI</strong>. And the in-app log viewing functionality has been moved to another new framework, <strong>AardvarkLoggingUI</strong>.</p>

<p>All of the existing functionality is still available through these new frameworks, with some simple changes to the dependency structure and a few small API updates. New areas of functionality can now more easily be added in new frameworks as extensions to these core frameworks.</p>

<h2 id="building-a-bug-reporting-platform">Building a Bug Reporting Platform</h2>

<p>Breaking the bug reporting functionality into smaller, more modular components makes it easier to build out customized bug reporting flows on top of the Aardvark platform while still providing simple-to-use, high level abstractions when desired. Currently, bug reports are filed by populating a mail composer with details about the device and app install. This email also includes attachments representing the app logs, a screenshot, and other custom attachments.</p>

<p>In internal Cash App builds, we’ve taken this functionality and extended it in a number of ways. In addition to the attachments automatically included in the bug report, we generate attachments containing data specific to the state of our app, such as a description of the feature flags that have been synced to the device and a compressed copy of our Core Data store. We also have a custom bug reporting prompt that shows a list of all the potential attachments and lets you select which ones to include when the bug report is filed. The default selection states in this list dynamically change based on which screen in the app you trigger the bug reporting prompt from. This is made possible by hooking into various delegate calls in the Aardvark email bug reporter.</p>

<p><img src="/assets/2021-02/AardvarkFlow.png" alt="" />
<em>Screenshots of Cash App’s bug reporting flow in internal builds. The screen on the left shows the initial prompt that lets the reporter input a description of the bug and select which attachments to include. The screen on the right shows the pre-populated email that contains the details of the bug report.</em></p>

<p>The latest release of Aardvark pulls much of this functionality out of the default email-based bug reporter into reusable utilities, making it simpler to write a completely custom bug reporter implementation that submits the report directly to an API instead of via email, or triggers the bug report via a mechanism other than a gesture, or collects the information for the bug report in a totally different way unique to your app.</p>

<h2 id="the-power-of-attachments">The Power of Attachments</h2>

<p>The concept of email attachments has been extended to generalized “bug report attachments” that can be generated and attached to any bug report, regardless of how it’s submitted. The code for generating the default attachments has been extracted into reusable, self-contained utilities. This makes it easy to build up a custom bug reporter from these components that fits the exact needs of your app.</p>

<p>Don’t worry, the email-based bug reporter isn’t going anywhere. You can keep using the standard flow with its simple one line setup and still optionally hook into the existing customization points to add attachments as needed.</p>

<p>We’re working on some exciting new attachment generators that will be available soon. And we can’t wait to see what new types of attachments you create to enhance the bug reports in your apps!</p>

<h2 id="adopting-the-latest-changes">Adopting the Latest Changes</h2>

<p>If you already use Aardvark in your app, we’ve created a simple <a href="https://github.com/square/Aardvark/blob/master/Documentation/Migration.md#migrating-from-aardvark-3x--coreaardvark-2x">migration guide</a> to help you update your dependency to the latest version. In most cases, there will be little to no code change required.</p>

<p>And if you don’t already have a bug reporting tool built into your app, there’s never been a better time to adopt <a href="https://github.com/square/Aardvark">Aardvark</a>.</p>]]></content><author><name>Nick Entin</name></author><category term="[&quot;ios&quot;]" /><summary type="html"><![CDATA[This week we released a new major update for Aardvark, our open source bug reporting framework for iOS. Aardvark makes it dead simple to create actionable bug reports from right inside your app. This latest release makes it even easier to build out powerful bug reporting flows that capture a greater amount of information than ever before and provide increased flexibility in how your bug reports are submitted.]]></summary></entry><entry><title type="html">Reverting Commits in Client Apps</title><link href="https://cashapp.github.io/2020-12-09/reverting-commits-in-client-apps" rel="alternate" type="text/html" title="Reverting Commits in Client Apps" /><published>2020-12-09T00:00:00+00:00</published><updated>2020-12-09T00:00:00+00:00</updated><id>https://cashapp.github.io/2020-12-09/reverting-commits-in-client-apps</id><content type="html" xml:base="https://cashapp.github.io/2020-12-09/reverting-commits-in-client-apps"><![CDATA[<p>When faced with a critical bug in a client app that’s already shipped to customers, one of the key first steps in the debugging process is finding the earliest commit in which the bug is present. Tools like <code class="language-plaintext highlighter-rouge">git bisect</code> can make finding this commit much easier. But what do you do once you’ve found the offending change?</p>

<p>The obvious answer is to revert the commit, verify that the bug is resolved, and ship a new version of the app. Sometimes this is all it takes - you found the commit that introduced the bug and removing it took the bug away. But sometimes reverting a commit can cause even bigger problems. <strong>Rolling back code without understanding <em>why</em> it fixes the problem can be dangerous.</strong></p>

<p>Discovering that the reversion resolved the issue is only one piece of understanding the effects of reverting that commit. There are multiple potential results when you roll back a commit, such as:</p>

<ul>
  <li>It fixes the bug you were seeing and changes nothing else. This is the ideal result. In practice, I’ve found that this rarely happens, at least organically, although you may see it around things like refactors.</li>
  <li>It fixes the bug you were seeing, but also reverts other changes that you would like to have kept. This is still an acceptable trade off in many cases, since the urgency of fixing the bug takes precedence. For example, we recently found a crash that was introduced by a purely cosmetic change to a screen. Reverting this was an easy decision, since the negative effects of a crash greatly outweighed the benefits of those minor visual changes. When it involves more functional differences, the decision is a bit tougher, but the same trade offs can be weighed.</li>
  <li>It looks like it fixes the bug, but actually it moves the issue to somewhere else. Maybe that issue is still mostly mitigated and it’s an improvement. Or maybe it’s now somewhere where it’s not easily detectable, but will still impact your customers. Either way, you’ve now lost time on pushing a new release that hasn’t really resolved the underlying issue.</li>
  <li>It fixes the bug you were seeing, but it causes a different (and maybe even significantly worse) bug somewhere else.</li>
</ul>

<p>That last case is obviously quite bad. But how can this happen? After all, you’re reverting back to code that you’ve already shipped to your customers in the past, right?</p>

<p>Unfortunately, that’s not entirely true. Yes, the specific lines of code you’re reverting to have shipped before, but in many cases other parts of the app it interacts with will have changed since then. <strong>Identical code is only guaranteed to behave the same if all of the code around it and the data it processes is also the same.</strong></p>

<p>There’s an important distinction here that comes up a lot when talking with our friends on the server side of things. Client rollbacks typically involve reverting a commit, while server rollbacks typically involve reverting <em>to</em> a commit. Reverting to a commit allows the project to be redeployed with all of the code in the same state it was when that (hopefully stable) commit was originally shipped. There’s still a danger of the data being different, but this makes the risk analysis much easier.</p>

<p>So what questions can you ask to determine whether it’s safe to revert a commit? Here’s some to get started:</p>

<ul>
  <li><strong>How recent is the commit?</strong> Is it the last commit in the project before shipping the app? Probably safe to revert. Is it a commit from a few months ago that’s had a lot of commits on top of it since then? Maybe not so safe - other code built on top of it may be depending on that new behavior.</li>
  <li><strong>What part of the app does the code affect?</strong> Is the code entirely in a top level view controller or some other object that’s very self-contained? Probably safe to revert. Are there changes in a more infrastructural layer of the app that lots of other code calls into? Maybe not so safe - code in other parts of the app may be depending on that new behavior.</li>
  <li><strong>Does the code affect data stored on device?</strong> Does the change affect only the presentation of the data and not the data itself? Probably safe to revert. Does it affect the data stored on device (e.g. Core Data or keychain migrations)? Maybe not so safe - the customer’s data will now be in a new state that your old code may not handle properly.</li>
  <li><strong>Does the code interact with a server (even indirectly)?</strong> Was the change made in coordination with a change on the server side? That server change probably expects the new client behavior. Was the change previously gated behind a feature flag (or combination of flags)? The state of those flags may have changed since this code last shipped.</li>
</ul>

<p>This is not meant to be a complete checklist, but it illustrates some of the ways that reverting a commit can have unintended consequences. When faced  with a crash and a potential commit to revert, your job is to perform the risk analysis to determine how likely you are to hit any of those consequences. When I’m unsure of the risk to revert, I find it best to think about a revert commit as if it were a newly written fix. <strong>Understanding the change you’re reverting is just as important as understanding new code you write.</strong></p>

<p>With the time pressure of a crash in the wild, it can feel like a waste of time to do this extra analysis before shipping a potential fix. There’s a saying that I think applies really well to this situation though: slow is smooth, smooth is fast. Iterating on mobile releases can be a somewhat slow process given the delays caused by the app review process. Taking the time to increase confidence in your first fix can make the entire process faster. If your team has the resources available, it can be helpful to prepare a new version and submit it for review, but wait to start rolling out to customers until you’re confident in the fix.</p>

<p>I realize this probably paints a fairly bleak picture of the debugging process. Unfortunately there is no quick and easy answer for what to do when you have a serious bug out in the wild that will work for all situations. Reverting is a great solution when it works, but needs to be considered based on the current situation. The good news is that there are steps you can take to increase the likelihood of being able to safely revert changes in the future. Things like:</p>

<ul>
  <li><strong>Keep changes small and contained.</strong> In general, smaller, more targeted changes are easier to understand, and therefore make it easier to understand the impact of reverting them.</li>
  <li><strong>Document why changes are being made.</strong> It’s important to be able to quickly gather context around the change in question in order to understand what reverting it will do. My recommendation is to document this context in commit messages, since this is the first thing you’ll see while tracking down the bugs, but linking out to any form of documentation is great.</li>
  <li><strong>When working on a risky change, plan ahead for a potential rollback.</strong> Hope for the best, plan for the worst. Spending a bit of time earlier in the process to think through how your changes could be undone can save time and stress later when there’s pressure to get a bug fixed ASAP. For example, you might be able to break out a small commit at the end of landing a new feature that can quickly be reverted to disable that feature.</li>
</ul>

<p>Reverting an old commit is a great tool in the bug-fighting toolbox, but it isn’t always the right tool for the job. It’s important to understand when it’s safe to revert a commit, so you can keep the option available while avoiding a potential disaster.</p>]]></content><author><name>Nick Entin</name></author><category term="[&quot;ios&quot;, &quot;android&quot;]" /><summary type="html"><![CDATA[When faced with a critical bug in a client app that’s already shipped to customers, one of the key first steps in the debugging process is finding the earliest commit in which the bug is present. Tools like git bisect can make finding this commit much easier. But what do you do once you’ve found the offending change?]]></summary></entry></feed>