A Multithreading Saga, Part 2
This is the second post of a three part series. In part one, we discussed the performance problems that we found with our activity feed and some of the changes we made to the way we use dispatch queues.
Solving our problems with thread proliferation bought us some headroom to unblock the current release, so things began to return to normal. We hadn’t gotten to the root of our problems, though; most people were seeing a fairly smooth experience, but the app was still freezing up for a few of our massive canary accounts. We profiled the app again and found that there were still more threads than seemed necessary, and many of those threads were getting stuck.
A screenshot from the Xcode memory graph showing a series of JSValue instances holding references to a single JSContext, which holds onto the JSVirtualMachine.
We finished our rewrite the Tuesday following the virtual alarm bells going off, one week from when we had begun the sprint to restabilize the app. The changes were rolled out to our internal build for thorough testing and we got feedback that scrolling through the activity feed was feeling much smoother. We knew we still had work to do though.
One common call we found out of JSC was to check the current time zone. The rendering logic deals with dates in multiple places, mainly in order to format the display versions of the dates properly based on how far in the future/past they are. We searched for time zone problems related to threading and found that the DarwinNotify mechanism used for caching the time zone in memory did not work as expected after forking. This meant that JSC had to read the time zone from disk far more often than it should.
Since we don’t have control over JSC, we’re required to work around the problem by decreasing the number of date reads in the logic. Much of this work around improving the rendering script is still in progress.
What We Learned
Mitigate the risk of large changes. Any big change comes with a certain amount of risk; but using techniques like pair programming, strategic scoping, and feature flags can raise confidence in the change and increase the likelihood of a successful release.
In the conclusion to our multithreading saga, we’ll talk about some of the final targeted improvements we made, as well as take a look at the resulting metrics.