Need Mavericks troubleshooting ideas…

(The post below lays out a system crash that had been plaguing my home system for a few months now. The updates are at the top – think of it like an email thread. Start from the bottom if you want the whole saga)

Update 5: OK, bug report submitted to Apple along with a consistent test case. The issue does really seem like a memory leak in the Spotlight plugin for emails, but I’ll leave it to the experts to sort out. It’s Radar rdar://problem/15695276 and I’ve submitted a copy to OpenRadar here (sans the email file, since it has real email addresses in there).

I went through and deleted every email larger than ~27MB, then turned on Spotlight indexing for mail. After that finished, I turned on Time Machine again. I haven’t seen the memory spike at all. So, this seems like the culprit.

On a related note, as I was submitting the bug report to Apple, I copied the email file to my MacBook. Immediately, it started feeling sluggish and stuttering. Looked at Activity Monitor and, sure enough, memory was going absolutely nuts. The MBP has twice as much RAM as the iMac, though, so I think it’s been able to recover when this happens (though it has locked up a few times that I can remember… probably because of this).

I’ll update if/when Apple confirms anything.

Update 4: no luck – crashed again after an hour or so. I found a bunch of other large emails still on the disk, so I think I need to clean them up. Or, I’m just wrong. Either way, more debugging later this week.

Update 3: Solved, maybe! So, I was able to narrow this down to files in the mail folder. I happened to inspect the mdworker processes in Activity Monitor and saw they were always in .emlx files around when the memory would spike. So, taking that as a clue, I told Spotlight to ignore that folder under the Spotlight Privacy tab and suddenly, machine stayed up. But… I also had to shut down Time Machine because that somehow uses mdworker or caused it to hit those folders, leading to another crash.

So, next step was to try and log what files mdworker was accessing. There’s probably a more elegant way to do this, but I ended up using fs_usage and then opensnoop, which are both part of OS X. They both let you see what files a process is interacting with while the process is running using DTrace hooks. The final command line was opensnoop -a -n mdworker | tee mdworker.log.

I then unblocked the Mail folder from the privacy settings and let Spotlight go (left Time Machine off for the first run). I let the machine crash a few times. After a few restarts, it was clear that largest mdworker processes last touched large emails that were in partial emlx (.partial.emlx) files. I manually ran mdimport against some representative files (mdimport -d4 /path/to/file) and was able to recreate the near 5GB kernel_task behavior. One email (~30MB on disk), in particular, added 7-8GB of SWAP space. It was crazy.

So, went and disabled TM & Spotlight again, went into Mail.app and tried to delete the files and kicked off spotlight again. It all worked. Time Machine just finished, too. I think this may be sorted out. Fingers crossed – will wait a few days before declaring victory.

So, the only bummer is that in my zeal to see if those emails were the issue, I deleted them before backing them up somewhere. So… no test case to send off to Apple. These sorts of emails show up now and again for me (they’re basically digests of an attachment heavy PR mailing list), so I will probably have another sample case soon.

Update 2: crashed again. FML. I posted a screenshot of the Activity Monitor at time of death: https://twitter.com/sujal/status/410482157644423168

Update: So, this last reboot, mdworker was still running, but memory was fine. Then, Time Machine kicked on and started prepping a backup. That’s when memory usage spiked and memory pressure went red. I killed the TM backup, memory returned to normal, but then a few moments later, went crazy again. Hmm – it looks like at least one mdworker is indexing Mail right now… wonder if this is a variation of the Gmail thing in Mavericks?


I’m hoping the Mac mavens among you can help me find some ideas on how to debug an issue I’m seeing now with both of my Macs running Mavericks. I’m going to file a bug report with Apple soon, but based on history, that will take a while and I really can’t deal with this for much longer. I may just downgrade.

Short summary:

  • after some amount of time, measuring in minutes to a few hours, my iMac 27″ (from 2010) will randomly freeze, hard. Tapping on the Magic Trackpad won’t do anything, hitting a key on the keyboard will sometimes get the backlight to go on, but no screensaver will be visible. Just a dark screen. Only way to recover is to reboot.

  • Disk Utility & DiskWarrior say everything is fine with the drives

  • memtest passed when I ran off the recovery partition, but running it in my normal logged in state, the whole memory pressure red/swap going nuts thing happened and the system froze.

  • the most consistent symptoms I can see pre-crash, based on logs and live monitoring using Activity Monitor are the following. This is the situation just before it crashes:
    – Activity Monitor shows memory pressure is high
    – kernel_task memory usage is listed at 4.68GB (or more, but in that ballpark)
    – mdworker has 3-4 processes running, each listed at ~500MB

So, why would mdworker make kernel_task use so much RAM?

(Also, I’ve tried resetting my spotlight cache, removing old unused Spotlight plugins… no luck)

Other observations:

  • I tried turning off Time Machine, which seemed to help. My current theory is that when this crash happens, my computer is in the high memory_pressure state, caused my mdworker, and then Time Machine kicks in trying to backup and the world just stops.

I’m trying to catch that, but I’m busy enough that I doubt I will catch it happening…

Any ideas on where to look next or something to try?

(iMac has 8GB of RAM, but today my MacBook with 16GB of RAM just exhibited the same symptoms, and has been less stable than I’d like with Mavericks… I’m wondering if it’s just more stable because it has more RAM…)

To clarify – Beastie Boys, GoldieBlox

Earlier tonight, I shared an article on Salon about the Beastie Boys/GoldieBlox controversy. I shared it because it was the first article I personally read that pointed out that GoldieBlox sued the Beastie Boys, not the other way around.

This prompted a few people to take me to task about “defending” the Beastie Boys. It all went downhill from there to the order of something like 60 tweets. I’m not even kidding about that number.

So, here’s a really short, you-don’t-need-to-reply explanation of what I was trying to say in a space where I’m not constrained by 140 characters and Twitter’s intrinsic “reply point-by-point” structure.

Here’s where I’m coming from: my timeline on Facebook, primarily, but also on Twitter, was filled with people admonishing the Beastie Boys for getting litigious with GoldieBlox over this super positive video. Initiating legal action made the Beastie Boys the bad guys in their eyes.

All I was trying to say was that given the little we actually know about the conversation these two parties had, I’m unwilling to judge either Beastie Boys or GoldieBlox as being “censorious thugs” or “thieves” in this case. I’m not really commenting on whether GoldieBlox should’ve sued or who’s right on the merits of the legal case.

My sense is that everyone that is jumping to one side or the other has some assumptions they’re bringing to the conversation. Even EFF, which defaults to defending fair use and thus takes GoldieBlox’s side, hedges about how serious the legal threat was:

It’s unclear how strong the legal threats were — they aren’t attached to the complaint — but GoldieBlox is clearly worried not just about an infringement lawsuit, but any effort to abuse the DMCA to take down the video just as the holiday shopping season gets under way.

It doesn’t look like they have any inside information – they’re also just speculating about what motivated GoldieBlox.

That little detail – “who’s the bad guy” – is different from “who’s in the right.” Sometimes, no one is the bad guy, even if one side or the other is right. Considering the lack of a DMCA takedown, which is trivial and free and their claims that they made no demands at all, it doesn’t seem like the Beastie Boys were being too aggressive about taking this down. Guess we’ll find out more as they get to court.

Buffy

Over the past few months, I’ve watched the entire Buffy the Vampire Slayer series. It’s a fun ride, and I’m now surprised I never even considered watching it when it was on the air.

One random observation: the show is a really interesting window into how much technology shifted in the late 90s and early 2000s. The show ran from 1997-2003. As you watch the seasons, you start seeing the computers go from desktops to a nice MacBook, the characters casually mention Google as a verb, and the arrival of cell phones in the hands of (some) high schoolers. Nice window into our changing tech over those 7 years.

It’s not all about the timeline

I tweeted a little about my sense that Twitter is misunderstanding it’s core platform. It felt like an obvious-ish observation, but it looks like more people are talking about this:

The other problem is perhaps the larger one: Twitter’s aversion to risk when it comes to its marquee product, the core Twitter timeline. According to sources, some of Twitter’s top people — including a few from the company’s executive ranks — fear any major changes will meddle with the company’s revenue products. These products, in just a handful of years, took Twitter’s revenue from zero to hundreds of millions of dollars.

As one insider put it: “Twitter hasn’t been playing to win. It’s playing not to lose.”

This is the thing that bugs me the most about Twitter’s product updates these days. They are really focused on bringing experiences into the timeline. It’s almost borderline obsessive. Imagine if every Facebook app had to have it’s UI be a part of the Facebook News Feed?

Of course it makes experimentation hard, of course it makes deployment tricky, and of course it makes people in charge nervous.

I claim no brilliant insight here. It just seems like Twitter could be more interesting if they started tweaking the core assumptions about what Twitter is instead of trying to innovate inside the constraints of the timeline. This is also the big drawback about them thinking about themselves as a media property rather than a backbone service platform for thousands of products…

The timeline experience defined Twitter, but it’s starting to look like a straightjacket.

Facebook Open Graph Testing Ideas?

Anyone have any suggestions on how to test open graph actions against a production Facebook app but have it pull content from a development server?

Basically, i’m working on cleaning up some sharing on an app that we launched recently, and I want to setup the open graph types, stories, and actions in one place. I’d like to avoid having a dev app and syncing open graph types, stories, and collections between the two.

I use localtunnel to let Facebook hit my dev box, but I was having trouble getting around Facebook’s domain validation rules. It requires the URLs for the objects to map to the same domain as the website URL, so I couldn’t publish objects on http://1111.localtunnel.com/ for an app that actually lives at http://www.example.com. (I’m only doing this with test users, so it shouldn’t matter).

So, my current solution is to run a reverse proxy on a subdomain (which is allowed) that points to my tunnel back to my dev box. It works, but it seems unnecessarily complicated. I feel like I’m over-thinking this. Anyone have any better ideas? Or should i just use a development app and just deal with manually migrating open graph setup from it to the production app?

In case there’s no other idea, if anyone wants the reverse proxy setup I’m using with localtunnel, the project is up on Github. It’s got some nice bells and whistles, including custom redirect domains based on the host requested. This way, you can configure a single Heroku instance with multiple domain names and map them to individual team members. See the README at the project for details.

Expectations

It’s been just 6 years since the iPhone was unveiled. The App Store was only months old when Barack Obama was elected. We’ve done a mobile version of the 386 to x86-64 evolution in that time. In a weird coincidence, hitting the 64bit era probably means we’ll see more evolution than revolution in our phones. I guess that’s why everyone is looking for a new hardware device from Apple (or anyone) to shake things up.

I’ve been trying to wrap my head around why the tech press speculation about watches, TVs, etc bothers me, because it really does. Looking for industrial design or microprocessor updates or whatever really isn’t anything but incremental evolution. Similarly, a watch (or whatever) that does the same thing as my phone doesn’t really matter.

To understand why, just take a look at what happened to Blackberry. If you haven’t read the Globe & Mail’s history of Blackberry’s decline, you should. It’s fascinating.

There are a lot of little insights, including this one:

To Mr. Lazaridis … the iPhone was a device that broke all the rules. … Unlike the BlackBerry, the iPhone had a fully Internet-capable browser. That meant it would strain the networks of wireless companies like AT&T Inc., something those carriers hadn’t previously allowed. RIM by contrast used a rudimentary browser that limited data usage.

“I said, ‘How did they get AT&T to allow [that]?’ Mr. Lazaridis recalled in the interview at his Waterloo office. “‘It’s going to collapse the network.’ And in fact, some time later it did.”

Even beyond the Blackberry, this goes to how we as a tech observing public get distracted. The iPhone was sexy, beautiful and great industrial design. But, it was also much more than that.

Think about life in 2007, pre-iPhone. We were all clamoring for this. Every person I know that had a phone, whether it was a Blackberry or a Palm or even a candybar/flip phone with “Internet” access hated two things about their phone:

  1. They only let you browse the “internet” instead of the Internet.
  2. Our data plans were stupid expensive with stupid small caps.

Everyone  with a smartphone hated these things. Every single person with a phone that touched the “mobile internet” hated these things. These problems were obvious.

So, along comes Apple with the iPhone. It shipped with a real browser and offered an unlimited data plan (and Wi-Fi).1. We all got it immediately, because we all wanted it for so long. It was obvious:

The iPhone let me surf the web, from anywhere, more or less as if I were sitting at my desk. I actually had the Internet in my pocket, and it was amazing.

It’s the breakthrough that’s obvious in hindsight that really disrupts markets. It’s obvious in hindsight because the demand is essentially pent up behind walls that the incumbents either can’t see or are afraid to dismantle. Solve the business problem, execute, and combine them with this kind of demand and you’ll have a revolutionary product.

Contrast this with the products our tech pundits are obsessed with. I can’t see, for example, what the smart watch products solve that’s an obvious problem for a lot of people. What will it let me do that I obviously want to do, that I can’t do today?

I’m coming up empty. That’s not to say I can’t see how having a nice screen on my wrist wouldn’t be convenient, but I’ve got no frustrations that a watch-like thing would seem to fix.

That’s ultimately what we should be looking for: the limitations that annoys a lot of us that don’t necessarily involve breakthrough science to solve but where there’s a business problem in the way.

One other thought: I was looking at a list of Apple’s disruptions and spent some time thinking about how the iPad measures up against this criteria. Obvious ahead of time: our laptops were too heavy, getting too powerful for the types of things most people do at work, didn’t have the battery life we wanted, and didn’t have the Internet connectivity we wanted when traveling.

The iPad solved all of those things, killed netbooks as a thing, and brought flat rate (at the time), no contract data to a mass market product. Maybe it isn’t as purely disruptive as the iPhone, but it’s up there.


PS. Coincidently, Ben Thompson posted on a very similar idea this weekend. I like his concept of “obsoletive” vs. “disruptive.” I apologize in advance of running afoul of his definition of disruption. :)

I disagree with his point that the iPhone was not disruptive because it was obsoletive. First, that doesn’t capture the business side of the iPhone which was, IMHO, truly disruptive2. Second, the iPhone was demonstrably worse at mail and phone calls than contemporary Blackberries, Nokias, etc – 2G only data connection, no keyboard, and limited mail integration & power user functionality. In other words, “worse on the vectors” that the existing business focused on but way “better on other vectors” (yay, real Internet!).

I’d go so far to suggest that truly revolutionary products exhibit both characteristics, which is why tech writing gets so confused. I can make the same argument about iPad, I think.


  1. I still wish I knew what those conversations were like. The unlimited data thing was just the start, too. Soon after, Apple got the carriers to let folks install apps that could access the data connection on the iPhone without going through costly certification. Another under-appreciated change…

    Classic Steve, persistent until he got results, so I’ve been told.

  2. maybe it’s sustaining in Thompson’s definition? or is the business change another obsoletive move? The data plan was the same price as a Blackberry, if I remember correctly, but the phones came with no subsidies (so likely the carriers thought they could be in front of adoption). A topic for further research. I don’t think it changes my thoughts that the iPhone is both disruptive & obsoletive, at least for now.

Why do we send people to jail?

That question has been on my mind quite a bit these days. Between shows like Orange is the New Black and talk about mandatory minimums, I’ve started thinking about why jail makes sense as a punishment for non-violent offenders. For violent offenders or people that need detox or mental care, I can understand locking them up for public safety or for treatment, respectively. Anyway, I got some book suggestions from lawyer friends and will read some of them in the next few months.

That’s the context for why this program, JusticeHome, caught my eye:

Sending a mother and two children to Drew House costs $34,000, compared with $129,000 to keep her in prison for a year and to put those children in foster care, Mr. Hynes said. The stay-at-home program will cost even less, $10,000 to $15,000 per woman, per year, Ms. Lerner said.

JusticeHome will be available only to women, mostly mothers, who plead guilty to felonies in which they face a minimum of six months in prison. They will be visited several times a week by case managers from the Women’s Prison Association, receive counseling about jobs, schools and management of their homes and children. Some will be required to have treatment for drug addiction and mental illness.

With a judge’s approval, a woman’s traditional sentencing will be delayed, and if she completes the program successfully, the charges against her will be dismissed. The association anticipates that an average woman will spend six to eight months under its supervision.

Ms. Lerner acknowledged that the women’s homes and neighborhoods are often the source of the sparks or enticements that lead to drugs or crime. But she said, “Our goal is to help them change the way they function in those environments so they are able to live in a safe way.” Even if a woman goes to prison, she said, she will eventually have to cope in her home environment.

It’s an interesting experiment, one I hope works. Found it via a post about Piper Kerman’s op-ed in the Times about stopping the relocation of all the female prisoners at Danbury penitentiary to other federal facilities.

Also, if you haven’t been watching Orange is the New Black, you really should give it a shot. It handles some tough material in a way that’s approachable for those of us that don’t deal with this reality all the time. Hopefully, it opens some eyes about prison life and our penal system. IMHO, we put too many people behind bars, and too often for stupidly minor offenses. I’d rather see people make good for their crimes than spend time behind bars. I’m not sure what that means exactly, in a practical, programatic way, but that’s why I’m starting to read about this.

Also, tangentially related, if you haven’t read Sarah Stillman’s article in the New Yorker on the proliferation and abuse of civil forfeiture laws, you really, really ought to. This should be another issue that liberals, conservatives and libertarian types should be able to find some common ground on.

Yeah, I got nothing

(Note: This will likely only be of interest to my local friends and social network buddies. It’s a response to Concerning The Suburbs, Their Raison D’etre, And How It Is Reflected In Their Planned Public Spaces, prompted by a discussion on Twitter earlier today. If you’re curious, or want the context, go read that first.

So, I’ve been staring at that essay for a good 30 minutes trying to figure out what my thesis is here. So far, I got nothing. I’m struggling because there are two separate arguments being made, intertwined in ways that make them difficult to separate. Here goes my attempt to understand, then respond.)

My impression walking away from the conversation this morning was that Josh had set an unattainable standard by which to evaluate the impact of Blue Back Square on the town of West Hartford. The essay reinforces that impression.

In particular, this rankles:

Compare it to, say, Inman Square in Cambridge, Mass.: It would not do in BBS to have buses passing right through and stopping right in front of the Cheesecake Factory, if those buses ran a route that included, say, the local district court and jail, and nearby housing projects. Nor would it do to have upscale boutiques interspersed with dollar stores and appliance repair stores and bodegas, the way you might see in a secondary commercial center in Brooklyn, like 5th Avenue in Sunset Park, or even on Manhattan’s Upper East Side.

I’d quibble on the details here. I consider Blue Back to be part of West Hartford’s town center. In that scope, most of the objections of how Blue Back fosters economic exclusivity fall apart. Major bus lines run right through WH center and they go right into Hartford, past the court, even – take the 60-66. It’s not like that bus stop is prohibitively far, it’s two mini blocks east of BBS.

Businesses are not corner bodegas, but they’re mid-tier chains & indy businesses at the low end: CVS, Harry’s Pizza, Cosi, Gyro Palace, Moe’s, Subway, Ben & Jerry’s, etc. Sure, there are more upscale places, but there’s more variety than you’re allowing for.

But this is all debating the details.

If you’re arguing Blue Back could have a more economically diverse customer and visitor base, I don’t disagree with that. But I also think it’s hard to argue that it hasn’t brought in more economic diversity and created more unpredictable interaction than what was there before it. In that light, and measured against any development project in the state, I think it’s done a net positive for the area and for the town on the diversity front.

Held up to the ideal of something like Inman Square or Brooklyn, the products of organic and centuries long growth of two major American cities and their environs… well, sure, it’s not going to measure up in terms of economic diversity and retail/residential integration. But what does?

It would help to have an example of a development plan or project that provided the kind of new urbanism or whatever you’re calling this that you’re describing. I don’t know of any, especially any that have done a better job than BBS that didn’t already start in already dense construction.

And it would be nice to talk about what urban means, and what counts as “authentic urban” — I don’t agree with the narrow definition because I don’t think it’s historically valid (I also am not well educated on this topic to make the argument here… so, worthy of a conversation, I think, perhaps with others).

Why we stopped using Trello, even though we love it

I tweeted this earlier …

… which prompted a few people to ask, “Why’d you stop using Trello?”

The answer is pretty specific to us and our particular organizational inertia (such as it is for a small company like us), but here it is:

We liked Trello a lot, but we ended up switching to use Github issues. While it’s somewhat inferior to Trello, it had two features that made it compelling.

  1. We use Github for source control, and Trello really doesn’t integrate with that workflow at all. For example, we can manage Github issues from our commit messages, reference them, and comment on them in a place we already have to look.

  2. We use Campfire, and Trello didn’t have any integration with it. We could’ve built that, but Github already has it, and so laziness won the day.

In truth, Github issues is pretty nice, too, so it’s not like we’re giving up that much.

I did like Trello’s visibility and the visual metaphor. It also is a lovely app. The other great thing about it was that it was easy to throw it up on the screen and use during meetings. Github issues (or any bug tracker, really), is merely OK projected.

Anyway, all of this might be moot now. Someone did the work we didn’t want to do, and built a service that integrates Github & Trello. We may have to take a look at this. :)

Great reads in this month’s @nsfwcorp

Almost forgot to mention this: I read the latest print edition of NSFWCorp on the flight here and ended up reading cover to cover. Two great stories. The first one, “The Life and Murder of Anna Loginova“, paints a really worrisome and sad picture of life outside of Moscow in modern day Russia.

The other piece, about immigrating to Canada, is an interesting story, even though it’s unfair to Canada.

Both are going to evoke some disagreement with their conclusions or ideas. Both are worth reading.