What I’m Looking for from Apple Tomorrow

Like everyone else, I’m curious to see if the leaks have nailed the updated iPhones and what the wearable gadget looks like, and what new capabilities are in both that we haven’t expected. I’m also pretty sure that this won’t be the most important info from tomorrow’s announcements.

I always go back to the original iPhone announcement when I try and think about evaluating what matters from these sorts of product announcements. By far, the most important details of the new iPhone were mobile Safari coupled with an unlimited data plan at a reasonable-ish price. We got a new capability (real Internet on a mobile device!) with the means to use it without friction.

So, tomorrow, if Apple announces a mobile payment solution, for example, the key will be scoping the limitations around its use1. If it’s like Google Wallet (“hey, we’ll figure out where you can use it… some day”), then it’s obviously useless. Apple solves business problems when they ship features.

Also worth linking to my earlier post about why I hate the smart watch rumors I’ve heard so far.

Can’t wait for the announcements tomorrow.


  1. And there will be limitations. The original iPhone only had 2G, only ran on AT&T, and had limited enterprise integration. Ultimately, those didn’t matter for adoption in the first year, and were soon remedied in subsequent iterations. So, a lot of tomorrow will be figuring out if the limitations matter, and why. So far, Apple has a good track with this, especially for their highest priority project at any moment.

SSL & other notes

Just a quick note that Fatmixx is now https only – please let me know if you see any weirdness or browser complaints. I’m working on cleaning up loose ends that I know about (right now, the photos are borked because I’m working on my service that aggregates them, for example).

Speaking of that service, I’m keeping announcements and details about its ongoing maintenance over on the Forche blog. I’ll probably retweet those posts into my main Twitter account, but from here on out I probably won’t mention it much here.

Changes a comin’

For those of you that pay attention to these sorts of things, Proxigram has had some bug fixes and will be getting an overhaul soon. This blog is also undergoing some much needed maintenance and improvements.

At the top of my priority list right now is getting SSL setup everywhere and consolidating some of my hosting. Shouldn’t see too much on the user side, except for the switch to https when I finally figure out how to do that with my current web host.

Sad face – blog hacked

So, it looks like this site and the other blogs I host on this server were defaced earlier today. I’ve repaired most things I can check and have engaged the support team at my host to help diagnose the attack vector and other details.

I’m posting this publicly, though, because I know many of my friends (and former contributors to this site) have accounts on FM, so I want to make sure you’re aware that it’s possible that the passwords stored in this WordPress install might be compromised.

I’m also tempted to just blow away the blog and start over again elsewhere… it’s safer, and really, this blog could use a reset button.

Need Mavericks troubleshooting ideas…

(The post below lays out a system crash that had been plaguing my home system for a few months now. The updates are at the top – think of it like an email thread. Start from the bottom if you want the whole saga)

Update 5: OK, bug report submitted to Apple along with a consistent test case. The issue does really seem like a memory leak in the Spotlight plugin for emails, but I’ll leave it to the experts to sort out. It’s Radar rdar://problem/15695276 and I’ve submitted a copy to OpenRadar here (sans the email file, since it has real email addresses in there).

I went through and deleted every email larger than ~27MB, then turned on Spotlight indexing for mail. After that finished, I turned on Time Machine again. I haven’t seen the memory spike at all. So, this seems like the culprit.

On a related note, as I was submitting the bug report to Apple, I copied the email file to my MacBook. Immediately, it started feeling sluggish and stuttering. Looked at Activity Monitor and, sure enough, memory was going absolutely nuts. The MBP has twice as much RAM as the iMac, though, so I think it’s been able to recover when this happens (though it has locked up a few times that I can remember… probably because of this).

I’ll update if/when Apple confirms anything.

Update 4: no luck – crashed again after an hour or so. I found a bunch of other large emails still on the disk, so I think I need to clean them up. Or, I’m just wrong. Either way, more debugging later this week.

Update 3: Solved, maybe! So, I was able to narrow this down to files in the mail folder. I happened to inspect the mdworker processes in Activity Monitor and saw they were always in .emlx files around when the memory would spike. So, taking that as a clue, I told Spotlight to ignore that folder under the Spotlight Privacy tab and suddenly, machine stayed up. But… I also had to shut down Time Machine because that somehow uses mdworker or caused it to hit those folders, leading to another crash.

So, next step was to try and log what files mdworker was accessing. There’s probably a more elegant way to do this, but I ended up using fs_usage and then opensnoop, which are both part of OS X. They both let you see what files a process is interacting with while the process is running using DTrace hooks. The final command line was opensnoop -a -n mdworker | tee mdworker.log.

I then unblocked the Mail folder from the privacy settings and let Spotlight go (left Time Machine off for the first run). I let the machine crash a few times. After a few restarts, it was clear that largest mdworker processes last touched large emails that were in partial emlx (.partial.emlx) files. I manually ran mdimport against some representative files (mdimport -d4 /path/to/file) and was able to recreate the near 5GB kernel_task behavior. One email (~30MB on disk), in particular, added 7-8GB of SWAP space. It was crazy.

So, went and disabled TM & Spotlight again, went into Mail.app and tried to delete the files and kicked off spotlight again. It all worked. Time Machine just finished, too. I think this may be sorted out. Fingers crossed – will wait a few days before declaring victory.

So, the only bummer is that in my zeal to see if those emails were the issue, I deleted them before backing them up somewhere. So… no test case to send off to Apple. These sorts of emails show up now and again for me (they’re basically digests of an attachment heavy PR mailing list), so I will probably have another sample case soon.

Update 2: crashed again. FML. I posted a screenshot of the Activity Monitor at time of death: https://twitter.com/sujal/status/410482157644423168

Update: So, this last reboot, mdworker was still running, but memory was fine. Then, Time Machine kicked on and started prepping a backup. That’s when memory usage spiked and memory pressure went red. I killed the TM backup, memory returned to normal, but then a few moments later, went crazy again. Hmm – it looks like at least one mdworker is indexing Mail right now… wonder if this is a variation of the Gmail thing in Mavericks?


I’m hoping the Mac mavens among you can help me find some ideas on how to debug an issue I’m seeing now with both of my Macs running Mavericks. I’m going to file a bug report with Apple soon, but based on history, that will take a while and I really can’t deal with this for much longer. I may just downgrade.

Short summary:

  • after some amount of time, measuring in minutes to a few hours, my iMac 27″ (from 2010) will randomly freeze, hard. Tapping on the Magic Trackpad won’t do anything, hitting a key on the keyboard will sometimes get the backlight to go on, but no screensaver will be visible. Just a dark screen. Only way to recover is to reboot.

  • Disk Utility & DiskWarrior say everything is fine with the drives

  • memtest passed when I ran off the recovery partition, but running it in my normal logged in state, the whole memory pressure red/swap going nuts thing happened and the system froze.

  • the most consistent symptoms I can see pre-crash, based on logs and live monitoring using Activity Monitor are the following. This is the situation just before it crashes:
    — Activity Monitor shows memory pressure is high
    — kernel_task memory usage is listed at 4.68GB (or more, but in that ballpark)
    — mdworker has 3-4 processes running, each listed at ~500MB

So, why would mdworker make kernel_task use so much RAM?

(Also, I’ve tried resetting my spotlight cache, removing old unused Spotlight plugins… no luck)

Other observations:

  • I tried turning off Time Machine, which seemed to help. My current theory is that when this crash happens, my computer is in the high memory_pressure state, caused my mdworker, and then Time Machine kicks in trying to backup and the world just stops.

I’m trying to catch that, but I’m busy enough that I doubt I will catch it happening…

Any ideas on where to look next or something to try?

(iMac has 8GB of RAM, but today my MacBook with 16GB of RAM just exhibited the same symptoms, and has been less stable than I’d like with Mavericks… I’m wondering if it’s just more stable because it has more RAM…)

To clarify – Beastie Boys, GoldieBlox

Earlier tonight, I shared an article on Salon about the Beastie Boys/GoldieBlox controversy. I shared it because it was the first article I personally read that pointed out that GoldieBlox sued the Beastie Boys, not the other way around.

This prompted a few people to take me to task about “defending” the Beastie Boys. It all went downhill from there to the order of something like 60 tweets. I’m not even kidding about that number.

So, here’s a really short, you-don’t-need-to-reply explanation of what I was trying to say in a space where I’m not constrained by 140 characters and Twitter’s intrinsic “reply point-by-point” structure.

Here’s where I’m coming from: my timeline on Facebook, primarily, but also on Twitter, was filled with people admonishing the Beastie Boys for getting litigious with GoldieBlox over this super positive video. Initiating legal action made the Beastie Boys the bad guys in their eyes.

All I was trying to say was that given the little we actually know about the conversation these two parties had, I’m unwilling to judge either Beastie Boys or GoldieBlox as being “censorious thugs” or “thieves” in this case. I’m not really commenting on whether GoldieBlox should’ve sued or who’s right on the merits of the legal case.

My sense is that everyone that is jumping to one side or the other has some assumptions they’re bringing to the conversation. Even EFF, which defaults to defending fair use and thus takes GoldieBlox’s side, hedges about how serious the legal threat was:

It’s unclear how strong the legal threats were — they aren’t attached to the complaint — but GoldieBlox is clearly worried not just about an infringement lawsuit, but any effort to abuse the DMCA to take down the video just as the holiday shopping season gets under way.

It doesn’t look like they have any inside information – they’re also just speculating about what motivated GoldieBlox.

That little detail – “who’s the bad guy” – is different from “who’s in the right.” Sometimes, no one is the bad guy, even if one side or the other is right. Considering the lack of a DMCA takedown, which is trivial and free and their claims that they made no demands at all, it doesn’t seem like the Beastie Boys were being too aggressive about taking this down. Guess we’ll find out more as they get to court.

Buffy

Over the past few months, I’ve watched the entire Buffy the Vampire Slayer series. It’s a fun ride, and I’m now surprised I never even considered watching it when it was on the air.

One random observation: the show is a really interesting window into how much technology shifted in the late 90s and early 2000s. The show ran from 1997-2003. As you watch the seasons, you start seeing the computers go from desktops to a nice MacBook, the characters casually mention Google as a verb, and the arrival of cell phones in the hands of (some) high schoolers. Nice window into our changing tech over those 7 years.

It’s not all about the timeline

I tweeted a little about my sense that Twitter is misunderstanding it’s core platform. It felt like an obvious-ish observation, but it looks like more people are talking about this:

The other problem is perhaps the larger one: Twitter’s aversion to risk when it comes to its marquee product, the core Twitter timeline. According to sources, some of Twitter’s top people — including a few from the company’s executive ranks — fear any major changes will meddle with the company’s revenue products. These products, in just a handful of years, took Twitter’s revenue from zero to hundreds of millions of dollars.

As one insider put it: “Twitter hasn’t been playing to win. It’s playing not to lose.”

This is the thing that bugs me the most about Twitter’s product updates these days. They are really focused on bringing experiences into the timeline. It’s almost borderline obsessive. Imagine if every Facebook app had to have it’s UI be a part of the Facebook News Feed?

Of course it makes experimentation hard, of course it makes deployment tricky, and of course it makes people in charge nervous.

I claim no brilliant insight here. It just seems like Twitter could be more interesting if they started tweaking the core assumptions about what Twitter is instead of trying to innovate inside the constraints of the timeline. This is also the big drawback about them thinking about themselves as a media property rather than a backbone service platform for thousands of products…

The timeline experience defined Twitter, but it’s starting to look like a straightjacket.

Facebook Open Graph Testing Ideas?

Anyone have any suggestions on how to test open graph actions against a production Facebook app but have it pull content from a development server?

Basically, i’m working on cleaning up some sharing on an app that we launched recently, and I want to setup the open graph types, stories, and actions in one place. I’d like to avoid having a dev app and syncing open graph types, stories, and collections between the two.

I use localtunnel to let Facebook hit my dev box, but I was having trouble getting around Facebook’s domain validation rules. It requires the URLs for the objects to map to the same domain as the website URL, so I couldn’t publish objects on http://1111.localtunnel.com/ for an app that actually lives at http://www.example.com. (I’m only doing this with test users, so it shouldn’t matter).

So, my current solution is to run a reverse proxy on a subdomain (which is allowed) that points to my tunnel back to my dev box. It works, but it seems unnecessarily complicated. I feel like I’m over-thinking this. Anyone have any better ideas? Or should i just use a development app and just deal with manually migrating open graph setup from it to the production app?

In case there’s no other idea, if anyone wants the reverse proxy setup I’m using with localtunnel, the project is up on Github. It’s got some nice bells and whistles, including custom redirect domains based on the host requested. This way, you can configure a single Heroku instance with multiple domain names and map them to individual team members. See the README at the project for details.

Expectations

It’s been just 6 years since the iPhone was unveiled. The App Store was only months old when Barack Obama was elected. We’ve done a mobile version of the 386 to x86-64 evolution in that time. In a weird coincidence, hitting the 64bit era probably means we’ll see more evolution than revolution in our phones. I guess that’s why everyone is looking for a new hardware device from Apple (or anyone) to shake things up.

I’ve been trying to wrap my head around why the tech press speculation about watches, TVs, etc bothers me, because it really does. Looking for industrial design or microprocessor updates or whatever really isn’t anything but incremental evolution. Similarly, a watch (or whatever) that does the same thing as my phone doesn’t really matter.

To understand why, just take a look at what happened to Blackberry. If you haven’t read the Globe & Mail’s history of Blackberry’s decline, you should. It’s fascinating.

There are a lot of little insights, including this one:

To Mr. Lazaridis … the iPhone was a device that broke all the rules. … Unlike the BlackBerry, the iPhone had a fully Internet-capable browser. That meant it would strain the networks of wireless companies like AT&T Inc., something those carriers hadn’t previously allowed. RIM by contrast used a rudimentary browser that limited data usage.

“I said, ‘How did they get AT&T to allow [that]?’ Mr. Lazaridis recalled in the interview at his Waterloo office. “‘It’s going to collapse the network.’ And in fact, some time later it did.”

Even beyond the Blackberry, this goes to how we as a tech observing public get distracted. The iPhone was sexy, beautiful and great industrial design. But, it was also much more than that.

Think about life in 2007, pre-iPhone. We were all clamoring for this. Every person I know that had a phone, whether it was a Blackberry or a Palm or even a candybar/flip phone with “Internet” access hated two things about their phone:

  1. They only let you browse the “internet” instead of the Internet.
  2. Our data plans were stupid expensive with stupid small caps.

Everyone  with a smartphone hated these things. Every single person with a phone that touched the “mobile internet” hated these things. These problems were obvious.

So, along comes Apple with the iPhone. It shipped with a real browser and offered an unlimited data plan (and Wi-Fi).1. We all got it immediately, because we all wanted it for so long. It was obvious:

The iPhone let me surf the web, from anywhere, more or less as if I were sitting at my desk. I actually had the Internet in my pocket, and it was amazing.

It’s the breakthrough that’s obvious in hindsight that really disrupts markets. It’s obvious in hindsight because the demand is essentially pent up behind walls that the incumbents either can’t see or are afraid to dismantle. Solve the business problem, execute, and combine them with this kind of demand and you’ll have a revolutionary product.

Contrast this with the products our tech pundits are obsessed with. I can’t see, for example, what the smart watch products solve that’s an obvious problem for a lot of people. What will it let me do that I obviously want to do, that I can’t do today?

I’m coming up empty. That’s not to say I can’t see how having a nice screen on my wrist wouldn’t be convenient, but I’ve got no frustrations that a watch-like thing would seem to fix.

That’s ultimately what we should be looking for: the limitations that annoys a lot of us that don’t necessarily involve breakthrough science to solve but where there’s a business problem in the way.

One other thought: I was looking at a list of Apple’s disruptions and spent some time thinking about how the iPad measures up against this criteria. Obvious ahead of time: our laptops were too heavy, getting too powerful for the types of things most people do at work, didn’t have the battery life we wanted, and didn’t have the Internet connectivity we wanted when traveling.

The iPad solved all of those things, killed netbooks as a thing, and brought flat rate (at the time), no contract data to a mass market product. Maybe it isn’t as purely disruptive as the iPhone, but it’s up there.


PS. Coincidently, Ben Thompson posted on a very similar idea this weekend. I like his concept of “obsoletive” vs. “disruptive.” I apologize in advance of running afoul of his definition of disruption. :)

I disagree with his point that the iPhone was not disruptive because it was obsoletive. First, that doesn’t capture the business side of the iPhone which was, IMHO, truly disruptive2. Second, the iPhone was demonstrably worse at mail and phone calls than contemporary Blackberries, Nokias, etc – 2G only data connection, no keyboard, and limited mail integration & power user functionality. In other words, “worse on the vectors” that the existing business focused on but way “better on other vectors” (yay, real Internet!).

I’d go so far to suggest that truly revolutionary products exhibit both characteristics, which is why tech writing gets so confused. I can make the same argument about iPad, I think.


  1. I still wish I knew what those conversations were like. The unlimited data thing was just the start, too. Soon after, Apple got the carriers to let folks install apps that could access the data connection on the iPhone without going through costly certification. Another under-appreciated change…

    Classic Steve, persistent until he got results, so I’ve been told.

  2. maybe it’s sustaining in Thompson’s definition? or is the business change another obsoletive move? The data plan was the same price as a Blackberry, if I remember correctly, but the phones came with no subsidies (so likely the carriers thought they could be in front of adoption). A topic for further research. I don’t think it changes my thoughts that the iPhone is both disruptive & obsoletive, at least for now.