Glad you noticed! Over the break, we migrated a number of customers from our original Mac mini servers over to much more powerful servers with a lot more I/O capacity.
We only did this migration for customers who were exclusely using the latest versions of our apps, since those versions have logic in place to make those migrations seamless. We’ll be migrating our remaining customers over to the new servers soon, but we plan to send email notices to each of those customers first since the migration won’t be quite as simple (they’ll be prompted to reenter their sync passwords for the new servers the next time they sync each device).
We very much do care, and we’ve been working on improving this both on the back end (improving our server hardware and network) and on the front end.
When we think about sync performance, there are two key metrics:
Sync duration: How long does a sync operation take from start to finish?
Sync latency: How long does it take for a change to propagate from one device to all devices, i.e. how often does the app perform a sync operation?
Load on our sync servers and network performance affect the first metric, and I think our new sync servers have mostly solved this for people who have a good connection to our servers and who have been migrated to the newer servers. If you’re seeing a single sync operation take more than 15 seconds, please let us know so we can investigate why that might be. (In my investigations to date, the most common issues have been that some customers haven’t been migrated to our new server hardware yet. But we’ve also seen some customers stuck behind a slow network link that is out of our control, in which case it would help to set up a server which is closer to them on the network.)
Improved sync duration is great, but that doesn’t help if the app isn’t actually trying to sync frequently enough—i.e., if there’s a lot of latency between when a change is made and when the app syncs. To improve sync latency in OmniFocus 2, we made it so that whenever one device running OmniFocus finishes syncing, it tells other synced devices running OmniFocus on the local network that now would be a good time for them to sync as well. This means that if I have OmniFocus open on my Mac and iPhone and I make and sync a change on one device, I’ll immediately see that change appear on the other device without having to wait for the next sync interval. (However, this only helps when OmniFocus is running. If it’s not running, it can’t receive the notification and doesn’t know that it missed a sync.)
That improvement is great, as far as it goes—but unfortunately, we introduced two problems in OmniFocus 2.4 for iPhone that made sync latency much worse. First, a little background: OmniFocus 2.3 for iPhone would sync automatically when leaving the app, and would also sync automatically when you opened the app if it had been more than an hour since it last synced. But in version 2.4, we accidentally broke both of those automatic syncs, so you could leave the app and your most recent edits would never sync—or you could come back some hours later and it wouldn’t realize that it was time to sync then either. We’ve fixed some of those issues in v2.4.1 and v2.4.2, and one more fix is coming in v2.4.3 that should get us back to where we were in v2.3.
Combined with the server side improvements, the fixes in v2.4.3 should mean that OmniFocus syncing will be better than it ever has been. But that doesn’t mean that we consider syncing to be a solved problem. We’re continuing to work on both sync duration and sync latency: we have some plans for changes to the sync format which will improve our sync duration, and we’ll be working on automatically initiating syncs more frequently to improve our sync latency. Some of these changes will be incompatible with OmniFocus 1, however, so we’ve been giving our customers time to transition from v1 to v2 on all their devices before we make those changes.
Very good to hear things are actively being worked on.
For anyone who needs an improvement right now - SwissDISK are offering free WebDAV space. http://www.swissdisk.com
I just set it up and it improves sync times significantly for me. It comes with all the concerns of a free cloud service, but it might be worth a shot if you are looking for an improvement while this issue is being sorted out at Omni.
I have to admit I remorsefully moved back to OmniFocus last night. I’ve been trying some other GTD apps (again) but nothing comes close to OF’s flexibility and feature set. For me and the way I work there simply is no real alternative out there.
To skip the sync issues I started a WebDAV service on our own web server and it performed pretty well. Because of security reasons I don’t want to have WebDAV services running on that server permanently, but it was good for testing.
@khaberz + @the: Thanks for the hints. I’ll give them a closer look if OF’s sync issues return.
@kcase: Thanks for replying. As mentioned above, I initially had a very good sync experience but it turned bad soon afterwards – for reasons I am unable to reproduce. No changes on my side and we have big pipes (50 Mbit/s down and 10 MBit/s upstream local and 42/8 on mobile).
Last night I tried OmniSync again and it was pretty good. Clients are syncing much more frequently now. I’d really like to avoid 3rd party WebDAV services and keep using OmniSync / OmniPresence. Hope the hiccups are gone now.
We all love OmniFocus and appreciate your work. Thanks :-)
Remorsefully? This is part of the growing pains when we switch to the newest version of a software package. Things 3 will probably have some growing pains of its own to experience as well.
Maybe you could implement an option for WebDAV-users to “Sync after every change”, this would help me and probably others a lot! So whenever something is changed, right away the sync is triggered…
@kcase: Thanks for detailed answer.
In order to fully understand: “this only helps when OmniFocus is running” means that OF needs to be the front application (currently visible on the home screen) on every of the iOS devices, not OF is running, but currently in the background?
Thomas
Yes, that’s correct. Unfortunately, iOS apps are not allowed to subscribe to Bonjour notifications when they’re in the background at the present time. (We’ve requested the ability for an app to ask iOS to wake it when a Bonjour notification happens, similar to the way iOS handles push and location- and time-based notifications. Perhaps in a future OS update. But in the meantime…)
We’re exploring adding a push notification service which devices could use to tell other devices when they’ve synced—which would not only let the iOS apps receive notifications when they’re in the background, it would also let them receive those notifications when they’re not connected to the same local network. (A downside to this approach is that it will require Internet access to a centralized server. We’ll probably make this optional to preserve the ability for OmniFocus to sync over private, isolated networks.)
Firing up an iPhone’s radio each time someone makes an edit will really cut into its battery life. But it’s probably worth considering as an option—I guess it’s no worse than using a web app!
@kcase: Thanks, that helps a lot to understand the current mechanism. I will try to leave my iOS devices more often with OF as the latest open application and see whether it works out.
Sounds promising. I think I am referring not only to myself, every improvement of the sync mechanism is really appreciated.
Thomas
One minor note: OmniFocus syncs fastest when all clients have recently synced. This means that when you switch sync servers (and sync up all your clients), you will get faster performance at first, degrading slightly over a few hours or days, depending on your usage. (You’d even get faster syncs by creating a new account on the same server.) In other words, whether you switch from Omni’s server to one in Germany or vice versa, you might see a temporary performance improvement. You probably need to use a server for a week or so to get a fair performance assessment.
Well, just my feedback after the server update: @kcase: I gave OF another shot this evening after returning from work after reading your update email sent this morning (ETC). Thanks for letting us know.
I am transferring projects and tasks from the past two weeks to OF now, and I can say that this time there really is a remarkable difference. It’s still not Things or Todoist fast (but OF makes up for that in other areas). My point is that sync time is now pretty good - esp considering that your server is in Seattle and I am in Copenhagen, Denmark.
So well done and thanks! I have to say that I really appreciate it since I am 100% dependent on my task management system in order for me to do my work (and keep it). I am really happy that you’ve got it sorted! Then now is also the time to get OmniOutliner - now that OF and me are best pals again :)
And this is the real problem: OmniFocus sync is VERY FINICKY. If you have 3-4 clients, you have to go to those devices and sync each and every one of them every couple days, or everything slows down.
When I was using OF, pretty much every day I would go to settings and look at the number of .zip files on iOS. And I’d try to manage this. Heck, maybe I should have made it a repeating task.
I SHOULD NEVER, EVER, HAVE TO MANAGE THE NUMBER OF ZIP FILES (OR EVEN KNOW ABOUT IT) WITH MY TASK MANAGER.
I have spent extensive time with Things, OmniFocus, and The Hit List. By the criteria of “which is the fastest and most reliable sync, without me needing to worry about it,” OmniFocus’s sync is BY FAR THE WORST. It’s an order of magnitude worse than the other options (I say this with absolutely no hyperbole).
Yes, OmniFocus is more flexible, and it’s the only one where you can choose your own sync server (if data security and privacy is an issue), but that’s not really an acceptable tradeoff for me, given how crappy the performance is. And throwing more hardware at the problem is just a band-aid (and will NEVER lead to an order-of-magnitude improvement in performance): the architecture of sync with a bunch of .zip files on a WebDAV server that have to be manually “checked off” and then merged on a client-by-client basis is architecturally crappy.
I really think the Omni folks should download and set up Things and THL and seeing what sync is like. It’s fast, it works, and it has no latency. It’s just better. There are other shortcomings to these products, but sync is not one of them. OF sync is a donkey in this syncing horse race.
Ken did outline what the plans are for sync improvement in an above post. These future improvements sound promising. He explained the challenges of changing the database format. They are exploring it further. I’d say leave them alone and let them, . They were slowed down by waiting for existing OmniFocus 1 users to upgrade to version 2. They are working on it.
Remember that OmniFocus was first based on an OmniOutliner data structure (kinkless GTD). Eventually, the database structure will be changed.
Well, aside the fact that I do like donkeys (especially in horse races) I agree insofar that I think OmniGroup should have improved their sync mechanism before rolling out OF2 – as a significant part of the latest OF1 updates.
Unfortunately they skipped that important step and hoped they would be able to improve the sync later on the fly. So they built OF2 on the same foundation with only minor changes and improvements because they wanted a smooth transition from OF1 to OF2.
They had a time frame of 12 to 14 month to release the 3 new versions one by one. During that time it was impossible to perform major changes on the sync method without leaving the customers in chaos.
The only alternative would have been to release all three versions of OF2 together on the same day – including a modern database architecture and a brand new sync mechanism. Drawback: No downgrade path and no option to use mixed versions (say, OF2 on iPhone and OF1 on Mac).
What do you think how successfull this strategy would have been for OmniGroup’s business?
If Omni had waited longer and released 3 simultaneous versions, customers would have abandoned OmniFocus a long time ago.
Damned if you. Damned if you don’t. It’s easy to play the head coach role after the game has been played.
I was reading J.D. Meier’s “Getting Results the Agile Way.” One recommendation is to release software version 1.0 and get it out the door. There is no time for perfection. Then plan for version 1.1 and incrementally improve the software. Then plan for version 1.2 and incrementally improve upon version 1.1.
I thought Omni had to keep the sync and database in place for the meantime until everything was in place.
I also appreciated at least having OmniFocus 2 iPhone which gave me an idea of what to expect in a redesigned Mac version. Then OmniFocus 2 for Mac built upon ideas that originated in the iPhone version. Then the iPad version built more upon ideas that were created in the Mac version.
Now, we’ll be getting a Universal app that will run on both the iPad and iPhone.
I think it was a smart idea to let us have growing pains as we slowly switched from the OmniFocus 1 architecture and eventually move into OmniFocus 2. Omni would’ve been blasted for “abandoning” OmniFocus 1 users too quickly if the database format and sync were changed so suddenly.
I applaud Omni for taking their lumps. I’m excited to see the Universal iOS app. That hopefully indicates a change where the database structure and sync services will change.
Just a quick update: OmniFocus for iPhone v2.4.3 is now available on the App Store, with improvements to background sync and related features:
Background Sync When Activated From a Suspended State — OmniFocus once again responds to background fetch requests after being suspended (for example due to low memory).
Sync When Returning to Foreground — When appropriate, OmniFocus once again syncs when returning to the foreground.
Sync in Nearby Perspective — Sync changes can now be incorporated while the Nearby perspective is open.
Hope you’re able to notice the improvements! If you’re still having issues with sync latency or duration, please be sure to let us know.
This has now moved beyond the exploration phase: the push notification service we’re currently building for OmniFocus will support syncing to the Omni Sync Server or your own private server.
