Reading a batch of tasks taking way too long

Automation OmniFocus Automation
#1

Hey, I’m experimenting with scripting a batch of tasks in JXA and running into some big speed issues. I think this is a simple misunderstanding - I’m expecting it to work like a single SQL query that loads all objects in memory but instead it seems to do each operation separately.

Here’s a simple example - let’s get the names of all uncompleted tasks:

var tasks = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})
var totalTasks = tasks.length
for (var i = 0; i < totalTasks; i++) {
    tasks[i].name()
}

[Finished in 46.68s]

Actually getting the list of 900 tasks takes ~7 seconds - already slow - but then looping and reading basic properties takes another 40 seconds, presumably because it’s hitting the DB for each one. (Also, tasks doesn’t behave like an array - it seems to be recomputed every time it’s accessed.)

Is there any way to do this quickly - to read a batch of objects and all their properties into memory at once? Alternatively, if I’m trying to do more complex batch processing, is it possible / preferable to hit the Omnifocus DB directly?

#2

I tested on my own database — 414 total flattened tasks —, and this script is finished in 0.809s.

(() => {
	'use strict'

    const
        of = Application('OmniFocus'),
        oDoc = of.defaultDocument,
        oTasks = oDoc.flattenedTasks();

    return (
        oTasks.map(x => 'Task: ' + x.name())
    )
})();

I don’t know why is taking so much time in your case; it shouldn’t.

If you share some examples of what you are trying to do, I can give some opinions.

P.S.: OmniJS is coming to OmniFocus 3; let’s hope a good implementation.

#3

Interesting - don’t fully understand the syntax but that is indeed faster for me, from 46 to ~7 seconds for about a thousand tasks, though still not as fast as you. The issue is that if I try adding other properties to the return line - say due date or flagged or parent - the time balloons with each property added, whereas I’d like to get them all at once.

Here’s a simplified version of what I’m trying to do - take all of my available tasks, and return the most important ones according to a value score that I assign:

// How important a task is
function taskValue(task) {
	value = 1
	if (task.flagged()) { value += 1 }
	if (task.dueDate()) { value += 1 }
	if (task.note().includes("#important")) { value += 1 }
	return value
}

// Get tasks
tasks = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false, blocked: false})

// Copy tasks into the array tasksFound; not sure why this is necessary but tasks is not a sortable array
numTasks = tasks.length
tasksFound = []
for (var i = 0; i < numTasks; i++) {
	tasksFound.push(tasks[i])
}

// Sort according to value
console.log("Sorting " + numTasks + " tasks")
sortedTasks = tasksFound.sort(function(a, b) { return taskValue(b) - taskValue(a) })

// Display first 10
for (var i = 0; i < Math.min(10, sortedTasks.length); i++) {
	console.log(taskValue(sortedTasks[i]) + " - " + sortedTasks[i].name())
}

This is hellishly slow:

Sorting 207 tasks
1 - task1
1 - task2
...
[Finished in 248.231s]

If you could help me sort this out I would be eternally grateful - I’ve been struggling with this for weeks.

1 Like
#4

This is the definition of an available task that I included in the full script.

const now = new Date()

// Get available tasks
const tasks = Application('OmniFocus')
    .defaultDocument.flattenedTasks.whose(
        { _and: [
            {blocked : false},
            {numberOfTasks : 0},
            {completed : false},
            { _match: [ObjectSpecifier().containingProject.status, 'active']},
        ]},
        { _or: [
            {deferDate : null},
            {deferDate : {'<' : now}}
        ]},
        { _or: [
            { _match: [ObjectSpecifier().containingProject.deferDate, null]},
            { _match: [ObjectSpecifier().containingProject.deferDate, {'<' : now}]}
        ]},
        { _or: [
            {context : undefined},
            { _match: [ObjectSpecifier().context.allowsNextAction, true]},
            { _match: [ObjectSpecifier().context.hidden, false]}
        ]},
    );
#5

User @draft8 did this brilliant script that I modified.

Full script (Sierra onwards)

(function () {
    'use strict';

    // compare :: a -> a -> Ordering
    const compare = (a, b) => a < b ? -1 : (a > b ? 1 : 0);

    // map :: (a -> b) -> [a] -> [b]
    const map = (f, xs) => xs.map(f);

    // unlines :: [String] -> String
    const unlines = xs => xs.join('\n');

    // mappendComparing :: [((a -> b), Bool)] -> (a -> a -> Ordering)
    const mappendComparing = fboolPairs =>
        (x, y) => fboolPairs.reduce(
            (a, fb) => {
                const f = fb[0];
                return a !== 0 ? (
                    a
                ) : (
                    fb[1] ? compare(f(x), f(y)) : compare(f(y), f(x))
                )
            }, 0
        );

    // zipWith4 :: (a -> b -> c -> d -> e) -> [a] -> [b] -> [c] -> [d] -> [e]
    const zipWith4 = (f, ws, xs, ys, zs) =>
        Array.from({
            length: Math.min(ws.length, xs.length, ys.length, zs.length)
        }, (_, i) => f(ws[i], xs[i], ys[i], zs[i]));


    // MAIN ------------------------------------------------------------------

    const now = new Date()
    
    // Get available tasks
    const tasks = Application('OmniFocus')
        .defaultDocument.flattenedTasks.whose(
            { _and: [
                {blocked : false},
                {numberOfTasks : 0},
                {completed : false},
                { _match: [ObjectSpecifier().containingProject.status, 'active']},
            ]},
            { _or: [
                {deferDate : null},
                {deferDate : {'<' : now}}
            ]},
            { _or: [
                { _match: [ObjectSpecifier().containingProject.deferDate, null]},
                { _match: [ObjectSpecifier().containingProject.deferDate, {'<' : now}]}
            ]},
            { _or: [
                {context : undefined},
                { _match: [ObjectSpecifier().context.allowsNextAction, true]},
                { _match: [ObjectSpecifier().context.hidden, false]}
            ]},
        );

    const [names, flags, dues, notes] = map(
        k => tasks[k](), ['name', 'flagged', 'dueDate', 'note']
    );

    return unlines(
        zipWith4(
            (strName, blnFlagged, dteDue, strNote) => {
                const v = 1 +
                    (blnFlagged ? 1 : 0) +
                    (Boolean(dteDue) ? 1 : 0) +
                    (strNote.includes('#important') ? 1 : 0);
                return [v, strName]
            },
            names, flags, dues, notes
        )
        .sort(mappendComparing([
            [x => x[0], false],
            [x => x[1], true]
        ]))
        .map(([a, b]) => a.toString() + ' - ' + b)
    );
})();
#6

Wow, @unlocked2412 and @draft8 you guys are awesome - very grateful for your time.

I will study this code bit by bit and try to understand what’s going on. My own logic is quite a bit more complex than the example (I’m prototyping a fairly ambitious layer of logic on top of Omnifocus)*, and this coding style is very different than what I’m doing (from a more OOP background), so the prospect of porting my logic to this style is somewhat daunting.

In addition to studying the code, it would be really helpful if you could point out which changes are essential to the speed vs. which are for stylistic preference / to make it concise. And let me know if there are any resources that would be helpful to understanding what’s going on / what I was doing wrong.

(*Some examples of other things my logic does that I’ll need to replicate, to give you an idea:

  • also consider a task flagged/due if any of its ancestors are flagged/due - the task’s property doesn’t seem to reflect this in the same way the UI does
  • check whether a task’s project is under a certain time budget - e.g. boost task X if I completed less than 2h of other tasks in its project Y this week
  • update the note field of a bunch of tasks to add a “tag”)
#7

You’re welcome. I’m glad you found it useful, @zambaccian.

Out of my house, today. When I come back, I will give some suggestions.

#8

That style is known as functional programming.

One of the thing I would suggest would be batch-fetching all properties at once, by calling it it as a method with trailing parentheses.

tasks = Application('OmniFocus').defaultDocument.flattenedTasks()

Instead of storing a reference

tasks = Application('OmniFocus').defaultDocument.flattenedTasks

and then fetch each item. I noticed a significant gain in speed there.

The use of const as a way to declare variables promotes immutability since that variable name can’t be used to declare another variable. Immutability is one of functional programming key concepts.

#9

Task properties containingProject and container hold the piece of data you are looking for.

Project property estimated minutes holds that piece of data.

My advice would be to wait until OF 3 ships with multiple tags before making a complex system depending on that.

#10

As a general footnote on performance – the main currency consists of Apple Events, and the trick is just to minimize the number of them that you use.

The scripting interface behaves like a queryable database. Something like:

Application('OmniFocus').defaultDocument.flattenedTasks

is not yet an Array – it’s just an interface which we can query, using the kind of expression which @unlocked2412 demonstrated.

(See https://developer.apple.com/library/content/releasenotes/InterapplicationCommunication/RN-JavaScriptForAutomation/Articles/OSX10-10.html)

If we add trailling parentheses, and call it as a method:

Application('OmniFocus').defaultDocument.flattenedTasks()

then we do obtain an Array as a return value (1 Apple Event),
but the next steps are going to be expensive – it will require one more Apple Event every time that we retrieve a particular property (.deferDate, .status, .allowsNextAction) from each individual task in that Array. This is why your first draft cost so many Apple Events.

The much cheaper strategy, as @unlocked2412 explained, is to cache a reference to .flattenedTasks (either raw or query-filtered, and holding off on those trailing parentheses)

You can then fetch the name of every single task with an expression like tasks.name() (just 1 Apple Event for the whole set of names)

and similarly fetch the whole batch of due dates (again at the low cost of just 1 Apple Event for the entire set) with tasks.dueDate() etc etc.

The only problem you then have to solve, is that for 100 tasks, you now have:

  • A list of 100 names, and
  • a separate list (in the same order) of 100 due dates,
  • and another list of 100 flag states …

etc.

This is when it is useful to have a function like zipWith4, which takes four lists, and moves through the index from start to finish, combining the index-matched values.

Peers of zipWith4 (like zipWith6, zipWith8) could also be constructed, or you could just hand-write your own loop.

1 Like
#11

PS as @unlocked2412 mentioned, OmniFocus 3 will have its own JSContext, allowing it to bypass the AppleEvent bottle-neck.
This should allow for a much faster scripting interface, as it already has, for example, in OmniGraffle.

#12

Understanding arrays vs. references, knowing that this is functional programming (which I’ve wanted to learn), and in particular understanding how to batch get properties and zip them up has made everything make sense. Have already made my script 8x faster and there’s still more to optimize. :)

2 Likes