Building TaskerArmy, Part II: The AI Wasn’t the Hard Part

When I published Part I of the TaskerArmy engineering journal, the central idea was simple: generating Shopify code is not the hard part. The hard part is building a system around that code so the work is well-defined, clear and recoverable.

I still believe that. The difference now is that we have had enough real failures to understand what those words actually cost.

The first article was mostly about the architecture we wanted: inspect the store, prepare a well-defined changeset, validate it, put it on staging, let the merchant review it and only then allow a deliberate move toward production. Since then, most of the important work has been less glamorous. We have been fixing all the places where a system can say something happened when Shopify says otherwise.

A deployment can return without proving that the remote theme contains the expected files. A process can die after Shopify accepted a write but before our database records it. A staging theme can look healthy while missing an asset. A merchant can approve one revision while a newer one is still open. A background inspection can get stuck waiting for an answer nobody is there to give and still look completed from the outside. Billing can show one number while the ledger says another.

Those are not AI problems in the usual sense. They are engineering problems created by giving AI enough authority to participate in real workflows.

That has been the biggest change since Part I. We are spending less time asking whether a model can write useful Liquid and more time asking whether TaskerArmy can prove what happened before it tells a merchant that work is done.

The product started becoming real when success had to be proven instead of inferred.

Production made the state machine real

The first journal talked about states like planned, staged, approved and live as if they were clean steps in a diagram. They are clean in a diagram. They are not clean once a remote platform, a database and a user can all change independently.

One of the first hardening passes after Part I was to stop treating deployment as a single action. We added a durable deployment-attempt journal before writes begin. An attempt can be pending, running, succeeded, failed, interrupted or reconciled. The important part is not the labels. The important part is that TaskerArmy now has somewhere durable to record intent before Shopify is touched.

That matters because the ugliest failure window is not 'Shopify returned an error.' That one is easy. The ugly window is when Shopify accepts the change and TaskerArmy crashes, times out or loses connectivity before recording the result locally.

Without recovery logic, the next process has no honest answer. Did the deployment happen? Should it retry? Would retrying duplicate work? Is the live theme already correct?

So interrupted deployments are reconciled by reading Shopify again. We do not mark a deployment successful because the previous process probably got far enough. We compare the remote theme against the expected result and classify what actually exists.

The same thinking was extended to rollback. Rollback is not 'write the old files back and hope.' It has its own durable attempt, drift checks, recovery behavior and post-write readback. If live changed after the changeset was built, TaskerArmy refuses to casually overwrite that newer state unless the action is explicitly overridden.

This was a major mental shift for me. A safe rollback is not the inverse of deploy. It is another production mutation with exactly the same need for evidence, concurrency control and recovery.

Staging was not the same as done

Another uncomfortable discovery was that some of our own lifecycle semantics were too optimistic.

We had cases where work could be treated as complete too early, or where the distinction between 'deployed to staging' and 'published live' was not strict enough across every part of the product. That is dangerous because a merchant should never have to understand our internal ambiguity to know whether production changed.

We corrected the lifecycle so staging is explicitly staging. Merchant approval is its own event. QA has to happen before approval. A revision request invalidates the previous staged result and forces the work back through the relevant steps instead of letting an old preview continue to look authoritative.

We also tightened session completion. A session should not become completed merely because somebody clicked Approve, or because a changeset carries a superficial live status. Completion is tied to verified publication evidence and the absence of newer unresolved work.

That change seems small until you think about what a merchant sees. If the UI says Completed, that word has to mean something stronger than 'the process reached the end of one branch.' It has to mean TaskerArmy has evidence that the relevant work reached the state we claim.

The preview experience itself also evolved. We added safer staging snapshots and an in-app review flow so the merchant can see the staged result without immediately leaving the product. Revision requests are persisted rather than treated like ephemeral chat context.

This is where the original staging-based idea stopped being a slogan and became a lifecycle we had to enforce line by line.

Evidence became a product feature

One of the most important additions since the first journal is publication evidence.

Earlier versions of the system could know that an API request was sent successfully. That is not the same thing as knowing the store now contains the intended result.

We added readback verification so TaskerArmy can compare what it intended to publish with what Shopify actually exposes afterward. That evidence is surfaced to the merchant instead of living only in logs. If the recorded status and Shopify disagree, we want the UI to show the conflict rather than smooth it over.

The same principle now applies to staging synchronization. If staging falls out of sync, that state should be visible. If publication needs to be re-verified, the merchant can trigger a read-only check rather than rerunning an implementation.

This sounds almost boring, which is good. In the first phase, the impressive moment was watching AI generate a diff. Now one of the most valuable screens is the one that says, in effect: this is what we expected, this is what Shopify contains, and these two things match.

Shopify kept changing the definition of ready

The Shopify app has been the other major source of progress and frustration.

TaskerArmy can work beautifully inside our own assumptions and still fail the moment Shopify's platform rules become the real boundary. We have hit that repeatedly.

Before Part I, theme access still depended on REST-era patterns. For a public app, that was not going to survive review. Shopify requires new public apps to use the GraphQL Admin API, so we migrated the entire theme I/O path: theme listing, file reads, file writes, deletes and theme duplication.

That migration was not just compliance work. It changed the architecture in useful ways. Theme duplication can happen server-side instead of copying a theme asset by asset through a long sequence of requests. That reduces the number of places where rate limits can leave staging half-built.

Naturally, the migration then exposed two bugs of its own. Shopify GraphQL throttling arrives inside a successful HTTP response, so code that only retries HTTP 429s can quietly stop retrying exactly when the store gets busy. We fixed the error classification so GraphQL THROTTLED responses enter the same retry path.

We also found that staging repair could report success even when a missing binary asset had been skipped. The repair path was technically finishing, but the staging theme was still incomplete. That was fixed so an unrepairable missing file remains a failure, not a silent success.

Then, weeks later, the current GraphQL contract for theme duplication forced another correction and its own regression test. That is the reality of building on a platform API: 'we migrated to GraphQL' is not a permanent achievement. The contract has to remain executable as Shopify evolves.

App Review was engineering work

I originally thought of Shopify App Review as a launch checklist around the product. I do not anymore.

The review requirements forced decisions deep into TaskerArmy: how a merchant enters OAuth, which scopes are declared, how compliance webhooks resolve, whether the app shows any intermediate TaskerArmy UI before Shopify authentication, how billing is disclosed, how a reviewer can reproduce a harmless staged change, and which human-owned approvals still block submission.

One bug was particularly instructive. Our configured Shopify App URL included a path. Shopify resolves relative webhook URIs against that URL, which meant the resulting webhook endpoint gained a duplicated path. The obvious fix was to make webhook URLs absolute. That hid the symptom but left the configuration model wrong.

The real correction was to make the application URL the bare origin and let the webhook paths resolve from there. We added a regression test that performs the same URL resolution Shopify performs instead of checking a particular string. That test now guards the rule rather than one accidental implementation.

That incident became a good example of a broader rule we are adopting: a regression test should protect the invariant that failed, not merely the exact line that happened to fix it.

The app later moved to a modern embedded Shopify Admin experience. That required App Bridge session tokens, token exchange, protected UI gating, race-safe tenant/store bootstrap and carefully scoped iframe policy.

Then came CSP and CORS. We had to enforce Shopify frame ancestors in the server that actually serves production on Railway, not in a configuration file that was not on the runtime path. We had to explicitly allow the App Home origin through backend CORS. We had to reject spoofed 'embedded' hints and fail closed when required client configuration is missing.

None of those fixes makes TaskerArmy look more intelligent in a screenshot. They make it a real Shopify app.

Engineering Runs had to become honest money

The Run model has also changed more than the pricing page suggests.

At first, Runs were mostly our way of avoiding token language and mapping engineering scope to something a merchant could understand. As implementation matured, they became an authorization and accounting boundary.

TaskerArmy now treats an implementation as a well-defined cost that can use 1, 2 or 4 Engineering Runs depending on scope. The merchant authorizes the ceiling before work starts; the final charge is based on the completed work that actually survives the workflow. Run Packs exist as separate capacity rather than pretending every customer needs a larger subscription.

We changed pricing more than once because the unit economics forced us to. One version gave too much capacity at the Scale tier relative to worst-case inference cost. We reduced the included Runs and removed 'Unlimited' language from Enterprise rather than keep a cleaner-looking pricing table that did not match the economics.

That was not fun. Pricing changes rarely are. But an autonomous engineering product that ignores the cost of failed attempts, retries and expensive tasks eventually becomes either misleading or unprofitable.

The billing implementation then moved toward Shopify App Pricing and Shopify-hosted plan management, with Run Packs modeled through Shopify usage events. That created another migration problem: new billing behavior had to coexist with legacy subscriptions and cancellation paths long enough to avoid breaking existing tenants.

The billing ledger has also caught us being wrong.

In one background inspection, the UI reported one Engineering Run charged even though the actual ledger showed zero consumption for an internal plan. The code was persisting the amount we intended to charge instead of the amount the billing RPC actually applied.

We fixed that distinction explicitly: requested amount, successful finalization and actual charged amount are different concepts. The UI should show the ledger truth, not the pre-charge estimate.

A background run completed with no report

Another incident was worse because it crossed product behavior and billing.

TaskerArmy has background inspection flows that can analyze a store without a merchant sitting in the chat. One of those runs called the same request-user-input capability available to the interactive agent. The model asked which finding it should prioritize.

There was nobody there to answer.

The underlying session waited for input, but the background service only checked whether the run had been interrupted. It did not check the awaiting-input state. The flow continued, charged an Engineering Run and marked the inspection Completed with no report.

That is exactly the kind of failure I want this journal to record because it is much more useful than saying 'we improved background automation.'

We fixed it structurally. The user-input tool is not offered to non-interactive runs at all. There is also a defense-in-depth guard if a model somehow attempts it anyway. And before any charge occurs, the background service now explicitly refuses to treat an awaiting-input result as completion.

The important lesson was not that the model asked a bad question. The lesson was that the system offered an impossible action in that execution context.

Autonomy is not asking the model to behave. It is removing actions that should not exist in that mode.

We also added live progress, explicit authorization, cancellation and honest charged-versus-authorized Run visibility to those inspections. Again, the theme is the same: less magic, more observable state.

The calendar still found us

Not every failure was architectural. One was simply time doing what time does.

A usage analytics test used a fixed future date for the next Engineering Run reset. Eventually that date stopped being future. The production logic correctly advanced the billing period, but the fixture still expected an August changeset to fall inside the old period. CI went red even though the feature had not changed.

The fix was straightforward: make the test dates relative to the time the test runs. The value was in the reminder. A system with billing periods, expiring credentials, staged themes and remote state cannot treat time as background scenery.

What broke changed TaskerArmy

When I look at the work since Part I, the progress is easier to understand through the failures than through the feature list.

There is a lot in that table that nobody would put in a launch video. I am increasingly convinced that this is the useful part of the story.

The first version of TaskerArmy was exciting because it could do things. The current version is more interesting because it is learning how to refuse, recover, verify and account for what it did.

Where TaskerArmy is now

TaskerArmy today is much closer to the product described in Part I, but not because the model suddenly became dramatically better.

The changes around the model are what moved the product forward.

• Theme operations have been moved to Shopify's GraphQL Admin API, including server-side theme duplication and hardened throttling behavior.
• Staging, QA, merchant approval, revision, live publication and session completion have stricter lifecycle semantics.
• Deployments and rollbacks leave durable attempt records and can reconcile after interruptions instead of guessing.
• Live-theme drift is checked before consequential writes, and publication is verified by reading Shopify back afterward.
• Merchants can see publication evidence, staging synchronization state and conflicts between recorded and remote state.
• The Shopify app now follows a modern embedded App Home flow with App Bridge session authentication and tighter CSP/CORS boundaries.
• Billing has moved toward Shopify-native App Pricing, with Engineering Runs and Run Packs treated as explicit commercial and accounting contracts.
• Background inspections have authorization, progress, cancellation and execution-mode constraints instead of borrowing interactive-agent assumptions.

There are still open edges. App review itself has human-owned dependencies. Shopify platform contracts will change again. Billing rules will keep getting tested by edge cases. New forms of store customization will create new drift and validation problems.

That no longer feels like evidence that the product is unfinished in the naive sense. It feels like the permanent operating condition of software that has permission to change somebody else's commerce system.

What I want to make boring next

My goal for the next phase is still boringness, but now I can define it much more specifically.

I want connecting a store through Shopify to be routine. I want a staging theme to be created or repaired without us discussing rate limits. I want the merchant to see exactly what TaskerArmy changed, exactly what it verified and exactly what will cost Runs before they authorize it.

I want a process crash in the middle of a production mutation to be an operational event, not an existential question. The next process should be able to read the durable attempt, inspect Shopify and determine the truth.

I want a revision request to invalidate the old result automatically. I want a rollback to be boring because drift detection and readback are normal. I want app authentication to be boring because spoofed states simply cannot reach protected UI. I want billing to be boring because the number on the screen is always the number the ledger actually charged.

And I want the AI itself to become almost less visible.

That may sound strange for an AI product, but the end state I care about is not a merchant admiring a model. It is a merchant asking for a well-defined Shopify improvement and receiving a staged, clear, verified result with a clear cost and a clear production decision.

The first journal argued that code generation is a capability and engineering execution is a system.

This second one has taught me something narrower and more practical.

A trustworthy engineering system is one that can prove success, survive ambiguity and admit when the remote world disagrees with its own state.

We have found more ways for TaskerArmy to be wrong since Part I than I expected.

That has been one of the most productive parts of building it.