End of sprint 3: ERP Integration Connector Platform (NetSuite-first)

Sprint retrospective. One connector, many independent customer accounts. Demo and design notes.

Sprint goal

The connector itself was built in earlier sprints. This one was about the tests behind it, and what happens under concurrent load.

What shipped

The connector predates this sprint; listed here as the surface the new tests and load scenarios cover.

• One connector, three ERPs. A single ErpConnector interface serves NetSuite, Dynamics 365 and QuickBooks. Adding an ERP is one connector plus one register() call.
• Four NetSuite APIs behind that interface. SuiteTalk REST + SuiteQL, SuiteTalk SOAP, RESTlets, and SuiteScript 2.x deployed inside the account.
• Sync that tells the truth. Server-sourced watermarks per (connection, entity, direction); per-tenant sealed credentials (HKDF-derived); idempotency records keyed by business identity; field-level conflict records; per-connection circuit breakers; throttling that honours Retry-After.
• A NetSuite-equivalence harness. A service shaped like the NetSuite API that injects faults Chaos Monkey style: throttling, a write that commits and then loses its response, an outage, a mid-run schema change.
• Multi-tenancy. PostgreSQL row-level security, every query scoped by tenant_id, with a resident worker draining an outbox under leases.
• Observability and ops. Prometheus metrics, a Grafana dashboard, self-throttling, backfill progress, and customer-facing diagnostics that fail loud instead of looking like "no data".

Evidence

1. The demo

Open the Stage, then press Play demo: the simulated NetSuite account, the account's own SuiteScript, and the connector picking up each change. Recorded at half speed.

2. Five-scenario walkthrough

Five scenarios, one connector. Recorded from the live console, at half speed.

3. Lessons Learnt

Twelve slides, five seconds each, aimed at engineers: real do / don't code from the build.

What went well

• Contract tests across interchangeable transports. One test asserting REST, SOAP and RESTlet return the same records found two bugs that no unit test could reach.
• The schema snapshot became the source of truth for how to talk to a given customer, so field-name differences are discovered and stored rather than assumed.
• Idempotency by business identity, not request id, made retries and duplicate prevention safe and testable.
• The demo reads the platform's own numbers. Printing raw evidence is what exposed the dashboard reporting a breaker as closed while it was open.
• Deployment found what the suite could not. Reachability bugs (a broken readiness probe, a static bundle missing a file) only show up behind Apache.

What we learned

Twelve bugs the tests and the deployment caught, each written up as do / don't code. The full tour is on the console's Lessons Learnt tile; the short list:

• An upsert whose defaults are the state resets the state (a circuit breaker that could never open).
• Retry-After is a floor, not a suggestion. Separate your backoff from their instruction.
• Contract-test interchangeable transports, not each one in isolation.
• Write the field name the account actually has, or the ERP drops your data silently.
• A failing test is a hypothesis. Here, the test was wrong: internalId is per-account.
• A UI that reads the wrong key is confidently wrong. Report the worst state across keys.
• RFC 5849 signs form-encoded bodies only. A JSON body in the signature fails every write and no read.
• A composite key one field short corrupts balances silently (missing lineSequenceNumber).
• A guard you only observe on a second run never fires. The cursor already moved.
• A running service never drops the schema it serves from. Reset rows, not tables, and lock the endpoint.
• rate() and increase() are hostile to churning labels. Use cumulative values there.
• Never name a package after a stdlib module, and probe every meta endpoint, not the cheapest one.

Numbers

Next sprint

• SugarCRM connector. The next ERP after NetSuite, Dynamics 365 and QuickBooks. Main Sugar surfaces to cover: Sugar Sell, Sugar Serve, Sugar Market and the open Community Edition, against Sugar's REST v11 API and the legacy SOAP v4_1 API, over the standard module model (Accounts, Contacts, Leads, Opportunities).
• Safer dead-letter replay. Replaying a dead letter needs the same idempotency-key discipline as a normal write, plus an audit trail, so a replay cannot duplicate or double-apply.
• Functional-currency conversion and mapping in OneWorld tenants. Mapping currency and subsidiaryId is not the same as converting the way the account does, and comparing two subsidiaries' amounts without it is how reconciliation arguments start.

Appendix: software stack and Python libraries

• HTTP API: fastapi served by uvicorn; pydantic v2 for request models and pydantic-settings for configuration.
• Data: sqlalchemy 2 ORM over PostgreSQL with psycopg2, scoped by tenant_id with row-level security underneath.
• Outbound HTTP and crypto: httpx for async ERP calls; cryptography for sealing per-tenant credentials.
• CLI and ops: typer + rich; a resident worker draining an outbox with leases; prometheus-client for metrics.
• Quality and load: pytest (~190 tests) and ruff; locust + gevent in a separate virtualenv.
• In-account: SuiteScript 2.x inside the simulated NetSuite; React + Vite for the admin dashboard.

Recorded from the live console: the simulated NetSuite account as a multi-window desktop, an Omni-style workspace of floating windows (draggable, resizable, minimize tray), with each window in its own browsing context, and the numbers read from the platform's own endpoints.