Developer Portals, Catalogs, and Provisioning

The previous article, Building a Golden Path with Explicit Service Contracts, defined a service contract with supported options, an exception process, and compatibility checks, and it described a template that produces a conforming service. This article assumes you took those ideas from it: a contract states what the platform guarantees and what the consumer must do, and a template is only the first step in a service’s life. What it does not cover is the machinery around the template. Who knows that the service exists? Who owns it? What actually happens when a person presses a button that says create a database, and what is left behind when that work stops halfway?

Those questions belong to three cooperating pieces: a software catalog that records what exists and who is responsible for it, a portal that presents the catalog and the available actions, and workflows that carry out provisioning. They are often shipped as one product and described with one word, which hides the fact that each has its own source of truth and its own failure modes. This article separates them, then works through catalog metadata, ownership, workflow orchestration, permissions, and provisioning failure and recovery.

The scenario data and code are illustrative. Team names, resource names, and dates are invented. The three scripts were written for this article and run locally with Python 3.12 using only the standard library. They use in-memory stand-ins for a repository host, a database service, a secret store, and a catalog, and no real service is called. They show mechanisms, and they are not measurements of any product.

Three layers that share a name

A developer portal is the entry point people use to find, create, and operate software. The CNCF platforms white paper lists web portals and APIs as separate platform capabilities, and it describes self-service as letting users request and receive capabilities autonomously, through a portal, a command-line tool, or an API, with minimal manual intervention (CNCF TAG App Delivery). A portal is one possible interface to those capabilities. It does not perform the work.

Layer	Question it answers	Source of truth	Typical failure
Catalog	What exists, who owns it, how does it relate to other things?	Records ingested from systems that are authoritative for each field	Stale owner, missing entity, dangling dependency, record that is accurate about intent and wrong about reality
Portal	How does a person find a record or start an action?	None of its own: it presents the catalog and calls workflows	A hidden button treated as access control, a screen that claims success the backend never confirmed
Workflow	What runs, in what order, with what retries, and did it finish?	A durable run record plus the state of the systems it changes	Partial completion, duplicate resources after a retry, a stuck run nobody notices

The service catalog entry makes the same split between metadata and execution: the catalog records software and its ownership, and a workflow is needed to create another instance. Self-service provisioning adds the user’s view: approval, execution, and interpretation can each leave a human in the loop, and an automated path should be judged by whether the user can request, receive, and understand an outcome without waiting for someone. Keep the three layers apart when designing, even if one product bundles them, because a design that blurs them tends to put authorization in the portal and truth in the screen.

Where this article refers to a concrete product, it uses Backstage, an open-source framework for developer portals, because its documentation is public and specific. Nothing here depends on running it. The same questions apply to any catalog and any workflow engine.

Catalog metadata

What a record needs to say

Backstage’s descriptor format shows what a catalog treats as essential. A Component entity has a required spec.type (common values include service, website, and library), a required spec.lifecycle (recommended values are experimental, production, and deprecated), and a required spec.owner, an entity reference to the owning team that produces an owned-by relation and its reverse. Optional fields tie the entity to a system, to the APIs it provides, and to the components and resources it depends on (Backstage descriptor format). Annotations carry non-identifying metadata that points to external systems, such as the repository, a monitoring dashboard, or a paging service.

Three design points follow. The type field is locally defined, so an organization must decide what each value means and validate it, or two teams will use one label for incompatible things. Lifecycle is a statement the platform can act on, since a deprecated component can trigger a notice to its dependents. And relationships such as depends-on are what turn a list into something you can query: without them the catalog can say that a database exists, but not who breaks when it is retired.

A minimal record for the contract in the previous article therefore carries a name, a kind, a lifecycle, an owner that resolves to a group, relationships to the things it depends on, and pointers to the repository and operational dashboards. Anything beyond that should have a named consumer. Every extra field is something a person must keep correct, and an unmaintained field is worse than an absent one because readers trust it.

Where the values come from

Backstage’s catalog processes entities in three stages. Entity providers fetch data from authoritative external sources, such as YAML files in version control, and seed it into the catalog’s database. Processors then validate, analyze, and modify each entity, and may emit relations and errors. Stitching finally merges the processed entity, its errors, and its relations from both directions into the entity that the catalog API serves (Backstage, life of an entity). Processing errors are exposed through backend events for operators and through status fields users can see. An entity that loses all its parent references is marked as orphaned, and the configured strategy decides whether it is kept or deleted, with deletion as the documented default.

Several operating consequences follow. The displayed record is a processed copy, not a live view of its source. A change to a file reaches the catalog only after ingestion and processing complete, so a record can lag its source, and the lag and failures need to be visible. Each field needs one authoritative source: the repository for the description, an identity system for group membership, a cloud inventory for resources. And because an orphaned entity can be deleted automatically, a deletion in the source system can remove a catalog record, which is useful for cleanup and dangerous if the source was edited by mistake.

Checking a record against the facts you have

The following script checks a small invented set of records for the problems that make a catalog untrustworthy: an owner that does not resolve to a known group, a lifecycle value outside the allowed set, a dependency that points to nothing, and an owner that has not been confirmed within a stated limit. It also computes the dependents of one resource, which is the query an owner needs before changing it. The limit of 180 days and the group names are arbitrary choices for the example.

"""Illustrative catalog-record checks (author-written, stdlib only)."""
import datetime
TODAY = datetime.date(2024, 8, 9)
GROUPS = {"team-billing", "team-search", "team-platform"}
LIFECYCLES = {"experimental", "production", "deprecated"}
MAX_AGE_DAYS = 180

ENTITIES = [
  {"name": "billing-api",  "type": "service", "lifecycle": "production", "owner": "team-billing",
   "depends_on": ["billing-db"], "owner_verified": "2024-06-01"},
  {"name": "billing-db",   "type": "database", "lifecycle": "production", "owner": "team-billing",
   "depends_on": [], "owner_verified": "2023-11-15"},
  {"name": "search-api",   "type": "service", "lifecycle": "production", "owner": "team-discovery",
   "depends_on": ["search-index", "ratings-db"], "owner_verified": "2024-07-20"},
  {"name": "search-index", "type": "database", "lifecycle": "prod", "owner": "team-search",
   "depends_on": [], "owner_verified": "2024-07-20"},
  {"name": "legacy-export","type": "service", "lifecycle": "deprecated", "owner": "team-platform",
   "depends_on": ["billing-db"], "owner_verified": "2024-02-01"},
]

def check(entities):
    names = {e["name"] for e in entities}
    findings = []
    for e in entities:
        if e["owner"] not in GROUPS:
            findings.append((e["name"], "owner", f"owner '{e['owner']}' is not a known group"))
        if e["lifecycle"] not in LIFECYCLES:
            findings.append((e["name"], "lifecycle", f"'{e['lifecycle']}' is not an allowed lifecycle value"))
        for d in e["depends_on"]:
            if d not in names:
                findings.append((e["name"], "dependsOn", f"'{d}' does not resolve to an entity"))
        age = (TODAY - datetime.date.fromisoformat(e["owner_verified"])).days
        if age > MAX_AGE_DAYS:
            findings.append((e["name"], "freshness", f"owner last verified {age} days ago (limit {MAX_AGE_DAYS})"))
    return findings

def dependents(entities, target):
    return sorted(e["name"] for e in entities if target in e["depends_on"])

if __name__ == "__main__":
    for f in check(ENTITIES):
        print(f)
    print("dependents of billing-db:", dependents(ENTITIES, "billing-db"))
    print("coverage: %d of %d records have no findings" % (
        len({e['name'] for e in ENTITIES} - {f[0] for f in check(ENTITIES)}), len(ENTITIES)))

Output from an actual run:

('billing-db', 'freshness', 'owner last verified 268 days ago (limit 180)')
('search-api', 'owner', "owner 'team-discovery' is not a known group")
('search-api', 'dependsOn', "'ratings-db' does not resolve to an entity")
('search-index', 'lifecycle', "'prod' is not an allowed lifecycle value")
('legacy-export', 'freshness', 'owner last verified 190 days ago (limit 180)')
dependents of billing-db: ['billing-api', 'legacy-export']
coverage: 1 of 5 records have no findings

Each finding corresponds to a different repair. The unresolved owner team-discovery and the unresolved dependency ratings-db point to a missing or misspelled record, so the fix is to register the missing entity or correct the reference. The lifecycle value prod is a vocabulary problem, fixed by validating at ingestion rather than by hand. The freshness findings are different in kind: the record is syntactically valid, and the question is whether anyone still stands behind it. A validator cannot answer that. It can only ask the owner to confirm.

The final line deserves caution. Only one of the five records has no findings, but this says nothing about the estate, because the script sees only the records that were registered. The more important coverage question is how many real services have no record at all. Measuring that requires an independent inventory, such as cloud accounts or deployment targets, compared against the catalog. Expose uncertain and missing coverage instead of presenting the catalog as complete.

Ownership

Ownership is the metadata most likely to decide an incident. A catalog entry with an owner field tells a responder whom to contact, and it tells a patching effort whom to ask. The ownership field becomes useful only when it names something that persists. Backstage documents spec.owner as typically a team, and a group rather than an individual is the sound choice in general because individuals change roles and leave. A group that reorganizes or is dissolved leaves its records pointing at nothing, which is exactly what the owner check in the script reports.

A name does not make an accountable owner. The service owner entry describes the role as accountable across the service’s life and says the mandate has to be usable: what the owner may decide, what needs another authority’s approval, and how unresolved issues escalate. Four areas need a stated mandate: readiness and lifecycle, the service promise such as support hours and change notice, escalation of missed commitments, and retirement. A catalog can store the pointer to that mandate. It cannot create the mandate.

Three operational rules make ownership data survive. First, assign it at creation. The template or workflow that creates a resource should require an owner and register it in the same run, because the creator is present and the resource has no history. Second, set a confirmation cycle, as the script’s freshness check does, and route the confirmation request to the group so that a missing response becomes a visible finding. Third, define transfer and retirement paths: when a team is dissolved, the catalog should show which records are affected, and someone with authority should reassign them or begin retirement.

Distinguish the accountable owner from the first responder. The catalog entry may point to an on-call rotation for first response while the owner is accountable for decisions. Conflating them leads to asking the wrong person at the wrong time. Also keep in mind what the catalog does not know: it records declared relationships, and a service can depend on something nobody declared. The dependents query in the script is only as complete as the declared dependencies.

Workflow orchestration for provisioning

What the workflow must do

Creating a useful capability is several actions against several systems. The scenario here is a request for a new database: create the repository entry, create the database, store its credentials in a secret store, and register the result in the catalog. Workflow orchestration is the layer that decides what runs, in what order, under what conditions, and what happens when something fails. For provisioning, that means a dependency order, state for every run, bounded retries, and an operational view that shows which runs are stuck. Dependency order matters here because the secret cannot be stored before the database exists, and registration should come last so the catalog never lists something that was not created.

The portal is the presentation layer over this. A backend plugin calls the workflow, and the cloud, the cluster, and the pipeline system perform and authorize the real changes. In a Backstage template the steps are actions, and the built-in ones include fetching content, registering entities in the catalog, and creating and publishing repositories (Backstage built-in actions). Actions can call external systems, so describing what a template does requires reading its actions and not assuming it only copies files.

Two execution models

Provisioning can be implemented as an imperative workflow, a sequence of steps with a run record, or as a declared desired state that a controller reconciles. Kubernetes’ documentation on custom resources describes the declarative approach: a user creates or updates a resource, a controller watches it, reads the desired state, reconciles the current state toward it, and writes status back (Kubernetes, custom resources). The control loop entry adds that a controller usually asks other components to do the work, and that partial progress is possible because the steps are not one atomic transaction.

Question	Workflow with run record	Declared state reconciled by a controller
What does the user submit?	A request for an operation	A description of the target
How is repetition handled?	The workflow must make each step safe to repeat, usually with keys	The controller compares target and observation on every pass, so repetition is the normal mode
Where is progress visible?	In the run record	In the resource’s status and conditions
Natural fit	Ordered, one-off actions that cross systems, such as creating a repository, then a database, then registering both	Long-lived resources whose configuration should keep matching a target
Main risk	A run that stops between steps with nothing responsible for finishing it	A resource that is stored but never acted on because no controller is running

Neither model removes the problem of partial failure. The workflow model puts it in plain view, because you must write the recovery path. The declarative model hides it inside the controller, but the controller still faces the same ambiguity when a provider call times out. The same Kubernetes page recommends a standalone service rather than a custom resource when the API is imperative, so the choice depends on what the consumer is asking for. Many platforms use both: a declared resource for the long-lived thing, wrapped by a workflow for the orchestration around it.

Permissions

Three identities appear in a provisioning request, and confusing them is the usual source of permission bugs. The requester is the person using the portal. The workflow identity is the service account the portal’s backend or workflow engine runs as. The downstream provider checks yet another identity, usually the workflow’s cloud credential. If the workflow identity is broadly privileged and the portal checks nothing, any requester gets the workflow’s power.

Backstage’s permission framework illustrates the decoupling. A permission names an action. A policy, written in code, receives the user and the permission and returns a decision, either allow, deny, or conditional, where conditional delegates the evaluation to the plugin that owns the resource. Policies decide, and plugins, typically backends, enforce (Backstage permission concepts). The developer portal entry adds the practical consequence: a hidden button does not prevent a direct backend request, so enforcement belongs on the request path, and a plugin that uses its own cloud credentials needs to preserve the initiating user in the audit trail or delegate identity downstream. Otherwise the cloud record shows only the service account.

The next script models the three checks in a simplified form. It first asks whether the requester belongs to the owning group, then whether the request falls inside a preapproved envelope, and finally, separately, whether the workflow identity may perform each action. The audit row records both the initiator and the actor. All names and limits are invented.

"""Illustrative request policy for a provisioning portal (author-written, stdlib only).
Three identities appear: the requester, the workflow's service identity, and the downstream provider."""
ENVELOPE = {  # requests inside this envelope need no human approval
    "database": {"sizes": {"small", "medium"}, "data_classes": {"internal"}},
}
MEMBERS = {"alice": {"team-billing"}, "bob": {"team-search"}}
SERVICE_SCOPES = {"workflow-svc": {"database:create", "secret:create", "catalog:register"}}  # not 'database:delete'

def decide(requester, req):
    # 1. Can this person act for this owning group at all?
    if req["owner"] not in MEMBERS.get(requester, set()):
        return "DENY", "requester is not a member of the owning group"
    # 2. Is the request inside the preapproved envelope?
    env = ENVELOPE.get(req["kind"])
    if env and req["size"] in env["sizes"] and req["data_class"] in env["data_classes"]:
        return "ALLOW", "inside the preapproved envelope"
    return "NEEDS_APPROVAL", "outside the envelope: route to a named approver"

def execute(requester, req, action, audit):
    # 3. The workflow identity is checked separately, and the audit row keeps the initiating user.
    ok = action in SERVICE_SCOPES["workflow-svc"]
    audit.append({"initiator": requester, "actor": "workflow-svc", "action": action, "allowed": ok})
    return ok

if __name__ == "__main__":
    audit = []
    cases = [
        ("alice", {"kind": "database", "owner": "team-billing", "size": "small",  "data_class": "internal"}),
        ("alice", {"kind": "database", "owner": "team-billing", "size": "large",  "data_class": "internal"}),
        ("alice", {"kind": "database", "owner": "team-billing", "size": "small",  "data_class": "customer"}),
        ("bob",   {"kind": "database", "owner": "team-billing", "size": "small",  "data_class": "internal"}),
    ]
    for who, r in cases:
        print(who, r["size"], r["data_class"], "->", decide(who, r))
    print(execute("alice", cases[0][1], "database:create", audit))
    print(execute("alice", cases[0][1], "database:delete", audit))
    for a in audit: print(a)

alice small internal -> ('ALLOW', 'inside the preapproved envelope')
alice large internal -> ('NEEDS_APPROVAL', 'outside the envelope: route to a named approver')
alice small customer -> ('NEEDS_APPROVAL', 'outside the envelope: route to a named approver')
bob small internal -> ('DENY', 'requester is not a member of the owning group')
True
False
{'initiator': 'alice', 'actor': 'workflow-svc', 'action': 'database:create', 'allowed': True}
{'initiator': 'alice', 'actor': 'workflow-svc', 'action': 'database:delete', 'allowed': False}

The first four lines show the policy distinguishing three outcomes. A small internal database for a member of the owning group is allowed without approval. The same group member asking for a large one, or for one holding customer data, is routed to an approver. A requester who is not in the owning group is denied outright, and the denial does not depend on the size. The self-service provisioning entry describes the same technique: encode the allowed cases as policy so that requests inside the envelope need no approval and only exceptions escalate.

The last lines show the second boundary. The workflow identity can create a database and cannot delete one, so even a correct request cannot cause deletion through this path. That choice has a cost, which the next section meets: a workflow that must compensate by deleting what it created needs delete permission, ideally scoped to resources created by that run. Least privilege and recovery pull in opposite directions here, and the design has to resolve them explicitly. The least privilege entry makes the related point that a broadly privileged service should not lend all its privileges to every user request.

The audit rows record that Alice initiated both actions and that the workflow service performed them, with the denial recorded as well. That record supports an investigation, which is the role the audit log entry assigns: it supports investigation and does not replace the control. An audit entry that says an operation was requested does not prove it completed, so the audit trail should record both the authorization decision and the observed outcome. The script’s audit rows hold the decision only, which is a limitation of the example.

The permission checks have limits worth stating. The group membership table is a stand-in. A real system derives membership from an identity provider, and stale membership is its own failure. The envelope is a policy, and a wrong envelope silently approves too much, so its limits need an owner and a review schedule. And authorization applies to the specific action and resource: a catalog record being visible to a person does not mean the person may delete the resource it describes.

Provisioning failure and recovery

Four kinds of failure

A provisioning run can fail in four distinct ways, and the recovery differs for each.

• Rejected before any work. The provider refuses the request, or validation fails. Nothing changed for this step, so a retry is safe once the cause is fixed.
• Partial completion. Some steps succeeded and a later one failed. The earlier effects exist and the run must either finish or undo them.
• Ambiguous outcome. The provider created the resource but the response was lost, or the call timed out. The caller does not know whether it acted. A naive retry may create a second resource.
• Duplicate submission. The user clicks twice, or a reconnecting client resubmits. Two runs start for one intention.

The partial-failure and lost-response cases are not exotic. Distributed calls fail between acting and answering, and the AWS Builders’ Library article on idempotent APIs explains the standard answer: the client supplies a unique request identifier, the service records it atomically with the effect, and a retry with the same identifier returns the same result, while a retry with the same identifier and different parameters is rejected as a validation error (Amazon Builders’ Library). The article also notes that late-arriving requests complicate the picture, since resources may have changed state by the time a retry lands.

A run record and four experiments

The script below implements a run as a durable record: a key for the logical operation, the requested name, and a map of completed steps to the resource identifiers they produced. The workflow loop skips steps already recorded and executes the rest. Each backend can be told to fail in a chosen way: reject before work, or do the work and lose the response. The backend honors a client token only if the caller supplies one, which lets the script compare the two retry styles. The run record is a Python dictionary here. A real system would persist it durably before each step, which this script does not do.

"""Illustrative provisioning workflow with a durable run record (author-written, stdlib only).
Fake in-memory backends stand in for a repository host, a database service, a secret store and a catalog."""
import json, uuid

class Backend:
    """A fake provider. create() is idempotent only if a client_token is supplied."""
    def __init__(self, kind):
        self.kind, self.items, self.by_token, self.fail_plan = kind, {}, {}, []
    def create(self, name, client_token=None):
        action = self.fail_plan.pop(0) if self.fail_plan else "ok"
        if action == "fail_before":
            raise RuntimeError(f"{self.kind}: rejected before doing work")
        if client_token and client_token in self.by_token:
            rid = self.by_token[client_token]
        else:
            rid = f"{self.kind}-{len(self.items)+1}"
            self.items[rid] = name
            if client_token: self.by_token[client_token] = rid
        if action == "lose_response":
            raise TimeoutError(f"{self.kind}: created {rid} but the response was lost")
        return rid
    def delete(self, rid):
        self.items.pop(rid, None)

def make_world():
    return {k: Backend(k) for k in ("repo", "database", "secret", "catalog")}

STEPS = ["repo", "database", "secret", "catalog"]

def run(world, record, use_token=True, log=print):
    """Execute pending steps. 'record' is the durable state: {key, name, done:{step:resource_id}}."""
    for step in STEPS:
        if step in record["done"]:
            continue
        token = f"{record['key']}:{step}" if use_token else None
        try:
            rid = world[step].create(record["name"], client_token=token)
        except Exception as e:
            log(f"  step {step}: {type(e).__name__}: {e}")
            record["status"] = f"failed at {step}"
            return record
        record["done"][step] = rid
        log(f"  step {step}: ok -> {rid}")
    record["status"] = "complete"
    return record

def compensate(world, record, log=print):
    for step in reversed(STEPS):
        rid = record["done"].pop(step, None)
        if rid:
            world[step].delete(rid); log(f"  compensate {step}: deleted {rid}")
    record["status"] = "rolled back (known resources only)"

def inventory(world):
    return {k: sorted(v.items.values()) for k, v in world.items()}

if __name__ == "__main__":
    print("== A. Partial failure, then resume from the durable record")
    w = make_world(); w["secret"].fail_plan = ["fail_before"]
    rec = {"key": str(uuid.UUID(int=1)), "name": "orders-db", "done": {}}
    run(w, rec); print(" ", rec["status"], rec["done"])
    print("  resume:"); run(w, rec); print(" ", rec["status"], inventory(w))

    print("\n== B. Lost response, retry WITHOUT a client token")
    w = make_world(); w["database"].fail_plan = ["lose_response"]
    rec = {"key": "k-2", "name": "orders-db", "done": {}}
    run(w, rec, use_token=False); run(w, rec, use_token=False)
    print(" ", rec["status"], "databases:", w["database"].items)

    print("\n== C. Lost response, retry WITH a client token")
    w = make_world(); w["database"].fail_plan = ["lose_response"]
    rec = {"key": "k-3", "name": "orders-db", "done": {}}
    run(w, rec); run(w, rec)
    print(" ", rec["status"], "databases:", w["database"].items)

    print("\n== D. Compensation after an unrecoverable failure, with an ambiguous step")
    w = make_world(); w["secret"].fail_plan = ["fail_before"]; w["database"].fail_plan = []
    rec = {"key": "k-4", "name": "orders-db", "done": {}}
    run(w, rec); compensate(w, rec); print(" ", rec["status"], inventory(w))
    w = make_world(); w["database"].fail_plan = ["lose_response"]
    rec = {"key": "k-5", "name": "orders-db", "done": {}}
    run(w, rec); compensate(w, rec)
    print("  after compensating an ambiguous step:", rec["status"], inventory(w))

Output from an actual run:

== A. Partial failure, then resume from the durable record
  step repo: ok -> repo-1
  step database: ok -> database-1
  step secret: RuntimeError: secret: rejected before doing work
  failed at secret {'repo': 'repo-1', 'database': 'database-1'}
  resume:
  step secret: ok -> secret-1
  step catalog: ok -> catalog-1
  complete {'repo': ['orders-db'], 'database': ['orders-db'], 'secret': ['orders-db'], 'catalog': ['orders-db']}

== B. Lost response, retry WITHOUT a client token
  step repo: ok -> repo-1
  step database: TimeoutError: database: created database-1 but the response was lost
  step database: ok -> database-2
  step secret: ok -> secret-1
  step catalog: ok -> catalog-1
  complete databases: {'database-1': 'orders-db', 'database-2': 'orders-db'}

== C. Lost response, retry WITH a client token
  step repo: ok -> repo-1
  step database: TimeoutError: database: created database-1 but the response was lost
  step database: ok -> database-1
  step secret: ok -> secret-1
  step catalog: ok -> catalog-1
  complete databases: {'database-1': 'orders-db'}

== D. Compensation after an unrecoverable failure, with an ambiguous step
  step repo: ok -> repo-1
  step database: ok -> database-1
  step secret: RuntimeError: secret: rejected before doing work
  compensate database: deleted database-1
  compensate repo: deleted repo-1
  rolled back (known resources only) {'repo': [], 'database': [], 'secret': [], 'catalog': []}
  step repo: ok -> repo-1
  step database: TimeoutError: database: created database-1 but the response was lost
  compensate repo: deleted repo-1
  after compensating an ambiguous step: rolled back (known resources only) {'repo': [], 'database': ['orders-db'], 'secret': [], 'catalog': []}

Experiment A shows partial failure with a rejected step. The secret store refuses the call after the repository and database exist. The run record holds the two completed steps, and resuming it executes only the remaining two. Nothing is created twice because the record says which steps already happened. This works when the failure is a clean rejection and the record is trustworthy.

Experiment B shows the ambiguous case without a client token. The database was created, the response was lost, and the retry created a second database. The run completes and reports success, yet two databases exist and the catalog knows about one of them. That is the dangerous outcome, because every individual step reported a plausible result. Experiment C repeats the failure with a client token derived from the run key and the step. The retry reaches the same database, and one database exists. The token is what turns the retry into a recognition of the same operation.

Experiment D shows compensation. After the secret store rejected the call, the script deleted the database and repository it had recorded, in reverse order. This leaves nothing behind. The second half of experiment D is the limit. When the database call returned an ambiguous failure, the run record never received an identifier, so the compensation had nothing to delete. The repository was removed and the database remained, an orphan that no record points to. Compensating only what the record knows is a useful rule and an incomplete one.

The recovery contract

The experiments suggest what a provisioning workflow must implement. These are design choices and not guarantees supplied by any portal.

1. A key for each logical operation. Generate it before the first submission, keep it across retries and reconnects, and use a new one for a deliberate second creation. The idempotency key entry lists what the server must decide: reconnect repeats to the same operation, compare the request and not only the key, record evidence with the effect, and state how long it is kept.
2. A durable run record written before each step. In the script a crash between the provider’s action and the record update is the weak point. Writing intent first lets recovery know which step to inspect.
3. Steps that ensure instead of create. A step should discover whether its effect already exists, by token or by a lookup on a deterministic name, and treat existence as success.
4. Reconciliation for unknown outcomes. When a step ends ambiguously, look up the provider for resources carrying the run’s token or tag before retrying or compensating, and keep the result in the record.
5. Cleanup with an owner. Decide who removes orphans, how they are found, and what permission they use. The earlier permission section shows why the workflow identity should not hold broad delete rights.

Idempotency makes retrying safe. It does not make the whole process atomic. The saga entry describes the standard model: local steps commit independently, with a defined way to undo earlier ones, and the process does not provide the isolation of a single transaction. Other people can observe a half-built environment, and a compensating transaction is a semantic reversal, not a restore. Deleting a database also deletes what someone may already have written to it, so compensation for steps that have been used is a business decision rather than a mechanical one. The catalog registration being last helps: if the run never reached it, no consumer found the resource through the catalog, though a direct connection string may still have been used.

Users need to see what is happening. A run that shows only a spinner and then a generic error pushes the user toward a support request, which defeats the point of self-service. DORA’s platform engineering guidance names clear feedback on the outcome of tasks as an important capability (DORA). For provisioning, the screen should show each step’s state, whether the run can be safely resumed, whether the outcome is unknown, and who handles a stuck run. Retry limits and timeouts also need definitions, so that a run that cannot proceed becomes visibly failed, not forever in progress.

Where this design fails

• The record and the catalog disagree. The catalog says a resource exists and the cloud account says otherwise, or the reverse. Compare them periodically against an independent inventory, and report the differences to owners.
• The portal becomes the only interface. If an outage in the portal blocks creation entirely, the platform has a single point of failure for new work. Provide an API or a documented fallback for critical paths.
• Concurrent runs. The script runs one request at a time. Two simultaneous requests with the same key need an atomic claim on the key, such as a uniqueness constraint in a database, or both can pass the check before either writes. The script does not demonstrate that case.
• Stale approvals and envelopes. An envelope that was reasonable when written can become wrong as the costs and data classes in the estate change. Review it on a schedule and record who approved the limits.
• Credential delivery. Provisioning a database whose credentials the user cannot receive safely is not a completed task. The outputs of a request, whether connection details, a secret location, or a workload identity binding, are part of the contract from the previous article.
• Manual steps hidden behind automation. A script that an engineer runs on request looks automated from the portal and is a ticket from the user’s side. Measure whether requests complete without a person.

A build order

A proposal for the sequence, not a standard:

1. Define the minimum catalog record and the authoritative source for each field. Validate it at ingestion and report ingestion failures where owners can see them.
2. Require an owner at creation and add a confirmation cycle with a visible finding when it lapses.
3. Choose the execution model for each capability, a workflow with a run record or a declared resource with a controller, and write down why.
4. Write the permission model with the three identities separated, a preapproved envelope with a named owner, and delegation or audit that preserves the initiating user.
5. Implement keys, a durable run record, ensure-style steps, and a reconciliation step for unknown outcomes. Test each failure kind deliberately with fault injection, as the script does with its fake backends.
6. Show users step-level status and a clear next action for each failure.
7. Compare catalog, run records, and an independent inventory on a schedule, and treat orphans and unlisted resources as defects with owners.

The next article, Designing Platform APIs and Extension Boundaries, treats the interface these workflows sit behind: lifecycle, tenancy, idempotency at the API boundary, and who may extend the platform.

References

• CNCF TAG App Delivery, Platforms White Paper
• Backstage, Descriptor Format of Catalog Entities
• Backstage, The Life of an Entity
• Backstage, Built-in Actions
• Backstage, Permission Concepts
• Kubernetes, Custom Resources
• Amazon Builders’ Library, Making retries safe with idempotent APIs
• DORA, Platform engineering