DEV Community

Cover image for Teaching an AI assistant to forget a deleted DocuSign agreement
Akshat Gupta
Akshat Gupta

Posted on Edited on

Teaching an AI assistant to forget a deleted DocuSign agreement

Here is a small mystery.

I moved a signed offer letter to the Deleted folder in DocuSign. Then I looked at what a normal "what changed?" listing said about it. Nothing. The status was still completed. The status date had not moved. As far as that listing could tell, the letter was fine.

If I had trusted that listing alone, the letter would have stayed in my AI assistant's memory, salary and all. Nothing would have crashed. The assistant would just remember a contract that was gone.

So for Mergetober I built a DocuSign connector for Cognee, and this puzzle ended up being the heart of it. The code is in PR #544 for issue #5716.

What it does

Cognee turns documents into a knowledge graph an agent can search. The connector feeds it from DocuSign:

  • Each envelope becomes one document. A short header says who sent it, who signed as what and when, plus custom fields and what signers typed into form fields. The text of the signed PDF follows.
  • You choose which envelopes: by status, by folder, or by a start date.
  • Later runs only look at envelopes whose status changed.
  • Envelopes that are voided, deleted or purged are forgotten.
source = docusign_source()  # DOCUSIGN_* from the environment
await cognee.remember(source, dataset_name="docusign",
                      write_disposition="merge")
Enter fullscreen mode Exit fullscreen mode

It signs in with a JWT grant (integration key, user id, RSA key) and only ever sends GET requests. The API host comes from DocuSign's userinfo call, not a hardcoded demo.docusign.net, because production accounts live on different hosts.

The catch with "deleted"

DocuSign does not really delete envelopes. And the obvious signal does not work.

I sync with from_date and from_to_status=changed, which lists envelopes whose status changed. But a completed envelope cannot be voided. Removing it means moving it to the Deleted folder, or purging it. Neither one changes the status date. A cursor walks right past.

The delete you cannot see in the listing

So every run does a second check. Any envelope the connector ingested earlier but the listing did not return is looked up again by id, with its folders and purge state. If it is binned, purged, gone or out of scope, it becomes a delete. dlt removes the row, and Cognee's orphan cleanup removes it from the graph and the vectors.

A wrong delete is worse than a missed one, so a failed page, a 401 or a long rate limit stops the run before anything is deleted or the cursor moves. An empty listing never wipes the dataset.

The real test

I built a fake DocuSign first. It sits under a real HTTP session, so the JWT exchange, paging and retries all run. 82 tests pass, including some that run the whole Cognee pipeline with a mocked LLM and show that an unchanged re-sync makes no LLM call.

Then I used a free developer account. I created five agreements through the API, signed them with a headless browser, and ran the example script with a real LLM.

Five signed agreements, two quiet deletes

I asked "Who signed which agreements, and when?" and all five signers came back with the right times.

Then I moved the Deltacorp offer letter to the Deleted folder. Its status stayed completed, just as I feared. The next sync reported forgotten: 1, and the dataset went from 5 documents to 4. Asking about the salary in that letter now returns "the context does not state it".

I purged another envelope. Status stayed completed there too. Only purgeState changed, to documents_queued. One sync later: forgotten: 1, 4 documents down to 3.

What only the real account showed me

It confirmed four things I had only read in docs. The by-id lookup works without from_date. The form data has the shape I assumed. The folder types are sentitems and recyclebin. Neither a binned nor a purged envelope moves the status date.

Then it broke two of my assumptions.

Folders and statuses do not mix. I asked for status=completed,voided,deleted so removals would show up in the listing. With a folder filter on top, DocuSign returns zero envelopes and no error. A sync scoped to a folder would have ingested nothing and said nothing. A custom folder name is also refused with HTTP 400. So names are now resolved to ids through GET /folders, and with a folder the status is checked in the connector.

Deleted means deleted, eventually. Six days after I moved Deltacorp to the Deleted folder, DocuSign had emptied the folder. The envelope was still in the listing, with no folders, and its PDF answered HTTP 400. A fresh sync would have crashed on it. Now that envelope is skipped, and an ingested one is forgotten.

I also added a request budget. max_requests=5 stops a run cleanly, keeps what it read and resumes at the cursor. From scratch it finished in three runs and never fetched an envelope twice.

What I did not test

  • Scanned PDFs. The demo stamps text onto every page, so my "scan" was never empty. That path has fake tests only.
  • A purge that removes the envelope from every listing.
  • Production. I ran the demo host only.
  • A real 429. The rate-limit paths are tested against the fake only.
  • Distinct signer emails. All five signers used my own address.

One more limit. The text comes from the PDF's text layer. There is no OCR.

Try it

The example is packages/connector/docusign/examples/example.py. You need a free developer account, an integration key with an RSA key pair, and one consent click. The README has every step.

If you point it at real agreements, keep the API user's permissions small and the dataset separate. Then tell me what breaks.

Top comments (0)