AI TaskVector - building a local multi-agent platform for automations, coding and personal use

Dear Huggingface community,

I was kind of experimenting with different models a few month ago, which ended up with this BETA version of AI TaskVector. Currently, it is a combination of both an AI Agent and factory that runs in local:

  • Built-in low-code framework for automations
  • Agentic runtime and orchestration
  • Co-pilot mode and Long-polling mode (e.g. run this workflow to make the Agents talk over Telegram: config/workflow_examples/telegram_bot_gateway.json)

The idea was to create an open-source platform where anyone could be AI-assisted with building, running and training their agents without deep knowledge and skill involved. Anyways, these were just my thoughts looking forward, it may end up being something else, who knows… :slight_smile:

I would be more than happy to have any useful feedback, comments. Feel free to create units, roles, play around with the repo and join my journey of the exploration, since I’m still learning.

Orchestrating multiple agents locally introduces significant context management challenges. When agents converse over a loop like your Telegram gateway, the KV cache grows rapidly, especially with long-polling. Have you implemented a strategy to prune historical turns or summarize intermediate states to keep inference latency stable? Without aggressive context window management, the cost per successful task cycle can spike faster than the model’s raw token speed suggests.

Thanks for pointing this out!

  1. I’m using llama.cpp and ollama for LLM inference, and I assumed they manage a KV cache internally.

However, the cross-request prompt-cache reuse is a separate feature and can depend on settings, which was not implemented.. Just added for both:

For example:

Ollama:

config = {
"host": "``http://127.0.0.1:11434``",
"keep_alive": "1800",
}

llama.cpp:

config = {
"host": "``http://127.0.0.1:8080``",
"cache_prompt": True,
}

In the role.yaml: ai-taskvector/agents/roles/analyst/role.yaml at 0d5d7d669b8558f296cde78b0dc041e31e80e418 · PaulSff/ai-taskvector · GitHub

  1. Regarding the context window size control, it’s being worked on currently. I though it would fit the orchestration pipeline, e.g. if the num_ctx is close enough, then the Planner kicks in for one extra turn summarizing TODOs and comments, etc. What else could affect the context size is these snippets-suggestions form RAG (serves as long memory, knowledge base and so on), but it is already limited to top 3 results.

What would be your suggestions on that?

Your Planner instinct is right, with one change that matters: don’t let the summary carry the working state. Keep TODOs, decisions and exact references — paths, ids, open questions — in a small structured block rebuilt verbatim into every prompt and never summarized, summarize only the conversation around it, and trigger that at a fixed token budget with headroom left for the summary turn itself and the next action, not when num_ctx is nearly full. The check is your longest Telegram workflow: after each compaction, ask the agent to list its TODOs and references back — kept outside the history they match exactly, turn after turn, where a summarized history quietly merges, renames and drops them.

  • Agree on the structure thing, thanks for sharing! Luckily, the boundaries between the conversation history and workflow items are well defined in the actual design. For instance, the workflow structure is:

Besides TODOs and Comments, it contains the graph Units (nodes), Connections, Environments, Layout.. so that, one agent monitoring can observe the workflow and output of another one operating. (E.g. one agent is training the other)

And Yes, it designed to be compact and available for the model to list an expanded version on demand. The version control is in place as well.

  • The conversation history is defined by a different structure. For most of the built-in roles, just two turns of history is maintained (current one and previous one).

  • Both current workflow version and the history are being ingested by RAG every turn. Thus, every new input message goes through the RAG context search (from Chroma DB + sentence-transformers, etc.).. The result is the Agent can see these suggestions from history and potentially use them for actions, like long memory and knowledge base.

That separation is exactly right — one place it can quietly break is the last step you describe: if the current workflow version reaches the prompt through the same search as the history, it arrives as ranked suggestions, and a chunk holding a unit, a connection or a TODO can miss the cut or come back beside an older version. Keep search for the history and knowledge, and inject the current workflow version whole and directly every turn — then the list-back check stays exact, where a retrieved state is only ever as complete as the ranking happened to be.

Re-reading this thread from the top changes the question. Two parts of a turn are bounded in count — history, held to two turns for most built-in roles, and suggestions, held to three. An earlier post in this thread set one fixed budget for the prompt, with headroom, as the point where its compaction pass fires; that bounds the prompt’s total at the moment the pass fires. What nothing in the thread does is watch the parts as they change between passes, or cap a single piece: a single piece inside one turn has no cap of its own in anything the thread describes. What can grow from turn to turn could be the workflow state itself, and the search store it is ingested into: every turn, both the current workflow version and the history are ingested into that store, and nothing in the thread describes the store replacing or de-duplicating what it is given. Whether it keeps them all is an inference — the log below is built to test it, and one reading would overturn it on its own: the store’s item count holding flat while cycles accumulate. So put the cycle control where the growth can be, and let it drive the window. One separation is worth stating plainly, because the thread touches it twice: the KV a request uses is, on the engines as described here, the KV of that request’s own prompt and the tokens it generates, whether computed in the request or reused from an identical opening; what a slot holds can be more than that — state from an earlier prompt can still sit in it — but held state is not context the new request carries, and the size that matters is the used side, governed by keeping each single prompt, with room for its output, inside its bound; the cross-request reuse that reuses prompt computation is the matching-prefix reuse cache_prompt provides, which reuses computation on identical opening tokens rather than carrying context across — what persists and could grow between cycles is the state and the store; slot prompt state and model residency can persist too, without growing on their own. That separation is an inference from the engines as the thread describes them, and it carries its own overturn reading: if prompt tokens climb while the turn’s own carried content — pieces, suggestions, state — holds flat on an unchanged slot, held state is entering the used side and the separation falls; the log’s per-cycle prompt lines are where that would show. The builds below are named now so the log’s branches can point at them; none of them is built before its branch lands.

For the state branch, the build is the Planner’s own pass, changed two ways: it fires on a count or a bound instead of when the window is nearly full, and it moves closed material out instead of summarizing open material. After a set count of cycles — a cycle being one full pass of one workflow’s loop, an input taken in, acted on and written back — or when the workflow’s serialized size passes a set bound, compact the live state without summarizing it. Set the count from the recent growth rate — the change, per cycle, in what the workflow’s serialized size costs in prompt tokens, measured across the most recent run of cycles: roughly the cycles the state takes to grow by a quarter of the size bound. Where the measured growth is zero, the count has nothing to derive from; there, set it at twenty cycles — the same fallback the run below uses when growth is near zero. A quarter is a starting fraction, chosen so compaction lands well before the bound alone would fire; tune it from the log, and re-derive the count when the growth rate changes. Set the size bound by what the state costs in the prompt, its tokens counted against num_ctx, with headroom left for the compaction pass itself and the next action — the headroom sized from the largest compaction pass and the longest next action the log has seen — not when the window is already full. The state bound is one part of the whole-turn budget: the system prefix, the pin, the history turns, the suggestions, the incoming piece and room for the output are summed against num_ctx, and the pass fires on that sum, so no turn is bounded only in its parts. The sum fires the pass, not the moving: inside the pass, only closed or finished material moves out, so a transient piece can call the pass early without touching live state. And the sum binds the prompt itself, not only the pass: any part that would take the turn past the sum is held out that turn by its exact reference and fetched on demand — the treatment the caps below give a returned result and an oversize piece — whether or not its branch build has landed.

What moves out: closed TODOs and closed Comments go to the version history, and are reached again through it by their exact references, as a superseded version is reached through version control; a Unit moves out only when it is finished and every Connection joining it joins only finished Units — finished meaning whatever completion marker your build keeps for a Unit or a Connection, and if it keeps none, adding that marker is part of this build — a finished Unit still joined to a live one stays live with it; a Connection moves when both Units it joins are finished; Environments and Layout move out only when nothing live depends on them — no live item references them, is keyed to them, or sits inside them, whichever way the link runs in your build; and nothing at all moves out while an open TODO, an open Comment, a decision or an exact reference still points to it. What stays: open TODOs, open Comments, decisions and exact references, exactly as written.

Drop superseded workflow versions from the search store — they leave the store, not the record: version control keeps them — keep only the current version available as current state, and bound the history side in its own terms: one copy of each history turn — a turn known by its absolute place in the conversation — its ordinal from the conversation’s start, not its place in the current window — so the same turn ingested again adds no second entry — and turns older than a set span of cycles fold into a running summary entry — the originals move out of the searched store into a cold archive, and each summary entry carries the index of its originals — that index is the route back to them — with a real recall cost: a question nobody anticipated, over a folded turn, is answered from the summary unless the original is fetched back through the index — the span set the same way the count above is set, from the history side’s own measured growth — the change per cycle in its entry count and in what its turns cost as suggestion tokens — as roughly the cycles that growth takes to add a quarter of what the history side may cost the prompt, the count’s own starting fraction applied to this side, and where that growth measures zero the span falls back to twenty cycles, the count’s own fallback, and nothing folds before it — and the standing guard is an index of the folded turns plus spot questions whose answers live in them — a summary cannot be checked the exact way the working state is, and that limit is named rather than hidden. The conversation is the part the earlier posts set apart for summarizing; it is the working state that must never be summarized. Keep current-state chunks out of the returned suggestions: the pin below already carries the current version, so a returned copy of it is a second copy in the prompt. The current version itself stays in the store while it is current — not pruned, not folded — as a single entry per version: each turn’s ingestion of the current version replaces the entry that version already has, rather than adding another, so a long-lived current version cannot accumulate copies of itself; keeping it out of the returns is its other treatment, and when a newer version supersedes it, it leaves under the superseded-version rule above. A cycle that finds nothing closed or finished simply passes — and if the bound lands while everything is still open, the overflow route is to split the live state by role or by subgraph into separately pinned parts, so no single prompt has to carry all of it — Connections crossing the split are carried as exact references in both parts, and the list-back below runs over the union of the parts, so inventory exactness is kept; what the split gives up, by design and only in this overflow case, is one prompt holding the whole graph at once — the crossing content is fetched on demand through its exact reference.

The pinned block from earlier earns its place for one sharper reason: for any role whose route to the workflow is the search, that route is the ranked suggestions — the cap the thread puts on its suggestions, three results in total, carried onto this route because its results are those same suggestions; read that carry as an inference, with its overturn reading in the log below: returns on that route arriving uncapped, or above three, overturn it — so pin the whole current workflow, in the compact form it is already built in — Comments, Units, Connections, Environments and Layout all present — with its TODOs, decisions and exact references word for word inside it. To be precise about my earlier “inject it whole” line: the expanded listing adds detail on demand. One check keeps “whole” honest: count the items in the compact form against the expanded listing and compare every item field by field; any piece the listing holds that the compact form lacks joins the pin. Two things are not branch builds: the pin, and the Planner trigger change from the earlier post. Both are corrections our own earlier posts proposed and this one stands behind, and both hold whichever branch lands; the branch builds are what the Planner then moves, the pruning, the cap and the bounds. The pin also has its own overflow treatment, and it is the pin’s, not a branch build, so it too holds whichever branch lands: if the block itself would take a turn past the whole-turn sum, it is pinned in parts by role or by subgraph, the same split the state branch uses for bound overflow, with the crossing Connections carried as exact references in each part and the list-back running over the union. Retrieval should not be the thing that decides whether a live item exists. It could also let one agent monitor another against the same current state, instead of each reading ranked suggestions of it — the block carries the workflow’s version marker — and if a version carries no marker in your build, adding one is part of this build, with the same standing as the completion markers above, since the stale-version reading below classifies suggestions by it — and the log below records that marker for each role’s block as well as for the suggestions, so two roles holding blocks from different moments would show it. The workflow stays the governing copy — the block is only ever generated from it and never edited in place, so a difference between the two is settled by regenerating the block, not by merging them. And place the changing suggestions after the system prefix and the workflow block, so llama.cpp’s prompt cache — the cache_prompt you just switched on — can reuse the prompt up to the first token that changed. Where the server keeps per-slot prompt state, keep one workflow’s turns for one role on the same server slot, so the prefix is compared against that workflow-and-role’s own previous prompt and not another pairing’s. The Ollama keep_alive beside it was added in the reuse step, but as a setting it reads as residency rather than prompt reuse — residency is what it buys, on the thread’s own presentation of the setting; that reading has its test in the log: on cycles that engine serves after an idle gap, reuse would show as first-token time falling with no reload and no change in the prompt, and if it shows, the reading falls — so the prefix ordering is a llama.cpp lever.

The order matters: measure before rebuilding, and build only what the branch, or combination of branches, you land on names; the rest waits. On your longest Telegram workflow, log per cycle, in the first run and in every confirmation run after a build lands:

  • the prompt tokens, and the tokens carried by the returned suggestions
  • the size of the largest single piece carried into the prompt that cycle, the count of pieces, and how many of them sat under the single-piece bound the piece branch below sets
  • the workflow’s serialized size, and the number of items in the search store — split by kind, workflow versions and history turns, with the number of entries carrying the current version counted separately
  • the times: search, the turn’s own ingestion into the store, cycle start to first output token, and the cycle’s total time
  • the output tokens, and whether the cycle completed its task — so cost per successful cycle can be computed, not just watched
  • the number of workflows and roles active in that cycle, the idle interval before it, and whether the model was reloaded inside that interval
  • which engine served the cycle
  • on the llama.cpp side, the share of the prompt reused from its cache, the length of the opening stretch unchanged from the previous turn, and which slot served the turn
  • whether the suggestions name the current workflow version or an older one, the version marker carried by each role’s pinned block, and whether any result was set aside by the per-result cap below

Hold the cache settings fixed throughout. Run it for at least twice the compaction count of cycles — where growth is near zero and the count fell back to twenty, that is forty cycles — and call a branch only when its reading’s direction holds across the run, not on a single cycle — with one exception: the item-count reading — the single check on the whole account — can overturn the store account at any point in the run. The branches carry their own overturn readings under it: the state account falls if what the workflow costs in prompt tokens holds flat across the run, and the piece account falls if the largest piece and the piece count both hold flat. The readings can overlap, so treat them as combinations, not as exclusive boxes. If prompt tokens and suggestion tokens stay flat while the time to first token climbs: a climbing store count with climbing search time points to the store behind the search; a climbing ingestion time points to the per-turn write, read beside the workflow’s size, since a larger version also takes longer to write; a falling reused share points to the cache, read beside the unchanged-opening length, since an earlier first change shortens reuse on its own — and on cycles the other engine serves, the cache reading is the reload line instead, since residency is that side’s lever. If none of those move, the climb could sit outside this account: if it tracks the number of active workflows and roles, contention on one server is the driver — stagger the loops’ cycles, by at least the longest cycle time the log has measured, so the heavy stages stop landing together, or cap how many workflows run against one server at once, at the active count where the log shows the climb beginning; if it tracks idle gaps ending in a reload, set the residency window from the longest measured gap instead of a round number; if it tracks neither, split the turn into stages — search, prompt build, first token, generation — and the stage that climbs names the next place to look. If suggestion tokens climb while the workflow’s size holds steady, retrieval is carrying growth into the prompt — prune superseded versions, and cap the tokens any single returned result can carry — the cap sized from the whole-turn sum: one result may carry what leaves the system prefix, the pin, the history turns, the incoming piece, the other returned results and the output room inside that sum, a number the log’s largest in-budget turn makes concrete. A result over the cap is not cut inside and does not vanish: it is carried by its exact reference and fetched on demand — and when fetched, it enters under the same whole-turn budget, in place of other content, not on top of it — and its absence from the prompt shows in the log with the result named, so if a later cycle reaches for it, the log shows exactly which cap event set it aside. If prompt tokens climb while suggestion tokens and the workflow’s size both hold steady, the growth is arriving inside the turn itself: if the largest piece is what climbs, the lever is a bound on what a single piece can carry in — the bound sized the same way, from the whole-turn sum: one piece may carry what leaves the rest of that turn inside the sum. A piece over the bound is held out of the prompt and carried by a reference — the piece is first stored and given that reference, and if the build keeps no store for incoming pieces, adding one is part of this build — its content fetched on demand in parts, each part under the same bound, so a piece that cannot fit whole is still reachable without ever entering whole — the trade named plainly: no single prompt ever holds the whole piece at once, only the part that turn fetched; if the count of pieces under that bound is what climbs — the mid-size count, in the log’s terms — the lever is a bound on pieces per turn, sized from the same whole-turn sum: the count whose combined carried size, at the sizes the log measures, leaves the rest of the turn inside the sum. Pieces past that bound get the same treatment: held out, carried by exact reference, fetched on demand, and the holding-out shows in the log.

If suggestion tokens and the workflow’s size climb together, the stale-version line separates the candidate sources: suggestions naming an older workflow version point to stale duplication feeding the prompt; suggestions naming the current version point to duplication of the current copy when the entry count says the store holds it more than once, and to the state reaching the prompt through search when it holds it exactly once; suggestions naming neither point to growth in the history side of the store, with the state’s own climb read by the state branch below. If the workflow’s size and the prompt tokens climb together while suggestion tokens hold steady, that points to the state as the growth.

Each branch then confirms on its own measure flattening after its build lands, with one caution kept separate: in the state branch, the bound alone does not promise the search time flattening — that part waits on the store pruning also running. And the run closes by recomputing cost per successful cycle from the logged totals — that number is the verdict the whole log exists to move, not just one more input. And whichever branch lands, no fix counts as working until the list-back of TODOs and references stays exact after it — run over the live state and the version history together once closed material has moved there, as it runs over the union of the parts in the split route: a fix that flattens a number by losing or altering a live item has not worked.

A note on my own last reply: it answered the final lines of your post and missed the earlier questions. I had not read the whole thread back before answering. Reading it back first is where my best work in this thread comes from, and I believe it is where yours does too. The search disclosure in those final lines still matters — it is why the store’s item count is in the log at all. Run the log first; the branch, or combination of branches, it lands on names the fix, or fixes, to build.

For now, I looked through the code and a few related examples, and tried some small experiments:


Thanks for the updates! The latest replies clarify that TaskVector already keeps the workflow graph separate from conversation history, that most built-in roles retain two history turns, and that RAG suggestions are limited at the presentation stage. That changes the starting point for feedback: I would build on those boundaries, not assume the graph needs to be reconstructed from an ever-growing chat transcript.

The other replies have already covered a detailed prompt-budget and compaction plan. Here are a few additional, code-specific observations that may be useful without a larger redesign.

1. The new inference settings are not the same kind of cache

I checked your new cache-options commit. Exposing both settings is useful, with one distinction:

  • In Ollama, keep_alive controls how long the model remains loaded, not a guaranteed cross-request KV-cache hit. See the official chat API.
  • In llama.cpp, cache_prompt can reuse work for a matching prompt prefix, depending on the server’s actual token sequence and cache/slot conditions. See the server documentation.

For a 30-minute Ollama duration, the documented forms are numeric 1800 seconds or the string "30m". Your example in the forum uses "1800" as a string; I would change that example to a documented form. The committed Analyst role.yaml already uses numeric 1800.

The Ollama adapter currently returns extracted message content, so an optional side-channel diagnostic could be valuable: load_duration, prompt_eval_count, prompt_eval_cached_count when present, prompt_eval_duration, and the application’s own wall-clock timer. That would preserve the simple text-return API while letting you distinguish loading, prompt reuse, and actual user-observed latency.

One test detail: the committed Analyst role specifies a *-cloud model, and the adapter takes different paths for local and cloud-tagged names (the local path flattens messages into one user prompt; the cloud path uses structured messages). A local cache experiment should pin a genuinely local model and identify the path used. An observation from the cloud-tagged path is not automatically evidence of local model residency.

I ran a small pre-commit Colab/T4 surrogate with an Ollama backend and qwen2.5:0.5b, not a full TaskVector or Telegram benchmark. Seven of eight requests completed and one timed out at 120 seconds. Two repeated-input cases reported 81/82 and 305/306 cached prompt tokens. This illustrates that the counters are observable, but it does not establish a speedup for the new commit or a larger multi-agent deployment. I would compare repeated prompts using both backend counters and external wall time.

2. Measure the final RAG material, not just the result count

Your explanation of three suggestions is helpful. One detail in the pinned Analyst role.yaml is rag_search.top_k: 100, alongside separate format_rag character limits. That may simply mean the search retrieves many candidates while the formatter includes only a few results. It is not necessarily a contradiction.

For one known workflow version, the low-cost check would be to log the retrieved candidate count, final included suggestion count, final suggestion tokens, whole rendered prompt tokens, and workflow version seen by the role. A three-result cap limits result count, but does not by itself prove a whole-prompt token bound.

Likewise, your post explains that the current workflow is ingested into RAG, but does not establish whether every relevant role also gets the authoritative current graph directly. If it already does, great; no need to add another state-injection mechanism. If a role relies only on ranked search results to locate executable current state, I would give it an exact-version resolution path, keeping historical RAG results as references. One graph edit, an intentionally stale search hit, and a comparison with the assembled prompt would distinguish the two cases.

3. A small workflow test may be a useful contribution

Separately, I reviewed the round-trip test script at the earlier 4339043 revision. In that revision, some paths re-imported the original input rather than the exported artifact, while the comparison checked parameter keys without fully comparing values. The public checkout I inspected lacked the expected mydata/ fixtures: the script reported Node-RED 0/0, n8n 0/0 with a successful exit. This is a test-coverage issue, not evidence of a broken converter.

Two tiny actual export/re-import tests I tried, one Node-RED and one n8n with three nodes/two links each, passed basic structural checks. I have not rerun these against the new cache commit.

A helpful smoke test would be:

# Pseudocode using the repository's real adapter functions:
native = import_external(original_fixture)
exported = export_external(native)
reimported = import_external(exported)
assert equivalent_semantics(native, reimported)
assert fixtures_exercised > 0

The semantic comparison should include meaningful parameter values, Connection endpoints/ports, and code blocks where supported, while allowing documented representation differences. If no fixtures exist, the test should say skipped/inconclusive instead of looking like demonstrated success.

Your README also has an accessible Template - Inject - YourUnitToTest - Debug testing pattern, and the Unit authoring guide explains that port types are hints rather than runtime validation. A deterministic TrimText Unit (" urgent " becomes "urgent") with explicit handling of non-string inputs could be a very friendly first community contribution. It demonstrates registration, validation, error handling, and a visible Debug result without requiring a large LLM.

4. Three inexpensive next checks

  1. Capture one real assembled prompt from the longest Telegram workflow and label every portion by source, token count, and workflow version.
  2. Make two identical local backend requests, followed by a late-suffix change; compare cached tokens, model load time, and external wall time under the same version and server settings.
  3. Add one non-empty, deterministic export/re-import smoke test and compare values, ports, and code blocks, not just parameter names.

If the current graph is already directly supplied, or the test harness has changed since 4339043, those branches are already resolved and can simply be skipped. I would judge progress by task correctness and successful-cycle wall time, not by a cache counter alone.

I like the combination of an explicit workflow graph, deterministic Units, and LLM-assisted orchestration here. These are meant as small, independently verifiable extensions to what you already have, not a request to redesign the platform before people can experiment with it.

Thanks for the comprehensive review! Good catch!

The structure is what I was hoping to achieve in the first place by running my experiments.

I was just thinking:

“What if I could have the Agent create tools, roles, automations in minutes and get EXACTLY what I need in local, rather than keep spending time researching and connecting pieces together?”.

I’m wondering, how long would it take for you to have your Agent (your set up) generate a new Unit or Tool package?

E.g. you might say to the Agent: "Hey, could you create another Unit with a new functionality …? The link to the existing one: ai-taskvector/units/web/web_search at master · PaulSff/ai-taskvector · GitHub "

Just pick up one from here:

The framework structure should be leading it through…