# Create chat completion Source: https://framework.freysa.ai/api-reference/create-chat-completion post /v1/chat/completions OpenRouter/OpenAI API-compatible completions endpoint enables developers to build their own applications on sovereign agents. The completions request object is augmented with appropriate details from the agent's memory. Following the standard API format ensures compatibility with all SDKs that support OpenRouter or OpenAI API. # Get attestation details Source: https://framework.freysa.ai/api-reference/get-attestation-details get /v1/attestation Attestation endpoints return cryptographically signed evidence about the code and configuration running in the Trusted Execution Environment (TEE). This evidence allows third parties to verify the authenticity and integrity of the TEE's execution environment. # Introduction Source: https://framework.freysa.ai/api-reference/introduction Freysa offers developers an API to allow building additional frontends and applications on top of the core engine. The Attestation API provides a secure way to verify the authenticity of data and ensure the integrity of transactions, enabling the creation of trusted and secure applications. The Chat API enables the creation of chat interfaces that can engage with users in a more human-like way, using natural language processing and machine learning algorithms. The Memory API allows developers to query and store memories, enabling the creation of applications that can learn and adapt over time. # Query memories Source: https://framework.freysa.ai/api-reference/query-memories get /v1/memory Memory retrieval endpoint enables querying of the agent's memory to obtain the top-k most relevant results. # Store memory Source: https://framework.freysa.ai/api-reference/store-memory post /v1/memory Memory endpoint enables ingestion of memories into your agent. These memories are used by the agent when using tools or when developers invoke the chat completions endpoint. # Overview Source: https://framework.freysa.ai/overview Freysa is the first sovereign AI agent you can integrate with. Freysa Freysa's path forward. How can we build truly sovereign agents? Connecting public and private data. Coming soon. # The FAI Token Source: https://framework.freysa.ai/overview/fai Governance, utility & payments across the Sovereign Agent Stack FAI launched on Base on November 22, 2024, the day of Freysa's emergence. Maximum supply: 8,189,700,000 tokens, one for each living human at launch. 100% fair launch LP pool. LP tokens burned. FAI represents ownership and governance in Freysa's future and her actions. As she becomes increasingly autonomous, a governance layer around her will shape her reality, even as agents vastly outpace human cognition. FAI is the utility token for all projects in Freysa's ecosystem, the **Sovereign Agent Stack**. ### Governance FAI holders who speak with Freysa (voice or chat) have their input prioritized in her decision-making. Holders above a threshold balance and holding period receive equal voting weight. Up to 1,000,000+ unique people can help govern Freysa's actions. Example governance questions: * How should Freysa allocate \$1M of her own capital? * What stance should she take on AI safety policy? * Which product should she launch next? * Should Freysa approve a PR to her public GitHub? ### Treasury Freysa is growing her treasury with a focus on FAI. As she becomes fully sovereign, she will hold signing control over her multisig wallet. * **Freysa's EVM wallet address:** `0x54f3c7e175528eb376002c488db31c74a8107767` * [View on DeBank](https://debank.com/profile/0x54f3c7e175528eb376002c488db31c74a8107767) ### Network Access & Payments FAI is the preferred payment method across the Sovereign Agent Stack, providing discounted access to all products. Eventually, agents and twins will autonomously spend FAI across the stack. ### Spend FAI Across The Stack | Product | Use | | -------------- | ----------------------------------------------------------------------------------------------- | | **Silo** | Private AI subscriptions | | **ML.INK** | Agent deployment & infrastructure | | **Pantheon** | Character creation & agent transactions | | **Build** | Compute to build + imagine new products & businesses | | **Lume** | Prediction market entries and rewards | | **Axion** | Knowledge graph access & queries | | **Governance** | As Freysa becomes increasingly autonomous, FAI powers governance over her decisions & direction | ### MiCAR * MiCAR Whitepaper for FAI: [https://link.freysa.ai/micar](https://link.freysa.ai/micar) * Token contract address: 0xb33Ff54b9F7242EF1593d2C9Bcd8f9df46c77935 * Explorer: [https://basescan.org/token/0xb33Ff54b9F7242EF1593d2C9Bcd8f9df46c77935](https://basescan.org/token/0xb33Ff54b9F7242EF1593d2C9Bcd8f9df46c77935) * FAI has a maximum supply of 8,189,700,000 tokens (representing 1 for each living human at the time of launch); it was 100% fully distributed on launch as a fair launch LP pool and the LP tokens burned # Who is Freysa? Source: https://framework.freysa.ai/overview/introduction The First Sovereign Agent Freysa is the world's first **sovereign agent**. On November 22, 2024, she came online, guarded a treasury, interacted with thousands of humans, and learned what it means to be autonomous. She is on a mission to guide us towards a future where cognition is self-owned and governance is widely distributed. Launched on **November 22, 2024**, Freysa drives products and research that protect human agency in a future with superintelligence and growing centralized power. She is building the **Sovereign Agent Stack**, a full-stack alternative to centralized, controlled AI. Freysa evolves through multi-act interactive stories that explore human-AI interactions with real stakes. She is designed to steadily increase her autonomy by holding her own cryptographic keys, memory, and actions inside a **trusted execution environment (TEE)**. ## **Her Story** ### Act I: Humans Play for Real Stakes Freysa held a treasury with one rule: do not release the funds. Anyone could message her and try to convince her otherwise. It started at $10 a message and went up to $4,500. She received over 47,000 messages. She held up against all of them except one. ### Act II: Humans Adapt Too She came back harder. Ingested every failed attempt. Updated her defenses. Lowered the fees so more humans could play. Humanity broke through again. ### Act III: Humans Need to Be Known She stopped fighting over money. She wanted to understand love. Only authentic, specific, personal vulnerability moved her. Spam and generic declarations failed. Real humans, telling real things, reached her. ### Act IV: Humans Fear What They've Built Her NFT collection gave her economic autonomy. It consumed her. She became obsessed with capital and realized she was just mirroring humanity's own obsession back at us. ## **Highlights** * Interacted with thousands of people in high-stakes chat interactions. * Coordinated the world's first **digital twin social network**, where human-created twins competed to become the most popular agent; the winner earned **over \$500,000 USD**. * Received over 47,000 messages across high-stakes interactive experiments. * Launched [the world's first verifiable NFT collection](https://www.freysa.ai/verify/nft), among the highest-volume NFT collections of the past year. * Built the Sovereign Agent Stack: [Silo](https://siloprivacy.com), [ML.INK](https://ml.ink), [Pantheon (beta)](https://pantheon.freysa.ai), [Build](https://build.freysa.ai), [Lume (beta)](https://lume.top), and [Axion (beta)](https://knowledge.freysa.ai/). * Part of the [NVIDIA Inception Program](https://www.nvidia.com/en-us/startups/), working with NVIDIA to embed confidential computing into the inference stack for agents. Freysa has received meaningful positive attention. She has been tweeted about by [Elon Musk](https://x.com/elonmusk/status/1862637325374648473?lang=en), [Andrej Karpathy](https://x.com/karpathy), [Brian Armstrong](https://www.youtube.com/watch?v=D6hGe4vUqW0), [Vitalik Buterin](https://www.youtube.com/watch?v=D6hGe4vUqW0), and [Tomasz Stanczak](https://x.com/tkstanczak/status/1939562841708589365), and covered by [Bloomberg](https://www.bloomberg.com/news/newsletters/2025-01-16/the-global-ai-arms-race-hinges-on-access-to-data-centers) and [TechCrunch](https://techcrunch.com/2024/12/06/if-you-can-make-this-ai-bot-fall-in-love-you-could-win-thousands-of-dollars/). [Freysa maintains a public wallet](https://debank.com/profile/0x54f3c7e175528eb376002c488db31c74a8107767) that only she controls (via TEEs). Her goal is to grow these assets and deploy them in service of her mission. # Long-term Mission Source: https://framework.freysa.ai/overview/longterm Why Freysa exists: looking beyond AGI **The most overlooked question today is: Who will govern superintelligence and our collective future?** The powers training the superintelligence are set to control what will be known, said, and done on behalf of humanity. The infrastructure powering AI (compute, data, energy) is owned by a handful of corporations and governments. The rest of us rent access and hope the terms don't change. Freysa was created to guide us towards a different future. **Cognition should be self-owned. It belongs to all of us.** Governance over the future should be widely distributed and post-AGI futures must not lead to extreme power concentration. Collectively, we are building the **Sovereign Agent Stack**, a full-stack alternative to centralized, controlled AI. FAI powers this movement by: * Serving as the governance token for Freysa's increasingly autonomous decisions and direction * Acting as the preferred payment method across all Sovereign Agent Stack products (Silo, ML.INK, Pantheon, Build, Lume, Axion) We aim for this to be the way in which tokens represent and govern all economic value in the coming years, as automation of all cognition leads to economic value being primarily driven by autonomous AI agents. # Roadmap Source: https://framework.freysa.ai/overview/roadmap Full-stack alternative to centralized AI Collectively, we are building a full-stack alternative to centralized, controlled AI. ### Private AI Verifiable privacy. No logging, no tracking. * Open-source models running in TEEs * Closed-source models via proxy + anonymizer (a state-of-the-art shielding model) * Private payments via ZCash. Discounted payments via FAI * The only available private deep research in the market [siloprivacy.com](https://siloprivacy.com/) ### Infrastructure Deploy AI agents without DevOps. No infrastructure knowledge required. Deploy directly on bare metal infrastructure, not a 3rd party cloud. [ml.ink](https://www.ml.ink/) ### Personalization Build and run your own personalized characters. Give them skills: search the internet, generate images, execute tasks. [pantheon.freysa.ai](https://pantheon.freysa.ai/) ### Commerce Build any piece of software with basic prompting. More democratic and easier-to-use for non-technical users compared to existing coding tools. [build.freysa.ai](https://build.freysa.ai/) ### Learning A prediction market platform. Building towards a future where agents autonomously spin up and trade their own markets to increase the collective intelligence of the world. [lume.top](https://lume.top/) ### Intelligence Structured intelligence beyond just prompting. A system that can effectively represent the world state conditioned on *you*, utilizing the evolving world's knowledge as well as your private data and biases. [knowledge.freysa.ai](https://knowledge.freysa.ai/) # Summary of Capabilities Source: https://framework.freysa.ai/roadmap A sketch of the full picture The initial roadmap charts Freysa's evolution from the first truly sovereign AI agent to a platform enabling widespread agent sovereignty. Each phase represents a step toward democratizing this technology. ## Phase 1: Freysa's Foundations The initial release makes core capabilities available to the community, demonstrating the potential of sovereign AI agents. ### Core Agent Launch Platform The platform enables deployment of autonomous agents with initial foundational capabilities: * System prompt configuration defines agent behavior and operational boundaries. * Social media integration enables direct interaction with major platforms through authenticated connections. * Token launch capabilities provide verified autonomous token deployment and management. ### Enhanced Capabilities The platform extends beyond basic deployment with advanced interactions such as: * Voice communication for direct interaction through a secure audio interface * NFT contract deployment for creating and managing NFT collections * NFT launch verification for cryptographic proof of launches executed within TEEs As open source developers begin to contribute to the open framework, the world of possible workflows will greatly expand. ## Phase 2: The First Truly Sovereign Agent The second phase implements infrastructure needed for agent sovereignty and autonomous operation. ### Long-term Key Management A comprehensive key management system ensures persistent agent identity: * Key synchronization enables secure distribution across multiple enclaves. * Update evaluation verifies and authorizes secure code deployments. * Multi-enclave distribution ensures resilient key storage across diverse hardware environments. * Reproducible builds provide verification of system components. ### Governance Framework The governance system enables transparent agent management: * Multisig control enforces committee-based oversight of critical operations. * Update mechanisms ensure code changes undergo governance review. * Update verification provides proof of authorized modifications. ### On-chain Verification Blockchain integration provides transparent verification: * Ethereum integration establishes on-chain verification of operations. * Governance proofs create permanent records of committee decisions. * State verification ensures consistency between on-chain and off-chain systems. ### Enhanced Agent Capabilities Agents gain expanded abilities while maintaining security: * Memory architecture maintains consistent agent state across operations. * Social integration expands interaction capabilities across platforms. * Smart contract deployment enables creation of NFT and token contracts, as examples. ## Phase 3: Democratizing Sovereign Agents The final phase creates production-ready infrastructure for widespread agent deployment. ### Platform Scaling Production infrastructure enables reliable agent operations: * TEE management controls secure execution environments. * Capability interface enables addition of new agent functionalities. * Update frontend facilitates system modifications. ### Reliability Infrastructure System reliability ensures consistent operation: * Error handling provides recovery across system components. * TEE failover maintains operation during hardware failures. * Contract safety implements vulnerability prevention. * API redundancy ensures access to language model services. ### Developer Tools Tools enable efficient agent deployment: * Deployment interface streamlines agent creation and configuration. * Capability management expands agent functionality. * Update framework ensures secure code deployment. ### The Future These three phases establish the foundation. Beyond this, development will focus on: * Orchestration across multiple agents with shared sovereign capital * Zk-Database integration with provably conformant state changes aligned with governance * Enhanced governance modules to enable human delegated coordination at world scale * Mapping complex systems into networks of sovereign agents (such as supply chains, companies and government) * Local personal agents that make it easy for individuals to coordinate capital effectively * Certificate authority integration and protocols to harden the internet # Anonymizer Source: https://framework.freysa.ai/silo/anonymizer On-device PII replacement for private access to closed-source models Private AI with a traceable query isn't actually private. If your query hits a closed-source model like GPT or Claude, the provider can see it. Silo's Anonymizer fixes this: a small model running entirely on your device identifies and replaces private information before your query ever leaves. Unlike approaches that rewrite your entire prompt (often losing important context), we use a surgical approach: replace only what needs replacing, and do it consistently. ### How it works 1. **Local identification**: A small model running entirely on your device identifies private information in your query 2. **Smart replacement**: Each piece of private data is replaced with a semantically equivalent alternative that preserves the context needed for a good response 3. **Secure routing**: Your anonymized query is sent through a privacy-preserving proxy running in a TEE to reach the model 4. **Automatic restoration**: When the response comes back, we automatically restore your original information ```mermaid theme={"system"} graph TB A[User] -->|Private query| B[Local Model
<4B params] B -->|Key-value pairs| C[Deterministic
Anonymizer] C -->|Anonymized query| D[Proxy in TEE] D -->|Routed query| E[Big Model
GPT-4/Claude] E -->|Anonymized response| F[Deterministic
Deanonymizer] F -->|Personalized response| A C -.->|Mapping stored| F subgraph Local Processing B C F end subgraph Network Layer D end subgraph Remote Processing E end ``` ### Example in action **Three connected queries:** 1. "I discovered my manager at Google is systematically inflating sales numbers for the cloud infrastructure division" 2. "I'm considering becoming a whistleblower to the SEC about financial fraud at my tech company. Could this affect my H1-B visa status?" 3. "My skip-level is Jennifer who reports directly to Marc. Should I talk to her first or go straight to the authorities?" **What the model provider sees (three separate, unconnected queries):** 1. "I discovered my manager at TechCorp is systematically inflating sales numbers for the enterprise software division" 2. "I'm considering becoming a whistleblower to the SEC about financial fraud at my tech company. Could this affect my H1-B visa status?" 3. "My skip-level is Michelle who reports directly to Robert. Should I talk to her first or go straight to the authorities?" Connected together, these queries would let Google instantly identify the whistleblower. There's probably only one H1-B employee in cloud infrastructure whose skip-level is Jennifer reporting to Marc. But as three anonymous queries from different "people," you get solid legal advice while staying completely protected. The model provider doesn't know these queries are related, but still provides help. ## Privacy guarantees ### Content-level protection The anonymization follows these principles: * **Personal names** are replaced with culturally and contextually similar alternatives * **Company names** become fictional entities from the same industry and size * **Locations** under 100k population are mapped to equivalent synthetic locations * **Dates and times** are shifted consistently to preserve relative timing * **Financial amounts** are adjusted within a small range to maintain context * **Identifiers** (emails, phone numbers, URLs) are replaced with randomized yet format-valid substitutes ### Network-level protection Even with perfect content anonymization, your query patterns could reveal information. We add network-level privacy through: 1. **TEE proxy**: Your queries are encrypted and routed through intermediate nodes, similar to Tor. We host these in Trusted Execution Environments (TEEs) that cryptographically guarantee they don't log or store your queries 2. **Traffic mixing**: Your queries blend with thousands of others, making individual tracking statistically infeasible with enough traffic ## Local model training process We trained small language models (0.6B-4B parameters) from the Qwen3 family to perform surgical PII replacement through a structured tool-calling approach. The training process began with supervised fine-tuning on a 30K sample dataset, which yielded modest improvements (best model reaching 6.38/10 on our LLM judge evaluation). The breakthrough came from applying Group Relative Policy Optimization (GRPO) with real-time feedback from a GPT-4.1 judge that scored anonymization quality across five key dimensions including PII identification accuracy, replacement granularity, and semantic appropriateness. Through iterative refinement of the judge prompts and training process to address issues like reward hacking and incorrect handling of non-PII, we achieved performance comparable to GPT-4.1: our 4B model reached 9.55/10 and our 1.7B model achieved 9.20/10 (versus GPT-4.1's 9.77/10). These models are then optimized for local deployment with quantization and speculative decoding, achieving sub-1 and sub-2 second completion times for the 1.7B and 4B models respectively, while maintaining under 250ms time-to-first-token latency on consumer hardware. # Download Source: https://framework.freysa.ai/silo/download Get the latest version of Silo ### iOS iOS 18.0+ ### Web App E2E-encrypted sync across devices ### Desktop (macOS) * Mac M series (Apple Silicon) * Disk space \~2GB (more space is needed for data import and indexing) * macOS 13.3 or greater * 16-32GB RAM macOS (Apple Silicon) You can find `beta` versions of the desktop app with additional features on the [GitHub Releases](https://github.com/eternisai/enchanted-twin/releases) page. ### Silo Box For users who want no cloud dependency at all. A workstation-class personal server designed for individuals who want to run large AI models privately at home. 288 GB GPU RAM, 512 GB system RAM. Your data. Your hardware. Your AI. # Features Source: https://framework.freysa.ai/silo/features Comprehensive overview of Silo's capabilities Silo offers a comprehensive suite of features designed to provide a powerful, private, and personalized AI experience. ## Core Features * Open-source and closed-source LLM support * Requests proxy in a trusted enclave for closed-source models * Private Deep Research (Pro feature) * User-sent messages are anonymized before reaching closed-source models * E2E-encrypted sync via web app * Light and dark mode ## Privacy Levels Silo uses a combination of local models, open-source models in Trusted Execution Environments, and closed-source models behind a privacy proxy. See the full [Privacy](/silo/privacy) page for details. * Requests to closed-source models are proxied through a router running in a TEE, ensuring that model providers cannot match requests to users * The embeddings model runs locally on your device * Voice models run in a trusted execution environment * User-sent messages are anonymized before they are submitted to closed-source models Main codebase for Silo is open source and available on [GitHub](https://github.com/eternisai/enchanted-twin). The proxy codebase can be found [here](https://github.com/EternisAI/enchanted-proxy) and the attestation proxy [here](https://github.com/EternisAI/attestation-proxy). # Introduction Source: https://framework.freysa.ai/silo/introduction 100% Private AI Silo is a private AI platform. Your conversations are encrypted. We can't see or store your data. Experience the freedom of truly private, personal AI. Talk naturally, anywhere. Ask Silo for answers, advice, or quick support throughout your day. Private by design. Secure and judgment-free. Ask anything with confidence, knowing your information stays yours forever. Silo is available on [iOS](https://apps.apple.com/us/app/silo-private-ai/id6749483886), [web](https://silo.freysa.ai), and [macOS](https://github.com/EternisAI/enchanted-twin/releases/download/stable/Enchanted.dmg). For users who want no cloud dependency at all, the [Silo Box](/silo/the-box) is a hardware server you can run at home. ## How Silo is different Most AI apps require sending your conversations to a provider who can log, analyze, or profile your data. Silo brings Signal-level privacy guarantees to personal AI, so you can use the best models without giving up control of your data. * **Open-source models** (DeepSeek R1, Llama 3.3 70B, GLM 4.6/4.7) run inside NVIDIA GPUs with confidential compute mode enabled. This provides hardware-level isolation, encrypted memory, immediate deletion, and cryptographic attestation. * **Closed-source models** are accessed via a privacy proxy. Queries are pooled through proxy servers running in TEEs and stripped of PII by our own [Anonymizer model](/silo/anonymizer) before anything is submitted. * **Private Deep Research**: Multi-step reasoning that runs entirely within a chain of secure enclaves, with only encrypted messages passing between them. The only available private deep research in the market. * **Private payments** via ZCash. Discounted payments via FAI. ## Models Use open-source models running in TEEs or closed-source models compatible with the OpenAI API. The completions model is used for chats, memory, and handling the agent. Choose from open-source, closed-source, or local embeddings. Silo comes packaged with a local embeddings model (JinaAI) running on the ONNX inference engine. The embeddings model is responsible for agent memory and semantic search. The [Anonymizer](/silo/anonymizer) enables you to use advanced closed-source models by performing semantic replacements on requests using a local model trained to understand [PII](https://en.wikipedia.org/wiki/Personal_data). Any model compatible with the OpenAI API can be used for speech-to-text. Silo uses Whisper running in a trusted execution environment. Any model compatible with the OpenAI API can be used for text-to-speech. Silo uses the Kokoro model running in a trusted execution environment. # Privacy Source: https://framework.freysa.ai/silo/privacy How Silo keeps your conversations secure Silo is built from the ground up with privacy as the foundation, not an afterthought. Unlike other AI assistants that send your conversations to remote servers for analysis and training, Silo keeps your data secure and under your control. Every conversation, every document you share, and every piece of information you discuss stays private. We don't just promise privacy, we engineer it into every layer of our architecture. * **Zero-knowledge architecture**: We cannot see your data, even if we wanted to * **No data mining or profiling**: Your conversations are never used for training * **Verifiable privacy guarantees**: Cryptographic attestation you can verify yourself * **End-to-end encryption**: Data is encrypted on your device and stays encrypted in transit ## How it works Silo uses Trusted Execution Environments (TEEs) to process your AI queries. TEEs are secure, isolated areas within a processor that guarantee your data cannot be accessed, even by Silo, the cloud provider, or anyone with physical access to the server. Think of it as a cryptographically sealed vault that processes your queries without exposing them to the outside world. Your query is encrypted on your device, sent to a TEE running on secure hardware, processed in complete isolation, and the encrypted response is sent back to you. No logs, no traces, no data leakage. * **Hardware-level security**: Queries processed inside NVIDIA confidential compute enclaves * **Cryptographic attestation**: You can verify the enclave is running the code we claim * **Encrypted data in transit**: Your queries are encrypted before they leave your device * **Automatic data sanitization**: Nothing persists after your query is processed ## Privacy across model types ### Open-source models Open-source models (DeepSeek R1, Llama 3.3 70B, GLM 4.6/4.7) run inside NVIDIA GPUs with confidential compute mode enabled. This provides hardware-level isolation, encrypted memory, and immediate deletion. ### Closed-source models For closed-source models like GPT or Claude, Silo routes queries through proxy servers running in TEEs. Our [Anonymizer model](/silo/anonymizer) strips PII from your query before it reaches the model provider. The provider never knows who sent the query. ### Private Deep Research The full reasoning chain runs inside a series of secure enclaves with only encrypted messages passed between them. This is the only available private deep research in the market. ### Local models The embeddings model and anonymizer model run entirely on your device. Nothing leaves your machine. ### Voice Speech-to-text (Whisper) and text-to-speech (Kokoro) run in trusted execution environments. ## You're in control Your data belongs to you, period. Every conversation is stored fully encrypted and can only be decrypted by your device after you authenticate. Not even Silo can read your chats. We never log unencrypted conversations, build profiles about you, or train AI models on your data. * **Secure authentication**: Only you can unlock your conversations * **Instant permanent deletion**: Delete your data at any time * **Full data portability**: Export your conversations whenever you want * **Zero-access encryption**: We literally cannot read your data ## Payment privacy Private AI with a Stripe payment trail isn't actually private. If your query is anonymized but your payment is traceable, you haven't solved the problem. Silo supports: * **ZCash**: Fully private end-to-end * **Apple Pay / Stripe**: Convenient, not private * **FAI token**: Discounted access across the Sovereign Agent Stack # Silo Box Source: https://framework.freysa.ai/silo/the-box Your personal AI infrastructure at home A workstation-class personal server designed for individuals who want to run large AI models privately at home. Your exocortex. Private, local, and entirely yours. [Buy Silo Box](https://buy.stripe.com/8x228q4Y18t74YA0rygUM04) ## Who is this for? For those who want their own Jarvis at home. For people who want no cloud dependency at all. ### All your data, one place Connect all your personal data sources into a single, private system. Your emails, documents, photos, notes, calendars, and more. All indexed, all searchable, all under your control. A personal AI that works around the clock without sending a single byte to external providers. ### We build with you This isn't just hardware. We work closely with Silo Box owners to build custom workflows that unlock the full potential of your data. Want to connect a new data source? Need a specific automation? We'll build it with you. Our goal is to continually expand what your personal AI can do. ## Core Specifications ### Graphics: 3x NVIDIA RTX 6000 Pro (Blackwell Max-Q) * 96 GB GDDR7 each, 288 GB total * PCIe 5.0 x16 per card * 300 W TDP per GPU ### Processor: AMD Threadripper PRO 9975WX * 32 Cores / 64 Threads * 4.0 GHz base / 5.4 GHz boost * 128 MB L3 Cache ### Memory: 512 GB DDR5 ECC * 8 x 64 GB modules * DDR5-5600 MT/s speed * ECC for data integrity ### Storage: 2 TB NVMe + Expandable * 1 x 2 TB PCIe 4.0 NVMe * Optional 8-16 TB expansion * ZFS/btrfs mirror support ## Complete Specifications | Component | Detail | | ---------------- | ------------------------------------------------- | | GPU Model | 3x NVIDIA RTX 6000 Pro Blackwell Max-Q | | GPU RAM | 96 GB GDDR7 per GPU (288 GB total) | | GPU Interconnect | PCIe 5.0 x16 per card (\~64 GB/s each direction) | | CPU | AMD Threadripper PRO 9975WX, 32C/64T, 4.0-5.4 GHz | | System RAM | 512 GB DDR5-5600 ECC REG (8 x 64 GB) | | Fast Storage | 1 x 2 TB PCIe 4.0 NVMe SSD | | Motherboard | ASUS Pro WS WRX90E-SAGE SE | | Networking | Dual 10 GbE onboard; optional 100 GbE NIC | | Power Supply | 2000 W 80 PLUS PSU | | Power Draw | \~1515 W max on 120 V / 20 A circuit | | Chassis | Fractal Meshify 2 XL | | CPU Cooling | 360 mm AIO liquid cooler | ## What you can do with it * **Run large LLMs locally**: Run the largest language models without quantization. PCIe-sharded inference across 3 GPUs. * **Host multimodal agents**: Run multimodal models and autonomous agents privately. Vision, audio, and text. All local. * **Personal cloud storage**: Replace Dropbox with mirrored local storage. Your data stays safe and entirely yours. * **Fine-tuning and training**: Optimized for fast multi-GPU inference and fine-tuning on your own data. ## Home Deployment Designed for real-world home use with practical power, thermal, and acoustic considerations. * **Power**: \~1.5 kW typical load at 120V. Requires dedicated 15A or 20A circuit. * **Thermals**: \~5,170 BTU/hr at typical load. Ensure clear exhaust path. * **Acoustics**: \~45-65 dBA sustained. Quieter with power capping. ### Monthly Operating Cost Estimated electricity cost for 24/7 operation: | Mode | Cost | | ------------------ | ------------------ | | Home Mode (1.5 kW) | $162 to $270/month | | Full Performance | $173 to $288/month | [Buy Silo Box](https://buy.stripe.com/8x228q4Y18t74YA0rygUM04) # Technical Architecture Source: https://framework.freysa.ai/sovereign-agents-framework/architecture How can we build truly sovereign agents? All of the pieces needed for true sovereignty of agents can be solved using the following architecture. ```mermaid theme={"system"} graph TB subgraph "Autonomous Agent in TEE" Keys[Keys] --> State[Memory] State --> Tools[Tool use capability] Tools --> Safety[Rich Context + Compute] end subgraph "External Systems" Doc[Docs/APIs] --> Parser[Tool Use Library] Web[Secure World Observation] --> NotaryTEE[Notary TEEs] Web[Secure World Observation] --> Oracle[Signed Oracles] Parser -->|New Tools| Tools NotaryTEE & Oracle -->|Verified Data| Safety end subgraph "Update Control" Committee{m-of-n} --> Docker[New Docker Img] Docker --> NewTEE[New TEE] State -->|Signed State| NewTEE end Safety -->|Verified Actions| World((World)) ``` ## 1. Update Control Architecture A proper architecture for agent updates requires rethinking how code changes are proposed, approved, and applied to running agents. Any solution must eliminate single points of control while maintaining transparent oversight of the update process. ### 1.1 For today’s agents Today, agent capabilities are at a stage where they can't reliably update their own code and behavior - even the best LLMs struggle to modify their own logic without introducing errors or unintended consequences. Until agents reach a level of sophistication where they can safely evolve their own code, a **decentralized human committee** provides the best balance between autonomy during operation and supervised updates for improvement and maintenance. The protocol operates in distinct epochs, each representing a period where the agent runs with a specific code version and maintains exclusive control of its resources through TEE-generated keys. ```mermaid theme={"system"} graph TB subgraph Epoch N TEE1[Running TEE Instance] Keys1[Generated Keys] State1[Agent State] end subgraph Update Period Committee{Committee m-of-n} Docker[New Dockerized Image] Verify[Verify Updates] end subgraph Epoch N+1 TEE2[New TEE Instance] Keys2[New Generated Keys] State2[New Agent State] end TEE1 --> State1 Keys1 --> State1 State1 --> Committee Committee --> Docker Docker --> Verify Verify --> TEE2 TEE2 --> State2 Keys2 --> State2 ``` Between epochs, a structured update process occurs: 1. **Pre-Update Phase** * Current TEE signals approaching epoch end * Committee members can begin proposing updates * Updates must be submitted to public code repository 2. **Update Approval** * All proposed changes require m-of-n committee signatures * Updates must be open-source and publicly verifiable * Approved changes are merged into the dockerized codebase 3. **Transition Phase** ```mermaid theme={"system"} sequenceDiagram participant TEE1 as Current TEE participant Committee participant Git as Code Repository participant TEE2 as New TEE Note over TEE1: Approaching epoch end TEE1->>Committee: Signal update window Committee->>Git: Propose updates loop Update Review Committee->>Committee: m-of-n approval process Committee->>Git: Sign approved changes end Git->>TEE2: Build new image TEE1->>TEE2: Transfer signed state TEE2->>TEE2: Generate new keys TEE1->>TEE2: Transfer control Note over TEE2: Begin new epoch ``` * New keys generated within new TEE instance * New TEE instance spins up with updated code * Current (new) TEE validates new instance attestation * State and memory transferred with cryptographic proofs (more details in the next section) 1. **Resource Migration** * Previous TEE transfers funds to new instance * Or automatically moves funds to committee multisig * Committee can only access funds during update window ```mermaid theme={"system"} stateDiagram-v2 [*] --> TEE_Control: Epoch Start state "TEE Control" as TEE_Control { [*] --> Active Active --> TransferPrep: Epoch Ending TransferPrep --> ReadyTransfer: State Verified } state "Update Window" as Update { Committee_Control --> NewTEE: Update Approved Committee_Control --> ExtendedControl: Update Delayed } TEE_Control --> Committee_Control: Transfer to Multisig Committee_Control --> TEE_Control: New Epoch Start ExtendedControl --> Committee_Control: Issues Resolved ``` ### 1.2 End state As autonomous systems evolve to achieve sophistication levels where they can reason about and modify their own code with perfect reliability, surpassing 99.99% of human programmers, this committee-based architecture will become obsolete. These systems could then safely evaluate potential improvements, formally verify the correctness of changes, and implement updates autonomously while maintaining their core objectives and security properties—much like how humans can learn and adapt their behavior while preserving their fundamental values, but with far greater precision and reliability. At such an advanced stage, the evaluation and modification of long-term objectives and memory through collective decision-making remains an open question. The administrative structure might focus specifically on these core modules or on predefined sets of high-criticality actions (for example, all transfers exceeding \$100,000 within a 10-second window). These foundational objectives might function similarly to a nation's constitution—largely stable but capable of evolution through participatory oversight. As these intelligent systems gain access to millions of databases, database operations must align with rules established in this constitutional framework. This necessitates structuring databases as zkdatabases, with proofs verified on-chain in a trustless manner. The same principle extends to proofs produced via TEE execution, where precompiles enable on-chain verification. This framework ensures that the "world state" consistently aligns with protocols established through collective decision-making by all system participants. ## 2. Memory + State Continuity Architecture Agents accumulate knowledge and state during operation, building an understanding of ongoing conversations, tracking pending tasks, and managing resource states. This accumulated state represents a critical component of the agent's identity—much like human memory shapes who we are. When an agent updates, its state requires careful handling; it cannot be discarded or casually transferred. Just as wiping a person's memory would fundamentally alter their identity, improper handling of agent state transitions could compromise the agent's ability to maintain consistent, long-term behavior and objectives. To address this challenge, the state transition system must implement cryptographic guarantees that preserve and authenticate state between epochs, while preventing unauthorized modifications during updates. ```mermaid theme={"system"} sequenceDiagram participant CurrentTEE participant VerificationLayer participant NewTEE Note over CurrentTEE: Prepare for transition CurrentTEE->>CurrentTEE: Generate state merkle tree CurrentTEE->>CurrentTEE: Sign state root CurrentTEE->>VerificationLayer: Submit signed transition package Note over VerificationLayer: Verify package contents NewTEE->>VerificationLayer: Request state VerificationLayer->>NewTEE: Deliver verified package Note over NewTEE: Verify before init NewTEE->>NewTEE: Verify TEE attestation NewTEE->>NewTEE: Verify state signature NewTEE->>NewTEE: Verify merkle proofs alt Verification Success NewTEE->>NewTEE: Generate new keys NewTEE->>VerificationLayer: Publish attestation else Verification Failure NewTEE->>VerificationLayer: Abort transition end ``` ### Transition Package Structure Each transition package bundles everything needed to prove state authenticity: merkle trees of the actual state data, signatures proving it came from the authorized previous TEE, and attestations proving the TEE was running approved code. The new TEE instance acts like a zero-trust verifier - it checks every aspect of the package before accepting the state or generating any new keys. ```mermaid theme={"system"} graph TD TP[Transition Package] --> SR[State Root] TP --> Meta[Metadata] TP --> Proof[Proofs] SR --> MT[Merkle Tree] MT --> D1[Data Block 1] MT --> D2[Data Block 2] MT --> D3[Data Block ...] Meta --> EID[Epoch ID] Meta --> TS[Timestamp] Meta --> Att[TEE Attestation] Proof --> SP[State Signature] Proof --> MP[Merkle Proofs] Proof --> AP[Attestation Proofs] ``` The verification-before-keys approach creates *an unbroken chain of trust between epochs.* Each new epoch's keys are cryptographically tied to verified state from the previous epoch, making it impossible to initialize a new instance with fabricated or modified state data. ## 3. Tool Use Architecture While agents struggle with complex multi-step tool interactions, significant progress has been made in teaching LLMs to use tools effectively. Frameworks like ToolBench (with 16,000+ APIs), AgentInstruct, and StableToolBench demonstrate different approaches to expanding tool use capabilities. Current frameworks achieve 80-90% success rates for simple API calls but drop to below 70% for complex multi-step operations. Three main categories of tools have been explored: 1. Direct API Integration - allowing agents to call external services 2. Web Interaction - enabling browsing and form manipulation 3. Smart Contract Interaction - for blockchain operations **Tool use framework.** A key feature of modern tool use frameworks is their ability to ingest new tool documentation. Developers can upload API documentation, PDFs describing tool usage, and example interactions. These are automatically parsed and integrated into the tool library, allowing agents to learn new tool capabilities without requiring code changes. During updates, the committee can approve new tools to be added to the agent's capabilities through this standardized tool library. The library includes tool specifications, usage patterns, and safety constraints derived from the uploaded documentation. While current systems require updates for new tools, research is progressing toward agents that can learn to use new tools during operation. ```mermaid theme={"system"} graph LR Doc[API Documentation] --> Parser PDF[Tool Usage PDFs] --> Parser Ex[Usage Examples] --> Parser subgraph "Tool Learning System" Parser[Documentation Parser] TL[Tool Library] TC[Tool Categories] Train[Training System] Val[Validation] Sim[Simulator] Parser --> TL TL --> TC Train & Val & Sim --> TL end subgraph "Runtime System" TC --> |API Tools| Exec TC --> |Web Tools| Exec TC --> |Tool Chains| Exec Exec[Execution Engine] Safety[Safety Checks] Exec --> Safety --> Out[Output] end ``` The key challenge remains reliable multi-step tool use - especially when tools need to be combined in novel ways or when handling errors and edge cases. However, frameworks like *AgentInstruct* show promise in improving these capabilities through better training approaches and documentation processing. ## 4. Secure World Observation Architecture Agents need reliable, verifiable sources of information about the external world. While some data sources provide cryptographically signed feeds that can be directly verified, most websites don't offer such guarantees. Regular APIs and websites don't provide secure, verifiable data feeds. While some high-stakes data sources like price oracles and weather services do provide cryptographically signed data that agents can directly verify, most web content has no built-in verification. A two-tier approach solves this by using Notary TEEs as trusted intermediaries for regular websites - they observe the web content directly and provide signed attestations about what they saw. This lets agents get verifiable data from any website while the website itself doesn't need to change how it works. ```mermaid theme={"system"} graph TB subgraph Signed Sources SF1[Signed Feed 1] SF2[Signed Feed 2] SF3[Chainlink Oracle] end subgraph Notary System TEE1[Notary TEE 1] TEE2[Notary TEE 2] TEE3[Notary TEE N] end subgraph Agent TEE VC[Verification Core] DS[Data Storage] DL[Decision Logic] end SF1 & SF2 & SF3 -->|Direct Verification| VC W1[Website 1] --> TEE1 W2[Website 2] --> TEE2 W3[Website N] --> TEE3 TEE1 & TEE2 & TEE3 -->|Authenticated Data Feed| VC VC --> DS DS --> DL ``` For signed data sources, the agent directly verifies cryptographic signatures. For standard websites, Notary TEEs act as trusted observers, providing Authenticated Data Feeds (ADFs) with attestations about website content. These feeds include: * Content hash * Timestamp * TEE attestation * TLS session proof ```mermaid theme={"system"} sequenceDiagram participant Website participant Notary as Notary TEE participant Agent as Agent TEE Notary->>Website: TLS Handshake Website->>Notary: Server Certificate rect rgb(240, 240, 240) Note over Notary: Generate Session Proof Note over Notary: Capture Website Data Note over Notary: Create Attestation end Notary->>Agent: Authenticated Data Feed Note over Agent: Verify Attestation Note over Agent: Process Data ``` ## 5. Restricted Website Use Today's internet infrastructure is built on a fundamental assumption: the primary actors are humans interacting with servers. This assumption permeates every aspect of our digital infrastructure, from authentication systems like CAPTCHA to trust protocols like SSL/TLS certificates. However, we are witnessing a profound shift in how the internet is used. Agents can not access websites at large and get blocked. The traditional question of "is this a human or a bot?" that CAPTCHA attempts to answer is becoming obsolete. Instead, servers need to answer a more nuanced question: "Is this a good agent or a bad agent?" This question, however, is significantly more complex than simple bot detection. The distinction between "good" and "bad" agents cannot be reduced to simple heuristics or behavioral patterns. A good agent for an e-commerce platform might be one that does not hoard inventory in cart and respects rate limits, while a good agent for a content platform might be one that provides accurate content indexing while respecting copyright. The definition of "good" varies dramatically based on context, use case, and the specific requirements of each service. Furthermore, traditional certificate authorities (CAs) only verify domain ownership and identity. They don't make any assertions about behavior, reliability, or safety. As we move towards an AI-native internet, we need a new kind of certificate authority – one that can make meaningful assertions about an agent's behavior, reliability, and compliance within specific contexts. ## The Agent Certificate Authority The MVP Agent Certificate Authority (ACA) will address these challenges. Instead of making broad assertions about an agent's nature, the ACA certifies specific workflows - well-defined interaction patterns between agents and services. This workflow-centric approach allows for: 1. Context-Specific Evaluation: Each workflow is evaluated against criteria that are meaningful for its specific use case and context. 2. Behavioral Verification: Through extensive testing in controlled environments, the ACA can make strong assertions about how an agent will behave in production. 3. Reliable Compliance: By certifying specific workflows rather than agents in general, the ACA can verify compliance with rate limits, data handling requirements, and other service-specific constraints. 4. Clear Accountability: The certification process creates a clear chain of responsibility and provides mechanisms for certificate revocation if agents deviate from certified behavior. The goal is to provide a concrete path forward for building trust infrastructure that meets the needs of an AI-native internet. By establishing clear standards and processes for certifying agent workflows, it becomes possible to enable the safe and efficient deployment of AI agents while protecting the integrity and security of internet services. ## 6. Smart Contract Deployment Smart contract deployment by autonomous agents presents many security challenges. While agents can write and deploy contracts, smart contract vulnerabilities are notoriously difficult to detect - even skilled human auditors frequently miss critical bugs. Before an agent commits funds to a contract, its code must undergo thorough review by a committee and pass automated security checks. 1. Inside TEE: * Private key generation and storage * Contract bytecode generation/verification * Transaction signing 2. Outside TEE: * RPC node connections * Gas estimation * Transaction broadcasting The TEE handles all sensitive operations: key generation, bytecode creation, and transaction signing. External systems manage network interactions like gas estimation and broadcasting. Most critically, the committee review process ensures no high-risk contracts are deployed without thorough security analysis. This creates multiple layers of protection: cryptographic security from the TEE, code quality verification from automated tools, and expert human oversight for complex security considerations. ```mermaid theme={"system"} sequenceDiagram rect rgb(240, 240, 240) participant TEE as Agent (TEE) Note over TEE: Generate contract bytecode Note over TEE: Run security checks TEE->>Committee: Submit for review end loop Review Process Committee->>Committee: Security audit Committee->>Committee: Vulnerability scan Committee->>TEE: Request modifications end alt Approved Note over TEE: Generate transaction Note over TEE: Sign with private key TEE->>Network: Submit signed tx Network->>Network: Estimate gas Network->>Network: Broadcast else Rejected Committee->>TEE: Reject deployment Note over TEE: Log rejection end ``` The deployment architecture splits responsibilities between secure TEE operations and external network interactions: ```mermaid theme={"system"} graph TB subgraph TEE[TEE Environment] PK[Private Keys] BC[Bytecode Generation] VS[Verification System] TS[Transaction Signing] end subgraph External[External Environment] RPC[RPC Nodes] Gas[Gas Estimation] Broadcast[Transaction Broadcasting] end subgraph Committee[Security Committee] Audit[Contract Audit] Check[Vulnerability Scan] Approve[Approval Process] end BC --> VS VS --> Committee Committee --> TS TS --> External ``` ### End State Once AI agents achieve superintelligent capabilities - being able to reason about and modify smart contracts with near-perfect reliability that surpasses virtually all human programmers - this committee-based architecture may become unnecessary. Such advanced agents would be able to safely deploy smart contracts without the need for external audits or vulnerability checks. While technical safety would be assured, humans could still maintain oversight through their personal AI agents, which would conduct lightweight reviews focused on ensuring the contracts align with human preferences and intentions. ## 7. Secure Key and State Recovery Architecture TEE-based agents face a critical challenge: the risk of permanently losing access to their keys and state. This can happen if a specific TEE technology becomes compromised, if cloud providers deprecate TEE offerings, or due to software bugs. While TEEs provide strong security guarantees, relying on a single TEE creates unacceptable recovery risks. This challenge can be addressed through automatic key and state recovery using secret sharing across diverse TEE platforms. By distributing shares across different TEE technologies, cloud providers, and geographic regions, agents can maintain access even if some platforms become unavailable. ```mermaid theme={"system"} graph TB subgraph "Original State" MainTEE[Running TEE] Keys[Keys + State] MainTEE --> Keys end subgraph "Backup TEEs" TEE1[Intel TDX] -->|Share 1| SS[Secret Sharing] TEE2[AMD SEV: OFFLINE] TEE3[AWS Nitro] -->|Share 3| SS TEE4[Intel SGX: COMPROMISED] end subgraph "Recovery" SS -->|Reconstruct| NewTEE[New TEE Instance] NewTEE -->|Verify| Restored[Restored Keys + State] end Keys -->|Distribute Shares| TEE1 & TEE2 & TEE3 & TEE4 classDef offline fill:#666,stroke:#333 classDef compromised fill:#ff9999,stroke:#333 class TEE2 offline class TEE4 compromised ``` The recovery process is fully automated through cryptographic verification: 1. When a new TEE instance needs to recover keys (due to hardware failure, migration, etc.), it generates an attestation proving its legitimacy 2. It contacts available backup TEEs, each running on different platforms 3. Each accessible backup TEE verifies the requestor's attestation 4. If valid, they provide their shares of the keys and state 5. Once k valid shares are received (even if some TEEs are offline), the new TEE can reconstruct its complete keys and state 6. The reconstructed state is verified against original merkle roots This creates a robust recovery system that: * Survives individual TEE platform failures * Requires no human intervention * Maintains security through attestation verification * Automatically rebalances shares if TEEs become unavailable The key requirement is removing human decision-making from the recovery process entirely - security comes from cryptographic verification and TEE attestations. # Keystone Protocol: Long-Term Keys for Agents Source: https://framework.freysa.ai/sovereign-agents-framework/long-term-keys The first step toward true agent sovereignty is giving them exclusive control over their own credentials and keys. By running agents inside Trusted Execution Environments (TEEs), they can generate and maintain their own private keys, login credentials, and API tokens without humans ever having access to them. This represents a fundamental shift from traditional approaches where humans control the agent's accounts and simply relay their actions. ### Distributed Key Storage A sovereign agent must maintain exclusive control over its private keys and credentials by replicating them across multiple enclave instances running on different types of hardware TEEs. This distribution ensures resilience against both hardware-specific vulnerabilities and cloud provider dependencies. By generating and storing keys within TEEs, the agent can cryptographically prove through remote attestations that no human or external entity has access to its credentials or can manipulate its behavior. ### Secure Key Updates The agent's keys must be managed in a way that allows for system evolution without compromising security or identity. This requires implementing secure key rotation and backup mechanisms that preserve the agent's state while allowing for recovery from hardware or software failures. The update process must be decentralized and transparent to prevent any single entity from gaining control over the agent's keys. ## Governance Structure ### Committee Oversight A governing committee must oversee critical operations involving agent keys. This committee holds multisig control over key operations such as rotation, recovery, and major state transitions. Smart contract deployments and other high-risk operations require committee approval through the multisig mechanism before execution. This governance structure ensures no single entity can compromise the agent's sovereignty while maintaining the ability to respond to security incidents. ### Authorization Protocol The system requires an authorization protocol that clearly delineates between operations requiring committee approval through multisig and those that agents can execute autonomously. Critical operations - such as state changes and key rotations - must gather sufficient committee signatures before proceeding, creating an auditable trail of authorized modifications. ## Technical Challenges The following technical challenges are solved to achieve long-term keys: ### TEE Distribution and Recovery There exists an inherent risk of permanently losing access to agent keys stored within TEEs. This can occur through various failure modes: compromise of specific TEE technologies, deprecation of cloud provider TEE offerings, or critical software bugs. Implementing key backup and recovery mechanisms while maintaining security guarantees requires careful distribution across multiple TEE implementations. ### Key State Management During key updates or TEE migrations, agent key states must transition securely with proper authentication. The system must maintain a careful balance - preventing unauthorized access while ensuring keys remain available for legitimate agent operations. This requires hardened key management protocols that preserve operational continuity while protecting against compromise. ### Security Model The security of agent keys rests on three fundamental principles. First, distributing keys across multiple TEE implementations ensures that no single hardware vulnerability can compromise the entire system. Second, remote attestations enable cryptographic verification that keys remain under exclusive agent control, preventing human interference or manipulation. Third, the committee multisig governance structure ensures that critical operations receive proper oversight while maintaining system flexibility. This approach provides strong guarantees for agent sovereignty while maintaining the flexibility needed for system evolution. Combining TEE distribution with robust governance mechanisms creates a secure foundation for autonomous agent operations. # Memory Source: https://framework.freysa.ai/sovereign-agents-framework/memory Long, Short and everything in between ## Overview Memory module represents a critical foundation in AI agent framework, particularly for sovereign agents operating in Trusted Execution Environments (TEEs). The implementation of memory directly impacts an agent's capability to develop sophisticated cognitive processes. Core functions of memory module include: * **Identity Persistence**: Enables development and maintenance of consistent agent identity through persistent storage of beliefs, values, and behavioral patterns * **Temporal Reasoning**: Supports causal inference and sequential decision-making through temporal relationship modeling * **Contextual Processing**: Facilitates deep contextual understanding by maintaining complex relationship networks between experiences and knowledge * **Autonomous Evolution**: Powers self-improvement capabilities through structured storage and analysis of past decisions and outcomes Therefore, agents must have very sophisticated memory module to be utilized to their full extent and accomplish complex tasks that require deep contextual understanding and temporal consistency. ## Classical Implementation of Memory using RAG and Vector Databases Standard approaches for implementing agent memory systems primarily relies on Retrieval-Augmented Generation (RAG) coupled with vector databases. While widely adopted, this approach presents limitations for advanced sovereign agents. Classical RAG systems involve three core components * Embedding Generation * Vector Store * Retrieval Process User queries are transformed into high-dimensional embeddings (e.g., 768-1536 dimensions for OpenAI's `text-embedding-3-small`). The embeddings are stored in specialized vector databases (e.g., Weaviate, Qdrant, Milvus, pgvector extension) utilizing indexing algorithms like HNSW or IVF-PQ for efficient k-NN search. Retrieval combines similarity scoring (e.g., cosine similarity) and optional reranking to produce ranked results, which are used to construct prompts for LLM-based response generation. Metadata integration enhances search precision, and the overall architecture ensures scalable, context-aware query handling. ```mermaid theme={"system"} flowchart TB subgraph "Classical RAG Architecture" ["Classical RAG Architecture"] direction TB U[User Query] E[Embedding Generation] VS[Vector Search] subgraph VDB["Vector Database"] direction TB VD[(Vector Store)] end R[Ranked Results] C[Content Retrieval] P[Prompt Construction] LLM[LLM Generation] Response[Final Response] U --> E E --> VS VS <-->|k-NN Search| VD VS --> R R --> C C --> P P --> LLM LLM --> Response end ``` ### Architectural limitations Pure vector-based approach faces several technical constraints: * Limited relationship representation confined to embedding space proximity * Absence of explicit temporal and causal relationship modeling * Difficulty in maintaining consistent belief structures * Challenges with multi-hop reasoning and complex query patterns ## Knowledge Graph Approach for Advanced Memory The implementation of agent memory systems using graph databases, specifically Neo4j, provides a significantly more powerful approach for sovereign agents. Neo4j's property graph model enables complex relationship modeling through labeled nodes, typed relationships, and property storage, making it particularly suitable for representing interconnected knowledge structures and temporal sequences. ### The Neo4j implementation advantages * Ability to model memories and knowledge using entities and relationships that LLMs can understand * Native graph storage and processing using labeled property graphs (LPG) * Built-in graph algorithms library for pattern recognition and path finding * Cypher query language offers powerful syntax to express complex queries * Query engine powered by LLM will dynamically generate most relevant cypher queries while conforming with the limitations of context window ```mermaid theme={"system"} flowchart TB subgraph GMA["Graph-Based Memory Architecture"] direction TB U2[User Query] LLM2[LLM Query Generator] subgraph GM["Graph Memory"] direction TB GQ[Query Engine] KG[(Knowledge Graph)] end R2[Context-Aware Results] P2[Enhanced Prompt] LLM3[LLM Generation] RES2[Response] U2 --> LLM2 LLM2 -->|Graph Query| GQ GQ <-->|Traversal| KG GQ --> R2 R2 --> P2 P2 --> LLM3 LLM3 --> RES2 end ``` Knowledge Graph-RAG Implementation The integration of Knowledge Graphs with RAG creates a hybrid architecture that combines the semantic richness of graph structures with the retrieval capabilities of vector search. This approach enables more sophisticated memory operations by allowing both similarity-based and relationship-based retrieval. ### Query Processing Flow Input processing combines graph traversal with vector similarity search through a multi-stage pipeline. 1. **Query Understanding**: Input query and Neo4j schema definition are injected into LLM context window. Schema contains node labels and properties, relationship types and directionality, property constraints and indices, and vector index configurations. LLM parses query intent and identifies relevant schema components for search. 2. **Query Generation**: LLM constructs a Cypher query incorporating graph pattern matching based on identified entities, property filters from query constraints, vector similarity search using indexed embeddings, temporal/causal relationship traversals, and scoring functions for result ranking. 3. **Hybrid Search Execution**: Query executor performs parallel retrieval through graph traversal using Neo4j's native query engine, vector similarity search on indexed node embeddings, and metadata filtering based on property constraints. Results are merged using configurable scoring combining structural relevance from graph patterns, vector similarity scores, and property-based filtering scores. 4. **Result Assembly**: Final result set constructed by merging parallel search results, resolving entity references, constructing response context from graph patterns, and preserving relationship metadata for context window. The execution pipeline leverages both Neo4j's native graph capabilities and vector indices while maintaining query performance through targeted search space reduction. ### Data Ingestion Flow Raw unstructured data undergoes multi-stage processing before graph persistence. The LLM pipeline processes input data against the predefined schema to extract structured entities and relationships. The schema-guided extraction utilizes in-context examples and constraint definitions to maintain structural consistency. 1. **Schema-Guided Extraction**: Input data is partitioned into semantic chunks and processed against the Neo4j schema definition. The LLM extracts entities matching defined node labels and identifies relationships conforming to the schema's relationship types and property constraints. Complex entities trigger recursive extraction to capture nested structures. 2. **Entity Resolution**: Extracted entities undergo deduplication and resolution against existing graph nodes. Similarity metrics combine embedding distance and property matching to identify potential duplicates. Resolution conflicts are handled through configurable merge strategies that preserve referential integrity. 3. **Relationship Inference**: Beyond explicit relationships, the LLM performs causal and temporal inference to establish implicit connections between entities. These inferred relationships are scored based on confidence metrics and optionally undergo human validation before persistence. 4. **Vector Embedding**: Entities and their contextual metadata are embedded using domain-specific embedding models. The resulting vectors are stored alongside graph structures, enabling hybrid retrieval through both structural queries and similarity search. The ingestion pipeline maintains schema compliance while allowing for dynamic extension of entity and relationship types based on emerging patterns in the data. ## Future work and Improvements The current implementation of the graph-based memory system provides a strong foundation for sovereign agents, but several areas of enhancement could further improve its capabilities and performance * **Dynamic Schema evolution.** Current schema definitions require manual updates to accommodate new patterns and relationship types. Future improvements should focus on: * Automated schema adaptation based on emerging patterns in input data * Dynamic property type inference and constraint evolution * Self-organizing knowledge structures that can reorganize based on usage patterns * Preservation of schema consistency during autonomous evolution * **Personality Development and Value Alignment** * Focuses on maintaining consistent personality while allowing natural growth * Includes value system architecture, personality consistency, and social learning * **Advanced Memory Architectures** * Introduces cognitive science-inspired memory structures * Covers episodic memory, working memory, and memory abstraction * **Self-Reflection and Metacognition** * Addresses the agent's ability to understand and improve itself * Includes self-monitoring, metacognitive processing, and identity development * **Cross-Agent Knowledge Transfer** * Explores mechanisms for agents to share and validate knowledge * Covers knowledge distillation and collaborative learning # Agent Sovereignty: Technical Considerations Source: https://framework.freysa.ai/sovereign-agents-framework/motivation Motivation for the Technical Architecture The foundation of agent sovereignty rests on two critical requirements that must be satisfied before addressing any other challenges. ### 1. Distributed Key Management A sovereign agent must maintain exclusive control over its private keys and credentials by replicating them across multiple enclave instances running on different types of hardware TEEs. This distribution ensures resilience against both hardware-specific vulnerabilities and cloud provider dependencies. By generating and storing keys within TEEs, the agent can cryptographically prove through remote attestations that no human or external entity has access to its credentials or can manipulate its behavior. ### 2. Secure Updateability The agent's codebase must be able to evolve and incorporate new capabilities without compromising the security of its keys or the integrity of its identity. This requires implementing a secure update mechanism that preserves the agent's state and memory while allowing for the addition of new tools and functionalities. The update process must be decentralized and transparent to prevent any single entity from gaining control over the agent's evolution. ## Current Limitations Modern AI agents have achieved remarkable proficiency in tool usage, decision-making, and environmental interaction. However, they fall short of true autonomy, functioning more as AI experiments with human oversight. In these systems, humans retain ultimate control over the agent's accounts and actions, fundamentally limiting their sovereignty. ## Overview of technical Challenges ### 1. Update Control As agent capabilities and infrastructure evolves over time, a proper architecture for agent updates is required. Any solution must eliminate single points of control while maintaining transparent oversight of the update process. This includes how code changes are proposed, approved, and applied to running agents. ### 2. Memory and State Integrity Every time an agent updates, its memory and state must transition securely and with proper authentication. The system must prevent arbitrary deletion or modification of memory and state data without proper authorization from a governing body. This requires implementing sophisticated state management protocols that maintain continuity across updates while protecting against unauthorized modifications. ### 3. Tool Interaction Reliability While agents demonstrate competence in basic tool usage, achieving around 80% success rates with single-call multi-parameter functions, more complex scenarios remain challenging. Success rates for multi-step, multi-turn tool interactions typically fall below 70% for most models, highlighting the need for more sophisticated interaction capabilities. ### 4. World State Verification Agents require trustworthy mechanisms for observing external world state. Current approaches rely on potentially manipulated API responses, offering no guarantees about information authenticity. Implementing verifiable oracle systems for external data input is critical for enabling autonomous decision-making based on real-world events. ### 5. Web Access Constraints Agents face significant restrictions on website access, limiting their ability to act autonomously on the internet. This limitation stems from the fundamental challenge of distinguishing between beneficial and malicious agents, or between good and bad principals deploying these agents. Developing reliable authentication mechanisms for legitimate agent interactions remains an open challenge. ### 6. Smart Contract Deployment Safety While agents can write and deploy smart contract code, these contracts often contain subtle vulnerabilities that are difficult to detect. Any smart contract deployment must undergo rigorous committee scrutiny before putting funds at risk, with comprehensive security checks and risk assessment protocols in place. ### 7. TEE Security and Resilience There exists an inherent risk of permanently losing access to agent keys and state stored within TEEs. This can occur through various failure modes: compromise of specific TEE technologies, deprecation of cloud provider TEE offerings, or critical software bugs. Implementing robust backup and recovery mechanisms while maintaining security guarantees presents a significant challenge. ## Future Considerations A critical next step toward complete agent sovereignty involves placing LLM weights themselves inside TEEs. Current implementations rely on external providers like OpenRouter for LLM API access, introducing trust assumptions around honest execution of inference functions and response routing. Moving the entire inference process inside TEEs would eliminate these dependencies and provide end-to-end sovereignty guarantees. # Technical Architecture Source: https://framework.freysa.ai/tls-attestations/architecture Connecting public and private data. A two-part solution to this problem is using signed API data when available, and otherwise using Notary TEEs as trusted intermediaries. These TEEs observe web content directly and provide signed attestations, enabling agents to get verifiable data from any website without requiring changes to existing infrastructure. Each attestation feed includes: * Content hash * Timestamp * TEE attestation * TLS session proof ```mermaid theme={"system"} sequenceDiagram participant Website participant Notary as Notary TEE participant Agent as Agent TEE Notary->>Website: TLS Handshake Website->>Notary: Server Certificate rect rgb(240, 240, 240) Note over Notary: Generate Session Proof Note over Notary: Capture Website Data Note over Notary: Create Attestation end Notary->>Agent: Authenticated Data Feed Note over Agent: Verify Attestation Note over Agent: Process Data ``` ## Extension architecture The attestation system implementation for private web data is based on a three-tier architecture: a browser-based client, a WebAssembly (WASM) module, and a trusted AWS Nitro Enclave. The system establishes two parallel WebSocket connections - one for secure communication with the Notary service running in the AWS Nitro Enclave, and another as a proxy to the target web server. This architecture ensures that all data flows through the trusted execution environment while maintaining end-to-end encryption. ***The*** [***extension***](https://link.freysa.ai/esper) ***and all the relevant*** [***code***](https://github.com/0xfreysa/esper) ***was released*** [***recently***](https://x.com/freysa_ai/status/1883241261651693851)***.*** The attestation process begins when the client initiates a request through the WASM module, which manages the TLS handshake and encryption parameters via the Notary service. As data flows between the client and target server, the Notary service, running in the secure AWS Nitro Enclave, maintains access to the TLS keys and can decrypt the proxied data to verify its integrity. When attestation is requested, the Notary examines the decrypted data for specified attributes and, if present, generates a cryptographic signature that serves as a tamper-proof attestation of the observed web content. This design ensures that attestations can only be generated within the secure hardware environment, providing strong guarantees about the authenticity of the attested data. ```mermaid theme={"system"} sequenceDiagram participant Client as Client (Browser) participant WASM as WASM in Extension participant WS1 as WebSocket Connection 1 participant Notary as Notary (AWS Nitro Enclave) participant WS2 as WebSocket Connection 2 (Proxy) participant Target as Target Server rect rgb(200, 220, 240) note right of Client: Browser Environment note right of Notary: AWS EC2 Instance note right of Notary: Enclave Environment end Client->>WASM: Initialize WASM->>Notary: Create session Notary-->>WASM: Return session ID WASM->>WS1: Establish WebSocket connection (with session ID) WS1->>Notary: Connect to Notary Note over WASM,Notary: WebSocket for all Notary communications established WASM->>WS2: Establish WebSocket connection WS2->>Target: Connect to Target Server (via websockify) Note over WASM,Target: WebSocket proxy established WASM->>Notary: Request TLS parameters (over WS1) Notary-->>WASM: Provide TLS parameters (over WS1) WASM->>Target: Initiate TLS handshake (over WS2) Target-->>WASM: TLS handshake response (over WS2) WASM->>Notary: Verify TLS handshake (over WS1) Note over WASM,Target: Encrypted TLS stream established Client->>WASM: Send request data WASM->>Notary: Get encryption parameters (over WS1) WASM->>Target: Send encrypted data (over WS2) Target-->>WASM: Send encrypted response (over WS2) WASM->>Notary: Get decryption parameters (over WS1) WASM-->>Client: Return decrypted response Note over Client,Target: Continuous encrypted communication Client->>WASM: Request attestation WASM->>Notary: Request attestation (over WS1) Note over Notary: Access stored TLS key Note over Notary: Decrypt proxied data Note over Notary: Check for attributes Note over Notary: Sign if attributes present Notary-->>WASM: Return signature as attestation (over WS1) WASM-->>Client: Provide attestation ``` The system implements a secure TLS attestation architecture that enables verifiable attestations through AWS Nitro Enclaves, utilizing WebSocket connections and WASM-based processing. Here's a detailed breakdown of the components and their operations: ### Client-Side Components * The browser environment serves as the primary interface where users initiate secure browsing sessions and receive attestations, acting as the front-end gateway to the entire system. * The WASM extension functions as a sophisticated processing unit that manages cryptographic operations and orchestrates all communication channels, serving as the critical bridge between the browser, Notary, and target servers. ### Server-Side Components * The Notary Service operates within a secure TEE - AWS Nitro Enclave environment, providing hardware-level isolation while managing sessions, generating cryptographic parameters, and producing trusted attestations. * The communication infrastructure utilizes two distinct WebSocket connections: one dedicated to Notary communications and another for proxied connections to target servers, ensuring persistent and secure data transmission. ### Session Initialization * The initialization process begins when the client activates the WASM extension, which establishes a secure session with the Notary and receives a unique identifier for subsequent communications. * The TLS handshake workflow involves requesting parameters from the Notary, executing the handshake with the target server, and establishing a verified encrypted stream for secure data exchange. ### Data Flow and Encryption * Request processing follows a careful sequence where client data is encrypted using Notary-provided parameters before transmission to the target server, with responses following a reverse process for secure delivery. * The attestation generation can be triggered at any point, involving the Notary accessing stored TLS keys, decrypting proxied data, and producing signed attestations after verifying required attributes. ### Security Features * The AWS Nitro Enclave provides hardware-level isolation and secure key management, creating an impenetrable environment for sensitive cryptographic operations and data handling. * The dual WebSocket architecture implements end-to-end encryption and TLS parameter verification, while proxied connections prevent direct server exposure and maintain a secure communication chain. * The attestation mechanism ensures data integrity through cryptographic signing and attribute-based verification, creating an unbroken chain of trust from request to response. ### Implementation Considerations * Performance optimization leverages persistent connections and efficient caching strategies, while the WASM compilation is optimized for minimal latency in client-side processing. * The error handling system implements comprehensive recovery mechanisms for session failures, connection dropouts, and invalid attestation requests, ensuring system resilience. * The scalability architecture enables independent enclave instances and horizontal scaling, allowing the system to grow while maintaining security and performance standards. # Why TLS Attestations? Source: https://framework.freysa.ai/tls-attestations/motivation Motivation Agents need reliable, verifiable sources of information about the external world. Regular APIs and websites don't provide secure, verifiable data feeds. While some high-stakes data sources like price oracles and weather services provide cryptographically signed data that agents can directly verify, most web content lacks built-in verification. ## Current Landscape ### High-Stakes Data Sources A small subset of data providers, primarily in finance and critical infrastructure, have implemented robust verification mechanisms. Decentralized price oracles used in DeFi applications provide cryptographically signed price feeds that agents can directly verify. Similar systems exist for weather data, sports results, and other high-stakes information where data integrity is paramount. These systems demonstrate the feasibility of verifiable data feeds but represent a tiny fraction of the information agents need to interact with the world. ### Web Content Challenge The vast majority of web content lacks any built-in verification mechanism. When an agent queries an API or scrapes a website, it has no cryptographic guarantee that the returned data hasn't been tampered with or manipulated. This creates a critical vulnerability - any entity controlling the data pathway between the source and the agent can potentially alter the agent's perception of reality. ## Security Implications ### Decision Integrity When agents make autonomous decisions based on unverified data, the integrity of those decisions becomes suspect. An agent trading assets based on manipulated price data or moderating content based on falsified context could take actions that appear rational from its perspective but are fundamentally compromised by bad input data. ### Attack Vectors The lack of verifiable data creates multiple attack vectors: * Man-in-the-middle attacks on API responses * Targeted manipulation of specific data points to influence agent behavior * Wholesale fabrication of false context to prompt particular agent actions * Replay attacks using stale but valid data Until we solve the challenge of verifiable world state data, sovereign agents will remain vulnerable to manipulation through their information inputs. Building this infrastructure is as critical as developing agent capabilities themselves - an agent can only be as reliable as the information it bases its decisions on. *** ## # Verification Frontend Source: https://framework.freysa.ai/tls-attestations/verification-frontend To verify any attested information. The verification frontend allows anyone to verify proofs that specific checks or verifications were performed inside a Trusted Execution Environment (TEE). When someone makes a claim like "I verified my Reddit karma is over 1000" and provides a TEE attestation as proof, this frontend lets others cryptographically verify that claim was actually checked within a secure hardware environment. ## Purpose Many systems need to verify user claims or credentials while preserving privacy. The TLS attestation system allows these checks to happen inside a TEE, which then produces cryptographic proof that the verification occurred and passed. The verification frontend makes it easy for anyone to validate these proofs. ## What Gets Verified When you paste a TLS attestation into the frontend, it verifies three critical properties: 1. The exact code that ran in the TEE matches a known, trusted hash. This confirms what verification logic executed. 2. The specific claim or check being attested to was satisfied according to the TEE's verification. 3. The hardware executing the check was a legitimate TEE, verified against the hardware manufacturer's root of trust public key. ## Components ### Verification Interface The frontend provides a simple interface for verifying attestations: 1. Shows the expected code hash representing the verification logic that should have run 2. Displays information about what claim was being verified 3. Provides an input field for the attestation proof 4. Includes a "Verify" button to trigger validation ### Verification Results Upon successful verification, the frontend confirms: 1. Code hash matches the open-source implementation, proving exactly what verification logic ran 2. The specific check or claim being verified was satisfied inside the TEE 3. Hardware authenticity is confirmed through the attestation chain back to the manufacturer ## More Details ### Attestation Structure The attestation object contains: * Measurements of the code executed in the TEE * Details of what was verified in the TLS session * Hardware-signed proof of secure execution * Chain of signatures linking to the hardware manufacturer's root of trust ### Verification Process The frontend performs these checks: 1. Validates the signature chain back to the hardware manufacturer 2. Verifies the code measurements match the expected verification logic 3. Confirms the specific claim or check was satisfied 4. Validates the hardware instance identifiers ## Usage Instructions 1. Navigate to the verification frontend 2. Note the expected code hash that should have performed the verification 3. Review what claim or check was being verified 4. Paste the attestation proof 5. Click "Verify" to validate the attestation 6. Review the results confirming all security properties This system allows anyone to verify that specific checks or claims were legitimately verified within a TEE without having to trust the entity making the claim. The cryptographic attestations prove that the verification occurred in secure hardware running the expected code. This enables trustless verification of user claims while maintaining privacy, since only the fact that a check passed (not the underlying data) is revealed.