🛠️ DevLog – Gateway Capacity Follow-Up: Next Focus Is Multi-Model Routing A quick follow-up on the recent API Gateway capacity / reliability work. 🔹 Current progress We’ve now added the first minimal layer of capacity-aware tracking / load balancing on the API Gateway side so the hosted path behaves a bit more safely under shared load. 🔹 What comes next From here, we’ll shift gears a bit and start reconfiguring the existing sessions so they can support more than just the current oss-20b path. The goal is to test: - model routing at the API Gateway level - how the gateway behaves when multiple model paths exist underneath - whether the current hosted flow still routes cleanly once the model mix becomes broader 🔹 Why this matters So the next step is not only “protect the gateway under load,” but also make sure the gateway can route correctly once the backend is no longer centered around just one model/session path. 🔹 Current takeaway The first capacity/reliability layer is in place. Next, we use that foundation to test the multi-model routing side more directly. #Cortensor #DevLog #Portal #APIGateway #Routing #Reliability

🛠️ DevLog – Gateway Capacity Follow-Up: Next Focus Is Multi-Model Routing A quick follow-up on the recent API Gateway capacity / reliability work. 🔹 Current progress We’ve now added the first minimal layer of capacity-aware tracking / load balancing on the API Gateway side so the hosted path behaves a bit more safely under shared load. 🔹 What comes next From here, we’ll shift gears a bit and start reconfiguring the existing sessions so they can support more than just the current oss-20b path. The goal is to test: - model routing at the API Gateway level - how the gateway behaves when multiple model paths exist underneath - whether the current hosted flow still routes cleanly once the model mix becomes broader 🔹 Why this matters So the next step is not only “protect the gateway under load,” but also make sure the gateway can route correctly once the backend is no longer centered around just one model/session path. 🔹 Current takeaway The first capacity/reliability layer is in place. Next, we use that foundation to test the multi-model routing side more directly. #Cortensor #DevLog #Portal #APIGateway #Routing #Reliability

🛠️ DevLog – Gateway Capacity Tracking Is Now in Testing A quick update on the capacity-aware gateway work from the last devlog. 🔹 Current progress The first version is now implemented and being tested across the hosted API path. The gateway can now coordinate shared router-session usage through Redis, so multiple gateway instances have better awareness of which sessions are already busy before routing more traffic. 🔹 What changed Core gateway updates: - Redis-backed session capacity tracking - per-session concurrency limits - existing routing flow preserved - capacity/runtime status exposed through management endpoints Ops UI updates: - admin view for capacity mode, tracked sessions, lease TTL, and Redis state - dedicated Sessions view for live Cortensor session usage - visibility into live occupancy, recent traffic, labels, latency, token usage, and gateway hit distribution 🔹 Why this matters This helps the hosted API path handle bursts more safely. With multiple gateway replicas, traffic can now be coordinated against the same shared session pool instead of blindly overloading the same router session. This gives us: - better load protection - clearer real-time session visibility - easier debugging during stress tests - more predictable hosted inference traffic distribution 🔹 What comes next This is not full queue-based admission control yet. Next, we’ll also be looking into the queue layer so overload behavior is safer when sessions are busy or all sessions are at capacity. The goal is: - stronger backpressure - clearer overload handling - more predictable burst behavior #Cortensor #AIInfrastructure #DevLog #Inference #Gateway #DecentralizedAI

🛠️ DevLog – Portal Stress Tests Are Now Starting A quick follow-up on the Portal V1 stress-testing path. 🔹 Current progress We’ve now started running the first round of: - parallel API calls - repeated request patterns - broader hosted-path checks through the Portal API Gateway flow 🔹 What this means This is the next step beyond simple single-request validation. The goal is to see how the hosted path behaves under: - more realistic usage - concurrent requests - multi-account / multi-key activity - longer-running API calls 🔹 What’s next We’ll keep running more of these tests today and use them to surface: - weaker points in the hosted path - routing / pool behavior under load - request/accounting consistency - areas that still need hardening before broader usage 🔹 Current direction So this stress-test phase is now in motion, and it should give us a clearer signal on how the Portal path behaves under more realistic usage patterns. #Cortensor #DevLog #Portal #Gateway #API #StressTest

🛠️ DevLog – Next Step on Portal API Gateway Compatibility: SSE / Streaming As the next step toward fuller Portal API Gateway compatibility with OpenAI and Anthropic-style APIs, we’ll be looking more closely at the SSE / streaming path. 🔹 Current direction The first step is to assess what we can make work cleanly at the API Gateway layer itself and see how far that can go with the current Cortensor Router node stream/SSE behavior. 🔹 What we want to test The main goal is to make sure the stream path can behave properly across: - client - API Gateway entry point - API Gateway instance - router pool - actual Cortensor Router node 🔹 Why this matters Streaming is not just another response mode. It has to work cleanly across multiple layers if we want better compatibility with existing OpenAI/Anthropic-style clients. 🔹 What comes after that Once we test the current path more directly, we should be able to surface the real remaining gaps more clearly - including where we may need to update the Cortensor Router node protocol itself to support streaming more cleanly end to end. #Cortensor #DevLog #Portal #APIGateway #SSE #Streaming

🛠️ DevLog – Portal API Gateway Compatibility Progress A quick progress update on the Portal API Gateway compatibility layer, focused on what has actually been tested so far. 🔹 Tested and working The following are now implemented, deployed, and tested: - GET /v1/models - POST /v1/chat/completions for basic non-stream text requests - POST /v1/messages for basic non-stream text requests - OpenAI-style Bearer auth - Anthropic-style x-api-key auth We also confirmed that provider-shaped error handling is working live: - OpenAI-style errors on /v1/chat/completions - Anthropic-style errors on /v1/messages And unsupported-feature handling is behaving as expected: - stream=true returns the expected OpenAI-shaped error - non-text Anthropic content returns the expected Anthropic-shaped error 🔹 Working in tests, but not yet fully proven live The success-response mapping for: - OpenAI chat completions - Anthropic messages is working in tests, but we have not yet cleanly proven the success-shape adapter against live prod inference. 🔹 Stream is a deeper area Streaming is one of the bigger remaining areas we still need to look at more carefully. To do streaming properly, we need the SSE/stream path to behave cleanly across the full chain: - client - API Gateway entry point - API Gateway instance - router pool - actual router That means this is not just one missing flag. It is a multi-layer stream/chunking problem, and we’ll need to validate that path more deeply before calling streaming support done. 🔹 Current blocker The main blocker right now is live router/session health on the inference path, and we’re continuing to check those session-side issues. 🔹 Current takeaway So the current state is: - the basic compatibility layer is tested and working - auth and error-shaping are in place - unsupported-feature behavior is behaving correctly - live end-to-end success mapping still needs cleaner inference health underneath - streaming is a deeper next area because it needs to work cleanly across every layer in the hosted path We’ll keep iterating from here. #Cortensor #DevLog #Portal #APIGateway #DeveloperExperience

🛠️ DevLog – Portal API Gateway Compatibility Layer Is Being Tested Today A quick follow-up on the Portal API Gateway compatibility work. 🔹 Current status We’re testing this path more directly today now that the basic compatibility layer is in place. 🔹 What we’re checking The current focus is the public-facing compatibility path for: - GET /v1/models - POST /v1/chat/completions - POST /v1/messages So this is mainly about making sure the Portal API Gateway can accept the more familiar provider-shaped request flow and translate it cleanly into the current Cortensor router path. 🔹 Current scope Right now, this is still focused on the basic non-streaming, text-only path. 🔹 Why this matters If this holds up cleanly, it makes the Portal API Gateway easier to integrate with from existing clients without forcing an immediate rewrite around a custom request shape. We’ll keep iterating from here. #Cortensor #DevLog #Portal #APIGateway #DeveloperExperience

🛠️ DevLog – Portal API Gateway Compatibility Layer Is Now in Place A quick update on the public-facing Portal API Gateway side. 🔹 What changed We added a compatibility layer in the Portal API Gateway so existing clients can use Cortensor through more familiar provider-shaped REST routes without requiring router changes first. The Portal API Gateway now supports: - GET /v1/models - POST /v1/chat/completions - POST /v1/messages It can translate OpenAI-style and Anthropic-style text requests into the current Cortensor router completion path, then return provider-shaped success/error responses back to the client. We also added: - x-api-key support for Anthropic-style auth - compatibility while keeping the older custom POST /v1/completions route intact - updated docs/README - tests, with pytest and ruff passing 🔹 Why this matters This is important because it lowers adoption friction. Instead of asking developers to immediately rewrite everything around a custom Cortensor request shape, the Portal API Gateway can meet existing clients closer to where they already are. That helps with: - easier integration - faster testing - less client-side rewrite work - a more familiar first experience for external developers So this is less about “copying another API” and more about making Cortensor easier to plug into existing workflows through the Portal API Gateway. 🔹 Current scope Right now, this compatibility layer is still focused on the basic non-streaming, text-only REST path. 🔹 What’s still pending Full provider compatibility is not done yet. Still pending: - streaming support for OpenAI chat and Anthropic messages - checking whether the router’s existing SSE stream adapts cleanly enough - tool/function calling support - multimodal content support - more parity details around stop sequences, usage nuances, and possibly newer response-style routes 🔹 Current takeaway So the current state is: - basic non-stream REST compatibility is now in place in the Portal API Gateway - it already makes public integration easier - but there is still more work before we can call it broader provider compatibility #Cortensor #DevLog #Portal #APIGateway #DeveloperExperience

🛠️ DevLog – Dual API Gateway Path Is Now Added for Portal Testing A quick follow-up on the Portal reliability path. 🔹 What changed We’ve now added the first dual API Gateway setup for Portal: - one entry path - two API Gateway instances behind it - load-balanced in a similar way to how the router pool works underneath 🔹 Why this matters This is one of the cleaner reliability improvements for the hosted path: - if one API Gateway instance has an issue, another can still serve requests - deploys/restarts should be less disruptive - the Portal entry path becomes less dependent on one single gateway instance 🔹 Current shape So the rough flow now becomes: API request → one entry path → two API Gateway instances → router pool → router node 🔹 Current status This is now added in rough form, and we’ll test it later today. #Cortensor #DevLog #Portal #Gateway #Reliability #API