Character Count
The charactercount
custom processor is a simple template processor that can be used to build your own custom processors. charactercount
demonstrates how to accept an incoming message to a processor and send it back to gecholog
.
- Receives field
ingress_payload
- Sends back field
character_count
Quick Start: Test Charactercount
1. Clone the GitHub repo
git clone https://github.com/direktoren/gecholog_resources.git
2. Set environment variables
# Set the nats token (necessary for broker to connect to gecholog)
setx NATS_TOKEN "changeme"
# Set the gui secret to be able to gecholog web interface
setx GUI_SECRET "changeme"
# Replace this with the url to your LLM API
setx AISERVICE_API_BASE "https://your.openai.azure.com/"
# Set the nats token (necessary for broker to connect to gecholog)
export NATS_TOKEN=changeme
# Set the gui secret to be able to gecholog web interface
export GUI_SECRET=changeme
# Replace this with the url to your LLM API
export AISERVICE_API_BASE=https://your.openai.azure.com/
3. Start gecholog
and the charactercount
processor
cd gecholog_resources/processors/charactercount
docker compose up -d
The Docker Compose command starts and configures the LLM Gateway gecholog
and the processor charactercount
. It builds the charactercount
container locally.
NOTE: To take the app down, run
docker compose down -v
4. Monitor the logs
nats sub --translate "jq .request.character_count" -s "%NATS_TOKEN%@localhost" "coburn.gl.logger"
nats sub --translate "jq .request.character_count" -s "$NATS_TOKEN@localhost" "coburn.gl.logger"
5. Make the calls
This example will use Azure OpenAI, but you can use any LLM API service.
setx AISERVICE_API_KEY "your_api_key"
setx DEPLOYMENT "your_azure_deployment"
export AISERVICE_API_KEY=your_api_key
export DEPLOYMENT=your_azure_deployment
Open a new terminal window. Send the request to the /service/standard/
router:
curl -X POST ^
-H "api-key: %AISERVICE_API_KEY%" ^
-H "Content-Type: application/json" ^
-d "{\"messages\": [{\"role\": \"system\",\"content\": \"Assistant is a large language model trained by OpenAI.\"},{\"role\": \"user\",\"content\": \"Who are the founders of Microsoft?\"}],\"max_tokens\": 15}" ^
http://localhost:5380/service/standard/openai/deployments/%DEPLOYMENT%/chat/completions?api-version=2023-12-01-preview
curl -X POST -H "api-key: $AISERVICE_API_KEY" -H "Content-Type: application/json" -d '{
"messages": [
{
"role": "system",
"content": "Assistant is a large language model trained by OpenAI."
},
{
"role": "user",
"content": "Who are the founders of Microsoft?"
}
],
"max_tokens": 15
}' "http://localhost:5380/service/standard/openai/deployments/$DEPLOYMENT/chat/completions?api-version=2023-12-01-preview"
Standard response
{
"id": "chatcmpl-8nZCiOLutrIDeVT94lyXkYzdKtkDe",
"object": "chat.completion",
"created": 1706824088,
"model": "gpt-35-turbo",
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"role": "assistant",
"content": "The founders of Microsoft are Bill Gates and Paul Allen. They founded Microsoft on"
}
}
],
"usage": {
"prompt_tokens": 29,
"completion_tokens": 15,
"total_tokens": 44
}
}
And see the output from the charactercount
processor in the first terminal window
11:10:57 Subscribing on coburn.gl.logger
[#1] Received on "coburn.gl.logger"
179
6. Learn about configuring gecholog
via web interface
The GUI_SECRET
is the password to login to the web interface of gecholog, available on http://localhost:8080/login.
Usage
Start gecholog
and charactercount
manually
# Set the nats token (necessary for mock to connect to gecholog)
setx NATS_TOKEN "changeme"
# Set the gui secret to be able to gecholog web interface
setx GUI_SECRET "changeme"
# Replace this with the url to your LLM API
setx AISERVICE_API_BASE "https://your.openai.azure.com/"
# Set the nats token (necessary for mock to connect to gecholog)
export NATS_TOKEN=changeme
# Set the gui secret to be able to gecholog web interface
export GUI_SECRET=changeme
# Replace this with the url to your LLM API
export AISERVICE_API_BASE=https://your.openai.azure.com/
Start and configure the gecholog
container
cd gecholog_resources/processors/charactercount
# Create a docker network
docker network create gecholog
# Spin up gecholog container
docker run -d -p 5380:5380 -p 4222:4222 -p 8080:8080 \
--network gecholog --name gecholog \
--env NATS_TOKEN=$NATS_TOKEN \
--env GUI_SECRET=$GUI_SECRET \
--env AISERVICE_API_BASE=$AISERVICE_API_BASE \
gecholog/gecholog:latest
# Copy the gl_config to gecholog (if valid it will be applied directly)
docker cp gl_config.json gecholog:/app/conf/gl_config.json
Optional: Check that the config file is applied (both statements should produce the same checksum)
docker exec gecholog ./healthcheck -s gl -p
versus
CertUtil -hashfile gl_config.json SHA256
shasum -a 256 gl_config.json
sha256sum gl_config.json
Continue with the charactercount
processor container
# Build the processor container
docker build --no-cache -f Dockerfile -t charactercount .
# Start the processor container
docker run -d \
--network gecholog --name charactercount \
--env NATS_TOKEN=$NATS_TOKEN \
--env GECHOLOG_HOST=gecholog \
charactercount