Tokenizer

tokenizer is a processor that applies GPT2Tokenizer to LLM traffic for a generic token count. This can be useful to see how tokens would be calculated differently between LLM services.

  • Routers and search paths for content fields are specified in the config data structure.
  • Both request and response traffic are processed.
  • A tokenizer field is added to the log.

tokenizer demonstrates how to apply a unified token calculation to different LLM models.

Quick Start: Test the tokenizer processor

1. Clone the GitHub repo

git clone https://github.com/direktoren/gecholog_resources.git

2. Set environment variables

# Set the nats token (necessary for mock to connect to gecholog)
setx NATS_TOKEN "changeme"

# Set the gui secret to be able to gecholog web interface
setx GUI_SECRET "changeme"

# Replace this with the url to your LLM API
setx AISERVICE_API_BASE "https://your.openai.azure.com/"

# Set the nats token (necessary for mock to connect to gecholog)
export NATS_TOKEN=changeme

# Set the gui secret to be able to gecholog web interface
export GUI_SECRET=changeme

# Replace this with the url to your LLM API
export AISERVICE_API_BASE=https://your.openai.azure.com/


3. Start gecholog and the tokenizer processor

cd gecholog_resources/processors/tokenizer
docker compose up -d

The Docker Compose command starts and configures the LLM Gateway gecholog and the processor tokenizer. It builds the tokenizer container locally.

NOTE: To take the app down, run docker compose down -v

4. Monitor the logs

nats sub --translate "jq .response.tokenizer" -s "%NATS_TOKEN%@localhost" "coburn.gl.logger"

nats sub --translate "jq .response.tokenizer" -s "$NATS_TOKEN@localhost" "coburn.gl.logger"


5. Make the calls

This example will use Azure OpenAI, but you can use any LLM API service.

setx AISERVICE_API_KEY "your_api_key"              
setx DEPLOYMENT "your_azure_deployment"         

export AISERVICE_API_KEY=your_api_key      
export DEPLOYMENT=your_azure_deployment


Open a new terminal window. Send the request to the /service/standard/ router:

curl -X POST ^
     -H "api-key: %AISERVICE_API_KEY%" ^
     -H "Content-Type: application/json" ^
     -d "{\"messages\": [{\"role\": \"system\",\"content\": \"Assistant is a large language model trained by OpenAI.\"},{\"role\": \"user\",\"content\": \"Who are the founders of Microsoft?\"}],\"max_tokens\": 15}" ^
     http://localhost:5380/service/standard/openai/deployments/%DEPLOYMENT%/chat/completions?api-version=2023-12-01-preview

curl -X POST -H "api-key: $AISERVICE_API_KEY" -H "Content-Type: application/json" -d '{
    "messages": [
      {
        "role": "system",
        "content": "Assistant is a large language model trained by OpenAI."
      },
      {
        "role": "user",
        "content": "Who are the founders of Microsoft?"
      }
    ],
    "max_tokens": 15
  }' "http://localhost:5380/service/standard/openai/deployments/$DEPLOYMENT/chat/completions?api-version=2023-12-01-preview"


Standard response

{
  "id": "chatcmpl-8nZCiOLutrIDeVT94lyXkYzdKtkDe",
  "object": "chat.completion",
  "created": 1706824088,
  "model": "gpt-35-turbo",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The founders of Microsoft are Bill Gates and Paul Allen. They founded Microsoft on"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 29,
    "completion_tokens": 15,
    "total_tokens": 44
  }
}

And see the output from the tokenizer processor in the first terminal window

11:10:57 Subscribing on coburn.gl.logger 
[#1] Received on "coburn.gl.logger"
{
  "gpt2": 14
}

6. Learn about configuring gecholog via web interface

The GUI_SECRET is the password to login to the web interface of gecholog, available on http://localhost:8080/login.

Usage

Start gecholog and tokenizer manually

# Set the nats token (necessary for mock to connect to gecholog)
setx NATS_TOKEN "changeme"

# Set the gui secret to be able to gecholog web interface
setx GUI_SECRET "changeme"

# Replace this with the url to your LLM API
setx AISERVICE_API_BASE "https://your.openai.azure.com/"

# Set the nats token (necessary for mock to connect to gecholog)
export NATS_TOKEN=changeme

# Set the gui secret to be able to gecholog web interface
export GUI_SECRET=changeme

# Replace this with the url to your LLM API
export AISERVICE_API_BASE=https://your.openai.azure.com/


Start and configure the gecholog container

cd gecholog_resources/processors/tokenizer

# Create a docker network
docker network create gecholog

# Spin up gecholog container
docker run -d -p 5380:5380 -p 4222:4222 -p 8080:8080 \
  --network gecholog --name gecholog \
  --env NATS_TOKEN=$NATS_TOKEN \
  --env GUI_SECRET=$GUI_SECRET \
  --env AISERVICE_API_BASE=$AISERVICE_API_BASE \
  gecholog/gecholog:latest

# Copy the gl_config to gecholog (if valid it will be applied directly)
docker cp gl_config.json gecholog:/app/conf/gl_config.json

Optional: Check that the config file is applied (both statements should produce the same checksum)

docker exec gecholog ./healthcheck -s gl -p

versus

CertUtil -hashfile gl_config.json SHA256

shasum -a 256 gl_config.json

sha256sum gl_config.json


Continue with the tokenizer processor container

# Build the processor container
docker build --no-cache -f Dockerfile -t tokenizer .

# Start the processor container
docker run -d \
        --network gecholog --name tokenizer \
        --env NATS_TOKEN=$NATS_TOKEN \
        --env GECHOLOG_HOST=gecholog \
        tokenizer