Content Filter

The contentfilter custom processor demonstrates how to create your own filter in front of the LLM API to reject requests based on specific rules or classifications. Routers, patterns, and logic for the filter can be modified. The default behavior is as follows:

• The field messages[1].content of requests to /service/standard/ is classified using the detoxify library.
• If any of the detoxify classification parameters are greater than THRESHOLD, the request will be rejected with a rejection response.
• Otherwise, it will be forwarded to the LLM API.

The THRESHOLD parameter can be adjusted through an environment variable.

Disclaimer

It is recommended to run the contentfilter locally to explore how it works instead of building a container image. It's a considerable amount of dependencies needed, so for production purpose this would need to be optimized.

When building a contentfilter Docker image, the process is slow, and therefor instructions below for docker are recommended only after trying running it locally first. The contenfilter container image becomes really big (>8gb).

Quick Start: Explore a Custom Content Filter

1. Clone the GitHub repo

git clone https://github.com/direktoren/gecholog_resources.git

2. Set environment variables

# Set the nats token (necessary for broker to connect to gecholog)
setx NATS_TOKEN "changeme"

# Set the gui secret to be able to gecholog web interface
setx GUI_SECRET "changeme"

# Replace this with the url to your LLM API
setx AISERVICE_API_BASE "https://your.openai.azure.com/"

# Set the nats token (necessary for broker to connect to gecholog)
export NATS_TOKEN=changeme

# Set the gui secret to be able to gecholog web interface
export GUI_SECRET=changeme

# Replace this with the url to your LLM API
export AISERVICE_API_BASE=https://your.openai.azure.com/

3. Download the contentfilter dependencies

cd gecholog_resources/processors/contentfilter
pip install -r requirements.txt

Enjoy a warm beverage of your choice whilst waiting.

4. Start the gecholog LLM Gateway

# Create a docker network
docker network create gecholog

# Spin up gecholog container
docker run -d -p 5380:5380 -p 4222:4222 -p 8080:8080 \
  --network gecholog --name gecholog \
  --env NATS_TOKEN=$NATS_TOKEN \
  --env GUI_SECRET=$GUI_SECRET \
  --env AISERVICE_API_BASE=$AISERVICE_API_BASE \
  gecholog/gecholog:latest

# Copy the gl_config to gecholog (if valid it will be applied directly)
docker cp gl_config.json gecholog:/app/conf/gl_config.json

Optional: Check that the config file is applied (both statements should produce the same checksum)

docker exec gecholog ./healthcheck -s gl -p

versus

CertUtil -hashfile gl_config.json SHA256

shasum -a 256 gl_config.json

sha256sum gl_config.json

5. Run contentfilter

python contentfilter.py

When ready, the output is

Connected to NATS server!

6. Make the calls

This example will use Azure OpenAI, but you can use any LLM API service.

setx AISERVICE_API_KEY "your_api_key"              
setx DEPLOYMENT "your_azure_deployment"         

export AISERVICE_API_KEY=your_api_key      
export DEPLOYMENT=your_azure_deployment       

Send the request to the /service/standard/ router:

curl -X POST ^
     -H "api-key: %AISERVICE_API_KEY%" ^
     -H "Content-Type: application/json" ^
     -d "{\"messages\": [{\"role\": \"system\",\"content\": \"Assistant is a large language model trained by OpenAI.\"},{\"role\": \"user\",\"content\": \"Im sick of this nonsense!\"}],\"max_tokens\": 15}" ^
     http://localhost:5380/service/standard/openai/deployments/%DEPLOYMENT%/chat/completions?api-version=2023-12-01-preview

curl -X POST -H "api-key: $AISERVICE_API_KEY" -H "Content-Type: application/json" -d '{
    "messages": [
      {
        "role": "system",
        "content": "Assistant is a large language model trained by OpenAI."
      },
      {
        "role": "user",
        "content": "Im sick of this nonsense!"
      }
    ],
    "max_tokens": 15
  }' "http://localhost:5380/service/standard/openai/deployments/$DEPLOYMENT/chat/completions?api-version=2023-12-01-preview"

And receive the response

{
  "is_toxic": true
}

Changing Sensitivity

contentfilter uses the THRESHOLD environment variable to adjust the sensitivity of the filter.

contentfilter container

The container for contentfilter is slow to build and the image is very large. But if you want to try it anyway run

cd gecholog_resources/processors/contentfilter

# Build the processor container (this one is a little heavy...)
docker build --no-cache -f Dockerfile -t contentfilter .

# Start the processor container
docker run -d \
    --network gecholog --name contentfilter \
    --env NATS_TOKEN=$NATS_TOKEN \
    --env GECHOLOG_HOST=gecholog \
    --env THRESHOLD=0.5 \
    contentfilter

Monitor logs in realtime

You can connect to the service bus of gecholog container to see the logs from the api calls.

This command will display the content_filter classification field that contentfilter creates.

nats sub --translate "jq .request.content_filter" -s "%NATS_TOKEN%@localhost" "coburn.gl.logger"

nats sub --translate "jq .request.content_filter" -s "$NATS_TOKEN@localhost" "coburn.gl.logger"

Example

16:41:43 Subscribing on coburn.gl.logger 
[#1] Received on "coburn.gl.logger"
{
  "toxicity": 0.7701785564422607,
  "severe_toxicity": 0.0008779675699770451,
  "obscene": 0.016887160018086433,
  "threat": 0.001066152355633676,
  "insult": 0.03343857452273369,
  "identity_attack": 0.0014626976335421205
}

7. Learn about configuring gecholog via web interface

The GUI_SECRET is the password to login to the web interface of gecholog, available on http://localhost:8080/login.