3. Test your configuration

This page demonstrates how to test the routing of gechoLog requests to either Azure OpenAI or OpenAI. However, gechoLog is designed to be compatible with any LLM-API provider. If you're interested in more tutorials, please provide feedback by registering through our Sign-up link and submitting your suggestions to our product team.

This section assumes you have set the environment variable AISERVICE_API_BASE to your real Azure OpenAI or OpenAI endpoint before running docker-compose up command. If not, you can either rebuild gecholog-ek-dev again or follow the tutorial here on how do update the /app/conf/gl_config.json configuration file.

cUrl to azure openai

Let's make a first request using cUrl to Azure OpenAI via gecholog. Set the azure OPENAI_API_KEY and DEPLOYMENT as environment variables:

setx OPENAI_API_KEY "your_api_key"              
setx DEPLOYMENT "your_azure_deployment"         

export OPENAI_API_KEY=your_api_key      
export DEPLOYMENT=your_azure_deployment       

Make the request

curl -X POST ^
     -H "api-key: %OPENAI_API_KEY%" ^
     -H "Content-Type: application/json" ^
     -d "{\"messages\": [{\"role\": \"system\",\"content\": \"Assistant is a large language model trained by OpenAI.\"},{\"role\": \"user\",\"content\": \"Who are the founders of Microsoft?\"}],\"max_tokens\": 15}" ^
     http://localhost:5380/service/standard/openai/deployments/%DEPLOYMENT%/chat/completions?api-version=2023-12-01-preview

curl -X POST -H "api-key: $OPENAI_API_KEY" -H "Content-Type: application/json" -d '{
    "messages": [
      {
        "role": "system",
        "content": "Assistant is a large language model trained by OpenAI."
      },
      {
        "role": "user",
        "content": "Who are the founders of Microsoft?"
      }
    ],
    "max_tokens": 15
  }' "http://localhost:5380/service/standard/openai/deployments/$DEPLOYMENT/chat/completions?api-version=2023-12-01-preview"

And receive a response like

{
  "id": "chatcmpl-8GQ88W04Yx6Z6kCYLbnf2NzTXm2WO",
  "object": "chat.completion",
  "created": 1698924384,
  "model": "gpt-4",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "\"Bill Gates and Paul Allen.\""
      }
    }
  ],
  "usage": {
    "prompt_tokens": 37,
    "completion_tokens": 7,
    "total_tokens": 44
  }
}

Congratulations! You have now routed your first Azure OpenAI request via gecholog. The request is sent via the /service/standard/ router in gecholog.

NOTE: In the default gecholog configuration, the api-key header is included in the masked_headers group. This means that its value will be obfuscated in the logs, with value shown as [*****MASKED*****]. For more information on how masked headers work, check out the Headers section.

Python openai library

gecholog is designed to require minimal code changes on the application side. For instance, you can continue using the standard OpenAI library for python, even with gecholog deployed.

Set the AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT as environment variables.

setx AZURE_OPENAI_API_KEY "your_api_key"                                   
setx AZURE_OPENAI_ENDPOINT "http://localhost:5380/service/standard"  # Without end /      

export AZURE_OPENAI_API_KEY=your_api_key
export AZURE_OPENAI_ENDPOINT=http://localhost:5380/service/standard  # Without end /

A standard python example (just replace the your_deployment part)

import os 
import openai

openai.api_key = os.getenv("AZURE_OPENAI_API_KEY")
openai.api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
openai.api_type = "azure"
openai.api_version = "2023-12-01-preview" # this might change in the future

response = openai.ChatCompletion.create(
    engine="your_deployment", # The deployment name you chose when you deployed a model
    messages=[
        {"role": "system", "content": "Assistant is a large language model trained by OpenAI."},
        {"role": "user", "content": "Who are the founders of Toyota?"}
    ],
    max_tokens=15
)

print(response)

import os 
from openai import AzureOpenAI

client = AzureOpenAI(
  api_key=os.getenv("AZURE_OPENAI_API_KEY"),  
  api_version="2023-12-01-preview", # this might change in the future
  azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
)

response = client.chat.completions.create(
    model="your_deployment", # model = "deployment_name"
    messages=[
        {"role":"system","content":"Assistant is a large language model trained by OpenAI."},
        {"role":"user","content":"Who are the founders of Toyota?"}
    ],
    temperature=0.7,
    max_tokens=15
)

print(response)

Store it as test.py and run it with

python test.py

And it should print something like

ChatCompletion(id='chatcmpl-8mOd2bJMFVEjrOcWu4gRjHf23zu4s', choices=[Choice(finish_reason='length', index=0, logprobs=None, message=ChatCompletionMessage(content='Toyota Motor Corporation was founded by Kiichiro Toyoda. It was established', role='assistant', function_call=None, tool_calls=None))], created=1706545108, model='gpt-4', object='chat.completion', system_fingerprint='fp_6d044fb900', usage=CompletionUsage(completion_tokens=15, prompt_tokens=29, total_tokens=44))

Congratulations, the Hello World tests are now officially complete. You're ready to begin routing your traffic through gecholog. All you need to change in your code is the target URL!