Standalone Container

Always make sure you comply with the License Terms before using gecholog.

Dependencies

We distribute our application via a public repository on Docker Hub which makes gecholog easy to deploy to any cloud, or locally on your laptop using Docker Desktop.

To install docker, visit docker.com.

Spin up gecholog

To pull the container, start it locally and run in detached mode:

docker run -d --name gecholog -p 5380:5380 -p 8080:8080 -e GUI_SECRET=changeme -e AISERVICE_API_BASE=https://your.openai.azure.com/ gecholog/gecholog:latest

az container create --resource-group <RESOURCE_GROUP> --name gecholog --image gecholog/gecholog:latest --environment-variables AISERVICE_API_BASE=https://your.openai.azure.com/ --secure-environment-variables GUI_SECRET=changeme --dns-name-label gecholog-changeme --ports 5380 8080

The GUI_SECRET is the password to login to the web interface of gecholog, available on http://localhost:8080/login (replace ´localhost´ with your hostname).

The AISERVICE_API_BASE environment variable is used to set the outbound url of the two standard routers /service/standard/ and /service/capped/. If AISERVICE_API_BASE is left out, gecholog will not add those two routers and only the /echo/ router will be active by default.

NOTE: You can create your own routers and specify your own environment variables via the web interface or in the /app/conf/gl_config.json configuration file of gecholog.

How to apply new configurations?

The gecholog container reads all the configuration files from the /app/conf directory. By modifying these files you control the behavior of gecholog. To accommodate the many different requirements and cloud stacks, gecholog supports multiple ways to deploy the configurations:

1. Modify the configuration files via the web inteface
2. Copy new configuration files to the container
3. Mount a volume with existing configuration files
4. Mount an empty volume (gecholog will copy the default configuration files to that directory)
5. Create your own container image that includes your configuration files

NOTE: All methods are not supported by all cloud technologies

Configurations supported by the web interface

It is possible to update, validate and deploy the gecholog configuration files via the web interface. This can be a quick an easy way change the behaviour of the LLM Gateway.

Currently these configuration files are supported via the web interface:

The web interface provides

1. Ability to modify the configuration files
2. Validation information
3. Ability to deploy & verify deployment of new configuration

Automatic updates

Some of the configuration files are continuously monitored for changes, and will be automatically ingested and applied if valid changes are made. Other configuration files require a restart.

These files are automatically updated if (a) a change is detected and (b) the new configuration file is valid:

NOTE: Automatic updates requires at least version 0.93.1 of gecholog.

Requires restart

For files that require restart, run the following command

docker restart gecholog

az container stop --name gecholog --resource-group <RESOURCE_GROUP>
az container start --name gecholog --resource-group <RESOURCE_GROUP>

How to validate new configuration file before changes

See the section Field Validation with all the details on how to validate your configuration files before uploading/submitting them.

Copy new configuration files to the container

Run this command to copy a local new_gl_config.json to the right path for the container. If the new_gl_config.json is valid, the changes will be applied instantly. Example for the service gl:

docker cp new_gl_config.json gecholog:/app/conf/gl_config.json   

Mounted volume

There are multiple benefits of mounting a local volume or cloud file share to the /app/conf directory of the gecholog container. If you add your configuration files before starting the container, gecholog will use the configuration files on the mounted path at startup. If no configuration files are present in the volume, gecholog will, as part of the startup process, copy the default configuration files to the volume, making it easy to use the default files as a starting point. Starting the container by mounting a volume to the directory /app/conf:

docker run -d --name gecholog -p 5380:5380 -p 8080:8080 -e GUI_SECRET=changeme -e AISERVICE_API_BASE=https://your.openai.azure.com/ -v your/path:/app/conf gecholog/gecholog:latest

az container create --resource-group <RESOURCE_GROUP> --name gecholog --image gecholog/gecholog:latest --azure-file-volume-account-name <STORAGE_ACCOUNT_NAME> --azure-file-volume-account-key <STORAGE_ACCOUNT_KEY> --azure-file-volume-share-name <FILE_SHARE_NAME> --azure-file-volume-mount-path /app/conf --environment-variables AISERVICE_API_BASE=https://your.openai.azure.com/ --secure-environment-variables GUI_SECRET=changeme --dns-name-label gecholog-changeme --ports 5380 8080

The content of the mounted volume will look like this:

ginit_config.json
gl_config.json
gui_config.json
license.json
nats-server.conf
nats2file_config.json
nats2log_config.json
tokencounter_config.json

You can then just edit and save the configuration files, and if the configuration files are valid and monitored for change, the changes will take effect instantly.

How to verify that a new configuration has been applied

gecholog offers a health check command ./healthcheck to be able to verify that new configuration files have been accepted and activated. This command will display the sha256 signature of the configuration file currently in use of the service gl:

docker exec gecholog ./healthcheck -s gl -p

az container exec --resource-group <RESOURCE_GROUP> --name gecholog --container-name gecholog --exec-command "./healthcheck -s gl -p"

And prints

9ac52f58f2c480abf7254657648acde0709e1b37c468e72e49714d4e2bebf1c7

You can verify yourself that the signature displayed from the ./healthcheck command is the same as the sha256 checksum of the configuration file you wanted to activate:

CertUtil -hashfile gl_config.json SHA256

shasum -a 256 gl_config.json

sha256sum gl_config.json

Ignored changes

What happens if gecholog detects a change in one of the configuration files, but the updated configuration file is not valid? gecholog will ignore the change and continue to operate with the settings unchanged. The ./healthcheck command will display the sha256 checksum of the configuration file before the change, since it is still applicable.

Un-healthy services

If the ./healthcheck command return a non-zero exit code or the ./healthcheck -p doesn't print a checksum, the service is not healthy and is not active.

docker exec gecholog ./healthcheck -s gl

az container exec --resource-group <RESOURCE_GROUP> --name gecholog --container-name gecholog --exec-command "./healthcheck -s gl"

Healthy:

echo $?                                                                                                     
0

Not healthy:

echo $?                                                                                                     
1

NOTE: Some un-healthy services will cause the entire container to exit. Read more in the description of ginit in the Section Configuration.

Services that support ./healthcheck

The ./healthcheck command supports the following services:

Create your own container image

See section Custom Dockerfile for a template to create your own Dockerfile with your own configuration files.

Run with access to service bus

If you want to expose the port 4222 for the service bus (for example to develop, test or connect a processor to interact with gecholog) run this command which also sets the NATS_TOKEN which is required to connect an external processor to the container.

docker run -d --name gecholog -p 5380:5380 -p 4222:4222 -p 8080:8080 -e GUI_SECRET=changeme -e AISERVICE_API_BASE=https://your.openai.azure.com/ -e NATS_TOKEN=changeme gecholog/gecholog:latest

az container create --resource-group <RESOURCE_GROUP> --name gecholog --image gecholog/gecholog:latest --environment-variables AISERVICE_API_BASE=https://your.openai.azure.com/ --secure-environment-variables NATS_TOKEN=changeme GUI_SECRET=changeme --dns-name-label gecholog-changeme --ports 5380 8080 4222

Test connection to service bus

Use the nats-cli and run try

nats sub -s changeme@localhost "coburn.gl.logger"

which should print if the token is correct and port successfully open.

16:33:45 Subscribing on coburn.gl.logger 

Services and initial settings

The standard standalone gecholog runs

• The gecholog binary called gl
• The gecholog web interface called gui
• The gecholog process manager called ginit
• nats-server light-weight service bus
• nats2log standard library processor to publish logs to a log store with modes file_writer rest_api_writer elastic_writer azure_log_analytics_writer. Idle by default.
• nats2file is a second nats2log processor with default mode file_writer. Disabled by default by listening to mute coburn.gl subject
• tokencounter standard library processor pre-configured for OpenAI but simple to configure to any LLM-service provider.
• healthcheck service available on request

There are pre-configured routers

• /service/standard/ which routes traffic to AISERVICE_API_BASE with no cap.
• /service/capped/ which routes traffic to AISERVICE_API_BASE but with a cap implemented by processor tokencounter. tokencounter will track and block traffic to this router when total tokens exceed 100 per 120s. These caps can of course be configured in /app/conf/tokencounter_config.json.
• /echo/ router echoes back the payload and headers.
• /restricted/ router that requires GECHOLOG_API_KEY and AISERVICE_API_KEY to be set. It's an example router that enables you to hide the LLM service API key and instead distribute your own static api key.

Stable releases and early access

gecholog is in early access with each release being a stable release. gecholog feature set is still under iteration and from v1.0.1 we will always strive for backwards compatibility or provide necessary migration tools.

Stateless

The gecholog container is stateless, meaning it can be restarted at any moment without interrupting its efficient operation.

NOTE: The tokencounter processor within gecholog is an exception to this statelessness. It maintains a temporary state for the purpose of throttling token traffic, as it keeps track of token consumption per path.